Mobile Game Testing | Why testing ‘Pokémon Go’ is different?

Mobile gaming is booming, and that’s awesome! A billion dollar industry with constant new efforts to capitalize on that. Mobile game segment is fiercely competed. The app store is filled with new games daily and it takes more than just addicting puzzles and complex levels to attain and keep users interested. There is no limit to the choices that a gamer can make. Any deficiency in smooth play, performance, graphics, logic, compatibility or the elusive “fun factor” will jeopardize its commercial success. Users too have a short attention span – if your game glitches, is buggy, doesn’t load, kicks players out, crashes or fails to save a high score, users will walk away angry, never come back and might even warn their friends not to bother. In order to compete and thrive in the market, you should take up the struggle of delivering the best possible quality products, i.e. Mobile Game Testing!

According to Forbes, 80% of smartphone owners have played a game on their device, while 46% play on a daily basis. The report further states that 65% of total online time is spent on either a smartphone or a tablet, with 32% of that time allocated to games.

When hearing about mobile game testing, one of the immediate questions that come to mind is, why is there such a distinction? How is mobile game testing different from a regular mobile application testing? Let’s see at some Insights and knowledge when it comes to Mobile Game testing.

‘Mobile Games’ don’t get a second chance

Unlike console-based video games, mobile game developers are often not given extended periods of testing. Users will not be likely to download a newer version of a game with less bugs after they have already deleted it. The first impression they have of the app is the place you hold in their mind. Without the proper amount of Mobile Game testing or QA, a great game may never get the popularity it deserves.

How Mobile Game Testing is different from App Testing?

When mobile technology and gaming converge, you get an entirely new space that requires a very unique and focused testing effort. Mobile game testing is typically very different compared to the regular mobile app testing. Unlike most other mobile applications, games are driven by their user experience rather than their functionality.

Gaming industry is growing exponentially due to increase in apps usage in mobile and tablets. A part of what makes the mobile game testing space so unique for gaming is that games can range from simple puzzles to elaborate first-person shooters. You wouldn’t test and evaluate Angry Birds the same way you would test Max Payne or NFS. It is hard to define a standard strategy and testing methodologies need to be tailored as each game differs from the other. To start with, aspects like Functionality, Usability, Multi-player functionality, Regression, Endurance, Compatibility, Localization / Internationalization, Performance, Content Testing, Hardware and Recovery Scenarios have to be analyzed.

Herein lies why Mobile Game testing possesses significant differences compared to most App testing. There are many unique aspects that make mobile game testing truly challenging – I tried to compile a list of types of testing that makes game testing especially troublesome.

Functional testing

This is undoubtedly the most common form of mobile game testing performed. Typically functional testing is associated with manual testing and playing ‘game through’. Automation is also a viable possibility in several areas of functional testing.

User Interface

The first impression! The User Interface has a direct impact on how successful your game will be. User Interface, including visual appeal and the game play are the most important thing to get right – and ensure device fragmentation doesn’t break any of these. Even when your game upholds a great idea with effective responses, if it fails to deliver a clean and sturdy UI, the game will have a hard time hitting the mark. In the UI various things needs to be tested – UI layouts and elements, Menu structure and functions, Screen orientation & Screen resolution.

Usability and User Experience (Good Entertainment)

Are the visual appearance and layout of the program correct, appealing and aid ease-of-use? Does your app have a command line feel to it or is it appealing to even the most non-technical of users? Other usability factors you need to consider on mobile games intended for both smartphones and tablets are swiping, tilting, button placement, and proportions. Testing usability, navigation flow, and eventually what user experience your game delivers to gamer simply cannot be ignored.

Performance testing

The key driver for user experience! Before you publish your game in Google Play or App Store, you should have run different types of performance tests for your game on all possible device configurations. What type of performance testing is needed varies on the nature of game, graphics, game play, multiplayer functionalities and many other things. Typical metrics from performance tests are CPU load, memory consumption and frames per second (FPS). Or the general stress tests to determine how the game performs in terms of responsiveness, stability and refresh rate under certain workload or user interaction pattern.

Graphics Performance

Does the game display show properly on all the devices? Do the UI and layout and other graphical elements (toolbars, buttons, icons etc.) display correctly? Animations? Remember! Animations require some performance capability from the device. It is important to test this in the context of low-end devices. Does your game support all possible screen resolutions and is it tested on real devices? What happens when user rotates the screen (Screen Orientation)?

Good graphics performance of any mobile game is tightly related to good user experience. Stunning graphics, pictures, animations and all those beautiful effects will make the mobile game shine. Performance of Game on Real Device needs to be consistent across all different device variants that your game’s users have. It can help game developers to understand those potential bottlenecks in performance, reveal unused potential with CPU and GPU resources, and help to push the boundaries of gaming and visuals, without forgetting the most important thing – a need to test the game on real devices.

Image Recognition

Despite of what game engine they have selected, over 90% of game developers use some form of image recognition. As many graphics cannot be recognized as control mechanism – buttons, menus, sliders etc. – image recognition helps to identify those that are meant for controlling (and not just as a graphical element). The test script should recognize graphic elements and should be able to access that. After that, test script need to interact with these graphical elements and figure out a way to validate if the view has changed after any interaction.

Multi-player Features

Nowadays, numerous games are multi-player oriented and includes features that promote team play. Multiplayer mobile game testing is a whole other beast in itself. Many players simultaneously interacting with the game, with computer-controlled opponents, with game servers, with each other. So many things that can go wrong. And it often requires a whole team of testers. Many difficult risk-based decisions to make if you don’t want to spend unlimited amounts of time testing different scenarios. Understanding of multiplayer game design, and how to test efficiently as a team is required for this type of testing.

Compatibility testing

Because of game complexity and the need to run on a myriad of devices and OSs, compatibility testing is of paramount importance. Early in development, software emulators representing various mobile devices can be used, but eventually, extensive tests on a variety of real, released mobile devices must be performed to ensure the game runs well everywhere.

Load testing

No matter how many players log in on launch day, the game needs to perform. Load Tests validate the limits of a system, such as the number of players on a server, the graphic content on the screen (e.g. frames per second, FPS), memory consumption (allocation and deallocation of it), Basic server upload/download function, database connection and queries, or different types of inputs given to the game. Furthermore, load testing can help you understand how game needs to ‘compromise’ when certain operations cannot be handled. Load testing can be built seamlessly in test automation context and heavy loads of data, usage or interactions can be simulated with test scripts and real content from backend servers.

Resource Utilization

Most mobile device users are pushing their memory limits to the nth degree on any given day. How will your app impact device memory? What impact will your app have on the battery life? Many application binaries are massive (up to 3Gb) and games consume a lot of memory, CPU, GPU and battery – all this need to be accounted for during mobile game testing so that the real end user experience does not suffer due to physical realities of mobile devices. The last thing you want your app to be associated with is making your user’s device so slow because it is a tremendous resource hog.

Sensors Usage

The majority of games utilize sensors or other hardware features that are not consistent from one device to another and if this is left unchecked there will be nasty surprises down the road. You can imagine what an inconsistent acceleration sensor would do to all games that rely on gesture recognition based on the data from acceleration sensor!

AI Tests

Most mobile apps do not employ artificial intelligence. But beyond the simplest games, it is an essential ingredient. Testing that computer-controlled opponents are working correctly, that the artificial intelligence is behaving according to design, can also become very difficult as the complexity of the behavior increases. Chess is a good basic example, and enemies in a first person shooter is a more modern one.  This type of mobile game testing requires the tester to understand what triggers different types of behavior, and how these triggers can be confused by different parameters. Understanding of AI and game design is critical to succeed in this field.

Localization testing

Highly important mobile game testing when your game is targeted for the global markets. The word ‘global’ means your game needs to be ‘local’ for everyone. Thorough tests needs to be conducted in all supported languages to ensure that the game can be played through. The process involves translation and verification of text and voice, local naming conventions and testing of user documentation.

Network Tests

Will your app be a data hog that causes users to watch their bandwidth usage as a result? How does your app perform on various carrier networks as opposed to WiFi? If a game is played with a backend server or played with others across various types and speeds of networks, this makes a huge impact on performance requirements that must be tested.

Social Integration

Social network integration is also very important part of Mobile games. In many games, it is essential to be able to share results across your ecosystem, friends or just for yourself. This should be thoroughly tested in the context of real Android and iOS devices, with different OS versions and different device configurations, to understand the functionality and easiness-of-use.

Security and Liabilities

Nearly all game developers use some open source components as part of their game. This is highly recommended and well-accepted development practice as it offloads the task of developing code for non-core functions in your mobile game. But these codes can compromise security and allow data breach. As part of your mobile game testing, identification and rectification of open source component vulnerabilities and license restrictions are necessary for a risk-free gaming experience.

Audio Testing

Audio testing is common in all software that creates some kind of sound or plays media. However Game music has to involve the user in the game and enhance the game play. Not only should the audio play without stuttering or missing elements, it should also add to the game play. This requires extensive audio skills and specific understanding of game audio. Very specific expert domain knowledge.

Physics Testing

Many modern 3D games have physics engines. Modern shooters such as Battlefield or Crisis and RPGs such as Skyrim, have destructible environment and possibilities to throw objects. Testing if the physics engine works requires an understanding of physics as well as how to implement that in a good way into a game. The complexity of the mobile game testing grows with the complexity of the engine.

Regression testing

Every time you make a change in your mobile game, another type of mobile game testing needs to be done, namely regression testing. Regression testing is really important where a client-server interaction is present and continuous changes are put into effect. Developers must confirm that these changes don’t interfere with the normal working of the game. Automated testing is really the only option in this case, manual doesn’t cut it.

Beta Tests

Real players aren’t QA professionals in a lab setting. They’re just ordinary people—and their experience and feedback could be the key to your success. People play games voluntarily, which means that they are really sensitive to negative user experiences. Beta tests allow you to get feedback from real players on not only their enjoyment and engagement, but also about Animations, Visual appeal, Playability on different devices, Menu navigation and more…

Realism tests

With the progress in various mobile device performance delivered by high power processors and better quality graphics hardware, video games now tends towards being more of real-world representations. Apart from graphics quality, realism is achieved by providing the right perceptional approximations, cause and effect relationship, right emotional expressions to characters etc.

Humans are very good at spotting inconsistencies within games related to how realistic are the physics of objects, motion the interactions thereof, especially in simulators or racing games. Game realism is constantly increasing, which further explodes the number of test cases. If a game does not “feel” right, it may impact ease-of-use. Test automation frameworks are without a doubt indispensable, but there is still no complete substitute for live black-box testing by expert players. This type of mobile game testing often requires specific domain expertise, for e.g. weapon techniques, component specific operations, vehicle controls, animal behaviors and instincts etc.

Fun Factor Testing

Fun Factor Testing is something unique to games, since they are an entertainment product. Games are supposed to work intuitively and should be fun to play. It is also important to take a step back and evaluate the game as a whole entity. What users are looking for is a game that is entertaining and fits seamlessly into the way they already use their device. If a game is not fun, it will not receive good reviews and it could be than no one is around to try the next version.

How do you assess if something is fun or not?  How do you know if a certain experience is something that will entice the proposed target group? Assessing fun-factor needs some aesthetic sense and critical thinking. This is the most subjective area of mobile game testing, since everyone’s definition of fun varies, but it is also the most important element for which testing is imperative. This requires a unique insight into game design, and vast experience and data about the user group and what that group enjoys.

The Mobile Game Tester

A good mobile game tester should also be a good mobile game player. They can use their experience in gaming to envision the perspective of other players, thus being able to rate the game better, and decide what’s best for its success.

Mobile game testing has unique challenges compared to most mobile app testing. Despite similarities, mobile apps and mobile games have significant differences that affect test methods. Mobile games contain subjective, crowd-pleasing qualities that need specific test approaches and the help of domain experts in technical areas such as physics, graphics and audio. As mobile gaming becomes more mainstream, proper mobile game testing can make or break your fortune. The bar is much higher for game testing overall. So, try to find the right game testers for your mobile gaming app.









  • You forgot to include testing interrupts and notifications either to occur while playing. Does the notification appear during game play? Can a user get a phone call or even a text message? Does the notification crash the game or does it stop the game and save where the user left off? How does this affect the use of memory? And don’t forget performance not only affects the web server but the device CPU too. I’ve seen a lot of problems with “commercials” affecting game play which interrupt and at various times during the playing of the game. Expect the unexpected. The combination of conditions can be overwhelming but taking the time to create a solid test strategy will help. Don’t just assume automation will be the answer. Manual tests based on critical thinking will be your friend and be cost efficient.

  • A Mobile game player, always scrolls down play store apps to find new games. Once user likes the interface and performance, user madly addicts to it. But, Most of the mobile games lack Responsive design, Connectivity issues and Moreover they are filling the screen with advertisements. Once the user clicks this ads, It is likely to connect some unknown website. Testing the games has lost its quality, should be tested with ads and if they are unresponsive, try to avoid them.

Leave a Reply

Your email address will not be published.