Our Top 10 Tips for Mobile Apps Testing

A recent study has shown that approximately 21% of users will abandon an App after first use. Crashes, bugs, poor performance, bad UX and lack of functionality are just some of the many factors that can get in the way of your customers being engaged with, and repeatedly using your mobile apps. Having a well defined Test & QA strategy in place is essential if you want to deliver high quality Apps to your customers and it’s something we’ve focused heavily on here at amaysim.

Over the last year and a half, we’ve been on an exciting, challenging and ultimately successful journey to re-build and then repeatedly test and release our Mobile Apps here at amaysim. I’d like to take this chance to share our top 10 tips on how to address some of the challenges we’ve encountered along the way and define an effective Apps Testing Strategy...

1. Define (and then follow!) your ways of working. Ensure quality is baked in at every stage.

Working for a growing, fast-paced and agile company means we need to be able to adapt and respond quickly to change. In a competitive and dynamic business environment, the ability to be able to quickly and effectively deliver a stream of new ideas and features into market is essential. This is especially true for mobile Apps.

So, how can we ensure we’re delivering high quality Apps in a short time-frame? Well, there is no right or wrong answer, but there are a number of practices and processes that you can follow as part of your ways of working and definition of done, that will stand you in great stead. For us, these include:

• Ensure you have a reliable set of Unit, Integration and End to End tests running as part of your Continuous Integration pipeline. This is the minimum required, you can’t succeed without this. Identify your App’s critical flows and automate tests around them. Do not over-test, we need to be smart with the tests we pick to run in our CI pipeline. Remember, we want fast-feedback. We need to know quickly if we have broken a critical flow.
• Ensure every task worked by the team has gone through at least a story Kick-Off, a story Walkthrough, Exploratory Testing and that of course your CI pipeline is green (both before and after you have merged your feature!). Check out this great post from Hans Dushanthakumar (our QA Lead) where he explains how our Quality Assistance approach works at amaysim.
• There are many device models, operating systems and screen sizes your app will probably need to support. You certainly won’t have time to carefully test every single one. What we did was to put an emphasis on identifying and testing with the most used devices and OS based on customer data. A strong analytics tool can be invaluable here. Also, consider testing the oldest and newest OS versions of each platform (iOS and Android) as your App is more likely to misbehave on those versions. Check the smallest and biggest device screens too. In our experience the majority of issues will be found in these scenarios.
• Once you have a stable Release Candidate build, use a Risk Based Testing approach to define any additional tests to be performed on your final build. A good start is to list all new changes and areas that are more likely to be impacted. Get together with your team, walk them through what needs to be tested and time-box some time for a final team effort testing.

2. Phase your rollout(s) to customers.

One of the key differences between App development and ‘normal’ web/software development is that if bugs or issues are detected following an App release, there is no such a thing as simple “rollback procedure”. There are no green/blue environments or simple DNS switches we can do to revert to a known good version. Once customers have installed a new (buggy) version, you have no choice other than to release a new version of your App. This will not only cause disruption as your customers won’t be happy to re-install a new version again, but also is dependant on your Apps being ‘re-approved’ before they can be released — something which does not always happen in a timely manner.

Both Apple Store and Google Play allow you to do “phased rollout releases”. Phased Release is the ability to be able to gradually release your App update to users over a period of time. You can start rolling out a new version to 1% of your user base and gradually increment along the days.

You can then use your Analytics & Monitoring tools to observe how your release is progressing and stop the rollout, if needed. Once you’re confident with the rollout, you can then release to everyone. This will significantly reduce chances of impacting many customers if a major issue is identified.

3. Watch your analytics and monitoring Tools. Obsessively.

“You can have data without information, but you cannot have information without data.” — Daniel Keys Moran, an American computer programmer and science fiction writer.

To define where we are heading to, first we need to understand where we are at. Analytics data plays an incredibly important role to help deciding how we are going to prioritise Test activities.

We’ve been using a set of different analytics and monitoring tools such as Firebase, Crashlytics, New Relic and SumoLogic to understand better our Apps and customer behaviours.

We pay close attention to a number of different metrics that help us validate both customer experience, as well as other key quality metrics. These include, amongst other: Crash rate, top crashes, most used App version, most used OS versions, most used devices, etc. These put together help provide a much clearer picture on where we potentially should be focusing more of our Testing effort.

4. Feature toggles are your best friend.

Feature Toggle is a powerful technique that allows you to hide, enable and disable features during build or run time without changing code. This is a common approach used as a way to de-risk Releases by putting code in Production without necessarily enabling it. This aligns well with our philosophy of Continuous Delivery and constantly releasing working code to Production in small increments.

You may have found yourself in a similar situation: stakeholders (or even yourself) may want to have a look at the current state of the App but there are features still under development which may not be fully functional. So you don’t necessarily want to have everything enabled/visible. You can make use of Feature Toggles to easily enable and disable specific features on the fly. We currently use three different types of toggles:

Build Toggle — this one is the main toggle in the code to enable and disable features. These toggles will define what features will be seen by users once they open the App. From Testing point of view, it would require code changes and a new build to get App features in the state you need. Not too efficient for Testing.

Run-time Toggle — this is my favourite! These toggles are located in our App’s config (non-Prod builds only) and can be enabled or disabled on the fly. You see the result straightaway in the App. This is quite useful for both technical and non-technical people. We use this technique quite often, not just to enable and disable features but also to point our App to different Test Environments. For instance, if we want to point an instance of our App to a Dev environment, we can easily do it by using our Run-time toggle instead of having to make code changes and re-build our App. Here is an example of what our Environment Run-time Toggle look like:

Remote Toggle — we use this technique to enable and disable features in Production. It can be used to override Build Toggles. This is also our safety net in case something goes wrong in production and we need to disable a feature. This mechanism helps our team to increase confidence level when doing Releases as we can quickly turn features ON and OFF and reduce customer impact.

5. End to End Automated Tests will save you. Again and again.

As mentioned earlier, it’s extremely important to have a suite of End to End Automated Tests (UI) in place and running as part of your CI pipeline. UI Tests are usually brittle and high-maintenance so a smart strategy is required.

Writing and maintaining UI tests for both iOS and Android can be quite challenging and time consuming. We currently have a Test Framework based on Ruby, Cucumber and Appium. Our Test Framework has been developed in a way that we can easily write new tests for both iOS and Android platforms at the same time which saves us quite a bit of time.

Our Test Framework uses Page Object Model and once iOS and Android element locators are defined in the code, we can simply write a test and it will work on both platforms (sometimes with minor code changes). This saves us significant overhead from needing to write separate tests for each platform.

We also store our App artefacts (.ipa and .apk files) in Amazon AWS S3 so our CI pipeline can quickly fetch artefacts for every build and trigger our suite of Automated Tests.

6. Don’t forget manual testing. It will also save you. (Again and again).

We all know the importance of having a reliable suite of automated tests in place to support development of new features and to avoid introducing issues.

Although Manual Testing may not be as efficient as Automated Tests, it’s still an important step to increase team’s confidence that the App will behave as expected. When manually testing your App, here are a few things to pay attention at:

• Upgrade test (from current version to new version)
• UI glitches
• User Experience (UX)
• Speed and Performance
• Typos
• Check your App in the smallest and in the biggest device screen available
• Open, close and restart your App; put your App in the background and bring it back to foreground; leave your App idle and interact with it after some time, etc.

7. Use a CMS to publish App content changes.

Having to do an App release to simply make minor content changes is not ideal. From the customer point of view, they will have to download a new version of the App and from the development point of view, team will have to deal with release related activities which usually take time, effort and introduce possible risks.

In order to solve this challenge, we’ve been using Contentful CMS which is a platform that lets you create, manage and distribute content to your App via an API without requiring a release. This also gives more control and flexibility to Stakeholders and Content Creators to make content changes quickly.

Having a CMS to deal with App content makes Testing activities much easier as we can quickly change content via CMS and see how it will look like in the App without requiring code changes and new builds.

8. Cross device testing is time consuming. Find ways to Optimise this.

There is an ever-growing list of different device models (e.g: Samsung Galaxy, Google Pixel, Nexus, iPhone, etc), several Operating System versions (e.g: Android 4, 5, 6, 7, 8, iOS 8, 9, 10, 11), different screen sizes, etc. That said, it’s nearly impossible to test all combinations.

Using a Cloud Testing service such as AWS Device Farm, SauceLabs, or BrowserStack can help you overcome this challenge. We’ve recently setup AWS Device Farm and we’re currently verifying whether it suits our needs. At the moment it’s looking promising!

In cases where this approach is not viable, I suggest to prioritise your customer’s most used device models and OS based on analytics data (which I hope you have :)).

9. Understand and test your unhappy scenarios.

We should always ask ourselves: “what could go wrong?”. Well, let’s think about a real life example: your customer is on the train on their way to work and using your App to purchase one of your products. The train goes through a tunnel for about 2 minutes with extreme low mobile signal. Your App is doing a few requests to your backend which eventually fail or timeout. How will your App behave? What will the user see? Will the App re-try the operation and possibly purchase a product twice? You need answers to these questions.

We’ve been using CharlesProxy to help us testing failure (unhappy) scenarios. CharlesProxy is a well known HTTP/HTTPs proxy that enables us to view all traffic between our App (or machine) and the Internet including requests, responses, headers, etc. The most interesting CharlesProxy features from the Testing point of view are:

• Rewrite requests — We use this option quite often to simulate different HTTP Response status codes (e.g: 500, 404, 400, 401) for different requests and check how our App behaves.
• Map Remote — This option is pretty handy to deal with different Test environments. For instance, in our case, we’ve got Dev, QA, Preprod and Prod environments. Map Remote allows us to quickly change our Requests to point to different Environment URLs.
• Map Local — We use this option to simulate different JSON Responses when hitting specific API endpoints. For instance, return a JSON that has a large number of content (characters) and observe how it looks like in the App; Return a malformed JSON; etc
• Throttle internet connection — This is useful to simulate different internet connections (e.g: 3G, 4G, ADSL, etc) and check how your App behaves. We’ve seen Requests timing out after 30s and App not behaving as expected. Give it a try and figure out how your App behaves in unhappy scenarios.

10. Dog-food your Apps internally.

You don’t want to be manually building your App and installing it in each test device, right? So you need to automate this process as part of your CI pipeline.

This is an immensely important step to test your App efficiently. You need to be able to easily distribute your Apps internally for testing so you can get early feedback from your stakeholders. This gives you time to identify, prioritise and fix issues as early as possible using a friendly and more forgiving audience.

We’ve been using HockeyApp to distribute our Apps internally for Testing purposes. We setup our CI pipelines in such a way that once we have a successful build, it will automatically push artefacts (.ipa for iOS and .apk for Android) straight to HockeyApp. Once this is done, we can easily download and install it in any test device. Here’s an example of our HockeyApp setup:

Wrapping up

Having a robust, optimised and efficient Test & QA strategy is fundamental in allowing you to rapidly deliver high quality Apps to your customers. Being able to prioritise your Testing effort, minimise risks, automate the right tests, identify issues early, distribute your App efficiently, measure and monitor your progress are just some of the factors to be considered to help you achieve outstanding App quality and to ultimately deliver an awesome experience to your customers.

Has any of the above piqued your interest? Does amaysim sound like the sort of place where you think you could make an impact? Do you thrive in organisations where you are empowered to bring change and constant improvement? Why not take a few minutes to learn more about our open roles and opportunities and if you like what you see then say hi, we’d love to hear from you..

Shout-out to all the lawyers..

The views expressed on this blog post are mine alone and do not necessarily reflect the views of my employer, amaysim Australia Ltd.