B. Testing mobile app — 1. Strategy
Why test?
- Rapid feedback: testing complex logic for all possible scenarios.
- Early failure detection: find the issue before it becoming bug.
- Safe code refactoring: optimize and refactor code without worrying about regression.
- Stable development velocity: make code changes with confidence, keeps stable velocity.
What to test?
- Functionality: does my app do what it’s supposed to?
UI layer:
~ Unit test state holder / business logic for UI layer (view model).
~ Integration test, user interactions (screen level) and UI state.
Data layer:
~ Unit test data layer with Test Doubles (check section below)
Domain layer:
~ Unit test domain layers, Unit test utility classes
Test edge cases:
~ Complex business logic
~ Network connection error
~ Corrupt data
~ User Interactions (phone level)
~~~ Rotate device
~~~ Switch to a different app. - Performance: does it do it quickly and efficiently?
- Accessibility: does it work well with accessibility services?
- Compatibility: does it work well on every device and API level?
~ App is working on smallest to biggest supported screen size
~ App is working for the minimum and maximum supported SDK version.
~ App is working for new devices out in the market.
How to test?
In an ideal world, you would test every line of code in your app on every device that your app is compatible with. Unfortunately, this approach is too slow and costly to be practical.
A good testing strategy finds an appropriate balance between the fidelity of a test, its speed, and its reliability.
Before you think about writing tests, make sure your code supports it. Check the section below Testable Architecture.
- Manual test:
~ QA engineers writing test plans after the start of release plan.
~ QA engineers writing test cases for the User story in each iteration.
~ QA engineers creating data, users required for testing in a test environment.
~ Reviewing test cases with developers, BA.
~ Testing test cases once code is ready.
~ Creating issues if code is not working as expected. - Automated test:
~ Instrumented tests run on an Android / iOS device, either physical or emulated.
~~~ Medium tests are in between and check the integration between two or more units.
~~~ End-to-end tests or big tests verify larger parts of the app at the same time, such as a whole screen or user flow.
~ Local tests execute on your development machine or a server, so they’re also called host-side tests.
~~~ Unit tests or small tests only verify a very small portion of the app, such as a method or class. - How much to test?
~ Check most important section below, Test Pyramid.
When to test?
- Unit test: For each Pull Request / develop branch merge (synchronous).
- Integration test:
~ New or updated tests for updated code (manually): For each PR.
~ Full test suite: nightly build / merge to develop branch (asynchronous). - E2E test: Full test suite: nightly build / merge to develop (asynchronous)
- Compatibility test: E2E tests on AWS Device Farm, for minimum / maximum supported SDK and minimum / maximum supported device screen size. Always lookout for latest devices / SDK versions.
Testable Architecture
With a testable app architecture, the code follows a structure that allows you to easily test different parts of it in isolation. Testable architectures have other advantages, such as better readability, maintainability, scalability, and reusability.
An architecture that is not testable produces the following:
- Bigger, slower, more flaky tests. Classes that can’t be unit-tested might have to be covered by bigger integration tests or UI tests.
- Fewer opportunities for testing different scenarios. Bigger tests are slower, so testing all possible states of an app might be unrealistic.
Approach:
Use decoupling using following code splitting techniques
- Multiple layers
~ UI
~~~ Keep business logic out of UIController
~~~~~No framework dependencies in business logic class e.g. In case of Android, no package android. * in ViewModel
~ Domain
~ Data - Multiple modules (per feature)
- Use DI
Test Doubles:
When you need to provide a dependency to a subject under test, a common practice is to create a test double (or test object). Test doubles are objects that look and act as components in your app but they’re created in your test to provide a specific behavior or data. The main advantages are that they make your tests faster and simpler.
Types of test doubles:
- Fake: A test double that has a “working” implementation of the class, but it is implemented in a way that makes it good for tests but unsuitable for production. Fakes don’t require a mocking framework and are lightweight. They are preferred.
Example: an in-memory database. - Mock: A test double that behaves how you program it to behave and that has expectations about its interactions. Mocks will fail tests if their interactions don’t match the requirements that you define. Mocks are usually created with a mocking framework to achieve all this.
Example: Verify that a method in a database was called exactly once. - Stub: A test double that behaves how you program it to behave but doesn’t have expectations about its interactions. Usually created with a mocking framework. Fakes are preferred over stubs for simplicity.
- Dummy: A test double that is passed around but not used, such as if you just need to provide it as a parameter.
Example: an empty function passed as a click callback. - Spy: A wrapper over a real object which also keeps track of some additional information, similar to mocks. They are usually avoided for adding complexity. Fakes or mocks are therefore preferred over spies.
Test Pyramid
The lower layers contain faster, cheaper and isolated tests. At the top level of the pyramid, we will have slower, more expensive and integrated tests.
our app should contain:
- 70% small or unit tests
- 20% medium or integrated tests
- 10% large or end-to-end tests
Following diagram illustrates how to implement test pyramid on both platforms.
Article from apple website: https://developer.apple.com/documentation/xcode/testing-your-apps-in-xcode
