A mobile security team should evaluate security validation by checking three things: that each test runs in an environment able to prove it, that any exception needed for testing stays out of the release, and that release criteria are written as expected signals before the first run. With Otukan, a protected build is validated on physical devices for real behaviour, in simulators and emulators for functional coverage where detection is expected, and in approved cloud labs only for what they can honestly show. This guide applies to Android and iOS apps, including Flutter and React Native. Performance measurement and pipeline set-up are separate topics.
The full test plan for protected builds, including functional and performance layers, is on protected mobile app testing.
Choosing the test environment
A protected app watches the environment it runs in. That makes the choice of test device part of the test, not a detail.
| Environment | What it can prove | What it cannot prove | What Otukan does there |
|---|---|---|---|
| Physical device, stock OS | Real user behaviour, clean-device acceptance, install and update | Detection of attack tooling, unless you add it | Stays silent on legitimate journeys |
| Physical device, prepared for attack testing | That a detection fires and your policy applies | How ordinary users experience the app | Raises the matching signal and applies the response for that flow |
| Android emulator or iOS Simulator | Functional flows, layout, automated regression | Anything about genuine devices | Reports an emulator or simulator signal, which is the expected result |
| Approved cloud device lab | Breadth of models and OS versions | Clean-device behaviour, if the lab modifies or instruments devices | Reports what it finds; your team decides whether that is expected |
Physical devices
- Do you have at least one untouched device per platform for clean-device acceptance?
- Are your attack-testing devices kept apart, labelled and never used for acceptance runs?
- Does your device set include the manufacturers and OS versions your users actually run?
Simulators and emulators
- Have you recorded that an emulator or simulator signal is the correct result on these environments?
- Are automated suites that run in emulators pointed at a test build, so enforcement does not block them?
Cloud device labs
Some labs install their own agents, change system settings or grant elevated access to drive devices remotely. A protected build may report that, and it should.
- Has the lab told you in writing how its devices are modified?
- Have you run one clean journey in the lab and recorded which signals appeared?
Exceptions without weakening production
Testing sometimes needs a check relaxed. The danger is an exception that outlives the test or reaches the store.
How Otukan scopes an exception
Otukan lets an exception apply to a named test build and a named set of devices, with an end date. The release protection profile is untouched, so a release built from the same pipeline carries every check. Observe mode is the other tool: controls stay active and record signals, but the journey continues, which is often all a validation run needs.
- Is every exception tied to a build, a device set, an owner and an end date?
- Does your release gate refuse a build that carries any validation exception?
- Are exceptions reviewed and closed when the test cycle ends?
What to rule out
- Shipping a release built with checks turned off to make a lab run pass.
- Adding production accounts or real customer data to a relaxed test build.
- Widening an exception from one device to a whole lab without a recorded reason.
Release criteria
A release decision is only as good as the criteria written before the tests ran. Express each one as an expected signal.
Expected events
| Test case | Environment | Expected signal | Expected response |
|---|---|---|---|
| Clean install and sign-in | Physical device, stock OS | None | Journey completes |
| Update from previous protected version | Physical device, stock OS | None | Journey completes; stored data intact |
| Instrumentation attached during payment | Prepared device | Hooking signal | Your chosen response for the payment flow |
| Run on emulator | Emulator or Simulator | Emulator or simulator signal | Your chosen response for that environment |
| Modified copy installed | Prepared device | Integrity signal | Your chosen response for a modified copy |
User journeys
- Do the cases cover install, update, sign-in, your most sensitive flow and recovery after a failed step?
- Has each journey run on both platforms, recorded separately?
Diagnostic results
For every run, Otukan provides diagnostic information that names the protection profile and the checks that fired, so a tester can tell a real detection from a misconfigured test.
- Does each result record the build, profile, device, OS version, date and reviewer?
- Did your backend receive the signals the table expects, and did its rules act on them?
- Were unexpected signals investigated and explained before sign-off?
Background on choosing a protection product sits in mobile app security evaluation.