How to test app shielding performance

Measure app shielding performance by comparing a baseline and a protected build from the same commit, on the same physical devices, with the same workload and controlled conditions. Record startup, download and installed size, memory, stability and the timing of sensitive flows. Repeat the runs and keep their spread: a single fast result is not a release decision. Two Android details catch teams out: Macrobenchmark needs a profileable build, which anti-debugging controls may react to, and code changed after a Baseline Profile is generated can stop that profile applying. Android and iOS are measured separately with their own tools.

Fix the baseline and measurement conditions

Build the baseline and protected candidates from the same revision. Keep dependencies, assets, compiler settings, feature flags and backend behaviour identical, and write down the protection configuration. If another difference is unavoidable, record it before attributing any change to protection.

Otukan compares a protected build with its unprotected baseline from the same commit, on the same physical devices, and fits your existing build and release pipeline. You choose which checks run and when, per user flow, so the configuration you measure should be the one you intend to ship.

Android build requirements

Google recommends benchmarking a build close to production: non-debuggable, minified and with release optimisations. Macrobenchmark also needs the target app to be profileable, declared in the manifest:

<application>
    <profileable android:shell="true" />
</application>

Two consequences for a protected app:

• Anti-debugging may react. A profileable build lets the shell attach profiling and tracing tools. Runtime protection that watches for debugging or instrumentation may respond to that state or to the tracing itself. Agree which checks are active in the benchmark build, confirm the protection is still running, and record any difference from the shipped configuration.
• Baseline Profiles can stop matching. A Baseline Profile lists classes and methods by name so Android can compile the startup path ahead of time. The build packages it with the APK or AAB. A step that renames or rewrites DEX code after the profile is generated can leave rules that no longer match, and the protected build then starts slower for a reason unrelated to its runtime checks. Generate or confirm the profile against the final code, and check it applies, for example with ProfileVerifier from the androidx.profileinstaller library.

Run Macrobenchmark with an explicit compilation mode, such as CompilationMode.Partial() for the Baseline Profile path, so both candidates are compiled the same way.

iOS build requirements

Use a Release configuration for both candidates, with the same optimisation settings, features and test data. Run the measured app without an attached debugger: a protected app that responds to a debugger will otherwise measure its own response. Apple’s XCTest provides XCTApplicationLaunchMetric, XCTClockMetric, XCTCPUMetric, XCTMemoryMetric and XCTStorageMetric for use with measure(metrics:options:block:).

Prepare each run

1. Choose physical devices that represent your users, including constrained models where they matter. Record model, OS build and free storage.
2. Install the candidate and prepare the same test account, dataset, permissions and app state. Define when reinstallation or data reset happens.
3. Hold network, charging state, display settings and background activity as steady as the lab allows. Record thermal state and interruptions rather than quietly discarding inconvenient runs.
4. Define the startup state (cold, warm or hot) and where measurement starts and ends.
5. Fix the warm-up, iteration count and run order before collecting results. Alternate or randomise the candidates to reduce drift.
6. Confirm that the intended protection configuration is active during measurement.

Keep failed runs with their reason. An app that crashes before the completion marker has not delivered a short completion time.

Collect startup, size, memory and stability

Use workloads that end in a user-visible outcome: a usable sign-in screen, an authenticated account view or a completed action. Keep server-side conditions controlled so network delay does not pass for local cost.

Never compare an Android bundle’s file size with an installed iOS app’s footprint. Compare like with like inside each platform.

Android sequence

Write a Macrobenchmark test for the launch or journey, using StartupTimingMetric for startup and FrameTimingMetric for scrolling or animation. Run the same test against each candidate. Macrobenchmark writes JSON results and a Perfetto system trace for each iteration. Archive both with the build and device identities, and open the traces of slow runs rather than keeping only the console summary.

Measure size, memory and stability separately where the benchmark does not cover them, and record the tool and units.

iOS sequence

Write an XCTest performance test for the journey and choose the metric deliberately: XCTApplicationLaunchMetric for launch, XCTClockMetric for a measured interval, XCTMemoryMetric for memory. Run both candidates on the same device and keep the result bundle with the test configuration. Use Instruments in separate diagnostic sessions; a trace explains a delay, but its timings should not replace the release-like measurement.

Record each run

Keep one record per run with:

• Build identity, baseline or protected label and protection configuration revision.
• Device model, OS build, tool versions and test identifier.
• Workload, startup state, compilation mode and run order.
• Metric, unit, value and completion status.
• Raw trace or result reference, exclusion reason if any, and reviewer.

Use non-production accounts and redact diagnostics before sharing a review packet.

Interpret variation and release impact

Report attempted and valid runs for each platform and device. Show the median and a high percentile with the spread; never publish only the best run. Keep exclusion rules so another engineer can reproduce the analysis.

Otukan reports medians and high percentiles per device and per flow rather than a single average.

Compare absolute differences first. If you also report a relative change, state the baseline and skip percentages where the baseline is too small or unstable to support them. Overlapping, noisy results call for more runs, not a claim of zero cost.

Agree user-impact budgets before reading the results. An acceptable launch time does not make up for a failed sign-in. Run authorised security-trigger tests separately from performance runs, and repeat installation, update, sign-in and the sensitive flow after any protection change.

• Baseline and protected candidates differ only as documented.
• Android benchmark builds are profileable, and active protection is confirmed.
• The Baseline Profile applies to the protected build.
• Android and iOS results keep their own devices, units and limits.
• Failed runs and exclusions are visible.
• Raw data supports the published summary and reviewer decision.

For product evaluation, see mobile app protection performance. Broader release checks are covered on protected mobile app testing.

Further reading

• Write a Macrobenchmark — Android Developers
• Baseline Profiles overview — Android Developers
• <profileable> — Android Developers
• Performance tests — Apple Developer Documentation
• Improving your app’s performance — Apple Developer Documentation