Buyers verify a device security deployment by testing the exact thing they will ship: named models, a named firmware build and a stated attacker privilege, with the unprotected result recorded next to the protected one. Otukan runs this evaluation with you on physical devices before you commit. You receive a scope statement, the scenarios and their results, every residual risk with an owner, and the obligations that keep the result valid after updates. The method applies to Android-based devices whose firmware an OEM or ODM builds. It verifies agreed protection, not the general security of any firmware.
Evaluation target
An evaluation without a fixed target produces results nobody can reuse. Otukan agrees three things in writing before testing starts.
Model and chipset
Each model is listed with its chipset and hardware security features. Two devices sold under one name can differ inside, so the list follows the hardware, not the marketing name.
Firmware version
The build number, security patch level and signing configuration are recorded. Results apply to that build, and the record shows which later builds were retested.
Tester privilege
What an attacker can already do decides what a result means. Otukan fixes the privilege for each scenario.
| Privilege level | What the tester starts with | Typical question it answers |
|---|---|---|
| Installed app | An ordinary app with no special permissions | Can a malicious app reach protected data or services? |
| Debug access | USB debugging enabled on a production build | What can someone with a cable and a few minutes do? |
| Unlocked test device | Bootloader unlocked, custom image possible | Does your server notice that the device is no longer yours? |
| Physical possession | The device in hand, powered off | What survives a lost or seized device? |
Test method
Every scenario runs twice on the same devices: once on the baseline build and once on the protected build. The comparison shows what Otukan’s protection changes, and what it leaves unchanged.
- Baseline behaviour. The scenario’s outcome on the build as it ships today, recorded before any change.
- Controls under test. Which Otukan controls and platform mechanisms apply, and what each is expected to do.
- Protected result. The outcome with protection in place, against the pass criteria agreed in advance.
- Residual risk. What still succeeds, how much it matters for your devices, and who owns the follow-up.
Choosing the scenarios
Scenarios come from your threat model, not from a generic list. Otukan starts with the attackers you name, such as a thief with a lost handset, an insider with debug access or a malicious app on a shared device, and picks the scenarios that show whether the proposed protection stops, slows or reports each one.
| Scenario family | Example goal for the tester |
|---|---|
| Boot and image | Start a modified or older image and keep access to your services |
| Privileged layer | Use a system component to reach data outside its role |
| Keys and evidence | Make an unlocked or modified device look genuine to your server |
| Update path | Install a release your programme has not approved |
Residual risk is part of the result. A scenario that still succeeds at a high privilege level can be an acceptable, documented outcome when your threat model says so.
Decision evidence
The evaluation ends with a decision record built for three readers: the security lead who accepts the risk, the engineering lead who owns the build, and the buyer who signs the agreement.
| Evidence | What it shows | Who relies on it |
|---|---|---|
| Repeatability | Scenarios rerun by your engineers, with the same outcome | Security lead |
| Performance | Start-up, battery and responsiveness of the protected build against the baseline, on your devices | Engineering lead |
| Maintenance obligations | What Otukan, the OEM and the institution must keep doing across updates | Buyer and programme owner |
| Limitations | Scenarios, models or privileges outside the evaluation | All three |
When to retest
The decision record names the changes that call for a retest: a new firmware build, a new version of Otukan’s protection, a chipset revision or a change in your threat model. Each change reruns the affected scenarios at the same privilege levels, and the new results are added to the record. Your approval stays tied to what your devices actually run.
The questions to settle before an OEM programme starts are listed on the OEM device security evaluation guide. How the integration itself is delivered is covered on OEM device security integration. For the product overview, see mobile device security.