What the Android 15 Compatibility Test Suite Actually Guarantees

Google updated the Android 15 Compatibility Definition Document last week, tightening the rules every manufacturer must satisfy before shipping a certified Android device. To earn that certification, devices must pass the Android 15 compatibility test suite and its companion CTS Verifier using the software that actually ships on the device not a test build. What neither tool does is verify whether the app a user just installed matches what a developer originally submitted to any store.

That gap has real consequences. A peer-reviewed study published earlier this year found that roughly 29% of matched app pairs same title, same version code differed meaningfully between Google Play and third-party markets, with off-Play versions more likely to request expanded permissions (Springer, Empirical Software Engineering, this year). A certified device can run the wrong build without the user ever knowing.

How Google checks Android app compatibility

To ship as a certified Android 15 device, a manufacturer must satisfy every requirement in the Android 15 Compatibility Definition Document (CDD) and pass the Android Compatibility Test Suite (CTS) on the software that actually ships, not a pre-release build (AOSP, last week). The bar shifts over time: devices must clear the latest CTS version available when software is finalized, not whatever version existed at the start of development.

The CTS is not a single automated run. Devices must also pass the CTS Verifier, which requires human interaction to confirm behaviors automated suites cannot catch, and must maintain compatibility in ambiguous cases or wherever manufacturers have reimplemented parts of the Android reference code (AOSP, last week). That last requirement addresses a real pattern: reimplemented components that pass the letter of the tests while behaving differently in practice.

The CDD is the governing document; the Android CTS for Android 15 is its primary enforcement instrument. The Android 15 revision actively adds, changes, and removes specific requirements confirming that Android 15 device compatibility requirements are a living standard, not a fixed checklist (AOSP, last week).

Certification means a phone should run Android apps correctly and consistently. It says nothing about whether a specific app downloaded from outside the Play Store is the same binary anyone reviewed. Device compatibility and app provenance are different problems, and only one gets a formal test suite.

Why app provenance is a harder problem

Passing the CTS tells you the device is trustworthy. The app is a separate question. That distinction becomes concrete at scale: analyzing 17,218 matched app pairs across Google Play and third-party markets, researchers found approximately 29% showed meaningful differences in at least one measured dimension despite carrying identical version codes (Springer, this year). Version codes are labels. They carry no integrity guarantee.

The divergence follows a pattern. Third-party market versions consistently requested more permissions than their Play Store equivalents, with games showing the sharpest differences (Springer, this year). More permissions means a larger potential attack surface, even when the app looks identical to the person installing it.

Some of that divergence is deliberate. The Pinduoduo case, cited in the same research, shows developers shipping meaningfully different code between Play and off-Play versions of the same app by design rather than accident. The researchers describe this as potentially "just the tip of the iceberg" (Springer, this year) a characterization grounded in the structural reality that nothing in a version code signals what the binary actually contains.

The Android 15 Compatibility Definition Document was not built to catch this. The CTS is a platform compatibility tool. Verifying that an installed APK matches what a developer submitted to a store is a distribution integrity problem, and the two require different solutions.

What Android's security stack offers and where it stops

Sideloading is entering a new era of friction and policy pressure, with a growing expectation that installers prove what they are doing before any APK touches a device (Scan.quest, two weeks ago). Best practice for managed environments means validating a package's cryptographic hash against a trusted manifest, confirming the signing certificate or certificate lineage, and blocking version downgrade attacks. That is not yet a universal Android enforcement mechanism, but it describes where the pressure is heading.

Android already has hardware infrastructure relevant to this problem. Google's Trusty OS runs as a Trusted Execution Environment completely isolated from the main Android system, protected from malicious apps and Android-level vulnerabilities alike, and is currently used for payments, biometrics, device reset protection, and secure PIN processing (AOSP, Trusty TEE, this year).

A separate paper published earlier this year proposed extending hardware-backed isolation to app manifest verification. The SM-AAPIV framework splits cryptographic checks across two independent hardware-backed segments stored in separate keystores, so an attacker must compromise both simultaneously to reconstruct the verification root (Journal of Cybersecurity, earlier this year). A three-tier fallback design covers an estimated 92% of active Android devices across hardware tiers. This is academic research, not a shipping Google product, and the distinction matters: it points to where the architecture could go, not what it does today.

The distance from proposal to practice shows up clearly in how certification currently works. The Play Integrity API, which the same research benchmarks as a detection baseline, is not listed in the Android 15 CDD as a mandatory certification requirement (Journal of Cybersecurity, earlier this year). Certification governs how devices behave. Whether the app running on that device came from a trustworthy source still depends largely on which distribution channel the user chose.

What this means for buyers, sideloaders, and enterprises

For most Android users, a certified Android 15 device is a meaningful baseline. The hardware passed behavioral testing on final shipping software and must hold compatibility even where manufacturers would otherwise cut corners (AOSP, last week). That floor is genuine and worth something.

For anyone installing apps outside the Play Store, it does not extend to the APK. Roughly 29% of identically versioned apps differ between Play and third-party sources (Springer, this year), with differences ranging from permission scope to intentionally divergent behavior. The device passed its tests. The app is a separate question with no equivalent answer.

For enterprises managing Android fleets, the operational hierarchy is direct. Device certification sets the floor; meaningful app integrity requires controls built above it. Signature verification, source policy, and audit logging sit on top of what Android 15 certification provides, not inside it (Scan.quest, two weeks ago; AOSP Trusty TEE, this year). A practical trust hierarchy for enterprise teams runs roughly: certified device first, then verified installer workflow, then per-package cryptographic checks each layer covering what the one below it cannot.

The Android 15 CDD update is a genuine tightening of the device-level standard. The app provenance problem it leaves open is not a failure of the test suite; it is a structurally different challenge that Android's broader security ecosystem is working to address through tools that are still evolving. Knowing which problem each tool was built to solve is the most useful thing anyone managing Android devices can take from this.