Running ART Tests with Atest / Trade Federation

ART Testing has early support for execution in the Trade Federation ("TradeFed") test harness, in particular via the Atest command line tool.

Atest conveniently takes care of building tests and their dependencies (using Soong, the Android build system) and executing them using Trade Federation.

See also README.md for a general introduction to ART run-tests and gtests.

ART run-tests

Running ART run-tests on device

ART run-tests are defined in sub-directories of test/ starting with a number (e.g. test/001-HelloWorld). Each ART run-test is identified in the build system by a Soong module name following the art-run-test-<test-directory> format (e.g. art-run-test-001-HelloWorld).

You can run a specific ART run-test on device by passing its Soong module name to Atest:

atest art-run-test-001-HelloWorld

To run all ART run-tests in a single command, the currently recommended way is to use test mapping (see below).

You can nonetheless run all supported ART run-tests with a single Atest command, using its support for wildcards:

atest art-run-test-\*

Note: Many ART run-tests are failing with the TradeFed harness as of March 2021, so the above Atest command will likely report many tests failures. The ART team is actively working on this issue.

ART gtests

Running ART gtests on device

There are three ways to run ART gtests on device:

  1. by building "standalone" ART gtests and running them against the active ART APEX on the device;
  2. by installing the Testing ART APEX (i.e. manually "updating" the ART APEX on device); or
  3. by setting up a chroot environment on the device, and "activating" the Testing ART APEX in that environment.

The first approach can be used to test the ART APEX presently residing on a device (either the original one, located in the "system" partition, or an updated package, present in the "data" partition).

The second and third approaches make use of the Testing ART APEX (com.android.art.testing.apex), and were previously the only options to run ART gtests on device, because of build- and link-related limitations (the ART gtests had to be part of the ART APEX package itself to be able to build and run properly).

Running standalone ART gtests on device

Standalone ART gtests are defined as Soong modules art_standalone_*_tests. You can run them individually with Atest, e.g:

atest art_standalone_cmdline_tests

You can also run all of them with a single Atest command, using its support for wildcards:

atest art_standalone_\*_tests

The previous commands build the corresponding ART gtests and their dependencies, dynamically link them against local ART APEX libraries (in the source tree), and run them on device against the active ART APEX.

Running ART gtests on device by installing the Testing ART APEX

You can run ART gtests on device with the ART APEX installation strategy by using the following atest command:

atest ArtGtestsTargetInstallApex

This command:

  1. builds the Testing ART APEX from the Android source tree (including the ART gtests);
  2. installs the Testing ART APEX using adb install;
  3. reboots the device;
  4. runs the tests; and
  5. uninstalls the module.

You can run the tests of a single ART gtest C++ class using the ArtGtestsTargetInstallApex:<art-gtest-c++-class> syntax, e.g.:

atest ArtGtestsTargetInstallApex:JniInternalTest

This syntax also supports the use of wildcards, e.g.:

atest ArtGtestsTargetInstallApex:*Test*

You can also use Trade Federation options to run a subset of ART gtests, e.g.:

atest ArtGtestsTargetInstallApex -- \
  --module ArtGtestsTargetInstallApex --test '*JniInternalTest*'

You can also pass option --gtest_filter to the gtest binary to achieve a similar effect:

atest ArtGtestsTargetInstallApex -- \
  --test-arg com.android.tradefed.testtype.GTest:native-test-flag:"--gtest_filter=*JniInternalTest*"

Running ART gtests on device using a chroot environment

You can run ART gtests on device with the chroot-based strategy by using the following command:

atest ArtGtestsTargetChroot

This sequence:

  1. builds the Testing ART APEX from the Android source tree (including the ART gtests) and all the necessary dependencies for the chroot environment;
  2. sets up a chroot environment on the device;
  3. "activates" the Testing ART APEX (and other APEXes that it depends on) in the chroot environment;
  4. runs the tests within the chroot environment; and
  5. cleans up the environment (deactivates the APEXes and removes the chroot environment).

Test Mapping

ART Testing supports the execution of tests via Test Mapping. The tests declared in ART's TEST_MAPPING file are executed during pre-submit testing (when an ART changelist in Gerrit is verified by Treehugger) and/or post-submit testing (when a given change is merged in the Android code base), depending on the "test group" where a test is declared.

Running tests via Test Mapping with Atest

It is possible to run tests via test mapping locally using Atest.

To run all the tests declared in ART's TEST_MAPPING file, use the following command from the Android source tree top-level directory:

atest --test-mapping art:all

In the previous command, art is the (relative) path to the directory containing the TEST_MAPPING file listing the tests to run, while all means that tests declared in all test groups shall be run.

To only run tests executed during pre-submit testing, use:

atest --test-mapping art:presubmit