Android CTS (Compatibility Test Suite) introduction

发表于   |   分类于


http://i-miss-erin.blogspot.com/2010/04/android-cts-compatibility-test-suite.html

What is CTS?

Compatibility Test Suite is a command mode tool to run a series of test cases in Android. They published all from Eclair branch on January 2010.

git repository
http://android.git.kernel.org/?p=platform/cts.git;a=summary

How to run CTS in host machine?

$ cd $MYDROID
$ . build/envsetup.sh
$ make cts
(it would generate all test plans, packages, cases, result report format and then zip to a android-cts.zip file)

  • Package CTS: out/host/linux-x86/cts/android-cts.zip
  • cts make file: mydroid/build/core/tasks/cts.mk
  • run cts program: mydroid/out/host/linux-x86/bin/cts
  • test plans: mydroid/out/host/linux-x86/cts/android-cts/repository/plans
  • test packages: mydroid/out/host/linux-x86/cts/android-cts/repository/testcases
  • test results: mydroid/out/host/linux-x86/cts/android-cts/repository/results
  • CTS program settings value: mydroid/cts/tools/utils/host_config.xml


run cts
$ cd $mydroid/out/host/linux-x86/bin/
$ ./cts
Screen would like this:

erin@midnight:~/eclair/mydroid/out/host/linux-x86/bin$ ./cts
Android CTS version 2.1_r1
cts_host > help
Usage: command options
Avaiable commands and options:
Host:
help: show this message
exit: exit cts command line
Plan:
ls –plan: list available plans
ls –plan plan_name: list contents of the plan with specified name
add –plan plan_name: add a new plan with specified name
add –derivedplan plan_name -s/–session session_id -r/–result result_type: derive a plan from the given session
rm –plan plan_name/all: remove a plan or all plans from repository
start –plan test_plan_name: run a test plan
start –plan test_plan_name -d/–device device_ID: run a test plan using the specified device
start –plan test_plan_name -t/–test test_name: run a specific test
start –plan test_plan_name -p/–package java_package_name: run a specific java package
start –plan test_plan_name -t/–test test_name -d/–device device_ID: run a specific test using the specified device
start –plan test_plan_name -p/–package java_package_name -d/–device device_ID: run a specific java package using the specified device
Package:
ls -p/–package: list available packages
ls -p/–package package_name: list contents of the package with specified name
add -p/–package root: add packages from root to repository
rm -p/–package package_name/all: remove a package or all packages from repository
Result:
ls -r/–result: list all result of sessions
ls -r/–result -s/–session session_id: list detail case result of a specified session
ls -r/–result [pass/fail/notExecuted/timeout] -s/–session session_id: list detail cases of a specified session by the specified result.
History:
history/h: list all commands in command history
history/h count: list the latest count records in command history
history/h -e num: run the command designated by ‘num’ in command history
Device:
ls -d/–device: list available devices




How to add test plan, test packages to CTS?

  1. Add test package name to cts.mk, it would generate apk file when we build cts.
  2. Add all source folder (includes src java files, Android.mk and its Manifest file) to mydroid/cts/tests/tests
  3. If you’d like to create a test plan for this test package. Modify this python script: mydroid/cts/tools/utils/buildCts.py

Adjust CTS program settings?

Modify $mydroid/cts/tools/utils/host_config.xml
  • Number of tests executed between reboots. A value <= 0 disables reboots. (maxTestCount)
  • Max size [tests] for a package to be run in batch mode. (maxTestInBatchMode)
  • Max time [ms] between test status updates. (testStatusTimeoutMs)
  • Max time [ms] from start of package in batch mode and the first test status update. (batchStartTimeoutMs)
  • Max time [ms] from start of test in individual mode to the first test status update. (individualStartTimeoutMs)
  • Timeout [ms] for the signature check. (signatureTestTimeoutMs)
  • Timeout [ms] for package installations. (packageInstallTimeoutMs)
  • Time to wait [ms] after a package installation or removal. (postInstallWaitMs)
Write your own testing package?

If you would like to write your own testing package, you may reference the Instrumentation Testing document from Android porting guide. Also, you could reference the source code from few Android application, like Browser, Messages, Gallery, Email, Camera, Calculator….etc. You can build its test apk file and upload it to device.

How to write test cases?
Each instrumentation test case is similar to an Android application with the distinction that it starts another application. For example, have a look in the $MYDROID/packages/apps/Music directory.
  • There should be a Makefile and an Android Manifest file
  • Tests are located in $MYDROID/packages/apps/Music/tests.
  • The Instrumentation Test Runner is located at packages/apps/Music/tests/src/com/android/music/MusicPlayerFunctionalTestRunner.java.

Build package apk file

erin@midnight:~/eclair/mydroid/packages/apps/Music/tests$ mm
Install: out/target/product/generic/data/app/MusicTests.apk

Install it to the device
erin@midnight:~/eclair/mydroid/packages/apps/Music/tests$ adb install ../../../../out/target/product/generic/data/app/MusicTests.apk

How to run test cases in device?

Running Tests

erin@midnight:~/$ adb shell pm list instrumentation
instrumentation:com.android.music.tests/.MusicPlayerStressTestRunner (target=com.android.music)
instrumentation:com.android.music.tests/.MusicPlayerFunctionalTestRunner (target=com.android.music)
instrumentation:com.android.music.tests/.MusicPlayerLaunchPerformance (target=com.android.music)

The am command is a command-line interface to the ActivityManager. ‘am’ is used to start and instrument activities using the adb shell command, as shown in the snippet below:


> adb shell am
usage: am [start|instrument]
am start [-a <ACTION>] [-d <DATA_URI>] [-t <MIME_TYPE>]
[-c <CATEGORY> [-c <CATEGORY>] …]
[-e <EXTRA_KEY> <EXTRA_VALUE> [-e <EXTRA_KEY> <EXTRA_VALUE> …]
[-n <COMPONENT>] [-D] [<URI>]
am instrument [-e <ARG_NAME> <ARG_VALUE>] [-p <PROF_FILE>]
[-w] <COMPONENT>

For example, to start the Contacts application you can use
> adb shell am start -n com.google.android.contacts/.ContactsActivity

Eg. verify Music player launcher performance

erin@midnight:~/$ adb shell am instrument -w -r com.android.music.tests/.MusicPlayerLaunchPerformance
Eg. verify Music player stress test

erin@midnight:~/$ adb shell am instrument -w -r com.android.music.tests/.MusicPlayerStressTestRunner
Here is a video about running a cts test case by Android emulator!