A brief guide to testing#

To ensure quality in our build we test our build regularly and intensively. Different levels of testing are used depending on what area is under test.

More details at Test reporting tool

Types of tests#

build time#

At build time checks from the Yocto toolkit and custom checks from .bbclass files are used.

runtime (hardware agnostic)#

For this we use the Automated runtime testing feature from Yocto.

Those tests can be run on virtualized hardware, hence can be performed on every build change.

opt-out for hardware dependent tests

Instead of specifying the tests for a specific hardware, all tests are loaded but then skipped if the expected hardware is not used/present.

This is done using e.g. @skipIfNotQemu() or @skipIfMachine(...) decorators.

Cases needs to be added manually

To add a new test suite to a run, the file needs to be added to the TEST_SUITES variable of the image under test. See also the Yocto project’s variable documentation.

runtime (hardware dependent)#

For hardware dependent tests we either use Test exports <https://docs.yoctoproject.org/5.0.9/test-manual/runtime-testing.html#exporting-tests> or if run from a build environment our

$ devtool simplecore-test-target

integration

release testing#

for releases and release candidates we use scotty-test (see Run a tests on real hardware for more information).

The run tests, are based on runtime (hardware dependent) test exports.

SDK testing#

SDKs are tested we are using SDK testing functionality <https://docs.yoctoproject.org/5.0.9/test-manual/intro.html#testsdk> from Yocto.

Cases are picked automatically

All valid test case implementation placed under lib/oeqa/sdk/cases are picked automatically. There is currently no way to exclude tests from a run.

Testing extensions#

If the build is based on simplecore distribution, we offer a variety of helpful extensions, that can be used from within a test case or as a decorator.

You can find more information in the code base at our SimpleCore™

Reporting#

Reporting on a pull request#

If a pull request is run, the test results from runtime (hardware agnostic) and the SDK testing test cases are posted as a comment to your pull request.

The full build results can be downloaded from the actual run in simplecore-tools.

Limited storage time of artifacts

The full test results are only stored for up to 24 hours, after that they will be deleted automatically by Github

Reporting in simplecore-tools#

For all push builds runtime (hardware agnostic) and the SDK testing test cases are added to the permanent test report storage in simplecore-tools.

Those can be enhanced by runtime (hardware dependent).

Out of those reports our Test Report is generated.