Lesson
Testing and Quality Assurance
How Yocto build QA, package tests, runtime image tests, and product validation fit together
The image built successfully. What does that actually prove?
It proves that BitBake could parse the metadata, run the required tasks, and produce the requested artifacts. That is necessary. It is not the same as proving that the generated product is correct, that the package metadata is clean, that the services start, or that the hardware behaves as expected.
That distinction matters on real projects. A build can complete while hiding a bad packaging decision, a disabled service, an image that only works under QEMU, or an application that starts but cannot do anything useful. Yocto gives us several places to catch those problems, but each one answers a different question.
Recipe
-> Build
-> Package
-> Package QA
-> Image
-> Runtime tests
-> Product validationThis lesson is about that chain. It is not a full testing course, and it is not a catalogue of every QA check OpenEmbedded can run. The useful skill is knowing what kind of evidence you have, what it proves, and where to look when it fails.
Build-Time QA
OpenEmbedded performs package QA checks as part of the normal build. These checks
are implemented through insane.bbclass, which adds quality checks to the
package generation process. In practice, you will often see the results around
do_package or do_package_qa.
The name is memorable, if not especially gentle. The checks are not there to annoy you. They catch common problems in the packaged output before those problems become runtime failures or strange release artifacts.
Representative QA problems include:
- files installed under
${D}but not included in any package - files installed into unexpected locations
- incorrect ownership or permissions
- runtime dependency problems
- shared library packaging issues
- target binaries with the wrong architecture
- binaries that have already been stripped
- patch warnings where source changes are not being applied cleanly
You do not need to memorise every check name. The important habit is to read the message as a clue about the metadata. A QA warning is usually telling you that the files produced by the recipe do not match what the recipe metadata says should happen.
For example, an installed-but-not-shipped error is not really about the error
text. It means a file exists in the recipe’s staged install tree, but no package
claimed it through FILES. The debugging path is:
- find the package and check name in the QA message
- inspect the path under the recipe work directory
- decide whether the file should exist
- fix the install path,
FILES,PACKAGES, or dependency metadata as needed
That connects directly to the packaging lesson: Recipes, Packages and Package Splitting. For general task logs and work directories, use the workflow from Debugging Yocto Builds.
INSANE_SKIP Is Not a Fix
INSANE_SKIP tells Yocto not to run a particular QA check for a particular
package. It does not fix the underlying issue.
That distinction is important enough to be slightly blunt about. If a QA check
reports a real packaging problem and you add INSANE_SKIP, you have not made the
package correct. You have made Yocto stop telling you that it is suspicious.
There are legitimate uses. Some products include third-party binary-only software where, for example, an unstripped binary is not available. In that case, skipping a specific check for a specific package may be a deliberate product decision. It should still be documented in the recipe or in the review history.
It should not normally be the first response to a QA failure.
A Small QA Example
Suppose a recipe installs a helper script like this:
do_install() {
install -d ${D}${libexecdir}/myproduct
install -m 0755 ${S}/factory-reset ${D}${libexecdir}/myproduct/factory-reset
}The build then fails during packaging with an installed-but-not-shipped error for
/usr/libexec/myproduct/factory-reset.
The tempting response is to hide the check:
# Bad response: this hides the symptom rather than fixing ownership.
INSANE_SKIP:${PN} += "installed-vs-shipped"That is the wrong fix. The file has been installed, and the package metadata does not say which package owns it. If the file belongs in the main package, say so:
FILES:${PN} += "${libexecdir}/myproduct/factory-reset"Now the metadata matches the installed output. The package QA check can keep running, and the package ownership is clear.
The same reasoning applies to other QA messages. If a shared library is in the wrong package, fix the package split. If a runtime dependency is missing, fix the runtime dependency. If a file has the wrong mode, fix the install command. Only skip the check after you understand why the check is not appropriate for that specific package.
Package Tests with ptest
Package QA checks the shape of the generated packages. It does not run the
software and decide whether it behaves correctly. For that, some components can
ship target-side tests using ptest.
A ptest packages tests with the component that they test. A recipe that supports
ptest normally inherits ptest.bbclass:
inherit ptestThe recipe may also need to build test binaries, install test data, or provide a
run-ptest script. Once included in an image, ptest files are normally installed
under a directory such as:
/usr/lib/<package>/ptestThe ptest-runner package then runs the installed ptest suites on the target.
For an image used for package testing, you might see:
DISTRO_FEATURES:append = " ptest"
EXTRA_IMAGE_FEATURES += "ptest-pkgs"
IMAGE_INSTALL:append = " ptest-runner"That belongs in a test or CI image, not usually in the production image you ship. ptest packages add size, test dependencies, and extra files that a deployed product normally does not need.
ptest is useful when a library, command-line tool, or service already has a reasonable target-side test suite. It is less useful when the component only works with real peripherals, requires complex external systems, or has tests that are unreliable in an embedded environment. Flaky ptests are still flaky tests; putting them in Yocto does not make them noble.
Image and Runtime Testing
There are several different levels of testing in a Yocto product, and it is worth keeping them separate.
- Recipe and package QA
Checks whether the recipe output and generated packages obey expected packaging rules.
- Package-level runtime tests
Runs tests that were packaged with a component, commonly through
ptest.- Image tests
Boots or accesses an assembled image and checks basic runtime behaviour.
- Product and system tests
Checks whether the actual product satisfies its requirements on representative hardware.
The testimage class is Yocto’s standard entry point for automated runtime image
testing. For QEMU-capable machines, it can boot the image and run tests through
the OpenEmbedded test framework. The tests run against the target system, often
over SSH, which means they can check behaviour that packaging QA cannot see.
A common manual setup is:
IMAGE_CLASSES += "testimage"Then run:
bitbake -c testimage my-imageIn CI you may instead see:
TESTIMAGE_AUTO = "1"That asks the tests to run automatically after the image has been created.
Image tests are good for smoke testing. They can check that the image boots, that networking works well enough for the test environment, that expected services are running, that an application responds, or that the package manager behaves as expected in images where runtime package management is enabled.
QEMU is particularly useful here because it gives fast feedback. It is also not a real board. Anything involving device trees, bootloader interaction, firmware, storage timing, field buses, radios, sensors, power behaviour, or product enclosure assumptions still needs hardware validation.
What This Looks Like in a Real Project
A production project usually ends up with several images or build modes.
A development image might contain SSH access, debug tools, package manager support, ptest packages, test users, and logging that would be inappropriate in a released product. That image is useful for engineering work, but it is not the same artifact as the production image.
A CI pipeline might build both:
- a test image, used for QEMU boot checks and ptests
- a production image, used for release artifact generation and final validation
The important point is to record which one was tested. “CI passed” is much less useful than:
image: my-product-image
machine: myboard
distro: mydistro
version: 2.3.0
tests: package QA, QEMU smoke tests, hardware boot testThat record does not have to be elaborate. It just needs to be good enough that six months later you can tell what was built, what was tested, and where the gaps were.
What Different Tests Prove
Different tests prove different things.
- “BitBake completed” means the build completed.
- Package QA checks whether generated packages obey expected rules.
- Runtime tests check behaviour.
- Product testing determines whether the actual system satisfies its requirements.
Confusing these is how projects accidentally treat a successful build as a validated product. Yocto can give you useful evidence at several points in the chain, but it cannot decide what your product requirements are.
What to Check in a Real Project
When reviewing a Yocto product, I would normally check:
- whether QA warnings have been fixed or merely suppressed with
INSANE_SKIP - whether any suppressed QA checks are narrow and documented
- whether important components have package or runtime tests
- which tests run in CI for normal changes
- which checks run against QEMU images
- which validation steps require actual hardware
- whether the release record says exactly which image was tested
That is the engineering message of this lesson. A successful build is the start of the evidence, not the end of the argument.
Check your understanding
Quick quiz: testing and QA
Check what different Yocto checks and tests actually prove.