Lesson

Build Infrastructure, Caches and CI

How downloads, shared state, mirrors, clean builds, and CI fit into a maintainable Yocto workflow

Practical Yocto Project DevelopmentMaintaining a Production Yocto ProjectLesson 21 of 26

Once a Yocto project moves beyond one developer machine, build infrastructure starts to matter.

The aim is not to make every build painfully slow in the name of purity. The aim is to make development builds fast enough to use, CI builds trustworthy enough to gate changes, and release builds controlled enough that you know what shipped.

That balance matters. A perfectly clean build that nobody can afford to run is not very useful. A very fast build that silently depends on random machine state is also not very useful. We are looking for the boring middle, which is where most good engineering lives.

Downloads and Shared State

Earlier, we set up a shared directory because Yocto builds reuse a lot of data. The two important caches are:

DL_DIR: The downloads directory. This stores fetched source archives, Git mirror data, and other downloaded inputs.
SSTATE_DIR: The shared state cache. This stores reusable task output so BitBake can avoid rebuilding work that has already been done.

The usual pattern is to keep these outside the build directory:

DL_DIR = "${HOME}/yocto/shared/downloads"
SSTATE_DIR = "${HOME}/yocto/shared/sstate-cache/${MACHINE}"

The setup lesson introduced this idea for a developer workstation: Setting up the System. In a production project, the same idea becomes part of the build infrastructure.

Yocto fetches a lot of source code.

Sharing DL_DIR avoids repeatedly downloading the same upstream source for every build directory, every developer, and every CI job. It also reduces your exposure to temporary upstream outages.

On a real project, I normally want a controlled downloads cache or mirror that CI can use. If upstream disappears, the build should not fail merely because a tarball moved or a Git server had a bad morning.

The shared state cache is about build output, not source downloads.

BitBake breaks a build into tasks. If a task’s inputs and signature match something already in sstate, BitBake can reuse the previous output rather than running the task again.

That is the useful conceptual model:

same task inputs
  -> same task signature
  -> reusable sstate object
  -> less rebuilding

You do not need to understand the internal storage format to use sstate well. You do need to understand that it is a performance tool, not a substitute for proper release control.

Trusting Shared Sstate

An sstate cache is an optimisation, not the definition of the build.

The cache should accelerate a reproducible build. The cache should not be the only reason the build works.

Sharing sstate between developers and CI is useful, especially on a large Yocto project. It can save hours of repeated work and make normal development much less painful. The cache still needs to be treated as build infrastructure, not as a mysterious directory that everyone points at and hopes for the best.

In practice, think about:

which Yocto release series the cache was created with
which project metadata and layer revisions it belongs to
which machines, distros, and package architectures are expected to use it
whether native output is compatible with the build hosts that will consume it
who is allowed to write to shared CI or release caches
how stale or unexpected cached results are investigated
when old cache entries are expired

BitBake’s signatures are designed to avoid reusing the wrong task output, but that does not remove the need for sensible cache policy. A cache produced by an untrusted build should not automatically become an input to a release build. A developer scratch cache should not quietly define what CI believes.

There are also times when invalidating cached output is the right decision. If a native tool changed in a way that affects output, a host environment changed, or you are investigating a suspicious cached result, forcing a recipe or a small set of recipes to rebuild can be useful. That is different from deleting every cache because the build did something surprising.

Periodically validate that the project still builds without relying on stale local state. That might mean a scheduled clean CI build, a release-candidate build from a fresh build directory, or a build that uses controlled downloads but starts with an empty sstate cache. The exact policy depends on build time and product risk, but the habit matters.

The reproducible release lesson covers the inputs you need to retain for a release: Reproducible Releases and Project Maintenance. If you need to understand why a task did or did not reuse sstate, use the signature-focused workflow in Analysing Changes Between Builds.

Mirrors

Mirrors let you control where BitBake looks for source.

Two variables are commonly involved:

PREMIRRORS: Locations BitBake tries before the original upstream URL.
MIRRORS: Fallback locations BitBake tries if the original fetch fails.

A project mirror can make builds faster and more reliable:

PREMIRRORS:prepend = " \
    git://.*/.* https://mirror.example.com/git/PATH;protocol=https \n \
    http://.*/.* https://mirror.example.com/sources/PATH \n \
    https://.*/.* https://mirror.example.com/sources/PATH \n \
"

The exact mirror layout is project-specific, so do not copy that fragment blindly. The important idea is that production builds should not rely entirely on the continued good behaviour of every upstream server.

Do Not Start by Deleting `tmp/`

When a build fails, one of the least useful instincts is:

rm -rf tmp

It sometimes makes the next build pass. It also destroys the evidence of what failed, throws away useful work, and teaches you very little. That is not debugging. That is making the crime scene tidier.

Start with the failing task, logs, generated run script, and recipe work directory. The debugging lesson covers that workflow: Debugging Yocto Builds.

There are times when removing build state is appropriate, but it should be a decision, not a reflex.

Cleaning Tasks

BitBake provides cleaning tasks with different levels of force.

clean: Removes most build output for the recipe so tasks can be rebuilt.
cleansstate: Removes build output and shared state for the recipe, forcing BitBake to recreate sstate for that recipe.
cleanall: Removes build output, shared state, and downloaded source for the recipe.

The commands look like this:

bitbake -c clean myapp
bitbake -c cleansstate myapp
bitbake -c cleanall myapp

Use clean when you want to force a recipe’s local work to be rebuilt.

Use cleansstate when you have a good reason to distrust the sstate for that recipe, for example while debugging a bad cached result or validating that a change rebuilds correctly from tasks.

Use cleanall when you also need to remove the fetched source, for example when testing fetch behaviour or dealing with a corrupted download. It is rarely the right first tool.

CI Build Environments

CI should make the project harder to break.

That usually means:

building from committed metadata
using pinned layer revisions or a manifest
starting from a known host environment
using controlled shared downloads and sstate
retaining logs and artifacts
running checks that match the product risk

The build host matters because Yocto relies on host tools. You can control that with dedicated build machines, virtual machines, or containers. The exact choice depends on the organisation. The goal is the same: the CI build environment should be boring and repeatable.

Containers are useful, but they are not magic. You still need to manage storage, permissions, supported host distributions, network access, and cache policy.

Clean Release Builds

A release build should be controlled, but that does not always mean it must avoid all caches.

There are two separate questions:

Did the release build use the correct inputs?
Did the release build avoid hidden local state?

Shared downloads are usually fine, provided they are controlled and match the expected source checks. Shared sstate can be fine too, but some projects choose to do release builds with an empty build directory and carefully controlled caches so the result is fast enough while still being reproducible.

For high-confidence release work, record:

CI job identifier
build host or container version
layer manifest
image and machine
key configuration
artifact checksums
logs
license and SBOM output

That gives you evidence if you ever need to explain or reproduce the release.

A simple CI job might archive a directory named from the product, version, machine, and CI job identifier:

product-2.3.0-myboard-ci-1842/

Inside that directory, keep the image artifacts, build logs, layer manifest, license output, SBOM output, CVE reports, and checksums. Checksums are cheap and useful:

cd tmp/deploy/images/myboard
sha256sum my-image-myboard.wic.gz my-image-myboard.wic.bmap > product-2.3.0-myboard.sha256

If the build generates SDK installers or package feeds that are part of the release, checksum and retain those too. The goal is not to keep everything forever; it is to keep the things needed to identify, verify, and investigate the release later.

Retaining Logs and Artifacts

Do not only keep the final .wic or .tar image.

Useful artifacts include:

image files
kernel, device tree, bootloader, and firmware outputs
package feeds, if used
SDK installers, if released
build logs
task failure logs from CI
build configuration or manifest
license manifests
SPDX/SBOM output
CVE reports

Retention policy depends on product needs, storage cost, and regulatory requirements. The engineering principle is simple: keep enough to investigate an old release without rebuilding the entire world before you can even ask the first question.

Useful CI Checks

A production Yocto CI pipeline often grows over time.

Useful checks include:

parse the metadata
build key images
build the SDK
run recipe or package QA
verify no unexpected dirty workspace changes were used
check that release builds use fixed layer and source revisions
generate license manifests and SBOMs
run CVE checks
boot smoke tests in QEMU where practical
collect image artifact checksums

Do not try to build the world’s most elaborate CI system on day one. Start with checks that catch real failures, then add more as the project matures.

Determinism Versus Performance

There is always a trade-off.

Disabling all shared state can prove that a build starts from nothing, but it is slow. Reusing every cache aggressively is fast, but it can hide problems if the cache is poorly controlled.

In practice, I like a tiered approach:

developer builds use shared downloads and sstate heavily
CI builds use controlled caches and retain logs
release builds use pinned inputs, controlled hosts, and archived artifacts
periodic clean builds prove the project can still rebuild without relying on stale local state

That gives the team performance during normal development without giving up the ability to trust the release process.

Summary

Build infrastructure is part of project maintenance.

The main ideas from this lesson are:

DL_DIR stores downloaded source inputs
SSTATE_DIR stores reusable task output
shared downloads and sstate improve build speed and reliability
mirrors reduce dependence on upstream availability
deleting tmp/ is usually a poor first debugging step
clean, cleansstate, and cleanall have different levels of force
CI should build from controlled inputs in a controlled environment
release builds should retain logs, artifacts, and build metadata
deterministic builds and build performance need to be balanced deliberately

Question 1What does `SSTATE_DIR` primarily store?

Only downloaded source archivesOnly final image artifactsReusable build task output that can avoid rebuilding work

Question 2Why is deleting `tmp/` usually a poor first response to a build problem?

It destroys useful evidence and turns a specific problem into a slow full rebuildIt permanently deletes all source repositories upstreamIt disables BitBake parsing

Question 3Which command removes fetched downloads as well as build output for a recipe?

`bitbake -c clean <recipe>``bitbake -c cleanall <recipe>``bitbake -c listtasks <recipe>`

Downloads and Shared State

Why Share Downloads?

Why Share Sstate?

Trusting Shared Sstate

Mirrors

Do Not Start by Deleting tmp/

Cleaning Tasks

CI Build Environments

Clean Release Builds

Retaining Logs and Artifacts

Useful CI Checks

Determinism Versus Performance

Summary

Quick quiz: build infrastructure

Your privacy matters

Do Not Start by Deleting `tmp/`