Lesson

Analysing Changes Between Builds

How to use source control, task signatures, and buildhistory to understand differences between Yocto builds

Practical Yocto Project DevelopmentValidating and Operating a Yocto ProductLesson 26 of 26

Yesterday’s build and today’s build are different. Why?

That question appears after a layer update, a recipe change, a distro configuration change, a security fix, a CI rebuild, or occasionally after a build that everybody expected to be identical. It is one of the normal questions in a maintained Yocto project.

The important thing is to ask a more specific question. “What changed?” can mean several different things:

what did humans change in the source or metadata?
why did BitBake decide to rebuild something?
what changed in the generated packages, image, SDK, or release artifacts?

Those questions overlap, but they are not the same question. The earlier Understanding What BitBake Did lesson showed how to inspect BitBake’s view of metadata, tasks, layers, and packages. This lesson is about comparing one build with another.

1. What changed in the metadata?

Start with normal source-control evidence.

Git, a manifest, or whatever your project uses to pin layers tells you what humans changed in the source and metadata. It can show that a recipe was edited, a .bbappend was added, a layer revision moved, a patch changed, or the project configuration selected a different branch.

For example, a release review might start with:

git diff product-2.3.0..product-2.3.1 -- layers/project/meta-my-distro
git diff product-2.3.0..product-2.3.1 -- layers/project/meta-my-software

Or with a manifest comparison:

meta-openembedded: 4c1a...f92 -> 9d02...a17
meta-vendor-bsp:   91ab...772 -> 91ab...772
meta-my-distro:    6f0d...e14 -> 73ba...901
meta-my-software:  21cd...8aa -> 44bf...19c

This is not Yocto-specific magic. It is normal release engineering. Git answers questions about what changed in files under version control. A manifest answers questions about which layer revisions were selected.

That ties directly back to Reproducible Releases and Project Maintenance. If you did not record the old layer revisions, the comparison becomes much more interesting, and not in the good sense.

Source control does not tell you the whole story, though. A one-line metadata change may cause many tasks to rebuild, or it may change the final image in a way that is not obvious from the diff. That is where BitBake evidence comes in.

2. Why did BitBake rebuild something?

BitBake uses task signatures to decide whether a task’s previous output can be reused. At a practical level, a task signature is derived from the inputs BitBake considers relevant to that task: metadata, variables, task dependencies, source references, and other values that affect the task.

The useful model is:

task inputs
  -> task signature
  -> matching signature can reuse sstate
  -> changed signature means the task may need to run again

This is why an apparently unrelated metadata change can cause rebuilding. It may not be unrelated from BitBake’s point of view. A class may alter a variable used by many recipes. A distro feature may change PACKAGECONFIG. A dependency may change a sysroot. A machine setting may affect compiler flags or package architecture. If those values are part of a task’s inputs, the signature changes.

That also explains the relationship with shared state. Sstate is useful because BitBake can reuse task output when the signatures match. If the signature no longer matches, reusing the old task output would be unsafe. The build infrastructure lesson covered sstate as a performance tool: Build Infrastructure, Caches and CI.

Deleting tmp/ is not the normal diagnostic answer to an unexplained rebuild. It removes evidence and makes the next build slower. If the question is “why did this task rebuild?”, look at signatures.

Using `bitbake-diffsigs`

bitbake-diffsigs compares BitBake signature data and reports why two task signatures differ.

If BitBake has recent signature data for a recipe task, a common starting point is:

bitbake-diffsigs product-app do_compile

You can also compare two specific signature files from the stamp area:

bitbake-diffsigs \
  tmp/stamps/cortexa53-poky-linux/product-app/1.4-r0.do_compile.sigdata.abc123 \
  tmp/stamps/cortexa53-poky-linux/product-app/1.4-r0.do_compile.sigdata.def456

The paths are deliberately illustrative. On a real build, the machine, distro, recipe version, task, and hash values will match your build directory.

A simplified report might tell you:

Variable CFLAGS value changed:
  -O2 -pipe
  -O2 -pipe -DPRODUCT_FEATURE_DIAGNOSTICS

Dependency on task my-diagnostic-lib:do_populate_sysroot was added

That gives you a useful direction. The question is no longer “why did everything rebuild?” It becomes “what changed CFLAGS?” or “what metadata caused this new sysroot dependency?” At that point, the tools from the diagnostic lesson become useful again: bitbake -e, layer inspection, append inspection, and task logs.

Do not turn signature comparison into a ritual. Use it when the rebuild itself is the mystery. For release review, you normally care first about what changed in the product.

3. What changed in the resulting product?

buildhistory records information about build outputs so you can compare them between builds or releases.

Enable it in project or CI configuration:

INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"

With BUILDHISTORY_COMMIT = "1", the buildhistory output is committed after each build. That makes comparisons easier because the history itself is a Git repository.

Buildhistory can help expose changes such as:

installed package lists
package versions
package dependencies
image size
package metadata
file lists and other output details
differences between builds or releases

Use:

buildhistory-diff

or compare the buildhistory repository directly with Git when that is clearer. The tool does not decide whether a change is good. It makes the change visible enough that an engineer can review it.

A Small Product Change Example

Suppose Release A contains:

product-app 1.4
openssl X
busybox Y

Release B contains:

product-app 1.5
openssl Z
busybox Y
libfoo

The obvious question is why libfoo appeared.

Start with the output evidence. Buildhistory or the image manifest tells you that libfoo is now installed. It may also show that product-app gained a runtime dependency:

packages/cortexa53-poky-linux/product-app/product-app: RDEPENDS changed:
  added "libfoo"

images/myboard/glibc/my-product-image: installed-package-names.txt changed:
  added "libfoo"

Then ask what changed in the inputs. Git or the release manifest might show that product-app moved from 1.4 to 1.5, or that a .bbappend enabled a new PACKAGECONFIG option.

If the question becomes why product-app rebuilt, or why its package metadata changed, use bitbake-diffsigs against the relevant task. If the question is why the final image installed libfoo, use the package and image investigation workflow from Understanding What BitBake Did.

The Tools Answer Different Questions

These tools are complementary:

Git diff: What source files, recipes, configuration files, patches, or layer revisions changed?
bitbake-diffsigs: Why did this task’s signature change, and therefore why might BitBake rebuild it instead of reusing sstate?
buildhistory: What changed in the generated packages, images, SDKs, and related build output metadata?

In a useful CI system, these pieces reinforce each other. The release manifest tells you the exact inputs. Sstate behaviour tells you whether BitBake could reuse previous task output. Buildhistory and image manifests show what the build produced. Release review decides whether the differences are expected.

This is also how you debug regressions without guessing. If a package appeared, disappeared, changed version, or gained a dependency, you can work from product output back to metadata and then down into the specific task only if needed.

Troubleshooting Flow: The New Build Is Unexpectedly Different

When a new build differs unexpectedly, I would normally work through it like this:

Confirm the comparison is meaningful: same image, machine, distro, branch, and build configuration.
Compare the recorded layer revisions or project manifest.
Use Git to review source and metadata changes between the two builds.
Use buildhistory or image manifests to identify what changed in packages, versions, dependencies, image size, or artifacts.
If a package appeared or disappeared, work backwards through packagegroups, IMAGE_INSTALL, RDEPENDS, and RRECOMMENDS.
If the mystery is why BitBake rebuilt a task, compare task signatures with bitbake-diffsigs.
Use bitbake -e, bitbake-layers, and task logs only once you know which metadata or task needs inspection.
Record whether the change is expected, accepted, or a regression.

That is the practical reason for keeping release manifests, CI artifacts, buildhistory output, and useful logs. They turn “today’s image feels different” into an engineering investigation with evidence.