Silicon Blade

Lesson

Understanding What BitBake Did

A practical diagnostic workflow for inspecting metadata, layers, packages, tasks, and recipe work directories

Practical Yocto Project DevelopmentUnderstanding the BuildLesson 11 of 22

This lesson is not a BitBake internals reference. There are manuals for that, and they are very good at becoming quite large.

This is about the questions you ask on a real project when the build does something you did not expect.

Questions like:

  • why does this variable have this value?
  • why was this recipe selected?
  • why is this .bbappend applied?
  • where did this file come from?
  • why is this package in my image?
  • why is this package missing?
  • what depends on this recipe?
  • what tasks will BitBake run?

The trick is to stop treating BitBake as a black box. You do not need to memorise every internal directory or generated file. You do need to know where to look when the answer matters.

Start With the Kind of Question

Most Yocto investigations fall into one of a few categories:

Metadata questions

What did BitBake think the variables, overrides, dependencies, and tasks were?

Layer questions

Which recipe or append file was selected from which layer?

Task questions

Which task ran, what script did BitBake generate, and what did the log say?

Package questions

Which package contains this file, and why did the image install it?

Once you know which kind of question you are asking, the right tool is usually fairly obvious.

Why Does This Variable Have This Value?

Use:

bitbake -e <recipe>

For example:

bitbake -e widget-tools | less

This prints the expanded BitBake environment for that recipe. It is large, but it answers questions that are very hard to answer by just reading one .bb file.

Search for the variable you care about:

DEPENDS=
PACKAGES=
FILES:widget-tools=
IMAGE_INSTALL=

For many variables, bitbake -e also shows where the value was set or changed. That is the useful part. It lets you see whether the value came from the recipe, a class, a distro configuration file, a machine configuration file, or a .bbappend.

If you only need one variable, the variables lesson introduces narrower tools and the assignment rules: Working with Variables.

Why Was This Recipe Selected?

Use:

bitbake-layers show-recipes <recipe>

For example:

bitbake-layers show-recipes busybox

This shows which layer provides the recipe and which version BitBake can see. If several layers provide recipes with the same name, this helps you confirm which one is actually available and from where.

Also use:

bitbake-layers show-overlayed

This shows recipes that are present in more than one layer where one overlays another. Overlayed recipes are not automatically bad, but they deserve attention. Sometimes they are deliberate. Sometimes they are a sign that a copied recipe has survived long after everyone forgot why it existed. That is rarely a good sign.

Why Is This .bbappend Applied?

Use:

bitbake-layers show-appends <recipe>

For example:

bitbake-layers show-appends busybox

This shows append files that BitBake found for the recipe.

If your append is missing, check:

  • the layer is enabled in conf/bblayers.conf
  • the append filename matches the recipe name and version
  • the layer’s BBFILES pattern includes .bbappend files
  • the append is under a path matched by that pattern

The recipe modification lesson covers this workflow in more detail: Changing Yocto Recipes. The point here is that show-appends tells you whether BitBake has even seen your change before you start debugging the contents of the append.

What Tasks Will BitBake Run?

Use:

bitbake -c listtasks <recipe>

For example:

bitbake -c listtasks widget-tools

This lists the tasks BitBake knows about for that recipe. It is useful when a class has added tasks or when you are not sure which task name to run directly.

To run a task manually, use the task name without the do_ prefix:

bitbake -c compile widget-tools
bitbake -c install widget-tools
bitbake -c package widget-tools

The task and class lesson explains the normal task flow: BitBake Tasks and Classes. Here, the practical point is that you can inspect and run tasks deliberately instead of rebuilding the entire image every time.

What Depends on This Recipe?

Use:

bitbake -g <target>

For example:

bitbake -g my-image

This generates dependency graph files in the current directory, including files such as:

pn-buildlist
task-depends.dot
recipe-depends.dot

pn-buildlist is often the first useful file because it is plain text and lists the recipes BitBake expects to build for the target.

The .dot files can be processed with Graphviz, but they can also be searched as text when you only need to answer a specific dependency question. I normally start with the smallest question I can ask rather than trying to admire the entire graph. The entire graph is usually not improved by admiration.

For example:

grep widget-tools pn-buildlist
grep widget-tools recipe-depends.dot

This helps you see whether a recipe is part of the build plan and what pulled it in.

Where Did This File Come From?

For files in built packages, use oe-pkgdata-util.

To find which package owns a target path:

oe-pkgdata-util find-path /usr/bin/widgetctl

To list files in a package:

oe-pkgdata-util list-pkg-files widget-tools-tools

To find the recipe that produced a package:

oe-pkgdata-util lookup-recipe widget-tools-tools

This is often much faster than guessing from recipe names. It also avoids the very common mistake of assuming the package name must be the same as the recipe name.

Package splitting is covered in the previous lesson: Recipes, Packages and Package Splitting.

Why Is This Package in My Image?

This can come from several places:

  • IMAGE_INSTALL
  • IMAGE_FEATURES
  • a packagegroup
  • RDEPENDS from another package
  • RRECOMMENDS, depending on image policy

Start with pkgdata and the root filesystem logs.

Useful commands include:

oe-pkgdata-util lookup-recipe <package>
oe-pkgdata-util list-pkg-files <package>

Then inspect the image metadata:

bitbake -e my-image | less

Search for:

IMAGE_INSTALL=
PACKAGE_INSTALL=
BAD_RECOMMENDATIONS=
NO_RECOMMENDATIONS=

IMAGE_INSTALL is the input you normally write. PACKAGE_INSTALL is closer to what the image task is actually going to install after expansion. You do not need to memorise every image variable, but seeing both values helps distinguish “I asked for it directly” from “it arrived through expansion or dependencies”.

Why Is This Package Missing?

The first question is whether the package exists.

oe-pkgdata-util lookup-recipe <package>

If pkgdata does not know about it, the recipe may not have been built, or the package name may not be what you think.

Then check whether the recipe generated the package:

bitbake -e <recipe> | less

Search for:

PACKAGES=
FILES:<package>=

If the package exists but is not in the image, inspect the image package selection:

bitbake -e my-image | less

Search for the package name in IMAGE_INSTALL, packagegroups, and the final package install list.

For files missing from a package, inspect the recipe work directory. That tells you whether the file was never installed, was installed but not packaged, or was packaged but not selected for the image.

Inspecting Work Directories

Recipe work directories are where BitBake leaves a lot of useful evidence.

They usually live under:

tmp/work/<machine-or-tune>/<recipe>/<version>/

Inside, useful places include:

temp/

Task logs and generated run scripts.

image/

The staged install tree produced by do_install(), normally the value of ${D}.

packages-split/

The files after do_package has split them into binary packages.

source/build directories

The unpacked, patched, configured, or built source trees, depending on the recipe and classes.

The exact directory names vary, especially with different build classes. Use bitbake -e <recipe> if you need the actual values of WORKDIR, S, B, D, ${RECIPE_SYSROOT}, or ${RECIPE_SYSROOT_NATIVE}.

Inspecting temp/log.* and temp/run.*

The temp/ directory is where many failed builds become much less mysterious.

You will often see files like:

log.do_compile
run.do_compile
log.do_install
run.do_install

The log.* file shows what happened when the task ran.

The run.* file shows the script BitBake generated for that task.

If a compile fails, read log.do_compile. If the command line looks strange, read run.do_compile. If installation puts a file in the wrong place, inspect run.do_install and then compare it with the image/ directory.

This overlaps deliberately with the later debugging lesson, which focuses on failure handling as a workflow: Debugging Yocto Builds. Here, the aim is to know which evidence exists and what question it answers.

Using devshell

Sometimes inspection is not enough and you want to stand inside the recipe build environment.

Use:

bitbake -c devshell <recipe>

For example:

bitbake -c devshell widget-tools

This opens a shell with the recipe’s environment set up. It is useful for checking:

  • compiler and linker flags
  • include paths
  • available native tools
  • source and build directory contents
  • whether pkg-config resolves the dependency you expected

Do not use devshell as a way to manually fix the build tree and pretend the recipe is fixed. Anything you type there is diagnostic. The durable fix belongs in metadata, patches, or source changes.

Task Signatures, Briefly

BitBake decides whether a task needs to rerun using task signatures.

You do not need to understand the whole signature system at this point. The useful introductory idea is this: a task’s signature is affected by the metadata, inputs, and dependencies BitBake believes matter to that task. When those inputs change, BitBake may need to rerun the task instead of reusing shared state.

When a task reruns unexpectedly, or does not rerun when you expected it to, the answer often lives in the variables and dependencies that feed that task. Start with bitbake -e, task logs, and the generated run scripts before diving into lower-level signature tooling.

A Practical Diagnostic Routine

When the build surprises you, use the smallest question that moves you forward.

For example:

  1. Which recipe or package am I actually talking about?
  2. Which layer provided the recipe?
  3. Which append files were applied?
  4. What is the final value of the variable I care about?
  5. Which task produced the wrong result?
  6. What does the task log or generated run script show?
  7. Did the file appear in ${D}?
  8. Did it appear in packages-split/?
  9. Did the image install the package that owns it?

That routine sounds slower than guessing. In practice, it is usually faster, because it avoids changing three unrelated things and then wondering which one made the build behave differently.

Summary

BitBake leaves a lot of evidence behind if you know where to look.

The main diagnostic tools from this lesson are:

  • bitbake -e <recipe> for final recipe metadata
  • bitbake -g <target> for dependency graph output
  • bitbake-layers show-recipes for recipe providers
  • bitbake-layers show-appends for applied .bbappend files
  • bitbake-layers show-overlayed for overlaid recipes
  • oe-pkgdata-util for package and file ownership questions
  • bitbake -c listtasks <recipe> for known recipe tasks
  • bitbake -c devshell <recipe> for an interactive recipe build environment
  • temp/log.* and temp/run.* for task evidence
  • recipe work directories for staged, built, and packaged output

Check your understanding

Quick quiz: understanding BitBake output

Check which tool answers which diagnostic question.

Question 1You need to know why a variable has its final value for one recipe. Which command is the best starting point?
Question 2You need to check whether a `.bbappend` is being applied. Which command should you reach for first?
Question 3A task failed and the terminal output is only a summary. Where should you normally look next?