Back to Articles

Articles

Getting Mender OTA Working on Zephyr

Getting Mender OTA working with Zephyr and MCUboot, from device authentication and TLS problems to A/B firmware updates and rollback.

Mender OTA Updates are something I have mainly used with embedded Linux systems, normally with Yocto and an A/B root filesystem. However, Mender also has an MCU client, which made me curious about what it would take to use it with Zephyr.

The aim was to get beyond simply connecting a Zephyr device to a Mender server. I wanted to be able to deploy an update from Mender, download it onto the target, install it into an inactive firmware slot, reboot into the new image and retain the ability to roll back if the new firmware did not work.

This article describes the process of getting that working, including some of the (many!) things that did not work along the way.

Starting With an ESP32

Like most embedded engineers, I have a random selection of boards lying around my office. So, my first target was an ESP32 running Zephyr. I already had a working Wi-Fi example for this board, so it provided a convenient starting point for understanding how the Mender MCU client fitted into Zephyr.

The initial aim was just to authenticate the board with the Mender server. Before looking at firmware updates, I needed the basic pieces working:

• Wi-Fi networking
• DNS
• TLS
• a stable device identity
• persistent Mender authentication keys
• Mender client authentication

The first problems were around integrating the Mender MCU client into the Zephyr build. The Mender and cJSON sources were already available locally, but Zephyr did not automatically know that they were modules. Until they were correctly added to ZEPHYR_EXTRA_MODULES, the relevant CONFIG_MENDER_* and cJSON Kconfig symbols did not exist.

The CA certificate also needed to be included in the application. This was added to the build using generate_inc_file_for_target() so that the DER certificate could be registered with Zephyr’s TLS subsystem.

Initially I also tried to expose more of the Mender authentication process from the application itself. This was useful for understanding what was happening, but it duplicated functionality already provided by the Mender client and increased both the code size and memory requirements.

I eventually simplified this so that the application logged the useful stages, such as network setup, certificate registration and key handling, while leaving the actual authentication exchange to the Mender client.

TLS Problems

The next issue was with mbedTLS.

The first failure was in the PSA-backed private-key path. The configuration that worked was to keep:

CONFIG_MBEDTLS_PSA_CRYPTO_C=y

but disable:

CONFIG_MBEDTLS_USE_PSA_CRYPTO

This allowed key generation to work while avoiding the problematic PSA-backed PK export path on the Zephyr 4.2/ESP32 combination I was using.

Entropy configuration also caused problems and had to be made explicit before TLS would work reliably.

At this point I was still only trying to authenticate a device. There was no firmware update involved yet, but the exercise was already showing how many other parts of the system an OTA client depends on.

Running Out of RAM

The ESP32 then ran into memory problems.

There were several significant users of RAM:

• the Mender heap
• the Mender workqueue stack
• Zephyr network buffers
• TLS
• application and system thread stacks

I reduced the network pools for the relatively simple workload and removed deployment logging that was not required for an authentication-only test. I also initially configured Mender to share Zephyr’s existing heap and workqueue rather than allocating separate ones.

This got the image to build, but the resulting firmware then crashed very early during startup with a fatal CPU exception and an invalid stack pointer.

There was also a large amount of kernel mutex debug logging being produced before the application reached its own startup messages. Reducing the global log level from debug to info helped with the very tight configuration.

By this stage the ESP32 had been useful for learning how the Mender client fitted into Zephyr, but I came to the conclusion the ESP32, Mender and me were just not going to get along.

Moving to the FRDM-MCXN947

I had recently obtained a FRDM-MCXN947 for another project and that had more headroom, so it was time to switch boards…

The FRDM-MCXN947 version used Ethernet rather than Wi-Fi, so I had to make some changes.

One of the first things that needed changing was the MAC address configuration. The board configuration was using:

zephyr,random-mac-address

I was using the MAC address as the Mender device identity, so a random MAC was not suitable. The identity needs to remain the same across reboots, otherwise Mender has no idea that this was the same board as before.

The board overlay was changed to remove the random MAC address and use NXP’s unique MAC support instead.

This gave the application a stable Ethernet MAC address that could also be used as the Mender identity.

Getting Ethernet Working

The first Ethernet configuration was wrong.

I initially tried configuration symbols for the older NXP ENET driver before establishing that the MCXN947 board was using the ENET QOS driver.

The relevant configuration became:

CONFIG_ETH_NXP_ENET_QOS=y
CONFIG_ETH_NXP_ENET_QOS_MAC=y
CONFIG_NET_BUF_FIXED_DATA_SIZE=y

Once the correct driver was enabled, the application built, but Ethernet was still not working correctly. The driver reported errors including:

No new RX buf available

along with receive descriptor errors.

This turned out to be buffer starvation rather than anything related to Mender. Some of the networking values had been reduced quite aggressively while trying to fit the earlier ESP32 version into RAM, and those values were not appropriate for the ENET QOS controller.

I increased the receive buffers and adjusted the fragment and DMA descriptor configuration.

Waiting for the Ethernet Link

DNS resolution then failed intermittently with EAI_CANCELED.

Looking at the order of the log messages showed that the application was trying to use DNS before the PHY had actually reported that the Ethernet link was up.

Checking net_if_is_carrier_ok() on its own was not sufficient for the startup sequence on this board.

I changed the application to wait for the real Ethernet link before continuing with the IP and DNS configuration. DNS was also changed to retry transient resolver errors rather than immediately treating them as fatal.

With this in place, the board could finally:

• bring up Ethernet
• configure its IP address
• resolve the Mender server
• register the CA certificate
• select a stable MAC-based identity
• load or generate its authentication key
• authenticate with Mender
• activate the Mender client

The logs now showed:

Mender authentication accepted

This was the first major milestone on the MCXN947, it could talk to the Mender server and finally authenticate.

Creating a Mender Artifact

The next step was to create a Mender artifact containing the Zephyr firmware.

I added a script to package the firmware, assigned the device type frdm-mcxn947, and generated an artifact containing the normal zephyr.bin.

Mender accepted the artifact and it could be uploaded to the server.

However, this exposed an important difference between creating a Mender artifact and actually having an OTA update system.

At this stage I discovered I had not created a mechanism for installing the firmware. There was no second firmware slot managed by a bootloader and the application was not using Mender’s Zephyr image update module.

The artifact was valid, but the board could not use it to perform an OTA update.

To do that, the boot architecture needed to change.

Adding MCUboot

I changed the project to use MCUboot and Zephyr sysbuild.

The resulting update path was:

The Mender zephyr-image update module writes the new firmware into the inactive MCUboot slot and then requests a test upgrade.

The artifact also changed. Instead of packaging:

zephyr.bin

it now contained:

zephyr.signed.bin

with a Mender payload type of:

zephyr-image

This gave Mender an update payload that the Zephyr update module actually knew how to install.

Moving to Sysbuild

Using MCUboot meant that the bootloader and application needed to be built together, so the project moved to Zephyr’s sysbuild infrastructure.

This exposed several assumptions in the build scripts.

The artifact creation script originally expected the application configuration under:

build/zephyr/

With sysbuild, the application is a sub-build and the directory structure is different. The script had to be updated to understand both layouts and to prefer the MCUboot/sysbuild build when it was available.

I also initially tried using a custom MCUboot signing key. This introduced problems with the key path in sysbuild.conf and MCUboot’s public-key build requirements. For this demonstration there was no benefit in spending more time on the custom key, so I went back to MCUboot’s default signing key while getting the update path working.

An Invalid Primary Image

One particularly useful failure happened after MCUboot had been enabled.

I built and flashed the application using a normal application-only west build. MCUboot started, but reported:

Image in the primary slot is not valid!
Unable to find bootable image

The problem was that once MCUboot controlled the boot process, the application could no longer be built and flashed as though MCUboot was not present.

The baseline image needed to be produced using the same MCUboot/sysbuild configuration as the OTA images.

The project now explicitly rejected a plain non-sysbuild application build when MCUboot is enabled. This avoids producing an image that compiles successfully but cannot be booted by the configured bootloader.

After flashing a complete sysbuild-generated baseline, MCUboot could boot the application and the Mender client could authenticate and start polling for deployments.

Memory Problems Again

Once the board started processing real deployments, memory usage became important again.

Looking through the generated configuration and map files showed several large allocations in the main SRAM:

• 64 KiB system heap
• about 36 KiB of network buffer payloads
• about 24 KiB of packet slabs
• about 12 KiB of transmit payloads
• 128 KiB mbedTLS heap

The MCXN947 also had a separate 96 KiB SRAMX bank that the application was not using.

Rather than continuing to reduce everything in the main SRAM, I moved the mbedTLS heap into SRAMX and reduced it from 128 KiB to 64 KiB.

The network pools and general heap were also reduced to values more appropriate for the actual workload, while Ethernet DMA-sensitive data remained in main SRAM.

SHA-384 support was disabled because the certificate chain being used only required P-256/SHA-256.

After these changes, main SRAM usage dropped from approximately 281 KiB to 92 KiB, with the 64 KiB TLS heap moved into SRAMX.

This gave the networking and application stacks considerably more room.

A Stack Overflow During Deployment Polling

The next failure happened when Mender checked for a deployment.

Zephyr reported a stack overflow in:

k_sys_work_q

Earlier in the project I had configured Mender to use the shared Zephyr system workqueue to save RAM. That had been adequate for the authentication test, but it was not enough once Mender started doing real deployment work.

I changed Mender back to using its own workqueue:

CONFIG_MENDER_SCHEDULER_SEPARATE_WORK_QUEUE=y
CONFIG_MENDER_SCHEDULER_WORK_QUEUE_STACK_SIZE=6

This is a good example of why reducing every memory setting is not necessarily an optimisation. Saving a few KiB by removing a dedicated stack is not useful if the shared stack then overflows.

With the separate workqueue in place, the deployment progressed further.

The Artifact Was Not Coming From My Mender Host

The board now received a real deployment from Mender and was given a URL from which to download the artifact.

It then failed to resolve:

s3.docker.mender.io

Initially this looked odd because I thought I had configured everything to use my Docker-based self-hosted Mender server. Actually reading the manuals, looking at the docker compose file and a lot of googling later, I had a correct server configuration that I could put into my local DNS server.

The device could now resolve both the local Mender service and the external artifact storage.

Running Out of File Descriptors

The next failure was:

Too many open files in system

By this stage the application had DNS resolver sockets as well as the connections needed by Mender and TLS. The configured limits were simply too low for the complete workload.

I increased them to:

CONFIG_NET_MAX_CONTEXTS=8
CONFIG_ZVFS_OPEN_MAX=8

This was another problem that only appeared once the whole update path was being exercised.

The Complete OTA Update

With these problems resolved, I was finally able to perform the complete OTA update.

The board booted through MCUboot and started the Zephyr application. The Mender client loaded its persistent authentication key, established a stable device identity, authenticated with the server and was able to poll for deployments.

When an update gets assigned, the board will download the zephyr-image artifact containing zephyr.signed.bin.

The Mender Zephyr image update module writes the new firmware to slot1_partition and requests a test upgrade using:

boot_request_upgrade(BOOT_UPGRADE_TEST);

The board then reboots.

MCUboot boots the new firmware in test mode. Once the new application has started successfully, the image is confirmed using MCUboot’s image confirmation mechanism.

If the new firmware does not successfully reach the confirmation stage, MCUboot retains the ability to roll back to the previous image.

This was the point at which I considered the experiment successful: the board could receive a deployment from Mender, install it into the inactive slot, boot the new firmware and confirm the update.

Lessons Learnt

Linux is my comfort zone. I understand microprocessor boards. Developing with Zephyr and microcontrollers is new to me, so there was a lot of learning along the way (which was part of the point)

If I was starting again, I would introduce MCUboot at the beginning.

The initial authentication work was useful for understanding the Mender MCU client, but the bootloader and flash layout ultimately determine what an installable firmware image looks like. Adding them later meant changing both the build and artifact format.

I would also establish a stable device identity early. Using the MAC address is convenient, but only if the MAC itself is stable across boots.

Memory layout is another area I would look at much earlier. TLS, networking and OTA all require significant amounts of RAM for an MCU. On the MCXN947, using SRAMX for the mbedTLS heap was much better than trying to force every large allocation into the primary SRAM bank.

Finally, I would test the complete network path rather than just connectivity to the Mender API. Successfully resolving and connecting to the Mender server does not prove that the device can reach the server hosting the artifact.

A Security Caveat

There is still one issue with this demonstration that needs highlighting.

The Zephyr configuration reported that there was no proper entropy device available for the MCXN947 in the version of Zephyr I was using. This meant that the TLS configuration did not have the entropy source I would require for a production system.

This does not stop the demonstration from showing how Mender OTA can work with Zephyr and MCUboot, but it does mean I would not use this exact configuration for a production device without first fixing the entropy source.

OTA is part of the security boundary of a device, so TLS, signing keys and random-number generation need to be treated as production requirements rather than optional improvements.

Source Code

The final project contains the Zephyr application, the FRDM-MCXN947 configuration, the MCUboot/sysbuild setup and the script used to generate the Mender artifacts.

The source is available here:

https://github.com/ming4real/mender-zephyr-demo

The final implementation is reasonably straightforward when viewed as a finished system. Most of the work was in finding all of the assumptions that only became visible as the project progressed from authentication, to artifact creation, and finally to a real OTA update.