1. Today’s topic

We build the path from a firmware commit to a safe release:

text 
source code
→ reproducible build in Docker
→ checks and unit tests
→ Flash/RAM size control
→ release bundle
→ firmware signing
→ HIL testing
→ staged rollout
→ OTA with rollback

The central idea: firmware is ready not because it once built on a developer’s computer, but because it can be rebuilt in a clean environment, checked automatically, identified, and deployed safely.

2. Why this matters

Without CI it is difficult to prove:

text 
- which ESP-IDF and compiler built the firmware;
- whether firmware.bin matches the Git commit;
- whether the image grew beyond the OTA partition;
- whether sdkconfig changed accidentally;
- whether NVS migration passed;
- whether rollback works;
- whether something broke in UART parser, watchdog, or phase detector.

3. Theory

CI, CD, and HIL

CI:

text 
compile
static checks
unit tests
size check
config check
artifact generation

CD creates the release bundle: firmware binary, manifest, SHA-256, signature, ELF/MAP, partition table, and release notes. HIL checks a real board: flash, boot, UART console, GPIO, I2C, CAN/RS-485, watchdog reset, OTA rollback, and power interruption.

Reproducible build

Pin:

text 
Git commit
ESP-IDF version
compiler/toolchain version
sdkconfig
component versions
partition table
build flags

Do not use latest for production. Use an exact Docker image/tag and lockfiles.

What to keep in the repository

text 
project/
├── CMakeLists.txt
├── sdkconfig.defaults
├── partitions_ota.csv
├── dependencies.lock
├── main/
├── components/
├── tests/
├── ci/
└── .github/workflows/

Do not store: build, private signing keys, device certificates, or production passwords.

Release bundle

Keep these for each release:

text 
app.bin
bootloader.bin
partition-table.bin
flasher_args.json
project.elf
project.map
sdkconfig
size.json
manifest.json
SHA256SUMS
release-notes.md

ELF is needed for backtraces, core dumps, and investigating old production crashes.

Signing is separate from building

A PR-job must not receive the signing key. Architecture:

text 
untrusted build job:
  source → unsigned firmware → tests
protected release job:
  download verified artifact
  check Git SHA
  sign firmware
  verify signature
  generate release bundle

HIL on Raspberry Pi

Raspberry Pi Zero 2W can act as a HIL orchestrator:

text 
USB → ESP32 UART/JTAG
GPIO → inputs through isolation
USB relay/MOSFET → DUT power
CAN adapter
RS-485 adapter
Python test runner

4. Common mistakes

  • Using latest.
  • Building a release on a laptop.
  • Keeping only .bin without .elf and .map.
  • Signing firmware in a PR-job.
  • Not controlling OTA-slot size.
  • HIL checks only ping.
  • The HIL-runner is exposed to untrusted code.
  • Updating the entire fleet immediately.

5. Practical assignment

Add firmware-ci.yml:

c 
name: Firmware CI
on:
  pull_request:
  push:
    branches: [ main ]
    tags: [ "v*" ]
permissions:
  contents: read
env:
  IDF_IMAGE: espressif/idf:v6.0.2
  IDF_TARGET: esp32
  PROJECT_NAME: control
jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          submodules: recursive
      - name: Build in ESP-IDF Docker
        run: |
          docker run --rm \
            -v "$PWD:/project" \
            -w /project \
            -u "$(id -u):$(id -g)" \
            -e HOME=/tmp \
            -e IDF_TARGET="${IDF_TARGET}" \
            "${IDF_IMAGE}" \
            bash -lc 'idf.py set-target "$IDF_TARGET" && idf.py build && idf.py size --format json2 > build/size.json'

Add a size budget: firmware must use no more than 85% of the OTA-slot.

6. What to try next

  • Add manifest generation.
  • Add SHA256SUMS.
  • Keep ELF/MAP/size.json.
  • Create a HIL smoke-test through pyserial.
  • Separate the unsigned build from the protected signing job.

Exercise

A pull request modifies the firmware build script. Draw the trust boundary that prevents its job from accessing the signing key or controlling the HIL runner.

Self-check criteria: Identify credentials, artifact verification, and runner access separately. Passing unit tests does not make arbitrary PR code trusted.

Show the supplied answer

Run the untrusted build and tests without signing credentials or direct HIL control. A protected release job verifies the artifact and Git SHA before signing; only reviewed trusted inputs are allowed to reach the hardware runner.

Exercise

Define the evidence needed to investigate a crash from an older deployed release and prove which binary was installed.

Self-check criteria: Use the matching historical ELF rather than a fresh build; include both reproducible inputs and deployed artifact identity.

Show the supplied answer

Keep the exact firmware binary, manifest, SHA256SUMS, ELF/MAP, configuration, and release identity tied to the Git commit/toolchain. Compare the device’s release identity with the retained bundle and use its matching ELF for the crash analysis.