OTA updates are high leverage and high risk. A weak update process can brick large parts of a fleet quickly. A strong one reduces support load and security risk while preserving device availability.

1. Update system requirements

Define non-negotiables:

  • authenticity verification
  • interrupted-update recovery
  • rollback support
  • staged rollout controls

If rollback is absent, update failures become incidents.

2. Image integrity and authenticity

Use signed manifests and image hashes. Device should verify:

  • signature chain
  • target hardware compatibility
  • version monotonicity policy

Do not trust transport channel alone for authenticity.

Concretely, this is the verification a device should perform before it ever marks a new image bootable. On ESP32 this maps to Secure Boot v2 plus signed app images; on ARM Cortex-M with MCUboot it maps to the image header, TLV trailer, and the signing key baked into the bootloader:

verify_candidate(image):
    if image.magic != EXPECTED_MAGIC:            reject   # not a valid image header
    if image.hw_id != DEVICE_HW_ID:              reject   # wrong hardware target
    if image.version <= running_version:         reject   # anti-rollback / monotonicity
    if sha256(image.payload) != image.hash:      reject   # integrity
    if not verify_sig(image.hash, image.sig, PUBLIC_KEY): reject   # authenticity
    accept

The public key lives in immutable bootloader storage (or eFuse on ESP32), never in the updatable application. The transport (TLS/MQTT) protects confidentiality and freshness, but the image signature is what actually authorizes execution.

3. Dual-slot or fallback partition model

Preferred pattern:

  • active partition (current firmware)
  • candidate partition (new firmware)
  • boot flag and health confirmation

Boot into candidate, run health checks, confirm success. If confirmation fails, revert automatically.

The A/B (dual-bank) flow I rely on, and the failure mode each step guards against:

1. Download new image into the inactive slot (B). Slot A keeps running the whole time.
2. verify_candidate(B). On failure, discard; A is untouched. No downtime.
3. Set boot flag = "try B once" (pending/test state), arm hardware watchdog, reboot.
4. Bootloader boots B because the flag is "try once".
5. B runs the health-check contract (see below) within a warm-up window.
   - health OK  -> app calls confirm(): boot flag = "B permanent". Update complete.
   - health FAIL or hang -> watchdog fires OR flag stays "try once".
6. On the next boot the bootloader sees an unconfirmed "try once" flag,
   marks B bad, and falls back to A automatically. Device recovers unattended.

The critical invariants: the confirm step is explicit and made by the application after it proves itself healthy — never by the updater before reboot — and a hardware watchdog covers the case where the new image hangs before it can either confirm or cleanly fail. This is exactly the "test/confirm/revert" state machine MCUboot and ESP-IDF's esp_ota_* API implement (esp_ota_mark_app_valid_cancel_rollback() is the confirm call on ESP-IDF).

4. Rollout strategy

Use rings/canaries:

  1. internal test devices
  2. small pilot subset
  3. gradual percentage rollout
  4. full rollout

Gate each stage by health metrics and error thresholds.

5. Health check contract

Post-update success criteria should be explicit:

  • boot completed
  • network connected
  • core services responsive
  • error rate below threshold within warm-up window

Without clear criteria, rollback logic becomes unreliable.

6. Handling partial connectivity

Many devices are intermittently online. Update agent should support:

  • resumable downloads
  • bandwidth throttling
  • schedule windows
  • deferred activation

Aggressive updates during weak links increase failure rate.

7. Operational visibility

Track rollout telemetry:

  • download success/failure by reason
  • install and boot outcome
  • rollback counts
  • firmware distribution across fleet

Visibility prevents blind rollouts.

8. Incident rollback protocol

Prepare a fast rollback path:

  • halt rollout centrally
  • force fallback image for affected cohort
  • isolate problematic hardware variants
  • publish incident summary and corrective action

Speed and clarity matter more than perfect initial diagnosis.

Final note

Safe OTA is mostly about process discipline and recovery design. Signed artifacts, staged rollout, and automatic rollback make firmware delivery sustainable at fleet scale.

Sources