Telemetry schema changes are inevitable: new sensors, renamed fields, unit corrections, derived metrics. If evolution is unmanaged, dashboards break silently and analytics lose trust.

1. Schema goals and constraints

Define what your schema must support:

  • backward compatibility window
  • low-bandwidth encoding options
  • easy parsing on constrained devices
  • clear unit and timestamp semantics

Optimization without explicit goals often creates long-term compatibility pain.

2. Version placement choices

Three common options:

  • version in payload field
  • version in topic path
  • version in content-type metadata

For MQTT telemetry, payload field plus documented topic lineage is usually easiest to operate.

3. Compatibility rules

Set and publish rules, for example:

  • additive optional fields are backward-compatible
  • field type changes are breaking
  • unit changes require new field or version bump

Rules should be machine-checked in CI.

4. Migration mechanics

When introducing a new schema:

  • dual-write old and new for transition period
  • validate parity in ingestion layer
  • migrate consumers incrementally

Never force all consumers to switch simultaneously unless system is tiny.

5. Validation gates

Add schema validation at multiple points:

  • device side before publish
  • broker-adjacent ingestion service
  • storage write boundary

Early rejection of invalid events protects downstream analytics.

6. Observability during rollout

Track rollout metrics:

  • events by schema version
  • parse failure rates by version
  • consumer adoption progress

Without these metrics, migration status is mostly anecdotal.

7. Deprecation and retention policy

Define when old versions are retired and how long data remains queryable. Historical comparability often requires conversion layers or normalized views.

Deprecation without timeline causes permanent legacy burden.

8. Governance process

Schema changes need lightweight governance:

  • proposal template
  • compatibility impact review
  • rollout plan and owner
  • rollback plan

Even a small process avoids expensive downstream breakage.

9. Concrete enforcement mechanisms

The rules above only hold if a machine enforces them. Each compatibility strategy has a concrete tool I actually reach for.

JSON Schema validation in CI. I keep the telemetry schema in the repo with an $id and an explicit version field, then run a validator (for example ajv or check-jsonschema) as a required CI step against a corpus of recorded sample payloads. A change that breaks an old sample fails the build:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://schemas.example.org/telemetry/reading/v2.json",
  "title": "SensorReading",
  "type": "object",
  "required": ["schema_version", "ts", "value"],
  "properties": {
    "schema_version": { "const": 2 },
    "ts": { "type": "string", "format": "date-time" },
    "value": { "type": "number" },
    "unit": { "type": "string" }
  }
}

Protobuf reserved field numbers. When telemetry is encoded as Protobuf, the dangerous mistake is reusing a field number after removing a field — old decoders will silently misread it. Marking removed numbers (and names) as reserved makes that a compile-time error instead:

message SensorReading {
  reserved 4, 7;
  reserved "legacy_flags";

  uint32 schema_version = 1;
  int64  ts_unix_ms     = 2;
  double value          = 3;
  string unit           = 5;
}

A schema registry with compatibility checks. For a fleet with many producers and consumers, I register each schema version in a registry (Confluent Schema Registry or Apicurio, depending on the stack) with the compatibility mode set to BACKWARD. The registry then rejects any newly published schema that would break existing consumers before it ever reaches production, which turns the compatibility rules from documentation into an enforced gate.

Final note

Telemetry schema versioning is a product reliability concern, not only a data team concern. A deliberate evolution process keeps devices, pipelines, and analytics aligned over time.

Sources

  • Imported from the previous version of the site (gaborl.hu). Original publication around 2024-05-04.