sdlc.dev

Kafka path · Building on it · Topic 9 of 12

Schemas and evolution

Not started

A Kafka record's key and value are byte arrays. The broker stores them without looking inside, so the format is an agreement between producers and consumers that Kafka itself does not enforce. Because a topic outlives many deployments of the services around it, that agreement changes over time while old records remain in the log. A schema registry turns the agreement into something checked automatically.

Why it matters: Kafka stores bytes and never looks inside them. Producers and consumers deploy on different schedules and old records stay in the log, so every change to a record's shape must be readable by code that is older or newer than the writer.

  1. 1Kafka only sees bytes

    Serialisers in the client turn objects into bytes and back. The broker does not check formats, so nothing stops a producer from writing something consumers cannot parse.

  2. 2A schema registry stores the contract

    Producers register schemas (Avro, Protobuf or JSON Schema) and put a small schema id in each record; consumers fetch the schema by id.

  3. 3Compatibility rules gate changes

    The registry rejects a new schema version that breaks the chosen rule: BACKWARD (the Confluent default), FORWARD, FULL, their TRANSITIVE variants, or NONE.

  4. 4The rule decides the deploy order

    With BACKWARD, upgrade consumers first; with FORWARD, upgrade producers first; with FULL, either order works.

1

How it works

The explanation, with examples you can run.

How a registry works

The common setup, with Confluent's Schema Registry:

  1. The producer's serialiser registers the record's schema under a subject (by default <topic>-value or <topic>-key) and gets back a numeric id.
  2. Each record starts with a version byte (0) and the 4-byte schema id, then the encoded data.
  3. The consumer's deserialiser reads the id, fetches and caches the schema, and decodes.

Registering a new version triggers a compatibility check against earlier versions of the same subject. An incompatible change is rejected at the producer, before any bad record reaches the topic.

Evolving a schema

Version 1 of an Avro schema:

{ "type": "record", "name": "OrderPaid", "namespace": "shop",  "fields": [    { "name": "orderId", "type": "string" },    { "name": "amountCents", "type": "long" }  ] }

Version 2 adds a field with a default:

{ "type": "record", "name": "OrderPaid", "namespace": "shop",  "fields": [    { "name": "orderId", "type": "string" },    { "name": "amountCents", "type": "long" },    { "name": "discountCents", "type": "long", "default": 0 }  ] }

A consumer on version 2 reading a version 1 record gets discountCents: 0. That makes version 2 backward compatible. Checking it against the registry's REST API (not run here; host and topic are examples):

curl -s -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \  --data '{"schema": "<version 2 as an escaped JSON string>"}' \  http://registry:8081/compatibility/subjects/orders-paid-value/versions/latest

Compatibility modes

ModeGuaranteesSafe changes (Avro)Upgrade first
BACKWARD (default)New schema reads data written with the previous oneDelete fields, add fields with defaultsConsumers
FORWARDPrevious schema reads data written with the new oneAdd fields, delete fields with defaultsProducers
FULLBothAdd or delete fields that have defaultsEither
*_TRANSITIVEThe same, against all earlier versions, not just the last
NONENo checksAnythingAll at once

Plain BACKWARD only compares with the latest version, so a consumer replaying a topic from the beginning may meet records written with much older versions; BACKWARD_TRANSITIVE protects that case. For a change that no mode allows (changing a field's type from number to string), create a new topic or a new record type and migrate consumers.

Without a registry

JSON without a schema works until someone renames a field. If you go without a registry, at least version your records (a schemaVersion field or a header), write consumers that ignore unknown fields, and never remove or repurpose a field in place. The same rules apply to events in any log or queue.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Under BACKWARD compatibility, you add a field discount_cents to an Avro schema. What must the new field have?
  2. 2.Why is BACKWARD (rather than FORWARD) the usual default for Kafka topics?
  3. 3.Is Schema Registry part of Apache Kafka?

0 of 3 right so far. Get all three to finish this topic.

Go deeper