MQTT in production: topic design, QoS, retain and Last Will

What MQTT does and does not guarantee

Through a broker, MQTT decouples publishers from subscribers and defines a message’s delivery assurance with QoS. It defines nothing else: how topic names are built, the format of the payload, the unit of a value, how long a message is kept or how a consumer should interpret a late message. These make up your project’s data contract.

The broker is also a design element: its outage can stop the whole flow. Redundancy, connection limits, queueing for offline clients and monitoring depend on the broker product you choose and should be assessed separately.

Topic design

A topic name is a hierarchical path and is case-sensitive. A few principles that work in production:

• Go from general to specific. For example site/area/line/equipment/measurement. Subscriptions such as “everything on this line” or “temperatures across all sites” can then be written with wildcards.
• Do not start with “/”. A leading slash creates an empty first level and is usually unintended.
• Use lower case, digits and hyphens; avoid spaces. A difference in case means two separate topics and leads to bugs.
• Do not put data in the topic. The topic identifies and locates; the value travels in the payload. Constantly changing parts such as an instantaneous value bloat the topic tree.
• Do not use wildcards when publishing. “+” and “#” are valid only in subscription filters.
• Do not use topics starting with “$”. They are reserved for the broker (for example $SYS) and are not matched by a root-level “#”.

The MQTT topic builder generates sample topics and subscription filters following these principles; for a long-term naming approach see the Unified Namespace article.

Choosing QoS

QoS applies per hop: publisher to broker and broker to subscriber. The level applied to the subscriber is the lower of the published message’s level and the level the subscription asked for. So even if a publisher uses QoS 2, a client subscribed with QoS 0 receives the message at QoS 0.

• QoS 0 (at most once): Suitable for frequently updated telemetry where the next sample replaces the last anyway. Loss must be acceptable.
• QoS 1 (at least once): The usual choice for events that must not be lost. A message may arrive more than once, so the consumer must be able to process a duplicate safely (idempotency).
• QoS 2 (exactly once): Needs a four-step handshake and is slower. Consider it in the few cases where duplicates cannot be handled on the consumer side.

QoS alone does not protect data during an outage: if the connection is down and there is no client-side queueing, the message cannot be published at all. For an outage buffer see the Store & Forward guide.

Retain and Last Will

A message published with the retain flag is stored by the broker for that topic and delivered immediately to a client that subscribes later. This suits “last known state” topics (equipment state, mode, configuration); it does not suit event streams, because a new subscriber only receives the most recent event. To delete a retained message, publish an empty payload with the retain flag to the same topic.

The broker does not tell you how old a retained value is, so the payload should carry a source timestamp.

Last Will is a message defined when a client connects and published by the broker when the client disconnects unexpectedly (for example when the keep-alive period expires). A common pattern: on connecting the client publishes “online” with retain, and its Last Will is “offline” with retain. A consumer can then read from the same topic tree whether the data source is alive. The Last Will is not published on a clean disconnect, so the client must publish “offline” itself when it shuts down.

Sessions, identity and reconnection

Every client needs a unique client ID: if a second connection arrives with the same ID, the broker disconnects the existing one. Two flows accidentally sharing an ID end up in a loop of constant disconnects and reconnects.

With a persistent session (cleanSession=false in MQTT 3.1.1; clean start plus session expiry in MQTT 5.0) the broker can queue QoS 1 and 2 messages while the client is offline. Queue limits depend on the broker; do not assume unlimited storage. A short keep-alive lets you notice a disconnect quickly but raises network load; the broker considers a connection lost if it receives no packet from the client for about one and a half times the keep-alive period.

On the client side retry reconnection with an increasing delay (backoff); otherwise all clients try to connect at once when the broker returns.

Payload and security

A practical starting point for the payload is a single JSON object: value, unit, source timestamp (UTC, ISO 8601), quality and schema version. If you use a specification such as Sparkplug B, the specification defines topic and payload structure.

On the security side, the conventional port is 1883 for unencrypted MQTT and 8883 for TLS-encrypted connections. Authentication (user name and password or a client certificate) and topic-level authorisation (which client may publish or subscribe to which part of the topic tree) are configured in the broker. Restricting each publisher so that it can write only to its own subtree stops a faulty client from corrupting other data. For network zones and certificate management see the article on secure data flows.