From MQTT to AMQP and back

The Advanced Message Queuing Protocol (AMQP) and the MQTT Protocol (OASIS chose to no longer expand the acronym) are often seen as mutually exclusive choices, especially in the Internet of Things (IoT) realm.

MQTT started as a lightweight companion protocol to IBM’s MQ messaging middleware for allowing easy integration of industrial equipment with each other, and with backend enterprise systems (via MQ). It has a strong focus on minimal wire footprint. The “TT” stands for “telemetry transfer” and that’s still, in very many use-cases, one of its key purposes.

AMQP is a general-purpose message transfer protocol suitable for a broad range of messaging-middleware infrastructures, and also for peer-to-peer data transfer. It’s a symmetric and bi-directional protocol that allows either party on an existing connection to initiate links and transfers, and has rich extensibility and annotation features at practically all layers.

Both protocols share that they can be tunneled over Web Sockets and therefore function well in environments that restrict traffic to communication over TCP port 443 (HTTPS).

After observing the emerging IoT space and the usage patterns for a while, I believe the two protocols fulfill very complementary roles in a greater systems context, with some areas of overlap. This post aims to provide a very simple conceptual framework for seamless integration of both protocols that allows for both, and messaging infrastructures centered around their core principles, to be used concurrently in a system, and where their relative strengths are best leveraged.

The “protocol” portion, how bits are laid out on the wire and what gestures are available, isn’t as important here as the underlying conceptual foundation and the assumptions that drive the shape of the protocol.

MQTT Concepts

In MQTT, all messages are published into a shared topic space at the broker level. A “topic” in MQTT is a filter condition on the consolidated message stream that runs through the MQTT broker from all publishers.

Publishing topics have a hierarchical structure (a path through topic space) and filters can be expressed as direct matching conditions (topic name and filter expression must match), or the filter can use wild-cards for single or multiple path segments.

Every published message from any publisher is eligible for delivery into any client session where a subscription exists with a matching topic filter.

The wildcard subscription capability and the fact that filtering occurs exclusively on the topic path makes topics and subscriptions in MQTT fast and very powerful, which is probably MQTT’s greatest appeal, apart from the small wire footprint and the relative simplicity of the wire protocol itself.

In addition to dropping telemetry data into some backend system, MQTT is very suitable for fast and “online” dispatch and distribution of messages to many subscribers in scenarios where it’s feasible for the entirety of the consolidated published message stream to be inspected on behalf of all concurrent subscribers. That’s true for very many scoped scenarios where a group of terminals or things share a MQTT broker for publishing and subscribing to information related to the group’s collaborative work.

I’m stressing the “online” aspect, because MQTT’s notion of a subscription is more akin to a conditional receive operation observing real-time flow through a topic space than the kind of more permanent subscriptions (or equivalent constructs like exchange bindings) that you will find in non-MQTT brokers; in the latter, subscriptions generally take the form of explicitly created, server-managed virtual queues, with message retention and the receive-side features of queues, meaning that they can often also be shared across consumers.

MQTT’s “subscribe” gesture is much lighter weight. It establishes a filter context and simultaneously initiates and unbounded receive operation on that context. If session recovery is used, to scope of undelivered messages is that individual filter context. Subscribing is receiving. In some brokers, such an MQTT subscription context may indeed be backed by a volatile queue to allow leveling between fast and slow subscribers and to allow for caching of messages while a subscriber is temporarily disconnected and if session recovery is supported; but that’s an implementation detail, not an explicit construct.

Where MQTT truly shines is in fast, ad-hoc distribution of data to parties that are concurrently online and that all observe a shared information graph. It has facilities to deal with occasional brief disconnects and allows establishing a clear notion of delivery failure or success in the case of connection failures.

AMQP Concepts

The AMQP protocol is, in version 1.0, very intentionally agnostic to particular messaging system topologies. Earlier drafts, today still used as the primary protocol in some message brokers, had a very specific notion of the internal broker structure. The version that finally emerged out of the standardization process shed, to some controversially, all concrete notions of “queue”, “exchange”, or “binding” that had existed in prior drafts and replaced them with universal abstractions.

AMQP is (surprise?!) not a publish/subscribe protocol in that it doesn’t explicitly allow creating subscriptions into message streams and doesn’t even explicitly enable publishing a message stream with an expectation that there will be subscribers. AMQP doesn’t have MQTT’s topology concept of a topic graph, and it doesn’t have inherent subscribe and unsubscribe gestures. It’s a very robust and compact message transfer protocol.

The topology-agnostic stance of AMQP 1.0 has proved quite advantageous because it’s enabling a broad variety of messaging entities much beyond the constrained topology set envisioned in the early drafts. AMQP-enabled products and services offer bi-directional connectivity for IoT devices, access to queues and several variations of publish/subscribe topics, datacenter-scale event ingestion infrastructure, and universal routing brokers across numerous products and vendors, all with the same basic set of protocol gestures for establishing security contexts, flow control management, message transfer.

The price for that broad flexibility is that management of the messaging entities and acts like creating subscriptions are specific to the respective messaging service or product, although they are often based on a common pattern that has been established in the AMQP Management specification draft. AMQP treats all such entities abstractly as “nodes” that can be asked to transfer messages.

The versatility of AMQP allows modeling the MQTT “subscribe” gesture as a conditional (per extensibility) flow control gesture on top of AMQP’s “flow” performative, setting quasi-unlimited link credit; but the gesture wouldn’t quite be as compact. Paolo Patierno, for instance, has recently modeled MQTT gestures over AMQP.

Efficiency and Scalability

MQTT is very wire-efficient as a protocol and requires less effort to implement on a client than AMQP because it focuses on a particular set of patterns.

MQTT’s inherent concept of allowing subscribers to use prefix and wildcard filters to tap into the totality of the consolidated message stream flowing through a broker’s topic space is enormously powerful, at the cost of putting a natural ceiling on the scalability of such a broker since all ingested messages must ultimately be tested against all existing subscriptions; even though clever matching algorithms will obviously aim to minimize the actual work.

The scale-out and fail-over ability over a single topic space is similarly limited by how many messages can be reasonably replicated between distinct broker instances that each accept published messages and yet each need to probe the totality of the cumulative message stream against their local subscriptions. And if session recovery were allows behind a load balancer to any of the brokers, all existing sessions and their subscriptions along with related state need to be replicated across the brokers, which poses a consistency and capacity challenge as scale grows.

Those scale concerns are largely immaterial for “local” scopes where several hundred or even a few thousand endpoints exchange information; they become a greater concern in cloud-scale infrastructure.

As a pure publishing protocol, where MQTT is used by clients just to publish messages into an infrastructure, MQTT imposes no obvious scalability limits. If the MQTT topic space is constrained to a particular system tenant or partition so that each of these scopes effectively gets an isolated view, scaling the resulting “broker-lets” were also much easier, but less powerful. You give up all potential subscribers having access to the broker’s complete topic space.

AMQP is a more verbose protocol on the wire as a cost for its greater flexibility. For publishing messages to a node, you first need to establish a link, allow flow over the link, and then transfer the message. MQTT doesn’t need any of that ceremony. However, AMQP rewards longer-lived sessions with amortizing the initial handshake effort by anchoring the target address for the transfer on the link rather than on the individual transfer operation. The effect is that the actual bytes transferred over an longer-lived AMQP link may quite well be fewer than for equivalent MQTT publish operations, especially if the topic name is very long.

AMQP transfers the “topic name” once per link and then uses a numeric link reference for all transfers. MQTT includes the topic name with each publish operation. If the topic path includes a GUID string, the bytes-on-wire advantage of MQTT’s basic frame structure over AMQP’s is practically eliminated.

Since AMQP doesn’t have pub/sub semantics built-in, it’s not possible to say that a broker that uses it will scale better or worse as a matter of principle. The protocol doesn’t impose a particular structural model, which provides opportunities in either direction. With session recovery, AMQP has the same session-state replication challenge as MQTT (or HTTP) when used behind a load-balancer or in failover situation of a multi-node system, leading to both protocols’ spec promise of being able to provide exactly-once delivery semantics to be impossible to fulfill.

Competing Use-Cases

There is quite a bit of potential use-case overlap between the two protocols, and I already alluded to some of those; it’s worth calling some of these out:

• Pushing telemetry from clients into some processing infrastructure - this is the core “telemetry transfer” use-case and MQTT is very well suited for it because it’s very compact. AMQP allows for the same set of transfer reliability semantics as MQTT. AMQP allows for descriptive metadata to flow outside of the core payload on the message frame. With MQTT, such metadata like on indication about encoding and purpose of the payload, will have to be, and often is, mangled into the topic path in MQTT. The upcoming MQTT 5 specification will address this and also the lack of negative acknowledgements; the current version doesn’t allow communicating errors back to the publisher.

• Bi-directional communication between devices and a backend system - MQTT and AMQP are both session oriented protocols, which means they can both be used for a client connecting into a backend and for that connection to be held onto for ongoing communication back to said client. MQTT has an inherent client/server bias based on who initiated the connection, with legitimate gestures clearly affiliated with those roles. A server can’t just establish a publishing path and send a message to a client; the client must first create a subscription matching the topic the server wants to publish to. A server also can’t issue subscribe commands to a client within the scope of the existing protocol, which means it can’t actively ask to receive messages. AMQP is fully symmetric and permits server and client to initiate communication links and transfers from either direction.

Both protocols can and are practically used for both of these cases in production today. Which of the two is the better choice for a given use-case often might come down to minute use-case details and also particular capabilities of the respective environment.

If you periodically publish very small data packets from a device to a processing system and the transfer volume size is very important (like on cellular M2M contracts), MQTT is probably the best candidate in general. An exception may be the case where long-lived connections can be regularly maintained and the topic names run very long, which is a case that may favor AMQP transfers, as discussed earlier. If connections drop a lot, MQTT has a further edge because its connection setup requires fewer gestures. The wire footprint differences will in either case be overshadowed by the overhead imposed by the surrounding TLS security tunnel.

If you transfer large, pre-encoded and potentially compressed batches of time-series data (like Apache Avro containers), the relative wire-overhead difference between the two protocols vanishes into some fractional percentage of the total transfer and you will care more about reliable transfers with diagnostics feedback, and being able to describe and route the payload onwards by metadata and without having to crack it open. Then AMQP is probably the best candidate.

For the bi-directional scenario, it’s straightforward to model unidirectional flows in either direction as well as an RPC model on top of AMQP; AMQP is also suitable as a brokerless peer-to-peer protocol. These patterns indeed require jumping through some hoops with MQTT because the protocol isn’t symmetric; it’s obviously doable, but either requires extra ceremony and/or taking an “out of the box” approach beyond what’s in the spec.

Either protocols will work for these scenarios; the choice comes down to available infrastructure options and, if you have a choice, the use-case specifics.

Combining MQTT and AMQP

Overlapping scenarios aside, MQTT and AMQP both have domains where they’re clearly superior choices, respectively.

As explained above, MQTT really shines for flexible, easy, and fast data sharing amongst groups of mostly connected clients that all share access to a broker or a set of brokers. Here, MQTT competes not so much against AMQP, but against other data distribution protocols like DDS (which leans on UDP multicast as a broker of sorts), the UDP-based, local-network pub/sub distribution model defined for OPC UA, and MQTT’s own UDP-based MQTT-SN variant.

AMQP is becoming increasingly important for messaging inside business systems of all kinds, and also for federating messaging systems amongst each other, across datacenter boundaries, between different regions of a cloud system, or between different systems running on different clouds.

If you’re building larger scale IoT solutions, you might quite well end up using both protocols in the same system for their relative strengths - and doing so will surface some conceptual differences that need to be resolved.

Let’s assume, as an example, an architecture where we have several distinct physical sites, which each host several dozen of device endpoints of different kinds. To do their work, some of devices need to know about the respective state of some of the other devices. Low latency matters more than reliability. The devices also each emit high-frequency diagnostics information that can be evaluated to determine the device’s health and potentially even predict a future state.

Let’s further assume there’s a hub (or gateway) on each site that observes and selects information as it becomes available, and aggregates it for local use as well as for forwarding it to a centralized system that provides an integrated operations view across all sites. The centralized system feeds into a variety of distinct backend functions realized in other applications, including billing, servicing, and others.

In the assumed architecture, we’re going to pick a local MQTT broker to facilitate that data exchange between all the on-site devices with each other and the hub. The hub itself might play the role of that broker.

For the link between the hub and the “site integration gateway” in the backend system, we can pick AMQP or MQTT following the discussion about competing use-case scenarios above. Either will work, my personal preference would be AMQP because it explicitly allows for description of payloads outside of the payload, has proper error feedback for failed publications, and allows for multiplexed flows in either direction. The upcoming MQTT5 version will address some of these differences.

On between backend resources and across system boundaries, we will transfer messages using AMQP using messaging fabric made up of one or multiple enterprise message brokers.

Concept mismatch

For any concrete implementation, you’ll be limited by what protocol the chosen backend system permits for data ingestion. If that ends up being HTTP or even some messaging system with a product-specific protocol like Apache Kafka, the resulting concept mismatch is still similar to when you choose AMQP.

With either of these options, you will direct messages that were initially published via MQTT to some addressable resource that accepts them for processing. The address of that entity might be amqps://example.com/myqueue or https://example.com/gateway/partition-1.

The concept mismatch we now have at hand is that a message that has been published to a context-centric MQTT topic like /devices/35AF67B4/accelerometer, and that has been retrieved from the broker with a subscription filter /devices/+/accelerometer/ now needs to be routed to (and through) an infrastructure that isn’t as dynamic and context-centric as an MQTT topic graph, but prefers organization of messaging resources by other primary criteria such as partitions or routes, and may require a predetermined allocation of resources for capacity management and reliability reasons. And yet we don’t want to lose the MQTT context information encoded in the topic name, or the ability to route and dispatch based on that information.

Conceptual alignment

Luckily, that’s fairly simple if you look at the scope of an MQTT broker as being the equivalent to any single network-addressable messaging entity in some other message broker, at MQTT “topics” as string expressions that declare the message subject, and MQTT subscription filters to be matching expressions for these subject declaration.

With “queued pub/sub”, IBM’s MQ product has introduced such a separation where a named entity (a “stream”) can have its own isolated topic space. MQTT’s global topic space notion more resembles MQ’s prior “integrated pub/sub” model.

Taking that perspective, you can easily construct direct mappings from/to MQTT from/to other messaging models.

An MQTT broker’s topic space scope might therefore correspond directly to, say, a Topic in Azure Service Bus, or a Topic in mostly any JMS broker, or to a similar concept like an Address in Apache ActiveMQ Artemis.

The MQTT topic on each published MQTT message directly corresponds to the core message property field “subject” of an AMQP message. It’s a string that describes the message. Another candidate for that mapping might be the “to” property that designates the logical message destination. Either option works, I’m going to pick “subject” for the further discussion.

The subscription filter expression of an MQTT subscription corresponds to a filter expression inside a Subscription Rule for an Azure Service Bus Topic, or to a message selector for a subscriber on a JMS Topic, with these filters operating on the “subject” property of the message.

Integration

With that alignment model, a mapping bridge from MQTT to AMQP, like the hub in the described architecture, is now straightforward:

MQTT to AMQP

• The hub creates subscriptions to all aspects of the MQTT topic space in which it is interested on behalf of the backend systems it bridges towards.

• Whenever a message is received via MQTT for any such subscription, the bridge creates an AMQP message directed at the backend’s ingestion entity, setting the “subject” property to the topic name expressed in the MQTT message. The MQTT payload is copied into the AMQP message as-is as a binary array. The bridge may also set the content-type on the AMQP message and other metadata, if known through the MQTT subscriber configuration.

• In the backend system, further routing of the now-AMQP messages is done on a combination of a pattern match on the “subject” property and potentially other message properties that may be added by the bridge or other intermediaries.

AMQP to MQTT

• The hub receives messages from some producer that’s publishing messages directly or via routing to an AMQP entity associated with the hub. The way to address the MQTT topic space, and any subscribers on it, is to send a message to that entity (like a queue), and the hub will pull the message and publish it locally.

• The “subject” field from the received message becomes the topic name for the MQTT message published into the MQTT broker. The AMQP message body binary array content is copied into the MQTT payload as-is. There may be a suffix-rule that appends the content-type information from the AMQP message to the MQTT topic name, like mapping “application/json” to a “/json” topic name suffix as it is customary in some MQTT systems to allow for picking encoders in spite of lack of a descriptive metadata field in the present MQTT version.

Summary

While there are a few competing, overlapping use-cases between MQTT and AMQP, many larger systems will use a combination of both. Mind also that the mapping presented here applies similarly to other message broker API standards (like JMS), alternate protocols (like the legacy AMQP 0.9 protocol), and closed, proprietary products.

MQTT is a great companion protocol to more flexible “grown up” messaging protocols for moving telemetry and providing fast and ad-hoc, broker-based pub/sub capability in constrained contexts and amongst the participants in that context, with scenarios not at all limited to IoT.

AMQP is a very versatile, standardized messaging protocol that is used for a broad variety of different kinds of messaging capabilities from many different vendors and service providers.

Both protocols are actively developed further under the OASIS umbrella and both have been adopted as international ISO/IEC standards, which makes them a great foundation for reference and inclusion into other standards, as this happens, for instance in the context of the Universal Architecture standard created by the OPC Foundation for many industrial verticals.