While I do co-chair the AMQP Technical Committee in OASIS, this blog post is not an official statement of the committee. It’s a summary of the work that is either being discussed amongst committee members or already in progress in form of current working drafts and aims to put the various work products into context.
The AMQP 1.0 messaging protocol standard was finalized in 2012 and achieved ISO/IEC JTC 1 blessing as ISO/IEC 19464 in 2014. Since then, the protocol has been adopted for a broad variety of messaging applications, from classic enterprise messaging to application telemetry handling to attaching IoT devices to the cloud.
One of the protocol’s core strengths is that it provides a very flexible foundation for creating multiplexed and reliable message transfer capabilities between peers, while leaving the interactions pattern choice wide open. AMQP can be used for unidirectional message transfers as well as for request/response interactions, and two connected parties can easily initiate new conversations from either end.
During the development of AMQP, the separation of the data transfer model from an initially envisioned rigid pub-sub topology model was a significant point of contention, and the history of AMQP-based products and services released since AMQP 1.0 was finalized shows that the decision to split these concerns was the right one. The broad variety of messaging patterns realized with AMQP today goes far beyond pub-sub message brokers.
After the initial release, the OASIS AMQP Technical Committee (AMQP TC) who owns the standard development had taken an intentional break from further development to allow for the protocol to prove itself, and much of the attention shifted to and still is on shared development efforts of AMQP protocol stacks. There are now compliant AMQP 1.0 protocol stacks for practically all major runtimes and platforms in use today, including support for embedded devices, and nearly all of those implementations are open source.
The concepts that had been split off from the core of AMQP during development, most notably the ability to manage entities of a messaging infrastructure via AMQP as well as a unified AMQP addressing model had been moved out into working draft documents that the committee held on to.
The “amqp://...” and “amqps://...” URI format that’s in fairly frequent
use today is defined in the “AMQP Addressing” working draft, some of the
request/response management interactions supported by several AMQP servers
and services lean on today are defined in the “AMQP Management” working draft.
Neither are formal standards as of today.
While the core AMQP TC was in sleep mode, the associated AMQP “BINDMAP” Technical Committee (BINDMAP TC) had defined and finalized the WebSocket transport binding for AMQP, and had then moved on to define a fully AMQP based binding for Java Message Service (JMS), in tight coordination with the Apache Qpid project’s JMS client implementation effort.
The AMQP TC has resumed a regular meeting schedule for about the past 18 months, during which it has produced a set of new working drafts based on new scenarios, or has worked on revisions of existing working drafts, which will be discussed in context in the following.
Out of these, two documents have now formally progressed to committee specification status: AMQP Anonymous Terminus for Message Routing, and AMQP Enforcing Connection Uniqueness, which are both required for the BINDMAP TC’s work on the JMS binding.
The Path Forward
The Apache Qpid JMS client is using AMQP 1.0 and is based on the current JMS binding specification draft, and it’s also working well against Apache Qpid. But all that doesn’t yet mean that the client is indeed interoperable.
The client does indeed not work well with other products and services beyond message transfers. Transfers are generally interoperable because they are fully covered by AMQP 1.0, but JMS is quite demanding.
Important features like message selectors are defined by JMS, and have similar implementations in many brokers, but there is no interoperable definition of message selectors that APIs and frameworks/runtimes outside of JMS/Java could lean on.
The JMS message selector specification also just provides very limited access to message metadata; it’s a least common-denominator approach. A broker can choose to implement JMS message selectors, but those, and their constraints, then necessarily bleed through into other runtimes and languages.
The JMS specification also requires implementation of APIs for creating volatile and durable objects inside the message broker topology. While some of the required operations are covered by the AMQP Management working draft, a modern and truly interoperable standard needs to provide such manageability outside of JMS in a manner that doesn’t only make sense for the idiosyncratic perspective of the JMS standard.
To provide messaging interoperability across products and services that works for any framework and runtime and goes beyond today’s message transfer capabilities, the AMQP TC will, over the course of the next 12 months, define a consistent set of layered specifications that build on the existing AMQP 1.0 specification and can therefore readily use existing AMQP 1.0 community implementations.
The goal of these specifications is to provide building blocks for AMQP implementers that allow for shared implementation of components without giving hard constraints for what can be done with AMQP. If you come up with a wholly novel messaging capability, it should be possible to implement it using AMQP, and if you need, for instance, a message filter model for that new capability, you should be able to use the message filter expressions specification and an implementation in your preferred language and runtime to realize it.
The AMQP 1.0 specification itself will remain unchanged, but the work will yield a handful of corrections and clarifications in form of an errata addendum.
Addressing
The most foundational new specification will be AMQP Addressing.
The core AMQP 1.0 specification alludes to an addressing model and to a notion of “AMQP networks”, but largely stays away from specifics. Many addressing elements in the core spec, including the “to” and “reply-to” bare message properties are effectively just strings expressions open to application-level interpretation. At the AMQP link level, source and target addresses generally correspond to a terminus on a node, and while nodes are named and the source’s or target’s address field usually corresponds to that name, the relationship is kept fairly loose. The core specification never makes a hard commitment to what an address is beyond being a string.
The AMQP Addressing specification will clarify the “AMQP network” notion introduced in the core specification and provide a three-level addressing model that covers
- network-level addressing of AMQP containers,
- logical addressing of containers when routing inside an AMQP network, and
- addressing of nodes inside an AMQP container.
These three levels of addressing will be expressible in URIs, either singly or in combination.
The introduction of logical addressing will be a significant addition to the AMQP arsenal, because it will allow for routing of link routes and messages across federated AMQP containers based on application-level routing rules, and largely independent of underlying IP networks.
For instance, logical routing will permit for AMQP traffic to safely cross IP segment boundaries, and AMQP networks can safely span across many segmented IP networks, which enables secure communication in environments where security is of paramount importance such as in safety critical infrastructure in transportation, industrial manufacturing, or energy production and transfer.
Link Pairing
The Link Pairing specification extends the unidirectional link model of AMQP with a simple mechanism that allows pairing links into bi-directional channels with clear rules for realizing request/response or other bi-directional communication patterns.
Link pairing will become particularly powerful when combined with the aforementioned addressing model, because it allows for pairing links along a logical route. On AMQP infrastructure that support it, applications will be able to establish fully bi-directional transfer routes between parties which sit on opposite far edges of segmented networks, beyond TCP/IP peer-to-peer reach, and whereby the settlement of transfers between those parties is direct.
In effect, link pairing provides a bi-directional data transfer tunneling capability, with full flow control, between any two points inside an AMQP network.
Request Response Annotations
The Request Response Annotations specification provides a functional equivalent to address masquerading as provided by network address translation (NAT) in IP networks.
In many environments, it’s desirable for the initiation of conversations to be reserved to particular parties. The other parties that respond to such conversation initiations may respond within the scope of an existing conversation, but they may not open a new one themselves. To prevent this, the routing information given to the other party should be crafted such that it is only useable in the context of that particular conversation and cannot be used for initiation.
In addition, privacy or trade-secret protection concerns might make it desirable for the identity of the initiating party and even the identities of all routing intermediaries to be masked such that that the receiver will only know the identity of its most immediate AMQP network routing partner. For instance, the unmasked routing information of maintenance telemetry information emitted by sensors on bearings inside robots might disclose most of the internal structure of a car manufacturing plant when aggregated by the bearing manufacturer, which the car manufacturer will most definitely not be comfortable with.
Routing annotations replace clear text routing information with encrypted routing cookies that need stepwise resolution as information is routed back to an initiator.
Filter Expressions
Many routing scenarios need to make decisions about whether a particular message can be routed, or is eligible for storage, or shall be eligible for a particular transfer request.
The AMQP Filter Expressions specification defines simple and fast matching filters for each of the AMQP message sections, and a set of logical meta-filters to combine them. In addition it defines a filter syntax that leans on the SQL-92 condition expression syntax just as JMS message selectors do, but AMQP SQL filter expressions provides full access to all details of AMQP messages.
The filter expressions specification simply provides new data types of the AMQP archetype filter, meaning that those filters can be used anywhere where such filters are expected. For instance, a manageable entity that has the notion of durable subscriptions may use one or more such subscription filters to define which messages are eligible to be selected for the subscriber.
Management
The core AMQP Management specification will provide a foundation for creating, reading, updating, and deleting any kind of entity belonging to a messaging topology inside an AMQP container.
The common operations and base metadata elements are concrete enough for a generic management client application to provide a user experience for basic handling of all entities inside an AMQP container, even with complex relationships of such entities, but it is intentionally not specific about the concrete shape of these entities, i.e. it does not define what a “queue”, a “topic”, a “partition”, or an “access control rule” is.
A specification like the AMQP JMS binding might mandate support for a particular set of entities with a particular minimal set of attributes. The metadata model allows for such extensibility. A messaging entity can expose its particular metadata and also expose a mapping to the required JMS metadata in parallel if so desired, or the AMQP container could offer up the management of JMS specific entities that are projected onto native capabilities.
As the management specification already leans on parts of HTTP, this work stream might also spawn a more formal mapping of HTTP content and semantics onto AMQP.
Claims Based Security
The Claims-Based Security (CBS) specification extends the AMQP security model with a mechanism that allows for securing AMQP entities with claims-based authorization rules (with claims carried in the form of security tokens).
The SASL handshake defined by AMQP 1.0 does establish a security context scoped to the connection but doesn’t allow for a differentiated access control scheme where an external access control management service might grant a client access to one node inside an AMQP container but not to another.
The alignment of security contexts with connections also makes it impossible for a client to establish a route to a secured destination on an AMQP network, because connections are scoped to containers.
The CBS specification enables two new security options, both of which can be combined with all existing SASL mechanisms:
- it allows associating security tokens with any AMQP node inside of a container and scoped to a connection such that any link created to one of those nodes is using the cached token. Expiring tokens can be refreshed in the cache and can be revalidated for all existing links without the links having to be torn down.
- for routing scenarios, it allows associating security tokens directly with a link. The token is passed onward as the link route is resolved and must be accepted by the route target for the link to be established. The lifetime of a link route is limited to the lifetime of the token.
Participation
Participation in defining and reviewing these specifications is open to all employees of OASIS member companies and to all individual OASIS members, and the AMQP TC is also open to receiving feedback from everyone else. The committee home page contains all required contact information and links.
The backlog of pending work items for the AMQP TC is tracked in the AMQP TC Wiki.