Clemens Vasters

Football. Cloud. Tech. Politics. Pictures. Principal Architect for Messaging/Eventing at Microsoft.

Tim Kellogg from 2lemetry felt compelled to write a response to the MQTT analysis I posted Monday. Debate is good.

I believe there is a difference, however, between being directly and openly critical of the product of an public standards body that gets broadly advertised as the “de-facto” standard by the specification’s main sponsor and individual name calling.

Tim chose to describe me as “a man who prefers flame wars over professional dialog” based on that post and in spite of having civilized exchanges with me on Twitter in the past. This didn’t strike me as a particularly good example of professional dialog by itself, so I felt like I ought to call out that sentence on Twitter.   

So … Dear Tim, I spent quite the better part of a week implementing my MQTT 3.1.1 protocol core and I spent 3 days furnishing a critical, in-depth technical analysis of MQTT, including providing what I believe is required industry context from my vantage point about the role of a particular major vendor. If you find this personally offensive and the analysis and backstory portrayal challenges you personally into a “flame war”, I don’t know quite know what to say about that.

Tim also states “Obviously Clemens misunderstands the goals of MQTT”, which is a fine rhetorical attempt to blanket-disqualify of my analysis. Maybe that is even true is a very narrow sense in that I care very little about the immediate goals of the OASIS MQTT 3.1.1 Technical Committee and instead have a laser focus on solving the immediate problems brought to me by a range of large consumer and industrial goods manufacturing companies that want to get going with realizing their Connected Home, Connected Car, Industrie 4.0, or Smart Energy solutions at scale. In that sense, I may “misunderstand” the goals in that I actually disagree with some of the goals for the benefit of these customers.

MQTT has no facility for an application to hint at the payload content, carry a key hint for end-to-end encrypted payloads, provide a message subject or label for dispatching the message before decoding the payload, or to even carry a timestamp for when the message was sent on its way at the origin. I am personally convinced custom metadata extensibility is part of the table-stakes for any messaging protocol that I will recommend to my customers, even if Andy Piper seems to believe that this insistence and the remainder of my analysis is laughable (without linking to it!) and asking for “the kitchen sink”.

Another underhanded questioning of the correctness of my analysis was:

Indeed, I do allude to chat, which was actually meant as a reference to the Facebook Messenger scenario in the abstract. The MQTT post is 9500+ words and I scoped out the interactive scenario. I actually have reached out to my friends in Facebook engineering to get insight into the exact MQTT usage profile, but I can already say that an interactive chat application on a $500 mobile phone with a flat rate data contract on 4G/LTE isn’t quite the same as the two scenarios (connected car and smart meter) I mentioned in my write up. The car scenario and the data volume constraints I mention are not a fabrication, and the national roaming network hopping issues don’t exist for phones on voice contracts, because providing cross-carrier roaming is massively constrained by regulation. Apples and Oranges.

Back to Tim, he’s also making the somewhat surprising statement in response to my IBM references on MQTT that “IBM has mostly left it alone” in the standardization process. I’m afraid I can’t follow that claim looking at the TC chair and the specification editor list.

Tech

Tim also has technical responses, of course. Sadly the slightly miffed undertone (“almost valid”, “didn’t seem to take time to fully understand that.”) continues:

“One complaint that is almost valid is the variable 1-4 byte remaining length field […]”

In this paragraph he says that I’m asking to constrain remaining length to 2 bytes. I’m not, I offer 2 or 4 bytes: “It could also just say that the length indicator is either two or four bytes long and if bit 15 of the two-byte integer were set, it'd be a 31-bit integer and you need to factor in the next two bytes and just mask out the most significant bit.”. I fact, I’m ok with both the 7-bit encoding or fixed-length prefixes if their use only were consistent in the protocol.

On my critique of the version identifier, Tim says my proposal were for the “MQTT” string to “be just the raw 4 bytes without the prefixed length”. That again reflects quite cursory reading of what I really wrote, because I am quite specifically asking for a different shape of those 4 bytes and I ask to move them to the beginning of the MQTT stream: “the UTF-8 characters for 'T' and 'T' and a trailing version counter of two bytes as you'll want to give yourself some room for spec revisions, instead of presenting this were an extensible thing? And then put that 4-byte preamble at the beginning of the stream, so a server can pick the implementation stack as the connection opens?”

In the next paragraph he objects to me adding the underlying protocol overhead of IPv4/6, TCP, and TLS to the size of the MQTT message wire footprint, pointing at the effects of Nagling (see RFC896). Nagling means that a network frame gets stuffed with data until either the designated output frame is filled up or until timer elapses (often 200-500ms). Which I explicitly refer to in the respective paragraph in my post.

Yet, in the next paragraph he also objects to me pointing to an overflow and collision risk on the packet identifier with 64K in-flight messages. It’s difficult to argue nagling and minimal wire footprint and then object to that being pointed out as a risk. Tim says that a device with 100Kbyte would never have as many messages in flight. I agree. An industrial machine funneling consolidated sensor observations to an analysis backend quite well might and that might want to take advantage of a low-overhead protocol over a long-haul stream transport (Tim offers MQTT-SN, an IBM protocol not in OASIS, as an alternative, but that’s exactly not for TCP as per mqtt.org).

Coming back to the extensibility point, Tim argues that “content-type” were non-sensical because all common use of MQTT is using a particular set of conventions, like the payload containing UTF-8 text for metrics values provided on (nonstandard) system topics implemented in some brokers. I agree that convention is good, but even in the given use-case I may prefer expressing numbers as binary integers and there’s no way I can express that preference. Content-type negotiation is such a foundational concept today that it’s really hard to successfully maintain that argument.

Discourse

The notion I reject and which not only Tim, but also several others have expressed is that I ought to have brought these points to the Technical Committee during the comment period on MQTT 3.1.1 and that my blog and Twitter (and thus a debate out in the open) are not the right forum to voice such concerns. The suggestion is that I ought to take all this to OASIS so that it can be discussed on the mailing list – which is open to the public to view, but it’s obviously not quite as visible.

There is a very odd sense of entitlement being expressed here. When I take a public specification, especially one that’s getting so much marketing push, and implement it as a programmer/architect, I owe the Technical Committee absolutely nothing, irrespective of my place of work. My job is to create platform solutions for large scale business applications. I evaluate protocols, interpret and implement them, and move on to the next protocol. I’m not a protocol standardization diplomat. Microsoft is a large company, and we also have quite excellent people in these roles. My role or career doesn’t hinge on a particular protocol; if AMQP (of which it’s said that I’m underhandedly promoting it) were to fall off the face of the earth tomorrow I’d shrug and implement the next thing, if there were an adequate replacement.

When I find that the reality I find is in grave dissonance with the marketing claims, however, I believe that I owe the public and specifically colleagues, partners, and customers who come to me and seek my immediate advice, an in-depth analysis of what I found, and share that on the stage I have. Which is this one. You can go back more than 10 years here and find out, this is how I roll, before and during my Microsoft career. The OASIS TC can find this consolidated feedback quite well here and it looks like it made it to the mailing list alright.

It’s also been criticized that I bring this forward just after the public comment period of MQTT 3.1.1 finished and that that timing is suspicious. It’s not. Turns out that the sort of changes I am proposing are largely out-of-charter for MQTT 3.1.1. The TC charter is very clear:

Changes to the input document, other than editorial changes and other points of clarification, will be limited to the Connect command, and should be backward compatible with implementations of previous versions of the specification such that a client coded to speak an older version of the protocol will be able to connect to, and successfully use, a server that implements a newer version of the protocol.

That’s been reinforced by Nick O’Leary’s comment in March:

There was actually no point in bringing any of these points to the TC before 3.1.1 is done and they are gearing up to actually fix the protocol deficiencies, which I am not alone in finding:

It’d be awfully nice if the folks invested in MQTT would be willing to have a serious technical debate and acknowledge that my analysis and proposals have merit instead of trying to ridicule it. I have an enormously thick skin for ad-hominem arguments; you can call me things if that helps the ultimate goal of getting to a solution that helps my customers.

I may have brought the firewood to the town square, but I didn’t light it.

Updated: