.NET Services March 2009 CTP - Service Bus Routers And Queues - Part 2: Queue Policies

In the previous post in this series I’ve discussed some of the foundational principles of the new Queue and Router features of the .NET Services Bus and the role that policies play in turning names in the namespace into messaging primitives. In this post I’ll start drilling into the capabilities of the Queue feature and will discuss what’s in the queue policy.

The capabilities of the Queue are best explored by looking at the Queue policy. Policies can be expressed in two ways: You can either you use the .NET Services SDK’s Microsoft.ServiceBus.QueuePolicy class with a .NET application or you can just whip up a bit of XML. The structure and the serialized representation of that class is exactly equivalent to an XML queue policy you’d wire up “by hand” and thus I’ll discuss the policy in terms of its XML elements:

The queue policy has the element name “QueuePolicy” within the XML namespace “http://schemas.microsoft.com/ws/2007/08/connect”. That’s the namespace we use for most protocols and data structures in the .NET Service Bus at this development stage and you should expect that we’re migrating to the “final” V1 namespaces in one of the next CTPs. If you use the .NET class you’ll just have to recompile to snap to new namespaces once they come around.

Below is a list of the policy’s elements and the permitted values. All elements are optional and can appear at most once. The default value applies when an element is omitted. It’s perfectly ok to send an empty QueuePolicy and accept the default values to keep things simple. I’d recommend to always set the ExpirationInstant value, however.

One required forward reference: Don’t get confused or distracted trying to figure out from here what it means “to provide an authorization token”. The docs explain this and I’ll also address this in the next blog post when I dive into the protocol.

Authorization – The authorization setting indicates whether sender and/or consumers must provide a security token when interacting with the queue. You can explicitly permit the .NET Service Bus to allow anonymous consumers and/or anonymous senders on a Queue. If a token is required, a sender must present a token with “Send” permission that is issued by the .NET Access Control service and the consumer must present a token with “Listen” permission that is issued by the .NET Access Control service. The respective rules are set up in the .NET Access Control service on your project’s “Service Bus” scope. Mind that all anonymous traffic is attributed to the “manager” of a Queue, i.e. to whoever sets the policy; all traffic is metered and accounted for. The permitted values for this element are:
- Required – (default) Senders and Consumers must provide an authorization token to send/retrieve messages.
- RequiredToSend – Senders must provide an authorization token with a “Send” claim to send messages, consumers don’t.
- RequiredToReceive – Consumers must provide an authorization token with a “Listen” claim to receiver messages, senders don’t.
- NotRequired – Neither senders nor consumers must provide a token, i.e. the queue is set up for anonymous traffic.
Discoverability – The discoverability setting defines under which conditions the Queue and its policy are visible in the Service Registry Atom feed and any forthcoming discoverability mechanisms. When you are browsing the Service Registry feed from within a browser, the only setting that will make the Queue visible is Public since the browser isn’t able to attach an authorization token without some scripting assistance.
- Managers - (default) The Queue and its policy are only discoverable if you provide an authorization token with the request and the token carries a “Manage” claim.
- ManagersListeners – The Queue and its policy are only discoverable if you provide and authorization token with the request and the token carries a “Manage” and/or “Listen” claim.
- ManagersListenersSenders – The Queue and its policy are only discoverable if you provide and authorization token with the request and the token carries a “Manage” and/or “Listen” and/or “Send” claim.
- Public – The Queue and its policy are discoverable without any authorization requirement, i.e. the general public.
ExpirationInstant – This is an XML dateTime value indicating the TTL (time-to-live) for the Queue. You can make Queues long-lived or short-lived. A Queue’s lifetime may be extended by changing this policy value and reapplying the policy. Once a Queue expires it is removed from the system along with all the messages that reside in it. The system may limit the Queue lifetime in the effective policy that you get back from the create/update request, but we’ll typically allow at least 24 hours without renewal. This value has a default of 24 hours, a maximum of 21 days and a minimum of 30 seconds. The value must represent a future dateTime and must be expressed in UTC.
MaxMessageSize – This is a numeric value defining the maximum size of any individual message that can be accepted into the Queue. The maximum message size is expressed in bytes and includes all infrastructure-imposed and encoding-related overhead. The actual overhead varies significantly based on whether messages are sent as SOAP 1.1, SOAP 1.2, or plain HTTP frames using text or binary encoding and whether a security token is required. A very defensive assumption is to reserve 10-12KB for protocol overhead in complex cases with WS* Security; the minimal allocation for protocol overhead should be around 4KB. The default and maximum values for the maximum message size is 60KB (60*1024 bytes). The minimum value is 8KB. We suggest that the payload size for an individual message does not exceed 48KB, even though you can try to push it a bit. The expectation is that these values will trend up but I don’t see them more than doubling due to throughput, timeliness and scale considerations. If your data is larger you should consider how you can chunk it up.
TransportProtection – The transport protection setting defines whether senders and consumers must use HTTPS to interact with the Queue. This is the default setting. You would modify this to accommodate clients – typically on devices - that don’t do HTTPS all that well. Permitted values:
- AllPaths - (default) Any interaction with the queue requires HTTPS.
- None – Any interaction may be performed using either HTTP or HTTPS.
EnqueueTimeout – The enqueue timeout is an XML duration value that specifies how long an enqueue operation will hang if the queue is at capacity (full). The default value is 10 seconds, the minimum is 0 seconds, and the maximum is 60 seconds. If the timeout expires and the message could not be added to the queue during that time, the queue will act according to the Overflow policy setting.
MaxConcurrentReaders - [this setting not supported/enforced in the March 2009 CTP] The maximum concurrent readers value defines how many concurrent readers are permitted on the queue. If this numeric value is smaller than the default value of 2^31 (max int), the queue protocol will switch into ‘session mode’ that grants a limited number of read-locks on the queue. The minimum value is 1.
MaxDequeueRetries - [this setting not supported/enforced in the March 2009 CTP] The maximum dequeue retries value defines how often a message may be peek/locked and put back into the queue until it is considered poisonous, i.e. after how many retries it should be expected the the consumer will not be able to ever consume the message successfully, because it is malformed or the consumer experiences an error condition that requires some form of manual intervention (including a bug fix). If the message is found to be poisonous it will be sent once and without any retries to the endpoint specified in the policy’s PoisonMessageDrop value.
MaxMessageAge – The maximum message age is an XML duration value indicating after what time any enqueued message is considered ‘stale” and will be automatically dropped and removed from the queue. The default value is 10 minutes (600 seconds), the minimum value is 0 seconds (which effectively means that all incoming messages get dropped), the maximum value is 7 days.
MaxQueueCapacity – This numeric value indicates the maximum size of the Queue in bytes. This is a system-calculated value that cannot be set on the .NET class and should not be set by clients creating policies from scratch. The default and maximum capacity of any queue is currently capped at 2MB. This is a limitation specific to the March 2009 CTP , it’s been painful to impose this constraint, and it’s absolutely expected that this limit will be expanded significantly. 2MB still allow for a several hundred notification messages of a 2-4KB. If you need to store more data you can absolutely create several queues and partition data across them either explicitly or using Routers with a load-balancing message distribution policy (more on that in a subsequent post).
MaxQueueLength – This numeric value indicates the maximum queue length. The maximum and default value is 2^31, the minimum value is 1. We’re not enforcing the exact queue length in the March 2009 CTP, but your code should assume that we do. The queue length value is the basis for the calculation of the MaxQueueCapacity, with MaxQueueCapacity = min(MaxQueueLength * MaxMessageSize, 2MB) enforcing the hard 2MB limit that is currently in effect. You don’t need to touch this value unless you’d really want a Queue that’s even more size-constrained.
Overflow – The overflow policy setting becomes relevant when the queue is at capacity and the EnqueueTimeout has expired. It tells the Queue what to do with the message that just came in and that can’t be put into the queue because it’s full. Permitted values:
- RejectIncomingMessage - (default) The message will be rejected and an error status code or SOAP fault will be sent to the sender.
- DiscardIncomingMessage – The sender will get an indication that the message has been accepted, but the message will be dropped on the floor.
- DiscardExistingMessage – The Queue will remove and discard messages from the head of the Queue until the new, incoming message fits in the Queue.
PoisonMessageDrop – [this setting not supported in the March 2009 CTP] This value is a WS-Addressing 1.0 EndpointReference referring to an endpoint that any poison messages will be sent to once the MaxDequeueRetries limit has been exceeded. The EndpointReference may point to any SOAP or plain HTTP endpoint that can accept ‘any’ message.

Phew! Lots of options. The good thing is that most apps should be ok with the defaults.

I know that the 2MB capacity limit is somewhat disappointing and I’m certainly not happy with it. There’s a particular behavior (with bug number) that may occur under very rare circumstances in the underlying replication system, which caused us to play it very safe instead of risking data loss. I don’t think the limit is a showstopper for apps that send notifications and events around – it is a showstopper for apps that want to exchange larger payloads and we’re working to relax that limit and make Queues much, much larger as soon as we can. You can obviously always spool larger data into SDS or one of the Azure storage systems and then send a reference to that data as a message, but it’d be strange for a messaging system to make that a required pattern for data of all sizes. If we’re talking hundreds of megabytes it makes sense, though.

With the due apology out of the way, let’s look at how a policy may be applied to a namespace name – or in other words, how a queue is created in the simplest case (this must be done via HTTPS). The model here is that the client proposes a policy and Service Bus is at liberty to adjust the policy.

If you are intimately familiar with Atom, you’ll notice that the <QueuePolicy> is an extension and isn’t carried as <content>. That’s by intent. <content> is for people, extensions are for apps. We’ll start using <content> in a later milestone, so consider that being reserved.

HTTP/1.1 POST /myapp/q1
Host: clemensv.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <QueuePolicy xmlns=”http://schemas.microsoft.com/ws/2007/08/connect”>
      <ExpirationInstant>2009-04-03T12:00:00</ExpirationInstant>
   </QueuePolicy>
</entry>

If the queue can be created, the response is a 200 and you get the policy back along with any adjustments that the service may make. This is called the “effective” policy, i.e. that’s what the server is using. You also learn about where you can modify the policy, since – as I explained before – the endpoint where you POST the policy to is morphing into the Queue’s tail.

HTTP/1.1 200 OK
Location: https://project-name.servicebus.windows.net/myapp/q1!(queue)
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <link rel=”self” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue)” />
   <link rel=”alternate” href=”https://project-name.servicebus.windows.net/myapp/q1” />
   <link rel=”queuehead” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue/head)” />
   <QueuePolicy xmlns=”http://schemas.microsoft.com/ws/2007/08/connect” >
         <ExpirationInstant>2009-04-03T12:00:00</ExpirationInstant>
         <MaxQueueCapacity>2097152</MaxQueueCapacity>
   </QueuePolicy>
</entry>

If you want to renew the Queue (extend the expiration), take the effective policy, adjust ExpirationInstant and do a PUT to the “self” location.

HTTP/1.1 PUT /myapp/q1!(queue)
Host: project-name.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <link rel=”self” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue)” />
   <link rel=”alternate” href=”https://project-name.servicebus.windows.net/myapp/q1” />
   <link rel=”queuehead” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue/head)” />
   <QueuePolicy xmlns=”http://schemas.microsoft.com/ws/2007/08/connect” >
         <ExpirationInstant>2009-04-04T12:00:00</ExpirationInstant>
         <MaxQueueCapacity>2097152</MaxQueueCapacity>
   </QueuePolicy>
</entry>

[Bug note: The PUT will return the new effective policy just like the POST response shown above, but the returned Atom <links> aren’t correctly formed. Keep the ones returned from the POST]

If you want to delete the queue, just nuke the policy:

HTTP/1.1 DELETE /myapp/q1!(queue)
Host: project-name.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Length: 0

So that’s the policy story for Queues. In the next posts I’ll discuss the REST Queue protocol and the SOAP Queue protocol for how you send message to the queue and get messages out. REST I’ll explain in protocol terms, the SOAP model in .NET programming model terms.

Clemens Vasters