Service Bus - Optimize consumers using prefetch and maximum concurent calls features

From same version ago, Windows Azure Service Bus supports 'event notification'. This means that we can register to an event that will be triggered each time when a new message is available for us.

QueueClient client = QueueClient.Create("queue1");
client.OnMessage(
    OnMsgReceived, 
    new OnMessageOptions());
...
void OnMsgReceived(BrokeredMessage message)
{
    ...
}

This is a great feature, that usually make our life easier. By default, when we are consuming messages in this way, we will make a roundtrip to the Service Bus for each message. When we have applications that handle hundreds of messages, the roundtrip to the server for each message can cost us time and resources.
Windows Azure Service Bus offer us the possibility to specify the number of messages that we want to prefetch. This means that we will be able to fetch 5, 10, 100, .. messages from the server using a single request. We could say that is similar to the batch mechanism, but can be used with the event notification feature.

QueueClient client = factory.CreateQueueClient("queue1");
client.PrefetchCount = 200;

We should be aware that the maximum numbers of messages that can be prefetch by one request is 200. I think that this an acceptable value, if we take into the consideration that this will trigger 200 notification events.
This value need to be set when we create and setup the subscription client. Theoretically you can set this value before receiving the first message from the server, but I recommend to make all this configuration at the setup and initialization phase.
When we are using this mechanism of prefetching, we need to know exactly how many concurrent calls  we can have. Because of this, the Service Bus client give us the possibility to specify how many concurrent calls we can have in the same time.

client.OnMessage(CalculateEligibility,new OnMessageOptions()
            {
                MaxConcurrentCalls = 100
            });

For example if we would have the prefetch count set to 200, and the maxim concurrent calls sett to 100, we will have the following behavior:
Make a roundtrip to the server
Receive 200 messages (we suppose that there are 200 messages available)
Consume first 100 messages
Consume the other 100 messages
Messages are consumed in a async way. This means that from the first 100 messages, when 1 will be processed on the client, another message receive event will be triggered.
We saw in this post how we can consume the messages in parallel when using Windows Azure Service Bus and event notification. Using this features we can increase our application performance.
Good luck with Windows Azure Service Bus.

PS: Bonus picture - This is a picture from Seattle airport, where I am now, waiting the flight back home.

Read More

Throttling and Availability over Windows Azure Service Bus

In today post we will talk about different redundancy mechanism when using Windows Azure Service Bus. Because we are using Windows Azure Service Bus as a service, we need to be prepared when something goes wrong.

This is a very stable service, but when you design a solution that needs to handle millions of messages every day you need to be prepared for word case scenarios. By default, Service Bus is not geo-replicated in different data centers, because of this if something is happening on the data center where your namespace is hosted, than you are in big troubles.
The most important thing that you need to cover is the case when the Service Bus node is down and clients cannot send messages anymore. We will see later on how we can handle this problem.
First of all, let’s see why a service like Service Bus can go down. Well, like other services, this has dependencies to databases, storages, other services and resources. There are cases when we can detect pretty easily the cause of the problem.
For example when we receive ‘ServerBusyException’, then we know that the service don’t have enough resources (CPU, memory, …) and we need to retry later. The default retry period is 10 seconds. It is recommended to not set a value under 10 seconds.
This problem can be resolved pretty easily with partition. When we are using partitioning, a topic or a queue is spitted on different messages brokers. This means that we have less chances to have our service down. Also, if something happen with one of our brokers, we will still be able to use the topic/queue without any kind of problems. Don’t forget that brokers will be on the same data center. Using this feature don’t increase your costs.
Enabling this feature can be done in different ways. One option is from portal, Visual Studio Server Explorer or from code.

NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString("...");
TopicDescription topicDesc = new TopicDescription("[topicName]")
{
  EnablePartitioning = true;
}

namespaceManager.CreateTopic(topicDesc);

It is so simple to use it. You should know that in this moment you can have maximum 100 topics/queues per namespace that has this feature activated, but I expect this value to change in the future. You should also know different behaviors that are happing when you are using sessions:

• Partition Key – Messages from the same transaction, that has the same partition key, but don’t has a session id will be send to the same broker.
• Session Id – All messages with a specific session if will be send to the same broker.
• Message Id – Messages that are send to a queue/topic with duplicated detection activated will be send to the same broker. Because of this I recommend to use messages duplication detection only where is necessary.
• None of above – Messages are send to all brokers in a round robin manner – one message to each broker.

Another downtown cause can be Service Bus service is upgraded. In this cases, the service will still work, but we can have 15-20 minutes latency until the message will appear in the queue/topic. The most important thing in this case is “We don’t lose any message”.
Also when the system is not stable (internal causes), brokers will be automatically restarted. The restart can take one or more minutes. In this case the service will throw MessagingException or TimeoutException. This problem are resolved build in by the clients SDK’s (if you are using .NET SDK). They have a retry policy build-in, that will retry to resend the message. If the retry policy is not able to send the message, an exception is throw that can be handled in different ways. Until now, all the issues related to this were handle by retry policy with success.
Custom configuration of retry policy can be made in the factory class of messaging.

MessagingFactory messagingFactory = MessagingFactory.Create();
messagingFactory.RetryPolicy = RetryExponential.Default;

The last main cause of failing is external causes like internet connectivity problem, electrical outage or human errors. This problem is handled with a very different approach. The client needs to detect this problem and handle it. Until now, this required a custom code to be written, that would redirect the messages to a topic/queue that is in another datacenter (namespace).
From now we can use paired namespace to handle this scenario. The paired namespace give us the possibility to specify a second namespace (that can be in a different data center) that will be used to send messages until the primary one will be up and running. When messages are send to the second namespace, messages will be persisted until the primary namespace will be up. In the moment when the primary namespace is running, all messages from the second one will need to be redirected to the first one. We can imagine secondary namespace as a buffer that is used to store messages until our main namespace is in good state.
 When we configure this feature, we can set also the failover interval. This is the time interval when our system will accept failovers before switching to the second namespace. The recommended (and default) value is 10 seconds. Also you will need to specify the number of queues that are used to store the messages in the secondary namespace (default value is 10). This value should be greater or equal to 10.
The last option that you should be aware is syphon (‘enableSyphon’ parameter). When you activate this on a client, you tell to the system that this is the system that will transfer the messages from the second namespace to the first one. Usually this value should be set on the consumers clients (backend), because usually clients only send messages to the topics/queues.

NamespaceManager primaryNM = NamespaceManager.CreateFromConnectionString("...");
MessagingFactory primaryMF = ...
NamespaceManager secondaryNM= NamespaceManager.CreateFromConnectionString("...");
MessagingFactory secondaryMF = ...
SendAvailabilityPairedNamespaceOptions sao=
    new SendAvailabilityPairedNamespaceOptions(secondaryNamespaceManager, secondaryMF);
primaryMF.PairNamespaceAsync(sao).Wait();

There are some small things that we should know related to this feature:

• The state and order is guaranteed only in the first queue. When using session, the order of the messages is not guaranteed when secondary namespace is used
• Messages are consumed only from the primary queue/subscription
• You will pay the extra cost of moving messages from the secondary namespace to the primary one
• The default name of the queues that are created on the secondary namespace is ‘x-servicebus-transfer/i’ (where ‘i’ can have a value from 0 to n)
• The queue from the secondary namespace is randomly chosen
• It is not recommended to change the configuration of the queues from the secondary namespace

We saw that we have different mechanism to handle this special scenarios. We don’t have a mechanism that handle all this use cases. Before starting to think about integrating all this feature ask yourself if you need all of them? There are cases when a 10-15 minutes downtime is acceptable.

Read More

Windows Azure Service Bus - What ports are used

Windows Azure Service Bus is a great mechanism to distribute messages across components of your own system or to different clients. When you want to use this service for enterprise projects you will meet the IT guys.
For them it is very important to control the ports that are used by applications. Because of one of the first’s question that they will ask you is:
What are the ports that are used by Service Bus?
When this question is asked you should be prepare to have the answer prepared. Looking over the documentation from MSDN, the following ports are used by Windows Azure Service Bus:

• 80 – HTTP connection mode
• 443 – HTTPS connection mode
• 5671 – Advanced Message Queuing Protocol (AMQP)
• 5672 – AMQP
• 9350 – WCF with connection mode Auto or TCP using .NET SDK
• 9351 – WCF with connection mode Auto or TCP using .NET SDK
• 9352 – WCF with connection mode Auto or TCP using .NET SDK
• 9353 – WCF with connection mode Auto or TCP using .NET SDK
• 9354 – WCF with connection mode Auto or TCP using .NET SDK

I recommend to use HTTP/HTTPS connection where is possible. IF you are using .NET SDK you don’t need to make any custom configuration. When you let connection mode set to AutoDetect, the SDK will check if a connection can be made using non-HTTP ports. If the endpoint cannot be reached, that he will try to go over HTTP.
If you want to control the connection method, than you will need to set the ‘Mode’ property of the SystemConnectivity. The supported mode are:

• AutoDetect
• Http
• Tcp

ServiceBusEnvironment.SystemConnectivity.Mode = ConnectivityMode.Http;

Enjoy!

Read More

Types of Subscriptions Filter on Windows Azure Service Bus

Service Bus from Windows Azure is a great services to distribute messages to multiple subscribers. A topic can have one or more subscriptions. A message added to the topic will be send to all subscriptions that are register in that moment.
To be able to control what kind of messages are send to each subscription we can define filters. A filter is added on each subscription and specify what messages will be accepted by subscription.
The types of filters that can used are:

Correlation Filter
This is used when you want to accept messages that has a specific correlation value. This value can be set directly to the BrokenMessage and is great if you have messages with different priority or rank.

CorrelationFilter filter = new CorrelationFilter(“low”);
namespaceManager.CreateSubscription(topicPath, subscriptionName, filter);

BrokenMessage message = new BrokenMessage();
message.CorrelationId = “low”;
…

SqlFilter
This filter give us the ability to specify what kind of messages will be accepted in a subsection based on a custom rules. This rule can be related to the properties list of a BrokenMessage that are specific by the producer or system properties (SessionId for example is a system property).
The syntax is very simple and is based on SQL92 standard and T/SQL. If you know to write a query in SQL than you will not have problem to use this filter. The syntax give us the possibility to use = != < > <> IS IN LIKE EXISTS ESCAPE NO + - * / AND OR …
For a normal use this is enough to define rules over subscriptions.

SqlFilter filter = new SqlFilter(“CustomProperty == 1”);
namespaceManager.CreateSubscription(topicPath, subscriptionName, filter);
…
BrokenMessage message = new BrokenMessage();
message.Properties[“CustomProperty”] = 1;

The “user” or “sys” prefix need to be used when you refers to properties that are in different scope. The “sys” refers to properties that are on BrokenMessage (like Correlationid or SessionId).

FalseFilter
This is a filter that can be used to block all the messages to the given subscription. This can be used if you create a subscription and you don’t want to accept message to it (temporarily). Also, this can be used when you have a subscription and you don’t want to receive message anymore to it, but you don’t want to delete it.

FalseFilter filter = new FalseFilter()
namespaceManager.CreateSubscription(topicPath, subscriptionName, filter);
…

TrueFilter
This filter is similar with SqlFilter. It will accept the message that respect the specific rule.

We looked over what kind of filter can be defined over a subscription. I think that FalseFilter can be very useful when you are in production and you need to control what messages are receive by one of subscription. For example in a scenario when a client didn’t pays the monthly subscription and you want to stop temporary his subscription. Is more simple that deleting it or set a SqlFilter – “ 1 <> 1”.

Read More

Service Bus and Shared Access Signature (SAS)

Until few months ago, Shared Access Signature (SAS) could be used only wilt Azure Storage (Blobs, Tables and Queues). From now one we can use SAS with Service Bus. We can define SAS to use in combination with topics, queues or notification hub
This security feature is pretty great, especially when you have an application that use 3rd party. You don’t need any more to share with 3rd parties the account name and key. From now one you can give them a unique token that give them access to some part of your namespace services and also limit what kind of operation they are allow to do.
In this moment you are allow to define only 12 rules in a namespace, but in the near future I expect to be able to define more than 12 rules.
The access rights that can be controlled in this moment are:

• Listen – to be able to receive message
• Send – to be able to send messages
• Manage – to be able to manage the resource

From Service Bus perspective, this access rights are enough and in combination with expiration date it works pretty good. Be aware, that you can set Manage access rights, only if you set also the Send and Listen rights.
The interesting thing is how the rules are defined. The rules are defined over a URL. This means that you can define a rule over a namespace or over a specific topic or queue. The SAS will be valid for all resources under the specific URL.
When you generate a SAS rule, two keys will be generated for the same rules. Both keys can be used in the same time. This is done to help customers in the moment when they need to generate new keys and they don’t want to block the access to Service Bus.
The SAS rules can be defined and manage not only from code (using REST API) but also from the management portal of Azure. The portal can be very helpful when you are in the development phase or you have some issues with SAS and you want to check what rules are already defined.

How to create a SAS from code
First step is to create a SharedAccessAuthorizationRule and set the specific rights.

SharedAccessAuthorizationRule saar = new SharedAccessAuthorizationRule(
    “myFooName”,
    SharedAccessAuthorizationRule.GenerateRandomKey(),
    new[] {
        AccessRights.Manage, 
        AccessRights.Listen, 
        AccessRights.Send }));

GenerateRandomKey method is used to generate random keys.
Once the rule is created, you can add it to the queue, topic or notification hub using the description class

QueueDescription qd = …
qd.Autothorization.Add(saar);

Don’t forget to save the name of the name and the key of the rule.

How to use a SAS key
Once you created the rule and have the name and the key of the rule, you will be able to use them in the moment when you create the MessagingFactory.

MessagingFactory mf = MessagingFactory.Create(
    "uri,
    TokenProvider.CreateSharedAccessSignatureTokenProvider"myFooName", "myFooKey"));
QueueClient queue = mf.CreateQueueClient("myFooQueue");

Enjoy!

Read More

More about 'OnMessage' method of Service Bus

If you are using Service Bus, you should know that you have the ability to be notified when a message is available in the queue or in a subscription. Using this feature you don’t need to call the "Receive" method and catch the timeout exception when there are no new messages are available.

OnMessageOptions messageOptions = new OnMessageOptions()
  {
    AutoComplete = true,
    ExceptionReceived += LogErrorsOnMessageReceived
  };
queueClient.OnMessage((message) => 
  {
                ...
  }, messageOptions);

As you can see, you register to an event that will be called when a new message is available. If there are more than 1 messages available the event will be triggered more than one. Each notification will run on a different thread. To be able to control the number of concurrent messages that can be received by a client you should change the value of "MaxConcurrentCalls".

OnMessageOptions messageOptions = new OnMessageOptions()
  {
    AutoComplete = true,
    ExceptionReceived += LogErrorsOnMessageReceived
    MaxConcurrentCalls = 5
  };
queueClient.OnMessage((message) => 
  {
                ...
  }, messageOptions);

But what about the costs?
When using Service Bus, you pay for each transaction. Because of this each call of “Receive” method will consume a transaction. Even if there are no available messages and the call finish with a timeout exception, you will have to pay for one transaction.
Base on this, we should know that a similar thing is happening when we are using “OnMessage”. The behind implementation of this method use “Reveive” method and has a timeout value. Because of this, even if you register to “OnMessage” once you will notify that more than one transactions are consumed even if you don’t have messages in the Service Bus.
This is happening because “OnMessage” calls “Receive” method that will consume a transaction when a message is available or when a timeout exception occurs. The default timeout value is 60 seconds. This timeout value can be easily change using MessagingFactory.
From the cost perceptive you should not have any kind of problems. The cost of each client that use “OnMessage” method and has the timeout value set to 60 seconds is 4.3 cents per month.
In conclusion “OnMessage” is very useful because offer out of the box a mechanism that give us the possibility to be notified when a message is available – until now we had to implement this mechanism every time. You should not forget that this mechanism will handle each new message on a different thread and there are times when you want to control the concurrent level.  Also it generates some costs, but the cost is similar with the “Receive” mechanism – in the end “OnMessage” use “Receive”.

Read More

Subscription count - (Part 7) Testing the limits of Windows Azure Service Bus

Let’s talk about Windows Azure Service Bus and the numbers of subscriptions that a topic can have. I started to prepare a POC for a possible client and it is possible to end up with hundreds of subscriptions on each topic.
Looking over the documentation from MSDN, each topic supports 2000 subscriptions. I wanted to see what is happening with the latency of each subscription if we have over 1000 of them per topic.
To find an answer to my question, I started to write a worker role that:
Creates a new subscription
Sends a message to the topic
Receives a message using the first subscription
I measured how long it takes to send a message to a topic that has 1, 2, … , 2000 subscriptions. The same think I’ve done for the first subscription of the topic. I measured how long it takes to read a message from it when we have 1,2, … , 200 subscriptions.
The code is extremely simple but the results are very interesting.
The first diagram shows how long the send and receive commands took. The blue color is for the send request to the topic and the orange one is for receive command for the first subscription of the topic. The AX represent the number of subscriptions per topic and the OY represent how long a command took (in milliseconds).

As you can see, the command latency doesn’t increase too much. Even for 2000 subscriptions the latency is under 100 milliseconds.
The next diagram show the average latency when we have 500, 1000, 1500 and 2000 subscriptions. Based on this result we observe that the latency increases with a factor under 1.5X.

We can say that we can use this service in our business scenarios for 2000 subscribers without a problem. The latency increases only with a factor that is under 1.5X. We could have a problem if the latency would increase with a factor of 10X or 100X. But we are in a safe zone.
It seems that Service Bus is a real cloud service that scales and works excellent.

Read More

Topic Isolation - (Part 6) Testing the limits of Windows Azure Service Bus

Some time ago I wrote about how I manage to process millions of messages over Windows Azure Service Bus. I discover that processing millions of messages every hour will increase the latency and response time of our topic.

I wanted to find out what I happening with the topics from the same account and data center when the load of one of them increase drastically. 

To be able to make this tests and discover the answer to my questions I made the following setup:

• Create 3 topics on the same data center. Two on the same account and another one on a different account.
• Create a worker role that will read messages from topics (via subscription) and monitor the delay time. The results will be written to Windows Azure Tables.
• Create 8 worker roles that will push hundreds of thousands of messages in a very short period of time on the same topic (multi-threading rules).
• Create 4 worker roles that will consume messages from our topic (via subscription).

In the end we end up with 12 worker roles that hit the same topic. Of course the latency of that topic increases, but we wanted to see if the rest of the topics are affected. I run this performance test for around 6 hours and …

The results of the test were extremely good. The second and 3rd topic where not affected by this load test. The latency remain the same for this 2 topics during the load test.

I’m very happy with this results. This is a confirmation that we will have a good performance on all of our topics. The topics are 100% isolated are not affected by the rest of out topics.

Read More

[Post-event] ITCamp 2013

These days I participated at ITCamp 2013. This year I had the opportunity to participate at a premium event with almost 400 attendees. For Romania, this is big number, it is hard to gather 400 people at a premium conference.

What we found this year? COOL stuff. We discover what is the future of tablets, what is happening on the cloud and the voodoo magic that is happening inside the team management... and so on.

I really enjoyed the conference, the sessions and especially the location – you could see all the city from above. Each year one of the most important thing at ITCamp is socialization – meeting people from different parts of the world, discover what they are doing and what are the trends of the IT industry.

This year I had the opportunity to be invited as a speaker at ITCamp. Last year I talked about background task on Windows 8, but this year I decided to go on the cloud and talk about messaging pattern and how we can implement those patterns using Windows Azure Service Bus Services.

My slides from ITCamp:

Messaging patterns in the cloud from Radu Vunvulea

Read More

Testing the limits of Windows Azure Service Bus

Part 1 – What is the maximum number of messages that can be processed in 30 minutes?

Part 2 – What is the optimum size of worker roles that process messages from Service Bus?

Part 3 – What is the maximum number of worker role that can process messages from Service Bus?

Part 4 – What are cost of Service Bus when we need to process millions of messages every hour?

Part 5 – What are the scalabilities points of Service Bus?
Part 6 - Topic isolation

Read More

How to iterage a Service Bus Queue using Azure SDK 2.0 & What are the concerns of this new feature

The new version of Windows Azure Service Bus came with a lot of new feature. One of the new feature that was added in Windows Azure SDK 2.0 is the support of message browsing.
What is message browsing? This is a functionality that gives the ability to a client to iterate and access messages from a Service Bus Queue without locking or removing a message from the actual queue.
There are pretty interesting thing that we can do using this feature. For example we can peek a message from a specific index in the queue:

BrokenMessage message = queueClient.Peek(3);

Or we can peek the first message that is available:

BrokenMessage message = queueClient.Peek();

Another option is to peek a specific number of message from the queue:

IEnumerable<BrokenMessage> messages = queueClient.PeekBatch(20);

The PeekBatch support also two parameters, where the first one is used to specify the starting index (similar with Peek(3)):

IEnumerable<BrokenMessage> messages = queueClient.PeekBatch(100, 20)

In the above example we peek the next 20 messages, starting from the index 100.
Interesting feature, from now one we can iterate a queue. Great! But I have an open question:
Why would we want to iterate a queue?
If we need a functionality like this, then maybe we don’t use the proper service. For example, we would say that we need this feature when we need to do monitoring or audit of a system. Well… true – we could use this feature to have audit system over a queue, BUT. Yes, there is a big BUT. Why would you use a queue when you need to support audit over it. You can use Service Bus Topic and have a dedicated subscription for the audit system. For the same price, you can have a system that was created for this scenario.
Other cases when people would find this feature useful would be in the moment when they need to debug the application. They could see the content of the queue – the investigation process would be easier. True, I had the “opportunity” to   debug a system that use queues and when you cannot control the content of the queue, then the nightmare begin. When problem like this occurs I would try to use the death letter feature – when a message cannot be processed X times, than the message is send automatically to death letter queue.
I see the value of this feature when we need to make debug or to investigate the content of a queue. In the same time this feature can open Pandora’s door. Why? Because developer will use this feature in strange way and from a queue we will end up with a list.

Read More

(Part 5) Testing the limits of Windows Azure Service Bus

In the last series of post about testing the limits of Windows Azure Service Bus we saw what is the best configuration for our problem, the best configuration is to have 4 medium instances.
The next question for us is: How we can decrease the time processing?
In our case we saw that from the cost and benefits perspective we reached the limitation on Service Bus. This is around 4 medium instances that are able to process around  1.000.000 messages in 34 minutes. But what we can do if we have 10.000.000 messages that needs to be processed in 1 hour.
We could increase the number of instances, but this will not improve our performance too much. In this moment we use only one topic. This topic, as Service Bus, has its own limitations – we cannot make an undefined number of request per second and expect to have a low latency.
A solution to decrease the number of request is to use batches – we already use them.
Another solution is to scale the topics. In this moment we have only one topic. If we would use 5 topics, than we could have 4 instances that will consume messages for each topic. In this way we could consume 5.000.000 messages in 34 minutes. The downside is from the cost perspective. The costs will increase with a 5X factor.
From the cost perceptive, this would be acceptable because all the instances work at maximum capacity. In the moment when we don’t need any more this instances we could stop them.
The only problem is how we can distribute the messages between different topics.
One solution would be to identify different attributes of our messages and try to group messages based on this attributes. In this way we could distributed the messages on more than one topic. In theory this could be a good solution, but how many times you could group items in an equal way. In real life this cannot be accomplish in normal cases.
What we could do is to create a mechanism that can select what topic to use. For example each producer, at a specific time interval or after he send a specific number of messages, check what is the load on each topic and decide what topic should be used.
In this way we could have a flat distribution of messages over our topics.
In this post we saw how we can scale on horizontal Windows Azure Service Bus. With a good design we should be able to support scaling in all the locations where we expect to have a bottleneck.

Read More

Windows Azure Service Bus - Control what actions can be made

Did you ever need or want to disable the send functionality from a queue or from a topic? All the people start to send messages to you even if you say STOP. How nice would be to be able to say STOP.
We have good news for people that use Windows Azure Service Bus (Queue, Topics and Subscriptions). With the new version of SDK (2.0), we can control the operation that can be made on Service Bus. How nice is that.
People would ask – why we would want to do something like this. Well, there are times, especially when we make tests, when we want to control this. I see this value in two situations:
When we want to make performance tests, to see how many messages we can consume from a queue in a specific time interval
When there a lot of messages in the queue or topic and we don’t want to accept new messages anymore.
This feature can be controlled from the description of the queue, topic or subscription.  Based on what kind of service we use from Service Bus we can have different status. It is important to not forger to call the UpdateQueue or UpdateTopic/UpdateSubscription after you change the service status.
The entity status of a services can have the following values:

• EntityStatus.Disable – Sending and receiving messages is disabled
• EntityStatus.SendDisabled – Sending is disable
• EntityStatus.ReceiveDisabled – Receiving is disabled
• EntityStatus.Active – Sending and receiving are active

Don’t try to use a combination of this status. All the possible combinations are supported with this 4 status.

TopicDescription topic = namespaceManager.GetTopic("fooTopic");
topic.Status = EntityStatus.SendDisabled; //can continue to de-queue
namespaceManager.UpdateQueue(topic);

This configuration can be made from the moment when you create the topic.

if (!namespaceManager.TopicExists("fooTopic"))
{
    TopicDescription topic = new TopicDescription("fooTopic");
    topic.Status = EntityStatus.SendDisabled; //can continue to de-queue
    namespaceManager.CreateTopic(topic);
}

Read More

Event-Driven message consumtion (aka pump model) - Windows Azure Service Bus

The new version of Windows Azure SDK was launched. I’m glad to see that new feature of Service Bus are available. For me, one of the most important features that was added to Service Bus is the one that give us the possibility to consume messages from a topic or queue using an event-driven model – “pump model”.
Until now, we had to have an infinite loop which check if a new message is available. This simple solution was good, but it was very primitive. In a real production application, a new message would trigger an event that would execute a specific code.
To be able to support something like this, the user would need to create an infinite loop, check if a new message is available and trigger a specific event. This infinite pooling creates a lot of problems, especially when something would go wrong – detecting an error and managing it.

while (true)
{
try
{
  if (isNotStopped)
  {
    break;
  }

  BrokeredMessage message = null;
  message = queueClient.Receive();

  if (mesage == null)
  {
    // No message available;
    continue;
  }

  Process(message);
  message.Complete();
}
catch (Exception ex)
{
  ...
}
}

With the new SDK, we don’t need to do this anymore. We can remote the infinite loop from our code and switch to something more elegant.

OnMessageOptions messageOptions =new OnMessageOptions()
  {
    AutoComplete = true,
    MaxConcurrentCalls = 7
  }
messageOptions.ExceptionReceived += (sender, e) =>
  {
    ...
  }

QueueClient client = QueueClient.Create(...);
client.OnMessage(
  (message)=>
    {
      ... process message
    },
  messageOptions);

As we can see, we need to specify an event action that will be executing when a new message is available. The parameter or this event is the message itself – BrokeredMessage. This message can be process in any way we want.
In the code, beside the message arrived event I think that you notice the exception received event. This event will be called when an error occurs during the message processing. For example, if during the message arrive event an error occurs, the error will trigger the exception received event.
It is important to know that you should not catch errors in the message received event. Doing something like this will not notify the Windows Azure SDK that something went wrong during the message processing. Because of this, in the case we want to consume message in the peek and lock pattern (and not remove messages that could not be processed with success) we will end up with an expected behavior - removing the messages that were not processed with success. This behavior will appear because we catch all the exception in the message arrive event and the SDK will not know that the message processing went wrong.
OnMessageOption class is used to specify different options about consuming model. Using this class we can specify the level of concurrency (MaxConccurentCalls) and how we want to consume the messages (peek and lock or received and delete). When the AutoComplete flag is set to true, the peek and lock mechanism will be used. This mean that in the moment when the message received event will end, our message will be marked as consumed and removed from Service Bus.
If we want to control this behavior, we need to set this flag to false. Because of this the Complete() method of the BrokeredMessage will not be called automatically. In the message received event we will need to call this method manually. Without calling this method and setting the AutoComplete flag set to false, message will not be removed from Service Bus anymore – Service Bus will expect the success confirmation call.

OnMessageOptions messageOptions =new OnMessageOptions()
  {
    AutoComplete = false,
    MaxConcurrentCalls = 7
  }
messageOptions.ExceptionReceived += (sender, e) =>
  {
    ...
  }

QueueClient client = QueueClient.Create(...);
client.OnMessage(
  (message)=>
    {
      ... process message
      message.Complete();
    },
  messageOptions);

These option class contains also an exception event that will be triggered in the moment when an error occurs. I recommend to always register to this event. Don’t forget – this event will be called when an exception occurs in the message received event.
This new feature of Service Bus will improve the user experience and our application code.
PS: Don't forget to use a a retry policy when you call the Complete() event.

Read More

[Post-Event] Windows Azure Boot Camp in Cluj-Napoca

These Saturday all over the world Windows Azure Bootcamp took place. Of course we had to put Cluj-Napoca on the map – because of this we organize this event in Cluj also.
 All the people that participate had the opportunity to discover and use Windows Azure. At every session there where around 40 people.
There were 3 different sessions that explain how we can work with Service Bus, Mobile Services and how we can monitor our cloud applications.
At the end of the post you can find my session slides.

Read More

(Part 4) Testing the limits of Windows Azure Service Bus

In the last post related to this subject we discover how we can process millions of messages over a network using Windows Azure Service Bus Topic. For the problem that we want to resolve we find out that 4 worker roles of Medium size is the perfect configuration for our case.
In this post we will talk about costs. We will see what the costs to process 10.000.000 messages are. We assume that messages are send to Service Bus from our on-premise servers. If the messages are send from our the same data center, the cost related to bandwidth would be 0.
Sending messages to Service Bus

• Sending messages: 10$
• Bandwidth cost: 27.46$ 

Receive messages from Service Bus

• Receiving messages: 10$
• Bandwidth cost: 0$ (we are in the same data center)

Worker role costs:

• 4 medium worker roles: 8.64$

Remarks: From our results, we need 8h and 40 minutes to process 10.000.000 messages from Service Bus. In this time we included the warm up of each instance – 20 minutes. We calculate the cost of running 4 instances for 9 hours.
Windows Azure Table Storage:

• Storage: 5$ (cost per month)
•         Transactions:  2$

Remarks: In the transaction cost is included the cost of reading other data from Table Storage. Otherwise the cost of transaction to Table Storage would be maximum 1$ - because of batch support this value can be smaller.
The final cost of processing 10.000.000 messages and store them to Table Storage for 1 month is:

• 63.1$ (when messages are send from on-premise servers)
• 35.64$ (when messages are send from the same data center)

In theory, using this configuration, the price of each message processing would be 0.0003564$. What do you think about this cost? I would say that this is a pretty good price.
If we would need to process 10.000.000 every day, during a month, our final cost would be:

• 924.2$ (when messages are send from the same data center)

In this post we saw what the costs of processing 10.000.000 messages are. Even if the costs look pretty good, I still have one thing that I don’t like - the time processing of all messages is pretty high – over than 1 hour and a half. In the next post we will see what we can do decrease the time processing.

Read More

(Part 3) Testing the limits of Windows Azure Service Bus

In the latest post series about the limits of Windows Azure Service Bus, we saw that is the maximum number of messages that we can process through a single topic (1.000.000 messages every 30 minutes) and what kind of worker role we should use to process this messages in the optimum way (Medium size).
In this post I will try to respond to another question that is normal to appear when we are using a cloud solution.
Can I scale the number of instances that consume messages from the topic?
Environment:
Each message that was added to Service Bus was pretty small. We had around 100 characters in UTF7 and 3 properties added to each BrokeredMessage.
We run the tests with one; two and three subscribers with different filter rules and the result were similar.
Each message that is received from Service Bus required a custom action to be executed. This action is pretty complicated and consumes CPU power. Also the logic requires to access remote services (that are stored in the same data-center).
For all this tests we used a medium size worker role.
Action:
In the first phase we pushed on the topic 100.000 messages and measure how long it takes to process all the available messages. We tried to identify what is the best number of instances that we can have in parallel from the perspective of costs and time.
The next phase was to see detect what is the optimum number of messages that can be processed using the given number of instances that we found in the first phase.
Results:
Phase 1: We tried to process 100.000 of messages with different number of worker roles instances. The size of all instances was Medium. The results of these tests were:

• 1 instance – 18 minutes and 10 seconds
• 2 instance – 11 minutes and 40 seconds
• 3 instance – 7 minutes and 15 seconds
• 4 instance – 4 minutes and 15 seconds
• 5 instance – 3 minutes and 20 seconds
• 6 instance – 2 minutes and 59 seconds
• 7 instance – 2 minutes and 27 seconds
• 8 instance – 2 minutes and 5 seconds

We can observe that at while scaling up, the time decrease with around 30% percent until we reach 4-5 instances, when the time start to decrease more slowly. This happens because when a topic is intensive used, the response time increase. Don’t expect to have the same latency when the topic is hit 10 times per seconds or 1000 per seconds.
When calculating the price, we should take into account to different prices. The first price represents the cost of running N instances for the period of time when messages from topic are processed. The second price is the obsolete price, which represents the cost of running N instances for the minimum period of time – in the case of Windows Azure the smallest time unit is hour.
Taking all this things into account, we observed that 4 instance of medium size can process our messages in the shortest period of time with the best costs.
Having this “magic” number of instances and the size of them we made the next step. We measure how long it takes for our application that is deployed on 4 worker roles with medium size to process different number of messages. The results of this test were:

• 100.000 messages – 4 minutes
• 200.000 messages – 8 minutes
• 300.000 messages – 11 minutes
• 500.000 messages – 19 minutes
• 1.000.000 messages – 34 minutes
• 3.000.000 messages – 2 hours and 32 minutes

The results were pretty interesting. The performance is also the same until we reach a critical point. Do you remember the first post when I mentioned that we observe that 1.000.000 messages per 30 minutes seems to be the maximum number of messages that we can process in an optimum way?
This is the point when the performance starts to go down. We can observe that the difference between processing 1.000.000 messages and 3.000.000 messages is pretty big. From the time perspective the last test requested almost 4.5x more time.
At the end of the tests we decided to use only one worker role of medium size to see how long it takes to process different number of messages. The time results were:

• 100.000 messages – 18 minutes
• 200.000 messages – 32 minutes
• 300.000 messages – 51 minutes
• 500.000 messages – 1 hour and 31 minutes
• 1.000.000 messages – 2 hour and 58 minutes

What I liked at this result it was how the duration increased. The duration time increases almost like the number of messages.
Conclusion
For our business problems we observed that having 4 worker roles of medium size of the same topic is the best configuration that we can have from time and costs perspective. This result is extremely important because we know where is the point when we need to scale in a different way.
In the next post related to this topic we will talked about costs.
Remarks: Usually all the duration values are rounded to minutes (without seconds and milliseconds). This does not mean that we don’t have these values.

Part 4

Read More

(Part 2) Testing the limits of Windows Azure Service Bus

In one of my last post we talked about the performance of Windows Azure Service Bus and what is the maximum number of messages that we could process in a given period of time.
In this post we will see what is the maximum numbers of consumers that we can have for subscription on Service Bus. Before staring, you should know that this refers directly to our own requirements and with our custom actions that we need to do for each message. Because of this, for each kind of application, the result can vary a lot.
Scope:
To see what is the “magic” number of Windows Azure Worker Roles instances that we can have on cloud that can process messages from only one topic.
Environment:
Each message that was added to Service Bus was pretty small. We had around 100 characters in UTF7 and 3 properties added to each BrokeredMessage.
We run the tests with one; two and three subscribers with different filter rules and the result were similar.
Each message that is received from Service Bus required a custom action to be executed. This action is pretty complicated and consumes CPU power. Also the logic requires to access remote services (that are stored in the same data-center).
Action:
First step was to use only one instance that consumes messages. We tried to determine what kind of instance is more suitable for us – from CPU perspective and number of messages that are processed. We tried the size of the instance type that offers us the best price per instance also.
Remarks: The processer level of each instance was at around 80-85%. We use an automatic scaling mechanism over threads that take into account the CPU power, number of processors and so on. During the tests, there were client that produces messages in the same time – this was required because we had to see how Service Bus behave in a real situation, not in a situation when topic already contains all the messages
Second step was to identify the size of instance that was most suitable for us from all perspective. The cost was one of the most important one. After this we scale from 1 to n instance to identity where what is the maximum number of instances that we can have.
Results:
After first step we identify that 100.000 messages can be processed by roles in the following time:

• Small – ~19 minutes
• Medium – ~18 minutes
• Large – ~18 minutes
• Extra large – ~17 minutes

The best performance related to costs was offered by the Medium size instances.
Next step was to see what is the processing time of 100.000 messages using different number of instances. All the instances that were used at this step is Medium. We end up with the following times:

• 1 instance – 18 minutes
• 2 instances – 11 minutes 
• 3 instances – 7 minutes
• 4 instances - 4 minutes
• 5 instances – 3 minutes
• 6 instances – 2 minutes
• 7 instances – 2 minutes
• 8 instances - 2 minutes

In our case we observed that the number of instances that would make us happy and the cost would be pretty okay also is around 4 instances.
Costs:
The costs for processing 1.000.000 messages are pretty good:  4 x 0.16$ + 1$ = 1.64$.
Conclusion:
In this post we saw what is the size of the worker role that is most suitable for us. Using a Medium size worker role will offer us the best performance, reported to price. In the next post we will take a look over the duration of processing different number of messages using Medium size instances and differet number of instances.

Read More

(Part 1) Testing the limits of Windows Azure Service Bus

At the end of the last year I had the opportunity to write an application that had to handle millions of messages every day. To distribute the messages between different components of the application we used Windows Azure Service Bus Service.
In the past I talked a lot about this distribution messaging system. In this post we will see what are the magic numbers of a Topic from Windows Azure Service Bus Service.
Scope:
To see how many messages can be process in as short period of time. What is the maximum number of messages that can be processed using Service Bus using only one Topic. What is the optimal number of consumers for a subscriber or topic.
Environment:
Each message that was added to Service Bus was pretty small. We had around 100 characters in UTF7 and 3 properties added to each BrokeredMessage.
We run the tests with one; two and three subscribers with different filter rules and the result were similar.
Action:
We created 5 to 10 worker roles that started to send hundreds of messages to our Service Bus Topic.  In a very short period of time we could send millions of messages.
Results:
We tried pushing on the topic around 5 million messages. We observe that after a part of the messages are send to the Topic, the server starts to be slower. By slower I mean that the response latency increased drastically. Not only on the send operation, but also on the subscribers – the messages started to be received by consumer slower and slower.
One interesting part was on the portal. Even if the Topic was almost full, the number of messages that were reported on the Topic was only around 200.000-300.000.
After different experiment, we observe that with the current version of Service Bus and with the current configuration, a Topic can handle around 1.000.000 messages every 30 minutes.
Note: This value was obtain based on our experiment with our topic and messages configuration. This value is not a generic value, for any kind of project and configuration.
Cost:
From the cost perspective, we end up with a cost around 1$ per day if we use Windows Azure Service Bus for processing 1.000.000 messages. The size of each message was calculated by us around 24kB. This means that for processing 10.000.000 messages every day for a month would cost us around 300$. Before thinking that this are a lot of money, you should take into account that Service Bus will not lose any messages and in the end we process 300.000.000 (300M) messages per month.

I would conclude that the obtained value is pretty good. It is more that we expect to process a 30 minutes job, using one topic.
In the next post I will present the number of consumers that we used to process the messages in the best way possible.

Read More