Service Bus - Optimize consumers using prefetch and maximum concurent calls features

From same version ago, Windows Azure Service Bus supports 'event notification'. This means that we can register to an event that will be triggered each time when a new message is available for us.

QueueClient client = QueueClient.Create("queue1");
client.OnMessage(
    OnMsgReceived, 
    new OnMessageOptions());
...
void OnMsgReceived(BrokeredMessage message)
{
    ...
}

This is a great feature, that usually make our life easier. By default, when we are consuming messages in this way, we will make a roundtrip to the Service Bus for each message. When we have applications that handle hundreds of messages, the roundtrip to the server for each message can cost us time and resources.
Windows Azure Service Bus offer us the possibility to specify the number of messages that we want to prefetch. This means that we will be able to fetch 5, 10, 100, .. messages from the server using a single request. We could say that is similar to the batch mechanism, but can be used with the event notification feature.

QueueClient client = factory.CreateQueueClient("queue1");
client.PrefetchCount = 200;

We should be aware that the maximum numbers of messages that can be prefetch by one request is 200. I think that this an acceptable value, if we take into the consideration that this will trigger 200 notification events.
This value need to be set when we create and setup the subscription client. Theoretically you can set this value before receiving the first message from the server, but I recommend to make all this configuration at the setup and initialization phase.
When we are using this mechanism of prefetching, we need to know exactly how many concurrent calls  we can have. Because of this, the Service Bus client give us the possibility to specify how many concurrent calls we can have in the same time.

client.OnMessage(CalculateEligibility,new OnMessageOptions()
            {
                MaxConcurrentCalls = 100
            });

For example if we would have the prefetch count set to 200, and the maxim concurrent calls sett to 100, we will have the following behavior:
Make a roundtrip to the server
Receive 200 messages (we suppose that there are 200 messages available)
Consume first 100 messages
Consume the other 100 messages
Messages are consumed in a async way. This means that from the first 100 messages, when 1 will be processed on the client, another message receive event will be triggered.
We saw in this post how we can consume the messages in parallel when using Windows Azure Service Bus and event notification. Using this features we can increase our application performance.
Good luck with Windows Azure Service Bus.

PS: Bonus picture - This is a picture from Seattle airport, where I am now, waiting the flight back home.

Read More

Extract relative Uri using MakeRelativeUri method

Did you ever need to compare two web address and extract the relative part of them?
For example if we have address "http://foo.com" and "http://foo.com/car/color/power" we would like to get "car/color/power". But if we would have "http://foo.com/car" and "http://foo.com/car/color/power" we would like to get "color/power".
For this cases we should use "MakeRelativeUri". This method is part of Uri class and determines the delta between two URL address. This simple method extract the difference from two URL.
Code examples:

Uri uri = new Uri("http://foo.com");
Uri result = Uri.MakeRelativeUri("http://foo.com/car/color/power");
// result: "car/color/power"

Uri uri = new Uri("http://foo.com/car");
Uri result = Uri.MakeRelativeUri("http://foo.com/car/color/power");
// result: "color/power"

Uri uri = new Uri("http://foo.com");
Uri result = Uri.MakeRelativeUri("http://foo.com/car/color/power/index.html");
// result: "car/color/power/index.html"

Uri uri = new Uri("http://foo.com/car/color/power/");
Uri result = Uri.MakeRelativeUri("http://foo.com/");
// result: "../../../"

Uri uri = new Uri("http://foo.com");
Uri result = Uri.MakeRelativeUri("http://secondFoo.com/car/color/power");
// result: "http://secondFoo.com/car/color/power"

When the base address of the second URL is not the same with the first one (base one), the call of this method will return the second URL that was send as parameter.
This method can be very useful in certain situations, when you know about it. In one project I saw an extension implemented by developers that was doing the same thing :-).

Read More

Sync Group - Let's talk about Performance

In one of my latest post I talked about synchronization functionality that is available for SQL Azure. There was a question related of the performance of this service.

So, I decided to make a performance test, to see what are the performance. Please take into account that this service is in the preview and the performance will change when the service will be released.

For this test I had the following setup:

• Database
• Size 7.2 GB
• 15 tables
• 2 tables with more than 30.000.000 of rows (one table had around 3.2 GB and the other one had 2.7 GB)
• 34.378.980 rows in total
• Database instances
• 1 DB in West Europe (Hub)
• 1 DB in West Europe
• 1 DB in North Europe
• 1 DB in North Central US
• Agent
• 1 agent in West Europe
• Configuration
• Hubs win
• Sync From Hub

Scenario One: Initialize Setup

I started from the presumption that your data were not duplicated yet on all the databases. First hit of the Sync button will duplicate the database schema of the tables that needs to be sync, table content and rest of resources to all the databases for the given table. This means that 7.2 GB were send to the 3 different databases.

Normally you can do this action in other ways. Exporting/Importing the database for example, but I wanted to see how long it takes to sync all the databases.

Sync action duration: 5 hours and 36 minutes (20160.17 seconds)

 

Scenario Two: Update 182 rows

In this scenario I updated 182 rows from one of the tables

Sync action duration: 53.63 seconds

 Scenario Three: No changes

In this case I triggered the synchronization action without any changes.

Sync action duration: 38.47 seconds

 Scenario Four: 23.767 rows updated

23767 rows were updated on the hub database.

Sync action duration: 1 minute and 16 seconds (76 seconds)

 Scenario Five: 4.365.513 rows updated

As in the previous scenario, I updated  I changed a specific number of rows.

Sync action duration: 1 minute and 41 seconds (101.6 seconds)

 Scenario Six: 76.353 rows deleted

From one of the tables I deleted 73.353 rows.

Sync action duration: 56.26 seconds
 

As we can see, the synchronization action itself takes a very short period of time. For 4.5M of rows that were updated, the synchronization action took less than 2 minutes. The only scenario that took a log period of time was the initial synchronization action. Usually this action is made only one time. Also we have other method to import the database content to all our database.

I would say that the performance of the sync service is very good and I invite all of you tot check it out. You have support for synchronization out of the box.

Great job!

Read More

[PostEvent] Slides from MSSummit 2013, Bucharest

Last week I had the opportunity to participate at MSSummit. During this event I had the opportunity to present and talk SignalR and load testing using Windows Azure and Visual Studio 2013.
More about this event: http://vunvulearadu.blogspot.ro/2013/11/postevent-mssummit-2013-bucharest.html
You can find my session slides below:

Real time fluent communication using SignalR and Cloud (Windows Azure) from Radu Vunvulea

Load tests using visual studio 2013 and Cloud from Radu Vunvulea

Read More

How to get the instance index of a web role or worker role - Windows Azure

When we have one or more instances of a specific web role or worker role in the cloud, there are moments when we want to know from the code how many instances we have of the specific type or the index of current instance.
The total number of instances can be obtained using
RoleEnvironment.Roles[i].Value.Instance.Count
To be able to detect the index of the current instance we need to parse the id of the role instance. Usually the id of the current instance ends with the index number. Before this number we have ‘.’ character if the instance is on the cloud or ‘_’ when we are using emulator.
Because of this we will end with the following code when we need to get the index of the current instance:

int currentIndex = 0;
string instanceId = RoleEnvironment.CurrentRoleInstace.Id;
bool withSuccess = int.TryParse(instanceId.Substring(instanceId.LastIndexOf(".") + 1, out currentIndex));
if( !withSuccess )
{
  withSuccess = int.TryParse(instanceId.Substring(instanceId.LastIndexOf("_") + 1, out currentIndex));
}

Take into account that when you increase and decrease the number of instances, there are situation when you can end up with the following name of the instances:

• […].0
• […].1
• […].3
• […].4

Two is missing because when we decreased the number of instances from 5 to 4, that instance was stopped.

Read More

Sync Group - A good solution to synchronize SQL Databases using Windows Azure infrastructure

Working with Windows Azure becomes something that is more and more pleasant. In this post we will see how you can synchronize multiple databases hosted on Azure or on premise using Sync Group.
First of all, let’s start with the following requirement: We have an application that contains a database that needs to be replicated in each datacenter. What should we do to replicate all the content in all the datacenters?
A good response could be Sync Group (SDS). Using this feature we can define one or more instances of SQL Databases (from cloud or on premise) that will be synchronized. For this group we can specify the tables and columns that will be synchronized.
Creating a SDS can be made very easily from Windows Azure portal. This feature can be found under the SQL DATABASES tab – SYNC. I will not start to explain how you can create this group, because it is very easily, you only need to know the databases server address, user names and passwords.
I think that more important than this is the options that we have available on a SDS.

HUB
One of the databases that forms the group needs to be the hub. The hub represent the master node of the group, from where all the data propagates.
Synchronization Direction
Once we add the hub, we can add more databases to the group. In this moment we will need to specify the synchronization direction. In this moment we have 3 options:

• From the Hub – All the changes that are made in the hub are replicated to the rest of the databases from the group. In this configuration, when data are different, the hub will win. Changes in the databases are not written to the hub
• To the Hub – Al changes from the hub are not written in the databases. All changes that are made in the databases are written in the hub
• Bi-directional – The synchronization is made in both ways – from Hub to databases and from databases to hub

Synchronization Rules
From the portal, we have the option to select the tables (and columns) from the hub table that will be synchronized. In this way you don’t need to have databases that has the same schema. The most important thing is to have the tables that you want to synchronize in the same schema/format.
Remarks: We don’t need to replicate the database schema to all the databases. Once we select the tables and columns that we want to synchronize (from Hub schema), all this tables will be replicated in the rest of the group.

Conflict Resolution Policies
When we are creating a hub, we have two option for conflict resolution.

• Hubs Wins – In this case all the changes that are written to the hub will be persisted and in a case of the conflict, the version that is on the hub will be the ‘good one’
• Client Wins – In tis case the changes that are written on the slaves (non-hub database) will win and the change from the slave will propagate to the hub and to the rest of the group.

For more information about this conflict resolution policies I recommend to search on MSDN.

Synchronization Frequency
The synchronization between the databases of a group is not made in real time. We can select the time interval when the synchronization needs to be made. This time interval can be between 5 minutes and 1 month. Also, we have a button that can trigger the synchronization action.

On-premise SQL Server
To be able to use this feature on SQL Server you will need to download and install SQL Data Sync. This is a tool that will integrate this functionality on SQL Server.

Logs
All the synchronization action that are between group nodes are logged. Using this information we can determine how long the synchronization action took, what nodes were synchronized and how the action ended.

I think that this feature has a lot of potential. Why? Because database synchronization can be made very easily now. This feature can really add value to your application with minimal costs and headaches.

Read More

Throttling and Availability over Windows Azure Service Bus

In today post we will talk about different redundancy mechanism when using Windows Azure Service Bus. Because we are using Windows Azure Service Bus as a service, we need to be prepared when something goes wrong.

This is a very stable service, but when you design a solution that needs to handle millions of messages every day you need to be prepared for word case scenarios. By default, Service Bus is not geo-replicated in different data centers, because of this if something is happening on the data center where your namespace is hosted, than you are in big troubles.
The most important thing that you need to cover is the case when the Service Bus node is down and clients cannot send messages anymore. We will see later on how we can handle this problem.
First of all, let’s see why a service like Service Bus can go down. Well, like other services, this has dependencies to databases, storages, other services and resources. There are cases when we can detect pretty easily the cause of the problem.
For example when we receive ‘ServerBusyException’, then we know that the service don’t have enough resources (CPU, memory, …) and we need to retry later. The default retry period is 10 seconds. It is recommended to not set a value under 10 seconds.
This problem can be resolved pretty easily with partition. When we are using partitioning, a topic or a queue is spitted on different messages brokers. This means that we have less chances to have our service down. Also, if something happen with one of our brokers, we will still be able to use the topic/queue without any kind of problems. Don’t forget that brokers will be on the same data center. Using this feature don’t increase your costs.
Enabling this feature can be done in different ways. One option is from portal, Visual Studio Server Explorer or from code.

NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString("...");
TopicDescription topicDesc = new TopicDescription("[topicName]")
{
  EnablePartitioning = true;
}

namespaceManager.CreateTopic(topicDesc);

It is so simple to use it. You should know that in this moment you can have maximum 100 topics/queues per namespace that has this feature activated, but I expect this value to change in the future. You should also know different behaviors that are happing when you are using sessions:

• Partition Key – Messages from the same transaction, that has the same partition key, but don’t has a session id will be send to the same broker.
• Session Id – All messages with a specific session if will be send to the same broker.
• Message Id – Messages that are send to a queue/topic with duplicated detection activated will be send to the same broker. Because of this I recommend to use messages duplication detection only where is necessary.
• None of above – Messages are send to all brokers in a round robin manner – one message to each broker.

Another downtown cause can be Service Bus service is upgraded. In this cases, the service will still work, but we can have 15-20 minutes latency until the message will appear in the queue/topic. The most important thing in this case is “We don’t lose any message”.
Also when the system is not stable (internal causes), brokers will be automatically restarted. The restart can take one or more minutes. In this case the service will throw MessagingException or TimeoutException. This problem are resolved build in by the clients SDK’s (if you are using .NET SDK). They have a retry policy build-in, that will retry to resend the message. If the retry policy is not able to send the message, an exception is throw that can be handled in different ways. Until now, all the issues related to this were handle by retry policy with success.
Custom configuration of retry policy can be made in the factory class of messaging.

MessagingFactory messagingFactory = MessagingFactory.Create();
messagingFactory.RetryPolicy = RetryExponential.Default;

The last main cause of failing is external causes like internet connectivity problem, electrical outage or human errors. This problem is handled with a very different approach. The client needs to detect this problem and handle it. Until now, this required a custom code to be written, that would redirect the messages to a topic/queue that is in another datacenter (namespace).
From now we can use paired namespace to handle this scenario. The paired namespace give us the possibility to specify a second namespace (that can be in a different data center) that will be used to send messages until the primary one will be up and running. When messages are send to the second namespace, messages will be persisted until the primary namespace will be up. In the moment when the primary namespace is running, all messages from the second one will need to be redirected to the first one. We can imagine secondary namespace as a buffer that is used to store messages until our main namespace is in good state.
 When we configure this feature, we can set also the failover interval. This is the time interval when our system will accept failovers before switching to the second namespace. The recommended (and default) value is 10 seconds. Also you will need to specify the number of queues that are used to store the messages in the secondary namespace (default value is 10). This value should be greater or equal to 10.
The last option that you should be aware is syphon (‘enableSyphon’ parameter). When you activate this on a client, you tell to the system that this is the system that will transfer the messages from the second namespace to the first one. Usually this value should be set on the consumers clients (backend), because usually clients only send messages to the topics/queues.

NamespaceManager primaryNM = NamespaceManager.CreateFromConnectionString("...");
MessagingFactory primaryMF = ...
NamespaceManager secondaryNM= NamespaceManager.CreateFromConnectionString("...");
MessagingFactory secondaryMF = ...
SendAvailabilityPairedNamespaceOptions sao=
    new SendAvailabilityPairedNamespaceOptions(secondaryNamespaceManager, secondaryMF);
primaryMF.PairNamespaceAsync(sao).Wait();

There are some small things that we should know related to this feature:

• The state and order is guaranteed only in the first queue. When using session, the order of the messages is not guaranteed when secondary namespace is used
• Messages are consumed only from the primary queue/subscription
• You will pay the extra cost of moving messages from the secondary namespace to the primary one
• The default name of the queues that are created on the secondary namespace is ‘x-servicebus-transfer/i’ (where ‘i’ can have a value from 0 to n)
• The queue from the secondary namespace is randomly chosen
• It is not recommended to change the configuration of the queues from the secondary namespace

We saw that we have different mechanism to handle this special scenarios. We don’t have a mechanism that handle all this use cases. Before starting to think about integrating all this feature ask yourself if you need all of them? There are cases when a 10-15 minutes downtime is acceptable.

Read More