Service Bus - Optimize consumers using prefetch and maximum concurent calls features

From same version ago, Windows Azure Service Bus supports 'event notification'. This means that we can register to an event that will be triggered each time when a new message is available for us.

QueueClient client = QueueClient.Create("queue1");
client.OnMessage(
    OnMsgReceived, 
    new OnMessageOptions());
...
void OnMsgReceived(BrokeredMessage message)
{
    ...
}

This is a great feature, that usually make our life easier. By default, when we are consuming messages in this way, we will make a roundtrip to the Service Bus for each message. When we have applications that handle hundreds of messages, the roundtrip to the server for each message can cost us time and resources.
Windows Azure Service Bus offer us the possibility to specify the number of messages that we want to prefetch. This means that we will be able to fetch 5, 10, 100, .. messages from the server using a single request. We could say that is similar to the batch mechanism, but can be used with the event notification feature.

QueueClient client = factory.CreateQueueClient("queue1");
client.PrefetchCount = 200;

We should be aware that the maximum numbers of messages that can be prefetch by one request is 200. I think that this an acceptable value, if we take into the consideration that this will trigger 200 notification events.
This value need to be set when we create and setup the subscription client. Theoretically you can set this value before receiving the first message from the server, but I recommend to make all this configuration at the setup and initialization phase.
When we are using this mechanism of prefetching, we need to know exactly how many concurrent calls  we can have. Because of this, the Service Bus client give us the possibility to specify how many concurrent calls we can have in the same time.

client.OnMessage(CalculateEligibility,new OnMessageOptions()
            {
                MaxConcurrentCalls = 100
            });

For example if we would have the prefetch count set to 200, and the maxim concurrent calls sett to 100, we will have the following behavior:
Make a roundtrip to the server
Receive 200 messages (we suppose that there are 200 messages available)
Consume first 100 messages
Consume the other 100 messages
Messages are consumed in a async way. This means that from the first 100 messages, when 1 will be processed on the client, another message receive event will be triggered.
We saw in this post how we can consume the messages in parallel when using Windows Azure Service Bus and event notification. Using this features we can increase our application performance.
Good luck with Windows Azure Service Bus.

PS: Bonus picture - This is a picture from Seattle airport, where I am now, waiting the flight back home.

Read More

Sync Group - Let's talk about Performance

In one of my latest post I talked about synchronization functionality that is available for SQL Azure. There was a question related of the performance of this service.

So, I decided to make a performance test, to see what are the performance. Please take into account that this service is in the preview and the performance will change when the service will be released.

For this test I had the following setup:

• Database
• Size 7.2 GB
• 15 tables
• 2 tables with more than 30.000.000 of rows (one table had around 3.2 GB and the other one had 2.7 GB)
• 34.378.980 rows in total
• Database instances
• 1 DB in West Europe (Hub)
• 1 DB in West Europe
• 1 DB in North Europe
• 1 DB in North Central US
• Agent
• 1 agent in West Europe
• Configuration
• Hubs win
• Sync From Hub

Scenario One: Initialize Setup

I started from the presumption that your data were not duplicated yet on all the databases. First hit of the Sync button will duplicate the database schema of the tables that needs to be sync, table content and rest of resources to all the databases for the given table. This means that 7.2 GB were send to the 3 different databases.

Normally you can do this action in other ways. Exporting/Importing the database for example, but I wanted to see how long it takes to sync all the databases.

Sync action duration: 5 hours and 36 minutes (20160.17 seconds)

 

Scenario Two: Update 182 rows

In this scenario I updated 182 rows from one of the tables

Sync action duration: 53.63 seconds

 Scenario Three: No changes

In this case I triggered the synchronization action without any changes.

Sync action duration: 38.47 seconds

 Scenario Four: 23.767 rows updated

23767 rows were updated on the hub database.

Sync action duration: 1 minute and 16 seconds (76 seconds)

 Scenario Five: 4.365.513 rows updated

As in the previous scenario, I updated  I changed a specific number of rows.

Sync action duration: 1 minute and 41 seconds (101.6 seconds)

 Scenario Six: 76.353 rows deleted

From one of the tables I deleted 73.353 rows.

Sync action duration: 56.26 seconds
 

As we can see, the synchronization action itself takes a very short period of time. For 4.5M of rows that were updated, the synchronization action took less than 2 minutes. The only scenario that took a log period of time was the initial synchronization action. Usually this action is made only one time. Also we have other method to import the database content to all our database.

I would say that the performance of the sync service is very good and I invite all of you tot check it out. You have support for synchronization out of the box.

Great job!

Read More

How to get the instance index of a web role or worker role - Windows Azure

When we have one or more instances of a specific web role or worker role in the cloud, there are moments when we want to know from the code how many instances we have of the specific type or the index of current instance.
The total number of instances can be obtained using
RoleEnvironment.Roles[i].Value.Instance.Count
To be able to detect the index of the current instance we need to parse the id of the role instance. Usually the id of the current instance ends with the index number. Before this number we have ‘.’ character if the instance is on the cloud or ‘_’ when we are using emulator.
Because of this we will end with the following code when we need to get the index of the current instance:

int currentIndex = 0;
string instanceId = RoleEnvironment.CurrentRoleInstace.Id;
bool withSuccess = int.TryParse(instanceId.Substring(instanceId.LastIndexOf(".") + 1, out currentIndex));
if( !withSuccess )
{
  withSuccess = int.TryParse(instanceId.Substring(instanceId.LastIndexOf("_") + 1, out currentIndex));
}

Take into account that when you increase and decrease the number of instances, there are situation when you can end up with the following name of the instances:

• […].0
• […].1
• […].3
• […].4

Two is missing because when we decreased the number of instances from 5 to 4, that instance was stopped.

Read More

Sync Group - A good solution to synchronize SQL Databases using Windows Azure infrastructure

Working with Windows Azure becomes something that is more and more pleasant. In this post we will see how you can synchronize multiple databases hosted on Azure or on premise using Sync Group.
First of all, let’s start with the following requirement: We have an application that contains a database that needs to be replicated in each datacenter. What should we do to replicate all the content in all the datacenters?
A good response could be Sync Group (SDS). Using this feature we can define one or more instances of SQL Databases (from cloud or on premise) that will be synchronized. For this group we can specify the tables and columns that will be synchronized.
Creating a SDS can be made very easily from Windows Azure portal. This feature can be found under the SQL DATABASES tab – SYNC. I will not start to explain how you can create this group, because it is very easily, you only need to know the databases server address, user names and passwords.
I think that more important than this is the options that we have available on a SDS.

HUB
One of the databases that forms the group needs to be the hub. The hub represent the master node of the group, from where all the data propagates.
Synchronization Direction
Once we add the hub, we can add more databases to the group. In this moment we will need to specify the synchronization direction. In this moment we have 3 options:

• From the Hub – All the changes that are made in the hub are replicated to the rest of the databases from the group. In this configuration, when data are different, the hub will win. Changes in the databases are not written to the hub
• To the Hub – Al changes from the hub are not written in the databases. All changes that are made in the databases are written in the hub
• Bi-directional – The synchronization is made in both ways – from Hub to databases and from databases to hub

Synchronization Rules
From the portal, we have the option to select the tables (and columns) from the hub table that will be synchronized. In this way you don’t need to have databases that has the same schema. The most important thing is to have the tables that you want to synchronize in the same schema/format.
Remarks: We don’t need to replicate the database schema to all the databases. Once we select the tables and columns that we want to synchronize (from Hub schema), all this tables will be replicated in the rest of the group.

Conflict Resolution Policies
When we are creating a hub, we have two option for conflict resolution.

• Hubs Wins – In this case all the changes that are written to the hub will be persisted and in a case of the conflict, the version that is on the hub will be the ‘good one’
• Client Wins – In tis case the changes that are written on the slaves (non-hub database) will win and the change from the slave will propagate to the hub and to the rest of the group.

For more information about this conflict resolution policies I recommend to search on MSDN.

Synchronization Frequency
The synchronization between the databases of a group is not made in real time. We can select the time interval when the synchronization needs to be made. This time interval can be between 5 minutes and 1 month. Also, we have a button that can trigger the synchronization action.

On-premise SQL Server
To be able to use this feature on SQL Server you will need to download and install SQL Data Sync. This is a tool that will integrate this functionality on SQL Server.

Logs
All the synchronization action that are between group nodes are logged. Using this information we can determine how long the synchronization action took, what nodes were synchronized and how the action ended.

I think that this feature has a lot of potential. Why? Because database synchronization can be made very easily now. This feature can really add value to your application with minimal costs and headaches.

Read More

Throttling and Availability over Windows Azure Service Bus

In today post we will talk about different redundancy mechanism when using Windows Azure Service Bus. Because we are using Windows Azure Service Bus as a service, we need to be prepared when something goes wrong.

This is a very stable service, but when you design a solution that needs to handle millions of messages every day you need to be prepared for word case scenarios. By default, Service Bus is not geo-replicated in different data centers, because of this if something is happening on the data center where your namespace is hosted, than you are in big troubles.
The most important thing that you need to cover is the case when the Service Bus node is down and clients cannot send messages anymore. We will see later on how we can handle this problem.
First of all, let’s see why a service like Service Bus can go down. Well, like other services, this has dependencies to databases, storages, other services and resources. There are cases when we can detect pretty easily the cause of the problem.
For example when we receive ‘ServerBusyException’, then we know that the service don’t have enough resources (CPU, memory, …) and we need to retry later. The default retry period is 10 seconds. It is recommended to not set a value under 10 seconds.
This problem can be resolved pretty easily with partition. When we are using partitioning, a topic or a queue is spitted on different messages brokers. This means that we have less chances to have our service down. Also, if something happen with one of our brokers, we will still be able to use the topic/queue without any kind of problems. Don’t forget that brokers will be on the same data center. Using this feature don’t increase your costs.
Enabling this feature can be done in different ways. One option is from portal, Visual Studio Server Explorer or from code.

NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString("...");
TopicDescription topicDesc = new TopicDescription("[topicName]")
{
  EnablePartitioning = true;
}

namespaceManager.CreateTopic(topicDesc);

It is so simple to use it. You should know that in this moment you can have maximum 100 topics/queues per namespace that has this feature activated, but I expect this value to change in the future. You should also know different behaviors that are happing when you are using sessions:

• Partition Key – Messages from the same transaction, that has the same partition key, but don’t has a session id will be send to the same broker.
• Session Id – All messages with a specific session if will be send to the same broker.
• Message Id – Messages that are send to a queue/topic with duplicated detection activated will be send to the same broker. Because of this I recommend to use messages duplication detection only where is necessary.
• None of above – Messages are send to all brokers in a round robin manner – one message to each broker.

Another downtown cause can be Service Bus service is upgraded. In this cases, the service will still work, but we can have 15-20 minutes latency until the message will appear in the queue/topic. The most important thing in this case is “We don’t lose any message”.
Also when the system is not stable (internal causes), brokers will be automatically restarted. The restart can take one or more minutes. In this case the service will throw MessagingException or TimeoutException. This problem are resolved build in by the clients SDK’s (if you are using .NET SDK). They have a retry policy build-in, that will retry to resend the message. If the retry policy is not able to send the message, an exception is throw that can be handled in different ways. Until now, all the issues related to this were handle by retry policy with success.
Custom configuration of retry policy can be made in the factory class of messaging.

MessagingFactory messagingFactory = MessagingFactory.Create();
messagingFactory.RetryPolicy = RetryExponential.Default;

The last main cause of failing is external causes like internet connectivity problem, electrical outage or human errors. This problem is handled with a very different approach. The client needs to detect this problem and handle it. Until now, this required a custom code to be written, that would redirect the messages to a topic/queue that is in another datacenter (namespace).
From now we can use paired namespace to handle this scenario. The paired namespace give us the possibility to specify a second namespace (that can be in a different data center) that will be used to send messages until the primary one will be up and running. When messages are send to the second namespace, messages will be persisted until the primary namespace will be up. In the moment when the primary namespace is running, all messages from the second one will need to be redirected to the first one. We can imagine secondary namespace as a buffer that is used to store messages until our main namespace is in good state.
 When we configure this feature, we can set also the failover interval. This is the time interval when our system will accept failovers before switching to the second namespace. The recommended (and default) value is 10 seconds. Also you will need to specify the number of queues that are used to store the messages in the secondary namespace (default value is 10). This value should be greater or equal to 10.
The last option that you should be aware is syphon (‘enableSyphon’ parameter). When you activate this on a client, you tell to the system that this is the system that will transfer the messages from the second namespace to the first one. Usually this value should be set on the consumers clients (backend), because usually clients only send messages to the topics/queues.

NamespaceManager primaryNM = NamespaceManager.CreateFromConnectionString("...");
MessagingFactory primaryMF = ...
NamespaceManager secondaryNM= NamespaceManager.CreateFromConnectionString("...");
MessagingFactory secondaryMF = ...
SendAvailabilityPairedNamespaceOptions sao=
    new SendAvailabilityPairedNamespaceOptions(secondaryNamespaceManager, secondaryMF);
primaryMF.PairNamespaceAsync(sao).Wait();

There are some small things that we should know related to this feature:

• The state and order is guaranteed only in the first queue. When using session, the order of the messages is not guaranteed when secondary namespace is used
• Messages are consumed only from the primary queue/subscription
• You will pay the extra cost of moving messages from the secondary namespace to the primary one
• The default name of the queues that are created on the secondary namespace is ‘x-servicebus-transfer/i’ (where ‘i’ can have a value from 0 to n)
• The queue from the secondary namespace is randomly chosen
• It is not recommended to change the configuration of the queues from the secondary namespace

We saw that we have different mechanism to handle this special scenarios. We don’t have a mechanism that handle all this use cases. Before starting to think about integrating all this feature ask yourself if you need all of them? There are cases when a 10-15 minutes downtime is acceptable.

Read More

Windows Azure Service Bus - What ports are used

Windows Azure Service Bus is a great mechanism to distribute messages across components of your own system or to different clients. When you want to use this service for enterprise projects you will meet the IT guys.
For them it is very important to control the ports that are used by applications. Because of one of the first’s question that they will ask you is:
What are the ports that are used by Service Bus?
When this question is asked you should be prepare to have the answer prepared. Looking over the documentation from MSDN, the following ports are used by Windows Azure Service Bus:

• 80 – HTTP connection mode
• 443 – HTTPS connection mode
• 5671 – Advanced Message Queuing Protocol (AMQP)
• 5672 – AMQP
• 9350 – WCF with connection mode Auto or TCP using .NET SDK
• 9351 – WCF with connection mode Auto or TCP using .NET SDK
• 9352 – WCF with connection mode Auto or TCP using .NET SDK
• 9353 – WCF with connection mode Auto or TCP using .NET SDK
• 9354 – WCF with connection mode Auto or TCP using .NET SDK

I recommend to use HTTP/HTTPS connection where is possible. IF you are using .NET SDK you don’t need to make any custom configuration. When you let connection mode set to AutoDetect, the SDK will check if a connection can be made using non-HTTP ports. If the endpoint cannot be reached, that he will try to go over HTTP.
If you want to control the connection method, than you will need to set the ‘Mode’ property of the SystemConnectivity. The supported mode are:

• AutoDetect
• Http
• Tcp

ServiceBusEnvironment.SystemConnectivity.Mode = ConnectivityMode.Http;

Enjoy!

Read More

Simple load balancer for SQL Server Database

Two weeks ago I started to work on a PoC where the bottle neck is the database itself. We had a lot of complicated and expensive queries over the database. Because of this, if we wanted to have good performance we had to find a solution to have more than one instance of SQL Server.
Because the solution was on Windows Azure, we wanted to test different configurations and solution. We wanted to see what the performance is if we use 1, 2 and 3 instances of SQL endpoints. Not only this, we had different times of SQL endpoints –Azure SQL (SAS), Virtual Machine with SQL Server and SQL Azure Azure SQL Premium. (SAS but with dedicated resources).
 The good part in our scenario was that the data don’t change very often. We could start from the assumption that the database will be updated only one time per day. Because of this we could use different methods to create a load balancing from our SQL instances.
Because the time was very limited and the only think that we needed was to distributed the load on all the machines we decided to replicate the data on all the SQL instances and create a simple load balancer from code. We divided the tick time (timpstamp) with the number of instances and based on this we selected the SQL instance that will be hit. A simple mechanism that can be very easily managed for the PoC. Of course the final solution will be more complicated, but for a PoC this solution was perfect. In 10 minutes we had a load balancer  :-).
We checked the load of each machine and we can say that the queries were pretty good distributed. The load of each machine was almost the same (+/-10%)

What others solution we could use?
Master-Slave – In this case we have multiple instances of SQL servers. Each slave can be used for reading operation and the master will be used for writing operations.
Sharding – The solution that we used to resolve our problem
AlwaysOn – The base concept is the same one as for Master-Slave
Transaction Replication – Is based on a snapshot and when data is changing, all the subscribers are notified about the change. You can filter individual database objects and publish changes to subscribers
Peer-to-Peer Replication – Is based on Transaction Replication, but with some new features
Change Tracking – Is a very base and simple tracking mechanisms.  The only thing that is notifing you is that a specific row had change. You don’t know any information related to the old or new value
Change Data Capture – Writes in the logs all the changes that are made over a table and are used to track and synchronize the rest of the database instances

Conclusion
We have a lot of solutions on the marker and for SQL Server. Based on our needs we needed to select the best one for us. I was surprised to see that a simple load balancer, written in 5 lines of code can do a pretty good stuff.

Read More

VM and load balancer, direct server return, availability set and virtual network on Windows Azure

In today post we will talk about how we can configure a load balancer, direct server return, availability set and virtual network when working with virtual machine on Windows Azure.
For virtual machines we cannot talk about load balancer without talking about endpoints. Endpoints give you the possibility to specify the port (endpoint) that can be public accessed. For each endpoint you need to specify the public port, private port and protocol (TCP,UDP). The private and public port can be very useful when you want to redirect specific calls to different ports. If you don’t specify the endpoints for a virtual machine, the machine cannot be accessed from outside (for example when you host a web page and you need to have port 80 open).  
For web and worker roles this load balancer is out of the box support. We could say that it is the same think for virtual machines also, but you need to make some custom configuration. You will need to specify that you will use the load balancer functionality when you create an endpoint. Once you create an endpoint that is load balanced, the only think that you will need is to specify to the second, 3rd … machine from that will be included in the load balancer the endpoint that use a load balancer set (“Add an endpoint to an existing load-balancer set”).
Direct server return is another feature of endpoints. It can be very useful when you have want each machine to response to the client directly, without the load balancer routing. In this way, once a client start to communicate with a machine, the connection will stick with only that virtual machine (stick connections) – useful when you have a DB on machines or you have an in-memory session that is not shared between machines.
Virtual network give you the possibility to create a private network formed from virtual machines from Windows Azure. When you need to share content between machines, this will be the best approach. Even if you have machines that are under the same subscription, you will not be able to access them using their private IP or name. You have almost the same access level from one machine to another as you would have machines under different subscription.
Because of this when you need to communicate between machines that are behind the same DNS name of public IP this will be the best approach. For example you will not be able to access a specific instance that is behind a load balancer to update content or call specific endpoints.
A virtual network can be created very easily from the Network tab. You have a lot of options from custom DNS names for specific machines to custom address spaces. Once a virtual network is created you can very easily create new virtual machines to it.
Remarks: You should take into account that you cannot add an existing virtual machine to a new virtual network. A fix for this problem is to stop and delete the machine without deleting the disk itself and recreate a new machine based on that disk
Not last, let’s talk a little about availability set. This is a pretty interesting setting. When you create two or more machines for the same purpose you don’t want the resources of the machine (memory, processor, and store) to be in the same rack. Why? If the rack will stop working that you will have both machines down and the load balancer will not be able to help you. In this case when you create a availability set you will have the guarantee that the machines under the same availability set will not be under the same rack. This will give you the availability of 99.95% that is offered by SLA.
The feature that I enjoy the most is the load balancer in combination with endpoints. It is a great feature that can help you a lot.

Read More

Custom Content Delivery Network Mechanism using Cloud (Azure)

Requirements:
Lets’ imagine an application that need to deliver Excel report in different location of the world. Based on the client origin the application should be able to deliver specific Excel version. The application should be deployed in different location on the world.
The current CDN solution cannot be used because the master node needs to push the content to the slaves (CDN nodes) and the total report size for each slave will be over 20GB.
Non-Cloud solution:
If we would go on a non-cloud solution we should develop an application that is deployed on different location all around the world. Each application should be able to detect the source of the request and provide the specific Excel report. We should also develop a redirecting mechanism/ load balancing solution that is able to redirect the user to a specific node.
Cloud solution:
If we go on a cloud solution, based on Windows Azure we can imagine a master slave solution.
Each slave of our application will have the Excel reports for the country in his region. This slaves will be able to provide reports for the countries in his own region.
Having an application deployed on different data centers give us the possibility to use Traffic Manager. Traffic Manger is an out of the box mechanism that redirect a call to the closest data center where our application is deployed.
We can have our slaves deployed on different data centers around the globe. Each slave will have the reports for the countries that are served by him. When a request is coming from a country that is not served by the specific slave, the request will be redirected to the global slave, which has the reports for all the countries.
This slaves could detect if there to many requests that are coming for a country that is not mapped for their location and trigger an alert of the provisioning action for reports for that country.
Each slave will have an endpoint that will be used to resolve an Excel report request and a storage (blobs) that will be used to store the reports itself. Based on the client attributes, this service will return a URL with a Shared Access Signature (SAS) of a blob storage where the report is stored. Using SAS the access to the content will be controlled.
The solution will contains a master that will manage all the Excel reports from all the nodes. The master will be able to deploy new version of reports, delete the old one and so on from each slave node. Beside this, the master will the one that can trigger the provisioning of a slave with additional countries. The master will contains a storage (blob) with all the reports that exists and are valid and a service that is able to manage and maintain all the slave nodes.
When the provisioning is triggered, the download process to a specific slave should not be done by the master node. Because we can have a lot of slaves, this action consume a lot of resources and can give us a lot of problems. The master node should send a notification to a specific slave that a specific report is available for download/update/delete. In that moment the slave node should receive the notification and trigger the specific action. In this way we are able to move all the load from master to slaves.
The notification mechanism can be done over Service Bus. Each slave node will be represented by a different subscription. When the download/update/delete action is finished, the slave node can send a notification to the master node using a queue or a Service Bus Topic.

Things that I like to this solution:

• Traffic Manager – Is able to automatically redirect request and when he detect a slave node is down redirect to the next slave
• SAS – The content of the blob can be shared in a secure manner
• Slave’s endpoint – If we have slaves that are hit by a lot of clients we can scale up the numbers of instances of that slave without affecting the rest of the slaves
• Redundancy – When a slave is down all the request will be redirect to the closest slave
• Report resolver – When a report cannot be resolve by a specific slave, the request is resolve by the global slave that is able not only to log this issues but also he can notify the master node about this incident. In this way the master node can trigger custom action like provisioning
• Scalability – Each slave can scale independently based on the load
• Provisioning mechanism – The provisioning is made using the slave processor resource. In this way the master node will not have peaks
• Service Bus – Notifications from master to slave can be made using Service Bus Topics. In this way we can have one or more slaves register to the same countries
• Download – The download itself will be made directly from Azure Storage. The load of the slaves itself will be minimal

We could have a similar approach and eliminate the endpoints from slave. Each slave could have only the storage part. This is a good solution when you the number of request that need to be handled is not very high. But when you have hundreds of request per minutes, that a solution like this is more suitable. The slave endpoints can be hosted on small instances.

Read More

CMS application for 3 different mobile platforms – Hybrid Mobile Apps + Mobile Services

This days I attended to a session where an idea of a great mobile application was presented. From the technical perspective, the application was very simple, the content that is deliver to the consumer has a great value.

The proposal of the application was 3 different application for each platform (iOS, Android and Windows Phone). When I asked why it is so important to have 3 different native application for an application that has the main purpose to bring content to the users I found up that they need push notification support and all the rest of the content that will be displayed will be loaded from the server (CMS).
Having 3 different native application means that you will need to develop the same application 3 times for 3 different platform…. 3X maintenance 3X more bugs and so on….
In this moment I have a rendevu. I have the feeling that I already wrote about this.
For this kind of application you develop a great HTML5 application. For push notification support it is very easily to create 3 different native applications that has a WebBrowser controller. In this controller you can display the HTML content of your application. Only custom behaviors, like push notification will need to be develop for the specific platforms.
In this way, the applications will be very simple and the costs of developing and maintenance will be very low.
For push notification I recommend to use Mobile Services for Windows Azure. It is a great service that can be used to push a notification to all devices and platforms with minimal development costs.

Read More

How to track who is accessing your blob content

In this post we’ll talk about how we can monitor our blobs from Windows Azure.
When hosting content on a storage one of the most commune request that is coming from clients is

• How I can monitor each request that is coming to my storage?
• For them it is very important to know 
• Who downloaded the content?
• When the content was downloaded?
• How the request ended (with success)?

I saw different solution for this problem. Usually the solutions involve services that are called by the client after the download ends. It is not nothing wrong to have a service for confirmation, but you add another service that you need to maintain. Also it will be pretty hard to identify why a specific device cannot download your content.
In this moment, Windows Azure has a build-in feature that give us the possibility to track all the requests that are made to the storage. In this way you will be able to provide all the information related to the download process.
Monitor
In Windows Azure portal you will need to go to your storage and navigate to the Monitoring section. For blobs you will need to set the monitoring level to Verbose. Having the monitoring level to Verbose all metrics related to your storage will be persisted. The main different between Minimal and Verbose is the level of monitoring.
This data can be persisted from 1 day to 1 year. Based on your needs and how often you collect the data you can set the best value that suites you. If your storage is used very often I recommend to set maximum 7 days. You can define a simple process that extract monitor information from the last 7 days, store it in a different location and analyze it using your own rules. For example you may way want to raise an alert to your admins if a request coming from the same source failed for more than 10 times.
This table contains all the monitoring information for your storage. In this moment we don’t have support to write monitoring data for a specific blob to a specific table, but we can make query over this table and select only the information that we need.
All the information related to this will be persisted in Azure table from your storage named ‘$MetricsCapacityBlob’.
Logging
The feature that we really need is logging. Using logging functionality we will be able to trace all request history. The activation of this feature can be done from the portal, under the logging section. You can activate logging for the main 3 operations that can be made over a blob: Read/Write/Delete.
All this data is stored under the $logs container:
https://<accountname>.blob.core.windows.net/$logs/blob/YYYY/MM/DD/hhm/counter.log
Everything that we can imagine can be found in this table:

• Successful and failed requests with or without Shared Access Signature
• Server errors
• Timeouts errors
• Authorization, network, throttling errors
• ...

Each entity from the table contains helpful information like:

• LogType (write, read, delete)
• StartTime 
• EndTime
• LogVersion (for future, in this moment we have only one version – 1.0)
• Request URL
• Client IP
• …

The most useful information for is usually found under the ‘Client IP’ and ‘Request URL’. Maybe you ask yourself why we have a start time and an end time. This can be very useful for a read request for example. In this way we will be able to know how long the download process took.

I invite you to explore this feature when you need to track the clients that access your blob resources.

Read More

Types of Subscriptions Filter on Windows Azure Service Bus

Service Bus from Windows Azure is a great services to distribute messages to multiple subscribers. A topic can have one or more subscriptions. A message added to the topic will be send to all subscriptions that are register in that moment.
To be able to control what kind of messages are send to each subscription we can define filters. A filter is added on each subscription and specify what messages will be accepted by subscription.
The types of filters that can used are:

Correlation Filter
This is used when you want to accept messages that has a specific correlation value. This value can be set directly to the BrokenMessage and is great if you have messages with different priority or rank.

CorrelationFilter filter = new CorrelationFilter(“low”);
namespaceManager.CreateSubscription(topicPath, subscriptionName, filter);

BrokenMessage message = new BrokenMessage();
message.CorrelationId = “low”;
…

SqlFilter
This filter give us the ability to specify what kind of messages will be accepted in a subsection based on a custom rules. This rule can be related to the properties list of a BrokenMessage that are specific by the producer or system properties (SessionId for example is a system property).
The syntax is very simple and is based on SQL92 standard and T/SQL. If you know to write a query in SQL than you will not have problem to use this filter. The syntax give us the possibility to use = != < > <> IS IN LIKE EXISTS ESCAPE NO + - * / AND OR …
For a normal use this is enough to define rules over subscriptions.

SqlFilter filter = new SqlFilter(“CustomProperty == 1”);
namespaceManager.CreateSubscription(topicPath, subscriptionName, filter);
…
BrokenMessage message = new BrokenMessage();
message.Properties[“CustomProperty”] = 1;

The “user” or “sys” prefix need to be used when you refers to properties that are in different scope. The “sys” refers to properties that are on BrokenMessage (like Correlationid or SessionId).

FalseFilter
This is a filter that can be used to block all the messages to the given subscription. This can be used if you create a subscription and you don’t want to accept message to it (temporarily). Also, this can be used when you have a subscription and you don’t want to receive message anymore to it, but you don’t want to delete it.

FalseFilter filter = new FalseFilter()
namespaceManager.CreateSubscription(topicPath, subscriptionName, filter);
…

TrueFilter
This filter is similar with SqlFilter. It will accept the message that respect the specific rule.

We looked over what kind of filter can be defined over a subscription. I think that FalseFilter can be very useful when you are in production and you need to control what messages are receive by one of subscription. For example in a scenario when a client didn’t pays the monthly subscription and you want to stop temporary his subscription. Is more simple that deleting it or set a SqlFilter – “ 1 <> 1”.

Read More

Why we have two accounts key for Windows Azure Services?

In today post I want to talk a little bit about the account keys from Windows Azure. There are a lot of services from Windows Azure that can be consumed using an account name and key. The account name is only one and in some situations is represented by the service namespace. In all cases the account key is duplicated. There are two account keys that can be used in parallel. Both of them can be used in the same time without problems.
But why we have two accounts key?
Well, we have two accounts key with a very good purpose. Let’s imagine the case when one of the accounts key is compromised. In this case you need to generate another key and distributed it to the clients. From the moment when you generate the new key, until you distributed it to all the clients, the specific service will not be able to be used – if you use only one key.
In the situations, when a key is not valid anymore, clients could fall back to the second key and start to use it. This mean that all the clients should access to the both accounts key.
The same scenario can be used when you need to regenerate the accounts key at a specific time interval (every 3 months). In this cases you can define a process to regenerate the accounts key without affecting the clients.

Read More

Service Bus and Shared Access Signature (SAS)

Until few months ago, Shared Access Signature (SAS) could be used only wilt Azure Storage (Blobs, Tables and Queues). From now one we can use SAS with Service Bus. We can define SAS to use in combination with topics, queues or notification hub
This security feature is pretty great, especially when you have an application that use 3rd party. You don’t need any more to share with 3rd parties the account name and key. From now one you can give them a unique token that give them access to some part of your namespace services and also limit what kind of operation they are allow to do.
In this moment you are allow to define only 12 rules in a namespace, but in the near future I expect to be able to define more than 12 rules.
The access rights that can be controlled in this moment are:

• Listen – to be able to receive message
• Send – to be able to send messages
• Manage – to be able to manage the resource

From Service Bus perspective, this access rights are enough and in combination with expiration date it works pretty good. Be aware, that you can set Manage access rights, only if you set also the Send and Listen rights.
The interesting thing is how the rules are defined. The rules are defined over a URL. This means that you can define a rule over a namespace or over a specific topic or queue. The SAS will be valid for all resources under the specific URL.
When you generate a SAS rule, two keys will be generated for the same rules. Both keys can be used in the same time. This is done to help customers in the moment when they need to generate new keys and they don’t want to block the access to Service Bus.
The SAS rules can be defined and manage not only from code (using REST API) but also from the management portal of Azure. The portal can be very helpful when you are in the development phase or you have some issues with SAS and you want to check what rules are already defined.

How to create a SAS from code
First step is to create a SharedAccessAuthorizationRule and set the specific rights.

SharedAccessAuthorizationRule saar = new SharedAccessAuthorizationRule(
    “myFooName”,
    SharedAccessAuthorizationRule.GenerateRandomKey(),
    new[] {
        AccessRights.Manage, 
        AccessRights.Listen, 
        AccessRights.Send }));

GenerateRandomKey method is used to generate random keys.
Once the rule is created, you can add it to the queue, topic or notification hub using the description class

QueueDescription qd = …
qd.Autothorization.Add(saar);

Don’t forget to save the name of the name and the key of the rule.

How to use a SAS key
Once you created the rule and have the name and the key of the rule, you will be able to use them in the moment when you create the MessagingFactory.

MessagingFactory mf = MessagingFactory.Create(
    "uri,
    TokenProvider.CreateSharedAccessSignatureTokenProvider"myFooName", "myFooKey"));
QueueClient queue = mf.CreateQueueClient("myFooQueue");

Enjoy!

Read More

Load Test - Binding Context Parameters

The new Load Tests features that are coming with Visual Studio 2013 are great. In combination with Windows Azure and TFS Controllers, you can run load test using cloud infrastructure without having to deploy or configure something things on the cloud.
When developers start to create different web test that will be used for load tests they usually them in sub-tests and reuse them when is possible. For example very easily you can end up with something like this:

• Foo1WebTest
• Foo2WebTest
• Foo3WebTest
• BigFooWebTest

Where BigFooWebTest make calls to Foo1WebTest and Foo2WebTest.
If you started to extract context parameters from each web test you will observer a pretty odd behavior. By default it is not possible to bind a context parameter. This means that you will not be able to fetch data for a context parameter from a data source like a csv file or data base.
Even if you will write to the value of the parameter context the binding command (“{{..#TableName.CollumnName}}”) you will observe that this will not work. When you will run the test, the value of the context parameter will be the string value itself “{{..#TableName.CollumnName}}”.
In this moment it is not possible to bind a context parameter. The only available solution in this moment is to write a plugin for the test that will load the values from data source in memory. This plugin will be able to set the value of context parameter with the memory values.
For more information about load test plugin: http://msdn.microsoft.com/en-us/library/ms243153.aspx

Read More

More about 'OnMessage' method of Service Bus

If you are using Service Bus, you should know that you have the ability to be notified when a message is available in the queue or in a subscription. Using this feature you don’t need to call the "Receive" method and catch the timeout exception when there are no new messages are available.

OnMessageOptions messageOptions = new OnMessageOptions()
  {
    AutoComplete = true,
    ExceptionReceived += LogErrorsOnMessageReceived
  };
queueClient.OnMessage((message) => 
  {
                ...
  }, messageOptions);

As you can see, you register to an event that will be called when a new message is available. If there are more than 1 messages available the event will be triggered more than one. Each notification will run on a different thread. To be able to control the number of concurrent messages that can be received by a client you should change the value of "MaxConcurrentCalls".

OnMessageOptions messageOptions = new OnMessageOptions()
  {
    AutoComplete = true,
    ExceptionReceived += LogErrorsOnMessageReceived
    MaxConcurrentCalls = 5
  };
queueClient.OnMessage((message) => 
  {
                ...
  }, messageOptions);

But what about the costs?
When using Service Bus, you pay for each transaction. Because of this each call of “Receive” method will consume a transaction. Even if there are no available messages and the call finish with a timeout exception, you will have to pay for one transaction.
Base on this, we should know that a similar thing is happening when we are using “OnMessage”. The behind implementation of this method use “Reveive” method and has a timeout value. Because of this, even if you register to “OnMessage” once you will notify that more than one transactions are consumed even if you don’t have messages in the Service Bus.
This is happening because “OnMessage” calls “Receive” method that will consume a transaction when a message is available or when a timeout exception occurs. The default timeout value is 60 seconds. This timeout value can be easily change using MessagingFactory.
From the cost perceptive you should not have any kind of problems. The cost of each client that use “OnMessage” method and has the timeout value set to 60 seconds is 4.3 cents per month.
In conclusion “OnMessage” is very useful because offer out of the box a mechanism that give us the possibility to be notified when a message is available – until now we had to implement this mechanism every time. You should not forget that this mechanism will handle each new message on a different thread and there are times when you want to control the concurrent level.  Also it generates some costs, but the cost is similar with the “Receive” mechanism – in the end “OnMessage” use “Receive”.

Read More

Load Test using Windows Azure and Visual Studio 2013

In a perfect world I would except to be able to run a load test or a stress test using a cloud provider without being forced to change any line code from my tests. 

In this article we will find out how to do this thing using the new cloud service. How a person without knowledge of cloud will be able to run a load test with 50.000 users for 24 hours with minim costs, without being obliged to purchase and configure 10, 20 nodes.

What is a load test?
What is a load test in fact? An expression that I think it fits when we need to answer this question would be: ”Are you ready to be so popular?”.  The main scope of this kind of test is to define and validate the maximum load that a system can have without affecting the performance. Even though the developer tells that the system can handle without any problem 1000 users in the same time, the load test can prove the contrary even on simple scenarios.
There is a tiny difference between the load test and stress test. Many times the load test finishes with a stress test that has the purpose to observe the behavior of the system in the moment when the load increases above the maximum supported capacity.

What the market offers us now?
At this moment it is full of different solutions for automated testing and for running load tests. Products like Selenium, LoadStorm or Neoload dominate the market. Besides them there are many others and I don’t think 10 pages will be enough to enumerate them.
If we use an on-premise solution then we need to be prepared to resolve things like the acquisition and reservation of the resources for the load-test. In a small company it will be very hard to buy 10 servers just to run the load-test. A big company, even if it has the financial resources it will take a while until this resources will be available and sometimes it might be too late. Besides all this factors the configuration of each machine and deploy can be the labyrinth from which we won’t be able to find a way out.

What is the Microsoft proposal?

On this extremely mature market, Microsoft appeared with an ingenious solution. If they already have a powerful and robust infrastructure for cloud why don’t use it to run load tests using Azure. Even though they are not the first ones that are offering this kind of solution, Microsoft has an advantage. They allow you to run load tests using Azure without using a new configuration. All you have to do is to have an account on Visual Studio Team Foundation Service (http://tfs.visualstudio.com/) which will be used for the receipt.
 Before a load test using Microsoft could have been done only through Load Test Ring. This if formed from a controller that controls the tests- Load Test Controller and one or more agents on which our tests run. An architecture that is based on master-slave. The new version of Visual Studio 2013 provides us a new option- instead of running our test in our own Load Test Ring we can run them on Azure, without a new configuration. We don’t have to deploy any virtual machine or to configure different services.
 
Load Test Web Service
Load Test Web Service is the new Azure service that helps us. Through this service, Visual Studio 2013 loads our tests on the cloud. Behind this service is a pool of test agents that is used to run our tests. This thing happens behind the scene and we don’t have to do anything for this to run. All the results from our test together with other performance counters are available for us.

Which are the main characteristics?
Before looking over different functionalities that are supported I propose to first take a look at the most important options that are supported now.
We have the possibility to define UI tests and more. Beside the fact that we can record our UI tests we can also write our custom tests and hit different endpoints. The tested endpoint doesn’t have to be only HTTP or HTTPS. We can also test a WCF or REST endpoint and also a web service. The only condition is that the endpoint must be accessible via internet.
Visual Studio 2013 brings a project template- Web Performance and Load Test Project. This type of project will allow us to define and run load tests on cloud.

How to create a new load test?
The easiest way to create a load test for a web application is to create one or more Web Performance Tests. This type of test can be UI tests which are very easy to create and to automate. Using the UI recorder that comes with Visual Studio 2013 we can create a Web Performance Test in just a few seconds. This type of test can be modified at any time and if we want we can generate code for the test. Through this method the ones that wish to write code to modify the code can do this very easily. Of course that functionality like automatically detection of the dynamic elements that the page has or to extract different constants as parameters is supported out of the box.
For each test of this type we can generate different sources like database, CSV or XML file. You can also use the tests written in Selenium for example. Each test can call other test and this way we can reuse the logic that we already have for testing.
Until now we saw how we can create a test can be used for the load test. It is time for us to see what a load test allows us to do.

Which are the main functionalities?
The first thing we need to do in order to create a load test is to THINK. We can define different profiles and time intervals through which we should be able to simulate a real user. For example we can simulate a delay of X seconds after each test. We have available different profiles which we can use.
The way we simulate a load test can be made through different methods. We have the possibility to run a test with a specific number of users or we can define the number of users to increase at every step. Of course that most of the times we will have to run different scenarios in the same time. That why we can select the tests that we want to run and in what proportion, but also the way that this test have to run. For example we can specify how many times a test should run in a time interval or which is the number of users that have to run a specific test in every moment while the load test is running.
We can simulate different browsers that the clients might have and different types of connections. The most interesting thing is that we have the possibility to add and access not just performance counters from the clients but also the ones from the servers on which our application is running. This way we can monitor and see the counters from both the client and the servers.

How to run a load test in cloud?

Until now we saw the main options that we have available to run a load test, but we didn’t saw how to run this type of test. All we have to do to run the load test on cloud is to open the Local.testsettings and on General tab to select “Run tests using Visual Studio Team Foundation Service”. This is the only thing we have to do to run the test on cloud. Of course that we have to be logged uni in Visual Studio with an account that is connected tp Visual Studio Team Foundation Service.

What is the price?
For this moment the service is on preview. Each user has 2000 virtual minutes per month that can use for the load tests. There aren’t fixed prices for this. If you need more minutes for load tests all you have to do is to enter in the early adoption program without additional costs. You will have 200.000 virtual minutes per month that you can use. Accepting the request takes very little time (in my case around 6 hours).
The first time I heard about virtual minutes I wondered what these are. A virtual minute is the duration of the load test multiplied with the number of users.
Once we run the tests we will have access to all the results including performance counters, failed tests, error messages and different diagrams. All the results can be exported into an Excel that will automatically contain the diagrams that you can show it to the clients.

I invite you to try the new cloud service for load tests. I was pleasantly surprised by this service and I think it will make our life easier.

Read More

[Post Event] Summer Codecamp at Cluj-Napoca (July 31, 2013)

Last afternoon we had another Codecamp event in Cluj-Napoca. Around 70 people came to find out more about Team Foundation Server 2012, Virtual Machine Manager 2012 and how we can make load tests using Visual Studio 2013 and Azure infrastructure.
Special thanks for THE sponsors – YONDER and CORESYSTEMS.

Recorded sessions:
Managementul testelor cu Team Foundation Server 2012 si System Center Virtual Machine Manager 2012, Adrian Stoian

Load Tests using Visual Studio 2013 and Azure Radu Vunvulea

The slides of my session:

Load tests using Visual Studio 2013 and Azure from Radu Vunvulea

And in the end you can find some pictures from the event:

Read More

Traffic Manager Overview

Starting from today we have a mechanism that give us the possibility to control the traffic that comes to our Azure services. The name of this service is Traffic Manager.
What does this means?

Performance Load Balancing
Well, the simplest scenario is when we have a service running on different data centers. In this case we want to be able to redirect users to the closest data centers. We could have a service that identifies the location of the user and based on this redirect him to a specific data center. This problem is resolved by Traffic Manager Service. Using the client IP, this service will identify the location of the client and will redirect him to the closest data center (the one that have the lowest latency).
To be able to monitor the performance of each endpoint you will need to specify a relative path to the resource that is monitored. The monitor part is pretty simple, the latency time of each endpoint resource is measure every 30 seconds. When one of the request exceed 10 seconds or the return request code is different than 200 for more than 4 times in a row the endpoint will be considered down.

Failover Load Balancing
Another scenario that is cover by Traffic Manager is the case when one of our services from a data center is down. In this case the Traffic Manager will be able to detect the failover of the service and redirect the traffic to another data center. In this way all the traffic will be redirect to a backup service. We can define the order of the endpoints. This means that if the endpoint one will be down, the Traffic Manager will try to redirect the traffic to the second endpoint. If the second endpoint is down, the traffic will be redirect to the 3rd one and so on.
The performance Load Balancing also monitors the status of the endpoint and will not redirect traffic to an endpoint that is down.

Round Robin Load Balancing
This is the classic case of load balancing. In this case we have 2 or more endpoints available. The first client is redirected to the first endpoint, the second client to the second one and so on. This is a simple and very efficient way to make load balancing.
Also in this case, the Traffic Manager Monitoring component will redirect traffic to the endpoints that are up and running.

A normal question is when does the Traffic Manager appear on the requested map. For example if we have a domain foo.com and we will create a traffic manager domain named foo.trafficmanager.net. When a request will come to our website DNS name the request will be redirect to the foo.trafficmanager.net. Based on the policy that we use the traffic manager will redirect the client request to one of our endpoint.
Of course the latency of our system will increase at first request, but this value will be very low. In normal cases I would consider this value equal to zero and is not relevant for normal web applications.
Also, you should know that the resources of the endpoint that is used to check if the latency of the service needs to be over HTTP or HTTPS protocol. If your services works with different protocols that you need to add a HTTP or HTTPS resource – this can be a simple resource like a small file.
Another important thing to do after you configure the traffic manager is to update the DNS resource record to redirect the request from foo.com to foo.trafficmanager.com.
What do you think about this service? Do you think that you will use it in the near feature?

Read More

Subscription count - (Part 7) Testing the limits of Windows Azure Service Bus

Let’s talk about Windows Azure Service Bus and the numbers of subscriptions that a topic can have. I started to prepare a POC for a possible client and it is possible to end up with hundreds of subscriptions on each topic.
Looking over the documentation from MSDN, each topic supports 2000 subscriptions. I wanted to see what is happening with the latency of each subscription if we have over 1000 of them per topic.
To find an answer to my question, I started to write a worker role that:
Creates a new subscription
Sends a message to the topic
Receives a message using the first subscription
I measured how long it takes to send a message to a topic that has 1, 2, … , 2000 subscriptions. The same think I’ve done for the first subscription of the topic. I measured how long it takes to read a message from it when we have 1,2, … , 200 subscriptions.
The code is extremely simple but the results are very interesting.
The first diagram shows how long the send and receive commands took. The blue color is for the send request to the topic and the orange one is for receive command for the first subscription of the topic. The AX represent the number of subscriptions per topic and the OY represent how long a command took (in milliseconds).

As you can see, the command latency doesn’t increase too much. Even for 2000 subscriptions the latency is under 100 milliseconds.
The next diagram show the average latency when we have 500, 1000, 1500 and 2000 subscriptions. Based on this result we observe that the latency increases with a factor under 1.5X.

We can say that we can use this service in our business scenarios for 2000 subscribers without a problem. The latency increases only with a factor that is under 1.5X. We could have a problem if the latency would increase with a factor of 10X or 100X. But we are in a safe zone.
It seems that Service Bus is a real cloud service that scales and works excellent.

Read More