Service Bus - Optimize consumers using prefetch and maximum concurent calls features

From same version ago, Windows Azure Service Bus supports 'event notification'. This means that we can register to an event that will be triggered each time when a new message is available for us.

QueueClient client = QueueClient.Create("queue1");
client.OnMessage(
    OnMsgReceived, 
    new OnMessageOptions());
...
void OnMsgReceived(BrokeredMessage message)
{
    ...
}

This is a great feature, that usually make our life easier. By default, when we are consuming messages in this way, we will make a roundtrip to the Service Bus for each message. When we have applications that handle hundreds of messages, the roundtrip to the server for each message can cost us time and resources.
Windows Azure Service Bus offer us the possibility to specify the number of messages that we want to prefetch. This means that we will be able to fetch 5, 10, 100, .. messages from the server using a single request. We could say that is similar to the batch mechanism, but can be used with the event notification feature.

QueueClient client = factory.CreateQueueClient("queue1");
client.PrefetchCount = 200;

We should be aware that the maximum numbers of messages that can be prefetch by one request is 200. I think that this an acceptable value, if we take into the consideration that this will trigger 200 notification events.
This value need to be set when we create and setup the subscription client. Theoretically you can set this value before receiving the first message from the server, but I recommend to make all this configuration at the setup and initialization phase.
When we are using this mechanism of prefetching, we need to know exactly how many concurrent calls  we can have. Because of this, the Service Bus client give us the possibility to specify how many concurrent calls we can have in the same time.

client.OnMessage(CalculateEligibility,new OnMessageOptions()
            {
                MaxConcurrentCalls = 100
            });

For example if we would have the prefetch count set to 200, and the maxim concurrent calls sett to 100, we will have the following behavior:
Make a roundtrip to the server
Receive 200 messages (we suppose that there are 200 messages available)
Consume first 100 messages
Consume the other 100 messages
Messages are consumed in a async way. This means that from the first 100 messages, when 1 will be processed on the client, another message receive event will be triggered.
We saw in this post how we can consume the messages in parallel when using Windows Azure Service Bus and event notification. Using this features we can increase our application performance.
Good luck with Windows Azure Service Bus.

PS: Bonus picture - This is a picture from Seattle airport, where I am now, waiting the flight back home.

Read More

Extract relative Uri using MakeRelativeUri method

Did you ever need to compare two web address and extract the relative part of them?
For example if we have address "http://foo.com" and "http://foo.com/car/color/power" we would like to get "car/color/power". But if we would have "http://foo.com/car" and "http://foo.com/car/color/power" we would like to get "color/power".
For this cases we should use "MakeRelativeUri". This method is part of Uri class and determines the delta between two URL address. This simple method extract the difference from two URL.
Code examples:

Uri uri = new Uri("http://foo.com");
Uri result = Uri.MakeRelativeUri("http://foo.com/car/color/power");
// result: "car/color/power"

Uri uri = new Uri("http://foo.com/car");
Uri result = Uri.MakeRelativeUri("http://foo.com/car/color/power");
// result: "color/power"

Uri uri = new Uri("http://foo.com");
Uri result = Uri.MakeRelativeUri("http://foo.com/car/color/power/index.html");
// result: "car/color/power/index.html"

Uri uri = new Uri("http://foo.com/car/color/power/");
Uri result = Uri.MakeRelativeUri("http://foo.com/");
// result: "../../../"

Uri uri = new Uri("http://foo.com");
Uri result = Uri.MakeRelativeUri("http://secondFoo.com/car/color/power");
// result: "http://secondFoo.com/car/color/power"

When the base address of the second URL is not the same with the first one (base one), the call of this method will return the second URL that was send as parameter.
This method can be very useful in certain situations, when you know about it. In one project I saw an extension implemented by developers that was doing the same thing :-).

Read More

Sync Group - Let's talk about Performance

In one of my latest post I talked about synchronization functionality that is available for SQL Azure. There was a question related of the performance of this service.

So, I decided to make a performance test, to see what are the performance. Please take into account that this service is in the preview and the performance will change when the service will be released.

For this test I had the following setup:

• Database
• Size 7.2 GB
• 15 tables
• 2 tables with more than 30.000.000 of rows (one table had around 3.2 GB and the other one had 2.7 GB)
• 34.378.980 rows in total
• Database instances
• 1 DB in West Europe (Hub)
• 1 DB in West Europe
• 1 DB in North Europe
• 1 DB in North Central US
• Agent
• 1 agent in West Europe
• Configuration
• Hubs win
• Sync From Hub

Scenario One: Initialize Setup

I started from the presumption that your data were not duplicated yet on all the databases. First hit of the Sync button will duplicate the database schema of the tables that needs to be sync, table content and rest of resources to all the databases for the given table. This means that 7.2 GB were send to the 3 different databases.

Normally you can do this action in other ways. Exporting/Importing the database for example, but I wanted to see how long it takes to sync all the databases.

Sync action duration: 5 hours and 36 minutes (20160.17 seconds)

 

Scenario Two: Update 182 rows

In this scenario I updated 182 rows from one of the tables

Sync action duration: 53.63 seconds

 Scenario Three: No changes

In this case I triggered the synchronization action without any changes.

Sync action duration: 38.47 seconds

 Scenario Four: 23.767 rows updated

23767 rows were updated on the hub database.

Sync action duration: 1 minute and 16 seconds (76 seconds)

 Scenario Five: 4.365.513 rows updated

As in the previous scenario, I updated  I changed a specific number of rows.

Sync action duration: 1 minute and 41 seconds (101.6 seconds)

 Scenario Six: 76.353 rows deleted

From one of the tables I deleted 73.353 rows.

Sync action duration: 56.26 seconds
 

As we can see, the synchronization action itself takes a very short period of time. For 4.5M of rows that were updated, the synchronization action took less than 2 minutes. The only scenario that took a log period of time was the initial synchronization action. Usually this action is made only one time. Also we have other method to import the database content to all our database.

I would say that the performance of the sync service is very good and I invite all of you tot check it out. You have support for synchronization out of the box.

Great job!

Read More

[PostEvent] Slides from MSSummit 2013, Bucharest

Last week I had the opportunity to participate at MSSummit. During this event I had the opportunity to present and talk SignalR and load testing using Windows Azure and Visual Studio 2013.
More about this event: http://vunvulearadu.blogspot.ro/2013/11/postevent-mssummit-2013-bucharest.html
You can find my session slides below:

Real time fluent communication using SignalR and Cloud (Windows Azure) from Radu Vunvulea

Load tests using visual studio 2013 and Cloud from Radu Vunvulea

Read More

How to get the instance index of a web role or worker role - Windows Azure

When we have one or more instances of a specific web role or worker role in the cloud, there are moments when we want to know from the code how many instances we have of the specific type or the index of current instance.
The total number of instances can be obtained using
RoleEnvironment.Roles[i].Value.Instance.Count
To be able to detect the index of the current instance we need to parse the id of the role instance. Usually the id of the current instance ends with the index number. Before this number we have ‘.’ character if the instance is on the cloud or ‘_’ when we are using emulator.
Because of this we will end with the following code when we need to get the index of the current instance:

int currentIndex = 0;
string instanceId = RoleEnvironment.CurrentRoleInstace.Id;
bool withSuccess = int.TryParse(instanceId.Substring(instanceId.LastIndexOf(".") + 1, out currentIndex));
if( !withSuccess )
{
  withSuccess = int.TryParse(instanceId.Substring(instanceId.LastIndexOf("_") + 1, out currentIndex));
}

Take into account that when you increase and decrease the number of instances, there are situation when you can end up with the following name of the instances:

• […].0
• […].1
• […].3
• […].4

Two is missing because when we decreased the number of instances from 5 to 4, that instance was stopped.

Read More

Sync Group - A good solution to synchronize SQL Databases using Windows Azure infrastructure

Working with Windows Azure becomes something that is more and more pleasant. In this post we will see how you can synchronize multiple databases hosted on Azure or on premise using Sync Group.
First of all, let’s start with the following requirement: We have an application that contains a database that needs to be replicated in each datacenter. What should we do to replicate all the content in all the datacenters?
A good response could be Sync Group (SDS). Using this feature we can define one or more instances of SQL Databases (from cloud or on premise) that will be synchronized. For this group we can specify the tables and columns that will be synchronized.
Creating a SDS can be made very easily from Windows Azure portal. This feature can be found under the SQL DATABASES tab – SYNC. I will not start to explain how you can create this group, because it is very easily, you only need to know the databases server address, user names and passwords.
I think that more important than this is the options that we have available on a SDS.

HUB
One of the databases that forms the group needs to be the hub. The hub represent the master node of the group, from where all the data propagates.
Synchronization Direction
Once we add the hub, we can add more databases to the group. In this moment we will need to specify the synchronization direction. In this moment we have 3 options:

• From the Hub – All the changes that are made in the hub are replicated to the rest of the databases from the group. In this configuration, when data are different, the hub will win. Changes in the databases are not written to the hub
• To the Hub – Al changes from the hub are not written in the databases. All changes that are made in the databases are written in the hub
• Bi-directional – The synchronization is made in both ways – from Hub to databases and from databases to hub

Synchronization Rules
From the portal, we have the option to select the tables (and columns) from the hub table that will be synchronized. In this way you don’t need to have databases that has the same schema. The most important thing is to have the tables that you want to synchronize in the same schema/format.
Remarks: We don’t need to replicate the database schema to all the databases. Once we select the tables and columns that we want to synchronize (from Hub schema), all this tables will be replicated in the rest of the group.

Conflict Resolution Policies
When we are creating a hub, we have two option for conflict resolution.

• Hubs Wins – In this case all the changes that are written to the hub will be persisted and in a case of the conflict, the version that is on the hub will be the ‘good one’
• Client Wins – In tis case the changes that are written on the slaves (non-hub database) will win and the change from the slave will propagate to the hub and to the rest of the group.

For more information about this conflict resolution policies I recommend to search on MSDN.

Synchronization Frequency
The synchronization between the databases of a group is not made in real time. We can select the time interval when the synchronization needs to be made. This time interval can be between 5 minutes and 1 month. Also, we have a button that can trigger the synchronization action.

On-premise SQL Server
To be able to use this feature on SQL Server you will need to download and install SQL Data Sync. This is a tool that will integrate this functionality on SQL Server.

Logs
All the synchronization action that are between group nodes are logged. Using this information we can determine how long the synchronization action took, what nodes were synchronized and how the action ended.

I think that this feature has a lot of potential. Why? Because database synchronization can be made very easily now. This feature can really add value to your application with minimal costs and headaches.

Read More

Throttling and Availability over Windows Azure Service Bus

In today post we will talk about different redundancy mechanism when using Windows Azure Service Bus. Because we are using Windows Azure Service Bus as a service, we need to be prepared when something goes wrong.

This is a very stable service, but when you design a solution that needs to handle millions of messages every day you need to be prepared for word case scenarios. By default, Service Bus is not geo-replicated in different data centers, because of this if something is happening on the data center where your namespace is hosted, than you are in big troubles.
The most important thing that you need to cover is the case when the Service Bus node is down and clients cannot send messages anymore. We will see later on how we can handle this problem.
First of all, let’s see why a service like Service Bus can go down. Well, like other services, this has dependencies to databases, storages, other services and resources. There are cases when we can detect pretty easily the cause of the problem.
For example when we receive ‘ServerBusyException’, then we know that the service don’t have enough resources (CPU, memory, …) and we need to retry later. The default retry period is 10 seconds. It is recommended to not set a value under 10 seconds.
This problem can be resolved pretty easily with partition. When we are using partitioning, a topic or a queue is spitted on different messages brokers. This means that we have less chances to have our service down. Also, if something happen with one of our brokers, we will still be able to use the topic/queue without any kind of problems. Don’t forget that brokers will be on the same data center. Using this feature don’t increase your costs.
Enabling this feature can be done in different ways. One option is from portal, Visual Studio Server Explorer or from code.

NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString("...");
TopicDescription topicDesc = new TopicDescription("[topicName]")
{
  EnablePartitioning = true;
}

namespaceManager.CreateTopic(topicDesc);

It is so simple to use it. You should know that in this moment you can have maximum 100 topics/queues per namespace that has this feature activated, but I expect this value to change in the future. You should also know different behaviors that are happing when you are using sessions:

• Partition Key – Messages from the same transaction, that has the same partition key, but don’t has a session id will be send to the same broker.
• Session Id – All messages with a specific session if will be send to the same broker.
• Message Id – Messages that are send to a queue/topic with duplicated detection activated will be send to the same broker. Because of this I recommend to use messages duplication detection only where is necessary.
• None of above – Messages are send to all brokers in a round robin manner – one message to each broker.

Another downtown cause can be Service Bus service is upgraded. In this cases, the service will still work, but we can have 15-20 minutes latency until the message will appear in the queue/topic. The most important thing in this case is “We don’t lose any message”.
Also when the system is not stable (internal causes), brokers will be automatically restarted. The restart can take one or more minutes. In this case the service will throw MessagingException or TimeoutException. This problem are resolved build in by the clients SDK’s (if you are using .NET SDK). They have a retry policy build-in, that will retry to resend the message. If the retry policy is not able to send the message, an exception is throw that can be handled in different ways. Until now, all the issues related to this were handle by retry policy with success.
Custom configuration of retry policy can be made in the factory class of messaging.

MessagingFactory messagingFactory = MessagingFactory.Create();
messagingFactory.RetryPolicy = RetryExponential.Default;

The last main cause of failing is external causes like internet connectivity problem, electrical outage or human errors. This problem is handled with a very different approach. The client needs to detect this problem and handle it. Until now, this required a custom code to be written, that would redirect the messages to a topic/queue that is in another datacenter (namespace).
From now we can use paired namespace to handle this scenario. The paired namespace give us the possibility to specify a second namespace (that can be in a different data center) that will be used to send messages until the primary one will be up and running. When messages are send to the second namespace, messages will be persisted until the primary namespace will be up. In the moment when the primary namespace is running, all messages from the second one will need to be redirected to the first one. We can imagine secondary namespace as a buffer that is used to store messages until our main namespace is in good state.
 When we configure this feature, we can set also the failover interval. This is the time interval when our system will accept failovers before switching to the second namespace. The recommended (and default) value is 10 seconds. Also you will need to specify the number of queues that are used to store the messages in the secondary namespace (default value is 10). This value should be greater or equal to 10.
The last option that you should be aware is syphon (‘enableSyphon’ parameter). When you activate this on a client, you tell to the system that this is the system that will transfer the messages from the second namespace to the first one. Usually this value should be set on the consumers clients (backend), because usually clients only send messages to the topics/queues.

NamespaceManager primaryNM = NamespaceManager.CreateFromConnectionString("...");
MessagingFactory primaryMF = ...
NamespaceManager secondaryNM= NamespaceManager.CreateFromConnectionString("...");
MessagingFactory secondaryMF = ...
SendAvailabilityPairedNamespaceOptions sao=
    new SendAvailabilityPairedNamespaceOptions(secondaryNamespaceManager, secondaryMF);
primaryMF.PairNamespaceAsync(sao).Wait();

There are some small things that we should know related to this feature:

• The state and order is guaranteed only in the first queue. When using session, the order of the messages is not guaranteed when secondary namespace is used
• Messages are consumed only from the primary queue/subscription
• You will pay the extra cost of moving messages from the secondary namespace to the primary one
• The default name of the queues that are created on the secondary namespace is ‘x-servicebus-transfer/i’ (where ‘i’ can have a value from 0 to n)
• The queue from the secondary namespace is randomly chosen
• It is not recommended to change the configuration of the queues from the secondary namespace

We saw that we have different mechanism to handle this special scenarios. We don’t have a mechanism that handle all this use cases. Before starting to think about integrating all this feature ask yourself if you need all of them? There are cases when a 10-15 minutes downtime is acceptable.

Read More

How to monitor clients that access your blob storage?

Some time ago I wrote about the monitor and logging support available for Windows Azure Storage. In this post we will talk about how we can use this feature to detect what clients are accessing storage.
Let’s assume that we have a storage that is accessed by 100.000 users. At the end of the month we should be able to detect the users that downloaded a specific content with success.
What should we do in this case?
(Classic Solution) Well, in a classic solution, we would create an endpoint that will be called by the client after he download with success the specific content. In this case we would need to create a public endpoint, hosted it, persists the calls messages, manage and maintain the solution and so on.
In the end we would have additional costs.

What should we do in this case?

(Windows Azure Solution) Using Windows Azure Storage, we can change the rules of the game. Windows Azure Storage offer us out of the box support for logging mechanism. All the request that are made to our storage will be logged.

Based on this idea, we activate this feature and log all the access to the storage. In the following example you can see how the logs look like for a file called ‘m.txt’ under a container named ‘container’.

1.0;2013-11-12T13:03:23.4277012Z;GetBlob;AnonymousSuccess;200;7;7;anonymous;;radudemo;blob;"http://radudemo.blob.core.windows.net/container/m.txt";"/radudemo/container/m.txt";98ebc2d2-249f-47d2-8a3b-e7bbcb5e3c59;0;86.124.100.155:21131;2009-09-19;379;0;284;7;0;;;"0x8D0ADBEAF6D6A3C";Tuesday, 12-Nov-13 13:03:15 GMT;;"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36";;

As you can see, we have all the necessary information to identify the client, inclusive the client IP. But we want to do more than that. We want to be able to detect clients based on aware own identification mechanism.
To be able to do something like this, we have to options:
Query parameters
We can add a custom query parameter that represent the client unique ID. For example we can make the following requests
http://radudemo.blob.core.windows.net/container/m.txt?clientID=123
In this case, the client will receive the file and in the logs we will have:

1.0;2013-11-12T13:03:36.3117012Z;GetBlob;AnonymousSuccess;200;14;14;anonymous;;radudemo;blob;"http://radudemo.blob.core.windows.net/container/m.txt?clientID=123";"/radudemo/container/m.txt";c90e928f-f3e7-4c4f-859b-4bb0051f0271;0;86.124.100.155:21131;2009-09-19;392;0;284;7;0;;;"0x8D0ADBEAF6D6A3C";Tuesday, 12-Nov-13 13:03:15 GMT;;"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36";;

In the client URL we can find the clientID that can be filter, processed and so on.
User Agent
Another option is to set a custom user agent. Using a custom user agent it will be very easily for us to identify witch client made the requests.

1.0;2013-11-12T13:03:36.3117014Z;GetBlob;AnonymousSuccess;200;14;14;anonymous;;radudemo;blob;"http://radudemo.blob.core.windows.net/container/m.txt";"/radudemo/container/m.txt";c90e928f-f3e7-4c4f-859b-4bb0051f0271;0;86.124.100.155:21131;2009-09-19;392;0;284;7;0;;;"0x8D0ADBEAF6D6A3C";Tuesday, 12-Nov-13 13:03:15 GMT;;"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36 ClientID/123";;

Both solutions are similar, we are only sending the unique client id in different locations. This files can be very easily processed and analyzed using different solutions. A good solution when you have a lot of logs file is by Hadoop.
The advantages of using such a solution is the out of the box support. You don’t need to create and manage another system for this. Also you eliminate the case when the client is able to download/access the content but is not able to confirm the download.
The only additional costs that are added to a classic solution is the storage cost for storing the logs data itself, plus the transactions costs. This costs are very low in comparison with the case when you would create all the infrastructure for logging. In both cases you would need to pay the storage costs of logs.

Read More

Digging through SignalR - Dependency Resolver

I’m continuing the series of post related to SignalR with dependency injection.
When having a complicated business you can start to group different functionalities in classes. Because of this you can end up very easily with classes that accept in constructor 4-5 or even 10 parameters.

public abstract class PersistentConnection
{
    public PersistentConnection(
        IMessageBus messageBus, IJsonSerializer jsonSerializer, 
        ITraceManager traceManager, IPerformanceCounterManager performanceCounterManager,
        IAckHandler ackHandler, IProtectedData protectedData, 
        IConfigurationManager configurationManager, ITransportManager transportManager,
        IServerCommandHandler serverCommandHandler, HostContext hostContext)
    {
            
    }
    ...
}

Of course you have a decoupled solution that can be tested very easily, but in the same time you have a fat constructor.
People would say: “Well, we have dependency injector, the resolver will handle the constructor and resolve all the dependencies”. This is true, the resolver will inject all the dependencies automatically.
In general, because you don’t want to have a direct dependency to a specific dependency injector stack, people tend to create a wrapper over dependency resolver. The same thing was done in SignalR also.

public interface IDependencyResolver : IDisposable
{
    object GetService(Type serviceType);
    IEnumerable<object> GetServices(Type serviceType);
    void Register(Type serviceType, Func<object> activator);
    void Register(Type serviceType, IEnumerable<Func<object>> activators);
}

Additionally, they done something more. In ctor, they don’t send all the dependencies that are already register in the IoC container. They send directly the dependency resolver, which will be used by the class itself to resolve all the external dependencies.

public abstract class PersistentConnection
{
    public PersistentConnection(IDependencyResolver resolver, HostContext hostContext)
    {
        Initialize(resolver, hostContext);
    }

    public virtual void Initialize(IDependencyResolver resolver, HostContext context)
    {
        ...
        MessageBus = resolver.Resolve<IMessageBus>();
        JsonSerializer = resolver.Resolve<IJsonSerializer>();
        TraceManager = resolver.Resolve<ITraceManager>();
        Counters = resolver.Resolve<IPerformanceCounterManager>();
        AckHandler = resolver.Resolve<IAckHandler>();
        ProtectedData = resolver.Resolve<IProtectedData>();

        _configurationManager = resolver.Resolve<IConfigurationManager>();
        _transportManager = resolver.Resolve<ITransportManager>();
        _serverMessageHandler = resolver.Resolve<IServerCommandHandler>();
        ...
    }

    ...
}

It is important to notify that you don’t need to inject everything through the resolver. You can have specific dependencies injected directly by constructor. For example, HostContext is something specific for each connection. Because of this is more natural to send this context using the constructor. Is something variable that is changing from one connection to another.
Why is the best approach to this problem?
It cannot say that one is better than another. Using this solution, the constructor itself will be lighter, but in the same time you add dependency to the resolver. In a perfect world you shouldn’t have constructors with 7-10 parameters… but when you have cases like this, this solution could be pretty interesting.

Read More

[Event] Global Day of Coderetreat in Cluj-Napoca! - December 14th, 2013 - Codecamp

We invite you to the Global Day of Coderetreat in Cluj-Napoca!
Global Day of Coderetreat is a world-wide event celebrating passion and software craftsmanship. Last year, over 70 passionate software developers in Cluj-Napoca joined the 2000 developers in 150 cities around the world in spending the day practicing the craft of software development using the coderetreat format. This year, we want to repeat the experience and extend it even more.
We will focus our practice on XP techniques like pair programming, unit testing,  or TDD and we'll give special attention to OOD and good code in general.
Find more about this event! Reserve your seat, here!

Co-organized by: RABS, Code Camp, Cluj.rb, Agile Works and Functional Programmers Cluj-Napoca

Read More

Windows Azure Service Bus - What ports are used

Windows Azure Service Bus is a great mechanism to distribute messages across components of your own system or to different clients. When you want to use this service for enterprise projects you will meet the IT guys.
For them it is very important to control the ports that are used by applications. Because of one of the first’s question that they will ask you is:
What are the ports that are used by Service Bus?
When this question is asked you should be prepare to have the answer prepared. Looking over the documentation from MSDN, the following ports are used by Windows Azure Service Bus:

• 80 – HTTP connection mode
• 443 – HTTPS connection mode
• 5671 – Advanced Message Queuing Protocol (AMQP)
• 5672 – AMQP
• 9350 – WCF with connection mode Auto or TCP using .NET SDK
• 9351 – WCF with connection mode Auto or TCP using .NET SDK
• 9352 – WCF with connection mode Auto or TCP using .NET SDK
• 9353 – WCF with connection mode Auto or TCP using .NET SDK
• 9354 – WCF with connection mode Auto or TCP using .NET SDK

I recommend to use HTTP/HTTPS connection where is possible. IF you are using .NET SDK you don’t need to make any custom configuration. When you let connection mode set to AutoDetect, the SDK will check if a connection can be made using non-HTTP ports. If the endpoint cannot be reached, that he will try to go over HTTP.
If you want to control the connection method, than you will need to set the ‘Mode’ property of the SystemConnectivity. The supported mode are:

• AutoDetect
• Http
• Tcp

ServiceBusEnvironment.SystemConnectivity.Mode = ConnectivityMode.Http;

Enjoy!

Read More

Debugging in production

Abstract
In this article we have discovered how we can create a dump and which the basic tools to analyze it are. By means of dump files we can access the information we could not normally access. Some data can only be accessed through these dumps and not by other means (Visual Studio debug).
We could state that these tools are very powerful, but they are rather difficult to use, as they require quite a high degree of knowledge.

How many times has it happened to you to have a problem in production or in the testing environment which you are unable to reproduce on the development machine? When this thing happens, things can go off the track, and we try out different ways of remote debugging. Without even knowing, helpful tools can be right near at hand, but we ignore them or simply don’t know how to use them.
In this article I will present different ways in which we can debug without having to use Visual Studio.
Why not using Visual Studio?
Though Visual Studio is an extremely good product, which helps us when we need to discover bugs and to debug, it won’t be of great help to us in production. The moment when we have a bug in production, the rules of the game change. In production, the application is compiled for release, and debugging is no longer possible.
When do we need these tools?
The moment we cannot reproduce the problem on our development machines. No matter what we do, we are not able to reproduce the problem we are dealing with. Since we cannot reproduce the problem, it’s like looking for a needle in haystack.
If by chance the problem appears, but we do not have a reproduction scenario, we find ourselves back in the situation above mentioned.
Another case is when the memory occupied by our application increases in time, the phenomenon emerging only on the production machines. We can only guess what the problem is, but we do not know the exact cause. That is why we may “fix” totally different code areas.
What solutions do we have?
Generally, there are two possibilities at hand. The first one is entirely based on logs. Through logs we are able to identify the application areas which do not function appropriately. But using logs can be two-edged. It is required that you know what exactly must appear in logs and how often. Otherwise, you may end up with thousands of pages of useless and almost impossible to analyze logs. If we end up having too many logs, we may be taken by surprise by the alteration of the behavior of the application.
When it is possible, we can send the PDBs on the production machine. This way we will have access to the entire stack trace generated by an exception.
Logs can be of great help for us to solve different problems that emerge in production. But even if logs are very useful, they won’t help us every time. There are different problems which can appear and which are extremely difficult to identify by using logs. For example, a dead-lock would be almost impossible to identify by means of logs.
Another alternative that is available for us is creating memory dumps and analyzing them.
What is a memory dump?
A memory dump is a snapshot of the process on a certain moment. Besides the information regarding the allocation of memory, a snapshot also contains information on the state of different threads, objects and cod. By using this information we can obtain very valuable information regarding the process that is running. This snapshot represents the image of the memory in 32 or 64 bits format, depending on the system.
Generally, there are two types of memory dump. The first one is minidump. This is the most uncomplicated memory dump that can be done, which consists of mere information on the stack – the state of the process or on the calls that are made and so on.

The second type of memory dump is full dump. It contains all of the information that can be obtained, including a snapshot on memory. It takes much longer to obtain a full dump compared to a minidump and the dump file itself is much bigger.
How can we generate a memory dump?
There are different applications which allow us to do this. Some of them allow us to automatically generate a dump, according to different parameters.
In the case we need to generate a memory dump, the easiest solution is the Task Manager. All we have to do is to click right on a process and select “Create dump file”. We can do the same thing also by using Visual Studio or “adplus.exe”. The last alternative is a debug tool for Windows which can be found on almost all machines on which Windows runs.
In the following example, we place an order in adplus to create a memory dump at this moment:

adplus –hang –o C:myDump –pn MyApp.exe

By means of pn option we specify the name of the process for which we wish to create a dump. If we want to create a dump automatically we can use the –crash option.

adplus –crash –o C:myDump –pn MyApp.exe
adplus –crash –o C:myDump –sc MyApp.exe

If it is necessary for us to automatically create a dump, besides “adplus.exe” we can use DebugDiag and “clrdmp.dll”. The three options we have in order to automatically create a dump are rather similar. DebugDump allows us to set up the system so that it automatically generates a memory dump the moment when the CPU level is higher than X% within a certain time span.
Besides these tools there are many others on the market. Depending on your requirements, you can use any tool of this type.
How do we analyze a dump?
The native debugger for a dump is represented by Windbg. This is a powerful tool, by means of which one can get very valuable information. The only problem of this tool is that it is not very friendly. We will see a little later what the alternatives to Windbg are. We must remember that in almost all the cases, the alternatives to Windbg are using this debugger behind – it’s just that they display a friendlier and more useful interface.
An alternative to Windbg is any Visual Studio that is more recent than Visual Studio 2010. Beginning with Visual Studio 2010, they offer us the possibility to analyze the dumps for .NET 4.0+. What we can do in Visual Studio is not as advanced as what Windbg allows us to do, but generally it can suffice.
Windbg

The first step we need to take after opening Windbg is to upload a dump (CTRL+D). Once uploaded, a dump can be visualized in different manners. For example, we can analyze the threads, the memory, the allocated resources and so on.
In order to be able to do more, for instance to visualize and analyze the managed code, we need to upload additional libraries such as Son of Strike (SOS) or Son of Strike Extension (SOSEX). These two libraries open new doors for us, as they are able to analyze the data from the dump in an extremely useful way.
Son of Strike (SOS)
SOS allows us to visualize the process in itself. It allows us to access the objects, threads and information from the garbage collector. We can even visualize names of variables and their value.
One must know that all the information that can be accessed is part of the managed memory. Therefore, SOS is highly connected to CLR and its version. When we upload the SOS module, we must make sure we are uploading the one that is correspondent to the .NET version of our application.

.loadby sos mscorks
.loadby sos clr

It the examples above, we have uploaded the SOS module for .NET 3.5-, and in the second example, we have uploaded SOS for .NET 4.0+.
All the SOS orders start with “!”. The basic order is “!help”. If we wish to visualize the threads list, we can employ the “!threads” order which has an output that is similar to the following:

0:000> !threads
ThreadCount: 5
UnstartedThread: 0
BackgroundThread: 2
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
Lock
     ID    OSID    ThreadOBJ Count Apt Exception
…

Debug a crash
So far we have seen there are many tools available for us to create and analyze a dump. Time has come now to see what we have to do in order to be able to analyze a crash.

• 1. Launch the process
• 2. Before it “crashes”, we order adplus to create a dump the moment when the process “crashes”
• adplus –crash –pn [numeProcesor]
• 3. Launch Windbg (after the crash)
• 3.1 Upload the dump
• 3.2 Upload SOS
• 3.3 !threads (to see which thread has crashed)
• 3.4 !PrintException (on the thread that has crashed in order to see the exception)
• 3.5 !clrstack (to see the stack of calls)
• 3.6 !clrstack –a (to see the stack together with the parameters)
• 3.7 !DumpHeap –type Exception (it lists all the exceptions that are not related to GC).

One must know that the results are according to the way in which the application is compiled. For instance, if there has been a code optimization performed during the compilation. Moreover, the exception list we can get may be quite long due to some orders such as !DumpHeap, which returns all the exceptions encountered – even the ones which have been pre-created, such as ThreadAbord.
How do we identify a deadlock?
A deadlock emerges when two or more threads are waiting for the same resource. In these cases, a part of the application, if not the entire application, gets blocked.
In this case, the first step is to create a dump using the order:

Addplus –hang –o –c:myDump –pn [NumeProces]

Then, it is necessary for us to analyze the stack trace for each thread and see whether it is blocked (Monitor.Enter, ReadWriteLock.Enter…). Once we have identified these threads, we can find the resources used by each thread, together with the thread that keeps these resources blocked.
For these final steps, the order “!syncblk” comes to our help. It lists for us the units of memory for a certain thread.

This article was written by Radu Vunvulea for Today Software Magazine.

Read More

Simple load balancer for SQL Server Database

Two weeks ago I started to work on a PoC where the bottle neck is the database itself. We had a lot of complicated and expensive queries over the database. Because of this, if we wanted to have good performance we had to find a solution to have more than one instance of SQL Server.
Because the solution was on Windows Azure, we wanted to test different configurations and solution. We wanted to see what the performance is if we use 1, 2 and 3 instances of SQL endpoints. Not only this, we had different times of SQL endpoints –Azure SQL (SAS), Virtual Machine with SQL Server and SQL Azure Azure SQL Premium. (SAS but with dedicated resources).
 The good part in our scenario was that the data don’t change very often. We could start from the assumption that the database will be updated only one time per day. Because of this we could use different methods to create a load balancing from our SQL instances.
Because the time was very limited and the only think that we needed was to distributed the load on all the machines we decided to replicate the data on all the SQL instances and create a simple load balancer from code. We divided the tick time (timpstamp) with the number of instances and based on this we selected the SQL instance that will be hit. A simple mechanism that can be very easily managed for the PoC. Of course the final solution will be more complicated, but for a PoC this solution was perfect. In 10 minutes we had a load balancer  :-).
We checked the load of each machine and we can say that the queries were pretty good distributed. The load of each machine was almost the same (+/-10%)

What others solution we could use?
Master-Slave – In this case we have multiple instances of SQL servers. Each slave can be used for reading operation and the master will be used for writing operations.
Sharding – The solution that we used to resolve our problem
AlwaysOn – The base concept is the same one as for Master-Slave
Transaction Replication – Is based on a snapshot and when data is changing, all the subscribers are notified about the change. You can filter individual database objects and publish changes to subscribers
Peer-to-Peer Replication – Is based on Transaction Replication, but with some new features
Change Tracking – Is a very base and simple tracking mechanisms.  The only thing that is notifing you is that a specific row had change. You don’t know any information related to the old or new value
Change Data Capture – Writes in the logs all the changes that are made over a table and are used to track and synchronize the rest of the database instances

Conclusion
We have a lot of solutions on the marker and for SQL Server. Based on our needs we needed to select the best one for us. I was surprised to see that a simple load balancer, written in 5 lines of code can do a pretty good stuff.

Read More

[PostEvent] MSSummit 2013, Bucharest

WOW Why? Because last two days (6-7 November 2013) I had the opportunity to participate at MSSummit 2013, that was held in Bucharest. It was one of the biggest IT event organized in Romania this year.
There were more than 1000 people, 5 simultaneous tracks and more than 45 speakers. It’s been a while since I’ve seen such an event in Romania. I was impressed about the event itself, location and the number of people.
The session list was extremely interesting, we had session held by David Chappell, Chris Capossela, Michael Palermo and so on. All the sessions were great, we had the opportunity to learn and discover new stuff. After this event we could say that Microsoft is full of surprising things (in the good way).
At this event I was invited as a speaker and I held two sessions where I talked about how we can write and load tests using Visual Studio 2013 and Windows Azure. In the second session I talked about fluent communication between client – server – client using SignalR.
Special thanks for the attendees that stayed until 18:30 in the last day of the event.
I hope that next year Microsoft Romania will organize another MSSummit. This is a great conference that can grow and become very big and powerful.
In this following picture I’m with my colleagues from iQuest Group.

Read More

How to read response time when you run a performance test

Measuring the performance of an application is mandatory before releasing it. Measuring the performance of a PoC is also mandatory to validate the base concepts and ideas.
Performance of a system can be measured in different ways, from the process time, to numbers of users, processor level, memory level and so on. Before starting a performance test you should know exactly what you want to measure.
When you want to measure the response time of a specific service/endpoint you should be aware how to interpret the results. Statistical information can be saw in different ways. Each of this view can give you a different perspective of the results.

Average
The average of the response time is calculated. This information is important if you want to know what is the average request time. Even if this information can be very useful, this information is misleading. For example from 1000 requests, you can have an average response time of 16 seconds even if you your chances to have a  requests that takes under 10 seconds is around 90% (you can have 10 requests that takes 300 seconds and 900 requests that takes only 10 seconds).

Distribution
This view will give you the possibility to see what is the distribution of requests based on the response time. For example you will be able to know that from 1000 requests, 300 requests took 1 seconds, 500 took 2 second, 100 took 9 second and so on.
When you are measuring the scalability of a system, distribution of response is more important than average response time. This information will help you to understand how the requests time change based on different configuration. You can have cases when the average time to be the same, but the distribution of the response time to be very different.
Here we could talk also about mean and standard deviation time.

Min/Max
When the response time needs to be in a specific time interval (usually a max of X) the min/max of request will offer you this data.

This days I had to measure the performance of a database and how scalable it is. When we started to measure the average execution time of each query with 1, 2 and 3 database nodes we observed that the average response time doesn’t improve so much. In contrast the distribution of query response time is changing a lot. From 30% of requests that take less than 3 second with one node, we ended up with more than 50% of requests that take under 2 seconds when having 3 database nodes.

Read More

VM and load balancer, direct server return, availability set and virtual network on Windows Azure

In today post we will talk about how we can configure a load balancer, direct server return, availability set and virtual network when working with virtual machine on Windows Azure.
For virtual machines we cannot talk about load balancer without talking about endpoints. Endpoints give you the possibility to specify the port (endpoint) that can be public accessed. For each endpoint you need to specify the public port, private port and protocol (TCP,UDP). The private and public port can be very useful when you want to redirect specific calls to different ports. If you don’t specify the endpoints for a virtual machine, the machine cannot be accessed from outside (for example when you host a web page and you need to have port 80 open).  
For web and worker roles this load balancer is out of the box support. We could say that it is the same think for virtual machines also, but you need to make some custom configuration. You will need to specify that you will use the load balancer functionality when you create an endpoint. Once you create an endpoint that is load balanced, the only think that you will need is to specify to the second, 3rd … machine from that will be included in the load balancer the endpoint that use a load balancer set (“Add an endpoint to an existing load-balancer set”).
Direct server return is another feature of endpoints. It can be very useful when you have want each machine to response to the client directly, without the load balancer routing. In this way, once a client start to communicate with a machine, the connection will stick with only that virtual machine (stick connections) – useful when you have a DB on machines or you have an in-memory session that is not shared between machines.
Virtual network give you the possibility to create a private network formed from virtual machines from Windows Azure. When you need to share content between machines, this will be the best approach. Even if you have machines that are under the same subscription, you will not be able to access them using their private IP or name. You have almost the same access level from one machine to another as you would have machines under different subscription.
Because of this when you need to communicate between machines that are behind the same DNS name of public IP this will be the best approach. For example you will not be able to access a specific instance that is behind a load balancer to update content or call specific endpoints.
A virtual network can be created very easily from the Network tab. You have a lot of options from custom DNS names for specific machines to custom address spaces. Once a virtual network is created you can very easily create new virtual machines to it.
Remarks: You should take into account that you cannot add an existing virtual machine to a new virtual network. A fix for this problem is to stop and delete the machine without deleting the disk itself and recreate a new machine based on that disk
Not last, let’s talk a little about availability set. This is a pretty interesting setting. When you create two or more machines for the same purpose you don’t want the resources of the machine (memory, processor, and store) to be in the same rack. Why? If the rack will stop working that you will have both machines down and the load balancer will not be able to help you. In this case when you create a availability set you will have the guarantee that the machines under the same availability set will not be under the same rack. This will give you the availability of 99.95% that is offered by SLA.
The feature that I enjoy the most is the load balancer in combination with endpoints. It is a great feature that can help you a lot.

Read More

Bugs that cover each other

This week I had the opportunity to work on a PoC that was pretty challenging. In a very short period of time we had to test our ideas and come with some results. When you need to do something like this, you have to:

• Design
• Implement
• Test
• Measure (Performance Test)

First two steps were pretty straight, we didn’t had any kind of problems. The testing of the current solution was pretty good, we find some small issues – small problems that were resolved easily.
We started the performance test, when we here hit by a strange behaviors. The database server had 3-4 minutes with a 100% load, after this period the load would go down to 0% for 5-6 minutes. This was like a cycle that used to repeat to infinity.
The load of the database should be 100% all the time… We looked over the backend server, everything looked okay. They were received requests from client bots and processed. Based on the load of the backend everything should be fine.
Next step was to look over the client bots machines. Based on the tracking information everything should be fine… But still we had a strange behaviors in the database, something was not right.
We started to take each part of the solution and debug it. We started with …SQL store procedures …backend and …client bots. When we looked over client bots we observed that we have a strange behaviors there. For each request we received at least to different responses.
After 1 hours of debugging we found out that we had 2 different bugs. The interesting part of this was that one of the bugs created a behaviors that masked the other bug. Because of this on the backend we had the impression that we have the expected behavior and the clients’ works well.
The second bug that was masked by the first one has big and pretty ugly.
In conclusion I would say that when even when you write a PoC and you don’t have enough time, try to test with one, two and 3 clients in parallel. We tested with one, 10 and 100 clients. Because of the logs flow for 10 and 100 clients we were not able to observe the strange behaviors before starting the performance testing.

Read More