How to monitor clients that access your blob storage?

Some time ago I wrote about the monitor and logging support available for Windows Azure Storage. In this post we will talk about how we can use this feature to detect what clients are accessing storage.
Let’s assume that we have a storage that is accessed by 100.000 users. At the end of the month we should be able to detect the users that downloaded a specific content with success.
What should we do in this case?
(Classic Solution) Well, in a classic solution, we would create an endpoint that will be called by the client after he download with success the specific content. In this case we would need to create a public endpoint, hosted it, persists the calls messages, manage and maintain the solution and so on.
In the end we would have additional costs.

What should we do in this case?

(Windows Azure Solution) Using Windows Azure Storage, we can change the rules of the game. Windows Azure Storage offer us out of the box support for logging mechanism. All the request that are made to our storage will be logged.

Based on this idea, we activate this feature and log all the access to the storage. In the following example you can see how the logs look like for a file called ‘m.txt’ under a container named ‘container’.

1.0;2013-11-12T13:03:23.4277012Z;GetBlob;AnonymousSuccess;200;7;7;anonymous;;radudemo;blob;"http://radudemo.blob.core.windows.net/container/m.txt";"/radudemo/container/m.txt";98ebc2d2-249f-47d2-8a3b-e7bbcb5e3c59;0;86.124.100.155:21131;2009-09-19;379;0;284;7;0;;;"0x8D0ADBEAF6D6A3C";Tuesday, 12-Nov-13 13:03:15 GMT;;"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36";;

As you can see, we have all the necessary information to identify the client, inclusive the client IP. But we want to do more than that. We want to be able to detect clients based on aware own identification mechanism.
To be able to do something like this, we have to options:
Query parameters
We can add a custom query parameter that represent the client unique ID. For example we can make the following requests
http://radudemo.blob.core.windows.net/container/m.txt?clientID=123
In this case, the client will receive the file and in the logs we will have:

1.0;2013-11-12T13:03:36.3117012Z;GetBlob;AnonymousSuccess;200;14;14;anonymous;;radudemo;blob;"http://radudemo.blob.core.windows.net/container/m.txt?clientID=123";"/radudemo/container/m.txt";c90e928f-f3e7-4c4f-859b-4bb0051f0271;0;86.124.100.155:21131;2009-09-19;392;0;284;7;0;;;"0x8D0ADBEAF6D6A3C";Tuesday, 12-Nov-13 13:03:15 GMT;;"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36";;

In the client URL we can find the clientID that can be filter, processed and so on.
User Agent
Another option is to set a custom user agent. Using a custom user agent it will be very easily for us to identify witch client made the requests.

1.0;2013-11-12T13:03:36.3117014Z;GetBlob;AnonymousSuccess;200;14;14;anonymous;;radudemo;blob;"http://radudemo.blob.core.windows.net/container/m.txt";"/radudemo/container/m.txt";c90e928f-f3e7-4c4f-859b-4bb0051f0271;0;86.124.100.155:21131;2009-09-19;392;0;284;7;0;;;"0x8D0ADBEAF6D6A3C";Tuesday, 12-Nov-13 13:03:15 GMT;;"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36 ClientID/123";;

Both solutions are similar, we are only sending the unique client id in different locations. This files can be very easily processed and analyzed using different solutions. A good solution when you have a lot of logs file is by Hadoop.
The advantages of using such a solution is the out of the box support. You don’t need to create and manage another system for this. Also you eliminate the case when the client is able to download/access the content but is not able to confirm the download.
The only additional costs that are added to a classic solution is the storage cost for storing the logs data itself, plus the transactions costs. This costs are very low in comparison with the case when you would create all the infrastructure for logging. In both cases you would need to pay the storage costs of logs.

Read More

Digging through SignalR - Dependency Resolver

I’m continuing the series of post related to SignalR with dependency injection.
When having a complicated business you can start to group different functionalities in classes. Because of this you can end up very easily with classes that accept in constructor 4-5 or even 10 parameters.

public abstract class PersistentConnection
{
    public PersistentConnection(
        IMessageBus messageBus, IJsonSerializer jsonSerializer, 
        ITraceManager traceManager, IPerformanceCounterManager performanceCounterManager,
        IAckHandler ackHandler, IProtectedData protectedData, 
        IConfigurationManager configurationManager, ITransportManager transportManager,
        IServerCommandHandler serverCommandHandler, HostContext hostContext)
    {
            
    }
    ...
}

Of course you have a decoupled solution that can be tested very easily, but in the same time you have a fat constructor.
People would say: “Well, we have dependency injector, the resolver will handle the constructor and resolve all the dependencies”. This is true, the resolver will inject all the dependencies automatically.
In general, because you don’t want to have a direct dependency to a specific dependency injector stack, people tend to create a wrapper over dependency resolver. The same thing was done in SignalR also.

public interface IDependencyResolver : IDisposable
{
    object GetService(Type serviceType);
    IEnumerable<object> GetServices(Type serviceType);
    void Register(Type serviceType, Func<object> activator);
    void Register(Type serviceType, IEnumerable<Func<object>> activators);
}

Additionally, they done something more. In ctor, they don’t send all the dependencies that are already register in the IoC container. They send directly the dependency resolver, which will be used by the class itself to resolve all the external dependencies.

public abstract class PersistentConnection
{
    public PersistentConnection(IDependencyResolver resolver, HostContext hostContext)
    {
        Initialize(resolver, hostContext);
    }

    public virtual void Initialize(IDependencyResolver resolver, HostContext context)
    {
        ...
        MessageBus = resolver.Resolve<IMessageBus>();
        JsonSerializer = resolver.Resolve<IJsonSerializer>();
        TraceManager = resolver.Resolve<ITraceManager>();
        Counters = resolver.Resolve<IPerformanceCounterManager>();
        AckHandler = resolver.Resolve<IAckHandler>();
        ProtectedData = resolver.Resolve<IProtectedData>();

        _configurationManager = resolver.Resolve<IConfigurationManager>();
        _transportManager = resolver.Resolve<ITransportManager>();
        _serverMessageHandler = resolver.Resolve<IServerCommandHandler>();
        ...
    }

    ...
}

It is important to notify that you don’t need to inject everything through the resolver. You can have specific dependencies injected directly by constructor. For example, HostContext is something specific for each connection. Because of this is more natural to send this context using the constructor. Is something variable that is changing from one connection to another.
Why is the best approach to this problem?
It cannot say that one is better than another. Using this solution, the constructor itself will be lighter, but in the same time you add dependency to the resolver. In a perfect world you shouldn’t have constructors with 7-10 parameters… but when you have cases like this, this solution could be pretty interesting.

Read More

[Event] Global Day of Coderetreat in Cluj-Napoca! - December 14th, 2013 - Codecamp

We invite you to the Global Day of Coderetreat in Cluj-Napoca!
Global Day of Coderetreat is a world-wide event celebrating passion and software craftsmanship. Last year, over 70 passionate software developers in Cluj-Napoca joined the 2000 developers in 150 cities around the world in spending the day practicing the craft of software development using the coderetreat format. This year, we want to repeat the experience and extend it even more.
We will focus our practice on XP techniques like pair programming, unit testing,  or TDD and we'll give special attention to OOD and good code in general.
Find more about this event! Reserve your seat, here!

Co-organized by: RABS, Code Camp, Cluj.rb, Agile Works and Functional Programmers Cluj-Napoca

Read More

Windows Azure Service Bus - What ports are used

Windows Azure Service Bus is a great mechanism to distribute messages across components of your own system or to different clients. When you want to use this service for enterprise projects you will meet the IT guys.
For them it is very important to control the ports that are used by applications. Because of one of the first’s question that they will ask you is:
What are the ports that are used by Service Bus?
When this question is asked you should be prepare to have the answer prepared. Looking over the documentation from MSDN, the following ports are used by Windows Azure Service Bus:

• 80 – HTTP connection mode
• 443 – HTTPS connection mode
• 5671 – Advanced Message Queuing Protocol (AMQP)
• 5672 – AMQP
• 9350 – WCF with connection mode Auto or TCP using .NET SDK
• 9351 – WCF with connection mode Auto or TCP using .NET SDK
• 9352 – WCF with connection mode Auto or TCP using .NET SDK
• 9353 – WCF with connection mode Auto or TCP using .NET SDK
• 9354 – WCF with connection mode Auto or TCP using .NET SDK

I recommend to use HTTP/HTTPS connection where is possible. IF you are using .NET SDK you don’t need to make any custom configuration. When you let connection mode set to AutoDetect, the SDK will check if a connection can be made using non-HTTP ports. If the endpoint cannot be reached, that he will try to go over HTTP.
If you want to control the connection method, than you will need to set the ‘Mode’ property of the SystemConnectivity. The supported mode are:

• AutoDetect
• Http
• Tcp

ServiceBusEnvironment.SystemConnectivity.Mode = ConnectivityMode.Http;

Enjoy!

Read More

Debugging in production

Abstract
In this article we have discovered how we can create a dump and which the basic tools to analyze it are. By means of dump files we can access the information we could not normally access. Some data can only be accessed through these dumps and not by other means (Visual Studio debug).
We could state that these tools are very powerful, but they are rather difficult to use, as they require quite a high degree of knowledge.

How many times has it happened to you to have a problem in production or in the testing environment which you are unable to reproduce on the development machine? When this thing happens, things can go off the track, and we try out different ways of remote debugging. Without even knowing, helpful tools can be right near at hand, but we ignore them or simply don’t know how to use them.
In this article I will present different ways in which we can debug without having to use Visual Studio.
Why not using Visual Studio?
Though Visual Studio is an extremely good product, which helps us when we need to discover bugs and to debug, it won’t be of great help to us in production. The moment when we have a bug in production, the rules of the game change. In production, the application is compiled for release, and debugging is no longer possible.
When do we need these tools?
The moment we cannot reproduce the problem on our development machines. No matter what we do, we are not able to reproduce the problem we are dealing with. Since we cannot reproduce the problem, it’s like looking for a needle in haystack.
If by chance the problem appears, but we do not have a reproduction scenario, we find ourselves back in the situation above mentioned.
Another case is when the memory occupied by our application increases in time, the phenomenon emerging only on the production machines. We can only guess what the problem is, but we do not know the exact cause. That is why we may “fix” totally different code areas.
What solutions do we have?
Generally, there are two possibilities at hand. The first one is entirely based on logs. Through logs we are able to identify the application areas which do not function appropriately. But using logs can be two-edged. It is required that you know what exactly must appear in logs and how often. Otherwise, you may end up with thousands of pages of useless and almost impossible to analyze logs. If we end up having too many logs, we may be taken by surprise by the alteration of the behavior of the application.
When it is possible, we can send the PDBs on the production machine. This way we will have access to the entire stack trace generated by an exception.
Logs can be of great help for us to solve different problems that emerge in production. But even if logs are very useful, they won’t help us every time. There are different problems which can appear and which are extremely difficult to identify by using logs. For example, a dead-lock would be almost impossible to identify by means of logs.
Another alternative that is available for us is creating memory dumps and analyzing them.
What is a memory dump?
A memory dump is a snapshot of the process on a certain moment. Besides the information regarding the allocation of memory, a snapshot also contains information on the state of different threads, objects and cod. By using this information we can obtain very valuable information regarding the process that is running. This snapshot represents the image of the memory in 32 or 64 bits format, depending on the system.
Generally, there are two types of memory dump. The first one is minidump. This is the most uncomplicated memory dump that can be done, which consists of mere information on the stack – the state of the process or on the calls that are made and so on.

The second type of memory dump is full dump. It contains all of the information that can be obtained, including a snapshot on memory. It takes much longer to obtain a full dump compared to a minidump and the dump file itself is much bigger.
How can we generate a memory dump?
There are different applications which allow us to do this. Some of them allow us to automatically generate a dump, according to different parameters.
In the case we need to generate a memory dump, the easiest solution is the Task Manager. All we have to do is to click right on a process and select “Create dump file”. We can do the same thing also by using Visual Studio or “adplus.exe”. The last alternative is a debug tool for Windows which can be found on almost all machines on which Windows runs.
In the following example, we place an order in adplus to create a memory dump at this moment:

adplus –hang –o C:myDump –pn MyApp.exe

By means of pn option we specify the name of the process for which we wish to create a dump. If we want to create a dump automatically we can use the –crash option.

adplus –crash –o C:myDump –pn MyApp.exe
adplus –crash –o C:myDump –sc MyApp.exe

If it is necessary for us to automatically create a dump, besides “adplus.exe” we can use DebugDiag and “clrdmp.dll”. The three options we have in order to automatically create a dump are rather similar. DebugDump allows us to set up the system so that it automatically generates a memory dump the moment when the CPU level is higher than X% within a certain time span.
Besides these tools there are many others on the market. Depending on your requirements, you can use any tool of this type.
How do we analyze a dump?
The native debugger for a dump is represented by Windbg. This is a powerful tool, by means of which one can get very valuable information. The only problem of this tool is that it is not very friendly. We will see a little later what the alternatives to Windbg are. We must remember that in almost all the cases, the alternatives to Windbg are using this debugger behind – it’s just that they display a friendlier and more useful interface.
An alternative to Windbg is any Visual Studio that is more recent than Visual Studio 2010. Beginning with Visual Studio 2010, they offer us the possibility to analyze the dumps for .NET 4.0+. What we can do in Visual Studio is not as advanced as what Windbg allows us to do, but generally it can suffice.
Windbg

The first step we need to take after opening Windbg is to upload a dump (CTRL+D). Once uploaded, a dump can be visualized in different manners. For example, we can analyze the threads, the memory, the allocated resources and so on.
In order to be able to do more, for instance to visualize and analyze the managed code, we need to upload additional libraries such as Son of Strike (SOS) or Son of Strike Extension (SOSEX). These two libraries open new doors for us, as they are able to analyze the data from the dump in an extremely useful way.
Son of Strike (SOS)
SOS allows us to visualize the process in itself. It allows us to access the objects, threads and information from the garbage collector. We can even visualize names of variables and their value.
One must know that all the information that can be accessed is part of the managed memory. Therefore, SOS is highly connected to CLR and its version. When we upload the SOS module, we must make sure we are uploading the one that is correspondent to the .NET version of our application.

.loadby sos mscorks
.loadby sos clr

It the examples above, we have uploaded the SOS module for .NET 3.5-, and in the second example, we have uploaded SOS for .NET 4.0+.
All the SOS orders start with “!”. The basic order is “!help”. If we wish to visualize the threads list, we can employ the “!threads” order which has an output that is similar to the following:

0:000> !threads
ThreadCount: 5
UnstartedThread: 0
BackgroundThread: 2
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
Lock
     ID    OSID    ThreadOBJ Count Apt Exception
…

Debug a crash
So far we have seen there are many tools available for us to create and analyze a dump. Time has come now to see what we have to do in order to be able to analyze a crash.

• 1. Launch the process
• 2. Before it “crashes”, we order adplus to create a dump the moment when the process “crashes”
• adplus –crash –pn [numeProcesor]
• 3. Launch Windbg (after the crash)
• 3.1 Upload the dump
• 3.2 Upload SOS
• 3.3 !threads (to see which thread has crashed)
• 3.4 !PrintException (on the thread that has crashed in order to see the exception)
• 3.5 !clrstack (to see the stack of calls)
• 3.6 !clrstack –a (to see the stack together with the parameters)
• 3.7 !DumpHeap –type Exception (it lists all the exceptions that are not related to GC).

One must know that the results are according to the way in which the application is compiled. For instance, if there has been a code optimization performed during the compilation. Moreover, the exception list we can get may be quite long due to some orders such as !DumpHeap, which returns all the exceptions encountered – even the ones which have been pre-created, such as ThreadAbord.
How do we identify a deadlock?
A deadlock emerges when two or more threads are waiting for the same resource. In these cases, a part of the application, if not the entire application, gets blocked.
In this case, the first step is to create a dump using the order:

Addplus –hang –o –c:myDump –pn [NumeProces]

Then, it is necessary for us to analyze the stack trace for each thread and see whether it is blocked (Monitor.Enter, ReadWriteLock.Enter…). Once we have identified these threads, we can find the resources used by each thread, together with the thread that keeps these resources blocked.
For these final steps, the order “!syncblk” comes to our help. It lists for us the units of memory for a certain thread.

This article was written by Radu Vunvulea for Today Software Magazine.

Read More

Simple load balancer for SQL Server Database

Two weeks ago I started to work on a PoC where the bottle neck is the database itself. We had a lot of complicated and expensive queries over the database. Because of this, if we wanted to have good performance we had to find a solution to have more than one instance of SQL Server.
Because the solution was on Windows Azure, we wanted to test different configurations and solution. We wanted to see what the performance is if we use 1, 2 and 3 instances of SQL endpoints. Not only this, we had different times of SQL endpoints –Azure SQL (SAS), Virtual Machine with SQL Server and SQL Azure Azure SQL Premium. (SAS but with dedicated resources).
 The good part in our scenario was that the data don’t change very often. We could start from the assumption that the database will be updated only one time per day. Because of this we could use different methods to create a load balancing from our SQL instances.
Because the time was very limited and the only think that we needed was to distributed the load on all the machines we decided to replicate the data on all the SQL instances and create a simple load balancer from code. We divided the tick time (timpstamp) with the number of instances and based on this we selected the SQL instance that will be hit. A simple mechanism that can be very easily managed for the PoC. Of course the final solution will be more complicated, but for a PoC this solution was perfect. In 10 minutes we had a load balancer  :-).
We checked the load of each machine and we can say that the queries were pretty good distributed. The load of each machine was almost the same (+/-10%)

What others solution we could use?
Master-Slave – In this case we have multiple instances of SQL servers. Each slave can be used for reading operation and the master will be used for writing operations.
Sharding – The solution that we used to resolve our problem
AlwaysOn – The base concept is the same one as for Master-Slave
Transaction Replication – Is based on a snapshot and when data is changing, all the subscribers are notified about the change. You can filter individual database objects and publish changes to subscribers
Peer-to-Peer Replication – Is based on Transaction Replication, but with some new features
Change Tracking – Is a very base and simple tracking mechanisms.  The only thing that is notifing you is that a specific row had change. You don’t know any information related to the old or new value
Change Data Capture – Writes in the logs all the changes that are made over a table and are used to track and synchronize the rest of the database instances

Conclusion
We have a lot of solutions on the marker and for SQL Server. Based on our needs we needed to select the best one for us. I was surprised to see that a simple load balancer, written in 5 lines of code can do a pretty good stuff.

Read More

[PostEvent] MSSummit 2013, Bucharest

WOW Why? Because last two days (6-7 November 2013) I had the opportunity to participate at MSSummit 2013, that was held in Bucharest. It was one of the biggest IT event organized in Romania this year.
There were more than 1000 people, 5 simultaneous tracks and more than 45 speakers. It’s been a while since I’ve seen such an event in Romania. I was impressed about the event itself, location and the number of people.
The session list was extremely interesting, we had session held by David Chappell, Chris Capossela, Michael Palermo and so on. All the sessions were great, we had the opportunity to learn and discover new stuff. After this event we could say that Microsoft is full of surprising things (in the good way).
At this event I was invited as a speaker and I held two sessions where I talked about how we can write and load tests using Visual Studio 2013 and Windows Azure. In the second session I talked about fluent communication between client – server – client using SignalR.
Special thanks for the attendees that stayed until 18:30 in the last day of the event.
I hope that next year Microsoft Romania will organize another MSSummit. This is a great conference that can grow and become very big and powerful.
In this following picture I’m with my colleagues from iQuest Group.

Read More