[Post-event] ITCamp 2013

These days I participated at ITCamp 2013. This year I had the opportunity to participate at a premium event with almost 400 attendees. For Romania, this is big number, it is hard to gather 400 people at a premium conference.

What we found this year? COOL stuff. We discover what is the future of tablets, what is happening on the cloud and the voodoo magic that is happening inside the team management... and so on.

I really enjoyed the conference, the sessions and especially the location – you could see all the city from above. Each year one of the most important thing at ITCamp is socialization – meeting people from different parts of the world, discover what they are doing and what are the trends of the IT industry.

This year I had the opportunity to be invited as a speaker at ITCamp. Last year I talked about background task on Windows 8, but this year I decided to go on the cloud and talk about messaging pattern and how we can implement those patterns using Windows Azure Service Bus Services.

My slides from ITCamp:

Messaging patterns in the cloud from Radu Vunvulea

Read More

Day 2 of Software Architecture 2012 & How an architect should be

This week I attended to Software Architecture 2012 conference from London. This was the second day of conference with 4 sessions per day on six tracks simultaneous. Here is the blog post of day one.
There were great sessions about how software design and architecture. I really enjoyed the keynote, where Simon Brown talked about how software architecture should be. One thing that he mentioned and I thing that is very important is what an architecture should do. I think that we know a lot of architecture that don’t write code anymore and don’t learn new technologies. They climb on the Ivory Tower and when we have a question throw a response like: “Implementation detail”. How you can define a solution when you don’t know the technology? Yep, that type of “guru” is only a PowerPoint architected and nothing more.

He is the perfect person that can draw components on a white paper and the first person that will run when a project will have problems. In that moment he will show the SAD (Software Architecture Documentation) but without giving you a real solution.
A pretty nice solution here (because you cannot be master of all technologies) is to let your ego at home and talk with technologies specialists. Design a good solution means making a team with this guys and with the rest of the developing team. They know the real problems that technologies have and what is the real limitation of a framework. On paper everything looks good.
As software architecture you should be able to share your knowledge, to couch and be mentor for other peoples. If you share your knowledge no one will come and steal your position. In reality, your value will increase because the team level will be higher and they will like working with you.
Don’t let people learn from their own experience. Try to share your experience with others, because we don’t want to reinvent the wheel again and again.

Day 1

Day 2

Day 3

Day 4 

Read More

Different methods to implement Message Aggregator pattern using Service Bus Topic – CorrelationId

In one of my last post I presented the Aggregator Pattern. The main purpose of this pattern is the ability of the consumer to combine (aggregate) messages. This pattern can be implemented in Windows Azure using Windows Azure Service Bus Queues or Topics.
There are two different implementation for this pattern. The implementations are extremely different and also can affect our performance.
The first implementation requires using the Session support from BrokeredMessage. What this means from the code perspective? We need to set the Session each time when we want to send a message. The consumer will start to consume the messages with the specific session id. This solution is simple and it works great when we don’t have a lot of messages. For example if we have only 9-10 messages.
The first advantage of this implementation is on the consumer side. We don’t need to create the consume before messages are added to the Service Bus. The messages will be persisted in the Service Bus infrastructure. One of the downside of this implementation is the numbers of the consumers that we can have for the same session. We can have only one consumer. Because of this, if we want to broadcast a message to more than one consumer … we will have a problem. Also, the current implementation of session doesn’t use any kind of hashing for the id field. Because of this, if we want to process thousands of messages per second, maybe we will have a problem.
Producer

Stream messageStream = message.GetBody<Stream>();
for (int offset = 0;
    offset < length;
    offset += 255)
        {                     
            long messageLength = (length - offset) > 255
                ? 255
        : length - offset;
            byte[] currentMessageContent = new byte[messageLength];
            int result = messageStream.Read(currentMessageContent, 0, (int)messageLength);
            BrokeredMessage currentMessage = new BrokeredMessage(
        new MemoryStream(currentMessageContent),
        true);
            subMessage.SessionId = currentContentId;
            qc.Send(currentMessage);
        }

Consumer

MemoryStream finalStream = new MemoryStream();
MessageSession messageSession = qc.AcceptMessageSession(mySessionId);
while(true)
{
    BrokeredMessage message = messageSession.Receive(TimeSpan.FromSeconds(20));
    if(message != null)
    {
                message.GetBody<Stream>().CopyTo(finalStream);
                message.Complete();
                continue;
    }
   break;
}

In the above example we are spitting a stream in small parts and sending it on the Service Bus.
But, what should we do if we want more than this. For example if we want to send messages to more than one consumer. In this case, if we need a way to group messages, like session we will need to think twice. We have some options for this situation. We can use the CorrelationId for this purpose of to add a custom property to the BrokeredMessage. CorrelationId use hashing, because of this is faster than the first option.
Both solutions will work, but we will encounter the same problem in both of them. How we can create a subscription for the given CorrelationId of property before starting receiving messages. This is the biggest problem that we need to resolve.
Before talking about this problem, I would like to talk a little about CorrelationId. This fields that can be set for each BrokeredMessage that is send on the wire. The advantage using is how we can define the filter. We have a pre-define filter for the CorrelationId that can be use when we want to create a subscription. Also each id is hashed; because of this the check is not made using string comparison.
But what is our problem using CorrelationId. We can broadcast a message to more than one subscriber, but… Yes, there is a but. We need to know the correlation id in the moment when we create the subscription. Why? Because the correlation id need to specify in the moment when we are creating the subscriber. Correlation id need to be specified for a subscriber as a CorrelationFilterExpression.
This is not the end of the road. We can find solutions for this problem. The trick is to notify the consumers before adding messages to the Service Bus Topic about the new group of messages that will be added to the system. For this purpose we can use the same Service Bus Topic, or another topic for this. We can have some special messages that have some custom property that describe what will be content of the next flow of messages with the given correlation id. Based on this information, each consumer will be able to decide if we want to receive the given messages.
The trick here is to register the subscribers before the moment when you start broadcast the messages with a given correlation id. The messages will be sending only to the subscribers that are already listening. Because of this, if you are not register from the beginning, there are chances to lose some of the messages.
You need some kind of callback or a timer. Because in a Service Bus pattern, the producer doesn’t know the numbers of subscribers, we cannot create a mechanism where each subscriber notifies the producer using another topic. We could have a waiting time (5s) on the producer side, after he sends the message that notify about the new correlation id.

Here is the code that we should have on the producer and on the consumer side.
-on the producer side, we need to create and send the message that contains the new correlation id. After this we will need to wait a period of time, until the consumers will be able to register to it.

BrokeredMessage message = new BrokeredMessage();
message.Properties["NewCorrelationId"] = 1;
message.Properties["Content"] = "new available cars for rent";
topicClient.Send(message);
Thread.Sleep(10000);

-creating the subscription that check if the message contains the property that specify the correlation id.

namespaceManager.CreateSubscription(
     “myTopic”,
     “listenToNewCorrelationIds”,
     new SqlFilterExpression("EXISTS(NewCorrelationId”));

-create the consumer that processes the message, by creating a new subscription.

SubscriptionClient client =
    SubscriptionClient.CreateFromConnectionString(
        connectionString,
        "myTopic",
        “listenToNewCorrelationIds”);
BrokeredMessage correlationIdMessage = client.Receive();
namespaceManager.CreateSubscription(
     “myTopic”,
     “listenToMyCorrelationId”,
     new CorrelationFilterExpression(correlationIdMessage.Properties["NewCorrelationId"]));
SubscriptionClient client =
    SubscriptionClient.CreateFromConnectionString(
        connectionString,
        "myTopic",
        “listenToMyCorrelationId);

... client.Receive() …

From performance perspective is would be better to use a different topic to send the messages that contains the new correlation id.
We can imagine a lot of implementation. What we need to remember when using correlation id that we need to create a subscription for the given correlation id before starting sending the messages to it. This waiting period that I described in the above paragraphs is the most challenging part. 
In conclusion, we should use session when we need to have only one consumer. If we need more than one consumer, than we should use correlation id.

Read More

Day 1 of Software Architecture 2012 & Unit Test Patterns

This week I attended to Software Architecture 2012 conference from London. This was the first day of the conference and I decided to participate to a full day workshop about design patterns that was held by Andrew Clymer and Richard Blewett.
After this day I made a clearer image in my mind about some design pattern and how we can use it. There are a lot of new things that I discovered today. A part of them will be covered in future blog posts.
I looked over my today notes and I’m trying to extract some information for this blog post, but there are a lot of great things, that I feel the need to dedicate an entire post about it.
I will make only a small introduction into Unit Test Patterns. Any programmer should write unit tests. As we study design patterns in general we should learn also about unit test patterns.
In the last period of time the software industry become mature from many perspectives. The testing area, especially unit testing has reached his maturity. With this maturity, some pattern for unit testing appeared and guys don’t thing if you are a developer you should not know about this. This things are not for testers, are for you, for developers. Developers write unit test and not testers. Also understanding unit testing and pattern that are related to unit testing will improve our production code.
The simplest pattern is “Simple-Test”. This is the most basic unit test pattern that validate that for a valid input the expected outcome is obtained. In the case if our code contains an error trap, than we need to create a test for this case.
Suppose that we have a class Calculate that calculate the sum of two numbers. We will need to write a test that validate that the sum for two numbers is calculated correctly. This test will not guarantee that our code is valid and will work for any kind of input data. It only validate the most simple and basic happy flow.

public class Calculator
{
        public int Sum(int a, int b)
        {
            if (a == 0)
            {
                return b;
            }
            if (b == 0)
            {
                return a;
            }

            return a + b;
        }
}

[TestClass]
public class CalculatorTests
{
    Calculator calculator=new Calculator();

    [TestMethod]
    public void SumOfTwoSimpleNumbersReturnsTheSumOfNumbers()
    {
        Assert.AreEqual(10,calculator.Sum(2,8));
    }
}

Another well know pattern is “Code Path”. In the first pattern we didn’t look over the code. We tested it with the happy flow. The Code Path Pattern is used to test all the paths from our code. In this way, we will be more confident about out code quality. When we are writing this kind of unit tests we don’t look over the requirements, because we could miss some paths from our code. Code Path Pattern requires looking through the code in a “white box” method and covering all the possible paths. When we are using this pattern over legacy code you may be shocked about the number of lines of codes that are not used anymore.

[TestClass]
public class CalculatorTests
{
    Calculator calculator=new Calculator();

    [TestMethod]
    public void SumOfTwoSimpleNumbersReturnsTheSumOfNumbers()
    {
        Assert.AreEqual(10,calculator.Sum(2,8));
    }

    [TestMethod]
    public void SumWhenFirstNumberIsZeroReturnsSecondNumber()
    {
        Assert.AreEqual(5,calculator.Sum(0,5));
    }

    [TestMethod]
    public void SumWhenSecondNumberIsZeroReturnsFirstNumber()
    {
        Assert.AreEqual(3, calculator.Sum(3, 0));
    }

}

The last pattern that I will describe now is “Parameter Range”. When we are writing unit test, we should test our code with more than one input data. Even if we tested all the paths of our code this will not guarantee that our code is perfect.
For example in the previous example we should test what is happening if we calculate the sum of two big numbers that will generate a value that exceed the maximum value.

    [TestMethod]
    public void SumOfTwoSmallNumbersReturnsTheSumOfNumbers()
    {
        Assert.AreEqual(7, calculator.Sum(3, 4));
    }

    [TestMethod]
    public void SumOfTwoBigNumbersReturnsTheSumOfNumbers()
    {
        Assert.AreEqual(180, calculator.Sum(100, 80));
    }

    [TestMethod]
    public void SumOfTwoPrimeNumbersReturnsTheSumOfNumbers()
    {
        Assert.AreEqual(24, calculator.Sum(11, 13));
    }

I would like to congratulate the speakers of this workshop. I think that you made us thinking and see the design patterns from other perspectives.

Day 1

Day 2

Day 3

Day 4 

Read More

Windows Azure Service Bus Patterns - a comprehensive look at patterns that can be used in combination with Windows Azure Service Bus

In the last period of time I posted a lot about pattern that can be used in combination with Windows Azure Service Bus. This is the list of all the patterns that I presented until now:

1. Message Splitter Pattern
2. Message Filter Pattern 
3. Message Aggregator Pattern
4. Recipient List Pattern  
5. Resequencer Pattern
6. Content-Based Router Pattern
7. Scatter-Gather Pattern 
8. Dynamic Router Pattern 

If you want to find more information about Windows Azure Service Bus Topic, please follow this LINK.
For more information about Windows Azure Serbice Bus Queue please follow this LINK.

Read More

Patterns in Windows Azure Service Bus - Dynamic Router Pattern

Last time we talked about Scatter-Gather Pattern and how we can implement it using Windows Azure Service Bus. Today, we will look over Dynamic Router Pattern and how can be integrated in Windows Azure Service Bus.
First of all, let’s see what Dynamic Router is. This is a pattern that can be used when we want to register at runtime different rules and based on this rules the messages are redirected to a specific consumers. This sounds pretty simple in the first moment, but we have 2 different situations that need to be handled.
The first one is from the consumer perspective.  In any moment we need to be able to change the rules or add new rules, without stopping the system. These rules can use properties from messages that didn’t exist when the system started.
The second situations are from the producer. At runtime we need to be able to add or remove properties that can be handled by the system when the routing is made without restarting the system or affecting the consumers.
Top on this, it would be nice to be able to extract all the properties that will be used by the routing mechanism automatically, without the need to make special configuration over the system.
Of course, this pattern can be implemented with success using Windows Azure Service Bus. This pattern can be implemented using Service Bus Queues or Service Bus Topics. The secret here to have the best implementation of this pattern in Windows Azure is to create a mechanism that can extract the properties that are needed by the dynamic routing from the message that is added to the Service Bus. This solution need to be very flexible and to give us the possibility at runtime to control this.
The solution of this problem can be the promoted properties that can be found in BizTalk. This kind of properties doesn’t exist yet in Windows Azure Service Bus, but can be implemented very easily using attributes. I describe this solution in the following blog post: http://vunvulearadu.blogspot.ro/2012/10/winduws-azure-service-bus-adding.htm
In this moment we know how we can dynamically add/remove properties from a BrokeredMessage. From now one, the rest can be implemented very easily. If we are using Windows Azure Service Bus Topic, we can create custom filters over each subscription.

TopicClient topicClient = TopicClient.CreateFromConnectionString(
CloudConfigurationManager.GetSetting(
    "ServiceBusConnectionString"),
    "myFooTopic");
SqlFilter sqlFilterReviewGroup = new SqlFilter(“Groups LIKE ‘%Review%’”);
topicClient.AddSubscription(“ReviewSubscription”, sqlFilterReviewGroup);
SqlFilter sqlFilterTestGroup = new SqlFilter(“Groups LIKE ‘%Test%’”);
topicClient.AddSubscription(“ReviewSubscription”, sqlFilterTestGroup);

This pattern can be used with success when we need a dynamic routing mechanism where rules and filters attributes can change anytime. This is a case that can appear when we offer a solution that is used by other solution providers.
Last edit: A list of all patterns that can be used with Windows Azure Service Bus, that were described by me LINK.  

Read More

Patterns in Windows Azure Service Bus - Scatter-Gather Pattern

I will continue the series of posts related to patterns that can be used using Service Bus from Windows Azure. In the last post from this blog series I talked about Content-Based Router Pattern and we saw how easily can be implemented using Service Bus Topics.
Today we will look over another message pattern. I don’t know if you ever heard about Scatter-Gather message pattern. It is not a widely used pattern. Also there are a lot of cased when we use this pattern without realizing.
The first step is to try to define this pattern. As you can see, this pattern is formed from two different words – scatter and gather.

• Scatter – refers how we can send a message to a list of receivers (consumers)
• Gather – refers how we receive a collection of messages from more than one source

Before talking more about this pattern let’s see an example where this pattern can be used with success. Very easily we can imagine that we are a company that wants to by surfaces (Windows 8 Tables) for all your employ. To get the best price on the market you want to have a price offer from all your suppliers. To be able to do something like this you will need to send a message with the price request for each supplier. After this you will need to gather all the offers and process them.
I thing that you already notified the “scatter” and “gather” in our example:
Scatter - sending the message to our suppliers
Gather – gather the prices from all the suppliers
As you can see, this pattern doesn’t represent only a small picture, but it a little more complicated. From some perspective we could say that this pattern is a combination of two different patterns – splitter and aggregator patterns.
Windows Azure Service Bus supports this pattern without any kind of problems. The Scatter-Gather pattern can be implemented with success using Service Bus Topic.
From the Scatter perspective we will each of our supplier to register to our topic. Each subscriber will receive our message with our request and we will be able to send to us their offer.
Creating the TopicClient:

TopicClient topicClient = TopicClient.CreateFromConnectionString(
CloudConfigurationManager.GetSetting(
    "ServiceBusConnectionString"),
    "myFooTopic");

Create a subscription for each supplier:

NamespaceManager namespaceManager = NamespaceManager                             
                 .CreateFromConnectionString(“ServiceBusConnectionString”);
namespaceManager.CreateSubscription( “myFooTopic”, “subscriptionNameForSupplier1”);

>Each supplier needs to listen the subscription (this action don’t need to be done 24/24h) – Windows Azure will preserve the messages until the subscriber will consume them.

SubscriptionClient subscriptionClient = SubscriptionClient.CreateFromConnectionString(
    CloudConfigurationManager.GetSetting("ServiceBusConnectionString"),
    "myFooTopic",
    "Property1Equal10");
BrokeredMessage message = subscriptionClient.Receive();
Sending the message to our subscribers:
BrokeredMessage message = new BrokeredMessage();
…
topicClient.Send(message);

Create a queue where each supplier can send the price offer and listen the given queue by our company:

NamespaceManager nm = NamespaceManager.CreateFromConnectionString(
     CloudConfigurationManager.GetSetting(“ServiceBusConnectionString”));
if (!namespaceManager.QueueExists("FooQueue"))
{
    namespaceManager.CreateQueue(qd);
}
QueueClient queueClient = QueueClient.CreateFromConnectionString(
    myFooConnectionString,
    "FooQueue");
BrokeredMessage offerReceived = queueClient.Receive();

When we create the infrastructure used to receive the offers from each supplier, we can use Service Bus Topics or Service Bus Queues without any kind of problems. For this case I think that Service Bus Queues is better, because we don’t need to distribute the messages to more than one receiver.
Last edit: A list of all patterns that can be used with Windows Azure Service Bus, that were described by me LINK.  

Read More

Patterns in Windows Azure Service Bus - Content-Based Router Pattern

In one of my latest post I talked about Resequencer Pattern, which can be used with success using Windows Azure Service Bus.  We saw how we can retrieve messages in the same order as they were sending to the Service Bus.
But what about Content-Based Router Pattern? The main scope of this pattern is the ability to route each message to different clients based on the data of each message. The system the process each message has to be able to redirect a message to a different consumer (client) based on the data that message contains.
One of the key features of this pattern is ability to change and maintained the rules that are used to redirect each message based on the content.
Windows Azure gives us the possibility to implement this pattern using Service Bus Topic. Each channel, where messages are redirected can be represented by a subscriber. As you already know each subscriber can have attached a filter that can filter messages based on the content. For this purpose we can use SqlFilter. Using SqlFilter we can look over the properties of a message and route them based on this purpose.
To be able to route messages based on the properties, we need to add the properties to our message. This step needs to be done on the producer side, on the system that produces messages, before adding them to the Service Bus Topic.
In the following example we will see how we can add these properties to our messages and how we can define these rules on the subscription side.
First step is to create the topic:

NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString(
     CloudConfigurationManager.GetSetting(“ServiceBusConnectionString”));

if (!namespaceManager.TopicExists("myFooTopic"))
{
    namespaceManager.CreateTopic("myFooTopic");
}

After that, from the client side, we can create a message and send it to the topic. In our case we add a custom property named “property1”.

TopicClient topicClient = TopicClient.CreateFromConnectionString(
    CloudConfigurationManager.GetSetting("ServiceBusConnectionString"),
    "myFooTopic");

BrokeredMessage message = new BrokeredMessage();
message.Properties["property1"] = 10;
topicClient.Send(message);

>Next, we need to create the subscription that accepts only messages that have property equal to 10.
topicClient.AddSubscription("Property1Equal10", new SqlFilter( "property1 == 10") );
The last step is on the consumer side. We will consume message that are sent our subscription.

SubscriptionClient subscriptionClient = SubscriptionClient.CreateFromConnectionString(
    CloudConfigurationManager.GetSetting("ServiceBusConnectionString"),
    "myFooTopic",
    "Property1Equal10");

while(true)
{
   BrokeredMessage brokeredMessage = subscriptionClient.Receive();

   if (message != null)
   {
       try
       {
           ...
           message.Complete();
       }
       catch (Exception)
       {
           message.Abandon();
       }
      }
   }
}

There are a lot of cases when we can use this pattern. We can imagine a system that need to route messages based on the information contained in the message. For example we can have a system that process newsletters. Based on the content of the newsletters we want to redirect messages to the specific group of email. This can be very easily done using this pattern.
Last edit: A list of all patterns that can be used with Windows Azure Service Bus, that were described by me LINK.  

Read More

Patterns in Windows Azure Service Bus - Resequencer Pattern

Today we will talk about another message pattern: Resequencer. In the last post I presented Recipient List Pattern. In comparison with this pattern, Resequencer Pattern is very different. The main scope of this pattern is to help us to put messages back in a specific order.
When we are talking about messages, we can talk about a stream of messages that need to be received in a specific order. It is very crucial for the receiver to retrieve the messages in the same order he receives.
Theoretically, in a simple case we will receive messages in the expected order. This is offer Service Bus by default. But what happen if an error occurs on the receiver and message is putted back in the queue. We will need to retry to consume that message one more time and not the next message.
Another case when the order can be broken is when we have more than one producer. For this case we can have two different situations.
In the first scenario, each producer will produce messages for different stream of messages. In this case we can very easily use the session id of the messages to be able to receive only messages for a specific stream. But we will still need a way to detect if the messages if the message that we expect. First step is to add two properties to each message. The first property will tell us how many messages are in this message stream and the other one will tell us the index of the current message. Base on this information, the receiver will know the index of the next message and will be able to validate it. If we will receive messages that don’t have a valid index id, we can throw them in the defer queue, from where we will be able to retrieve them anytime.
Producer:

QueueClient  queueClient = …
BrokekedMessage message = new BrokeredMessage();
message.Properties[“index”] = 1;
message.Properties[“count”] = 10;
message.SessionId = 123;
queueClient.Send(message);

Consumer:

MessageSession messageSession = queueClient.AcceptMessageSession(123);
int currentIndex = 1;
while(true)
{
    BrokeredMessage message = messageSession.Receive();
    if(int.Parse(message.Properties[“index”]) != currentIndex)
    {
        message.DeadLetter();
        continue;
    }
    …
    message.Complete();
    if(int.Parse(messsage[“count”]) == currentIndex)
    {
        break;
    }
    currentIndex++;
}

Next we need to take message that were marked as dead letters and moved automatically to the dead letter queue.

QueueClient deadLetterQueue = QueueClient.CreateConnectionString(
    connectionString,
    QueueClient.FormatDeadLetterPath(“FooQueue”));
while (true)
{
    BrokeredMessage message = deadLetterQueue.Receive();
    // same logic as in the normal queue.
    // we need to abandon the message and not to mark him as dead letter.
    // we already process the dead letter messages
}

The second scenario is a little more complicated. In the same queue, we have more than one producer that produce message for a specific stream. The chances to have messages in the expected order are very low. The solution is similar to the first one. We can add the index of each message and total number of messages as properties to the message that is added to the queue. The consumer can check this values and when the message index is wrong, the message will be added to the defer queue.
In the both solutions, the most time consuming is retrieving the message from the defer queue. The good part that usually the messages are in a kind of order, even if is not perfect (eq. 1 3 5 2 6 7 10 8 9). Because of this we will not need to “iterate” through the defer queue to many times.
This pattern can be used for cases when it is critical to process messages in a specific order. In a system that sells tickets for a baseball game this is not so critical. But for a system that receives commands using messages it is very important to execute the commands in the same order – for example a nuclear power station.
This is a pattern that is not very common. When you reach a case where you need this pattern, try to double check again if you need it, because this pattern can be very expensive – from the perspective of processing time and resources.
Last edit: A list of all patterns that can be used with Windows Azure Service Bus, that were described by me LINK.  

Read More

Patterns in Windows Azure Service Bus - Recipient List Pattern

I will continue the blog series of post about patterns that can be user with Service Bus of Windows Azure with Recipient List Pattern. Did you ever try to send an email to a list of users? Using Exchange Server is quite simple. We can create a group list and send an email to the specific group list. For example we can have groups of email for different products categories. One can be for TVs, another one for Notebooks and so on. If we create a complex system for a company that have different products that want to send we will need to be able to send notifications about new products that are available. To be able to do something like this we will need a system that permits us to send notifications to a “group” – to a list of subscribers. In a system that is based on messages we will not know the list of subscribers for each group. Because of this we can decorate each message with some meta-information about the groups should receive the message. In Windows Azure, we can implement this pattern using Service Bus Topics. For each message we can add a property to the message that specifies the groups of subscribers that will receive the message. This list of messages can be separate with a comma or any kind of character.

BrokeredMessage message = new BrokeredMessage();
message.Properties.Add(“Groups”,”Review, Test”);
…

Each subscriber will need to create a custom filter. For this purpose we can use the ‘LIKE’ operator of SqlFilter.

TopicClient topicClient = TopicClient.CreateFromConnectionString(
CloudConfigurationManager.GetSetting(
    "ServiceBusConnectionString"),
    "myFooTopic");
SqlFilter sqlFilterReviewGroup = new SqlFilter(“Groups LIKE ‘%Review%’”);
topicClient.AddSubscription(“ReviewSubscription”, sqlFilterReviewGroup);
SqlFilter sqlFilterTestGroup = new SqlFilter(“Groups LIKE ‘%Test%’”);
topicClient.AddSubscription(“ReviewSubscription”, sqlFilterTestGroup);

Even if this solution will work without any kind of problem, we should be aware that we use the “LIKE” operator. From performance perspective we know that this is not the fastest solution. If we don’t have a lot of messages that are send on the wire that this is not a reals issue. For better performance we can find different solutions, from complicated one that use bits to simpler one like adding different property for each group. In this way the subscriber of a filter will only need to check if the property is set or not. For this purpose we can use the “EXISTS” operator or SqlFilter.

SqlFilter sqlFilterReviewGroup = new SqlFilter(“EXISTS Review”);
topicClient.AddSubscription(“ReviewSubscription”, sqlFilterReviewGroup);

If you don’t want to have a lot of properties you can a prime numbers. The only problem with this solution is related to how easy the code can be read and maintain. Each prime number can represent a different group. And a message should be received by a subscription if the group property can be divided to the prime number that represents our group. We could imagine another solutions also.
To be able to use this pattern we need to define the list of recipients. This list can be a static list of can be created dynamically and can be change on runtime. Using Service Bus Topics from Windows Azure the list can be dynamically created and changing the list will not require changing the code that send or receive the messages. All this can be done in the configuration files.
The important thing that we need to remember about Recipient List Pattern is when we need to use it and Windows Azure Service Bus supports an implementation of this pattern.
Last edit: A list of all patterns that can be used with Windows Azure Service Bus, that were described by me LINK.  

Read More

Patterns in Windows Azure Service Bus - Message Aggregator Pattern

In the last post I talked about Message Filter Pattern that can be used on Windows Azure Service Bus. Today I will describe how we can use Message Aggregator Pattern. From some points of view, this pattern is the opposite pattern in comparison with Message Splitter Pattern.
Do you remember when Message Splitter Pattern can be used?… when we want to split messages based on some rules. Message Splitter Pattern does the opposite thing. This pattern can be used when we want to aggregate different messages.
This pattern can be used with success in cases when we send some information in more messages. For example if send GPS position of cars based on the type. Until the car will reach the destination it will send a lot of messages. We will need to mark with a custom property the last message that is send by the cars of a given type – or to define a timeout period. When the last message will be received we will be able to process the data – for example to calculate the total distance and the speed of the car.
Using Windows Azure Service Bus we have two possibilities to implement this pattern. The first one is to use sessions. I already described a solution based on session in the following post: http://vunvulearadu.blogspot.hu/2012/08/service-bus-queues-from-windows-azure_20.html
This is not the only way to implement this pattern. If we are using Windows Azure Service Bus Topic we can use with success CorrelationFilterExpression.
Why to use CorrelationFilterExpression and not session? Because usually this pattern is used when we work with a lot of messages that need to be aggregate. The correlation id is stored in a hash table and the matching is faster. In comparison, the session id comparison is not hashed and is not optimized for 1 to 1 match – we only compare two strings using a SQL expression filter. From this perspective, using CorrelationFilterExpression is better.
Let’s see how we can use CorrelationFilterExpression to make our life easier. For this solution we will need to use Service Bus Topics. On the client side we will need to set the correlation id. Every BrokeredMessage has a property named “CorrelationId” that can be set to our specific value.

BrokeredMessage message = new BrokeredMessage();
…
topicClient.Send(message);

Remember one thing and this is very important when we want to use the CorrelationFilterExpression. We specify the correlation id in the moment when we create the subscription, through the rule that we are creating. Because of this a subscription will be only for one correlation id. This is the cause why the session id is preferred when we don’t have many messages from the same “group” (session). If you remember the example with the cars and GPS location we will have a lot of messages from different cars type that will be grouped based on the car type. In our case this will represent the correlation id. This id can be any string, we are not limited to a int or some values.

namespaceManager.CreateSubscription(
     “myTopic”,
     “sedanSubscription”,
     new CorrelationFilterExpression(“sedan”));

This is the only thing that we need to do from the configuration perspective. Usually there is not a problem with the subscription that is “glued” to a specific correlation id because from this perspective, the ids should not change in time. Also there will be a constant flow of messages.
As a conclusion let’s see when we can use Messaging Aggregation Pattern. We can use it when we need to aggregate messages based on a specific flag. From the performance perspective it is recommended to use the CorrelationFilterExpression that hash the id that is specified.
Last edit: A list of all patterns that can be used with Windows Azure Service Bus, that were described by me LINK.  

Read More

Patterns in Windows Azure Service Bus - Message Filter Pattern

In the last post I tacked about Splitter Pattern.Today we will continue with Message Filter Pattern. This pattern can be used with success with Windows Azure Service Bus.

As the name says, all the messages are filtered based on specific rules. Any message that will reach the consumer will be filter based on this rules. All the producers will use the same entry point where they will add messages. They don’t have to know that messages are filtered based on same rules. In this way, the message system will create a decoupling between the producer and the consumer. This pattern is used to be able to control the messages that where not routed to any subscriber.

Windows Azure Service Bus Topics can be used for this purpose. It gives us the ability to define rules that can check the messages content based on meta-information. These rules will be added to each subscriber and will specify if the messages will be accepted or not.

SqlFilter myCustomFilter =
    new SqlFilter("grade < 5");
namespaceManager.CreateSubscription(
               "StudentsGradesTopic",
               "StudentsWithProblemsSubscription",      
               myCustomFilter);  

 In the following example I created a subscription that accept messages that have the grade under 5. For more information about defining custom rules: http://www.vunvulearadu.blogspot.hu/2012/08/service-bus-topics-how-to-use-it-part-2.html

When we are using this pattern, consumers will only receive and process messages that where filtered. Because of this, we can have messages in the system that will not pass any filter. This messages need to be tracked in one way or another. For these situations we can define a custom rule in Service Bus Topic that will receive messages that didn’t pass the rest of the rules of the subscribers. The name of the filter expression is “MatchNoneFilterExpression”. In the following example we setup a rule that accept messages that didn’t pass the rest of the rules.

RuleDescription notConsumedMessagesRule = new RuleDescription()
{
     FilterAction = new SqlFilterAction(“set isNotConsumed = true;”),
     FilterExpression = new MatchNoneFilterExpression()
};
subscription.Add(notConsumedMessagesRule);

For more information about this time type filters: http://www.vunvulearadu.blogspot.hu/2012/09/service-bus-topics-define-custom-rules.html

This pattern can be used with success when we need to control what kind of messages is received by each consumer. We can imagine that we need to manage grades from a university. For this purpose each department want to receive information related to them. The history department doesn’t want to receive grades from the mathematic department. For this case the Message Filter Pattern can help us a lot because it created only one entry point for the applications and services that add these messages.
Last edit: A list of all patterns that can be used with Windows Azure Service Bus, that were described by me LINK.  

Read More

Patterns in Windows Azure Service Bus - Message Splitter Pattern

In one of my post about Service Bus Topics from Windows Azure I told you that I will write about a post that describe how we can design an application that can route messages for a house which is controlled remotely. Before we can talk about this we need to see some design patterns that can be used in combination with Service Bus. I will write a series of post about this.
We will start with Splitter pattern. This pattern refers to the ability to have a collection of messages parts that form one message or an entity for us. This pattern gives as the ability to receive and process messages from the related messages separately. In this way all the messages that belong to that message part will be sent to the same consumer.

 

How we can use this pattern in Service Bus? Hmm, is pretty simple. We can use SessionId property of a message for this. In this way, when a client (consumer) start to receive a message with a given session id, we can specify to receive only messages with the given id. Service Bus guarantees that the messages from the given session will be received in the same order they were added.
When a client start to receive messages of a specific session, the session is automatically locked. Only that client will be able to receive messages with the given session id. This action is a transactional action. Because of this, if something will happen with the consumer (crash for example), all the messages for the specific session will be received, even if the client that consumed a part of the messages from that session crashed.
To be able to lock the messages from Service Bus to be consumed only by the client that started to receive messages with a given session id we need to the RequiresSession property of the QueueDescription or SubscriptionDescription. This is the only configuration that needs to be done on the Service Bus. On the consumer don’t forget to use the AcceptMessageSession() method to receive a reference to a session from ServiceBus. This method can be found under the abstract class MessageClientEntity that is implemented by all the crucial classes that are used to receive messages from Service Bus:

• QueueClient
• TopicClient
• SubscriptionClient
• MessageSender
• MessageReceiver
• MessagingFactory

Because of this we can write the same code that will be used when we use queues or topics. More information about how you can write code that can consume Service Bus Queues and Service Bus Topics can be found on the following link: www.vunvulearadu.blogspot.com/2012/08/service-bus-topic-how-we-can-migrate.html
In the next example I will get a reference to MessageSession from a queue, a topic and a MessageReceiver.

MessageReceiver
// Service Bus Queue
QueueClient queueClient =QueueClient.CreateFromConnectionString(
     myFooConnectionString, 
     "FooQueue");
MessageSession sessionFromQueue = queueClient.AcceptMessageSession();
// Service Bus Topic
SubscriptionClient subscriptionClient = SubscriptionClient.CreateFromConnectionString(
     CloudConfigurationManager.GetSetting(
          "ServiceBusConnectionString"),
          "myFooTopic",
          "seccondSubscription");
MessageSession sessionFromSubscriber = subscriptionClient.AcceptMessageSession();
// MessageReceiver
Uri serviceAddress = ServiceBusEnvironment.CreateServiceUri("sb", “myFooNamspace”, string.Empty);
MessagingFactory messagingFactory  = MessagingFactory.Create(serviceAddress, credentials);
MessageReceiver messageReceiver = messagingFactory.CreateMessageReceiver(“myFooQueueName”);
MessageSession sessionFromMessageReceiver = messageReceiver.AcceptMessageSession();

Don’t forget to activate the session support from the queue or topic of the Service Bus.

QueueDescription queueDescription = new QueueDescription("FooQueue");
queueDescription.MaxSizeInMegabytes = 5120;
queueDescription.DefaultMessageTimeToLive = new TimeSpan(0, 10, 30);
queueDescription.RequiresSession = true;
if (!namespaceManager.QueueExists("FooQueue"))
{
    namespaceManager.CreateQueue(queueDescription);
}

Once we have the reference to MessageSession we can consume messages with the same session id using Receive method of the MessageSession (as a hint: the base class of this class is MessageReceiver).

while ( true )
{
     BrokeredMessage message = session.Receive();
     ...
     message.Complete();
}

This pattern can be used when we the messages are a specific order that is important for the received. We want to be able to specify where the messages need to be sent. As an example we can imagine a hyper-market that sell a lot of things. Based on the type of the product we have different services that need to consume these messages and process this. For this case, each session can be a different product category. Another case when slitter pattern is very useful is for the case when we want to send content that is too big for a message. For this case we need to use a splitter.
Tomorrow we will continue with another pattern where Windows Azure Service Bus can be used.
Last edit: A list of all patterns that can be used with Windows Azure Service Bus, that were described by me LINK. 

Read More