Event-driven jobs

For scheduling jobs on Windows, the Windows Task Scheduler can do the job (see http://en.wikipedia.org/wiki/Windows_Task_Scheduler). However, depending on the complexity of the job, the Windows Task Scheduler may not be enough to deal with it. For instance, if a holiday schedule needs to be checked before starting the job and if the job has dependencies on other events and services happening before it runs, you may run into limitations with the task scheduler. The Windows Task Scheduler is a great tool, but it is not event driven, as is an ESB. With ESBs, time is an event.

Even though a job may be easily done by Task Manager, you will not get the global management of services, the messaging of events, commands, and messages, and the feedback of the processing from a managed perspective, unless you use a framework such as NSB.

NSB extends the design to allow many facets of event-driven designing where building a simple website, scheduling a daily job, or sending payments can easily grow into something much larger in order to have, monitor, manage, extend, secure, and maintain great software quality. It is not uncommon that when these simple designs become complex, a rewrite is warranted. That is simply because software quality and extending the design were not taken into consideration ahead of time.

The ESB is an architectural software design pattern and framework for designing SOA architectures. However, there are many additional software design patterns that an ESB can bring to the table as well. These will be software design patterns that NSB supports, but not all ESBs support these patterns.

Additional SOA patterns

There are many additional software design and messaging patterns that NSB supports. Many such patterns can be found at http://www.enterpriseintegrationpatterns.com. The purpose of design patterns is to provide reusable frameworks that have been hardened through reuse in the industry, to rapidly deploy what's been designed before. NSB has several design patterns that are popular in any robust and solid ESB solution.

The publish-subscribe pattern

One of the biggest benefits of using ESB technology is the availability of the publish-subscribe message pattern (for more details, see http://en.wikipedia.org/wiki/Publish-subscribe_pattern). The publish-subscribe pattern has been around for a very long time, and its simplicity is the cornerstone of many, many systems.

The pattern makes use of message queues, and NSB stores its messages through the use of queues, be it MSMQ, SQL queues, RabbitMQ, Azure queues, or ActiveMQ. The basic premise is that a message is published to a queue; in NSB, it uses the Bus.Publish(message) function that places an event message in a queue that does not need to be sent to specific receivers. There can be multiple output channels called subscribers listening in on the publisher. The benefit is that the services do not depend on any relationship with each other except the message, and can act independently of each other. This is a service which is decoupled. This means that dependencies are separated from each other.

Of course, normal NSB features still apply. For instance, the message will stay in the queue until the subscribers pull the message off the queue to handle it. For NSB, this means that the machine can be rebooted and the message will still be available in the queue until the subscribers process the message, which ensures that a message is never lost.

The subscribers will process the message, and there could be one or many subscribers. For instance, the publisher may put 10 different types of messages in its queue, and there could be a subscriber for each type of message.

The NSB framework has plugins for Windows Performance Monitor. In Service Level Agreement (SLA) code of the endpoint, we can specify that new services will be created or alerts will be sent if the service does not meet performance criteria. So if one of the subscribers is not performing well, it will force the other subscribers to take over, as they are performing better across multiple machines. Because of this, NSB is a highly available, durable-messaging, high-performance ESB engine that can be developed in a multitude of ways. A design of the publish-subscribe pattern appears as follows:

In NSB, there are special types of messages used specifically for publish-subscribe, called events. This forces the code to handle the message in a publish-subscribe scenario, and not in a request-reply scenario.

The publisher-subscriber mappings are done through the application configuration file. This will define the type of messages from the publisher queue that it will subscribe to, to handle the messages. When these programs execute, the subscription mappings will be further saved into a local database. By default, they are saved in RavenDB.

The request-reply pattern

The request-reply pattern is different from the publish-subscribe pattern as it sends a message to a directed endpoint.

The NSB framework will use the Bus.Send(message) function for request-reply. The message remains durable as it is still queued. The sender may or may not be available to handle the response, and the replier may not send a response, but the functionality in NSB is available to have these components easily added to handle the request-reply on their endpoints. The sender could easily be a website and the replier could send a response to update the website, for instance, when a credit card has been processed. The request-response pattern doesn't require subscription information to be stored, but it does require the endpoints to be defined along with the message types in the application configuration file. Sagas automatically store the sender's original information and reply directly to the requester, without storing extra information.

This pattern still has high availability and high performance, and still uses a multitude of topologies with message durability. Please see the scale out pattern in this chapter on how request-reply will cluster across machines.

The gateway pattern

There are cases when services may be partly stored on one part of an organization's LAN, and other services are stored on another LAN, and the only transport to each other is an HTTP or HTTPS tunnel to pass messages to NSB.

The main purpose of the gateway is to allow you to do the same durable fire-and-forget messaging that you are accustomed to with NServiceBus across physically separated sites, where sites are locations in which you run IT infrastructure and not websites.

The gateway only comes into play where you can't use normal LAN-to-LAN VPN tunnels or use internal LAN servers to communicate MSMQ to MSMQ. The purpose of the gateway is to create messages that communicate through HTTP, but it would be preferable to use HTTPS to ensure that messages are secured. The following architecture diagram represents a gateway built using the NSB:

 <GatewayConfig>
    <Sites>
      <Site Key="SiteA" Address="http://localhost:25899/SiteA/" ChannelType="Http" />
      <Site Key="SiteB" Address="http://localhost:25899/SiteB/" ChannelType="Http" LegacyMode="false" />
    </Sites>
    <Channels>
      <Channel Address="http://localhost:25899/Headquarter/" ChannelType="Http" />
      <Channel Address="http://localhost:25899/Headquarter2/" ChannelType="Http" Default="true" />
    </Channels>
  </GatewayConfig>

  <connectionStrings><add name="NServiceBus/Transport" connectionString="cacheSendConnection=true"/>
    <add name="NServiceBus/Persistence" connectionString="Data Source=.\SQLEXPRESS;Initial Catalog=nservicebus;Integrated Security=True"/>
  </connectionStrings>
  <!-- specify the other needed NHibernate settings like below in appSettings:-->
  <appSettings>
    <!-- dialect is defaulted to MsSql2008Dialect, if needed change accordingly -->
    <add key="NServiceBus/Persistence/NHibernate/dialect" value="NHibernate.Dialect.MsSql2008Dialect"/>
    <!-- other optional settings examples -->
    <add key="NServiceBus/Persistence/NHibernate/connection.provider" value="NHibernate.Connection.DriverConnectionProvider"/>
    <add key="NServiceBus/Persistence/NHibernate/connection.driver_class" value="NHibernate.Driver.Sql2008ClientDriver"/>
  </appSettings>

The DataBus pattern

The DataBus is used to send large chunks of data or files across as an attachment because MSMQ is limited to 4 MB. For this reason, a reference can be passed on a local file to transfer data using the databus method. The message will provide a reference to a larger data file to be accessed that exceeds the message queue size due to size constraints.

In this section, we will use the gateway solution.

The path of the data bus has to be set in the configuration of the endpoint. We will be using a relative path to where the gateway project is running. Both SiteA and SiteB will also have relative paths. There will be a relative path to the binary file with a data bus subdirectory containing the files that will have a lot of data. The following is the code:

When we execute the gateway project, we can mock the SomeLargeString variable to simulate data larger than 4 MB, as shown in the following code:

If we execute the gateway project, it will create a message to the relative path of its binary file, save the message under data bus, and use it as a reference to send to SiteA and SiteB. Here, we see the message saved to the local relative path.

The data bus is very useful for processing files or chunks of data that are too large for MSMQ.

Timeout patterns

In ESB systems, the need for timers and timeouts cannot be underestimated. Many developers use the Microsoft Task Scheduler. The Microsoft Task Scheduler looks like this:

In NSB, we have the ability to use event-driven timer messages. This is the ability of a saga to start a function, run a command, or perform many other tasks, based on its timer. In the event-driven timer function, we can set a timeout for any time in the day, from seconds after the process starts, to minutes, hours, days, and even use a holiday table for the process. By having a managed service, the usage of timers can be more complex, for instance, adding holiday tables, not executing timers on weekends, and more business functions that task schedulers cannot handle. Also included in NSB, is the installation and management of the service itself. Time saved just in deploying services and having NSB set up DTC for the administrator may already pay for the NSB licenses.

The source code in this section will be in NSB Version 5.0 in the TimerSaga – v5 directory. Here we have the following projects:

In this solution, we have a Windows form, where we enter a variable from 1 to 100, based on which the TimerSubmit project will generate a message to the Timer saga to create a timeout for those seconds. After those seconds expire, the Timer saga will handle the timeout message and respond back to the TimerSubmit window saying that the timer has expired. This is a simple exercise in an event-driven timer that could have many uses.

The program will start by submitting the number of seconds between 1-100 to TimerSubmit as shown in the following screenshot:

When we add a value between 1 and 100 seconds, the message will be sent to the queue of TimerSaga with the number of seconds and a RequestId to keep track of the message ID, as shown in the following screenshot:

The message's RequestId is used to map the message to the saga data. The saga is started by SubmitRequestCommand with the number of seconds in it. It will save the saga data in the TimerRequestData object, which will allow the original client data to respond to it. The number of seconds will be set in the timeout, and when the timeout expires, it will execute a TimeoutMessage instance that will be handled by the saga. The saga's object class definition that defines this mapping appears as follows:

The starting message SubmitRequestCommand will be handled by the saga's message handler in the following code:

This code will set the timer that will send a TimeoutMessage instance with the message's RequestId. All the messages that are part of the same starting message will have the same RequestId. This will keep track of which message started the next message, as well as map the saga data to save and retrieve it from the saga. The RequestId is handled like a primary key to the saga data that NSB will use to map the data to the messages. NSB handles the mapping, but we must define the unique identifiers and keep processing them in the messages. If this seems complicated, keep reading, as it is broken down in subsequent chapters.

When the timeout that we passed to the RequestTimeout<>() code is reached, an instance of a timeout message is passed back to the saga. If it was set for 10 seconds, after 10 seconds have passed, the following message handler in the saga is called to process the timeout message that we set in the timer. The Timeout message handler appears as follows:

This code will process the TimeoutMessage instance that we called state. We had mapped the RequestId in the version 5.0 Saga mapper code, ConfigureHowToFindSaga(SagaPropertyMapper<TimerRequestData> mapper) to map the message's RequestId to the data, so the state.RequestId will be the same as the Data.RequestId data instance of saga. The RequestId is checked to ensure that when we execute MarkAsComplete(), the correct saga store is deleted.

This can be broken down into simple steps:

The saga data is deleted, and a message is sent to the originator saying that the timeout has been completed. In this message handler, we create a SubmitRequestReplyMessage instance with the RequestId, which will be the same RequestId as the message that started this process. We reply to the originator, which will send this message to the original TimerSubmit program.

When the TimerSubmit process receives the reply message, it will pop a Timer in Seconds window as follows:

This code has all the pieces of a simple saga. The code could be created with a combination of other frameworks as well, such as Quartz for the timer and TopShelf to create services, but NSB is a complete framework that provides end-to-end pieces of architecture and many patterns that have all the pieces.

The end message that showed the Timed status in the Windows form could easily be the execution of a job, another program, an e-mail of system status, a report of the system status, and many more timed tasks. The number of seconds used earlier was just a demonstration. The RequestTimeout() function could take seconds, minutes, hours, days, or could be executed at a specific time of day. We could execute the function with a weekend and holiday calendar and extend the functionality further. We could create a centralized saga timer to literally schedule all the tasks like Microsoft Task Scheduler, but create a managed service that can be monitored, managed, perform reporting, and improve functionality in a much further detailed solution.

Message mutation patterns

Message mutators allow you to change messages by plugging custom logic into a couple of simple interfaces. For instance, you can encrypt a part or all of a message. The encryption message mutator is part of the NServiceBus library, and can be used at any time. You can intercept the incoming message and then mutate it before sending it as an outgoing message. This is the process of changing messages as they leave a client and enter a server.

The source code

In this section, we will be using the MessageMutators solution with the following projects:

• Client: The client will send messages to the server.
• Server: This will receive the mutated message.
• Messages: This is the message format being passed between client and server.
• MessageMutators: This project will contain the mutation code to compress and decompress the messages in TransportMessageCompressionMutator.cs and validate the message annotation in ValidateMessageMutator.cs.

The client and server needs to be running. The client will prompt to send a good or bad message. The good message is compressed so that it will pass the 4 MB MSMQ buffer size, as shown in the following screenshot:

The queue's data will be validated and compressed from the client before processing it in the MSMQ. This is shown in the following diagram:

Then, the server will receive the message from MSMQ, but before processing it, this will decompress and validate the message. It will restore the message that the client mutated. This is shown in the following diagram:

This is just a simple compression and data annotation validation to ensure that MSMQ will process the message. One of the reasons for mutating the message may be the encryption of a credit card within a payment message.

Message encryption patterns

NSB supports the AES (Rijndael) encryption algorithm. AES stands for Advanced Encryption Standard. This is a symmetric key algorithm, so both the program encrypting data and decrypting data must share a secret key for their functioning. Visit http://en.wikipedia.org/wiki/Advanced_Encryption_Standard for more information on AES.

Encrypting data will depend on the needs of the organization, but common items include passwords, financial information, or personal customer identification information. AES is the strongest symmetric encryption algorithm, and most languages, such as Java and C#, have API support for its use.

We know that part of the configuration on both sides will be a secret key.

The source code

In this section, we will be using the Encryption solution with the following projects:

• Client: This will send an encrypted credit card message to the server
• Server: This will receive the credit card message and decrypt it
• Messages: This is the message format being passed between client and server

Both, the client and server must be running. The client will have a prompt to send messages to the server, as shown in the following screenshot:

After pressing Enter, we see that the message is encrypted on the server queue:

When running the server, NSB will decrypt the message before passing it to the server's message handler.

All that is really needed is to enable both ends for AES in the IBus by the configuration .RijindaelEncryptionService();. We set the part of the message that we want to encrypt by the public WireEncryptionString Secret { get;set; } where WireEncryptionString defines that the string will be encrypted. Also, the secret key has to be in the App.config file of both the client and server. The following screenshot shows the code:

The ScaleOut pattern

As mentioned earlier, one of the many benefits of using NSB is that you can distribute the load or the NSB services or processes. This is commonly known as scaling out the services. The idea is that you can deploy the same service across a farm or multiple servers. This is used to create an environment of high availability.

This model is a form of round-robin clustering, where a handler can distribute its workload to additional workers doing exactly the same work. A distributor is used with MSMQ. If an endpoint has a critical time set for performance and requires more processing help, this clustering is used to spawn off work to the same services living on other machines to share the load. If the machine processing the message crashes, the message rolls back to the queue and other machines can then process it.

Worker services send messages through a control queue saying that they are ready for work. The distributor stores these messages, and when it receives messages, it fetches them out of the available queues. All pending work stays in the distributor's queue so that messages can be timed for performance.

The saga design pattern

The saga design pattern starts with a message to the saga service. The saga service requires several components, as follows: