Distributed Applications with RabbitMQ and Redis - Part 1

We recently completed a rewrite of a client's application which handled generation and delivery of statements. With 6 months of hindsight, we had the opportunity to do this properly.

In this series, we will be talking about some issues we faced and how we solved them.

Choosing our Technology Stack

The entire process is broken down into the following steps:
  1. Create a job defining batches of companies to process.
  2. Each batch contains a list of items representing each customer in that company.
  3. Each item is responsible for generating the customer's statement, uploading it to S3 and sending the customer an email.
  4. When items in a batch are complete, the customers' statements are combined into slices of 100, uploaded and an email sent to the company's representative.
  5. When all batches are done, the job is marked as complete.

The Foolishness of Youth

Our initial solution used Hangfire to handle our queues and background processing. It seemed like a good idea at the time.
After all, it was an easy way to "an easy way to perform background processing in .NET" with support for continuations, batches (pro) and batch continuations.
We promptly whipped up a proof of concept and when we were satisfied, purchased the professional license.
When the time came for it to handle the entire process, it failed abysmally.

We quickly came to understand that while Hangfire is not a bad tool by any means, we simply were using it for something it wasn't designed to do. It excels at being a quick and easy way to handle background processing such as sending emails in your website and some other stuff with medium to moderately heavy load. But what were were trying to do just simply wouldn't work.

The Wisdom of Age

So this time, after careful consideration, proper testing and research, we decided to go with Masstransit backed by and a proper and robust queuing system RabbitMQ.
Our choice of masstransit was heavily influenced by our love of opensource as well as it being built primarily around rabbitmq.

We also decided to replace our backing store as we found that using an RDBMS was a bottleneck when increasing the number of workers.
We went with Redis which is possibly the best decision we could have made.

In the next part of this series, we will look at how we structured our application.


Popular posts from this blog

Demystifying pglogical - Tutorial

Using an Integer as the Primary Key in ASP.Net Core Identity