Modern applications are very complex. They have to meet the demands of platforms’ holders, and customers’ expectations. They are also time and resource-consuming. And even more than that; they gather and process data, communicate across different services, share resources. It’s complex, it’s complicated, it’s not easy to optimize. Some of these challenges can be solved with a solution called a message broker. One of them is called “Amazon MQ” and it’s designed to simplify the life of AWS developers in companies like ours.
What is a message broker, anyway?
A message broker is a software dedicated to communication and information exchange between systems, applications, and services. Software translates messages between messaging protocols. This solution allows services to talk with each other; even if they are written in different languages and sit on different platforms.
We can think of it as an electronic translator from the “Star Trek” franchise. Different races talk to each other and understand other species on the fly, even if they are not familiar with one another.
Message brokers can store, route, and deliver messages to different recipients and addresses. Even without the information where the receivers are, if they are active or not, and how many exactly are there. This allows the decoupling of processes and services.
But there is a message queue and informations are waiting in line in the order in which they are transmitted. They wait until the delivery is completed.
Two message distribution patterns of message brokers
There are two models of delivering information by message brokers.
Point-to-point messaging: Here the communication is very straightforward and happens only once. It’s used when a given action needs to happen only once. A good example would be a payment confirmation or a payroll sent to an employee. The system needs confirmation that the money had been processed and the amount landed on an account.
Publish or subscribe messaging: In this pattern, the sender has no information about receivers. All messages published to a topic are distributed to all the applications subscribed to it. This is a broadcast-style distribution method, in which there is a one-to-many relationship between the message’s publisher and its consumers.
Advantages of using message brokers
There are some clear benefits of using message brokers.
Communication between services that are not necessarily running at the same time. The message can be sent regardless of whether the recipient is active or not.
Improved system performance through asynchronous processing. Here complex tasks can be broken down and distributed to separate processes. This will accelerate the performance of the application and optimize user experience.
Increased reliability through the assurance of delivery. Each message broker solution has sewn in a redelivering mechanism. If the recipient fails for some reason, the message can be resent. Right away or after the defined time. Message brokers also have a routing mechanism for messages that have been facing a constant dead end. It has been called a dead-letter mechanism.
Disadvantages of using message brokers
Not everything is shiny. Since this solution uses asynchronous processing, it faces challenges similar to asynchronous calls.
Steep beginning. There’s a learning curve to this, mainly related to setup and configuration. Size and contents of the messages, queues, their behavior, delivery… There are lots of options and every single one should be defined and tested.
System complexity. By adding a message broker, you are increasing the complexity level of architecture. That’s why you need to keep tabs on the reliability of communication between components and security. Additionally, some elements of the system may not yet be updated, ruining the flow of the messages.
Debugging can be problematic. Since there are many moving parts, something can go wrong (like not receiving confirmation for a received message). Because every element keeps its own log file, you have to monitor all of them to dig into the root of the problem. A good practice would be to implement a tracing system, in accordance with the already existing messaging highway.
Use cases for message brokers
Implementing message brokers can solve many burning business, technical, and even operational problems. They are introduced in these, exemplary cases:
Financial transactions, payment processing, money laundering tracking. Being sure the payment is sent only once, is only a part of it. Making sure where the money is going and, the delivery notice is not lost or duplicated, is equally important.
Data protection. There are basically two states of sensitive data: at rest or in transit. You want to make sure that both of these states have safety mechanisms. Solutions with end-to-end encryption capabilities are crucial.
E-commerce operations. Order processing, fulfillment, and payments are the backbone of selling online. You want to make sure everything runs smoothly. A brand’s image can be hurt very easily and user comments can derail your sales and marketing efforts.
Usage of microservices. Large and complex systems may run on many microservices at the time. Growing numbers can be difficult to track, and maintain. Plus, if a microservice goes down, it becomes unavailable and you’re faced with error messaging. You can put a message broker to work and go with event-based communication.
Web and mobile apps. Especially mobile apps, because they are heavy users of push notifications. Imagine a pretty neat mobile development feature, where mobile devices subscribe to the same message broker’s feed. For example, many phones at a given location want to know about promotion in the coffee shop around the corner. There you go…
Message brokers under a cloud architecture
Scalability, flexibility, accelerated development. These are perks of applications running in the Amazon Cloud environment. Microservices are applications that run independently. Meaning, that they can be deployed, run, scaled, and fixed regardless of system performance and other applications that are present around them. This offers unparalleled options for performance and optimization.
In the cloud, message brokers are used not only for messaging but also interconnection between multiple systems, even those that are present in different cloud environments. Also, message brokers are great for serverless computing, where services hosted by a cloud run on demand.
How to implement message brokers under AWS?
As says Karol Piątkowski, our Senior Fullstack Developer:
Amazon Simple Queue Service is definitely simple to learn but hard to master. In the following example, we will focus on the most desired use-case which is event stream processing.
The overall goal is pretty straightforward – we need to process events that come from various sources. Simple as that. But things get more complicated as we add more requirements, after all this needs to be reliable. This solution needs to take the order of events into consideration, it should also be able to handle a burst of events as fast as possible and most importantly it cannot lose a single event.
To go over through process we will show snippets of AWS CDK and TypeScript for AWS Lambda handlers.
As a starting point, we need two queues for that, one will be designated as a dead letter queue. This is a place where messages that fail to process will end up after three attempts. That number is usually enough – if data in a message is corrupted or there is a bug in code it will fail with each retry and setting this number higher will only cause unneeded runtime, but sometimes fails can be caused by other intermittent issues like network errors and having a retry usually helps with that. Three is a pretty good number, but as with all things, this can be configured in queue settings. Take note of visibility timeout too, this is a period of time during which Amazon SQS prevents other consumers from receiving and processing the message. What that means is once a message is delivered there is some time to finish processing and return the correct status or delete the message from the queue, otherwise, it will be delivered again.
```ts
const app = new cdk.App();
const deadLetterQueue = new aws_sqs.Queue(app, 'deadLetterQueue', {
contentBasedDeduplication: true,
encryption: aws_sqs.QueueEncryption.KMS,
fifo: true,
removalPolicy: RemovalPolicy.DESTROY,
});
const queue = new aws_sqs.Queue(app, 'queue', {
contentBasedDeduplication: true,
encryption: aws_sqs.QueueEncryption.KMS,
deadLetterQueue: {
maxReceiveCount: 3,
queue: deadLetterQueue
},
fifo: true,
removalPolicy: RemovalPolicy.DESTROY,
visibilityTimeout: Duration.minutes(1),
});
```
We have two queues ready, now let’s add lambda that will consume the messages. The most important part of that is reporting about batch item failures. To talk about this setting we need to take a step back and look at how messages are delivered and marked as consumed.
Amazon SQS can batch messages (and by default, it will batch up to 10 messages in a window of up to 5 minutes) so we need to plan around that as we want to leverage batching to increase efficiency and reduce costs. Let’s imagine a scenario where we got 10 messages in a batch (but any number greater than 1 will do). When all messages are processed without errors, which is our happy path, lambda returns without error so Amazon SQS knows that batch was processed and removes the messages from the queue. But what will happen if the first message fails? Lambda returns with an error code so Amazon SQS will try to redeliver this batch once the visibility timeout wears out. Both of those scenarios are ok, but what if a failed message is, for example 5th? Lambda will still return with an error code, which will tell Amazon SQS to redeliver the batch again (up to our retry limit), but the first four messages were processed and will be processed again. The worst case scenario is when a failed message is the last one in a queue as this will result in 9 messages being processed 3 times. This is not what we want at all and reporting batch item failures is here to help us with that.
```ts
const processingLambda = new aws_lambda_nodejs.NodejsFunction(app, "processingLambda", {
entry: path.join(__dirname, "./processingLambdaHandler.ts"),
handler: "handler",
memorySize: 128,
runtime: aws_lambda.Runtime.NODEJS_18_X,
timeout: Duration.minutes(3),
});
const eventSource = new aws_lambda_event_sources.SqsEventSource(queue, {
reportBatchItemFailures: true,
});
processingLambda.addEventSource(eventSource);
In the following implementation, we are processing all messages from a batch up until one of it will throw an exception. At this point we are stopping processing of the batch and returning failed items. You might ask why we don’t note which message failed and move onto next ones and only after the loop is done return failed items. That’s a great question I would say! The answer lies in our requirements – we need to make sure the messages are processed in the correct order, as subsequent messages might rely on the outcome of previous ones.
```ts
export const handler: SQSHandler = async (sqsBatch): Promise<SQSBatchResponse> => {
for (let idx = 0; idx < sqsBatch.Records.length; idx++) {
const sqsRecord = sqsBatch.Records[idx];
try {
await processSQSRecord(sqsRecord); // this is a placeholder function in which you put all your logic
} catch (error) {
const failedItems = {
batchItemFailures: sqsBatch.Records.slice(idx).map(
(record) => ({
itemIdentifier: record.messageId,
})
),
};
return failedItems;
}
}
return {
batchItemFailures: [],
};
};
```
What happens if the message fails 3 times? It will be moved to the dead letter queue (you will want to have set some CloudWatch Alarms on that). Messages that will end up there will most likely require manual attention to resolve the issue and then can be published to the main queue again and removed from the dead letter queue using redrive feature.
Because we are using a FIFO queue there is also room to improve as Amazon SQS messages can be grouped using some kind of identifier and there is many benefits of using them. Amazon SQS will still make sure that messages in the same group are processed in the correct order, but messages with different group can be sent in different batch to another lambda instance thus allowing for parallel processing whenever possible. You can even rebuild the processing lambda to group incoming messages from batch by group, process them in parallel using Promise.all, etc.
Key examples of message brokers
Let’s start with Amazon-based services since Code & Pepper is an AWS consulting company.
Amazon MQ. This is a message broker provided by Amazon. It’s a version of RabbitMQ and Apache ActiveMQ. The company recommends migration from the existing message brokers that rely on compatibility with APIs such as JMS or protocols. Amazon lists AMQP 0-9-1, AMQP 1.0, MQTT, OpenWire, and STOMP as examples.
Amazon SQS. This is a managed queuing service by Amazon. It has something called a pull mechanism – the receivers have to pull messages from SQS queues by themselves.
Amazon SNS. A push notification service by Amazon. Just like SQS, it’s a part of AWS services. It can be run within the publish-subscribe pattern.
Rabbit MQ. It’s easy to deploy on cloud environments, on-premises, and locally. It offers four types of exchanges. Here messages are sent to exchanges, not to queues.
Redis. It’s an open-source message broker. It has three major features: streams, pub/sub, and lists. It also offers transaction and pipeline capabilities.
What’s the difference between message brokers and APIs?
REST APIs are used for communication between microservices. There is a set of principles and rules that developers follow to build web services. The services that have foundations compatible with these rules are able to communicate through shared operators and requests. Application Programming Interface (API) underlines code, so that Representational State Transfer (REST) rules allow services to talk to each other.
Message brokers, on the other hand, enable asynchronous communication between services. The sending service doesn’t have to wait for the receiving service’s reply. That equals better fault tolerance and resiliency among places where message brokers are deployed.
Summary
When going with message brokers, pay attention to a few factors. Reliability, tool popularity, message semantics, and the solution’s track record. And to specialists, who implement them. With centaur developers (programmers equipped with AI tools) you are on the path to your application’s success.
We have the formula to hire only the best talents out there. With it, you are on the fast track to market readiness. We use microservices and message brokers to speed up development, and code written by fast engineers who take care of the application’s performance.
