If you are already familiar with messaging systems like Apache ActiveMQ or RabbitMQ, you might ask yourself how AWS SQS compares to your well-known solution. As I am just involved in a migration from Apache ActiveMQ to AWS SQS for a dozen of queues, I decided to write about my experiences and thoughts on that.
First of all, I claim that your current broker has a powerful concept that is so self-evident that most people don’t even notice its existence. I am talking about implicit queue creation. It means that queues are created automatically when producers or consumers use them. There is no need to explicitly create queues. Neither by an operations guy nor by a developer. Queues will just be there.
This feature really pays off when queues are created often. For example, in queue-based microservice architectures that are deployed to multiple environments. Or when queues are created at runtime, based on end-user interactions with the system.
It seems ironic but exactly this flexibility of traditional messaging systems, even reinforces their self-deprecation.
If queue creation is cheap, queue usage increases. If usage increases, broker limits will be hit earlier. And at exactly this point, AWS SQS takes over. In SQS, there is no central infrastructural unit like a broker. In SQS, the queue itself becomes the infrastructural unit. Many SQS queues equal to many independent infrastructural units. SQS queues do not affect each other. High throughput on one queue will never slow down another queue. Many messages on one queue will never block another queue. Even according to the official resource quotas, the number of messages that an Amazon SQS queue can store is unlimited. Regarding the number of SQS queues per account: there is no official limit.
These aspects emphasize the scalability of SQS. Another important characteristic is the serverless nature. Typical operational tasks such as monitoring of CPU, RAM and disks or maintaining TLS certificates are completely managed by AWS.
What are the trade offs?
In the introduction, I already described the concept of implicit queue creation which makes queue management in broker based systems lightweight. With SQS the effort of creating and editing queues depends in your general approach to provision cloud resources. Examples are the AWS Management Console, the aws cli or IaC tools like terraform.
SQS has a hard limit on the message payload size which is 256 KB. Traditional messaging systems are much more cooperative on this point.
The only realistic limit to the size of a message that can be sent or consumed is the amount of disk space you have available
https://activemq.apache.org/components/artemis/documentation/1.0.0/large-messages.html
While message brokers usually also support more advanced communication patterns such as Publish/Subscribe or request-response, SQS is limited to simple message queueing.
Conclusions
While traditional messaging systems provide an honorable degree of flexibility within their resource boundaries, SQS shines with impressive scalability by making a hard limit on the payload size and the overall feature set. If this limit is a concern, AWS suggests to support SQS queues with S3 buckets for payload offloading.
Header image source
