A Diskless Kafka® on S3, Offering 10x Cost Savings and Scaling in Seconds.

📑 Documentation 🔥 Free trial of AutoMQ on AWS

Prerequisites

Before running AutoMQ locally, please ensure:

Docker version 20.x or later
Docker Compose v2
At least 4 GB RAM allocated to Docker
Ports 9092 and 9000 are available on your system

[!Tip] Deploying a production-ready AutoMQ cluster is challenging. This Quick Start is only for evaluating AutoMQ features and is not suitable for production use. For production deployment best practices, please contact our community for support.

The docker/docker-compose.yaml file provides a simple single-node setup for quick evaluation and development:

curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.5/docker/docker-compose.yaml && docker compose -f docker-compose.yaml up -d

This setup features a single AutoMQ node serving as both controller and broker, alongside MinIO for S3 storage. All services operate within a Docker bridge network called automq_net, allowing you to start a Kafka producer in this network to test AutoMQ:

docker run --network automq_net automqinc/automq:latest /bin/bash -c \
"/opt/automq/kafka/bin/kafka-producer-perf-test.sh --topic test-topic --num-records=1024000 --throughput 5120 --record-size 1024 \
--producer-props bootstrap.servers=server1:9092 linger.ms=100 batch.size=524288 buffer.memory=134217728 max.request.size=67108864"

After testing, you can destroy the setup with:

docker compose -f docker-compose.yaml down

The docker/docker-compose-cluster.yaml file offers a more complex setup with three AutoMQ nodes, ideal for testing AutoMQ’s cluster features, and can be run in the same way.

There are more deployment options available:

🗞️ Newest Feature - Table Topic

Table Topic is a new feature in AutoMQ that combines stream and table functionalities to unify streaming and data analysis. Currently, it supports Apache Iceberg and integrates with catalog services such as AWS Glue, HMS, and the Rest catalog. Additionally, it natively supports S3 tables, a new AWS product announced at the 2024 re:Invent. Learn more.

🔶 Why AutoMQ

AutoMQ is a stateless Kafka alternative that runs on S3 or any S3-compatible storage, such as MinIO. It is designed to address two major issues of Apache Kafka. First, Kafka clusters are difficult to scale out or in due to the stateful nature of its brokers. Data movement is required, and even reassigning partitions between brokers is a complex process. Second, hosting Kafka in the cloud can be prohibitively expensive. You face high costs for EBS storage, cross-AZ traffic, and significant over-provisioning due to Kafka’s limited scalability.

Here are some key highlights of AutoMQ that make it an ideal choice to replace your Apache Kafka cluster, whether in the cloud or on-premise, as long as you have S3-compatible storage:

Cost effective: The first true cloud-native streaming storage system, designed for optimal cost and efficiency on the cloud. Refer to this report to see how we cut Apache Kafka billing by 90% on the cloud.
High Reliability: Leverage object storage service to achieve zero RPO, RTO in seconds and 99.999999999% durability.
Zero Cross-AZ Traffic: By using cloud object storage as the priority storage solution, AutoMQ eliminates cross-AZ traffic costs on AWS and GCP. In traditional Kafka setups, over 80% of costs arise from cross-AZ traffic, including producer, consumer, and replication sides.
Serverless:
- Auto Scaling: Monitor cluster metrics and automatically scale in/out to align with your workload, enabling a pay-as-you-go model.
- Scaling in seconds: The computing layer (broker) is stateless and can scale in/out within seconds, making AutoMQ a truly serverless solution.
- Infinite scalable: Utilize cloud object storage as the primary storage solution, eliminating concerns about storage capacity.
Manage-less: The built-in auto-balancer component automatically schedules partitions and network traffic between brokers, eliminating manual partition reassignment.
High performance:
- High throughput: Leverage pre-fetching, batch processing, and parallel technologies to maximize the capabilities of cloud object storage. Refer to the AutoMQ Performance White Paper to see how we achieve this.
- Low Latency: AutoMQ defaults to running on S3 directly, resulting in hundreds of milliseconds of latency. The enterprise version offers single-digit millisecond latency. Contact us for more details.
Built-in Metrics Export: Natively export Prometheus and OpenTelemetry metrics, supporting both push and pull. Ditch inefficient JMX and monitor your cluster with modern tools. Refer to full metrics list provided by AutoMQ.
100% Kafka Compatible: Fully compatible with Apache Kafka, offering all features with greater cost-effectiveness and operational efficiency.

✨Architecture

AutoMQ is a fork of the open-source Apache Kafka. We’ve introduced a new storage engine based on object storage, transforming the classic shared-nothing architecture into a shared storage architecture.

Regarding the architecture of AutoMQ, it is fundamentally different from Kafka. The core difference lies in the storage layer of Apache Kafka and how we leverage object storage to achieve a stateless broker architecture. AutoMQ consists of below key components:

S3 Storage Adapter: an adapter layer that reimplements the UnifiedLog, LocalLog, and LogSegment classes to create logs on S3 instead of a local disk. Traditional local disk storage is still supported if desired.
S3Stream: a shared streaming storage library that encapsulates various storage modules, including WAL and object storage. WAL is a write-ahead log optimized for frequent writes and low IOPS to reduce S3 API costs. To boost read performance, we use LogCache and BlockCache for improved efficiency.
Auto Balancer: a component that automatically balances traffic and partitions between brokers, eliminating the need for manual reassignment. Unlike Kafka, this built-in feature removes the need for cruise control.
Rack-aware Router: Kafka has long faced cross-AZ traffic fees on AWS and GCP. Our shared storage architecture addresses this by using a rack-aware router to provide clients in different AZs with specific partition metadata, avoiding cross-AZ fees while exchanging data through object storage.

For more on AutoMQ’s architecture, visit AutoMQ Architecture or explore the source code directly.

🌟 Stay Ahead

Star AutoMQ on GitHub for instant updates on new releases. star-automq

💬 Community

You can join the following groups or channels to discuss or ask questions about AutoMQ:

Ask questions or report a bug by GitHub Issues
Discuss about AutoMQ or Kafka by Slack or Wechat Group

👥 How to contribute

If you’ve found a problem with AutoMQ, please open a GitHub Issues. To contribute to AutoMQ please see Code of Conduct and Contributing Guide. We have a list of good first issues that help you to get started, gain experience, and get familiar with our contribution process.

👍 AutoMQ Enterprise Edition

The enterprise edition of AutoMQ offers a robust, user-friendly control plane for seamless cluster management, with enhanced availability and observability over the open-source version. Additionally, we offer Kafka Linking for zero-downtime migration from any Kafka-compatible cluster to AutoMQ.

📜 License

AutoMQ is under the Apache 2.0 license. See the LICENSE file for details.

📝 Trademarks

Apache®, Apache Kafka®, Kafka®, Apache Iceberg®, Iceberg® and associated open source project names are trademarks of the Apache Software Foundation