True-Serverless Event Infrastructure
There is no denying that the new Apple M1 MacBooks are impressive. I wouldn’t call myself a fanboy, but those machines are great—the processors perform very well in many areas.
Of course, there’s a “but”: not everything is ready for a seamless transition from Intel to ARM.
As a heavy Docker user, I learned the hard way that running Kafka on my local Docker instance wasn’t straightforward on ARM. Searching for Kafka Docker images with ARM support left me empty-handed.
Take a look at these GitHub issues and you ‘ll see what I mean:
Maybe there’s support by the time this article is published—if so, please let me know in the comments.
Time to look for other solutions.
I had never tried a cloud-based Kafka solution much before; my projects usually relied on local or on-premises setups. I’m also wary of cloud providers that require purchasing a quota up front—when a project ends, that money often just sits unused.
Three Reasons for Using Upstash
I wanted an on-demand, pay-as-you-go solution without managing hardware, VMs, or Docker containers.
Second, it shouldn’t cost anything when not in use— it should scale to zero, meaning nothing to pay at all.
Third, it should be free to try for low-traffic side projects and learning purposes.
The solution is Upstash, a true serverless offering for Kafka and Redis.
See their post “AWS MSK and Confluent. Are they really Serverless?” for an in-depth comparison of cloud providers and cost characteristics.
Creating a Cluster and a Topic
After creating an account (no credit card required), go to the Kafka section and click “Create Cluster”.
Enter a name, choose the region, and select single- or multi-zone.
After the cluster is created, create your first topic.
Don’t worry about the options—use the defaults and adjust later as needed.
Once done, you can run a quick test to confirm the cluster works.
Testing the Cluster
Upstash provides ready-made examples you can copy and paste.
The values for bootstrap.servers and sasl.jaas.config are generated dynamically for your cluster. Don’t copy the configuration below—it belonged to my tutorial cluster, which I deleted after writing.
You don’t need to write code for a quick smoke test. Use their REST API with curl or Postman.
The Upstash team supplies complete curl commands for producing and consuming sample messages.
So, let’s try it:
On the left, I ran the producer command and on the right the consumer. I published an event to tutorial-topic; it wasn’t my first message—the message had offset 18.
Conclusion
My brief experiment with a true serverless Kafka infrastructure was a success.
All my requirements were met and trying it out didn’t cost me anything.
I’ll continue to test the service and keep you updated.
Thank you for reading!