❗Do not miss today, May 16, at 7PM CEST/12 PM CDT our webinar "*Building high-scale backend systems with Kafka*" with Dineshkumar B, Founder at Scalescape. Learn to overcome tech debts and legacy architecture for reliable backend system scalability. Upgrading monolith to microservices is not the only way. Join our talk and discover how Kafka and event-driven architecture can be leveraged to achieve scalable systems reliably. Gain practical tips and strategies to stay ahead in the constantly evolving tech landscape. During this session, Dineshkumar will cover: • Bottlenecks and scalability issues in monolithic and microservices architecture • What is an event-driven system, and how it allows for teams and features to be independently scaled • Leveraging Kafka with best practices to handle high scale After a 30-minute talk, there’ll be a 15-minute Q&A, for which we encourage you to submit questions in advance. https://www.meetup.com/platform-engineers-atx-online/events/292875262/

duck danceWe are live with @gopher to talk about Building high-scale backend systems with Kafka

I’ve not used ActiveMQ/SQS in prod, so sharing details with limited information i’ve for 1,2. 1. The scale is different and ActiveMq is limited to messaging, but Kafka can handle much higher scale and has features like streaming, reliability/retry and more baked in. Probably https://www.confluent.io/blog/apache-kafka-vs-enterprise-service-bus-esb-friends-enemies-or-frenemies/ is a good read. 2. SQS queue seems limited than Kafka topics N:N producer/consumers architecture, SNS seems like pub sub, but it doesn’t have message retention for consumers to handle messages later. 3. seems to be an answer for the above Question 4. Interesting question, If we go into details of kafka, it’ll be clear. But kafka itself has multiple brokers with Zookeeprs and it’s a cluster, topic/pubsub can still be reliable even if some kafka brokers are down. It has replication across brokers, topic retention period, scaling brokers at runtime, etc. So it’s more complex piece than a single database. 5. It’s a group of instances (say 5 machines/instance of service with same consumer-group Id) for which kafka distributes/splits the messages from topic across without duplication https://docs.confluent.io/platform/current/clients/consumer.html 6. Keep track of the metrics and latency and each part in the chain of message consumption, so you can optimise the service processing time. Usually kafka performance’s high so it doesn’t cause high latency. Again depending on the context and with more details, we could do optimisation. 7. We can add brokers to kafka cluster, but we need to redistributed topics and do some maintenance activities. It’s documented, but difficult wrt the scale you’ve. You would need a DevOps team to manage the cluster.

@gopher Thanks for sharing