| Welcome to the InfoQ Software Architects' Newsletter! Each month, we bring you essential news and experience from industry peers on emerging patterns and technologies. This month, we focus on "event-driven architecture." Technologies, patterns, and practices from this topic span the entire "diffusion of innovation" graph in our 2023 Software Architecture and Design InfoQ Trends Report. We see event-driven architecture (EDA) being adopted by the late majority and a related topic, "correctly built distributed systems," being adopted by the early majority. Key challenges remain in this space, including “designing for resilience” and incorporating “workflow and decision automation platforms” into IT systems. We believe both topics are currently at the early adoption stage. News Datadog Creates Scalable Data Ingestion Architecture Datadog created a dedicated data ingestion architecture offering exactly-once semantics for their third-generation event store, Husky. The event-driven architecture (EDA) can accommodate bursts in traffic in the multi-tenant platform with reasonable ingestion latency and acceptable operational costs. The architecture of Husky separated data ingestion, data compaction, and data reading workloads, which allows them to be scaled independently. All three workloads leverage a shared metadata store built on FoundationDB and a blob storage service that uses AWS S3. The data ingestion workload uses Apache Kafka to deliver events into the storage platform and route them internally to data writers. Apache Pulsar 3.0 Delivers a New LTS Version and Efficiency Improvements The Apache Software Foundation has released version 3.0 (LTS) of Apache Pulsar, the distributed messaging and streaming platform. Pulsar 3.0 introduces the Long-Term Support (LTS) release. Apache Pulsar is a high-performance, multi-tenant messaging and streaming platform with support for multiple clusters, low latency, seamless scalability, guaranteed message delivery with persistent message storage provided by Apache BookKeeper, and serverless connector frameworks for data processing and connectivity. The Guardian Optimizes Mobile Push-Notification Delivery Architecture The technology team at the Guardian has used event-driven architecture to make mobile push notifications faster to improve readers’ experiences. The original architecture, optimized for concurrency, has been suffering from delays in notification delivery. The engineers used improved observability to make significant gains through experimentation. Guardian readers can use the mobile app to access content and can register to receive breaking news alerts via push notifications. The event-driven architecture (EDA) behind that has been operating since 2009, and over time, notification delivery speeds have decreased, taking more than five minutes for some users. Momento Topics: A Serverless Event Messaging System for Pub-Sub Momento, a Cache-as-a-Service platform, now offers Momento Topics, a serverless event messaging system that supports publish-subscribe communication patterns. This service is designed to provide a messaging pipeline for event-driven architectures. Subsequent feature releases will allow direct AWS Lambda invocations and change data capture events triggered from Momento Cache. Sponsored | | In the latest episode of the Big Ideas in App Architecture podcast, Chick-fil-A's Brian Chambers shares the fast food giant's infrastructure recipe for a perfect customer experience.
Listen and subscribe today! Sponsored by Cockroach Labs |
|
Case Study Tales of Kafka at Cloudflare: Lessons Learned on the Way to 1 Trillion Messages In this summary of a related QCon London talk, Matt Boyle and Andrea Medda discuss how Kafka clusters are used at Cloudflare to process large amounts of data, with a general-purpose message bus cluster developed to decouple teams, scale effectively, and process trillions of messages. There are four lessons to be learned: - Always find the right balance between flexibility and simplicity: while a configurable setup may offer more flexibility, a simpler one allows standardization across different pipelines.
- Visibility: adding metrics to the SDK as soon as possible can help teams understand how the system behaves and make better decisions, especially during incidents.
- Contracts: enforcing one strong, strict contract gives great visibility into what is happening inside a topic, allowing one to know who is writing and reading from it.
- Document the good work you do so that you don’t have to spend time answering questions or helping people debug production issues. This can be achieved through channels like Google Chat and wikis.
By following these rules, Boyle and Medda could improve their systems and make their customers happy, even in high-stress situations. This content is an excerpt from a recent InfoQ article by Matt Boyle and Andrea Medda, "Tales of Kafka at Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages". To get notifications when InfoQ publishes content on these topics, follow "Event-Driven Architecture", "Distributed Systems", and "Asynchronous Programming" on InfoQ. Missed a newsletter? You can find all of the previous issues on InfoQ. Upcoming events QCon: For practitioners, by practitioners
Introducing new 1-day tickets for teams of 3+ attending QCon San Francisco (Oct 2-6).
Divide the conference days between team members to provide a flexible learning path for individuals to bring back collective new ideas to guide your decision-making. This tailored option allows focused learning, less time away, and optimizes development budgets.
Our special launch price of just $990 is valid until August 2nd.
Senior software developers rely on the InfoQ community to keep ahead of the adoption curve. One of the main reasons software architects and engineers tell us they keep coming back to InfoQ is because they trust the information provided and selected by their peers. We’ve been helping software development teams adopt new technologies and practices for over 15 years through InfoQ articles, news items, podcasts, tech talks, trends reports, and QCon software development conferences. We hope you find this newsletter useful. If not, you can unsubscribe using the link below. Forwarded email? Subscribe and get your own copy. | 
