Calculating the Value of Pie: Real-Time Survey Analysis With Apache Kafka®

Track: Practices and other tech

Abstract

These days, it’s difficult to keep up with friends and family, even more so when important things are involved––like pie. Do you find yourself wishing that you could quickly and efficiently poll your loved ones, peers, and colleagues for their pastry preferences? Look no further. This talk explores how to create an interactive, real-time survey bot with Telegram and Apache Kafka® and how to analyze the survey responses using ksqlDB. Kafka Producers and Consumers are the main ingredients that come together to make an interactive Telegram bot that issues on-demand surveys to friends and family. Survey responses are written to Kafka in real-time where ksqlDB helps to measure, mix, and knead the data––serving up a fresh and delicious result that we can consume immediately! In this session, you’ll see the entire pipeline from start to finish, learning how to think about data and use schemas, how to configure Producers and Consumers, and how to use ksqlDB for both stateless and stateful processing. Come along to see how this recipe for real-time survey analysis comes together––and how you, too, can calculate the value of pie!

Danica Fine

Danica began her career as a software engineer in data visualization and warehousing with a business intelligence team where she served as a point-person for standards and best practices in data visualization across her company. In 2018, Danica moved to San Francisco and pivoted to backend engineering with a derivatives data team which was responsible for building and maintaining the infrastructure that processes millions of financial market data per second in near real-time. Her first project on this team involved Kafka Streams – she never looked back. Danica now works as a Developer Advocate with Confluent where she helps others get the most out of their event-driven pipelines. Outside of work, Danica is passionate about sustainability, increasing diversity in the technical community, and keeping her many houseplants alive. She can be found on Twitter, tweeting about tech, plants, and baking @TheDanicaFine.