Скачать с ютуб Flink SQL: The Challenges to Build a Streaming SQL Engine в хорошем качестве

Flink SQL: The Challenges to Build a Streaming SQL Engine

Flink SQL is a powerful tool for stream processing that allows users to write SQL queries over streaming data. However, building a streaming SQL engine is not an easy task. In this session, we will explore the challenges that arise when building a modern streaming SQL engine like Flink SQL. We will discuss the following challenges and how Flink SQL resolve them: - Late Data: Handling late arrival data and guaranteeing result correctness. - Change Data Ingestion and Processing: How to ingest change data from databases in real-time and apply complex operations on the change events. - Event Ordering: Shuffle may disrupt the order of data updates and get the wrong result. - Nondeterminism: Nondeterministic functions and external system lookups may produce different results on change data and get the wrong result. - State Storage: How to effectively process infinite datasets with limited storage without losing the correctness of results. We will also show real-world examples of using Flink SQL to solve common stream processing problems. By the end of this session, you will better understand the challenges involved in building a streaming SQL engine and how to overcome them. ABOUT CONFLUENT Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io. #confluent #apachekafka #kafka