Unified Stream and Batch Processing with Apache Flink
10 likes•4,644 views
The document discusses Apache Flink, an open source stream processing framework. It provides high throughput and low latency processing of both streaming and batch data. Flink allows for explicit handling of event time, stateful stream processing with exactly-once semantics, and high performance. It also supports features like windowing, sessionization, and complex event processing that are useful for building streaming applications.
Unified Stream and Batch Processing with Apache Flink
- 1. Ufuk Celebi Hadoop Summit Dublin April 13, 2016 Unified Stream & Batch Processing with Apache Flink
- 2. What is Apache Flink? 2 Apache Flink is an open source stream processing framework. • Event Time Handling • State & Fault Tolerance • Low Latency • High Throughput Developed at the Apache Software Foundation.
- 3. Recent History 3 April ‘14 December ‘14 v0.5 v0.6 v0.7 March ‘16 Project Incubation Top Level Project v0.8 v0.10 Release 1.0
- 4. Flink Stack 4 DataStream API Stream Processing DataSet API Batch Processing Runtime Distributed Streaming Data Flow Libraries Streaming and batch as first class citizens.
- 5. Today 5 DataStream API Stream Processing DataSet API Batch Processing Runtime Distributed Streaming Data Flow Libraries Streaming and batch as first class citizens.
- 6. Counting 6 Seemingly simple application: Count visitors, ad impressions, etc. But generalizes well to other problems.
- 7. Batch Processing 7 All Input Batch Job All Output Hadoop, Spark, Flink
- 9. Continuous Counting 9 Time 1h Job 1 Continuous ingestion Periodic files Periodic batch jobs
- 10. Continuous Counting 9 Time 1h Job 1 Continuous ingestion Periodic files Periodic batch jobs 1h Job 2
- 11. Continuous Counting 9 Time 1h Job 1 Continuous ingestion Periodic files Periodic batch jobs 1h Job 2 1h Job 3
- 12. Many Moving Parts 10 Batch Job 1h Serving Layer
- 13. Many Moving Parts 10 Batch Job 1h Serving Layer Data loading into HDFS (e.g. Flume)
- 14. Many Moving Parts 10 Batch Job 1h Serving Layer Periodic job scheduler (e.g. Oozie)
- 15. Many Moving Parts 10 Batch Job 1h Serving Layer Batch processor (e.g. Hadoop, Spark, Flink)
- 16. High Latency 11 Latency from event to serving layer usually in the range of hours. Batch Job 1h Serving Layer Schedule every X hours
- 17. Implicit Treatment of Time 12 Time is treated outside of your application. Batch Job 1h Serving Layer
- 18. Implicit Treatment of Time 12 Time is treated outside of your application. Batch Job 1h Serving LayerBatch Job 1h
- 19. Implicit Treatment of Time 12 Time is treated outside of your application. Batch Job 1h Serving LayerBatch Job 1h Batch Job 1h
- 20. Implicit Treatment of Time 13 DataSet<ColorEvent> counts = env .readFile("MM-dd.csv") .groupBy("color") .count(); Time is implicit in input file
- 21. Batch Job Serving Layer Continuously produced Files are finite streams Periodically executed Streaming over Batch 14
- 22. Streaming 15 Until now, stream processors were less mature than their batch counterparts. This led to: • in-house solutions, • abuse of batch processors, • Lambda architectures This is no longer needed with new generation stream processors like Flink.
- 23. Streaming All the Way 16 Streaming Job Serving Layer
- 24. Streaming All the Way 16 Streaming Job Serving Layer Message Queue (e.g. Apache Kafka) Durability and Replay
- 25. Streaming All the Way 16 Streaming Job Serving Layer Stream Processor (e.g. Apache Flink) Consistent Processing
- 26. Building Blocks of Flink 17 Explicit Handling of Time State & Fault Tolerance Performance
- 27. Building Blocks of Flink 17 Explicit Handling of Time State & Fault Tolerance Performance Explicit Handling of Time
- 28. Windowing 18 Time Aggregates on streams are scoped by windows Time-driven Data-driven e.g. last X minutes e.g. last X records
- 29. Tumbling Windows (No Overlap) 19 Time e.g.“Count over the last 5 minutes”, “Average over the last 100 records”
- 30. Tumbling Windows (No Overlap) 19 Time e.g.“Count over the last 5 minutes”, “Average over the last 100 records”
- 31. Tumbling Windows (No Overlap) 19 Time e.g.“Count over the last 5 minutes”, “Average over the last 100 records”
- 32. Tumbling Windows (No Overlap) 19 Time e.g.“Count over the last 5 minutes”, “Average over the last 100 records”
- 33. Tumbling Windows (No Overlap) 19 Time e.g.“Count over the last 5 minutes”, “Average over the last 100 records”
- 34. Sliding Windows (with Overlap) 20 Time e.g. “Count over the last 5 minutes, updated each minute.”, “Average over the last 100 elements, updated every 10 elements”
- 35. Sliding Windows (with Overlap) 20 Time e.g. “Count over the last 5 minutes, updated each minute.”, “Average over the last 100 elements, updated every 10 elements”
- 36. Sliding Windows (with Overlap) 20 Time e.g. “Count over the last 5 minutes, updated each minute.”, “Average over the last 100 elements, updated every 10 elements”
- 37. Sliding Windows (with Overlap) 20 Time e.g. “Count over the last 5 minutes, updated each minute.”, “Average over the last 100 elements, updated every 10 elements”
- 38. Sliding Windows (with Overlap) 20 Time e.g. “Count over the last 5 minutes, updated each minute.”, “Average over the last 100 elements, updated every 10 elements”
- 39. Explicit Handling of Time 21 DataStream<ColorEvent> counts = env .addSource(new KafkaConsumer(…)) .keyBy("color") .timeWindow(Time.minutes(60)) .apply(new CountPerWindow()); Time is explicit in your program
- 40. Session Windows 22 Time Sessions close after period of inactivity. Inactivity Inactivity e.g. “Count activity from login until time-out or logout.”
- 42. Notions of Time 24 12:23 am Event Time Time when event happened.
- 43. Notions of Time 24 12:23 am Event Time 1:37 pm Processing Time Time measured by system clock Time when event happened.
- 44. 1977 1980 1983 1999 2002 2005 2015 Processing Time Episode IV Episode V Episode VI Episode I Episode II Episode III Episode VII Event Time Out of Order Events 25
- 45. Out of Order Events 26 1st burst of events 2nd burst of events Event Time Windows Processing Time Windows
- 47. Explicit Handling of Time 28 1. Expressive windowing 2. Accurate results for out of order data 3. Deterministic results
- 48. Building Blocks of Flink 29 Explicit Handling of Time State & Fault Tolerance Performance
- 49. Stateful Streaming 30 Stateless Stream Processing Stateful Stream Processing Op Op State
- 50. Processing Semantics 31 At-least once May over-count after failure Exactly Once Correct counts after failures End-to-end exactly once Correct counts in external system (e.g. DB, file system) after failure
- 51. Processing Semantics 32 • Flink guarantees exactly once (can be configured for at-least once if desired) • End-to-end exactly once with specific sources and sinks (e.g. Kafka -> Flink -> HDFS) • Internally, Flink periodically takes consistent snapshots of the state without ever stopping computation
- 52. Building Blocks of Flink 33 Explicit Handling of Time State & Fault Tolerance Performance
- 53. Yahoo! Benchmark 34 • Storm 0.10, Spark Streaming 1.5, and Flink 0.10 benchmark by Storm team at Yahoo! • Focus on measuring end-to-end latency at low throughputs (~ 200k events/sec) • First benchmark modelled after a real application https://yahooeng.tumblr.com/post/135321837876/ benchmarking-streaming-computation-engines-at
- 54. Yahoo! Benchmark 35 • Count ad impressions grouped by campaign • Compute aggregates over last 10 seconds • Make aggregates available for queries in Redis
- 55. 99th Percentile Latency (sec) 9 8 2 1 Storm 0.10 Flink 0.10 60 80 100 120 140 160 180 Throughput (1000 events/sec) Storm and Flink at low latencies Latency (Lower is Better) 36
- 56. 99th Percentile Latency (sec) 9 8 2 1 Storm 0.10 Flink 0.10 60 80 100 120 140 160 180 Throughput (1000 events/sec) Spark Streaming 1.5 Spark latency increases with throughput Storm and Flink at low latencies Latency (Lower is Better) 36
- 57. Extending the Benchmark 37 • Great starting point, but benchmark stops at low write throughput and programs are not fault-tolerant • Extend benchmark to high volumes and Flink’s built-in fault-tolerant state http://data-artisans.com/extending-the-yahoo-streaming- benchmark/
- 58. Extending the Benchmark 38 Use Flink’s internal state
- 59. Throughput (Higher is Better) 39 5.000.000 10.000.000 15.000.000 Maximum Throughput (events/sec) 0 Flink w/ Kafka Storm w/ Kafka
- 60. Throughput (Higher is Better) 39 5.000.000 10.000.000 15.000.000 Maximum Throughput (events/sec) 0 Flink w/ Kafka Storm w/ Kafka Limited by bandwidth between Kafka and Flink cluster
- 61. Throughput (Higher is Better) 39 5.000.000 10.000.000 15.000.000 Maximum Throughput (events/sec) 0 Flink w/o Kafka Flink w/ Kafka Storm w/ Kafka Limited by bandwidth between Kafka and Flink cluster
- 62. Summary 40 • Stream processing is gaining momentum, the right paradigm for continuous data applications • Choice of framework is crucial – even seemingly simple applications become complex at scale and in production • Flink offers unique combination of efficiency, consistency and event time handling
- 63. Libraries 41 DataStream API Stream Processing DataSet API Batch Processing Runtime Distributed Streaming Data Flow Libraries Complex Event Processing (CEP), ML, Graphs
- 64. 42 Pattern<MonitoringEvent, ?> warningPattern = Pattern.<MonitoringEvent>begin("First Event") .subtype(TemperatureEvent.class) .where(evt -> evt.getTemperature() >= THRESHOLD) .next("Second Event") .subtype(TemperatureEvent.class) .where(evt -> evt.getTemperature() >= THRESHOLD) .within(Time.seconds(10)); Complex Event Processing (CEP)
- 65. Upcoming Features 43 • SQL: ongoing work in collaboration with Apache Calcite • Dynamic Scaling: adapt resources to stream volume, scale up for historical stream processing • Queryable State: query the state inside the stream processor
- 68. Queryable State 46 Query Flink directly
- 69. Join the Community 47 Read http://flink.apache.org/blog http://data-artisans.com/blog Follow @ApacheFlink @dataArtisans Subscribe (news | user | dev)@flink.apache.org