Why can Apache Flink be considered the best choice for processing data streaming?

Dawid Wysakowicz [DW]: One reason is that it addresses all streaming use cases: bounded stream – aka batch, streaming analytics, event-driven applications, etc. It also has the best of class support for state management and event time. It is also industry proven as it runs at Netflix-scale.

In contrast to other open-source stream processors, Flink provides not only true stream processing at the event granularity, i.e., no micro batching, but also handles many batch use cases very well.

What are the main directions of Flink development?

DW: Right now Flink community focuses on true unification of batch and streaming, both in the sense of query semantics, but also on the runtime level as well. That means it will be possible to perform operations like, e.g., bounded-to-unbounded stream join, enrichment with static data, etc. in an optimized way. Later on, this will expand to more use cases such as machine learning on data streams.

Another area of focus is to expand the scope and features of SQL support.  We plan to fully support TPC-DS in the nearest future but also further define industry-wide standards for Streaming SQL. Moreover, you can expect significant performance improvements in the future.

What are you working on right now as a Flink committer?

DW: Recently I was driving the effort of supporting the MATCH_RECOGNIZE clause that allows querying for row sequence patterns with SQL, which I’m giving a talk about. Right now I am involved in the efforts of further improving SQL support.

How did you join the community and started contributing to Apache Flink?

DW: I started contributing because I wanted to improve my coding skills and found Apache Flink’s community the most welcoming one among open-source projects I looked into. I was also lucky enough that I met people that allowed me to work on this project as part of my job, first at GetInData and now at Ververica.

Contributing to open-source projects is something I would recommend to everybody as it a great way to meet and work with super intelligent people on extremely interesting problems.