AGENDA 2019

Changes in the order of presentation might occur

8.00 - 9.00 -

Registration and welcome coffee

9.00 - 9.15

Conference opening

9.15 - 10.45

Plenary Session

Stream Processing for Business Insights at Netflix

130 million members from over 190 countries enjoy the Netflix service. This leads to trillion events and petabyte of data flowing through the Keystone stream processing infrastructure to help glean business insights and improve customer experience. The self-serve infrastructure enables the users to focus on extracting insights, and not worry about building out scalable infrastructure. I’ll share my experience building this platform with Flink, and lessons learnt.

 

Monal Daxini

Engineering Lead, Netflix

10.45 - 11.15

Coffee break

11.15 – 15.30 Simultaneous sessions

Architecture, Operations and Cloud

This track is dedicated to architects, administrators and people with DevOps skills who are interested in technologies, techniques and best practices for planning, building, installing, managing and securing their Big Data infrastructure in enterprise environments – both on-premise and the cloud.

Data Engineering

This track is the place for engineers to learn about tools, techniques and battle-proven solutions to collect, store and process large amounts of data. It covers topics like data collection and ingestion, ETL, job scheduling, metadata and schema management, distributed processing engines, distributed datastores and more.

Artificial Intelligence and Data Science

This track includes real-world case-studies demonstrating how data & technology are used together to address a wide range of business problems such as product recommendations, predictive analytics, decision optimization and automation. You find here talks about innovative analytics applications and systems for machine learning, statistics, visualization, natural language processing and deep learning.

Streaming and Real-Time Analytics

This track covers technologies, strategies and valid use-cases for building streaming systems and implementing real-time applications that enable actionable insights and interactions not previously possible with classic batch systems. This includes solutions for data stream ingestion and applying various real-time algorithms and machine learning models to derive valuable insights from the flow of events coming from IoT sensors, devices, users, and front-end applications.

11.15 - 11.45

From legacy to cloud: an end to end data integration journey

Raw data collection from cloud and legacy data centers. Standard data preparation (e.g. binary conversion, partitioning). User driven analytics and machine learning. Challenges and experiences of building and operating data pipelines and computation as a service for hundreds of teams operating at petabytes scale.

Transitioning a full company to the cloud is hard. Starting a Data Lake half way through is even harder. Integration of data sources from both cloud and legacy data centers has to be done tailored to the past, with the future in mind. Inside of Zalando we are building an end to end data integration platform in order to serve analytical use cases and machine learning throughout the company. Starting with raw data collection, a set of standard data preparation steps can be applied on a use case basis, after which user driven analytics generate the aggregated datasets which are of highest value throughout the company. This talk will cover the technical challenges and experiences of building and operating data pipelines and computation as a service for hundreds of teams operating at petabytes scale.
Learn more
Less

Keywords: dataLake, dataPipelines, infrastructure, Spark

Max Schultze

Data Engineer, Zalando SE

11.15 - 11.45

Building Machine Learning platform for Real-Time Bidding

How to build platform for fast machine learning model development? How to serve machine learning models under heavy load and tight timing constraints? How to establish effective cooperation between data scientists and data engineers?

In area of Real-Time Bidding (RTB) our goal is to deliver clients campaign goals like number of clicks or impressions on the Internet for the lowest possible price. We’ve been using machine learning for this purpose. Our use case is uncommon because we’re constrained by time (10ms for execution of algorithm) and under heavy load (over 500K requests/second). To mitigate those constrains we’ve developed our internal platform for Data Scientists where they can easily build their models which will be retrained and redeployed dozen times per day to our also internally build serving service without any downtimes or restarts.

This presentation will show our architecture, describe problems we’re facing and give rationale for some of the design decisions we’ve made.

Learn more
Less

Keywords: machinelearning, bigdata, fastdata, rtb

Tomasz Kogut

Technical Lead, Adform

11.15 - 11.45

Large Scale Land use of Satellite Imagery

Leveraging Convolutional Neural Network models in a Streaming pipeline for Segmentation of satellite images for agricultural use.

In this talk, we’ll be looking at leveraging Sentinel-2 satellite imagery datasets along with OpenStreetMap labels to be able to classify land use as agricultural or non-agricultural. With these datasets, many different ML tasks can be performed like – image segmentation into two classes (farm land and non-farm land) or more challenging task of identification of crop type being cultivated on fields.

For this talk, we’ll be looking at leveraging Convolutional Neural Networks (CNNs) built with TensorFlow/PyTorch to train Deep Learning models for identifying tulip fields in Netherlands based on Sentinel-2 satellite images. We’ll be covering the different Deep Learning Architectures considered for this particular use case along with the appropriate metrics. We’ll be also leveraging streaming pipelines built on Apache Beam Python SDK with Apache Flink runner for model inference.

Atendees will come away with a better understanding of how to analyze satellite imagery and the different Deep Learning architectures along with their pros/cons when analyzing satellite imagery for land use.

Learn more
Less

Keywords: Computer Vision, Apache Beam, Convolutional Neural Networks

Suneel Marthi

Principal Technologist - AI/ML, Amazon Web Services

11.15 - 11.45

Streaming Visualization

Batch and streaming visualization in big data reference architecture, architecture blueprints for streaming visualization, implementations of the blueprints in a fast data solution.

Most data visualisation solutions today still work on data sources which are stored persistently in a data store, using the so called “data at rest” paradigms. More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. These data stream publish with high velocity and messages often have to be processed as quick as possible. For the processing and analytics on the data, so called stream processing solutions are available. But these only provide minimal or no visualisation capabilities. Therefore, one can use a dedicated Streaming Visualisation solution. They are specially built for streaming data and often do not support batch data. A much better solution would be to have one tool capable of handling both, batch and streaming data. This talk presents different architecture blueprints for integrating data visualisation into a fast data solution and highlights some of the products available to implement these blueprints.
Learn more
Less

Keywords: streaming visualization, kafka, bigdata architecture

Guido Schmutz

Solution Architect, Trivadis AG

11.45 - 11.50

Technical break

11.50 - 12.20

The Data Analytics Platform or how to make data science in a box possible

State of data platforms in the tech industry. ING WBAA vision on the future of data analytics. Highlights on the ING Data Analytics Platform main components and features.

The Hadoop and FOSS revolution has reshaped the data analytics landscape. In the search for creating a cutting-edge data platform at ING, we are faced with challenging new requirements such as cloud-ready deployments and frictionless progressions of Machine Learning models into production, whilst ensuring proper data governance and security principals.
Can Docker move beyond stateless, small applications, into the world of dynamic, petabyte-scale workloads and GPU computing? Has Kubernetes gained momentum and matured enough to meet our needs?
Please join us in this session, where we will share ING WBAA’s experience on how to make a data platform based on open source components both enterprise and cloud ready, with an overview of current state and vision of our platform. We hope our demonstrations will inspire you about the potential of the evolving K8s ecosystem.
Learn more
Less

Keywords: kubernetes, data platform, data governance, spark

Rob Keevil

Data Analytics Platform Lead, ING

Krzysztof Adamski

Data Infrastructure Architect, ING

11.50 - 12.20

8 Takeaways from building Rakuten Analytics

Introduction – what is Rakuten Analytics, introduce team and roles. Motivation in building an on-premise petabyte-scale analytics service. 8 takeaways from building Rakuten Analytics. Tech and business impact.

This presentation will cover the background, challenges, architecture and takeaways from building a petabyte-scale analytics service (Rakuten Analytics). It will focus on the following takeaways:

– Build self-sustaining pipelines – pre-flight checks, automatic failure recovery, idempotency, self-validation, fault-tolerance
– Pick (kinda) static over dynamic – balance between flexibility and maintainability (2-3 examples)
– Approximate when possible – 90% of customers are good with approximate (data sketches usage examples)
– Give options to the remaining 10% – some customers require accuracy over speed
– Design architecture to scale
– Aim for real-time – valuable insights are unlocked as analytics move closer to realtime (2-3 examples)
– Listen to your customers
– Monitor everything but alert only on symptoms

Learn more
Less

Keywords: analytics, data sketches, microservices, real-time

Juan Paulo Gutierrez

Lead Data Engineer and Architect, Rakuten

11.50 - 12.20

Data Science at PMI - The Tools of The Trade

Introduction to Data Products, CI & CD, Modus Operandi and Agile Data Science on the Data Ocean. Best Practices: Docker Containers, Project Templates, Programming Style Standards etc. Reproducible Data Science

Data Science is not a one man show. It is a team effort that requires every team member to master the tools of the trade. This is extremely important for effectively putting data science to work in a global organization. In this talk we would like to share with you the best practices to start, develop and ship data science products developed inside PMI – the best practices and tools, currently in use by 30+ data scientists across four different location, where data science labs of PMI were established in 2017.

We would like to share with you how the technologies (Docker, Artifactory, Jenkins) and methods (templates in Cookiecutter, CI/CD with GitFlow) well-known from software engineering are helping us in creating data science workflow that adapts to specific needs of every peculiar use case we need to deal with, provides transparency at all times, is reproducible not only at the data science but also data engineering and devops dimensions and allows at the same time frictionless development of data products and gives us the freedom to experiment.

We also would like to discuss the ‘transition challenges’ and share some practical hints – for moving from pure exploration in Jupyter to building pip packages that will be put into production as well as for moving from data-to-code to code-to-data approaches in data science challenges.

If you’re interested in how Python, Jupyter notebooks, Docker, AWS, Hadoop ecosystem, Spark, Artifactory, Jenkins, Atlassian suite, etc. are setup to support our collaborative work, devoted to building predictive models, this talk is for you.

Learn more
Less

Keywords: CI/CD, Best Practices for Data Science, Data Product, Reproducible research

Maciej Marek

Enterprise Data Scientist , Philip Morris International

Michał Dyrda

Senior Enterprise Data Scientist, Philip Morris International

11.50 - 12.20

Driving your marketing automation with multi-armed bandits in real time

Multiarmed bandits vs simple A/B testing. Architecture of solution – how to connect Flink, Nussknacker and R? Other uses cases – what are other good fits for similar architecture.

We observe that many of our customers are actively adapting various marketing automation solutions. While most of them offer some basic A/B testing modules they are often too simple for highly dynamic conditions. Better outcomes can be achieved using e.g. multiarmed bandits algorithms, however, it’s not so straightforward to deploy them in a realtime production environment.

In our presentation, we will use a platform based on Apache Flink, Nussknacker – our custom GUI and R Studio + R Suite – everything deployed on Kubernetes.The main goal of our talk is to show how using proposed tools we can create complete flow – from model creation, through deployment and reinforcement learning – that helps to automate marketing communication without the need for custom code development.

The talk is partially based on our former deployments of similar solutions, many ideas are new, however.

Learn more
Less

Keywords: Multi-armed bandit, Marketing automation, Streaming/Flink, R

Wit Jakuczun

founder and co-owner, WLOG Solutions

Maciej Próchniak

Software Lead Developer, TouK

12.20 - 12.25

Technical break

12.25 - 12.55

Scalable machine-learned model serving

Online evaluation of machine-learned models (model serving) is difficult to scale to large data sets. Vespa.ai is an open source solution to this problem in use today on some of the largest such systems in the world, such as the content pages on the Yahoo network and the worlds third largest ad network.
This talk will explain the problem and architectural solution, show how Vespa can be used to implement the solution to achieve scalable serving of TensorFlow and ONNX models, and present benchmarks comparing performance and scalability to TensorFlow Serving.

Applying machine learning in online applications requires solving the problem of model serving: Evaluating the machine-learned model over some data point(s) in real time while the user is waiting for a response. Solutions such as TensorFlow Serving are available to solve this problem where the model only needs to be evaluated over a one data point per user request, but what about the case where a model needs to be evaluated over many data points per request, such as in recommendation systems?

This talk will show that this is a bandwidth constrained problem, and outline an architectural solution where computation is pushed down to data shards in parallel. It will demonstrate how this solution can be put into use with Vespa.ai – the open source big data serving engine – to achieve scalable model serving of TensorFlow and ONNX and show benchmarks comparing performance and scalability to TensorFlow Serving.

Model serving with Vespa is used today for some of the worlds largest recommendation systems, such as serving personalized content on all Yahoo content pages and personalized ads in the worlds third largest ad network. These systems evaluate models over millions of data points per request for hundreds of thousands of requests per second.

Learn more
Less

Keywords: bigdata, opensource, vespa, ml

Jon Bratseth

Distinguished architect, Oath (former Yahoo)

12.25 - 12.55

Enterprise Adoption at Klarna. Software Engineering methods bringing order to the Big Data Galaxy

Klarna provides instant consumer credit at the point of sale, and allows flexible credit lines after purchase. Credit is issued using Klarna’s Checkout product, which is integrated with almost 100’000 merchants on many markets. In order to perform automated decisions on credit and fraud with low latency guarantees, it is mission critical for Klarna that all required data is available at all times. Furthermore, it is important that all decisions are traceable. Finally, our data infrastructure must facilitate engineer and analyst productivity.

I will articulate these requirements, and show how we met the requirements with the following practices: (i) The immutability and recomputation properties of the Lambda/Kappa architectures (ii) continuously delivered and automated infrastructure, (iii) tooling to empower producers and consumers of data to be accountable and self-sufficient, and (iv) proactively improve efficiency of data users. In our journey through the Big Data Galaxy, it has become evident that unit testing, continuous delivery, and versioning are essential travel companions.
Learn more
Less

Keywords: Teams, Tools, Processes, Performance and scalability, Validation

Erik Zeitler

Lead software engineer, Klarna

12.25 - 12.55

Evolution of search: From a complicated problem to a simplified search experience

What goes into making Booking the leading player to book place to stay? At Booking, with millions of users visiting our platforms every day we have very rich behavioural data about our users – what they type, what they search for, what properties they look at, what filters they apply, how much time they spend on a property page that they book compared to pages of properties that they don’t book, etc. Each of these behavioral data points have helped us build a powerful, personalized search, making the experience of booking a trip easier for our users.

In this talk, we will take you through the journey of how our search evolved from being a simple database query to a data driven search experience. We will explain how we process almost 1 billion user interactions every day to find the top 5 results for any given search query, and how we ensure that 90% of our reservations come from properties found on the first page on search results.
Learn more
Less

Keywords: search, ranking, bigdata

Arihant Gupta

Software Developer, Booking.com

Priyanka Prakash

Product Owner, Booking.com

12.25 - 12.55

Detecting Patterns in Event Streams with Flink SQL

Introduction to SQL on streams concepts, explain how the new SQL MATCH_RECOGNIZE clause brings the power of pattern matching to (streaming) SQL and demonstrate advanced stream analytics with Flink SQL.

Apache Flink is one of the first open source stream processors that was able to address the full spectrum of stream processing applications, ranging from applications with low latency requirements to applications that process millions of events per second. On top of this powerful processing engine, the Flink community built APIs for complex event processing and streaming analytics, namely the CEP library and support for streaming SQL.

Since recently, the Flink community is integrating both APIs by extending Flink SQL to support the MATCH RECOGNIZE clause for row pattern matching that was introduced with the SQL:2016 standard.

In my talk, I will discuss the new MATCH RECOGNIZE feature and present use cases that benefit from pattern matching support in streaming SQL, such as process monitoring or anomaly detection. I will demonstrate the feature with a few example queries and take a brief look under hood to discuss a few details of the implementation.

Learn more
Less

Keywords: Apache Flink, SQL, stream analytics, stream processing

Dawid Wysakowicz

Software Engineer, data Artisans

12.55 - 13.50

Lunch

13.50 - 14.20

Reliable logging infrastructure: Building trust on logs @ Slack

Overview of logging infrastructure at Slack and why should you care about it, a summary of why reliability is critical for logging infrastructure to gain customer trusts.  Lesson learned and best practices for building logging infrastructure.

Slack is a communication and collaboration platform for teams. Our millions of users spend 10+ hrs connected to the service on a typical working day.

The Slack data engineering team goal is simple: Drive up speed, efficiency, and reliability of making data-informed decisions. For engineers, For people managers, For salespeople, For every slack customer.

Logs are the critical components of our systems. They help us to understand how a system works, debug, analysis performance and improve the operational efficiency. Logging infrastructure is aslo a critical component for Slack. Slack logging pipeline drives customer billing, usage pattern and performance analysis of the business critical systems.

The talk walks through the first generation logging infrastructure and some of the problems we encounter. The talk walkthrough the second generation logging infrastructure design, how we add reliability as a built-in feature of the systems, the overall design principles and some of the best practices for designing scalable logging infrastructure.

Learn more
Less

Keywords: logging, kafka, reliability

Ananth Packkildurai

Senior Data Engineer, Slack Technologies Inc

13.50 - 14.20

Metadata Driven Access Control in Practice

The importance of data access governance is continuously growing due to new regulations, such as GDPR, and industry policies. Manage access policies for each individual dataset is a hassle. In this talk, we will show how Svenska Spel use metadata about datasets to generate access policies. We use it to create policies for access, retention, and anonymization.

To create our policies from metadata we developed and open sourced cobra-policytool. Thanks to the tool we can use git and metadata repository as the source of truth for our policies. Policies are deployed to Apache Atlas and Ranger via a CI/CD pipeline. With our work deriving policies from metadata we have privacy and security by design, low maintenance burden using automatization, easy to review of policies, traceability of policy changes and more.
Learn more
Less

Keywords: DataGovernance, GDPR, Security

Magnus Runesson

Data engineer, Tink

13.50 - 14.20

I’m a data scientist and engineers don’t hate me

What’s the role of data scientist in the development of a new project? Do they only produce charts while engineers are doing the real work? In this presentation, I will share a story of data science and engineering collaboration at Twitter that resulted in shipping a successful new product feature. I’ll talk about our tools and environment and specifically focus on inter-team dynamics: what practices did we use? how did we divide tasks? what benefits did each team derive? why do we still want to work together?

Keywords: collaboration, cross-functional, best practices, data scientist’s environment

Mateusz Fedoryszak

Data Scientist, Twitter

13.50 - 14.20

Streaming topic model training and inference with Apache Flink

How to use stateful stream processing and Flink’s Dynamic processing capabilities to continuously train topic models from unlabelled text and use such models to extract topics from the data itself.

Analysing streams of text data to extract topics is an important task for getting useful insights to be leveraged in subsequent workflows. For example extracting topics from text to be continuously ingested into a search engine can be useful to tag documents with important keywords or concepts to be used at search time. Another use case is doing analysis of support tickets to get insights on the most common problems for customers.

Most Topic Modeling today is actually done as a Batch process despite the existence of Near-approximate algorithms that can actually leverage Stateful Stream processing.

In this talk, we discuss a new approach to streaming topic modeling and also look at other implementations like Online LDA leveraging Apache Flink stateful streaming. We illustrate how to use Flink’s Dynamic processing capabilities to continuously train topic models from unlabelled text and use such models to extract topics from the data itself. Such topic models will be built leveraging distributed representations of words and documents.

Learn more
Less

Keywords: Streaming, Topic Modeling, NLP, Keyword extraction

Suneel Marthi

Principal Technologist - AI/ML, Amazon Web Services

Jörn Kottmann

Senior Software Developer, Sandstone SA

14.20 - 14.25

Technical break

14.25 - 14.55

We are going to add all topics of each presentations soon.

14.25 - 14.55

We are going to add all topics of each presentations soon.

14.25 - 14.55

Data Science in Roche Diagnostics: From Exploration to Productionization

Data Science applications in Roche Diagnostics – from exploration to productionization of DS initiatives. Real use case #1: machine learning and image processing for quality control. Real use case #2: time series analysis on financial data for business planning.

Data Science has been seriously evolving during the last decade at the enterprise level across variety of industries including healthcare. There is no surprise that Roche, which is pioneer in the field of pharmaceuticals and diagnostics, would also intend to become a leader in digitalization and leveraging data and analytics. In this talk, I would like to show you how data science can significantly generate medical and business value in a non-IT company like Roche. I will take you through a journey in which two real data science use cases in Roche Diagnostics will be demonstrated from the exploratory phase to the industrialization phase. I am going to share with you some real challenges in that journey and how to come up with a way to get through them. The first use case is in the context of image processing, where machine learning can be used to automate in-process control during production. The second one would then be in the context of time series analysis with the application in financial forecasting.
Learn more
Less

Keywords: Data Science in Roche Diagnostics, Financial Time Series Analysis, Automated Quality Control, Deep Learning on Product Images, Machine learning, Image Processing

Dr Mohammadjavad Faraji

Data Scientist, Roche

14.25 - 14.55

The Changing Face of ETL: Event-Driven Architectures for Data Engineers

The power of events and unbounded data. Streaming is not just for real-time applications – it’s for everyone. Where a streaming platform fits in an analytic architecture. How event-driven architectures can enable greater scalability and flexibility of systems both now and in the future.

Data integration in architectures built on static, update-in-place datastores inevitably end up with pathologically high degrees of coupling and poor scalability. This has been the standard practice for decades, as we attempt to build data pipelines on top of databases that do a poor job modeling the fundamental objects that drive our businesses and systems: events.

In this talk, we’ll discuss the concepts of events, their relevance to software and data engineers and their ability to unify architectures in a powerful way. We’ll see how stream processing makes sense in both a microservices and ETL environment, and why analytics, data integration and ETL fit naturally into a streaming world. The talk will conclude with a hands-on demonstration of these concepts in practice using Apache Kafka and commentary on the design choices made.

Learn more
Less

Keywords: kafka, event-driven architecture, streaming, integration

Robin Moffatt

Developer Advocate, Confluent

14.55 - 15.00

Technical break

15.00 - 15.30

We are going to add all topics of each presentations soon.

15.00 - 15.30

We are going to add all topics of each presentations soon.

15.00 - 15.30

We are going to add all topics of each presentations soon.

15.00 - 15.30

We are going to add all topics of each presentations soon.

Architecture, Operations and Cloud

This track is dedicated to architects, administrators and people with DevOps skills who are interested in technologies, techniques and best practices for planning, building, installing, managing and securing their Big Data infrastructure in enterprise environments – both on-premise and the cloud.

11.15 - 11.45

From legacy to cloud: an end to end data integration journey

Raw data collection from cloud and legacy data centers. Standard data preparation (e.g. binary conversion, partitioning). User driven analytics and machine learning. Challenges and experiences of building and operating data pipelines and computation as a service for hundreds of teams operating at petabytes scale.

Transitioning a full company to the cloud is hard. Starting a Data Lake half way through is even harder. Integration of data sources from both cloud and legacy data centers has to be done tailored to the past, with the future in mind. Inside of Zalando we are building an end to end data integration platform in order to serve analytical use cases and machine learning throughout the company. Starting with raw data collection, a set of standard data preparation steps can be applied on a use case basis, after which user driven analytics generate the aggregated datasets which are of highest value throughout the company. This talk will cover the technical challenges and experiences of building and operating data pipelines and computation as a service for hundreds of teams operating at petabytes scale.
Learn more
Less

Keywords: kubernetes, data platform, data governance, spark

Rob Keevil

Data Analytics Platform Lead, ING

Krzysztof Adamski

Data Infrastructure Architect, ING

12.20 - 12.25

Technical break

12.25 - 12.55

Scalable machine-learned model serving

Online evaluation of machine-learned models (model serving) is difficult to scale to large data sets. Vespa.ai is an open source solution to this problem in use today on some of the largest such systems in the world, such as the content pages on the Yahoo network and the worlds third largest ad network.
This talk will explain the problem and architectural solution, show how Vespa can be used to implement the solution to achieve scalable serving of TensorFlow and ONNX models, and present benchmarks comparing performance and scalability to TensorFlow Serving.

Applying machine learning in online applications requires solving the problem of model serving: Evaluating the machine-learned model over some data point(s) in real time while the user is waiting for a response. Solutions such as TensorFlow Serving are available to solve this problem where the model only needs to be evaluated over a one data point per user request, but what about the case where a model needs to be evaluated over many data points per request, such as in recommendation systems?

This talk will show that this is a bandwidth constrained problem, and outline an architectural solution where computation is pushed down to data shards in parallel. It will demonstrate how this solution can be put into use with Vespa.ai – the open source big data serving engine – to achieve scalable model serving of TensorFlow and ONNX and show benchmarks comparing performance and scalability to TensorFlow Serving.

Model serving with Vespa is used today for some of the worlds largest recommendation systems, such as serving personalized content on all Yahoo content pages and personalized ads in the worlds third largest ad network. These systems evaluate models over millions of data points per request for hundreds of thousands of requests per second.

Learn more
Less

Keywords: bigdata, opensource, vespa, ml

Jon Bratseth

Distinguished architect, Oath (former Yahoo)

12.55 - 13.50

Lunch

13.50 - 14.20

Reliable logging infrastructure: Building trust on logs @ Slack

Overview of logging infrastructure at Slack and why should you care about it, a summary of why reliability is critical for logging infrastructure to gain customer trusts.  Lesson learned and best practices for building logging infrastructure.

Slack is a communication and collaboration platform for teams. Our millions of users spend 10+ hrs connected to the service on a typical working day.

The Slack data engineering team goal is simple: Drive up speed, efficiency, and reliability of making data-informed decisions. For engineers, For people managers, For salespeople, For every slack customer.

Logs are the critical components of our systems. They help us to understand how a system works, debug, analysis performance and improve the operational efficiency. Logging infrastructure is aslo a critical component for Slack. Slack logging pipeline drives customer billing, usage pattern and performance analysis of the business critical systems.

The talk walks through the first generation logging infrastructure and some of the problems we encounter. The talk walkthrough the second generation logging infrastructure design, how we add reliability as a built-in feature of the systems, the overall design principles and some of the best practices for designing scalable logging infrastructure.

Learn more
Less

Keywords: logging, kafka, reliability

Ananth Packkildurai

Senior Data Engineer, Slack Technologies Inc

14.20 - 14.25

Technical break

14.25 - 14.55

We are going to add all topics of each presentations soon.

14.55 - 15.00

Technical break

15.00 - 15.30

We are going to add all topics of each presentations soon.

Data Engineering

This track is the place for engineers to learn about tools, techniques and battle-proven solutions to collect, store and process large amounts of data. It covers topics like data collection and ingestion, ETL, job scheduling, metadata and schema management, distributed processing engines, distributed datastores and more.

11.15 - 11.45

Building Machine Learning platform for Real-Time Bidding

How to build platform for fast machine learning model development? How to serve machine learning models under heavy load and tight timing constraints? How to establish effective cooperation between data scientists and data engineers?

In area of Real-Time Bidding (RTB) our goal is to deliver clients campaign goals like number of clicks or impressions on the Internet for the lowest possible price. We’ve been using machine learning for this purpose. Our use case is uncommon because we’re constrained by time (10ms for execution of algorithm) and under heavy load (over 500K requests/second). To mitigate those constrains we’ve developed our internal platform for Data Scientists where they can easily build their models which will be retrained and redeployed dozen times per day to our also internally build serving service without any downtimes or restarts.

This presentation will show our architecture, describe problems we’re facing and give rationale for some of the design decisions we’ve made.

Learn more
Less

Keywords: analytics, data sketches, microservices, real-time

Juan Paulo Gutierrez

Lead Data Engineer and Architect, Rakuten

12.20 - 12.25

Technical break

12.25 - 12.55

Enterprise Adoption at Klarna. Software Engineering methods bringing order to the Big Data Galaxy

Klarna provides instant consumer credit at the point of sale, and allows flexible credit lines after purchase. Credit is issued using Klarna’s Checkout product, which is integrated with almost 100’000 merchants on many markets. In order to perform automated decisions on credit and fraud with low latency guarantees, it is mission critical for Klarna that all required data is available at all times. Furthermore, it is important that all decisions are traceable. Finally, our data infrastructure must facilitate engineer and analyst productivity.

I will articulate these requirements, and show how we met the requirements with the following practices: (i) The immutability and recomputation properties of the Lambda/Kappa architectures (ii) continuously delivered and automated infrastructure, (iii) tooling to empower producers and consumers of data to be accountable and self-sufficient, and (iv) proactively improve efficiency of data users. In our journey through the Big Data Galaxy, it has become evident that unit testing, continuous delivery, and versioning are essential travel companions.
Learn more
Less

Keywords: Teams, Tools, Processes, Performance and scalability, Validation

Erik Zeitler

Lead software engineer, Klarna

12.55 - 13.50

Lunch

13.50 - 14.20

Metadata Driven Access Control in Practice

The importance of data access governance is continuously growing due to new regulations, such as GDPR, and industry policies. Manage access policies for each individual dataset is a hassle. In this talk, we will show how Svenska Spel use metadata about datasets to generate access policies. We use it to create policies for access, retention, and anonymization.

To create our policies from metadata we developed and open sourced cobra-policytool. Thanks to the tool we can use git and metadata repository as the source of truth for our policies. Policies are deployed to Apache Atlas and Ranger via a CI/CD pipeline. With our work deriving policies from metadata we have privacy and security by design, low maintenance burden using automatization, easy to review of policies, traceability of policy changes and more.
Learn more
Less

Keywords: DataGovernance, GDPR, Security

Magnus Runesson

Data engineer, Tink

14.20 - 14.25

Technical break

14.25 - 14.55

We are going to add all topics of each presentations soon.

14.55 - 15.00

Technical break

15.00 - 15.30

We are going to add all topics of each presentations soon.

Artificial Intelligence and Data Science

This track includes real-world case-studies demonstrating how data & technology are used together to address a wide range of business problems such as product recommendations, predictive analytics, decision optimization and automation. You find here talks about innovative analytics applications and systems for machine learning, statistics, visualization, natural language processing and deep learning.

11.15 - 11.45

Large Scale Land use of Satellite Imagery

Leveraging Convolutional Neural Network models in a Streaming pipeline for Segmentation of satellite images for agricultural use.

In this talk, we’ll be looking at leveraging Sentinel-2 satellite imagery datasets along with OpenStreetMap labels to be able to classify land use as agricultural or non-agricultural. With these datasets, many different ML tasks can be performed like – image segmentation into two classes (farm land and non-farm land) or more challenging task of identification of crop type being cultivated on fields.

For this talk, we’ll be looking at leveraging Convolutional Neural Networks (CNNs) built with TensorFlow/PyTorch to train Deep Learning models for identifying tulip fields in Netherlands based on Sentinel-2 satellite images. We’ll be covering the different Deep Learning Architectures considered for this particular use case along with the appropriate metrics. We’ll be also leveraging streaming pipelines built on Apache Beam Python SDK with Apache Flink runner for model inference.

Atendees will come away with a better understanding of how to analyze satellite imagery and the different Deep Learning architectures along with their pros/cons when analyzing satellite imagery for land use.

Learn more
Less

Keywords: CI/CD, Best Practices for Data Science, Data Product, Reproducible research

Maciej Marek

Enterprise Data Scientist , Philip Morris International

Michał Dyrda

Senior Enterprise Data Scientist, Philip Morris International

12.20 - 12.25

Technical break

12.25 - 12.55

Evolution of search: From a complicated problem to a simplified search experience

What goes into making Booking the leading player to book place to stay? At Booking, with millions of users visiting our platforms every day we have very rich behavioural data about our users – what they type, what they search for, what properties they look at, what filters they apply, how much time they spend on a property page that they book compared to pages of properties that they don’t book, etc. Each of these behavioral data points have helped us build a powerful, personalized search, making the experience of booking a trip easier for our users.

In this talk, we will take you through the journey of how our search evolved from being a simple database query to a data driven search experience. We will explain how we process almost 1 billion user interactions every day to find the top 5 results for any given search query, and how we ensure that 90% of our reservations come from properties found on the first page on search results.
Learn more
Less

Keywords: search, ranking, bigdata

Arihant Gupta

Software Developer, Booking.com

Priyanka Prakash

Product Owner, Booking.com

12.55 - 13.50

Lunch

13.50 - 14.20

I’m a data scientist and engineers don’t hate me

What’s the role of data scientist in the development of a new project? Do they only produce charts while engineers are doing the real work? In this presentation, I will share a story of data science and engineering collaboration at Twitter that resulted in shipping a successful new product feature. I’ll talk about our tools and environment and specifically focus on inter-team dynamics: what practices did we use? how did we divide tasks? what benefits did each team derive? why do we still want to work together?

Keywords: collaboration, cross-functional, best practices, data scientist’s environment

Mateusz Fedoryszak

Data Scientist, Twitter

14.20 - 14.25

Technical break

14.25 - 14.55

Data Science in Roche Diagnostics: From Exploration to Productionization

Data Science applications in Roche Diagnostics – from exploration to productionization of DS initiatives. Real use case #1: machine learning and image processing for quality control. Real use case #2: time series analysis on financial data for business planning.

Data Science has been seriously evolving during the last decade at the enterprise level across variety of industries including healthcare. There is no surprise that Roche, which is pioneer in the field of pharmaceuticals and diagnostics, would also intend to become a leader in digitalization and leveraging data and analytics. In this talk, I would like to show you how data science can significantly generate medical and business value in a non-IT company like Roche. I will take you through a journey in which two real data science use cases in Roche Diagnostics will be demonstrated from the exploratory phase to the industrialization phase. I am going to share with you some real challenges in that journey and how to come up with a way to get through them. The first use case is in the context of image processing, where machine learning can be used to automate in-process control during production. The second one would then be in the context of time series analysis with the application in financial forecasting.
Learn more
Less

Keywords: Data Science in Roche Diagnostics, Financial Time Series Analysis, Automated Quality Control, Deep Learning on Product Images, Machine learning, Image Processing

Dr Mohammadjavad Faraji

Data Scientist, Roche

14.55 - 15.00

Technical break

15.00 - 15.30

We are going to add all topics of each presentations soon.

Streaming and Real-Time Analytics

This track covers technologies, strategies and valid use-cases for building streaming systems and implementing real-time applications that enable actionable insights and interactions not previously possible with classic batch systems. This includes solutions for data stream ingestion and applying various real-time algorithms and machine learning models to derive valuable insights from the flow of events coming from IoT sensors, devices, users, and front-end applications.

11.15 - 11.45

Streaming Visualization

Batch and streaming visualization in big data reference architecture, architecture blueprints for streaming visualization, implementations of the blueprints in a fast data solution.

Most data visualisation solutions today still work on data sources which are stored persistently in a data store, using the so called “data at rest” paradigms. More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. These data stream publish with high velocity and messages often have to be processed as quick as possible. For the processing and analytics on the data, so called stream processing solutions are available. But these only provide minimal or no visualisation capabilities. Therefore, one can use a dedicated Streaming Visualisation solution. They are specially built for streaming data and often do not support batch data. A much better solution would be to have one tool capable of handling both, batch and streaming data. This talk presents different architecture blueprints for integrating data visualisation into a fast data solution and highlights some of the products available to implement these blueprints.
Learn more
Less

Keywords: Multi-armed bandit, Marketing automation, Streaming/Flink, R

Wit Jakuczun

founder and co-owner, WLOG Solutions

Maciej Próchniak

Software Lead Developer, TouK

12.20 - 12.25

Technical break

12.25 - 12.55

Detecting Patterns in Event Streams with Flink SQL

Introduction to SQL on streams concepts, explain how the new SQL MATCH_RECOGNIZE clause brings the power of pattern matching to (streaming) SQL and demonstrate advanced stream analytics with Flink SQL.

Apache Flink is one of the first open source stream processors that was able to address the full spectrum of stream processing applications, ranging from applications with low latency requirements to applications that process millions of events per second. On top of this powerful processing engine, the Flink community built APIs for complex event processing and streaming analytics, namely the CEP library and support for streaming SQL.

Since recently, the Flink community is integrating both APIs by extending Flink SQL to support the MATCH RECOGNIZE clause for row pattern matching that was introduced with the SQL:2016 standard.

In my talk, I will discuss the new MATCH RECOGNIZE feature and present use cases that benefit from pattern matching support in streaming SQL, such as process monitoring or anomaly detection. I will demonstrate the feature with a few example queries and take a brief look under hood to discuss a few details of the implementation.

Learn more
Less

Keywords: Apache Flink, SQL, stream analytics, stream processing

Dawid Wysakowicz

Software Engineer, data Artisans

12.55 - 13.50

Lunch

13.50 - 14.20

Streaming topic model training and inference with Apache Flink

How to use stateful stream processing and Flink’s Dynamic processing capabilities to continuously train topic models from unlabelled text and use such models to extract topics from the data itself.

Analysing streams of text data to extract topics is an important task for getting useful insights to be leveraged in subsequent workflows. For example extracting topics from text to be continuously ingested into a search engine can be useful to tag documents with important keywords or concepts to be used at search time. Another use case is doing analysis of support tickets to get insights on the most common problems for customers.

Most Topic Modeling today is actually done as a Batch process despite the existence of Near-approximate algorithms that can actually leverage Stateful Stream processing.

In this talk, we discuss a new approach to streaming topic modeling and also look at other implementations like Online LDA leveraging Apache Flink stateful streaming. We illustrate how to use Flink’s Dynamic processing capabilities to continuously train topic models from unlabelled text and use such models to extract topics from the data itself. Such topic models will be built leveraging distributed representations of words and documents.

Learn more
Less

Keywords: Streaming, Topic Modeling, NLP, Keyword extraction

Suneel Marthi

Principal Technologist - AI/ML, Amazon Web Services

Jörn Kottmann

Senior Software Developer, Sandstone SA

14.20 - 14.25

Technical break

14.25 - 14.55

The Changing Face of ETL: Event-Driven Architectures for Data Engineers

The power of events and unbounded data. Streaming is not just for real-time applications – it’s for everyone. Where a streaming platform fits in an analytic architecture. How event-driven architectures can enable greater scalability and flexibility of systems both now and in the future.

Data integration in architectures built on static, update-in-place datastores inevitably end up with pathologically high degrees of coupling and poor scalability. This has been the standard practice for decades, as we attempt to build data pipelines on top of databases that do a poor job modeling the fundamental objects that drive our businesses and systems: events.

In this talk, we’ll discuss the concepts of events, their relevance to software and data engineers and their ability to unify architectures in a powerful way. We’ll see how stream processing makes sense in both a microservices and ETL environment, and why analytics, data integration and ETL fit naturally into a streaming world. The talk will conclude with a hands-on demonstration of these concepts in practice using Apache Kafka and commentary on the design choices made.

Learn more
Less

Keywords: kafka, event-driven architecture, streaming, integration

Robin Moffatt

Developer Advocate, Confluent

14.55 - 15.00

Technical break

15.00 - 15.30

We are going to add all topics of each presentations soon.

15.30 - 16.00

Coffee break

16.00 – 17.25 Roundtables sessions

16.00 - 16.05

Intro

Parallel roundtables discussions are the part of the conference that engage all participants. It has few purposes. First of all, participants have the opportunity to exchange their opinions and experiences about specific issue that is important to that group. Secondly, participants can meet and talk with the leader/host of the roundtable discussion – they are selected professionals with a vast knowledge and experience.

There will be 2 rounds of discussion, hence every conference participants can take part in 2 discussions

 

16.05 – 16.45    1st round

16.50 – 17.25    2nd round

Big Data on Kubernetes

Maciej Bryński

Big Data Architect, XCaliber

Analytics and Customer Experience Management on top of BigData

Darko Marjanović, Things Solver & Goran Pavlović, Vip Mobile

Going to production:we have the best model in the world and….

Radosław Kita

Team Lead, Adform

How to choose the best performing model? Accuracy, Precision, Recall or something wild?

Radosław Kita

Team Lead, Adform

17.25 - 17.45

Coffee break

17.45 - 18.15

Panel discussion

18.15 - 18.30

Closing & Summary

Przemysław Gamdzyk

CEO & Meeting Designer, Evention

19.00 - 22.00

Networking party for all participants and speakers

At the end of the conference we would like to invite all the attendees for the informal evening meeting.