Have you ever wondered how gaming companies build their analytics pipelines? Particularly scalable ones that are able to collect terabytes of data every day? At Outfit7, this is done with a little help from Google Cloud's top services, including Kubernetes, Dataflow, BigQuery, and Cloud Composer. In this presentation, you'll see how the pipeline is built, starting from ingestion in Kubernetes, through to ending in Jupyter, Tableau, and other BI dashboards. You’ll also find out how the team fights downtime with proactive monitoring and integration tests. And last but not least, you’ll hear about the challenges that Outfit7 faced when the amount of data it had to handle skyrocketed during the peak of the COVID-19 quarantine.

Data Quality is paramount -we all agree on that- and isn't straightforward even with small data sets. When working with over 120PB of data on Hadoop and thousands of jobs, I can tell you firsthand, it's a challenge! We started to tackle this at Criteo 2 years ago, and I have some tangible results I'll be happy to share. We'll go through this journey, from collecting data, detecting suspect behaviors, and alerting users on data quality incidents, while integrating these new checks into the Criteo Data Platform.

In this session you will learn about how H&M evolves reference architecture covering entire MLOps stack addressing a few common challenges in AI and Machine learning product, like development efficiency, end to end traceability, speed to production, etc. This architecture has been adapted by multiple product teams managing 100”s of models across the entire H&M value chain and enables data scientists to develop model in a highly interactive environment, enable engineers to manage large scale model training and model serving pipeline with fully traceability.
The team presenting is currently responsible for ensuring that best practices and reference architecture are implemented on all product teams to accelerate H&M groups’ data driven business decision making journey.

Machine learning-powered systems have become an essential part of most businesses. One such example are recommender systems that adapt to customer behavior to provide an organic way to make domains like clothes, books, or music explorable. In order to successfully put such systems into production, we need to bridge the gap from the raw mathematical models and algorithms to robust and scalable software systems. In this talk, we start out with core approaches to recommender systems like collaborative filtering or click probability prediction, and follow this journey to explore how theory and practice come together.

Operationalizing Machine Learning operations (features delivery, model training, deployment and serving) is nowadays one of the most challenging areas in fast-growing data-driven companies. Variety of open source components (Kubeflow, Mlflow, Kedro to name a few) and set of specialized managed services provided by every major cloud provider drive solution architects nuts.
In GetInData we have a solution for it - we used to call it GetInData MLOps Platform. Set of reusable components, following the Unix toolset pattern ("do one thing and be best at it") and portable to any environment. Also - thanks to loose coupling - adjustable to current and future clients' ML-related challenges, like a candy shop where the first person needs super-fast online predictions, the second one requires robust hyperparameter tuning for best possible models and the third person aims for scalable collaboration on features extraction within many data science teams.
During the presentation we will show you two components we're really excited for - Kedro-Kubeflow integration and Feast-based feature store - how we implement these and what clients' use them for. Welcome to our MLOps candy shop that no pandemic can close 😉

Tracking model performance is crucial to guarantee that a model running in production behaves as designed initially. Changes in the incoming data can affect the performance and make predictions unreliable. Given that input data often change over time, it is important to track changes in both input distributions and delivered predictions periodically, and to act on them when significantly different - for example, to diagnose and retrain an incorrect model in production. To make monitoring both more consistent and semi-automatic, at ING WBAA we have developed a generic Python package called popmon to monitor the stability of data populations over time, using techniques from statistical process control, at: https://github.com/ing-bank/popmon popmon works with both pandas and spark datasets. popmon creates histograms of features binned in time-slices, and compares the stability of the profiles and distributions of those histograms using statistical tests, both over time and with respect to a reference. It works with numerical, ordinal, categorical features, and the histograms can be higher-dimensional, e.g. it can also track correlations between any two features. popmon can automatically flag and alert on changes observed over time, such as trends, shifts, peaks, outliers, anomalies, changing correlations, etc, using static or dynamic monitoring business rules.

Modern software development requires a comprehensive approach with well-defined processes of code development, testing, and deployment. That is where DevOps methodology comes in. It helps software developers in the seamless moving of their newly implemented features from development to production. But what about other areas of information technologies like advanced analytics, artificial intelligence, and machine learning? That’s where ModelOps comes to play – an approach that takes what’s best in DevOps and expands it to apply in the analytical world. During the presentation, we will show how continuous integration and deployment with the help of specialized tools, specifically designed for analytical purposes, can be leveraged to implement such an approach. This strategy can dramatically reduce the time-to-value for analytical assets developed within the organization, ensuring all those assets are methodically managed and safely updated with the most advanced, well tested analytical models.

Machine Learning is an essential part to solve use cases in cybersecurity, operations and various lines of business. This talk provides you with an overview of Splunk’s big data and machine learning technologies that are used to solve real-world use cases. For example, we dive into the technical details of two selected customer examples of applied machine learning. First, we explain how a datacenter division of Siemens uses Splunk with unsupervised and supervised machine learning approaches in cybersecurity to uncover anomalies and to automate the classification of security events. We highlight the technical details of how this use case has been addressed with Splunk’s Machine Learning Toolkit. Second, we explain how BMW Group’s Innovation Lab developed a Predictive Testing Strategy with Splunk and a deep learning approach. Details are provided on how the Deep Learning Toolkit App for Splunk was used to build and evaluate a TensorFlow based model to solve that use case. We conclude the session with an outlook and a wrap up on all available technical resources.

At Nielsen Identity, we leverage Druid to provide our customers with real-time analytics tools for various use-cases, including inflight
analytics, reporting and building target audiences. The common challenge of these use-cases is counting distinct elements in real-time at scale. We’ve been using Druid to solve these problems for the past 5 years, and gained a lot of experience with it.
In this talk, we will share some of the best practices and tips we’ve gathered over the years. We will cover the following topics:
* Data modeling
* Ingestion
* Retention and deletion
* Query optimization

You will learn about a tool that can improve your big data projects on GCP. Unified structure, configuration, versioning, build, deployment, and more, available for Dataflow/Dataproc/BigQuery.

In my presentation I would like to present some approaches, good practices and Google Cloud components that we use in Sotrender to effectively train and deploy our machine learning models, which are used to analyze Social Media data. I will discuss which aspects of DevOps we focus on when developing machine learning models (MLOps), and how these ideas can be easily implemented in your company or startup using Google Cloud Platform.

Many BigData projects focus on implementing technological solutions, forgetting their purpose, i.e. the needs or applications they are to serve.
Investments (BigData projects are quite high budgets) are often made mainly in the IT area, ignoring individual areas of business activities and do not generate much profit from a business point of view, causing a high risk of failure of the entire project. During the presentation, we will tell a bit about this kind of factors that pose a threat to BigData projects. Together, we will consider what are analyst’s needs and who is BIGData's “client” and what they expect. How to lead to effective cooperation between the analyst and the "client" by implementing data management, but most of all - by implementing an interface that creates a bridge between the accumulated knowledge about the data and the data recipient. We will show how much AI is able to support the use of Big Data's potential. Can you get information about data while you have a morning coffee? Yes. By using the Clarite AI Data Assistant all you need to do is ask a question about data in natural language. Using that, you will easily enter the era of human-data communication and big data.

Data is increasingly becoming core to many products. Whether to improve recommendations for users, getting insights on how they use the product or using machine learning to improve the experience. This creates a critical need for understanding how data is flowing through our systems. Data pipelines must be auditable, reliable and run on time. Tracking lineage and metadata is the underlying foundation that enables many use cases related to data. It provides understanding of dependencies between many teams consuming and producing data and how constant changes impact them. It enables governance and compliance and generally helps you keep you data running. Marquez is an open source project part of the LF AI which instruments data pipelines to collect lineage and metadata and enable those use cases. It provides context by making visible dependencies across organisations and technologies and enables lineage governance and discovery.

Successful data projects are built on solid foundations. What happens when we’re misled or unaware of what a solid foundation for data teams means? When a data team is missing or understaffed, the entire project is at risk of failure. This talk will cover the importance of a solid foundation and what management should do to fix it.

Nowadays the problem of speed of processing is seemingly solved. Unless you process tens of petabytes an off-the-shelf toolset will suffice for most of the problems. Currently the main challenges in data lake systems are in the field of data governance:
• how do you make sure data is discoverable, reusable, up to date and of high quality?
• how to avoid huge technical debt when developing massive number of complex data flows?
• how to guarantee that the project can scale despite having access to very scarce human resources and technical talent?

The goal of this talk is to showcase how to design a data lake management system scalable in all the broadest meaning of the word: that is not only scales with the growth of the data, but as well that it scales with the growth of the complexity of the whole enterprise. The talk will outline the business reasoning, key design principles as well as technical solution. Expect some (but not too much) nerdy details related to Apache Spark implementation 😊

Story of a year-long journey with an Asian telecom of creating a positive feedback loop between building a data analytics platform and moving analytics into it.

Like in every great story, you can expect:
introduction - reasonably well defined scope and set of main characters,
turning points - how life took us on a journey of changed ownership, discovering new needs, learning by teaching and bold goals achieved by doing progress over perfection,
conclusion - how it all came together to change the way the analytics team works and productizes it's models.

This story includes real-world analytics use-cases such as: cost of network incidents, ARPU (Average Revenue Per User), NBO (Next Best Offer), cost of service, RFM (Recency, Frequency, Monetary Value), churn and a few more.

Everything seasoned with open-source (Spark, Presto, Nbdev) and latest ML Ops technologies (KubeFlow, Feast).

Digital Transformation takes a lot of our processes to new levels. Especially, since the eruption of the pandemic, we need to look for new solutions for old problems. For example, if we open for remote IT specialists, we can easily get 10x-100x more candidate, while having the same recruitment procedures.
During my presentation, I would like to share ideas of how AI-driven tools are and can be used in HR and Talent Management:
- Current possibilities of what is feasible (including some demos),
- Where we can take these tools in the forseeable future,
- What are the current technical challenges,
- Impact challenges, especially from ethical standpoint.

The session will focus on:

• AppDynamics platform with massively scalable big data infrastructure components to handle large numbers of events, metrics, and metadata.
• Machine learning baselining for APM metrics and advanced business analytics.
• Anomaly Detection and Automated Transaction Diagnostics
• Intuitive, powerful drill-down data visualizations

Going to the cloud sounds fantastic, giving people new opportunities - priceless. Public cloud gives you a lot of ready systems that you want to use. But when you have, already working environment based on your data center, a lot of data, and plenty of people using these tools, your work will have a lot of fascinating challenges. Transferring data, moving from one tool to another, selecting tools without consternation, cost optimization, policies, security, and many others.

Data has to be processed fast, so that a firm can react to changing business conditions in realtime. Streaming SQL gives us a
possibility to make stream processing available for a broader audience but also makes it easier to access data streams. This
presentation will not only give you an brief overview of the data and streaming architecture at InnoGames but also introduces you
to the idea of Streaming SQL in general and how it is implemented in Apache Flink. Furthermore it shows actual examples of how
to use Flink SQL so that you hopefully are inspired to consider this rather new technology to tackle your data challenges.

During this session we will reflect upon the importance of personalization in e-commerce. What challenges accurred as a results of bridging the gap between Google AlphaGo and real world? Furthermore, we will discuss the Vowpal Wabbit: the Swiss army knife of ML algos.

To sum up, we will introduce a case study exercise, during which participants will be creating a personalized user experience on a webpage.

Review of existing frameworks/approaches to data quality and principles behind designing data quality system at OLX.

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale at organizations including, Airbnb, Comcast, Fcaebook, FINRA, GrubHub, LinkedIn, Lyft, Netflix, Twitter, Uber, and Zalando, Presto has experienced unprecedented growth in popularity in both on premise and cloud deployments over Object Stores, HDFS, NoSQL, and RDBMS data stores.

Delta Lake, a storage layer originally invented by Databricks and recently open sourced, brings ACID capabilities to big datasets held in Object Storage. While initially designed for Spark, Delta Lake now supports multiple query compute engines. In particular, Starburst, developed a native integration for Presto that leverages Deltaspecific performance optimizations.

Join this session and hear how Starburst Presto deployed on Azure Kubernetes Service (AKS) serves as a fast SQL query engine versus data in ADLS, and enables query-time correlations between IoT data in Delta Lake, customer data in SQL Server, and web log data in Elasticsearch.

You will also gain best practice and real-life insights and lessons learned from production deployment of this integration.

At Simply Business we have built a rich stateful application using Kafka Streams to manage leads that are served by our 300-people strong UK call centre. This application combines many different data points from different services and has a few internal data stores to operate. Our initial design of the application became quite complex, so we had to make changes to ensure scalability and reliability . In the talk I'll present what techniques we used to simplify it.
But that's not all. I'll also talk about other key components like schema registry, which helped with the robustness of the solution and then lead scoring that helped to increase the return on the call spent by over 50%.

Weather forecasting is important in many fields, and minor improvements in accuracy can have a considerable business impact. Today, weather forecasting is performed using computationally expensive mathematical models based on the Navier-Stokes and mass continuity equations, the first law of thermodynamics, and the ideal gas law. These models simulate the physical world and are making use of expensive supercomputers. Alternatively, AI can be used to learn from data and produce forecasts in a fraction of the time the physical simulations require, and in many cases at a higher degree of accuracy. In this presentation, I will show how an AI was used to produce competitive forecasts, using state-of-the-art AI models and neural architecture search, and how I used React to prototype a weather forecasting service.

This session will be focused on the strategic side of our big data investments, from two major angles – instruments that we have – and partnerships – how we are impacting and enriching the external ecosystem. It will be a session in-between tech and business.

Analytical data processing has become the cornerstone of today's businesses success, and it is facilitated by Big Data platforms that offer virtually limitless scalability. However, minimizing the total cost of ownership (TCO) for the infrastructure can be challenging. By analyzing spot instance price history using ARIMA models, it is feasible to leverage the discounted prices of the cloud spot market with a limited risk of analytical job termination. In particular, we evaluated savings opportunities when using Amazon EC2 spot instances comparing to on-demand resources. During the presentation we show the evaluation of univariate spot price regression models to forecast future prices, and we confirm feasibility of short-term spot price prediction through real data from AWS. This confirms cost savings opportunities up to 80% compared to on-demand and within 1% of the absolute minimum.

We would like to invite you to discuss how Bolt migrated from batch and synchronous to real-time and asynchronous. During our session we will review and evaluate the obstacles we faced along the way and lessons we have learned. We will also focus on the unleashed value of real-time data.

A typical Software 2.0 improvement cycle at NoMagic:
* analyse when a data driven algorithm performs poorly
* gather and label data that will improve the algorithm or propose a modified algorithm
* train the model
* test the model
* deploy to production

Making this cycle as frictionless as possible was the focus of NoMagic ML team in 2020.
I will share with you what we achieved and how it changed the way we work at NoMagic.

During this session, we aim to review the technical and organisational challenges we faced, while building a complex AI-based app with a short time-to-market. How it was additionally influenced by the dispersion of involved teams around the world (10 time zones)? What unpredictable events that affected our plans, e.g. the COVID pandemic or the development vendor changing in the middle of the project? We will evaluate the examples of failures and successes. Lessons we have learned in the process. We invite you to a broader discussion.