The Data Mesh paradigm is a strong candidate to supersede the centralized data lake and data warehouse as the dominant architectural patterns in data and analytics. It promotes the concept of domain-focused Data Products which go beyond sharing of files and towards guarantees of quality and acknowledgment of data ownership.

Through personal experience with applying the Data Mesh concept in practice, as well as dedicated field research, the presenter discovered the most common pain points at different stages of the journey and identified successful approaches to overcome those challenges. In this talk, you will gain both technical and organizational insights ranging from companies that are just starting to promote a mindset shift of working with data, to companies that are already in the process of transforming their data infrastructure landscape, to advanced companies that are working on federated governance setups for a sustainable data-driven future.

Using components of the public cloud seems to be pretty straightforward. You have a nice and modern UI. If you're a console oriented guy (or freak) - there's even a built-in web shell to make you happy. Problems start to occur when each team uses various components, each in a different way and your task is to make those components meet the company's policies, normalize a process of a deployment and in the end - make this deployment user-friendly.
Let me show you how we've automated this process with the help of Terraform and provided a user-friendly way to build an infrastructure putting all of its complexities out of sight.
All the examples will be based on the infrastructure creation tasks for the big data processing projects.

#public #cloud #terraform #python #infrastructure #data

It is known that Data Science, Software Engineering, ML/AI are highly intertwined fields. When building future-proof advanced analytical solutions, originally devised by data scientists, not only certain best practices from Software Engineering are handy, e.g. pair programming, but also they help achieving e.g. low coupling and high cohesion for ML/AI systems. In this talk we show how this has been achieved on a concrete example of a predictive solution for understanding the impact of production process parameters on final product attributes.

#softwareenginnering #SEbestpractices #ArtificialIntelligence #MachineLearning #injdustry4.0 #crossfunctional

In this talk I’m going to present to you the design process behind Nielsens Data Auditing system, Life Line. From tracking and producing , to analysing and storing auditing information, using technologies such as Kafka, Avro, Spark, AWS Lambda functions and complex SQL queries. The data auditing project was one of main pillars in 2020, the extensive design process we went through paid off, and tremendously raised the quality of our data. We’re going to cover:
* A lot of data arrival and integrity pain points
* Designing your metadata and the use of AVRO
* Producing and consuming auditing data
* Designing and optimizing your auditing table - what does this data look like anyway?
* Creating an alert based monitoring system and some other add-ons
* Answering the most important question of all - is it the end of the day yet?

#data #auditing #kafka #architecture #sql

• User queries at an e-commerce site exhibit a plethora of information ranging from brands, size, intent or even desires. A better understanding of users intent leads to a better user experience.
• How can we exploit plain natural language texts to extract semantic and syntactic information. And how can such information help us towards improving the site behaviour?
• We will talk about enhanced language processing and understanding techniques for semantifying queries using advanced sequence learning techniques.
• We will also discuss how to design an offline evaluation to quantify the model performance at a large scale.

#nlp #queryunderstanding #ecommerce #sequencelearning #ml

Becoming a data-driven organization is a hot topic that keeps busy a lot of companies. Many leaders and executives see value in unleashing analytics potential and using it to impact business growth.
We are going to talk and share experiences about how we…
• democratize data across the different levels of the organization by consolidating, integrating and automating data workflows into a single Data Lake;
• designed and implemented cloud-based scalable & secure architecture;
• impacted business performance by unlocking various use cases;
• executed project based on the consolidated data for one of the largest nutrition brands.

#Datademocratization #businessimpact #Spark #datalake #digitization

Hearing aids nowadays can be treated as a cloud of IoT devices that produce a huge number of events.
However, getting from the devices to the report is not a simple path.
Especially that we need to additionally take into account GDPR, transformations or cost optimizations in Azure.

#Azure #EventHub #Datalake #Databricks #Lakehouse #ETL

The ability to analyze data in real time for mobile network is crucial for diagnostics and ensuring the quality of the service for end customers. To achieve this we have built a real-time ingestion and analytics platform that processes 2.2 billions messages a day from mobile networks hardware. During the talk we will show how we used Flink and Flink SQL to build this platform. The solution includes calculation of more than 5000 KPIs and 1500 aggregation defined in SQL, on 750 Kafka topics. We will describe how we manage Flink jobs at scale using Ververica and Kubernetes, how we monitor the platform using Clickhouse and what problems we need to overcome in the project.

#streaming #flink #real-time #operationalmonitoring #telco

• What is an analytics translator?
• What skills and competencies are required?
• How to build a data-driven business with analytics translators?
• Do you fit the profile of an analytics translator?
• Do you like quizzes?

#AnalyticsTranslator  #DataDrivenBusiness #Transformatio #PeopleAnalytics #Data-Science

Microsoft is a data driven company. All client-side software is well instrumented and emits telemetry. Designing, developing and operating (DevOps model) a big data pipeline gathering this data at the Microsoft scale (the pipeline has: 100k+ Azure cores, 13 Data Centers, hundreds of PBs) is a great learning opportunity. In this presentation I will show what we've learned over the last 7 years and describe the DevOps process we use. This will be a journey spanning: our design principles, testing approach, ops mindset (monitoring, automation, continuous-improvement), rollout across 13 Data Centers strategy and more.

#devops #telemetry #continuousimprovement #scale

- Common issues with data lakes
- What is Apache Iceberg? and what problems does it solve
- Building CDC archive at Shopify using Iceberg
- Management / considerations when using Iceberg
- Brief intro into whats next on deck for Shopify + Iceberg (Type-1 dimensions using Iceberg's V2 spec with row-level deletion)

#iceberg #datalake #columnardata #dataplatform #CDC

TerrariumDB is a column and row store engine designed specifically for behavioral intelligence, real-time data processing, and is the core of the Synerise platform. It simultaneously processes data heavy analytics while executing various business scenarios in realtime. TerrariumDB was designed to analyse behavioural data, where data order and time are important to make business decisions. During the talk, there are described why we are developing our distributed database engine, where were the challenges and pitfalls, for which use cases does TerrariumDB fits best, and how it handles billions of queries per day where 99 percentile does matter.

#Streaming Database #RealtimeAnalytics #ML #OperationalAnalytics

What is a feature store ? Why do we need it ? How to use it ? In this session, I would like to show how to use the Feast feature store to build a complete MLOps process. Starting with fetching historical data and model training, thought model versioning and deployment process and finally online features materialization and real-time model inference.

#featurestore #feast #mlops

Keep up with the increasing trend in the software world! Learn more about Digital Twins and what role Big Data/AI/XR/IoT and Robotics play in this story. Digital Twin is a broad topic with a number of business areas where it can be applied and wide list of technologies under the hood. On this session you will learn about a framework, intended to handle underlying complexity, facilitate design and adoption of Digital Twins; technical aspects and considerations; real-life examples, covering success stories and delivered value. This session can be interesting for different groups: Architects, tech-savvy business leaders, Product Managers, technology experts and consultants.

#DigitalTwins #AI/XR/IoT #Robotics

Let's discover together how COVID-19 affected the cloud adoption and what are the most common cloud security mistakes that team are doing.

#security #cloud #adotion #architecture

Data Engineering used to be a hard problem that only people with a software engineering background could solve. Additionally, a number of use cases and analytics needs in companies waiting to be solved outnumber data engineers teams, burdening them and making business departments wait for ages for their use cases to be implemented. On the other side, Business departments would love to implement the data pipelines themselves. But for a long time, they couldn't do it well, mainly because they lacked the engineering skill required for them to work efficiently and deliver technical quality. Today, we witness the change of this obstacle getting away thanks to the maturity of modern data platforms and thanks to tools that make it easy to implement pipelines according to the best DataOps practices. Like DBT if you have heard about it.

Tools like DBT are great but are just puzzles of the bigger picture. What we at GetInData do is take those puzzles, these advancements of data tools, and combine them into a coherent, unified data pipelines framework that guides analytics engineers by hand in developing the pipelines end-to-end. From the idea to production. 

Come to our presentation to listen about modern data platforms, DBT as well as our framework.

#analytics engineering, #self-service data pipelines, #cloud, #DBT, #SQL

Stream processing has become vastly popular in the past couple of years. As you're already familiar with Kubernetes, you have the necessary toolbox for deploying and operating low latency mission-critical streaming data pipelines with Apache Flink. After this talk, you'll get a broad understanding of how the complex stream processing applications fit into the cloud-native era and get comfortable with introducing them to your organization.

Over the last years, multiple data transformation projects included significant investments to provide a centralized data lake, which could swallow any type and quantity. There was a promise for a single point of data for analytics and democratization of data. However, the concept brought many expensive problems in both implementation and maintenance. Other issues include a lack of clear ownership and division between data contexts, which lowered the return on investment.

These issues were well known to software architects. Many lessons could be shared from software architecture patterns, including sociotechnical, strategic, and implementation. For example, the concept of Data Mesh connects really well with Domain-Driven Design on multiple levels; both are completing each other. The point of our discussion will be to share learnings from applying software architecture patterns to data architecture paradigms.

Processing Electronic Medical Records (EMRs) is super complex in itself, but it gets infinitely more complex when you want to do it at scale, covering multiple countries (and even continents). How did IQVIA EMRs Factory team build the probably largest global Electronic Medical Records data platform in the world? How was a combination of Cloudera, Spark, Kafka and Looker used to build a solid foundation and where are we taking it now on a hybrid-cloud architecture? How and why are we evolving - adopting Snowflake and Databricks to enable rapid business growth? We'll talk about what drove our design decisions (including requirements critical in healthcare like privacy, security and governance) and what we learned over the years running and modernising the platform. Importance od data residency and its impact on the system architecture.

#Cloudera #Spark #Scala #Java #Looker #Cloud #Snowflake #Databricks

Kubernetes is an excellent vehicle for driving your analytics, big-data, and ML workloads, but a one-size-fits-all configuration won’t give you all the benefits platform provides.
I want to walk you through the things to consider when creating a high-performance Kubernetes cluster for your workloads, from resource utilization, topology, application settings, and consumption of hardware components.

One of the more annoying challenges product businesses face is in establishing causal relationships while trying to determine the impact of making product changes. Frequently, A/B testing is used to this end, but that is expensive and time consuming for engineering and design teams, and often requires tooling as well as randomization. Here at Willa (a payments and invoicing app for US-based freelancers), we used an econometric technique called Difference-In-Differences (D-I-D) regression to tease out the causal impact of simplifying the invoicing process using our app, on the invoice creation rate per user. We took advantage of natural variation in product usage between different kinds of users, and were able to reach statistically significant results cheaper and faster than via A/B testing.

The technologies used were BigQuery and Jupyter/Databricks, and GCP more broadly, although the approach is platform agnostic. Overall, the presentation will be interdisciplinary in nature, so anyone interested in economics, data science and engineering, cloud technologies and general dynamics of product businesses is welcome to attend.

Streaming trillions of messages per day (and meeting the agreeed SLA's) was a challenging job. In this session I will present how are we handling some critical aspects like reliability, performance, skewed data, debugging and performance testing. This session is targeted to software engineers passioned about performance/handling large amount of data.

#spark #streaming #trillions

Retrospective overview of the process, methodology and implementation details of Data Platform in new and young digital company vs old fashion global wall street financial instirution.

#BigData #TechLeadership #Global #Projectimplemenations

What's the optimal strategy towards operations team building in the multi- and hybrid cloud era? Are we going to see all ops-related engineering roles like DevOps, DataOps or MLOps to specialize even more and gravitate out of each other in the responsibilities spectrum? Or should we rather expect them to consolidate, which will mandate a new way of leveraging cross-discipline skilled CloudOps talent as a foundation of the operations teams? And how does it actually relate to hyper automation, software-defined infrastructure and massive adoption of data analysis across organizations?
Let's meet and talk about the challenges we will face as Big Data specialists and managers in this vivid and emerging area of high-complexity technology, limited human resources and endless business opportunities.

As a global bank we need to follow strict financial regulations and certifications. Our goal is an agile organization, where developers can provision and operate cloud resources, while we keep in control and stays compliant. We provide a controlled development environment, where guardrails force high level security, while staying agile. This includes custom pipeline tasks, template repositories and automated change control. Get an insight in how we control resources, security and cost in automated processes

#azure #devops #compliant #agile #selfservice

The targeting of online advertising is like gold seeking - in our case, the interpretable machine learning models serve as the diggers that discover the most profitable targeting criteria for a particular campaign to maximize the final profit. In an environment consisting of multiple advertising products and traffic counted in millions of PV daily, they need to be fast and reliable to find the best spot before the others do.

In this talk, I will present how we bridged the gap between business goals of KPI maximization, while ensuring scalability and stability of the solution. The journey starts with presenting the kind of gold that we search for, defined by the business context of online advertising. Next, I will present our gold-digging machine learning techniques and discuss technical details of architecture design and implementation in the AWS environment integrated with our in-house ad server. Finally, I will discuss the performance of the deployed models that are constantly monitored on production.

#Machine Learning #Realworldsystem #Adoptimization #Modelinterpretability

• What is a framework?
• How can the framework help you achieve a faster start?
• How can working with frameworks help you achieve long-term speed and agility?

Since Data infrastructure was set up at Uber, we have been managing our own server fleet. Age old practices of managing hosts posed several challenges that stood in the way of innovation.
We did an entire ground up re-architecture of our deployment stack, embraced the DevOps model and automated away operational tasks. This effort gained us a lot of benefits across several areas (efficiency, security, etc) and strategically positioned us to leverage the cloud.

In this talk, we'll briefly discuss the challenges we faced as part of our containerization journey, our strategies/solutions to overcome these challenges and mainly focus on lessons we learned along the way.

#containerization #architecture #devops #automation #massive-migration

The primary source of bias in machine learning is not in the algorithms deployed, but rather the data used as input to build the  predictive models. In this talk we will discuss why this is a huge problem and what to do about it. Different sources of bias will be identified along with possible solutions for remedying the situation when deploying machine learning. We will also speak about the importance of transparency when using machine learning to predict outcomes that impact critical decisions.
• Learn why most predictive models are biased.
• Learn about the sources of bias in predictive models.
• Learn how to reduce the negative impact of potential bias in predictive models.

#MachineLearning #ArtificialIntelligence #EliminationofBias