Pain points are frequently encountered when aiming to orchestrate dbt (Core or Cloud) in a self-serve manner in a Data Mesh architecture. As part of this webinar we will examine a number of methods of orchestrating dbt including:

• YML files that create Airflow DAGs.
• Terraforming dbt Cloud jobs.
• YML files that create dbt Cloud jobs via the dbt Cloud API.

These options offer a number of advantages:

• When these YML files live in the dbt repo, dbt developers can orchestrate their project from within the project itself, this keeps orchestration as close to the code as possible and allows dbt project changes to be rolled out immediately.
• As the file is YML, and dbt already uses YML, dbt developers are not forced to learn a new programming language. Hence they are comfortable updating their orchestration settings without any dependency on engineering teams.
• Terraform is already a widely used Infrastructure-as-Code (IaC) tool, adding dbt Cloud jobs to Terraform encourages developers to treat their jobs as infrastructure.
• In a Data Mesh setup, self-service and scalability are requirements. Any new dbt project can include the YML file or Terraform code and automatically get their dbt project orchestrated.

My hope is that by sharing my experiences I can inspire other data practitioners to democratise their dbt orchestration and make their organisations more data driven.

Data Warehouses (RDBMS-OLAP) have been foundational to democratizing data and enabling analytical use cases such as business intelligence and reporting for many years. However, RDBMS-OLAP systems present some well-known challenges. Cloud data lakes have addressed some of the shortcomings of RDBMS-OLAP systems, but they present their own set of challenges. More recently, organizations have often followed a two-tier architectural approach to take advantage of both these platforms, leveraging both cloud data lakes and RDBMS-OLAP systems. However, this approach brings additional challenges, complexities, and overhead. This session will present how a data lakehouse, a new architectural approach, achieves the same benefits of an RDBMS-OLAP and cloud data lake combined, while also providing additional advantages. Through this session, the idea would be to break down the architecture of a data lakehouse & draw a comparison to the traditional data warehouse. We will focus on the following aspects:

• defining a standard terminology for data warehouse - breaking down/categorizing DWH into 3 parts:
  • technical components - storage, compute, table/file format, etc.
  • technical capabilities - concurrency, latency, ad hoc queries, etc.
  • tech-independent practices - modeling, ETL, MDM, SCD, etc.
• show how a lakehouse satisfies these 3 components & the additional value that it brings

In this webinar you will get:

• Technical deep-dive of the Lakehouse architecture
• Comparison of the technical components, capabilities and technology-independent practices of data warehouses & lakehouses to prove that they are similar
• Additional benefits of a lakehouse architecture
• How to get started with architecting lakehouses