We're going to take a look at a demo of Unity Catalog's data lineage service, which automatically captures data lineage across all languages, down to the table and even column level. But first, why is data lineage so important? Well, it helps with use cases across three broad areas.

The first is compliance. Data lineage can help us track the spread of sensitive data across the lake. From a discovery perspective, this will help analysts, data scientists, and ML engineers understand the context and trustworthiness of data before they use it in analytics.

And from a data observability perspective, this will help data engineers understand the impact of changes, failures, and data quality issues, as well as debug these issues as they arise. It's this last topic that we're going to use to explore a demo of the

Unity Catalog service. In this scenario, we're going to imagine that a data analyst has reported to me that this table, called `Car Sales Features`, is suffering from poor data quality. Now, this is a table that describes cars, their sales, and the different features that they have.

The first thing I might want to do is begin by running a data profile. When I do that, I can see that this one particular column, called `premium_feature_set`, is missing 50% of the time. So I might want to go about debugging this particular issue.

So I can start my analysis by searching for the table and opening it up in Data Explorer. From there, I can dig into the lineage. So here's my table on the right-hand side, and I can see that it's a combination of two other tables. Furthermore, I can expand the schema.

And now Unity Catalog is able to capture data lineage down to the column level and display that in the same graph. So we can see here that my column called "premium feature set" is derived from a column of the same name in this upstream table. So I might want to dig into the lineage

of this particular table a little bit more. So I can expand that a level further, and again I can explore the lineage of this one column that I'm interested in. So that's a little bit more revealing. I can see that the column I'm interested in is made up of four other columns: the

X, four, and five, and one and two from these two other tables. Well, let's have a look at how these tables are loaded, and we can do that because Unity Catalog captures the different jobs and workflows that are responsible for reading and writing to these tables.

So here I can see that there's one particular workflow which is responsible for loading this table, and I can open that up. I can see that most of the time this job runs without error, as indicated by the green bars here, but the last time this ran, which was

today, we have a red bar which indicates an error. So actually, I can drill down into this particular run of the job and find out my issue. And at this point in time, I could go ahead and contact the owner of this job or actually dig into the notebook and

diagnose the issue myself. So hopefully you've seen in this demo how data lineage can be a really effective tool in helping data engineers understand the root cause of data quality issues.

Automated Data Lineage with Unity Catalog

Databricks is the Data and AI company. More than 10,000 organizations worldwide — including Block, Comcast, Conde Nast, Rivian, and Shell, and over 60% of the Fortune 500 — rely on the Databricks Data Intelligence Platform to take control of their data and put it to work with AI. Databricks is headquartered in San Francisco, with offices around the globe, and was founded by the original creators of Lakehouse, Apache Spark™, Delta Lake and MLflow. 
