We're going to take a look at a demo of Unity Catalog's data lineage service, which automatically captures data lineage across all languages, down to the table and even column level. But first, why is data lineage so important? Well, it helps with use cases across three broad areas.

The first is compliance. Data lineage can help us track the spread of sensitive data across the lake. From a discovery perspective, this will help analysts, data scientists, and ML engineers understand the context and trustworthiness of data before they use it in analytics.

And from a data observability perspective, this will help data engineers understand the impact of changes, failures, and data quality issues, as well as debug these issues as they arise. It's this last topic that we're going to use to explore a demo of the

Unity Catalog service. In this scenario, we're going to imagine that a data analyst has reported to me that this table, called `Car Sales Features`, is suffering from poor data quality. Now, this is a table that describes cars, their sales, and the different features that they have.

The first thing I might want to do is begin by running a data profile. When I do that, I can see that this one particular column, called `premium_feature_set`, is missing 50% of the time. So I might want to go about debugging this particular issue.

So I can start my analysis by searching for the table and opening it up in Data Explorer. From there, I can dig into the lineage. So here's my table on the right-hand side, and I can see that it's a combination of two other tables. Furthermore, I can expand the schema.

And now Unity Catalog is able to capture data lineage down to the column level and display that in the same graph. So we can see here that my column called "premium feature set" is derived from a column of the same name in this upstream table. So I might want to dig into the lineage

of this particular table a little bit more. So I can expand that a level further, and again I can explore the lineage of this one column that I'm interested in. So that's a little bit more revealing. I can see that the column I'm interested in is made up of four other columns: the

X, four, and five, and one and two from these two other tables. Well, let's have a look at how these tables are loaded, and we can do that because Unity Catalog captures the different jobs and workflows that are responsible for reading and writing to these tables.

So here I can see that there's one particular workflow which is responsible for loading this table, and I can open that up. I can see that most of the time this job runs without error, as indicated by the green bars here, but the last time this ran, which was

today, we have a red bar which indicates an error. So actually, I can drill down into this particular run of the job and find out my issue. And at this point in time, I could go ahead and contact the owner of this job or actually dig into the notebook and

diagnose the issue myself. So hopefully you've seen in this demo how data lineage can be a really effective tool in helping data engineers understand the root cause of data quality issues.

Automated Data Lineage with Unity Catalog

Databricks is the Data and AI company. More than 10,000 organizations worldwide — including Block, Comcast, Conde Nast, Rivian, and Shell, and over 60% of the Fortune 500 — rely on the Databricks Data Intelligence Platform to take control of their data and put it to work with AI. Databricks is headquartered in San Francisco, with offices around the globe, and was founded by the original creators of Lakehouse, Apache Spark™, Delta Lake and MLflow. 


Databricks

데이터 계보의 중요성과 그 활용

민감한 데이터 확산을 추적하여 규정 준수에 도움을 줍니다.

분석가가 데이터 컨텍스트와 신뢰성을 이해하는 데 도움을 줍니다.

데이터 엔지니어가 변경, 장애 및 데이터 품질 문제의 영향을 이해하는 데 도움을 줍니다.

데이터 품질 문제를 쉽게 디버깅할 수 있습니다.

데모 시나리오: '자동차 판매 기능' 테이블의 데이터 품질 불량 디버깅하기

데이터 프로필에 'premium_feature_set' 열에 50% 누락된 데이터가 표시됩니다.

데이터 탐색기를 사용한 데이터 계보 탐색은 다른 두 테이블에서 테이블의 출처를 표시합니다.

열 수준의 계보에는 업스트림 테이블에서 파생된 'premium_feature_set'이 표시됩니다.

추가 계보 탐색을 통해 두 개의 표에 있는 다른 4개의 열에서 열의 구성을 확인할 수 있습니다.

Unity 카탈로그는 테이블 읽기 및 쓰기를 위한 작업과 워크플로를 캡처합니다.

테이블 로드를 담당하는 워크플로우에 최근 오류가 표시됩니다.

작업 소유자에게 연락하거나 노트북을 검토하여 오류를 조사할 수 있습니다.

결론 효과적인 디버깅 도구로서의 데이터 계보

Automated Data Lineage with Unity Catalog

Automated Data Lineage with Unity Catalog

스크립트