And thank you everyone for coming to today's session. Before we delve into our discussion, let me first give a very brief introduction about myself.

I am Ilana Makover and I am a machine learning engineer at BlueVine. Now let's get back to the real reason everyone's here, which is to let your data tell its story.

Let's talk about a fictional streaming service.

Let's call this service Flicks. Every time you hit that play button, Flicks quietly collects information.

It tracks things like how long you watched, whether you paused, or skipped ahead. One data point they store, let's say it's called average watch duration.

That one data point behind the scenes has a whole story. When you press Flick's play button, an event is sent by the playback service.

That event is then collected into the streaming logs. Then a daily analytics job reads that event to calculate the user's average watch time.

Then the recommendation model reads this data point, average watch duration, in addition to others, to determine what TV shows and movies to suggest to you tomorrow.

Next, your marketing team reads this data point, average watch duration, in addition to others, to decide which email you receive in the next email campaign.

And finally, the BI dashboard reads this data point, average watch duration, aggregates over all users, and then shows executive engagement trends.

This one data point, average watch duration, quietly flows through many different systems, powering recommendations, marketing, and business decisions.

Oftentimes, people look at data at face value. They check things like the data type, data integrity, and trends in the data. But Many forget to zoom out and look at the bigger picture.

Let's go back to our Flicks example. Let's say Flick changed its data point of average watch duration from seconds to minutes.

Would you know the downstream impact of such a change? Would you know which teams have to be involved in such a change?

Which models have to be retrained, dashboards need to be updated, and let's say Flick's recommendation engine starts acting weird suddenly.

Would you be able to identify upstream causes of such behavior and trace it back to this updated data point? And finally, this is an aspect some of us overlook.

Is this data even used? Perhaps Flick is still getting data from an external provider, yet no one is using it.

Perhaps Flick is generating data for a research project that ended ages ago. Now there is data floating around, taking up space, costing money.

This is where data lineage enters the picture. Data lineage allows us to understand and even visualize this network of dependencies and allows us to quickly answer these questions in front of you.

Data lineage is a story of listening to your data as it jumps across your data pipeline.

Listening to things like where it came from, how it moved, and how it was used.

These stories allow us to see the data's true value. Today we'll explore how to listen to that story and together we'll build a lightweight data lineage solution.

Our agenda for today will be as follows. We will first discuss what we want to listen to.

This will answer three major questions: What data? Which data interactions? And what details? Then we will go into our data lineage solution step by step, which of course answers the above questions.

And finally, we'll top it off with a great visualization.

The first step when we want to plan our data lineage solution is to think about what data you’d like to track. For better or for worse, not all data is created equal.

Some data might be more useful to your organization than others.

Data can be obtained from an external provider, and it can be generated in-house. Right? Companies may want to know some information about their users.

They could buy this information from an external provider, and other information they can generate in-house. Do you want to track data obtained from an external provider?

Data generated in-house? Data can be stored across multiple databases and data warehouses.

Do you want to track data from specific databases? Data from specific data warehouses? Some of them, all of them.

For our example purposes, we will say we want to track all of our processes that read and write our in-house generated data that is stored in our tabular DB and our graph.

Amazing. Now we get on to our next section, which is data interactions. We will now start exploring data interactions in our system. Most data journeys are composed of many interactions, whether it be reading from a database, dumping a file onto some cloud storage, or interacting with a model.

To understand the data journey, we will need to identify our interactions between agents in our system and the data itself. You could say these data pipeline interactions help shape the data's character, like in a story.

Let’s walk back to our original example, which includes a recommendation model.

This model makes a decision on what TV shows and movies to recommend to you based on input data.

This interaction between data and the model is something we want to track. This is vital. In addition to tracking this specific interaction between this data and this specific model, we also want to track other interactions between our data and all of our models.

So we want to find the common entry point where this data interaction happens, where all of our data and our models interact in this one common area.

And here, our engineering best practices come into play. We will be under the assumption that your team created some generic API to handle model inference that is model agnostic.

Amazing. And this generic API, right, it could be that some models call some of the methods here, while other models call other methods, but at the end of the day, based on our engineering best practices, all models will begin their model inference journey at a common entry point.

And for our purposes, let’s say it all begins.

All the models, their common entry point is in this predict method. This predict method is where we want to locate our data interaction for data models.

Now we have located one such example of a data interaction. Now we simply have to locate all the other major data interactions for your APIs, microservices, and internal tools which leverage your data.

This is the key. This will make implementing your solution much easier.

Once you have located and leveraged these central entry points, you can then record valuable information about your data and its current location in its journey.

Once you localize this part, the storytelling bit will seamlessly fall into place. One final aspect we need to discuss is categorization.

Every data interaction will be categorized in a very simplistic manner. Either it is a read interaction or it is a write interaction.

Either it is consuming data, reading data, or writing data; it is generating data.

Now let's tie our section together here with a clean example, and this example will follow us through today's session.

So let's say we have a central method called retabular DB, and our group uses this method in order to query from our tabular database.

This is our central entry point. We have now localized this. Now we want to categorize it as a read interaction because it is consuming data.

It is reading data. Again, this method is used to query our tabular databases.

The hardest part of all this will be deciding what information is critical to understanding your data. Every story needs a good genesis. What is your data's hero's journey?

We will go over some possible examples now, but keep in mind you will have to answer the overarching question of what information will be useful to understand your data.

Again, some examples. What process is requesting this data? If we go back to our example, we'll say a recommendation engine is requesting this data.

When did this request occur? November 4th, 2025.

Which main entry point was called? The retabular DB method. What data was requested? We can say two columns were requested: average watch duration and average time paused.

What is the source of the data? We can say a table called user watching metrics.

What is the source type of the data? It is a database. And finally, what type of interaction is this? This is a read interaction.

Now before we dig into the implementation aspect, let's take a little step back and discuss the functional requirements of our data lineage solution.

Data lineage should trace, visualize, and understand the journey of data across our systems from the moment that data is consumed through its transformations to its ultimate points of use. We have five functional requirements. The first one is that we want to list every possible downstream process, model, or system that uses a data point.

Next, we want to identify exactly where a data point starts. Who creates this data point? Which process, model, or system creates this data point? Next, we want to visualize the full data journey. This will help us both understand the possible downstream impact of data failure or data change and also identify upstream causes of odd behavior.

Put simply, we want to understand if a data point is even used, and if it’s not used, let's eliminate it and improve efficiency.

And finally, our last functional requirement: we want lineage to be automatically collected and updated as our jobs evolve, as our systems evolve. So we want to have the latest view of our pipelines.

Now we get to our high-level solution. Now we can focus on listening to our data as it moves across our pipeline. In front of you is the lightweight in-house data lineage solution. Our solution will require two things: one of which is a queuing system of your choice and a table.

The queuing system will need to have a mechanism to ensure that no two processes interfere with each other.

Identical messages can be queued. So to break this down even further, let's say a message is in the queue. An identical message cannot be queued at this point.

And the table will store all of our data interactions. Now let's go step by step into our solution.

Once the information details are agreed upon and finalized, the main entry points which we identified earlier, each main entry point will have to obtain the relevant data details, and they will get these data details, package them as one message, and send it to a queue.

We mentioned an entry point earlier called read tabular DB. As mentioned, all of our queries to our tabular database enter this location.

And let's say our recommendation engine calls this entry point three different times for three different queries.

Let's follow the light green here. We can see three different times for three different queries. That means we see three different messages in the raw interactions part.

So now we have three different messages. Let's break this down even further. Let's talk about a very specific example.

Let's say our read tabular DB gets this query as input. This query is very straightforward, very simple.

We can see select two columns. One is called average watch duration, and the other is average time paused from user watching metrics. The read tabular DB entry point has to get this query, parse it, extract the relevant information in order to write something like this to extract this message from that data interaction.

And we see here again this is all happening in read tabular DB. It takes this one query. It extracts the relevant data details.

It extracts that the process that called this entry point is called the recommendation engine. It extracts the request time, which is November 4th, 2025.

It extracts two columns that were called. One is called average watch duration, and the other is called average time paused. It also extracts that the table that was requested.

The source is called user watching metrics, etc. This is an example of one process of one entry point that has to obtain this information and send it as a message to a queue.

This is all happening at the same time that the existing process is happening.

So, the relational database and all other main entry points have to extract this information and send it as a message to a queue very quickly because once this happens, it needs to continue on with its regularly scheduled programming.

Now, this one message here can be a bit sloppy and not so easy to read.

We will call it the data's first draft of storytelling. Right? It looks clean to you now, but imagine the query were slightly different in our case.

Imagine we were querying 20 columns. Then the message would look very sloppy, right?

So now we have a very sloppy message, a lot of information. And now we get on to our next part of the process.

Now we have a new independent process. This process, what it does, is it dequeues from that queue which has all of our messages that talk about many data interactions, and this process takes a message and splits it into more bite-sized pieces where each piece talks about exactly one data point.

Right? So why do we do this? Because When you want to tell a story, you want to talk clearly and concisely.

Let's go back to our example at this stage. Our example of one message, by the time it gets to this independent process, it gets split into two different messages.

If we look at the top part of the screen, this message is more or less the same message that we have seen earlier.

The only difference is that what is marked in red now talks about exactly one data point.

Now we only talk about the column average watch duration in the message, and then if we look at the bottom message, we only talk about average time paused.

So again, this part of the process simply takes a message and breaks it down into multiple messages where each message talks about exactly one data point.

Both these messages are now sent to a new queue. Now we get to one of the last parts of the architecture.

Now we get to populate the data lineage table. This new independent process will dequeue messages from that second queue, and it will update and insert these messages into a table.

This is how it will look like. Those two messages that we have seen in the previous step will be two rows now in our data lineage table.

Amazing. We can see here we have in our table the process which is called the recommendation engine. The entry point is called the tabular DB.

The data point is average time paused. The source is user watching metrics. Amazing. Again, this is done in update and insert format when we interact with the table.

This is an example of just that one message that we saw turned into two messages and now into two rows.

But imagine all of our data interactions for all of our processes are in the table. Now we'll have much more information and with this information at the moment we can do so much with this when it's just in the table format.

Ilana Makover: Let Your Data Tell Its Story | PyData Tel Aviv 2025

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.

We aim to be an accessible, community-driven conference, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.

Ilana Makover: Let Your Data Tell Its Story | PyData Tel Aviv 2025

Scripts