We are going to be hearing from Avik Basu, who is a data scientist and software engineer on causal machine learning. All right.

Okay. Loud and clear. Cool. Yeah. Hey everyone. Uh, you know, I hope you're doing well. Uh, this is probably the last like talk of today, I think.

>> Yes. >> Okay. So, I'm so I hope you guys are not too tired.

This always happens to me, but but it's fine. So, I mean, today I'm going to talk about, you know, uh, you know, causal thinking in machine learning and, uh, now now this is something, you know, full disclosure, this is something new to me as well, but it's something that I have gained real interest lately.

So we'll dive some, uh, dig into some, you know, concepts together and see how that goes.

Right. So, a little bit about me. Um, I'm based in the Bay Area. I am a data scientist at Intuit.

Um, I actually call myself a full stack data scientist because I I kind of do the whole the whole thing, the whole pipeline.

I'm also an editor at PyOpenSci, which is a nonprofit for broadening the acceptance of scientific Python packages.

Um, and I come from, as I said, come from both an engineering and a data science background. I'm an avid gamer. So, I love PS5 and, you know, Steam Deck.

Like, for some reason, I forgot my Steam Deck. I don't know why, but I'm not very happy about that.

Uh, I love, uh, you know, tennis and, uh, you know, soccer as well. Um, but yeah, I play tennis. I just love it.

Uh, and then driving is something that I feel is very therapeutic, given some conditions are met. Uh, you know, conditions like these.

Miata is always the answer for me. So, yeah, okay, the best condition is that it has to be a stick shift. It has to be a rear-wheel drive and it has to be lightweight.

So, with that, let's move forward. So, the agenda for today is that, uh, and really the goal of today's talk is to give you an intuition behind causal thinking and how you can use it for your own use cases.

So, we'll go over the correlation versus causation, the classic, uh, you know, concept. And then some, you know, concepts which are very specific to, you know, causal inference.

And then we'll talk about uplift modeling, uh, and finally, we'll go through evaluation and some of the key takeaways.

So, um, let me start with a scenario many of you have probably encountered, okay? And it's, uh, and it's basically this, okay?

We build a model to predict, you know, customer churn, okay? And, uh, it's got a great F1 score of 0.9, and you're feeling good about it.

Um, and you can see that the model learned that, you know, probability of churn given call support is 0.45, and the probability of churn given no call is, you know, 0.15, okay?

Now, um, given this, you're very confident about this pattern, right? So, you can actually have an insight that customers who call support are 3x more likely to churn, right?

Um, so, well, I mean, so I mean, someone from business team says that, great, you know, how about we reduce the support calls to prevent churn, right?

Um, and this is where we need to pause because our model can be 90% accurate, but you know, still lead to completely wrong business results.

Okay, so the question we need to really ask ourselves is that did support calls cause churn or did unhappy customers call support?

Okay, so this is the classic example of, you know, correlation versus causation there.

Right? So the key insight here is that high accuracy does not really translate to an actionable insight there. Okay. So, um, now let's talk about the or in other words, let's just formalize the distinction between them.

Right? So we have the predictive models where it estimates, you know, expectation of Y given X.

Okay. And it answers the question who will buy and it also answers, you know, what was going to happen.

Okay. So who will buy and what's going to happen. And in case of the causal model, you have a different kind of a whole thinking basically.

So you're actually estimating what is the uplift.

The uplift meaning that what is the expectation of Y given that there is a treatment minus of Y given that there is not a treatment.

We'll actually go into these notations later on, but that's just to give the basic idea. And the question is that who will buy because of the treatment?

Okay. And the answer that we are trying to look is that what is the effect of our action.

So as you can see that these are fundamentally different questions requiring different approaches, right? So let's go through this problem and we'll we'll we'll actually go through this, you know, same problem, you know, over and over in this talk.

So let's just get this, you know very clear, right?

So we take a concrete example, um, and in this that you run an email marketing campaign and want to affect and want to really measure its effectiveness.

Okay. So, um, you have a prediction model, and this prediction model basically says that, you know, 20% of customers who got emails purchased.

Okay, but this is just an observation, by the way, right? And then in the intervention, you have, you know, something like, okay, sending the email increases the purchase rate by 5%.

Okay, that's the actual effect of the intervention. So you can see that the expectation of purchase given T equals 1, which means the email is sent, minus expectation of purchase given that it's the control, which means no email is sent, uh, is, you know, 0.05, okay, which means that, you know, sending email increases the purchase rate by 5%.

So same data, but we're asking, you know, some different questions here, right?

So, um, now, you know, here's your, you know, campaign results, right? So you have your received email, which is the treatment group, and the purchase rate is 20%.

No email, you have the control group where the purchase rate is 8%. Right? So it's a simple math, you know, email boosts, you know, purchases by 12 percentage points, and we are just ready to ship it.

That's that's what, you know, I would I would actually vote for, right? So email boosts purchases by 12%. So let's just ship it.

Well, not so fast because we need to really figure out who got the emails in the first place, right? So, um, the problem is that your marketing team was not sending these emails randomly.

They actually targeted customers who were recently active and people with very high engagement history.

Okay. So that 12% difference that we saw, it's not actually from that email that you sent.

Okay? Because some of it is that, you know, engaged customers are going to buy more anyway. So the problem here is that, you know, that the observed 12% difference is actually, you know, some true effect.

It is there, you know, no doubt about it.

But then there is also a big amount of selection bias that we need to really, you know, subtract to get the true effect.

So the true effect is what we want. So let's look at the real picture of the selection bias. Okay. So, uh, now we actually break it down by the customer type.

So here you can see that the high engagement and the low engagement are actually features. So you can, you can think of it as a feature to your model, right?

And let's just focus by each customer type. So if you look at the high engagement, so the people who received the email, they actually did, you know, like, you know, 20% purchase.

And then the no email group, which is the control, they did 16%.

So the true effect of the high engagement is actually 4% if you subtract the two, right? And the same and you know, a similar goes for the low engagement as well.

It's, you know, 6%, uh, you know, for the treatment group and then the 3% for the no email group, right?

So it is actually 3%. So, you know, here one thing to really, uh, you know, think about is that the customer engagement feature is actually a confounder.

It affects both who gets the treatment or, you know, who gets the email in this case, and who purchases.

So it affects both the target variable and also the treatment variable. All right. So, I mean, how do we, how do we really solve this problem?

Well, the best way to do that, or I would say the gold standard of this, is randomized control trials, right?

Or some randomized A/B tests. So, the key idea is that it just ensures that treatment and control groups are identical at the baseline.

So, we're not going to, we're not going to do the same mistake that the marketing team did last time, right?

That we need to make sure that we assign these things randomly, right?

So what really happens after randomization is that the treatment group, well, random 50% of all customers, the control group, the same thing. Now, when groups are identical on average, we, we can really see the true effect.

Okay. So, you know, any difference we observe is actually the true effect of the email.

Right? So randomization breaks the link between the customer characteristics and the treatment assignment.

So, but what is the problem then? Well, the problem is that A/B tests are really expensive and depending on the industry you are, it can be very unethical as well.

Okay, they also take much more time to run. I mean, by unethical, you mean, I, you can't just randomize medical treatments, right?

And often we've already collected the data as well. So the campaign already has happened.

Now we need to make use of that, that data as well. So what do we do? Well, you know, first, thing is that we need to ask, we need to ask this question that how do we estimate treatment effects from observational data.

Okay, and this is where uplift modeling really comes in.

Okay. So, um there are actually three components of, you know, of uplift modeling that we need to really take care of.

The first thing is the potential outcomes. How to think about treatment effects, right? So there is there are some distinctions that we should really know about when we think about that.

The second one is uh, you know, conditional average treatment effects or, you know, CATE, uh where we are trying to, so this is exactly what we're trying to estimate here.

Okay. And then the third part is is the metalearners, is actually the how, how do we estimate uh the conditional average treatment effects.

So let's just dive uh right in into the potential outcomes framework.

So this is really about a framework for thinking about treatment effects. The idea is pretty simple but uh actually powerful, right? So what do we do for each individual, and this comes to the notation that I was referring to earlier.

Um, there are actually two potential outcomes. Okay, Y of zero, okay, for a person I, right, which is that they are not treated, which is they are in the control group.

Y of one, um, okay, of a person I, which is the outcome of a person if treated.

So both of these are Y as you can see. So this is actually the target variable or the outcome.

The the superscript zero and one, they are actually the treatments that you're giving them. Okay. So the individual treatment effect, okay, is simply the difference, which is uh, you know, Y1 minus Y0.

That's all, okay? And individual treatment effect is really what we are looking into, meaning that in an ideal world, this is what we need, right?

On an individual level, how much of a lift are we getting by assigning the treatment or by, you know, intervening with any kind of treatment, right?

So that's more about it. Uh, but here's the problem.

Um, we can never observe both outcomes for the same person. Okay, so let's take an example. So Maria got an email, T equals 1, and did a purchase, Y equals 1.

Okay, very clear. Um, then, so it's very clear that we observe Y of 1 equals 1 because the, you know, she got a treatment, right?

But we do not observe Y of 0. Why? Because that thing never happened, right?

So that is known as the counterfactual. Okay. Uh, so we can't really rewind time and try another scenario.

I mean, I don't know if you know that, but I don't know. Okay. Um, um, right, but we, we cannot really answer this question that would she have purchased without the email?

Um, right, and well, if she did, well, now you can see that Y of 0 equals 1.

Meaning that they, that I mean, she, you know, purchased it anyway. In that case, our individual , treatment effect is actually zero because nothing changed.

I mean, she would have bought it anyway. But if it's zero, Y of 0 equals 0, which means that there was some lift.

There was some lift because the email caused the purchase there, right?

So that's about that example and this is a key distinction that we need to really remember, right?

So, the motivation, I mean, why all of this, uh, you know, why does all of this matter, right? So, just like these two people respond completely different to olives, okay?

One loves it, one hates it, right? Individuals respond differently to interventions. Um, and if you look at the average treatment effect, we might see zero and conclude that the intervention does not work, but that, you know, could not be any farther from the truth, right?

So, the goal is to provide, uh, you know, intervention to the right customers, and I'll go into that details, right?

Individuals respond differently to interventions, and, uh, you know, as I said, if you just sum everything over, the average treatment effect might be zero, and but that's not, you know, what we're looking at.

So, in fact, there are actually four distinct types of customers, okay?

Or, you know, four distinct types of people. Let me show you how this hypothetical scenario where we can see both potential outcomes. Okay?

This is not real, this, this cannot happen, but, um, let's just imagine that we can see both potential outcomes. Okay? So, we have the first customer, Mr. Grayson.

Avik Basu - Beyond Just Prediction: Causal Thinking in Machine Learning | PyData Seattle 2025

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.

We aim to be an accessible, community-driven conference, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.

Avik Basu - Beyond Just Prediction: Causal Thinking in Machine Learning | PyData Seattle 2025

Scripts