welcome back. So in this video, I'm going to introduce this idea of a physics informed neural network or a PINN, which was first presented in this paper by Maire Raissi, Paris Perdikaris, and George Karniadakis.

And since its introduction, it's become one of the workhorse uh algorithms and kind of ideas in physics informed machine learning.

So it's based on a kind of uh neural network idea that also incorporates known physics uh in the form of a partial differential equation in the loss function.

So I'm going to walk you through this, some examples of how it's been extended, some kind of caveats and cautionary tales, uh and some big success stories as well of what it can do.

So kind of a traditional approach to um using a neural network to predict something like a fluid uh flow field, you might build a neural network um some kind of, you know, deep feed forward network that predicts the field variables of interest, maybe U, V, W, and P.

These would be the X, the Y, and the Z velocity field components and P might be the pressure of a fluid and those are functions of the spatial coordinates X, Y, Z, and time T.

This picture is just a two-dimensional fluid, but you can imagine uh if I wanted to model a three-dimensional fluid with three velocity components and a pressure in terms of three spatial coordinates and time, uh I might be able to do that by building a big feed forward neural network and training it on a bunch of examples of flow data.

I might be able to learn that mapping uh from space and time to some uh coordinates U, V, W, and P.

Okay, this is kind of the naive approach to using a neural network to predict some physics and what uh what these authors did in this kind of classic now PINN paper is they extended this uh this naive neural network picture where you have just a simple loss function um of your kind of predicted fields from your true measured fields.

They extend this by computing things like partial derivatives of these output variables in terms of the input spatial and temporal variables.

And this is a really simple but very powerful idea that you can compute these terms essentially using the same kind of auto differentiation, automatic differentiation, and back propagation that you would normally use to train a neural network.

You can use that same idea in a modern language like PyTorch or JAX to compute these partial derivatives of the outputs with respect to the inputs.

And if you have those partial derivatives computed, you can add in this additional loss function that your partial differential equation should be satisfied.

So if you know, for example, that this uh velocity field should be divergence-free, you can add a term in purple here that is literally the divergence of your velocity field, and if it's non-zero, it means that that physics is being violated.

So I have an example here of uh the Navier-Stokes equations, which is one of the most popular examples used um with PINNs.

And again, just to reiterate, uh what this this big picture does is it takes a standard kind of neural network representation of the field variables you want to predict or estimate, things like velocity fields and pressure fields, as a function of space and time.

It takes that standard kind of generic neural network uh model, and it computes the partial derivatives that are relevant for the physics in this problem, again using automatic differentiation and back propagation, things like that.

And from those computed partial derivatives, you can create this additional loss term.

that tells you, uh, kind of if this known physics is being violated or not. Basically, you know, if there are things that should be zero, like the divergence, or you could take the right-hand side of this equation and move it over to the left-hand side, so all of that adds up to zero.

Those would be terms in this purple, uh, virtual loss function. And what that does is it adds this, um, this penalty when your neural network is violating known physics.

So this works really well for systems where we know, um, something about the physics. We know we're dealing with a fluid flow or a quantum system or an electromagnetic system, and we're still trying to predict kind of our field variables as a function of space and time.

And you'll notice that, um, I have these sums labeled. This is a sum over actual data, so that's training data. That's actual, you know, velocity fields that you have either measured or computed.

And then this purple sum is summed over virtual points. So this is one of the cool things about this network is if you know the physics, if you know that you're looking at a Navier-Stokes fluid flow, then even if I only have pretty limited data here, I can still test if my network is physical, meaning does it satisfy, you know, these equations?

Is this L function small? I can evaluate that on test points that are not actually in my training data.

So that's a huge and powerful idea. Also, is that oftentimes I've seen these PINNs trained on very small amounts of actual measurement data, and you offload a lot of the training to these kind of virtual points where you can evaluate, is this network physical on those virtual points?

Because even on a virtual point where I don't have real measurement data, I can still evaluate if my network is, you know, conserving mass and momentum.

Okay, pretty cool idea here. And if we uh kind of go back to this big picture of physics inform machine learning, it's pretty clear that um PINNs or this physics inform neural network sits here in this kind of stage four of crafting a loss function that promotes what we know about the physics.

Now of course there's some architecture, you know, we're computing those partial derivatives uh and we're modeling field variables as a function of space and time, but I think the real uh kind of special aspect of PINNs is here in this stage four crafting a loss function.

Okay, so back to this picture here. Now a couple of things I want to point out. Um there are reasons I like this architecture a lot and there's things that I think um you should be aware are maybe a little bit of like a cautionary tale about using an architecture like this.

So in general I think this is a really clever idea because it takes something that is very simple and powerful idea in machine learning, this kind of big feedforward neural network, a big MLP, and it adds a really simple way of incorporating physics into that uh problem.

So this loss function here and the fact that we're using kind of automatic differentiation to compute these quantities of interest that go into this virtual PD loss function, it's a very, very clever and also a very simple way of adding in known physics to this neural network process.

So I like that and I think that's probably one of the reasons that this has been so widely adopted, um you know, cited thousands and thousands of times and used by tons of researchers um across the world is because it's a really simple and intuitive way of adding physics into, you know, the standard way you would build a neural network to model physics.

Another thing I really like, and I mentioned this before, is that this really allows you to work with relatively small data sets.

You don't need necessarily that much data here. Uh, for the actual, um, database loss function, you can use virtual points to evaluate if your network is physical or not.

Um, and so I've seen lots of these networks trained with tons of virtual particles and not that much data. And so I think that's a really, really powerful part of this as well.

Now, the things that, um, this isn't, you know, exactly a negative, it's just a feature of the way that this is, um, that this is developed is that because the physics is added as a loss function, that's a strength because it's very intuitive and easy, but it also only suggests that your physics is being satisfied.

These two terms are always going to be fighting with each other.

You're always going to be having this pull between your possibly noisy data measurements and this kind of virtual term here that says that your PDE should be satisfied, your physics should be satisfied.

And so when you train this thing, it might be pretty stiff, um, and difficult optimization, and you're almost never going to actually have this purple term here be exactly zero.

So your physics is not being exactly satisfied by this, uh, this PIN mass is not exactly being conserved, momentum is not exactly being conserved.

And you know, that may or may not, um, become important in different applications. In tons of applications, it is perfectly fine to have a small but nonzero physics loss, and it still is an improvement over not adding that loss at all.

But in some applications, you might want to have a more exact, uh, enforcing of your of that physics, maybe through an architectural choice or through an actual constrained optimization.

And that's something we've talked about a little bit before and we'll talk about again, but I just want to point that out here.

Um, very simple and intuitive. It allows you to work with relatively small data sets, and those are both, you know, huge positives. But the fact that this is a loss function means you're only suggesting that the physics is informed.

You're not enforcing that the physics is informed. And so you should, or sorry, enforced, uh, and so you should just be aware of that.

I think that's, that's super important. Um, and I want us to be thinking about what this kind of architecture is going to be good for.

Okay, so it's really, really good for estimating things like whole flow fields with fairly sparse sensor measurements.

It's a really good way of doing that, and it improves on some of the past methods that we used to do before PINNs existed, and I'll talk a little bit about that.

So data assimilation and kind of flow estimates from, uh, limited or sparse measurements, that's a pretty cool application of this.

So you can picture maybe you have like a, you know, a fusion reactor, some kind of Sphera fusion reactor, and you can't put sensors everywhere inside of that reactor because they'll melt.

So you maybe only have a few, you know, probes on the surface.

This could be a pretty cool architecture to take that limited data on the surface and add in the physics loss because we know that the physics is, you know, magnetohydrodynamics or some kind of plasma physics and estimate the internal flow that's most consistent with those measurements and with the fact that we know our physics is governed by this PDE.

So that's just one example. Um, good. It might not be the most useful thing in the world for speeding up your high fidelity simulations, though, you know, that's something, uh, I think you should really be thinking about.

How can I use this architecture and what are ways that I maybe don't want to use this architecture?

Good. Um, and so actually one of the best and kind of most interesting papers, uh, that was published, you know, at around the same time as the original PINNs paper is, um, this this paper kind of, I think it's called something about hidden physics by many of the same authors in science.

So very high-profile paper where they essentially show that if you have relatively limited data from a fluid flow, um, I think this is, you know, some kind of a fluid flow in an artery or like a blood flow.

I don't know if this is called like a thrombosis, but it looks like there's some recirculation of blood flow here.

You can take very limited information and sometimes information that is, um, not even like a velocity field, like a smoke visualization or some kind of, uh, like a movie, and you can infer what the actual velocity field, the quantitative velocity field that is kind of closest to satisfying the Navier-Stokes equations and satisfying the measurement data that you have.

And you can see that you actually get relatively good reconstruction, uh, of the kind of PINNs reconstruction and the ground truth reference, uh, velocity field.

And you can kind of see quantitatively here, you should download this paper and actually read it.

It's very interesting paper and it tells you how to take, uh, other types of measurements and infer these velocity fields. But again, I think that's one of the things that, uh, PINNs are really, really good for is taking limited sensor data and inferring between the lines, kind of reading between the lines of what the velocity field should look like to be consistent with your measurement data and to be consistent with your governing equations.

And it's funny, that's actually pretty close to what a common filter does. If you think about it, it kind of optimally balances this idea between you want to, um, agree with your data, but you also want to agree with your model.

And depending on how much you trust your model or your data, you might change this kind of hyperparameter that balances those two.

So there's also a hyperparameter in PINNs that balances how much you trust your data versus how much you kind of want to be enforcing your governing partial differential equation.

Now, I'll point out because we're computing those derivatives, those partial derivatives that go into the kind of physics informed loss function here, this is going to have a harder time on systems that are, you know, discontinuous with things like shock waves or, you know, chaotic convecting flows where this neural network function of time might be really diverging and really chaotic.

It's not going to probably be amazing at those kinds of flows. So again, in the back of your mind, you have to be thinking this is not magic.

This is a neural network with a custom loss function and so you need to know when it is and is not going to work.

Okay, um, but it, you know, does really, really cool things in certain circumstances for things like flow reconstruction from limited measurements.

Okay, um, and in the past we used to use a method called 4D VAR. It was like kind of a variational method in space and time, so that's where the 4D comes from and it was this horrific physics-based constrained optimization.

It was an optimization constrained by the physics.

to again find a flow field that was consistent with your measurements and with the physics, and all of that now in PINNs is kind of abstracted from the human.

It uses modern machine learning architectures and optimizations, and it typically does, you know, as well in a lot of cases. So, for some of you who remember 4D-VAR, you'll see the parallel pretty quickly.

For those of you who don't, maybe just do a quick Google on 4D-VAR and you'll see kind of, uh, morally how these are related and different.

Okay. Um, now I'll put some links in the in the video description. There's some great resources out there that I'm going to link to.

Um, so a few really good, um, really good videos here. Um, this is one by, uh, Juan Tuscano, and I believe this is, uh, Benjamin Mosley on, uh, on this YouTube channel.

These are both really good kind of tutorial videos. Um, you know, this one, um, Juan actually works, this, uh, a training, how to train a, a physics informed neural network out in code, so I would highly recommend this one.

Um, and there's some great tutorials online also for how to code this up, uh, in Torch.

Okay, so I'll put links to these in, uh, in the video description, but really it's super important for you to try this yourself.

I think that's one of the main points here is if you want to know what this is good for and what this is not good for, you got to actually try it on a couple of examples and see, oh, was it, did it take forever to train?

Was it fast to train? Would did it take forever to execute? Was it fast to execute?

Did it generalize? Did I need a bunch of data or a little data? Those are all the things you would you would play.

around with and roughly speaking, um, you know, these methods, again, they're good for filling in lots of missing data or or kind of, um, inferring if you have very sparse measurement data.

They're good for speeding up the inference step. So training might be expensive, uh, compared to something like, you know, CFD or or or fluid flow simulations, uh, and you in fact need the training data from those high fidelity sources, but the inference step once you have a trained model can be very, very fast.

And so that's another, you know, good use of these methods. And among these resources, again, this video was by, uh, Benjamin Mosley and they have a really, really nice, um, kind of blog you should check out, um, all about scientific machine learning and they have a really cool blog post specifically about PINNs with, uh, code on how to actually, you know, train these and test these yourself.

So I think this is one of my favorite, uh, to follow along and they contrast PINNs with the kind of classic naive neural network approach where you don't add that physics informed loss.

And so I believe the example that they train on here is a really simple one-dimensional physics where you basically take a mass on a spring and you let that spring go or sorry, you let the mass go and you have this kind of decaying oscillation of your your spring mass damper system and there is some pretty sparse training data, these blue points here.

And essentially these are movies, so go to the website and actually, you know, check out the movies, but essentially this shows the training of a naive neural network over, uh, training step and it basically never trains.

It goes into this kind of suboptimal kind of crummy solution because it can't generalize past the blue data that it was trained on because it doesn't know anything about the physics.

And in contrast, if you have this kind of physics informed loss function where you actually say, hey, I know that this is a, you know, F equals MA, a spring system.

Now, from those blue training points, you can train a physics informed neural network that is much, much better at generalizing into the future because it got the physics right, because that was in the loss function.

Okay, um, so really simple example, but it's one you can code up yourself, you can follow it.

All makes intuitive sense. Um, and it's a little easier than the PTE version. Now, I will point out I noticed here that this is trained to 620 steps and this is trained to 12,000 steps.

So again, that should be making you wonder, hey, how much training do I really have to do to get this PIN to converge?

But again, maybe the training is an offline computation, maybe you're willing to spend a lot of resources to train this thing as long as it learns the physics and executes quickly, uh, in the future.

And again, this generalizes way better because it incorporates the physics.

That's kind of the upshot here. Good. Um, and there's a lot of cool extensions to PINNs. That's one of the things I like, you know, my lab and my collaborators, we develop a lot of methods, um, you know, similar kind of machine learning and physics informed machine learning methods.

And when I think of my favorite methods, things like, uh, DMD and Cindy and, you know, PINNs and things like that, they tend to be highly extensible.

They're such a simple idea that you can often add a lot of kind of add-ons to make them more performant for different kinds of scenarios.

And so this is just a very, you know, select few examples out in the literature, there are dozens of custom types of PINNs for different types of physics, for different types of data, different types of scenarios.

Physics Informed Neural Networks (PINNs) [Physics Informed Machine Learning]

Physics Informed Neural Networks (PINNs) [Physics Informed Machine Learning]

Scripts