Welcome back. So in this video, I'm going to introduce the idea of a physics-informed neural network, or a PIN, which was first presented in this paper by Maziar Raissi, Paris Perdikaris, and George Karniadakis. And since its introduction, 

it's become one of the workhorse algorithms and kind of ideas in physics-informed machine learning. So it's based on a kind of neural network idea that also incorporates known physics in the form of a partial differential equation

in the loss function. So I'm going to walk you through this, some examples of how it's been extended, some kind of caveats and cautionary tales, and some big success stories as well of what it can do. So kind of a traditional approach to

um, using a neural network to predict something like a fluid flow field, you might build a neural network, um, some kind of, you know, deep feedforward network that predicts the field variables of interest, maybe U, V, W, and P. These would be the X, the Y, and the Z

velocity field components, and P might be the pressure of a fluid. And those are functions of the spatial coordinates X, Y, Z, and time T. This picture is just a two-dimensional fluid, but you can imagine if I wanted to model a three-dimensional fluid with three velocity components and a pressure in

terms of three spatial coordinates and time, I might be able to do that by building a big feedforward neural network and training it on a bunch of examples of flow data. I might be able to learn that mapping from space and time to some coordinates U, V, W, and P.

okay this is kind of the naive approach to using a neural network to predict some physics and what uh what these authors did in this kind of classic now hidden paper is they extended this uh this naive neural network picture where you have just a simple loss function um of your kind of

predicted fields from your true measured fields they extend this by computing things like partial derivatives of these output variables in terms of the input spatial and temporal variables and this is a really simple but very powerful idea that you can compute these terms essentially using the same kind of auto

differentiation automatic differentiation and back propagation that would you would normally use to train a neural network you can use that same idea in a modern language like PyTorch or Jax to compute these partial derivatives of the outputs with respect to the inputs and if you have those

partial derivatives computed you can add in this additional loss function that your partial differential equation should be satisfied so if you know for example that this uh velocity field should be divergence free you can add a term in purple here that is literally the divergence of your velocity field

and if it's non zero it means that that physics is being violated so I have an example here of uh the Navier-Stokes equations which is one of the most popular examples used um with PINs and again just to reiterate at uh what this this big picture does is it

takes a standard kind of neural network representation of the field variables you want to predict or estimate things like velocity fields and pressure fields as a function of space and time it takes that standard kind of generic neural network uh model and it computes the partial derivatives that are relevant

for the physics in this problem again using automatic differentiation and backpropagation things like that and from those computed partial derivatives you can create this additional loss term that tells you uh kind of if this known physics is being violated or not basically you know if there are things

that should be zero like the divergence or you could take the right hand side of this equation and move it over to the left hand so all of that adds up to zero those would be terms in this purple uh virtual loss function and what that does

is it adds this um this penalty when your neural network is violating known physics so this works really well for systems where we know um something about the physics we know we're dealing with a fluid flow or a quantum system or an electromagnetic system and we're

still trying to predict kind of our field variables as a function of space and time um so it's a really really cool idea and you'll notice that um I have these sums labeled this is a sum over actual data so that's training data that's actual you know velocity fields

that you have either measured or computed and then this purple sum is summed over virtual points so this is one of the cool things about this network is if you know the physics if you know that you're looking at a Navier-Stokes fluid flow then even if I only

have pretty limited data here I can still test if my network is physical meaning does it satisfy you know these equations is this L function small I can evaluate that on test points that are not actually in my training data so that's a huge and powerful idea also is

that often times I've seen these PINs trained on very small amounts of actual measurement data and you offload a lot of the training to these kind of virtual points where you can evaluate is this network physical on those virtual points because even on a virtual point where I

don't have real measurement data I can still evaluate if my network is you know conserving mass and momentum okay pretty cool idea here and if we uh kind of go back to this big picture of physics informed machine learning it's pretty clear that um PINs or this physics informed neural

network sits here in this kind of stage four of crafting a loss function that promotes what we know about the physics now of course there's some architecture you know we're computing those partial derivatives uh and we're modeling field variables as a function of space and

time but I think the real uh kind of special aspect of PINs is here in this stage four crafting a loss function okay so back to this picture here now a couple of things I want to point out um there are reasons I like this architecture a lot and there's

things that I think um you should be aware are maybe a little bit of like a cautionary tale about using an architecture like this so in general I think this is a really clever idea because it takes something that is very simple and powerful idea in machine

learning this kind of big feedforward neural network a big MLP and it adds a really simple way of incorporating physics into that uh problem so this loss function here and the fact that we're using kind of automatic differentiation to compute these quantities of interest that go

into this virtual PD loss function, it's a very, very clever and also a very simple way of adding in known physics to this neural network process, so I like that and I think that's probably one of the reasons that this has been so widely adopted, um, you know, cited thousands and

thousands of times and used by tons of researchers um across the world is because it's a really simple and intuitive way of adding physics into, you know, the standard way you would build a neural network to model physics. Another thing I really like and

I mentioned this before is that this really allows you to work with relatively small data sets. You don't need necessarily that much data here uh for the actual um database loss function. You can use virtual points to evaluate if your network is physical or not um

and so I've seen lots of these networks trained with tons of V virtual particles and not that much data, and so I think that's a really, really powerful part of this as well. Now, the things that um this isn't, you know, exactly a negative, it's

just a feature of the way that this is um that this is developed is that because the physics is added as a loss function, that's a strength because it's very intuitive and easy, but it also only suggests that your physics is being satisfied. These two terms are always

going to be fighting with each other. You're always going to be having this pull between your possibly noisy data measurements and this kind of virtual term here that says that your PDE should be satisfied, your physics should be satisfied, and so when you train this thing it

might be pretty stiff, um, and difficult to optimize and you're almost never going to actually have this purple term here be exactly zero, so your physics is not being exactly satisfied by this, uh, this pin mass is not exactly being conserved, momentum is not exactly being

conserved, and you know that may or may not, um, become important in different applications. In tons of applications, it is perfectly fine to have a small but nonzero physics loss, and it still is an improvement over not adding that loss at all. But in some applications, you might

want to have a more exact, uh, enforcing of your, of that physics, maybe through an architectural choice or through an actual constrained optimization. And that's something we've talked about a little bit before, and we'll talk about again, but I just want to point that out

here. Um, very simple and intuitive, it allows you to work with relatively small data sets, and those are both, you know, huge positives. But the fact that this is a loss function means you're only suggesting that the physics is informed, you're not enforcing that the physics

is informed, and so you should, or sorry, enforced, uh, and so you should just be aware of that. I think that's, that's super important, um, and I want us to be thinking about what this kind of architecture is going to be good for. Okay, so it's really, really good for

estimating things like whole flow fields with fairly sparse sensor measurements. It's a really good, good way of doing that, and it improves on some of the past methods that we used to do before pins existed, and I'll talk a little bit about that. So data assimilation and kind of

flow estimates from uh limited or sparse measurements that's a pretty cool application of this so you can picture maybe you have like a you know a fusion reactor some kind of spherical fusion reactor and you can't put sensors everywhere inside of that reactor

because they'll melt so you maybe only have a few you know probes on the surface this could be a pretty cool architecture to take that limited data on the surface and add in the physics loss because we know that the physics is you know Magneto hydrodynamics or some kind of

plasma physics and estimate the internal flow that's most consistent with those measurements and with the fact that we know our physics is governed by this PDE so that's just one example um good it might not be the most useful thing in the world for speeding

up your high fidelity simulations though you know that's something uh I think you should really be thinking about how can I use this architecture and what are ways that I maybe don't want to use this architecture good um and so actually one of the best and kind of most interesting

papers uh that was published you know in at around the same time as the original PINNs paper is um this this paper kind of I think it's called something about hidden physics by by many of the same authors in Science so very high-profile paper where they essentially show that

if if you have relatively limited data from a fluid flow um I think this is you know some kind of a flu fluid flow in an artery or like a blood flow I don't know if this is called like a thrombosis but it looks like there's some recirculation

of blood flow here you can take very limited information and sometimes information that is um not even like a velocity field like a smoke visualization or some kind of uh like a movie and you can infer what the actual velocity field the quantitative velocity field that is kind of closest to

satisfying the Navier-Stokes equations and satisfying the measurement data that you have and you can see that you actually get relatively good reconstruction uh of the kind of pins reconstruction and the ground truth reference uh velocity field and you can kind of see quantitatively

here you should download this paper and actually read it it's very interesting paper and it tells you how to take uh other types of measurements and infer these velocity fields but again I think that's one of the things that uh pins are really really good for is

taking limited sensor data and inferring between the lines kind of reading between the lines of what the velocity field should look like to be consistent with your measurement data and to be consistent with your governing equations and it's funny that's actually pretty close to what a Kalman filter does if

you think about it it kind of optimally balances this idea between you want to um agree with your data but you also want to agree with your model and depending on how much you trust your model or your data you might change this kind of hyperparameter that balances

those two so there's also a hyperparameter in pins that balances how much you trust your data versus how much you kind of want to be enforcing your governing partial differential equation now I'll point out because we're computing those derivatives those partial derivatives that go into the the

kind of physics informed loss function here. This is going to have a harder time on systems that are, you know, discontinuous with things like shock waves or, you know, chaotic convecting flows where this neural network function of time might be really diverging and really chaotic. It's not going to

probably be amazing at those kinds of flows. So again, in the back of your mind you have to be thinking this is not magic, this is a neural network with a custom loss function, and so you need to know when it is and is not going to work.

Okay, um, but it, you know, does really really cool things in certain circumstances for things like flow reconstruction from limited measurements. Okay, um, and in the past we used to use a method called 4D VAR. It was like kind of a variational method in space and time.

So that's where the 4D comes from, and it was this horrific physics-based constrained optimization. It was an optimization constrained by the physics to again find a flow field that was consistent with your measurements and with the physics and all of that. Now in

PINNs is kind of abstracted from the human. It uses modern machine learning architectures and optimizations, and it typically does, you know, as well in a lot of cases. So for some of you who remember 4D VAR, you'll see the parallel pretty quickly. For those of you

who don't, maybe just do a quick Google on 4D VAR and you'll see kind of, uh, morally how these are related and different. Okay, um, now I'll put some links in the in the video description. There's some great resources out there.

that I'm going to link to um so a few really good um really good videos here um this is one by uh Juan Tuscano and I believe this is uh Benjamin Mosley on uh on on this YouTube channel these are both really good kind of tutorial videos

um you know this one um Juan actually works this uh a training how to train a a physics informed neural network out in code so I would highly recommend this one um and there's some great tutorials online also for how to code this up uh in torch okay so I'll put links to these

in uh in the video description but really it's super important for you to try this yourself I think that's one of the main points here is if you want to know what this is good for and what this is not good for you got to actually try

it on a couple of examples and see oh was it did it take forever to train was it fast to train would did it take forever to execute was it fast to execute did it generalize did I need a bunch of data or a little data those are

all the things you would you would play around with and roughly speaking um you know these methods again they're good for filling in lots of missing data or or kind of um inferring if you have very sparse measurement data they're good for speeding up the inference step so

training might be expensive uh compared to something like you know CFD or or or fluid flow simulations uh and you in fact need the training data from those high fidelity sources but the inference step once you have a trained model can be very very fast

and so that's another, you know, good use of these methods and among these resources again, this video was by uh Benjamin Mosley and they have a really, really nice um kind of blog you should check out um all about scientific machine learning and they have a really

cool blog post specifically about PINs with uh code on how to actually, you know, train these and test these yourself. So I think this is one of my favorite uh to follow along and they contrast PINs with the kind of classic naive neural network approach where you don't add that

physics-informed loss and so I believe the example that they train on here is a really simple one-dimensional physics where you basically take a mass on a spring and you let that spring go or sorry you let the mass go and you have this kind of decaying oscillation of

your your spring mass damper system and there is some pretty sparse training data, these blue points here, and essentially these are movies, so go to the website and actually you know check out the movies, but essentially this shows the training of a naive neural network over uh training step and it

basically never trains, it goes into this kind of suboptimal kind of crummy solution because it can't generalize past the blue data that it was trained on because it doesn't know anything about the physics and in contrast if you have this kind of physics-informed loss

function where you actually say hey I know that this is a, you know, f equals ma spring system, now from those blue training points you can train a physics-informed neural network that is much, much better at generalizing into the future because it got the physics right because

that was in the loss function okay um so really simple example but it's one you can code up yourself you can follow it all makes intuitive sense um and it's a little easier than the PDE version now I will point out I noticed here that

this is trained to 620 steps and this is trained to 12,000 steps so again that should be making you wonder hey how much training do I really have to do to get this PIN to converge but again maybe the training is an offline computation

maybe you're willing to spend a lot of resources to train this thing as long as it learns the physics and executes quickly uh in the future and again this generalizes way better because it incorporates the physics that's kind of the upshot here good um and there's a

lot of cool extensions to PINs that's one of the things I like you know my lab and my collaborators we develop a lot of methods um you know similar kind of machine learning and physics and form machine learning methods and when I think of my favorite methods things like

uh DMD and Cindy and you know PINs and things like that they tend to be highly extensible they're such a simple idea that you can often add a lot of kind of add-ons to make them more performant for different kinds of

scenarios and so this is just a very you know select few examples out in the literature there are dozens of custom types of PINs for different types of physics for different types of data different types of scenarios one of the ones I think is really cool uh are

fractional PINs for um partial differential equations that have things like fractional derivatives. What if I have the 1/2 uh power diffusion term or some, you know, hyperdiffusion term, or I have an integral differential uh partial differential equation? This happens all the time, like not everything

is as simple as Navier-Stokes, just simple mass and momentum conservation with, you know, easy integer power differential operators. Often times, you have these nasty integral differential equations with fractional derivatives in your physics uh um, and so there's this really cool kind of uh split

architecture here where your traditional PINs terms here can be handled using auto differentiation, using kind of the automatic differentiation of the, you know, PyTorch or JAX environment, and then the other terms here, your fractional derivatives, your integral terms, things like that, those can be handled using your traditional numerical

discretization schemes, things like finite difference, finite element, spectral methods, and you can kind of combine those in this fractional PIN architecture. So pretty cool idea um, and actually that broadens it out to a massive, massive class of PDEs. Um, another one I really, really like

is this idea of Delta PINs um, and this is an idea that's been kind of floating around um, you know, a lot. I know some of, some of um, my posts and collaborators talk about similar ideas where um, if you have a geometry, like if I know that my

physics is evolving on a sphere or on this spiral or on um, like let's say I look at the CA of an ear, I know that that's kind of this spiral shape. So if I know the geometry, kind of the determine the boundary conditions and and where my

PDE is living you can actually you have a lot of prior information you can bake into the the PINs or or other machine learning algorithms. And so specifically if you look at the Laplace-Beltrami operator on that geometry, its eigenfunctions give you a very good natural

coordinate system in which to represent solution functions of that PDE. So kind of the simplest example that we've all seen before is that if I look at something like the heat equation on a rectangular geometry, the eigenfunctions of the Laplace operator on that geometry are sines and

cosines, and that we know that the Fourier transform of those sines and cosines are a good coordinate system that linearize or simplify that heat equation. So it's a very similar idea but for more complex geometries, that eigenfunctions of the Laplace-Beltrami operator on an

interesting geometry will be a very good coordinate system in which to represent your solutions of your PDE. This can be useful in things like cardiac flows. If I take a weird irregular geometry like my heart with all the veins and arteries, again I can look at the eigenfunctions

of the Laplace-Beltrami operator, and that gives me a good basis in which to represent and sum uh to approximate my PDE solution. And so you know there's a lot going on under the hood here. Um, I'm not going to get into, but this Delta

PINs kind of uses that idea to incorporate some idea of the geometry into the representation of the PDE and you get these improved representations. So um, you know, some kind of cool examples here. This is a heat transfer problem where you have um some

heat convection and some geometry with, you know, specified boundary conditions and you can see that you know the regular pin doesn't agree with ground truth as well as this Delta pin. So having, uh, kind of the the some knowledge of the geometry and of the PDE physics

lets you do better than just having one or the other. Um this is kind of a cool example where, um they use this method to compute geodesics on this kind of this geometry of a bunny. Geodesics is just a fancy word for the shortest distance path on a

manifold. So you know if I look at the shortest distance path from my location to, um, you know, Switzerland, it's the great circle that connects those two points on our sphere of Earth, uh, you know, if you believe that the Earth is in fact a sphere, um, and so these geodesics

are actually, um, the solution of a certain partial differential equation. I'm not going to get into that, um, but so you know there is some PDE that governs geodesics that you can add as a loss function and this just shows that when

you use this kind of geometry, these Laplace Beltrami eigenfunctions for this geometry, you can very accurately get those solutions, uh, for this geodesic problem. It's a really cool, uh, example I think and it's kind of a clever example to benchmark their their problem

on, um, and I don't really know exactly what this example is, but I think it's super cool that there is, you know, a dragon in their paper, which I like. Okay, good. Um, so those are just two of many, many, many examples of, you know, powerful extensions to this idea of pins.

again because it's such a simple idea you're just adding a loss function that is literally the pte how much it's being satisfied or not um there's a lot of ways of extending this to other uh kind of more complicated systems but there are some caveats

these don't always train that well sometimes they're stiff sometimes they overfit there's issues anybody who has you know worked a lot with pins knows you know there's ways of massaging them to work and sometimes it doesn't work okay and that's like every method no

method is a silver bullet that just doesn't exist um but it's good to understand what are the limitations uh and actually study them uh in a very methodical and process driven way and so that's one of the things I really really like about this paper um out of Michael

Mahoney's uh kind of group and and collaboration on characterizing the actual failure modes in pins and I'm not going to expect you to read all of this you should download the paper and check it out because they have open source code and you can play with it yourself

but you know some of the main upshots here is that um the main contributions as stated in this paper is that they they tried to train pins on relatively simple uh but relevant problems things like convection reaction diffusion you know simple pdes and they

basically found that um a standard pin often doesn't train and gives very large error except for you know really really simple parameter regimes of of these these pdes okay so they basically uh set up a problem a set of physics pde problems where they can show under

certain parameter regimes the method fails to train um and then essentially they looked at you know adding how what if you balance that that pte loss term so you can crank that pte loss term up or down that's a hyperparameter of how important the

physics is or is not and they found um that the loss function by increasing or decreasing that uh pte based soft constraint actually makes it more complex to optimize and harder to find a good solution in some cases um and I think this is a really

careful paper I really like um the work um by Michael Mahoney and his colleagues because they are very very careful in setting up this problem I don't think they're just criticizing pins they're actually trying to systematically understand when they

don't work and fix the problem and so they look at you know the actual neural network itself without the physics informed loss and they do in fact find that it does have the capacity to represent the solutions so it's not a capacity issue it's not the neural

Physics Informed Neural Networks (PINNs) [Physics Informed Machine Learning]

Physics Informed Neural Networks (PINNs) [Physics Informed Machine Learning]