Welcome back. So in this video, I'm going to introduce the idea of a physics-informed neural network, or a PIN, which was first presented in this paper by Maziar Raissi, Paris Perdikaris, and George Karniadakis. And since its introduction, 

it's become one of the workhorse algorithms and kind of ideas in physics-informed machine learning. So it's based on a kind of neural network idea that also incorporates known physics in the form of a partial differential equation

in the loss function. So I'm going to walk you through this, some examples of how it's been extended, some kind of caveats and cautionary tales, and some big success stories as well of what it can do. So kind of a traditional approach to

um, using a neural network to predict something like a fluid flow field, you might build a neural network, um, some kind of, you know, deep feedforward network that predicts the field variables of interest, maybe U, V, W, and P. These would be the X, the Y, and the Z

velocity field components, and P might be the pressure of a fluid. And those are functions of the spatial coordinates X, Y, Z, and time T. This picture is just a two-dimensional fluid, but you can imagine if I wanted to model a three-dimensional fluid with three velocity components and a pressure in

terms of three spatial coordinates and time, I might be able to do that by building a big feedforward neural network and training it on a bunch of examples of flow data. I might be able to learn that mapping from space and time to some coordinates U, V, W, and P.

okay this is kind of the naive approach to using a neural network to predict some physics and what uh what these authors did in this kind of classic now hidden paper is they extended this uh this naive neural network picture where you have just a simple loss function um of your kind of

predicted fields from your true measured fields they extend this by computing things like partial derivatives of these output variables in terms of the input spatial and temporal variables and this is a really simple but very powerful idea that you can compute these terms essentially using the same kind of auto

differentiation automatic differentiation and back propagation that would you would normally use to train a neural network you can use that same idea in a modern language like PyTorch or Jax to compute these partial derivatives of the outputs with respect to the inputs and if you have those

partial derivatives computed you can add in this additional loss function that your partial differential equation should be satisfied so if you know for example that this uh velocity field should be divergence free you can add a term in purple here that is literally the divergence of your velocity field

and if it's non zero it means that that physics is being violated so I have an example here of uh the Navier-Stokes equations which is one of the most popular examples used um with PINs and again just to reiterate at uh what this this big picture does is it

takes a standard kind of neural network representation of the field variables you want to predict or estimate things like velocity fields and pressure fields as a function of space and time it takes that standard kind of generic neural network uh model and it computes the partial derivatives that are relevant

for the physics in this problem again using automatic differentiation and backpropagation things like that and from those computed partial derivatives you can create this additional loss term that tells you uh kind of if this known physics is being violated or not basically you know if there are things

that should be zero like the divergence or you could take the right hand side of this equation and move it over to the left hand so all of that adds up to zero those would be terms in this purple uh virtual loss function and what that does

is it adds this um this penalty when your neural network is violating known physics so this works really well for systems where we know um something about the physics we know we're dealing with a fluid flow or a quantum system or an electromagnetic system and we're

still trying to predict kind of our field variables as a function of space and time um so it's a really really cool idea and you'll notice that um I have these sums labeled this is a sum over actual data so that's training data that's actual you know velocity fields

that you have either measured or computed and then this purple sum is summed over virtual points so this is one of the cool things about this network is if you know the physics if you know that you're looking at a Navier-Stokes fluid flow then even if I only

have pretty limited data here I can still test if my network is physical meaning does it satisfy you know these equations is this L function small I can evaluate that on test points that are not actually in my training data so that's a huge and powerful idea also is

that often times I've seen these PINs trained on very small amounts of actual measurement data and you offload a lot of the training to these kind of virtual points where you can evaluate is this network physical on those virtual points because even on a virtual point where I

don't have real measurement data I can still evaluate if my network is you know conserving mass and momentum okay pretty cool idea here and if we uh kind of go back to this big picture of physics informed machine learning it's pretty clear that um PINs or this physics informed neural

network sits here in this kind of stage four of crafting a loss function that promotes what we know about the physics now of course there's some architecture you know we're computing those partial derivatives uh and we're modeling field variables as a function of space and

time but I think the real uh kind of special aspect of PINs is here in this stage four crafting a loss function okay so back to this picture here now a couple of things I want to point out um there are reasons I like this architecture a lot and there's

things that I think um you should be aware are maybe a little bit of like a cautionary tale about using an architecture like this so in general I think this is a really clever idea because it takes something that is very simple and powerful idea in machine

learning this kind of big feedforward neural network a big MLP and it adds a really simple way of incorporating physics into that uh problem so this loss function here and the fact that we're using kind of automatic differentiation to compute these quantities of interest that go

into this virtual PD loss function, it's a very, very clever and also a very simple way of adding in known physics to this neural network process, so I like that and I think that's probably one of the reasons that this has been so widely adopted, um, you know, cited thousands and

thousands of times and used by tons of researchers um across the world is because it's a really simple and intuitive way of adding physics into, you know, the standard way you would build a neural network to model physics. Another thing I really like and

I mentioned this before is that this really allows you to work with relatively small data sets. You don't need necessarily that much data here uh for the actual um database loss function. You can use virtual points to evaluate if your network is physical or not um

and so I've seen lots of these networks trained with tons of V virtual particles and not that much data, and so I think that's a really, really powerful part of this as well. Now, the things that um this isn't, you know, exactly a negative, it's

just a feature of the way that this is um that this is developed is that because the physics is added as a loss function, that's a strength because it's very intuitive and easy, but it also only suggests that your physics is being satisfied. These two terms are always

going to be fighting with each other. You're always going to be having this pull between your possibly noisy data measurements and this kind of virtual term here that says that your PDE should be satisfied, your physics should be satisfied, and so when you train this thing it

might be pretty stiff, um, and difficult to optimize and you're almost never going to actually have this purple term here be exactly zero, so your physics is not being exactly satisfied by this, uh, this pin mass is not exactly being conserved, momentum is not exactly being

conserved, and you know that may or may not, um, become important in different applications. In tons of applications, it is perfectly fine to have a small but nonzero physics loss, and it still is an improvement over not adding that loss at all. But in some applications, you might

want to have a more exact, uh, enforcing of your, of that physics, maybe through an architectural choice or through an actual constrained optimization. And that's something we've talked about a little bit before, and we'll talk about again, but I just want to point that out

here. Um, very simple and intuitive, it allows you to work with relatively small data sets, and those are both, you know, huge positives. But the fact that this is a loss function means you're only suggesting that the physics is informed, you're not enforcing that the physics

is informed, and so you should, or sorry, enforced, uh, and so you should just be aware of that. I think that's, that's super important, um, and I want us to be thinking about what this kind of architecture is going to be good for. Okay, so it's really, really good for

estimating things like whole flow fields with fairly sparse sensor measurements. It's a really good, good way of doing that, and it improves on some of the past methods that we used to do before pins existed, and I'll talk a little bit about that. So data assimilation and kind of

flow estimates from uh limited or sparse measurements that's a pretty cool application of this so you can picture maybe you have like a you know a fusion reactor some kind of spherical fusion reactor and you can't put sensors everywhere inside of that reactor

because they'll melt so you maybe only have a few you know probes on the surface this could be a pretty cool architecture to take that limited data on the surface and add in the physics loss because we know that the physics is you know Magneto hydrodynamics or some kind of

plasma physics and estimate the internal flow that's most consistent with those measurements and with the fact that we know our physics is governed by this PDE so that's just one example um good it might not be the most useful thing in the world for speeding

up your high fidelity simulations though you know that's something uh I think you should really be thinking about how can I use this architecture and what are ways that I maybe don't want to use this architecture good um and so actually one of the best and kind of most interesting

papers uh that was published you know in at around the same time as the original PINNs paper is um this this paper kind of I think it's called something about hidden physics by by many of the same authors in Science so very high-profile paper where they essentially show that

if if you have relatively limited data from a fluid flow um I think this is you know some kind of a flu fluid flow in an artery or like a blood flow I don't know if this is called like a thrombosis but it looks like there's some recirculation

of blood flow here you can take very limited information and sometimes information that is um not even like a velocity field like a smoke visualization or some kind of uh like a movie and you can infer what the actual velocity field the quantitative velocity field that is kind of closest to

satisfying the Navier-Stokes equations and satisfying the measurement data that you have and you can see that you actually get relatively good reconstruction uh of the kind of pins reconstruction and the ground truth reference uh velocity field and you can kind of see quantitatively

here you should download this paper and actually read it it's very interesting paper and it tells you how to take uh other types of measurements and infer these velocity fields but again I think that's one of the things that uh pins are really really good for is

taking limited sensor data and inferring between the lines kind of reading between the lines of what the velocity field should look like to be consistent with your measurement data and to be consistent with your governing equations and it's funny that's actually pretty close to what a Kalman filter does if

you think about it it kind of optimally balances this idea between you want to um agree with your data but you also want to agree with your model and depending on how much you trust your model or your data you might change this kind of hyperparameter that balances

those two so there's also a hyperparameter in pins that balances how much you trust your data versus how much you kind of want to be enforcing your governing partial differential equation now I'll point out because we're computing those derivatives those partial derivatives that go into the the

kind of physics informed loss function here. This is going to have a harder time on systems that are, you know, discontinuous with things like shock waves or, you know, chaotic convecting flows where this neural network function of time might be really diverging and really chaotic. It's not going to

probably be amazing at those kinds of flows. So again, in the back of your mind you have to be thinking this is not magic, this is a neural network with a custom loss function, and so you need to know when it is and is not going to work.

Okay, um, but it, you know, does really really cool things in certain circumstances for things like flow reconstruction from limited measurements. Okay, um, and in the past we used to use a method called 4D VAR. It was like kind of a variational method in space and time.

So that's where the 4D comes from, and it was this horrific physics-based constrained optimization. It was an optimization constrained by the physics to again find a flow field that was consistent with your measurements and with the physics and all of that. Now in

PINNs is kind of abstracted from the human. It uses modern machine learning architectures and optimizations, and it typically does, you know, as well in a lot of cases. So for some of you who remember 4D VAR, you'll see the parallel pretty quickly. For those of you

who don't, maybe just do a quick Google on 4D VAR and you'll see kind of, uh, morally how these are related and different. Okay, um, now I'll put some links in the in the video description. There's some great resources out there.

that I'm going to link to um so a few really good um really good videos here um this is one by uh Juan Tuscano and I believe this is uh Benjamin Mosley on uh on on this YouTube channel these are both really good kind of tutorial videos

um you know this one um Juan actually works this uh a training how to train a a physics informed neural network out in code so I would highly recommend this one um and there's some great tutorials online also for how to code this up uh in torch okay so I'll put links to these

in uh in the video description but really it's super important for you to try this yourself I think that's one of the main points here is if you want to know what this is good for and what this is not good for you got to actually try

it on a couple of examples and see oh was it did it take forever to train was it fast to train would did it take forever to execute was it fast to execute did it generalize did I need a bunch of data or a little data those are

all the things you would you would play around with and roughly speaking um you know these methods again they're good for filling in lots of missing data or or kind of um inferring if you have very sparse measurement data they're good for speeding up the inference step so

training might be expensive uh compared to something like you know CFD or or or fluid flow simulations uh and you in fact need the training data from those high fidelity sources but the inference step once you have a trained model can be very very fast

and so that's another, you know, good use of these methods and among these resources again, this video was by uh Benjamin Mosley and they have a really, really nice um kind of blog you should check out um all about scientific machine learning and they have a really

cool blog post specifically about PINs with uh code on how to actually, you know, train these and test these yourself. So I think this is one of my favorite uh to follow along and they contrast PINs with the kind of classic naive neural network approach where you don't add that

physics-informed loss and so I believe the example that they train on here is a really simple one-dimensional physics where you basically take a mass on a spring and you let that spring go or sorry you let the mass go and you have this kind of decaying oscillation of

your your spring mass damper system and there is some pretty sparse training data, these blue points here, and essentially these are movies, so go to the website and actually you know check out the movies, but essentially this shows the training of a naive neural network over uh training step and it

basically never trains, it goes into this kind of suboptimal kind of crummy solution because it can't generalize past the blue data that it was trained on because it doesn't know anything about the physics and in contrast if you have this kind of physics-informed loss

function where you actually say hey I know that this is a, you know, f equals ma spring system, now from those blue training points you can train a physics-informed neural network that is much, much better at generalizing into the future because it got the physics right because

that was in the loss function okay um so really simple example but it's one you can code up yourself you can follow it all makes intuitive sense um and it's a little easier than the PDE version now I will point out I noticed here that

this is trained to 620 steps and this is trained to 12,000 steps so again that should be making you wonder hey how much training do I really have to do to get this PIN to converge but again maybe the training is an offline computation

maybe you're willing to spend a lot of resources to train this thing as long as it learns the physics and executes quickly uh in the future and again this generalizes way better because it incorporates the physics that's kind of the upshot here good um and there's a

lot of cool extensions to PINs that's one of the things I like you know my lab and my collaborators we develop a lot of methods um you know similar kind of machine learning and physics and form machine learning methods and when I think of my favorite methods things like

uh DMD and Cindy and you know PINs and things like that they tend to be highly extensible they're such a simple idea that you can often add a lot of kind of add-ons to make them more performant for different kinds of

scenarios and so this is just a very you know select few examples out in the literature there are dozens of custom types of PINs for different types of physics for different types of data different types of scenarios one of the ones I think is really cool uh are

fractional PINs for um partial differential equations that have things like fractional derivatives. What if I have the 1/2 uh power diffusion term or some, you know, hyperdiffusion term, or I have an integral differential uh partial differential equation? This happens all the time, like not everything

is as simple as Navier-Stokes, just simple mass and momentum conservation with, you know, easy integer power differential operators. Often times, you have these nasty integral differential equations with fractional derivatives in your physics uh um, and so there's this really cool kind of uh split

architecture here where your traditional PINs terms here can be handled using auto differentiation, using kind of the automatic differentiation of the, you know, PyTorch or JAX environment, and then the other terms here, your fractional derivatives, your integral terms, things like that, those can be handled using your traditional numerical

discretization schemes, things like finite difference, finite element, spectral methods, and you can kind of combine those in this fractional PIN architecture. So pretty cool idea um, and actually that broadens it out to a massive, massive class of PDEs. Um, another one I really, really like

is this idea of Delta PINs um, and this is an idea that's been kind of floating around um, you know, a lot. I know some of, some of um, my posts and collaborators talk about similar ideas where um, if you have a geometry, like if I know that my

physics is evolving on a sphere or on this spiral or on um, like let's say I look at the CA of an ear, I know that that's kind of this spiral shape. So if I know the geometry, kind of the determine the boundary conditions and and where my

PDE is living you can actually you have a lot of prior information you can bake into the the PINs or or other machine learning algorithms. And so specifically if you look at the Laplace-Beltrami operator on that geometry, its eigenfunctions give you a very good natural

coordinate system in which to represent solution functions of that PDE. So kind of the simplest example that we've all seen before is that if I look at something like the heat equation on a rectangular geometry, the eigenfunctions of the Laplace operator on that geometry are sines and

cosines, and that we know that the Fourier transform of those sines and cosines are a good coordinate system that linearize or simplify that heat equation. So it's a very similar idea but for more complex geometries, that eigenfunctions of the Laplace-Beltrami operator on an

interesting geometry will be a very good coordinate system in which to represent your solutions of your PDE. This can be useful in things like cardiac flows. If I take a weird irregular geometry like my heart with all the veins and arteries, again I can look at the eigenfunctions

of the Laplace-Beltrami operator, and that gives me a good basis in which to represent and sum uh to approximate my PDE solution. And so you know there's a lot going on under the hood here. Um, I'm not going to get into, but this Delta

PINs kind of uses that idea to incorporate some idea of the geometry into the representation of the PDE and you get these improved representations. So um, you know, some kind of cool examples here. This is a heat transfer problem where you have um some

heat convection and some geometry with, you know, specified boundary conditions and you can see that you know the regular pin doesn't agree with ground truth as well as this Delta pin. So having, uh, kind of the the some knowledge of the geometry and of the PDE physics

lets you do better than just having one or the other. Um this is kind of a cool example where, um they use this method to compute geodesics on this kind of this geometry of a bunny. Geodesics is just a fancy word for the shortest distance path on a

manifold. So you know if I look at the shortest distance path from my location to, um, you know, Switzerland, it's the great circle that connects those two points on our sphere of Earth, uh, you know, if you believe that the Earth is in fact a sphere, um, and so these geodesics

are actually, um, the solution of a certain partial differential equation. I'm not going to get into that, um, but so you know there is some PDE that governs geodesics that you can add as a loss function and this just shows that when

you use this kind of geometry, these Laplace Beltrami eigenfunctions for this geometry, you can very accurately get those solutions, uh, for this geodesic problem. It's a really cool, uh, example I think and it's kind of a clever example to benchmark their their problem

on, um, and I don't really know exactly what this example is, but I think it's super cool that there is, you know, a dragon in their paper, which I like. Okay, good. Um, so those are just two of many, many, many examples of, you know, powerful extensions to this idea of pins.

again because it's such a simple idea you're just adding a loss function that is literally the pte how much it's being satisfied or not um there's a lot of ways of extending this to other uh kind of more complicated systems but there are some caveats

these don't always train that well sometimes they're stiff sometimes they overfit there's issues anybody who has you know worked a lot with pins knows you know there's ways of massaging them to work and sometimes it doesn't work okay and that's like every method no

method is a silver bullet that just doesn't exist um but it's good to understand what are the limitations uh and actually study them uh in a very methodical and process driven way and so that's one of the things I really really like about this paper um out of Michael

Mahoney's uh kind of group and and collaboration on characterizing the actual failure modes in pins and I'm not going to expect you to read all of this you should download the paper and check it out because they have open source code and you can play with it yourself

but you know some of the main upshots here is that um the main contributions as stated in this paper is that they they tried to train pins on relatively simple uh but relevant problems things like convection reaction diffusion you know simple pdes and they

basically found that um a standard pin often doesn't train and gives very large error except for you know really really simple parameter regimes of of these these pdes okay so they basically uh set up a problem a set of physics pde problems where they can show under

certain parameter regimes the method fails to train um and then essentially they looked at you know adding how what if you balance that that pte loss term so you can crank that pte loss term up or down that's a hyperparameter of how important the

physics is or is not and they found um that the loss function by increasing or decreasing that uh pte based soft constraint actually makes it more complex to optimize and harder to find a good solution in some cases um and I think this is a really

careful paper I really like um the work um by Michael Mahoney and his colleagues because they are very very careful in setting up this problem I don't think they're just criticizing pins they're actually trying to systematically understand when they

don't work and fix the problem and so they look at you know the actual neural network itself without the physics informed loss and they do in fact find that it does have the capacity to represent the solutions so it's not a capacity issue it's not the neural

Physics Informed Neural Networks (PINNs) [Physics Informed Machine Learning]

Steve Brunton

라이시, 페르디카리스, 카르니아다키스가 처음 소개한 PINN은 물리학 기반 머신 러닝의 핵심 알고리즘입니다.

PINN은 알려진 물리학(편미분 방정식)을 신경망의 손실 함수에 통합합니다.

이 동영상에서는 PINN과 그 확장 기능, 주의 사항 및 성공 사례를 다룹니다.

유체 흐름 예측을 위한 기존의 신경망 접근 방식은 공간과 시간의 함수로서 필드 변수(U, V, W, P)를 예측하는 것입니다.

PINN은 출력 변수의 부분 도함수를 손실 함수에 통합하여 이를 확장합니다.

나비에-스토크스 방정식 사용 예: 속도 필드의 발산이 0이 아닌 경우(물리학 위반) 손실 항이 추가됩니다.

PINN은 자동 미분과 역전파를 사용하여 관련 부분 파생상품을 계산합니다.

추가 손실 조건은 알려진 물리학을 위반한 신경망에 불이익을 줍니다.

알려진 물리학(유체 흐름, 양자 시스템 등)이 있는 시스템에서 잘 작동합니다.

실제 데이터와 가상 포인트를 모두 사용하여 네트워크의 물리적 특성을 평가합니다.

간단하고 강력한 기능: 자동 차별화를 사용하여 표준 신경망에 물리학을 추가합니다.

가상 포인트를 사용하기 때문에 상대적으로 적은 데이터 세트로도 훈련할 수 있습니다.

물리학은 손실 함수로 추가되며, 물리성을 강제하는 것이 아니라 제안할 뿐입니다.  훈련 중에 데이터와 물리 용어가 충돌할 수 있습니다.

드문 센서 측정값으로 유량장을 추정하는 데 적합하며, 4D VAR과 같은 이전 방법을 개선합니다.

제한된 측정값(예: 핵융합로)에서 데이터 동화 및 유량 추정.

제한된 데이터, 심지어 연기 시각화와 같은 정성적 데이터(예: 과학 논문)에서도 속도 필드를 추론할 수 있습니다.

데이터와 모델 합의의 균형을 맞추는 칼만 필터와 유사합니다.

한계: 불연속적인 시스템(충격파, 혼란스러운 흐름)으로 인한 어려움.

4D VAR과의 비교: PINN은 보다 현대적이고 종종 비슷한 접근 방식을 제공합니다.

유용한 동영상 및 튜토리얼 링크(후안 투스카노, 벤자민 모슬리).

PINN의 장단점을 이해하기 위한 실제 실험의 중요성.

누락된 데이터를 채우고, 희박한 측정값에서 추론하고, 추론 속도를 높이는 데 유용합니다(학습 후).

PINN과 나이브 신경망을 비교한 블로그 게시물(예: 스프링 질량 댐퍼 시스템).

높은 확장성: 다양한 물리, 데이터 유형 및 시나리오에 대한 많은 사용자 지정 PINN이 존재합니다.

분수 도함수와 적분 항이 있는 PDE용 분수 PINN입니다.

델타 핀은 기하학적 정보(라플라스-벨트라미 연산자)를 통합하여 복잡한 기하학에 대한 표현력을 향상시킵니다.

열전달 및 측지 계산에 적용된 델타 PINN의 예시입니다.

교육이 어려울 수 있으며 과잉 피팅 및 기타 문제가 발생할 수 있습니다.

Mahoney 등의 논문에서는 실패 모드를 분석하고 개선 사항(커리큘럼 정규화 및 시퀀스 간 학습)을 제안합니다.

또 다른 논문에서는 하이퍼파라미터 튜닝(데이터와 물리 손실의 균형)의 파레토 전선을 살펴봅니다.