Symmetry-Aware Neural Networks in Materials Science Tutorial

Tess Smith introduces the tutorial on Symmetry Aware Neural Networks, emphasizing their importance in materials science and machine learning applications.

### Symmetry Aware Neural Networks for Materials Science: MRS 2021 Fall Meeting Tutorial
* 👩 ‍🏫 Tess Smith introduces the tutorial on symmetry-aware neural networks, emphasizing the goal of making **equivariant neural networks more accessible** for applications in materials science.
* 📚 The tutorial will cover the **importance of symmetry** in machine learning models, the necessary **mathematics**, and examples of applications such as **molecular dynamics** and **material properties**.
* 🌐 Participants can access resources, including schedules, videos, and code, at **e3n.org/mrs**, and are encouraged to reach out via GitHub or email with questions.
* 🔍 The morning session will focus on **symmetry considerations**, with talks from Mario Geiger on **group theory** and Martin on using **equivariant functions in E3NN** for atomic geometry representations.
* ⚛ The afternoon will delve into applications of equivariant models, with presentations from Simon, Alby, Josh Rackers, and Nina Gentile on **molecular dynamics**, **electron densities**, and **phonon properties**.
* 📈 Tess highlights the significance of **symmetry** in achieving **high accuracy** in models, referencing DeepMind's AlphaFold2 and the Gordon Bell Prize-winning DeepMD model as examples of symmetry's **impact on performance**.
* 📊 Equivariant models are noted for their **data efficiency**, allowing for **accurate results with less data**, which is crucial for training on **expensive calculations** like coupled cluster methods.
* 📖 The tutorial aims to equip participants with the knowledge to develop computational tools that **understand and predict structure-property relationships** in atomic systems, ultimately **accelerating materials discovery and design**.

Hello, hello. My name is Tess Smith, and I'm here to start us off at MRS 2021 Fall Meeting, specifically the Symmetry Aware Neural Networks for Materials Science tutorial using the open-source package E3NN. Um, so I have the pleasure of introducing kind of

the whole goal of this tutorial, which is to make equivariant neural networks more approachable for those of you who have killer applications in the material sciences. And hopefully, this tutorial will help you get up to speed on kind of why would we want to have symmetry

in our machine learning models. Um, what is the mathematics we need to know to do that? And then what are some example ways that we can use these models for various applications, whether it's molecular dynamics, um, material properties, or electron densities, anything like that. And so we have a bunch of really amazing

tutorial instructors here ranging from EPFL to Harvard to Sandia National Labs to here at MIT. Um, and for all information regarding this tutorial, please go to e3n.org/mrs, which will redirect you to the full website, and that has the schedule, videos, slides,

Collab notebooks and code that we're using for this tutorial. So please feel free to go there. And since I'm guessing if you're watching this video, you might not be in person at MRS, if you have any questions, please feel free to reach out to us via GitHub. We

answer issues regularly, and you're welcome to also email any of us that provided tutorials for this. So with that said, I'm going to go ahead and give us a quick overview of what's to come. So the morning session, or the first three tutorials, are again focusing on

symmetry considerations and machine learning for the physical sciences uh that'll be given by me uh then we're gonna have Mario Geiger coming up next and he's gonna talk to us about kind of an introduction of group theory and equivalent interactions that we're going

to need in order to build machine learning models with symmetry built in and how this is done in our open source package E3NN and then Martin is going to give a guide on how to use equivariant functions as constructed in E3NN to give

invariant representations of atomic geometry that you can then feed to any sort of machine learning model and various properties of these invariant representations then in the afternoon or the fourth fifth and sixth tutorial we're going to discuss applications of equivariant models for molecular dynamics electron

densities and phonon properties of crystals so these will be given by Simon and Alby Josh Rackers and Nina Gentile and so that's sort of an introduction to the tutorial and without further ado I'm going to go ahead and jump in to give us a bit of an

introduction to why we care about symmetry and how it might affect our decisions for building machine learning models for material sciences okay so once again my name is Tess Smidt I'm a new assistant professor here at MIT and these blobs that you'll see here will come up

later in the talk but it's supposed to allure you to the fact that atomic systems tend to be symmetric but they also exist in 3D space and there's various types of considerations that we have for these types of systems and without further ado let's go ahead and

Introduce this problem, okay. So just to give some context as to why symmetry might be important here's some news items where symmetry has come up to be an important factor in the success of machine learning models on atomic systems. So this is from about last year.

You may have heard that DeepMind's AlphaFold2 got a really stellar accuracy of over 90% on the CASP14 challenge, which is a every two year challenge, and the goal is to go from amino acid sequences to a 3D folded structure, and

AlphaFold2 really jumped in accuracy. AlphaFold1 had about 60% accuracy. AlphaFold2 had about 90% accuracy. So really stellar jump in performance, and a really important aspect of the AlphaFold2 architecture is the fact that AlphaFold2 uses local and global reference frames

in such a way that it preserves symmetry on protein structures. And so that was taken into account in the way that the model was constructed. Another news item, again last year, the Gordon Bell Prize, which is often called the Nobel Prize of supercomputing, was awarded to the team DeepMD for pushing

molecular dynamics with quantum mechanical or ab initio accuracy to 100 million atoms with machine learning. So this is a really stellar accomplishment to be able to do quantum mechanical level accuracy molecular dynamics on 100 million atoms. Now the DeepMD model has some symmetry

considerations taken into account, but it's not a complete equivalent description of a system, which we'll define later. Um, the methods that we're going to describe today in this tutorial, importantly, are actually a thousand times more data efficient than DeepMD. What this means is that we can get more

accurate models by training on less data, and this is because equivalent models have Euclidean symmetry built into them. And one reason why you might care about that for your own models is so in the case of deep MD, these models are trained on density functional theory data, which

scales as n to the third, where n is the number of electrons. Um, but there are more accurate calculations out there, notably coupled cluster, which has a much significantly, um, significantly higher scaling, so n to the sixth or n to the eighth. And if you have a

machine-learned model that can get really accurate results with less data, then you can actually afford to train on really expensive to acquire calculations like coupled cluster. And a discussion of this implication will come up in the afternoon session with Simon and Alby, they talk about n-equip, which is an equivalent

interatomic potential model. But let's zoom out even further. What's our overall motivation for using machine learning in materials science? And one motivation, and probably one of the main motivations, is to build computational tools for understanding and designing the structure-property relationships of atomic systems. So

we already have some methods that can take 3D atomic coordinates and go to the properties of those material structures, like density functional theory or coupled cluster. These are some methods, or you know, even if we have a material structure that we've made in the lab, so

in actual synthesized materials, we can get some of those properties via characterization. But you know, we'd like to develop computational approaches or aid characterization methods so we can do this a lot faster. It currently takes a really long time computationally or just in the lab to actually like go from

structured properties we want to do this faster. Furthermore, you know, we have certain structures that we know about, but there's so many atomic structures out there that are waiting to be discovered. And so we would like some help hypothesizing structures that could

exist and could be synthesizable, such that we can actually, you know, make them ourselves. Furthermore, we might have conditions that we want a material to satisfy. We have properties that we need for various applications, whether it be for different types of memory and computers

or solar cells, and so we might have that laundry list of properties, and we would like some help by our machine learning algorithms to help us figure out which arrangements of atoms would possess those properties. Last but not least, it'd be really nice

to have a map that actually helps us understand which aspects of atomic structures lead to which types of properties and vice versa. So this kind of brings up the question: if we want these machine-learned functions that can take us from structures to properties, properties to structures, structures and

structures, and to combine it all in maybe a more approachable map of this really broad space, um, it seems like these functions need to be able to represent atomic structures and properties. So one question that kind of comes up is: how do we represent atomic structures and

properties? What is the form of the data that we're going to feed these algorithms? And how does this inform the assumptions we want built into our models? When we give it data, it has a lot of context. It's not really just a bunch of random

numbers in a file, a lot of these numbers carry specific meaning, so how do we actually carry that into our machine learning model? Okay, so oops, there we go, start there. So in general, when we describe physical systems, one of the ways that we do it, especially when we're describing it to a

computer, is that we'll use coordinate systems and 3D coordinates. So if I want to say where a molecule is and where those atoms are, I might use atomic coordinates. I may want to describe where the electrons are at, so I'll use atomic orbitals.

I might want to describe different forces that these atoms experience, or dipole moments that might occur in a molecule or material. I might also have more complicated properties that take the form of three by three matrices, such as polarizability susceptibilities, basically how does a

material respond when interacting with a magnetic or an electric field, and there's even higher order, more complex descriptions of these systems like elasticity tensors, which are rank four tensors. So we use coordinates and coordinate systems to describe what's going on in materials.

Now, coordinate systems don't actually exist in real space. They're sort of figments of our imagination that we find useful for conveying information for computers or to jot down on a piece of paper. So it's really important that we can always choose our coordinate system, and

we can change it if we like, because well, if it's an arbitrary coordinate system, and I choose a different one, I I better get an answer that still means the same thing. So we can always transform and choose our coordinate system, um, and we can change between them.

using rotations, translations, and inversions. And these are the symmetries of 3D space. Physics doesn't really matter, it doesn't really matter to physics where in space you are, how you're rotated, or whether you're using a right-handed or left-handed coordinate

system. Physics will still mean the same thing. Okay, so given this, how do we actually make symmetry-aware models for 3D data? And there's essentially three different ways of doing this before I get to that, just to emphasize again, here's a benzene molecule, here are

its coordinates written down, and these coordinates would change a lot if I decided to translate or rotate or invert my coordinate system. So the three different ways of imbuing your models with symmetry are the following. Sorry that the slides were skipping. Okay, three different approaches and my

feelings about each of them. So we have data augmentation, which is the brute force approach, and in general, this approach makes me sad, and I'll explain why. Second approach is invariant models, which is what most current models are doing, and again I'll explain this in a

bit. And then the focus of these tutorials that you'll hopefully continue to watch will be focused on equivalent models, and those are the models that I think really faithfully capture the complexity and richness of atomic systems that we're so interested in discovering and developing.

Okay, so machine learning models that are not built to handle symmetry, they don't understand the context of 3D coordinates and the fact that if I rotate a molecule, it's still the same molecule. Um, so these machine learning models that don't have this context require something called

data augmentation you have to show it the same object, let's say, with different rotations in order for it to get the idea that it should produce the same answer for each of them. Now, in two dimensions, when you have like images of rabbits or anything, you can get away with an augmentation of

about 10-fold. You can show it 10 different rotations of the images, and it roughly gets the idea that, you know, it can identify these objects in different rotations. But we're dealing with 3D space, and so this data augmentation gets a lot more expensive. It actually

requires more like 500-fold augmentation, and it can be even worse than that, um, if you really need high precision of angular, high precision. So, for example, if I wanted a network to be able to recognize this cube, I would have to show it the cube in many

different rotations versus if I have a model that sort of understands that coordinate systems are arbitrary. I can choose different coordinate systems, and I need to be able to get an answer that still means the same thing. Um, I just have to show it one cube, and it'll get the idea because that's

what it was designed to do. So, training with symmetry is a lot more efficient. It requires that you don't need as many parameters, and there's a lot of benefits to be gained from using models with symmetry when it's appropriate for your system. Sorry about the skipping. Okay, now let's

talk about invariant models versus equivariant models. So, invariant models compute invariant features of a system and then throw away the coordinate system. An invariant feature is something that doesn't change regardless of how I define my coordinate system. So, some examples are the distance between two atoms or two molecules is just a

distance and if I change my coordinate system that doesn't change so that's an invariant feature another invariant feature might be the angle between these three molecules um I can compute that angle and for that I do actually need a coordinate system but then once I

compute the angle I can throw away the coordinate system so an invariant model as I'm defining here is a model that only operates on invariant features so even though we may have needed coordinate systems to compute this angle we don't use that information once we get the angle

now equivalent models keep their coordinate systems around and if the coordinate system if we change it the output will change accordingly so if I give the inputs in this original coordinate system and I have these molecules but then I decide okay the second time I run the model I'm going to

use this coordinate system the model knows that it's still the same system just rotated and translated and inverted so interactions and equivalent models are more complex than in invariant models so in invariant models all we have is scalar multiplication so for example a number times a number gives me another

number or a 3D vector times a number just gives me the same vector scaled up or down so it's either bigger or smaller but the directional information is remains unchanged and so I'm defining that any models that only have these scalar interactions are

invariant models so how do we interact equivariant objects how do you multiply a vector times a vector well you might think in your head well okay maybe that's not so well defined I could do a dot product I could do a cross product I could do an outer product and in fact

you know it becomes more complicated so in the the most general operation for these vectors is to do an outer product and what's interesting is that when you do this outer product those other products that you might have thought of the dot product and the cross product

are actually hiding in that matrix so that three by three matrix has a trace the sum of the diagonals and that trace is actually equivalent to the dot product and it is invariant it doesn't depend on your choice of coordinate system you can rotate this three by three matrix and it

will still have the same trace furthermore the cross product is actually represented in the anti-symmetric components of the matrix and last but not least we have a symmetric traceless component and each of these require so the trace is only one number the cross product

requires three numbers to describe and the traceless symmetric part requires five numbers to describe so our three by three matrix has nine degrees of freedom and they're distributed across these different parts in a specific way and turns out that this is actually a very general phenomena

that we exploit in these equivalent neural networks and we'll go more into detail about that but if you're a bit familiar with um kind of the mathematics of addition of angular momentum or selection rules if that's ringing a bell this is exactly the same mathematics that plays out

in our equivalent neural networks okay so why equivalent functions again one of the reasons why you might want them is that they can substantially shrink the space of functions that you need to optimize over if you're training a machine learning model and this means that you need less data

to constrain your function space, your data becomes much more powerful in this more constrained function space. So, if we have all learnable functions, then we have all learnable equivariant functions, and we have all learnable functions constrained by your data, really the functions that you actually want to

learn in the case of, you know, a physical system that has equity in symmetry is really in this small slice. So, if you already limit yourself to this more constrained function space, your data is going to be much more powerful. Okay. What if we go even further and say

we just want to use invariant models that only use scalar multiplication? Um, you have to guarantee that your input features already contain any necessary equivariant interactions, like cross products or even dot products. You need to make sure that those operations have already happened and that your inputs, your

invariant inputs already encode these details and complexities in order for the model to be able to accomplish the task. So, again, looking at the picture, you have all learnable equivariant functions, and then there's all learnable invariant models that are only taking in

invariant features, and then there's all invariant models constrained by your data. Now, in some cases, it might be able to do it perfectly; the functions that you actually wanted to learn are contained within the space, but it's also possible that in order to accomplish

the task at hand, whether it's predicting energies or breaking forces, anything like this, that it's just not going to be expressive enough to to do the task accurately. And so, this is a downside potentially of invariant models; they are much simpler to implement, um, but sometimes they

lack the expressivity needed for complex systems okay so what do you get when you have equivariance to get at a high level um if we can identify a specific pattern in this case this rabbit if we can train a model that can identify this pattern we're guaranteed

to be able to identify any translation rotation or mirror or inversion of that same pattern this is guaranteed just by properties of the model given a molecule and a rotated copy let's say we're predicting forces the model will predict the same forces for the molecules just rotated

additionally the networks are going to be able to generalize to similar motifs it's aware of sort of local geometry it's not just sensitive to global geometry but also local geometry so it'll learn that certain motifs are correlated with certain properties and

so it's going to be able to generalize better Chris let's say we have crystal structures there isn't just one way to represent a crystal and there's many different choices of unit cell or unit cell conventions and so if you have assumed periodic boundary conditions and

you represent your model with in this case silicon with these three different unit cells an equivalent model is guaranteed to give you the same appropriate output for each of them the outputs will be on a different number of atoms because you gave it a different number of atoms but the values

and meaning of those outputs will be the same one really neat thing we can do is that we can use Euclidean neural networks these uh that are equipped with this um ability to operate and manipulate equivalent objects we can express tensors of atomic orbitals which occur a

a lot in electronic structure theory, and we can predict, for example, molecular Hamiltonians in any orientation from seeing just a single example. So on the left-hand side, I have a molecule, in this case, water, and it's just rotating around, and that's a relatively simple rotation.

But here's the Hamiltonian matrix on the right, where the rows and columns are atom-centered orbitals, and this matrix is indicating whether the interactions of electrons in these various orbitals are going to contribute positively or negatively to the total energy. And you can see that while this

rotation looks a bit simple, this rotation of the Hamiltonian looks pretty complicated. And so it's nice that our networks automatically handle these types of rotations due to their property of being equivariant. Another nice feature is that because we keep the coordinate system around, we can

actually actively manipulate geometry throughout networks. So we can use them potentially for generative, uh, models of even point geometry. Um, you can also do this for, uh, voxelized data as well, but you can also do it for points. Um, and for example, one kind of nice trick that we can do is

that we can convert local geometry into these equivariant features and vice versa via spherical harmonic projections. So let's say I have a local geometry of a tetrahedron. So we have these four points, and I want to convert this instead into equivariant features. I can do a spherical harmonic projection and

get these features. And this is going to feature prominently in Martin's tutorial, so the third tutorial. Uh, when we then use these types of features and operate on them in an equivariant manner to produce invariants that we can then plug into various types of models, not just

symmetry equivalent ones okay all right sorry for the computer lag I think my 5k monitor is giving my little MacBook a run for its life okay so how does uh Euclidean neural networks actually achieve um equivariance to Euclidean symmetry well

this is based on several works um that are some are listed here and there's even some that you know I've unfortunately just due to space have not included um but there's many people out there working in this space so there's many relevant papers if you're interested in

this and for the sake of this tutorial we're actually going to focus on a specific implementation of Euclidean neural networks in E3NN which is an open source repository that uses PyTorch on the back end for constructing for the neural networks and general

Euclidean equivalent operations and at a high level Euclidean neural networks are sort of the marriage of the flexibility of neural networks and the rigor of representation theory so just a quick reminder on neural networks there you go neural networks which is synonymous with deep learning is a

subset of machine learning which itself is a subset of artificial intelligence now any machine learned model can be abstracted to be some function f with inputs x learnable parameters w and some predicted output y now generally when we make a machine learned model we need to evaluate the

performance using some form of loss or error function if I was doing supervised learning where I have x's pairs of x's and y's and they're labeled I might use the mean squared loss so I take what my model gave me what I wanted my model to give me

subtract it, square it, and say that that's the loss. Now what's specific to neural networks is how we update the weights using the loss. So for neural networks, um, they are functions that have to be differentiable so we can update the weights with the following equation.

using gradient descent, we essentially take what our weights are currently and we subtract the gradient of um, we take the loss and we take the gradient with respect to every single parameter in the neural network. We slap a learning rate onto it, that's eta, saying okay how much in this

direction are we going to go? We subtract and then that becomes our new weight for our next iteration. And so we do this again and again and again until our model converges on our training data. Okay. What about representation theory? So representation theory is a branch of

mathematics that basically studies how do things change when you act on them with certain operations. In this case, our operations are specifically elements of a group. So the elements of the Euclidean group are rotations, translations, mirrors, inversions, screws, glides, any of these operations are elements of the

Euclidean group, and so how do things transform under the action of this group when we apply these group elements to our objects? This is also the branch of mathematics that brings us things like point groups, space groups, selection rules for spectroscopy, symmetry allowed forbidden properties.

You know, if you've ever heard the argument that you can't have a polarization, a vector polarization in a non-polar material, this is the branch of mathematics that helps you arrive at that realization. So when we want to construct our function f, our machine learned model

and we want it to be equivalent it has to satisfy this specific condition which is that all operations in this neural network so not just the overall model but even everything in between all the hidden things in the black box they have to be constructed such that they commute with the group action d of g

so even let's imagine f could be either the big machine or model or just an individual operation in my model d of g is representing let's say a rotation so d of g is how do I express how this rotation acts on x which might be let's

say a three vector so just it's a just some sort of vector x y z so d of g in this case would be a 3x3 rotation matrix so I can either apply the rotation to the inputs of f or I can apply it to the outputs of f

which you know maybe we want to predict something like a 3x3 matrix and so d of g now needs to act on both of those indices so essentially I can either rotate my inputs or I can rotate my outputs and I get the same answer and this is

guaranteed by the property of equivariance so my neural network has to satisfy this condition now importantly the form of d of g depends on what it's acting on so if it's acting on a vector it'll look different than if it's acting on a 3x3 matrix or or d

orbitals versus f orbitals um so all of these objects you need to know how to represent that rotation on that object but as long as you've satisfied this condition it's equivariant so it's a really easy unit test that we employ in

an e3, and so given this constraint, given that we have to be able to act on the inputs or the outputs and have it mean the same thing, what does this constrain our model to be? And it constrains the objects that exist inside the model, and it constrains

what operations we have. So we are applying our models onto geometry and geometric tensors, and we do that through geometric tensor algebra. So tensor products and related operations. Okay, so just to make that a bit more concrete, looks like my computer's having a bit of

problems. And okay, I think we're good. So the input to our network is geometry and features on that geometry. So let's say I have two point masses, they're at two different locations, let's say x1 and x2, but they're 3D coordinates.

So they have a y and z as well, and then we have features on these two point masses. We have their two masses, we have vectors of velocity and vectors of acceleration. So we have input into this, our Euclidean neural network, but then we also have to categorize our

e3nn Tutorial MRS 2021 Fall Meeting -- Tutorial 1/6 -- Tess Smidt

e3nn

Introduction to the MRS 2021 Fall Meeting Tutorial on Symmetry Aware Neural Networks

Tess Smith introduces the tutorial on equivariant neural networks for materials science using the E3NN package.

The tutorial aims to make equivariant neural networks more accessible for materials science applications.

Topics covered include the benefits of symmetry in machine learning models, relevant mathematics, and applications in molecular dynamics and material properties.

Tutorial information, including schedule, videos, slides, and code, is available at e3n.org/mrs.

Questions can be submitted via GitHub or email to the tutorial instructors.

The morning session focuses on symmetry considerations and machine learning for physical sciences.

Mario Geiger will discuss group theory and equivariant interactions.

Martin will explain how to use equivariant functions in E3NN to create invariant representations of atomic geometry.

The afternoon session covers applications of equivariant models for molecular dynamics, electron densities, and phonon properties.

Importance of Symmetry in Machine Learning Models for Materials Science

AlphaFold2's success is attributed to its use of local and global reference frames preserving symmetry.

The DeepMD model, while considering symmetry, is not fully equivariant.

Equivariant models are significantly more data-efficient than DeepMD.

The afternoon session will discuss n-equip, an equivariant interatomic potential model.

The overall motivation is to build computational tools for understanding and designing structure-property relationships of atomic systems.

Representing Atomic Structures and Properties

Physical systems are described using coordinate systems in 3D space.

Coordinate systems are arbitrary and can be transformed using rotations, translations, and inversions.

The choice of coordinate system should not affect the physical meaning of the results.

Approaches to Creating Symmetry-Aware Models

Data augmentation is a brute-force approach that is inefficient for 3D data.

Invariant models compute invariant features and discard the coordinate system.

Equivariant models retain the coordinate system, and the output changes accordingly with coordinate system changes.

Euclidean Neural Networks: Combining Neural Networks and Representation Theory

Neural networks are differentiable functions that update weights using gradient descent.

Representation theory studies how objects transform under group actions (rotations, translations, etc.).

Equivariant neural networks must satisfy the condition that operations commute with group actions.

The models operate on geometry and geometric tensors using geometric tensor algebra.