Automating PCB Design with Reinforcement Learning Techniques

Sergey Neno's Quilter AI automates PCB design using reinforcement learning, drastically reducing design time and improving reliability.

### Revolutionizing PCB Design with Reinforcement Learning
* 🎙 In this episode of Gradient Descent, host Lucas Bwal interviews Sergey Neno, CEO of Quilter AI, a company that **automates PCB design using reinforcement learning**.
* 🖥 PCBs, or printed circuit boards, are **crucial components in electronics**, connecting various components like **CPUs and GPUs** through complex wiring.
* ⚡ The design of PCBs is intricate due to physical constraints, such as **resistance and inductance**, which can lead to **signal degradation** and issues like **cross talk between wires**.
* 🛠 Traditionally, PCB design is a **manual and time-consuming process** that can take **months**, often requiring **multiple prototypes** to ensure functionality.
* 🤖 Quilter AI aims to automate the layout process of PCB design, **significantly reducing design time from weeks to mere hours**, allowing engineers to focus on other critical tasks.
* 📊 The company employs reinforcement learning to optimize the placement and routing of components, treating PCB design as a **game to improve efficiency and reliability**.
* 🔬 Although the initial simulations relied on traditional physics solvers, Quilter AI is evolving to incorporate **faster, heuristic-based simulations** to enhance the **reinforcement learning process**.
* 🚀 Ultimately, the goal is to **not only save time but to potentially design better PCBs than humans**, addressing the historical challenges of automation in this field.

### Exploring PCB Design Challenges and Reinforcement Learning Applications
* 🔌 Designing printed circuit boards (PCBs) can be complex, especially when dealing with **multiple layers** and ensuring wires do not **intersect**, which often requires using **vias to manage wire crossings effectively**.
* 📐 The Kuratowski theorem indicates that certain graphs cannot be embedded in a 2D plane, highlighting the **limitations of single-layer boards** and the **increased possibilities with two-layer designs**.
* ⚡ Power considerations are crucial in PCB design, as **current levels and voltage** can lead to **overheating and dielectric breakdown**, necessitating **careful calculations to ensure safe operation**.
* 🚀 The transition to high-speed boards involves using **differential pairs for signaling**, which requires **precise trace geometry** to **minimize noise and maintain signal integrity**.
* 🔍 As the design process evolves, regression testing becomes essential, with the team focusing on **improving speed and reliability** while ensuring users understand the **capabilities and constraints** of the software.
* 💻 The engineering team, comprising about 20 members, emphasizes a **balance between research and practical application**, transitioning from **pure exploration to focused development**.
* 🔍 The challenge of **generalizing across various board designs** remains significant, as the team continues to refine their models and processes to accommodate a **wide range of PCB complexities**.
* 👥 The company is actively hiring for roles in machine learning, reinforcement learning, and C++ development, particularly seeking candidates with backgrounds in **robotics and self-driving technologies** to enhance their capabilities.

you're listening to Gradient Descent, a show about making machine learning work in the real world, and I'm your host, Lucas Bwal. So, I'm out of my garage and in our beautiful office in the Mission, and today I'm talking to Sergey Neno, who is the CEO and founder of Quilter AI. Quilter is a super cool company because it automates PCB design, and PCBs are in pretty much every piece of electronics that you might use. It's an amazing example of a company that's actually using reinforcement learning for real-world applications. I remember Andrew in his course years ago telling me that there really were no reinforcement learning applications working in the real world, and well, here's one that's actually working. Sergey talks about not only his company but how he got into the field of machine learning and became an expert in reinforcement learning, and some of that dates back to Andrew's class, which I took as well. I hope you enjoy this [Music] interview. All right, so Sergey, um, you work

on AI applied to PCBs. And so, when I was first researching you, I was kind of realizing I have this vague sense that PCBs are those green things inside computers and other electronics, but kind of realized I don't really exactly know what a PCB is and why they're important. So, maybe you could start by talking about that. Yeah, yeah, gladly. So, I brought props. They don't have to be green; they can be any other color. But this is what we're talking about, right? So, this actually is just a computer motherboard. And what it really is, is it's basically fiberglass on the inside, and then a bunch of different layers of copper and insulating materials. And the job of the PCB is to basically connect a bunch of the components. So, if you look closely at the PCB, there's like, you know, all these capacitors, power components, there's a CPU here, GPU plugs in, all that good stuff. And you can really imagine that a PCB is just like a bunch of wires.

right, connecting those components together. Um, but of course, uh, it's not quite that easy. And why is it not quite that easy? I actually just imagine essentially a bunch of wires and places to solder things into or push, uh, chips into. What's what makes it more complicated? Yeah, so um, basically physics, right? Um, so all of the communications happening between those chips, um, you know, you cannot assume that the connections the PC is making are ideal, right? They have some resistance, they have some inductance, and so uh, you have potential signal degradation over those signals. The board can potentially heat up, um, and in general, like especially once you get into multiple gigahertz, you know, two, four, five gigahertz, your wavelength of the communication is about on the order of the size of the PCB, which means that all of the wave dynamics that happens in electromagnetics is now occurring on your PCB. And so you kind of have to um, account for all of the imperfections that are going to happen as a result of

you having a physical object, uh, and account for them in the design. Interesting. So, what's an example of some way you might modify the design to make the board work better? Yeah, so, um, in general, like, the big issue with designing the PCB is just like, are you confident it's going to work at all, right? So, this kind of board that I'm talking about, right? So, you know, this is, um, I'm actually, I haven't counted, but probably hundreds of components and thousands of connections, and you're talking about drawing all of these little wires on the back, and then there's probably another 10 or 16 layers inside of there that are vertically kind of looking similar to that, um, so it's very time-consuming. Um, this is done manually today, uh, in almost all cases, and so somebody's sitting at something that looks like Photoshop or Paint, uh, but like more specific for PCB design, of course, and is drawing all of these connections, and that could take, for a board like that, that could take months.

um, so um, what you really want to do is you want to make sure that when you're finished, the board actually works right. And this is quite that you know you fire up the board for the first time and not everything works on it right. And so you're trying to figure out why, trying to debug it um, and then eventually you'll have to do things like past FCC testing to make sure that the board is not kind of creating too much interference with everything else. So an example of something that might go wrong that isn't obvious um, imagine you have two of these wires running next to each other and they have some sort of high-speed communication uh, going along them. Maybe it's like a DDR connection to a memory or something like that. You'd think that you know wire goes from A to B, different wire goes from C to D, and that's it, and they stay isolated. In fact, that's actually not the case right. The wire is basically an antenna. It creates electromagnetic energy that electromagnetic energy couples into the

other wire. All of a sudden, you've got noise on that wire, and so that could then ruin some sensitive signal or ruin that communication or whatever else. Um, so this is called cross talk. So, figuring out how to make that not happen is one of the things that you might be thinking about as you're designing a PCB. Um, and there's way many different ways that it could kind of happen, many ways that the geometry that you set up kind of leads to it. Uh, but in general, like, you know, keeping them far apart is a good start, and really the only way to test it is by manufacturing it and trying it. I mean, we've talked to a lot of different roboticists on this show and talked a lot about, you know, kind of simulating um mechanics in the real world. Why is it so hard to simulate um, I guess the uh electromagnetic interactions here? Yeah, so um, it's it's practically difficult, not theoretically difficult. Um, so like all of the physics is known, right? It's just electromagnetics. It's

just max equations. There's a million different solvers for them. You can set them all up and that kind of stuff. That's fine um but practically speaking like you know there isn't just a tool where you like Submit Your Design click go and like cicd runs and checks all of the different things that could go wrong and you get a thumbs up or thumbs down. Instead like today as a designer you have to know what question to ask. You have to say oh I'm worried about cross talk specifically from this Trace to this Trace. Let me then go into a simulation tool and model it out and run the simulation and so on and so forth um and practically speaking like most folks that are designing boards you know have limited or no experience actually doing the simulations um so a lot of times people just say Well it I followed the rules I typically follow it's probably going to be fine. Let's just order it. It'll be on my desk in a few days and we'll find out. So it's definitely possible but just um the tooling isn't

so good that you can actually click a button and go through all of it that easily. But isn't it pretty bad if they fail? Like, doesn't your old device then not work? Definitely. Yeah, absolutely. Um, so I mean, this is, uh, this is where the iteration begins, basically, right? So if you ordered, you typically don't go and straight, like, order a million units for your device and go into production. Um, instead, you would, you know, do a prototype run where you do a few boards, you validate that everything works, anything that didn't, you, you know, debug, figure out why, make changes, go through a few revisions, and then when you're kind of really confident that your design is good, you go and ramp up to production. Um, for sure. Okay. And so where, where do you come in? Like, what are H, how are you improving this process? Yeah, so what we're trying to do is to automate um, the layout process. So um, to elaborate uh, designing a board is roughly three major steps, right? The first is creating uh, what's called a circuit schematic. Um, so you can

picture a circuit schematic, kind of like a block diagram or a flowchart, um, where you basically have, like, these are the chips I'm going to use, here's how they're connected, and so on and so forth, um. So, by analogy to software, I think of that kind of like writing code, right? That's where you're specifying, like, what is the inputs and outputs of your board, um. You can think of the chips as kind of like libraries that you're using with some off-the-shelf functionality and so on and so forth. The second step is called layout, so that's where you take that schematic and actually convert it into a blueprint for, you know, actual fiberglass and copper that implements that schematic. So, I think of that as kind of compiling your code, right? The job of the circuit board is just to make the schematic real, and then once you have those blueprints, you can send it to the manufacturer. The manufacturer can put it together, you get it back on your desk, and you know, you find if there's issues and you repeat, um. So, what we're trying

to do is um we're trying to take layout, which is today completely manual and time-consuming, and we're trying to automate it completely. And how are you doing that? So the approach that we take overall is by using reinforcement learning. So the way that we see this problem is that you can kind of think of PCB design as a game, right? Like at a simple level, you've got a big box, and you're putting little boxes in that big box, and then you're connecting them, right? And so the way that we are setting this up is that we've got this kind of simulation or engine that's running this ability to place components, to route components, so on and so forth. And then eventually, once the board design is completed, it gets passed to a bunch of physics tests. So that's us building out the kind of like CICD of enumerating every single concern and making sure that they all actually work. And then you can feed that signal back as a reward.

and then train an agent to get better and better at this game. That's roughly how we're approaching this. So, is the physics simulation also done using machine learning, or is that like a set of tests based on laws of physics? I guess, yeah. So, this is something where we're evolving over time. So, the first thing we actually did when we started this, we used a full wave-like Max equation solver, just classic off-the-shelf open-source implementation that's used by a lot of researchers and well-validated. But we found that it's just too slow, right? So, we have this issue that running even a single simulation, despite it being GPU-accelerated, can take, you know, 20 or 30 minutes, something like that. And you would have many of those to do throughout the board because you have many, many different questions. And, of course, in a reinforcement learning context, that's quite painful and slow. So, what we're doing now is we're going back and seeing whether we can use

either heuristics or much simpler simulations along the way, uh, as those reward signals. So, um, you know, you can encode conservatively some of the concerns. It's just raw geometry, um, or by doing analytical approximations to the max equations, or by doing approximations like quasi-static, um, things of that nature. And like, that's a part of the research that we're in right now, is figuring out what that's all going to look like exactly. Interesting. And, and so what's the end result of this? The more reliable PCB? Yeah, so I think that there's roughly two pieces of value that we'll focus on overall. Um, the first one and the one that we can actually kind of deliver on today is time, right? So, the most important thing in my opinion right now is that, um, you know, even a simple board could take a week to design by hand. Um, and that's a week that you're not testing it, that you're not building the firmware for it, that you know, you're waiting to ship your product basically.

and so we can compress that from, you know, a week or two weeks or whatever it is, down to an hour or two, uh, and release that bottleneck. That's probably the first thing. The second benefit of that, of course, is now you have an engineer who has time to go focus on something else, right? They can be focusing on, um, you know, validating their parts list or on, uh, initial bring-up or on writing the firmware or on their next schematic design or whatever else, um, so time is really the first major benefit that we're focused on. Uh, I think eventually, like, not today, but eventually, I do think that we'll be able to design better boards than people, um, and the reason for this is that when you design a board and you know it's going to take a long time, you stack a lot of margin, right? Like, you make the board a bit bigger than it has to be because you're worried about running out of space, um, you know, the things like crosstalk and whatnot, you assume some conservative rules to make sure it's not

going to bite you um and because of that I think that you know there's probably 10% or more margin in basically every single board out there uh and So eventually when we can do this job better than humans can there's potential to design better board boards than humans can. It's funny I would think it would be easy to design better boards than humans can right if humans are just sort of laying things out by hand. This doesn't seem like a task that humans are designed to do very well is it is it really so hard to get super hum performance. It is um so this problem um it's very interesting. I mean this has been researched for 60 years now right. So the first papers on automating board design are from like 1961 62 63 um. They were doing things like graph embeddings and planes to try to model this problem and like practically speaking those tools just still don't work right. Like nobody does their layout with an automated tool um Auto routers are notoriously hated in this industry.

uh, so, so it's, you know, um, making a machine that can actually do this has proven difficult just historically. The basic intuition for why it's hard, like, it's funny because we as humans are actually really good at spatial reasoning, and so to us this problem seems pretty easy, and it would seem like a computer would be good at doing it, but it's definitely tricky. So, I'll give you one piece of intuition for how, um, let's forget about all the physics, let's forget about, you know, imperfections in manufacturing, all that stuff. Let's just focus on placing components and routing them right, like connecting the dots. Um, if you've placed your components, you know, routing your first trace is really easy, right? You've got a start point, you've got an end point, you've got some obstacles, that's a maze solver, right? CS1A kind of problem. The difficulty comes up in that when you've solved that problem, that solution is now an obstacle to your next puzzle, to your next maze. And so, the real

challenge is how do you make sure that your 10,000's Trace still has enough space to be completed right. If you're just going to greedily connect the dots, you'll surely block yourself off and not be able to complete. And if you don't complete the board, definitely doesn't work. Uh, and so that's a deceptively challenging problem to solve that humans are actually pretty good at. OD, that's really amazing that um computers can beat the best Go player, but they can't lay out a bunch of wires better than a human. It's kind of blowing my mind. Is part of the sort of like fuzzy constraints of like you might move, you know, something on the board or something like that. I mean, I don't think that there's any kind of like fundamental reason for this, right? Like I'm sure if DeepMind had chosen to tackle PCB layout rather than Go, they probably would have done a good job. Um, and also you don't necessarily even need a superhuman ability in order to

make a useful product right, like even if you have the ability to do, you know, the easier half of all the board designs out there, that's already very useful. So, so I think that that, you know, it's definitely a problem worth solving and can be solved. I think that the difficulty is, um, I mean, there's a few pieces of difficulty here, right? So one is just the raw combinatorics, right? Like you have, you know, tens of thousands of traces to eventually do some of the bigger boards, like, uh, you know, these server cards for many GPUs could have hundreds of thousands of traces, uh, and for every single one of those traces, there's a lot of different ways that it could go. And just exploring all of them is, is, is not possible, right? So you have to have some level of um, intelligence, some level of pruning to figure out which strategies are going to work. Um, of course, just connecting the dots is insufficient. You have to account for the physics, and that's kind of the second big difficulty, is even if you've connected the dots.

that's still not good enough, right? You still have to go check those constraints. Um, so yeah, I think that's kind of what it is, just huge combinatorial space and a lot of important manufacturing and physics constraints. And now you've started to build a company around this and really take on real-world challenges. What's happened in that process? Like, I mean, you've laid out a very clear, well-defined, um, you know, reinforcement learning problem that I think, you know, every grad student would be jealous of. But I'm quite sure that as you've built a company around this, you've come across surprises. Like, what have been some of the surprises and learnings along the way? Oh, I mean, lots. There's no part of starting a company that isn't full of, uh, full of lessons and surprises. Um, I mean, I think, um, probably the most interesting thing for us to balance and try to figure out is like how do you simultaneously invest in, you know, basically research of like solving this

very difficult problem, but also make progress on finding applications early, delivering something that customers can actually start to use. Um, you know, and that's definitely tricky. That's something that we spent a lot of time focusing on. Um, so we've released our tool in an open Beta that people can just kind of upload boards and try it out, and we're encouraging people to build these things so we can see what happens. Um, and, uh, yeah, I mean, I see a lot of people submitting designs. Uh, that's super exciting. A lot of the designs are things I didn't think people would be doing. Um, people are learning to use this in interesting ways. So, like, I'll give you an example. Um, you know, uh, we saw a person submit basically the same board. Uh, 10 different times. Um, and we're wondering, like, what are they doing? Why are they doing this? Um, because, you know, you typically think, like, I have one codebase. I click compile. It comes out.

and there's my board like I'm done, uh, and what this person was telling us is that they were trying to figure out how small they can make the board. So what they end up doing is just giving it like 10 different constraints in terms of sizes and just say R of them. Let's see which one works, um, and they were saying that like it's changing how they're thinking because now the layout is kind of free, whereas before they had to like pay someone and ask someone to go and do this, and so they would never like explore a bunch of board sizes, uh, if they had to kind of pay for it and wait and all that stuff, um. So, uh, I think that is showing that like when a compiler starts to exist for electronics, like the way that we design electronics is going to change pretty drastically, uh, and I think that is the most exciting thing to me, and that's where I'm, you know, most looking forward to all the lessons. Well, why do you say when a compiler exists? I mean, it sounds like

you've made a compiler and released it. Is it? Are there things that hold it back from, um, you know, being from everyone using it? Absolutely, um, so we, uh, do not claim that we've solved the whole problem, uh, we're, we're, we're far from it, frankly, to get to the point where, uh, we can solve any kind of board out there, right, um, so there's two axes of complexity that we're measuring ourselves against, roughly speaking, um, one is just the raw combinatorics, so, you know, how many components can you fit on the board, how densely, how many pins are there to PR, right, that's kind of one set of problems, and the other is the physics, right, so what kinds of physics concerns can you, um, can you take care of, so, you know, on the combinatoric side, uh, we are currently solving boards that are about, you know, 1 to 2,000 pins at the maximum, which is like, okay, you know, for, for simpler boards, that's certainly fine, for some test boards, is certainly fine, but

like that's not nearly enough for like a motherboard, right? Um, you know, your CPU alone has like 1,400 pins, never mind all the other components. So, we have to improve by an order of magnitude to get to the point where we're doing like a computer motherboard. Even combinator, can I ask, can I interrupt? So, of course, you know, I've put together, I don't know, I mean, I've looked at a motherboard. The CPU is like 128 pin or something, isn't it? Why? I mean, why do I have so many? Um, if I'm not mistaken, I'm thinking about the, uh, was the LGA 1147 socket. Um, and I think that's the one that has that many pins. Um, it's pretty extraordinary. Um, I'll have to double-check the exact number at the end, but it's see F. I don't think it's multiple thousands, but it's over a thousand for sure just to connect the CPU. Yeah, in wow, okay, yeah, yeah, exactly. And then I guess there's tons of other

things that it connects to, which makes for more pins exactly. Um, yeah, exactly. So, you know, your CPU is connecting to your memory, to your audio system, to your HDMI video, all of that kind of stuff eventually ends up going through the CPU. And so, uh, you're doing all of that in parallel through all of these pins. Uh, and then, of course, each of those um components has their own, you know, breakouts, their own complexity, so on and so forth. So, it's quite easy to get into the tens of thousands of pins on a complicated board. And the wires, can they cross? They can't cross, right? Like, what do they do? They go underneath each other or above each other or something like that? Yeah, you can imagine it kind of like um, what's a good analogy, like a building or a parking lot or something like that. So, you have these layers um, and the layers are just planes of copper and they're isolated by insulating materials. And so, you can uh, take arbitrary shapes uh, into

two dimensions on any given layer, and then you basically drill a hole in the PCB and install kind of like a rivet, um, and so that's a via, and that's what allows you to jump between layers, and so the puzzle is to basically arrange all these wires across all of these layers such that they obviously don't intersect when they're not allowed to, um, but of course connect all of the start points to all of the end points. And can you always add layers? Like, I guess I would imagine if you could always add layers, you could eventually do it, right? Yeah, certainly, like, yeah, if you just added as many layers as you had nets on the board, then this is a trivial problem, uh, is, uh, because you still have to find a place for the vias to jump down to those layers, but layers is, um, they, uh, increase your manufacturing cost, cost, and complexity, right? So a very simple board will have just two layers, right? And if you're trying to make a really, really cheap board, that's what it'll be, so like

if you've ever seen an Arduino Uno, right, that one will be a two-layer. Well, the old one, the R3, was a two-layer board. Um, and that just makes for cheap, fast manufacturing. Uh, four layers more expensive, six layers more expensive, and practically speaking, like, probably the biggest stock ups that are around today are in the like 50, 60, 70 layers kind of thing. That's usually where you top out at that Uno. I think I had one of those back in the day, maybe not Uno, but I think so. I mean, Arduino has quite a lot of pins, doesn't it? Uh, yeah. So, how does that work? Like, I would think with two layers, it would be impossible to connect all the pins to the um, uino. What do you even call it? Microcontroller? The chip? Yeah. Wouldn't they have to cross somewhere with two layers? I mean, how does that work? No, um, definitely possible. I mean, a lot of boards are two layers, and even relatively complicated boards like like

I've definitely seen boards that have thousands of pins that are um two layers um. Even more fun challenge is when you make a board at home so like you can actually do the process of eding a board yourself pretty easily um at home uh. And the easiest way to do it is a one layer board uh. And so at that point then that becomes really really challenging how to not intersect at some point you can only handle very low complexity with that um. But roughly speaking these wires right they're relatively thin and so you have a lot of space to kind of put them side by side with each other and then where you have to have a Crossing one of the wires takes a via jumps under goes on the second layer and comes back up uh. And so that's roughly how you de intersect them is where you were going to have a collision jump one down to a different layer cross over and then jump back up. Oh I see and it can do that at any time yes exactly because I'm pretty

Sure, with the one-layer board, I'm digging deep into my math undergraduate, but I'm pretty sure there's some proofs that you can't do arbitrary graphs on a 2D plane, right? 100% yeah. There's a theorem called the Kuratowski theorem, which states the kinds of graphs that you cannot embed in planes. And I think it's actually an if and only if kind of statement where you basically show that a fully connected five graph and some types of three graphs, if you have those, it's impossible to embed. And so, yeah, at that point, it's definitely impossible. But I guess with two layers, everything is possible. It's just more like physics that gets in your way, is that right? Yeah, physics plus manufacturing tolerances. So, all of these wires are real. They have to be manufactured with real dimensions. And so, for example, a typical manufacturer can make a wire

or Trace um that's you know say five Ms wide or 5 thousand of an inch um they also have to respect some separation uh because if you get too close the etching process might not itch them apart correctly and you could have a short where wires are connecting when they shouldn't have been connecting uh that might also be 5,000 of an inch um and so at some point if you have a lot of density you know these wires end up stacking with each other end to end to end to end and at some point you just run out of space okay so I took you down a long digression I apologize but um it sounds like you could do boards up to to a few thousand connections which sounds like that would be like a lot of the boards out there but I mean what what else is um is kind of keeping you from making this in production everywhere yeah um the physics is definitely the second biggest part um so at this point we are um we are basically one consideration at a time adding the

different constraints that we can handle on these boards. Um, so for example, um, you know, if you go to quilter today, you'll see that it has consideration for power, right? So you can do like a 2 Amp or 4 Amp kind of power supply, um, which is nothing crazy, right? Like you're not going to run an electric car with that kind of current, um, you know, but, uh, you know, but for like a separate motor or something like that, that's kind of plenty. And so that's something that you'll see, and quilter will intake, it'll check that it does it correctly, and then when it gives you the result, um, if we weren't able to guarantee that that Trace isn't going to basically burn in your hands, um, then we'll tell you that we failed to generate the design. Okay, wait, I'm thinking, sorry. I mean, this is like a basic primer on PCP, I guess, but when I think about power lines, it's the high voltage lines that need like special treatment and bigger things. Why did you say it's the amps that

constrain you on a PCB. Yeah, so it's both um so the issue with high voltage um is that above a certain level like above 50 volts you have to start considering uh dielectric breakdown uh and arcing right so if you have two you know 100 volt um pieces of copper next to really really close to each other um they will break down and ionize air and it'll spark and that's obviously not not good uh and so that's not something we currently support actually uh and so you know please don't upload a 100 volt board to quilter and and hope that it works because it probably won't um the second is the current because the current what drives the heating um so your wire is basically a resistor right uh it has some amount of um of resistance and so as the current is kind of going through it uh it's going to heat up and so typically PCB designers will will do a little bit of math to figure out um hey I want to make sure

this you know copper doesn't rise more than 10° Celsius above ambient, and then you have to figure out how wide to make that wire basically, um, to then drop the resistance to a level where, uh, it doesn't overheat. I see, so the thicker wires are for more current. Yes, exactly. Got it, got it. Well, that seems like something you could handle. Is that, that's just another implementation detail? Yeah, exactly, uh, I mean, so that, that one's pretty easy, pretty straightforward. Um, what we're focusing on now, um, is going to get us into higher-speed boards. Um, so there's this, um, type of signaling that happens on high-speed boards called differential pair. Um, so where you're sending a positive signal in one wire and a negative signal in the other wire, uh, and then at the destination, one of them gets inverted, you add them together, it cancels out the noise, and you can communicate faster, basically. Uh, and so there, there's some kind of special geometry to how thick to make those traces such that basically

the speed of light in them, uh, is matched to the starting and end points, um, how far away they are from each other, to control what's called the differential impedance between them, uh, and then you have to run them really, really close together because if they do come apart, then they'll capture a magnetic field going between them, and that'll introduce opposite noise on the two different wires, and then when you invert them, that noise will amplify and, uh, do bad things, um, and so that's something we'll ship here in a few weeks, um, that'll allow us to do much faster designs, which will be fun, but after that, there's a lot more things we'll have to add, and now what about just making your models work because I think that's an interest of a lot of people that, um, listen to this podcast, like when you're trying to adapt your models and, and kind of make them ready for prime time. If you had to kind of change the way, um, you build them, test them, deploy them, all that, um, I think probably the like one of

The benefits of reinforcement learning is that you can give your models as much or as little control as you like, right? So, for us, you know, the agent playing the game doesn't look like, um, you know, what a human's doing. It doesn't look like it's like taking a trace and, you know, moving one pixel right, one pixel up, one pixel right, one pixel up, um, that kind of thing. Um, we take a lot of that complexity and just actually move it into classic code, and really what the models are doing is kind of higher-level planning, right? So, instead of saying one pixel right, one pixel up, it's go from here to here, go roughly this way, and then a classic set of code actually fills in the details of how to get that done. Um, so, we handle a lot of kind of constraints not by hoping that the model gets it right, but by just like enforcing it as a constraint in the game where it's just like an action that the model can't take if it's going to be wrong. Um

now testing is definitely a problem. Whenever we make changes, whether it's to models or to actual code, we have to run a lot of regression testing, and that's challenging because we want to run at least 20, 30, 450 boards through this kind of regression testing and more over time. Each of those takes substantial amounts of time, right? So for even 6 months ago, a lot of those were taking 10 hours a piece on a cluster with a whole bunch of compute resources. And so that makes for a not very fun development experience to sit around and wait for a day and then see that one board dropped by 1% completion and like that's a regression and you have to figure out why and that's not easy either. So yeah, we invest a lot in trying to make those regression tests much faster, but also getting a lot more information out of them so that when they fail, we have a lot more information.

about why rather than guess and checking why well how do you make these regression tests faster because you actually have to run this um reinforcement learning algorithm which famously takes a long time. Yeah yeah I mean uh the biggest liver that we've pulled over the last six months is taking a lot of our like um you know research grade python implementations of these model of these well not the models but the um the environments that that we running for the RL um and switching them to C++ and paying a lot more attention to speed uh and so that's been a huge set of speedups um you know primarily on the kind of routing side. Our Placer was already pretty fast because it was already written in like Cuda kernels uh pretty much directly um so the nice thing about that is you know if you make your environment faster it's much easier to train uh you know your your cycle times get a lot lower um you can run faster in production you can explore more candidates uh and uh we've taken a

lot of boards that were taking us like tens of hours before to now you know a few minutes. Uh, with these changes. Um, so that's probably the biggest way. It's just invest in making the environment as fast as possible. Um, is there anything else that you've added to your stack as you've tried to get that um last level of reliability so you could deploy it to customers. Uh, I mean, I think it's um in terms of the reliability to customers. We're going to be adding stuff all the time. Like I think the biggest thing that we are um currently not doing well enough is making it very clear to users like what kinds of considerations we can handle and what kinds we can't. Right. So a lot of people um will come to quilter and upload a board and they'll just assume that it's like magic and that it can like read and understand all the schematics and that like it'll figure out all the high-speed stuff and all the you know high current.

stuff and everything, and then like they see the result, and they're like, That's clearly wrong. It didn't do any of that. Um, and that's, I think, that's our fault for not basically saying, like, Hey, these are the things we can do. If it's not on this list, we can't do it. So, if your board requires that, like, Sorry, like, upload the feature, and we'll get to you soon. Um, so that's the part that we're kind of fleshing out. And now, in terms of the user experience, is a much clearer dialogue with the user about what constraints are on your board, what physics is on your board, what we can do, what we can't do, and then grading our homework and saying, Hey, we have met these constraints, or We haven't. Um, but that's going to be a journey that continues for years, I'm sure, as we get into, you know, faster and faster boards, more and more difficult boards, so on and so forth. Totally. And I guess at this moment, how do you divide up

your team between people kind of like researching the reinforcement learning to people kind of doing the engineering to to other yeah um so we're uh between full-time and part-time about 20 people at this point uh a little bit less um the vast majority of that is engineering obviously um so uh what we think of as our compiler team is currently 11 or so people and a few offers that have that have gone out with people starting soon uh which is super exciting uh we've got I think about four people right now on what we think of as the application team um so that's you know UI backend infrastructure uh all of that kind of stuff um and then we've got some folks on the product team uh in terms of like design some electrical engineers who are using the product informing it building boards with it so on and so forth now how do you run a team this is a question I get a lot like how do you run a team that's doing really new research like are you able to actually

operate to a schedule. Do you do Sprints and Sprint planning? Yeah, that's a tough question. Um, I don't know that I have great answers for this, to be honest. And if people do, I'd love to hear them. Um, I think that there's um, we still try very hard to to keep schedules and to keep uh Focus um, despite a lot of it being research, right? So um, we had especially earlier in the company um, you know, we were in just pure research mode, and so that was a lot more exploration, finding papers, replicating papers, coming up with their own experiments, casting a wide net with a lot of different bets to see what works out. Um, now we're kind of transitioning into a more focused, more fast kind of mode of development, where we're a lot more clear about the goals, a lot more clear about the projects. U, you know, have a higher bar to kicking something new off um, and I think with research in particular, like, there's still a lot of it that is actually just good engineering, right.

like if you want to run a good experiment, right, you have to set up, you know, your environment. You have to ask a good question. You have to set up all your instrumentation to track the question that you're asking. You have to collect the data. You have to actually make it run fast, all that kind of stuff. And a lot of that is actually predictable, good engineering work, um, but it's serving some, you know, specific question that you don't know if it's going to go well or not, um, and yeah, I mean, I guess we just have to accept that, like, some of these things don't work out, um, and uh, we try to focus on um, finding places where we think there's a clear path to make things better, uh, clear path to improve, you know, uh, the models, the environments that will go straight into production, uh, but also allowing some amount of time for for bigger bets that might not work out, um, well what um, what fraction of the things you try really don't work out or really clear failures, uh, at the beginning we had a

lot um so one of the things that we tried to do um was to borrow heavily from the chip industry um so uh there's a similar problem to this within chip design uh which is actually laying out insides of a chip um and on the surface of it that problem looks really really similar to this problem of PCB layout um and it's a lot more successful in automation right like chips like um you know like an Nvidia GPU that has 80 billion transistors certainly cannot be done by hand U it certainly must be uh at least partially automated uh and so we've looked for a lot of research to try to duplicate right uh different placing techniques different rout and techniques so and so forth um and unfortunately the vast majority of them just did not work out for us um for a bunch of different uh bunch of different reasons so we probably reimplemented I don't know 15 20 different papers that uh you know we got similar results to what the paper showed applied to the PCB.

problem, and it just didn't quite work. Um, so was quite a lot um by now. Uh, that we've kind of formed our own internal um approaches and thinking for how to do placement, how to do routing, how to couple it with uh with agents. Um, and I think our success rate on ideas is a bit higher uh than you know 99% failure. Nice. I guess like one of the interesting things about your background is that you're not originally. You didn't study uh reinforcement learning or even I think computer science at all, right? Like you know how did you um get into the field and how did you get up to speed? Yeah um so I I got really interested in it like in 2015 or 2016 or so um and at that point um you know I saw Andrew in's courses. I was like all right this is going to be great uh so I went through you know the whole all right here's what a neural network is, here's how the math works, here's back propagation, you know all of that kind of

stuff wrote it all out from scratch to make sure I understood it. Um, and then I just looked for places to apply it. Um, that's really kind of it, like on kind of interesting problems, different types of things. Is that yeah, exactly. Um, so I think, uh, you know, I played around with a lot of the, um, uh, what was it called, Kaggle. Kaggle challenges, that was super fun. Learned from some of the people who were succeeding at those, um, you know, and then eventually when we got into starting this company, uh, obviously, you know, uh, became a full-time job and had to step it up a notch. And when you say you wrote it all out, the back propagation, do you mean you kind of wrote it out in math or you wrote it out in code? Uh, both. Um, so engineering courses, I don't know, probably a lot of viewers here have gone through those. They're excellent. Uh, and he does, he does both, right? He works through all of the raw, the math, the linear algebra, and you know how it gets.

represented and then uh walks you through actual submissions of code. I forget if it was in MATLAB last or something. Maybe it was Python. I don't remember uh but yeah you actually write everything out as well uh which is great. Nice. I think it used to be MATLAB. Probably moved to Python by now I'd imagine. Yeah um awesome. Well um well I guess two final questions um what what's something what's like another topic in machine learning or something else that you think is underrated or something that you'd look into if you didn't have this job um yeah so uh I think we alluded to this earlier but I'm really excited by um AI applications for physics directly for physics simulations uh. It's one of the things that is um like a lot of the papers published for PCB design and for chip design with AI are actually driven around exactly that of like simulating you know electromagnetics concerns or thermal concerns and then once you have a model you can do it you know thousand

10,000 times faster, and then the generation of candidates that can fulfill that, uh, increases. And so, you know, I'd love to see, uh, a large language model-esque approach to all of physics, right? Because that's a reusable language, it's a reusable thing that's going to apply almost everywhere, uh, and so I'd love for there to be like an open AI of physics, totally. And you do see lots of different people working on different aspects of that, but I guess not all of the laws of physics at the same time, typically. Yeah, yeah, I agree, uh, super cool. Um, and then, you know, final question, which you kind of got into a little bit, but um, you'd be really somewhat to toine on this, I think, like, you know, what has been the biggest challenge of kind of taking this kernel of a machine learning algorithm and making it work in the real world? I mean, you mentioned, um, you know, sort of explaining to customers when um, when things are failing and sort of putting a product around it that that

you know prevents people from shooting themselves in the foot. Has that been the biggest thing, or is there like something else that you spent your time on? Yeah, no, I mean, that's definitely a big one, right? I think that one of the convenient things about our problem and reinforcement learning in general is that we can actually have a pretty clear division between the thing that creates candidates, right? And that thing is allowed to be faulty. It's allowed to not have perfect precision. But then you have a whole bunch of kind of classic code that checks every single rule that it can be met. And so we definitely focus a lot on that to make sure that hallucinations, for example, aren't a problem and never would be. You know, I think all of our other challenges around this are probably similar to everybody else doing research in AI in general, right? Like, you know, generalizing across boards, proving very, very difficult, for example, right? Boards

are everything from a tiny little thing with two components to these, you know, 10,000 component monsters. And so, how do you get an observation and a stock that, like, can actually vary across that and have a wide range of different actions available to it across that? Um, that's something we're spending a lot of time on and then definitely haven't fully cracked yet. Interesting. That's super cool. Do you have any, um, favorite applications of circuit boards built on your platform? Favorite applications? Um, yeah, so this is, uh, one of the boards that we designed that I was particularly excited about. Um, so this is a computer vision board, um, that's got an STM32 processor on it, a camera, uh, USB connection, uh, and then it runs right to a screen. And so, this is a particularly fun thing to work on because we actually got to build something that can run, uh, onboard algorithms on it. Um, and it's quite an old board, probably two or three years old, but, um, it's actually fun to see, to, like, bring it up, play with it, and

then um you know actually be able to do some fun things like put a mustache on yourself because as an O to the Silicon Valley show love it super cool and I hear you're hiring in in different roles is that right yeah definitely um always hiring uh you know uh our our careers page you know at co. a uh please come visit if you're interested uh we're looking for folks who are obviously experts in machine learning reinforc learning always welcome uh but also in like C++ uh you know running Kuda kernels for for some of these uh geometry algorithms and physics simulations um and so on and so forth there's anything particular that you look for in candidates that might be specific to you and and different from other machine learning companies yeah I mean we have a pretty strong focus on reinforcement learning in particular um you know so uh that's kind of an interesting thing where we look we actually look for a lot of candidates who are coming from like robotics uh where a lot of this work is

is being done um and especially where um you know sampling the environment is expensive um because that kind of is the case for us is that the environment is not you know super fast and trivial to sample um and so robotics deals with some of the similar issues and and that's why we look in that field in particular um we've had some folks also join from uh from like self-driving car companies uh because a lot of kind of path planning routing and so on and so forth smells a little bit similar to to this problem that makes well nice thank you very much this is a really interesting um interview all right thank you Lucas appreciate it thanks so much for listening to this episode of gr and descent please stay tuned for future episodes.

AI in electronics: Quilter’s journey in PCB design

Weights & Biases helps AI developers build better models faster. Quickly track experiments, version and iterate on datasets, evaluate model performance, reproduce models, and manage your ML workflows end-to-end.

Please subscribe to our channel and turn on notifications for future videos!


Weights & Biases

Automating PCB Design with Reinforcement Learning

PCBs, or printed circuit boards, are crucial for connecting components in electronics, but their design is complex due to factors like signal degradation and electromagnetic interference.

Quilter AI, founded by Sergey Neno, is using reinforcement learning to automate PCB design, a process that is traditionally manual and time-consuming.

The company's approach aims to address common design issues, such as crosstalk between wires, and to improve the overall efficiency and effectiveness of PCB production.

The process of PCB design involves creating a circuit schematic, laying out the design, and then manufacturing the board.

The speaker's company aims to automate the layout process using reinforcement learning, treating PCB design as a game to optimize component placement and routing.

The ultimate goal is to reduce the design time from weeks to hours, allowing engineers to focus on other important tasks and eventually design better boards than humans.

Automating PCB design has been a complex problem for over 60 years, with historical tools proving ineffective.

The difficulty lies in the combinatorial nature of the task and the need to account for physical constraints, making it challenging for machines to outperform humans.

The founder's company is working to solve this problem by offering an open beta tool for users to submit designs, leading to unexpected and interesting use cases.

The Impact of a Compiler on Electronics Design

The speaker is exploring how to optimize board sizes for electronics by using a compiler, which is changing their approach to layout design.

The compiler currently handles boards with 1,000 to 2,000 pins, but needs to improve to tackle more complex boards like motherboards.

Layering is used to manage wire connections, with vias allowing jumps between layers, but adding layers increases manufacturing cost and complexity.

Creating circuit boards at home can be challenging, especially with a one-layer design, as wire intersections are difficult to manage.

The use of vias to jump between layers helps to de-intersect wires, while manufacturing tolerances and real dimensions pose additional constraints.

Power supply considerations and current-driven heating are key factors in PCB design, with thicker wires needed for higher current.

Improving Model Adaptation and Testing Efficiency

The speed of light is controlled by the differential impedance of wires running close together, with a new design set to launch soon.

Reinforcement learning is used for model adaptation, with higher-level planning and classic code handling the details.

The transition from Python to C++ for environment implementation has significantly sped up training and production cycles, while also improving regression testing and user experience.

Building a Team for Reinforcement Learning Research

The team consists of about 20 people, primarily focused on engineering, with a mix of roles in the compiler, application, and product teams.

The approach to running the team involves a balance of research and engineering, with a transition from pure exploration to a more focused development mode.

The founder's background in reinforcement learning started with online courses and practical applications, leading to the establishment of the company.

AI's Role in Physics Simulations and Real-World Machine Learning Challenges

AI applications for physics simulations are gaining traction, with the potential to revolutionize processes like PCB and chip design.

The speaker emphasizes the importance of a clear division between candidate creation and rule-checking to mitigate errors in machine learning algorithms.

They highlight the unique challenges of generalizing across diverse board designs and the value of candidates with a background in robotics and self-driving cars for reinforcement learning roles.