Good day. My name is Brendan. Thanks for the introduction, and it's great to be here in Sydney at the ya CTO Summit. I work at Netflix. Hands up if you have Netflix. Alright, that's basically the whole room, so thank you for your support. I'm a senior performance architect, and at the Y conference on Friday, I'm giving a talk where I'm going from clouds to roots, and it's very detailed and very technical. But for today, I'm going to talk about working at Netflix. It's a topic that I really enjoy, and I've written a number of blog posts about, and so this is a great opportunity to discuss it here. I'll start with the story. On my third day at Netflix, I had a meeting. This was about a cent us to auntu migration, and we're going to discuss issues with kernel versions. I was warned beforehand that this meeting would include intense debate. Now, I've been around operating system migrations before, and I can understand how this can go. People spend years or decades working on an operating

system, and they become very invested, and they can become very defensive. This is almost similar to asking people to change the major programming language of the company. Let's say you're on Java; let's rewrite everything in Go or Rust. So, I expected this meeting to be quite heated, a quite heated debate. There would be the CentOS engineers battling the Iunu engineers, and there'd be factions, and we would sort it out, and we'd see who would win. But the meeting was not like that at all. Everyone was so nice. At Netflix, when you join, it's common for the new hires to say, Everyone at Netflix is so nice all the time, and it's not like we're living in a movie like The Twilight Zone. There's some sort of secret reason everyone's so nice. People are just nice at Netflix. In the meeting, everyone was professional and respectful. Facts were stated, and we made a decision, and we committed to that decision. I could tell that some of the engineers had a personal preference. People had

worked on Senta, some people who worked on Auntu, but their preference to do what was right for Netflix far out-seeded, exceeded their personal preference. And so, as a team, we became committed to the right decision and working together and using our knowledge together as a team. This was an amazing first experience for me at Netflix. So, that's not how I would expect an operating system migration meeting to go at all, but it struck me afterward that this is how it should always be. We should always be working together respectfully and professionally. So, how did Netflix get it right? I'm going to talk about my experiences working at Netflix. I'm coming up to 5 years. I should say that no one at Netflix asked me to do this, and KN one that Netflix has asked me to write blog posts. I really like the culture at Netflix. Netflix has put a lot of effort into coming up with a good culture and then sharing that publicly. And I'm an engineer, I'm not a hiring manager, so I've been able to

share what it's like to be the front line as an individual contributor and how that works. There is a document; it used to be a slide deck for the Netflix culture. It's had 18 million views as a slide deck. That's now turned into a memo where we were able to describe it in more detail. But I normally give talks on technology, and when I encounter a technology that's exciting, I want to go and give a talk about it and share it. And it's the same I feel with the Netflix culture. It's exciting, and I want to share and talk about it in the posts I've written in the past. However, one of the comments I had was someone saying, So it's basically sunshine and rainbows all the time at Netflix. And I understand where this is coming from, talking about how well it is. It can sound a little bit like it's corporate marketing. And I have had firsthand experience in previous companies where the company has asked people, Please write posts and tell people how awesome it is to work at the company, even if it's not.

awesome to work at the company because they're trying to do it to help recruiting. So, that's not why I'm doing this. I'm doing this because it really is worth sharing. And to talk more about Netflix being awesome, I encountered this in the job interview. I spoke to different staff for hours, and it just sounded too good to be true. So, what do you do when you're faced with this? You try and find a catch. It's like, Well, what's the negative? There must be some bad thing. And so, I came up with even more bizarre and ridiculous questions and scenarios for the Netflix staff of, Maybe if this happened, I would discover the catch, and I discover the bad thing about working at Netflix. They were polite, and they answered all of these questions. They didn't seem to be a catch. I went through many things. Can I do my own personal blog posts? Yes. Can I go and speak at conferences? Yes. Can I maybe continue to write books? Yes. Could all these different scenarios? At one point, I was basically

defeated, and I was with one of the leaders from talent, and I said, Look, I'm trying to find the catch. There has to be a catch somewhere. And she said, Why? We believe at Netflix you can have everything. It doesn't need to be a catch. And this really challenged what I believed about work for so long. I believed that you can have a really awesome job, but it would probably be low pay, or you could have a really high-paid job, but it would be long hours. So, there was always a but. There was always some compromise or balance. So, it's very unusual to have to experience this where they're saying there's no compromise. If there was a compromise, we'd fix it. Four years on, I can understand that totally now, being immersed in Netflix. If there was something bad about Netflix, we would fix it. Why would we leave it as something bad? There doesn't need to be anything bad. The worst thing for me at Netflix is where we run out of Coke Zero on the F2 fridge, so I either have to walk

downstairs to the F1 fridge or upstairs to the F3 fridge. I don't know why there's so many Coke Zero drinkers on building F level two. This is where I work. We have the performance and operating systems engineering where I work. We also have engineering tools security, but for some reason, this is the platform Engineers. The platform Engineers really like Coke Zero. Perhaps the real worst thing for me at Netflix is we have a culture where you're free to work on anything. There's no lines in the sand about Brendan. You're just doing operating systems and kernels and the JVM. You should just do that. You can't go over and work on Cassandra or talk to API. One of the principles of Netflix is highly aligned and loosely coupled. And so we're aligned for the mission of Netflix, but we're freed to do what's right. We're loosely coupled. So each year when I look at what I want to work on at Netflix, I can easily identify 10 different projects and then the sad

thing is I get to the end of the year, and I've only done three of them. And so, perhaps that is actually the worst thing at Netflix. I sometimes overcommit. I tell my manager I'm going to do all these things, and at the end of the year, I haven't done all those things. I've only done some of those things because I'm only one person. When I talk about all the amazing things you can work on at Netflix, that doesn't mean we work long hours. I took this picture from my office area at 5:30 p.m. on a Wednesday, and I count, if you look very carefully, there are six people in the picture, and that's 5:30 p.m. The guy who's waving on the right is Jason Kosh. He's from Melbourne, another Rosie who's working at Netflix. I can't say every team looks like this, but in my experience, working at Netflix is not about punching a clock and being at your desk for set hours. It's about getting

your deliverables done, and teams are flexible about how you do that. It's also about preserving a healthy work-life balance. The culture is great, in part because it's shared online, and people understand it when they walk in the door. We tell candidates to read the culture when they're starting off in the job interview pipeline, and so it's one of the very early things we tell them: Please read the culture. And it used to be a slide deck; now it's the memo. It might seem a little bit odd to tell people, You have to read this and understand it, but it works really great. It helps people self-select. The Netflix culture is great, but it's not great for everybody. And so, some people may read it and think, This is not where I want to work, and then they don't waste time continuing in that pipeline or joining Netflix and discovering that, yes, the culture deck is true; that really is how it's going to be. So, that's a positive. Another positive of publishing it online is that people know how to

behave the minute they walk in the door because you've studied it and you've read it. So, I've seen the companies where there's cultures from team to team and so, and it depends on the manager of the team and the leadership chain, and you never know until you've joined that team and you've spent a few months there or you know someone on that team. What's it like? What's the microculture like for that team? Netflix is not like that. So, all the teams have the same culture because we document it, we share it publicly. Another thing that I like about it is that with other companies, you do learn the culture by trial and error, which means error. You make those mistakes, and people tell you, We don't do that at this company. We don't do that. Whereas Netflix, you're not likely to make those mistakes because you've read the culture before you start. I had an experience with this very recently. We had a new hire. He started. It was very in the first few

days of work. I was describing a performance area with the Intel CPUs and the way it was doing registers, and this is an area I have expertise in. He said, I disagree with you. I don't think that's how it works. It's like, Wow, you've really read the culture deck, because sometimes people can be, This is my area of expertise, and to challenge my thinking. Some people may hesitate, but at Netflix, part of the culture is you should be prepared to disagree respectfully if you think something is wrong. So being called down on it is good. I'm human. I can make mistakes, and so I could say something that's wrong, and so it's helpful to have people be able to call that out. At the top of the culture, at the top of the culture memo, now I'll summarize some of the culture memo, but also want to talk about my own experiences. At the top of the culture memo, it talks about how we share information openly and deliberately. We encourage independent decision-making and so on. The culture deck itself is an example of

sharing information openly and deliberately. And so, that's obviously open. If you work at Netflix, you are getting memos all the time about what's happening in different areas and business decisions that are being made. And I get lots in my inbox so I can learn a surprising amount about the company within reason. There's going to be some sort of, I imagine there will be some sort of financial news that they cannot share exactly with everyone, but they go as far as they can to share information with employees so that you're aligned with the mission of Netflix. You know, it's not like I've worked at other companies where you discover projects by reading it in the news. Have you ever had this where you look at the news, everyone's talking about this new project launch. It's like, I work at that company. I didn't know we were doing this. I seem to be the last person that hears about these things, and they're very restrictive about information within the company. That doesn't happen in Netflix.

you know what's going on. You know what other teams are working on. As an experience of Netflix is an experience of independent decision-making. A few years ago, there was a leap second bug in Linux, and what that could do is either panic the kernel or it could freeze the kernel, which in some ways is worse because once you freeze the instance, it's stuck until something terminates it or reboots it. So leap seconds are a thing just like leap years, and a few teams asked because it's my area of expertise as the Linux kernel. They asked me, Are we affected by the leap second bug? Might Netflix be facing a single second where our entire fleet of over 150,000 instances freeze at the same time? That's a little bit scary, actually, because we do regular chaos testing where we kill instances, we shut down availability zones, but I'm not sure we've tested freezing every single instance Netflix has at the exact same second. That would be quite scary. Since it was my area of expertise, I

decided I'll handle it, so I looked at the kernel source code, which is harder than it sounds because it's a patched version from Canonical, and I need to make sure I've got that code. I ran traces on the kernel functions to see if we were running unpatched or patched, and I can put my finger on kernel functions and do dynamic tracing, and also ran some tests, obviously. So, I took some machines and I fast-forwarded it to that time to see if that panic. So, after my research, I concluded that we were already patched. We're past the patch needed to prevent these really bad panics. So, I told people that Netflix is fine. That was it. There was no more work, so everyone just continued on. And then the leap second came late one night, and I remember thinking, Wow, I've been empowered to make this decision independently, which could bring down Netflix. Now, at other companies, I can imagine maybe there might be some posturing by management and teams about this really important

decision for Netflix. So, we need to have maybe a steering committee or a project that will research it. And even though you have a Linux kernel expert on staff, you still want to maybe question their judgment. And so, Netflix, no one's going to question my judgment if I'm an expert in an area and you make a decision. That's it, you're owed to make that decision. So, the second happened in the middle of the night, and I'm wondering, Well, I hope I got it right because if I got it wrong, Netflix is down. So, it was fine. I got it right in my area of expertise. We had no problems. But it was one of those moments where I realized how no one gets in the way of your expertise. Netflix likes to hire experts in different areas and then remove all barriers between you and doing and applying that expertise. The Netflix philosophy. So, this is often you'll hear these phrases used: freedom and responsibility, context not control, highly aligned and loosely coupled. Freedom, responsibility. This is about you are

free to do what you think is right, but you have to take responsibility for it. So, if I'm running a microservice and I say, I think we're going to run it in Go instead of Java, well, now Go is not the paved road language at Netflix. So, now you might be responsible for debugging it when there are memory leaks or performance issues. Other teams will try and help you out, but you need to think about that. If I'm going to launch a new microservice and put it in production, I'm free to do that. I don't have to get authorizations to get instances or anything. I can just do it. But then you're responsible for how is it, how do you manage it on call, and how does that look like as another experience. When I first started at Netflix, we had no good way to profile Java. So, CPU flame graphs are a thing now. Is there anyone using CPU flame graphs? So, a few people. I invented them a long time ago. They're really awesome. I think they're really awesome, but when I

started Netflix. We couldn't really do them for the whole stack user and kernel because Java didn't work with system profilers. And I looked at the source code and thought, well, the problem is that Java doesn't honor the frame pointer, and someone needs to go and fix that. I thought I could actually maybe, I can fix that. J. There's open JDK. I can go and look at the source code. I can figure out how the hotspot C2 compiler works, and then I can change the assembly for the function prologs to do the breakpoint register. Maybe can fix it. I have no experience with JVM internals or hotspot, but I know how to code, so I will give it a shot. So, I told my manager, it's like, I want to try this out, and I want to put in production if I can fix it. I'll put it in production, and then we can do these great profiles, CP flame graphs for Java. So, freedom and responsibility. My manager says, sure, go for it, even though I'm committing to

doing something that is not my area of expertise. Netflix also is part of our values. We like taking intelligent risks, and so there was a risk that I could spend a few weeks and say, I just can't fix it; the hotspot C2 compile is too complicated. So, I spent a few weeks, I got it to work, and I had my patch open JDK. And then we deployed it in production, and it was great. I was able to profile things to CF graphs for the first time, but I was also responsible for now I'm running my own runtime in production that I've patched. But it turned out giving me the freedom to go and explore that, to like go and patch some runtime I've never coded in before, then go and run my patched version in production, has really paid off. CF graphs find lots of performance wins for Netflix, and nowadays it's easy because I convinced Oracle to make it an official feature flag for the JVM. So, that's now minus XX plus preserve frame pointer. Apart from those Netflix

philosophies. There's on Netflix values judgment communication curiosity. In the original slide deck, there was a picture of an Enron Lobby which had similar values listed, and it made the point that at Netflix, these aren't nice-sounding values we print on a wall. These are things we live by. And you might wonder, how do you convince staff to live by certain values? At Netflix, we have 360-degree reviews where once a year, we provide feedback on each other. It's not Anonymous, so people know your name when you're sending them feedback, and it's part of being a great team, a supportive team. I play on the Netflix cricket team as well. We have Cricket in the US, like some of us do, and it's similar in that if I'm bowling and I'm stepping over the line to make a delivery, people give me feedback straight away and say, Burn this a no ball. You've done it again. You've done it like 20 times in a row. And so, it's similar at Netflix. You want to give each

other feedback on how they're performing so that we can improve each other as a team. But as a way to think of what to tell your co-workers, the Netflix values is something that's that we're reminded of at this time. You can go through the Netflix values and then identify how your co-workers have supported them or not supported them and then give them some constructive feedback. And so, this is great. It actually works really well because with that time of year comes along and I have to think, Okay, so how do the values apply for my co-workers and people around me? And so, you have to think about those values and you have to think about work situations where they apply. And so, that helps remind you of the values and why they're important. And people will send you feedback as well, touching on those values. So, I think that's a strategy Netflix uses that helps reinforce this. We don't forget about the values. We have to think about them because they're part of giving each other feedback as another experience early on at Netflix.

I was this is an experience about values, curiosity, and courage. I wanted to try a tracer on a production system. This tracer had a reputation of possibly panicking the colonel, but I thought it was really important to try this risk, and so I run the trace. I SSH onto an instance. I run it in production, one of the major instances at Netflix, and then my cursor stops after I hit enter, and it's like tap tap tap tap tap. It's like hopefully the Wi-Fi is down, hopefully I didn't just panic a production server at Netflix. I had panicked a production server at Netflix. So I go into the major core chat room and confess. I say I'm sorry, I've panicked production. I run this command, I shouldn't have, and now it's gone down. Everyone was fine with it, everyone was supportive, and the Netflix architecture is designed to be resilient, so that didn't cause a major issue. My colleague Dave Hun gave me a button, killed production, achievement unlocked for my experience, but you can see how this is

about I was curious about this tool. I had the courage to try it. I communicated to the team that IID made a mistake, and I had the integrity to own up to my own mistake. So, you get good at identifying the values and applying them to work situations. So, I mentioned the Netflix Cloud architecture is resilient to this kind of failure. Chaos engineering helps because we do regularly test. That's a diagram of our microservices. It gets pretty complicated. And also, the architecture and the culture are complimentary because there's no approval process for deploying a new application. It gives you the freedom to try new things out. I have a diagram here which I'll be using at the Y conference about how specifically the architecture is resilient to failure. So, we have elastic load balancers. We have autoscaling groups so that when load increases, they can scale themselves out of an issue. If that happens in the middle of the night, an engineer doesn't necessarily need to be paged. We have availability zones for that type of failure. We have hiss and

ribbon, and we're also using grpc. These all provide fault tolerance as well, and timeouts and backups, and also chaos engineering regularly does tests, so you're in a very safe environment to try new things, and you know it works really well. And I think this is part of how the Netflix culture is able to work so well, as we have this environment. I want to mention a personal favorite from the culture deck to finish with. This is no brilliant jerks. It's one of the many things Netflix has. I really like this a lot. It's very similar to the no rule by Robert Sutton, a Stanford professor. And at Netflix, there aren't no brilliant jerks. Brilliant jerks hurt productivity. I've written a blog post about it where some people don't understand it, and some people do understand it. And I think it's helpful to split brilliant jerks into whether they're acting selflessly or if they're acting selfishly. If they're acting selflessly, then you some people will try and excuse it and say, Well, they a joke.

to everyone, but they're doing it for the company. They're providing so much company value, but then if you have a selfish jerk who's hurting people, bullying, humiliating, manipulating people, exploiting others, and they're doing it just for themselves, that it's just pain. It's not good for the company. It silences technical opinions, IT demoralizes staff, and reduces company efficiency. So, it's really awesome at Netflix. Stafford professional and respectful and supportive. The DI, the picture I've got there, is actually this resonates with other companies as well. That's from the Ericson headquarters at Stockholm, Sweden. Someone has printed out a picture of our CEO Reed Hastings with the do not tolerate brilliant jerks quote. So, that was great to see. Lastly, the Netflix culture is not for everyone. The memo is clear about that. And Cloud engineering is also in Los Gatos. Only that also doesn't work for everyone. Takeaways. So, I think the Netflix culture does work really well, and maybe that can improve other companies as well for

example no brilliant jerks. That should be an easy one. And if you're having that debate, please look at my blog post where I split it into selfish jerks and selfless jerks. That might help you understand the dangers, freedom, and responsibility. It helps if you're on the cloud architecture similar to ours. It's fault-tolerant to support that freedom. Openly disagree. The cloud, microservices, and redundancy, as I said, supports the culture and so on. So, some key takeaways I think could be learned. And thank you. That's my T. Thanks for.

Working at Netflix • Brendan Gregg • YOW! 2018

GOTO gathers the brightest minds in the software community to help developers tackle projects today, plan for tomorrow and create a better future.

Learn from thought leaders and innovators with top-rated videos released daily, and at our year-round conferences, masterclasses and meetups.

Get started: https://gotopia.tech

GOTO is produced by Trifork, a leading full-service supplier of high-quality, custom-built applications and end-to-end solutions.

GOTO YouTube Channel Code of Conduct: https://blog.gotocon.com/yt-coc


Working at Netflix • Brendan Gregg • YOW! 2018

Scripts