Day 366: Managing Long-Term Technical Debt in Infrastructure

Poto emphasizes the importance of addressing long-term infrastructure challenges beyond initial setup and maintaining a proactive culture.

### Navigating Day 366: Long-term Strategies for Infrastructure and Operations
* 🌟 Welcome to the HashiCorp Ambassador Thought Leader Series, where experts share insights on **infrastructure, networking, applications, and security** for the community's benefit.
* 🛡 Community guidelines emphasize **respectful participation**, and recordings will be available on **YouTube and HashiCast** for future reference.
* 🗣 P, a HashiCorp Ambassador since 2020, discussed the concept of **"Day 366,"** focusing on what happens after initial setup and operationalization, especially regarding the **longevity of systems** and the **importance of planning for future challenges**.
* 🔑 Key points included the necessity of **evolving systems beyond their initial setup**, the risks of **ignoring technical debt**, and the importance of **follow-up actions to maintain system integrity**.
* 🐘 The discussion also highlighted the "**elephant in the room**," which refers to issues that arise when teams fail to address **long-term implications** of their decisions, such as **certificate expirations** and **outdated practices**.
* 📅 P provided concrete examples from personal experience, illustrating the need for a **proactive approach** to avoid pitfalls that come with **neglecting ongoing maintenance and updates**.
* 📜 The importance of creating a **culture of accountability** was emphasized, where team members are encouraged to **document changes and follow up on tasks** to prevent future complications.
* 💡 Ultimately, the session underscored that developers are both the **problem and the solution**, advocating for a **collaborative approach** to address challenges and improve infrastructure management.

All right, welcome everyone to the HashiCorp Ambassador Thought Leader Series, where we have our most esteemed community members and experts talk to us about the big ideas in our knowledge space. This means everything: infrastructure, networking, applications, and security, and the techniques, principles, and lessons learned that we hope y'all, the audience, can make use of. Before we get too deep into it, obviously, there is another human being here on the stream with me, but beforehand, I do just want to remind everyone of our community guidelines. They are pretty simple: just don't be a troll. If that is confusing to you, check out the link there. It has very fine green details on how to not do that. And another thing is, while we are doing this live, we will have this recording up on YouTube, and we'll also have it in podcast form on our HashiCast, so be forward to look forward to that. Now, obviously, I have someone to the right of me, yeah, on the stream, and would you introduce yourself? Yeah, uh, hey Cole, hey everybody.

uh, my name is P. Um, um. Should I also say what I do, what I work, and all that? Yeah. Oh, absolutely. Yeah, totally. So, uh, I'm based out of New York City. I've been um, HashCor Ambassador since the beginning of the program, which was 2020, I believe. Um, work for a company that I'm not going to bore you with all that because it's not a sales pitch or anything. And it's even if it was a sales pitch, we're like business to enterprise, so probably never heard of us. But we do AI, uh, uh, conversation systems for uh, companies that need to have customer support or customer contact. So, you know, when you have to call your bank, your airlines, etc. So, like, we have a couple of services there, you know, from just the chat platform to voice recognition to transcription and a bunch of different things. So, all that is done by the smarter people that like KN know how to do the math and the science and all that. I just, you know, play with servers. Yeah, yeah, you're being very humble because we

know that all that other stuff is just a couple of files on the box that you're maintaining, right? Yeah, yeah, yeah. Um, right. So, uh, that's, that's Poto, right? I've known him for a while, and so I wanted to get him on here for this episode for sure. Um, if you are here, you've seen the title, and it is quite quizzical, uh, to say the least. In fact, when Poto brought it up, he's like, Day 366. I was like, You know that, like, that's not a thing, right? There's only 365. And then he explained it, and then he started talking about elephants, and I was like, Okay, we're going down a pipeline. But it wound up being a very fascinating topic, really cool idea. And in a nutshell, P, what was the idea that you wanted to get on and talk about today? Yeah, so, I mean, there's a lot of, or a couple of different, uh, thoughts there in the title, and, and they'll become clear as, as we start going, but, um, it's so, first of all, there's the

idea of you know we've all heard about day one, day two, or day two plus if you want, and how like you know sometimes things are easy to set up with like following the hell World um, and then like that's how you you know show it running, but then you have when you have to operational it, that operationalize it, that's day two uh, but then what happens when you know down the road there's things that happen you know a year from now, and like one of the examples, and we can talk about it later, is you know when certificates expire after 365 days. So what happens the next day? What happens when like something's kind of static but then you have to make a change, you know like how does that how does that happen down the road? Yeah um uh, there's also the thought of like when you build something that again like works for the first you know, you first phase of the project, and then you start building stuff around it like how do you evolve the the outside layers as well as

like the core of the thing that you started working with, right, um, and um, kind of also the ideas of like, you know, we all have to make trade-offs and decisions when you're building stuff, right. So like, how do you deal with the decision like at the moment but also later on, and how do you like, you know, manage that risk between, you know, end quotes, but those are the main things, yeah. I mean, it's to me, what's neat about this is we have, obviously, we have a ton of great content, and I see Jasmine, Jasmine White's jumped in, so hello to our community manager. The what's neat about this to me is a lot of this content that you'll see out there is about that beginning process, about day one, week one, year one, right, and then we forget, oh hey, what comes next, how do you maintain that next, and all the stuff that PO talked about is thinking about that from that long-term perspective, so really, really, uh, really neat, really looking forward to get into.

that and in terms of importance PTO, you sort of relayed sort of three concrete examples, SL stories to me of why this was something that came from personal experience. And like, could you go through those? Yeah, um, you mean you want to like talk about specific examples? Oh yeah, just like I remember we, this all began with the story about the certificates, right? Maybe related again. Then we also had the credentials thing with the database. Yeah, yeah. So, I mean, with the certificate one, it's uh, this happened some time ago. Um, we had set up, you know, a cous cluster with some automation. We had like a good set of uh, cous controllers. We had set up things with um, with terraform. We had Vault running and everything, you know, was running fine. And there was one certificate that was used, I think, for for kuber web hook or something that um, suddenly started to fail, you know, sometime after we set up, sometime after things had been running perfectly. And obviously, it happens when like somebody really needs it. And it's like, but it was working yesterday. So like you

can't real because you didn't commit a thing. You didn't run a pipeline, so uh, you know, after some investigation, it became clear that it was a certificate that expired 365 days. So this was dat 366, that's part of the title of the um of the the talk um, and it was because we kind of took a shortcut and just created created a certificate and just put it there as a secret but didn't take the time to either use C manager on Kubernetes or use Vault PKI or whatever like any of that automation uh, we just kind of set it up, put it you know, release it into the wild and didn't consider that the long-term um consequence. So we we thought about day one, stand it up, it works, thought about day two, these are the pipelines, this is the repo, this is the giops operations, didn't think about down the road, yeah like day 366, and uh just in case that you all haven't caught on, we don't mean that this literally all gets built in.

three days. It's just sort of a bucket that we sort of a category we use it to say like where is it. Is it the initial phase or you're standing it up you know when you're getting it set up or like afterwards when you're maintaining and poto of course is talking about what comes after that. Um so we can we can talk about some of those more of those stories but just for time. Uh you had three main Big Ideas and principles in this thought idea and what happens after a year. How do you maintain that and one of the very first ones had to do with elephants right. Actually never heard this so why don't you break us into this first first principle an idea. Yeah so I mean the first thing is uh I'm sure people have heard of it. I remember um hearing about it when I was I don't know five six years old and I I actually forget when uh funny because this has to do with memory but the idea that that elephants when they're babies

They are tied to, you know, in circuses. We know that circuses nowadays don't have animals anymore, and that's a good thing. But back in the day, the baby animal, baby elephants in the circus were tied to a stake that was in the ground, was maybe about as big as the actual elephant, and tied to a chain, right? So, but because the elephant, the baby elephant, was kind of very weak and like was trying to pull and couldn't pull, so it realized that it couldn't get away. So, as the elephant grew, it tried to, it didn't try to get away from the stake, even though the stake was the same size it always was, and the chain was like quite weak for a big elephant. But it goes to reflect the memory of the elephant that we, you know, reference often, that you think you keep a fixed mindset. I'm sure people have heard about this. This is in psychology, is a kind of a more recent sort of school of thought of fixed

mindset versus growth mindset. So, why does that have to do with software, right? And that's the part that people are here for. So, again, coming from experience, both your personal experience and in the organizations that I've been with, how we tend to not worry about what's the path forward, but as long as things are working, to not touch it. And like the idea of if it's not broken, then I'm trying to fix it. Like, that's actually not a good thing in software. Like, it's maybe a good thing in other fields, you know, to different degrees, but in software, it's like you can't expect that things are static all the time. If you think about it, even if you turn on some old computer that you had, you know, that's in the attic some time ago, even if you turn it on right now, like, it's not going to work the same as it did, you know, a few years ago. So, um, yeah, avoiding that.

that kind of um um don't try to fix it if it's not broken mentality. Mhm. And uh, at times, this gets reflected in you know, it's a good thing that for example in GCP and EKS, sorry on AWS, the um upgrades of Kubernetes for example, they are forced, right? They're forcing you to to get get. But do you actually, so like, right? So so how do you, so so quick, so quick kind of question like when those updates are forged, how do you how do you play CU? Sure, you don't like that every time, right? Well, I mean no, but yeah, the fact that it's that's imposed on us, it's um, there is a risk. And now also GCP, I think Azure too, any us, they offer extended support and you know, you have to pay a little extra, but it's not forever, right? Right, it reaches a point where you either cannot operate um on the cluster or cannot updated or it's it's updated automatically. And with Kubernetes, you like the U deprecations and removals are are a thing, then you

might you know find yourself in a tough spot. So, um, one of the things that happen often is that, well, so stepping back a little bit for other things like, you know, just if we talk about the HCH core stack, you have Vault, Console, Nomad, whatever running, or even just the Terraform binary. M, um, you can keep working with those for a long time, and they're not going to be like the updates are not going to be enforced. M, actually, I know that um, HCP boundary is enforced like that upgrade happens automatically. Yeah, but other than that, um, then at some point you find yourself trying to mitigate the risk of upgrading by adding layers around it. Then those layers are things that are kind of like not necess standard. They tend to be sort of, you know, like glued together. Yeah, and uh, then go ahead. No, I was going to say then a later point in time maybe there's functionality for those same things out of the box that make your middleware.

layer redundant, but you never take the time to go and, uh, like look at the change dog, or, you know, check for updates, or even just security updates, or things that, or even just for sanity, say like we are running the latest Vault, for example, right? Yeah, that's a gets overlooked. So, in other words, it's like, hey, like you've made some sort of change that maybe at the point in time was pretty good. Now we fast forward a year later, and it's either not optimal, or it's got some sort of major issue. And so, there's a leak in the ceiling now, and you're like, well, the temptation just throw a bucket down and catch the water, right? And then you have that bucket, you're like, well, now I got to think about getting a better bucket, and we got to think about bucket Ops, and all these other things. Yeah, yeah. So, I mean, the way that, um, we'll talk about like that, that kind of ties into something we want to talk, talk later on maybe, but um, the way in general that I've seen

this kind of addressed in organizations. This is, and by the way, most of the things that we're going to talk about today are not easily fixed with technology or with automation because a lot of this is like just the human aspect of things. Yeah, um, but, and like we are, like humans are flawed. We are flawed. Like, it's like we, we need to be aware of that, uh, of those flaws to, to be able to, to address them, right? But, um, so organizations sometimes have, uh, policies about like updates in general between you, just platforms like, um, Kubernetes, Nomad, etc., just language libraries, dependencies, binaries, like Terraform, for example. So, establish a give story around everything. You can't depend on one guy or woman, um, being in charge of the, uh, updates for their kind of like piece of the ecosystem, right? You have to have a, like, a concrete story that makes sense organizationally. Why? Yeah, that makes sense. And so, that, that was kind of that because I think we were talking about sort of these main three points here, right? So, the practical side of this is

you can't just rely on one person, right? You need to have this something that is and almost bleeds into our second big idea and principle that you wanted to talk about. But in terms of practical advice, just sort of wrap this one up, unless of course you had more on it. What so here's the no matter where you are, you probably have a fixed thing somewhere that could be improved, has issues, and maybe you've become blind to it, right? And poto your advice. If we're going to change it, obviously we can't necessarily do that through automation, but what should people, what steps can people take, right? I mean, there are some tools for uh. I don't think I've found a tool that just this like across the board like because it's very different uh detecting an upgrade in Kubernetes control plan versus a like go library, right? So, these are kind of different realms. There are tools like Renovate, for example. There's a like a kind of like a Cron job thingy.

that runs that you can run on Prem, and then we detect, we scan all your GitHub repost, and give you updates in something, generate automatic PRS, and even automatic merges for things. There's a sneak or SNY YK or GitHub, what's it called, GitHub Advanced security, something like that to detect security patches. So, there are some things that you can do there. Yeah, um, it, I don't think there's a tool that scans your, you know, Nomad console, Vault things, or like any of the other controllers that you made run Kubernetes. It's hard to find something that, so it's a matter of understanding what are like the big buckets of your dependencies and like address those, uh, as technically as you can. Yep, and it seems like, and it's got to be a people thing, right? Has to, you got to get the whole company on board with it, because, or AI in the future maybe, but robots, yes, well, yeah, and so that sort of plays into the next big idea in principle, right? So, we just

came from Yo. You've got a fixed mindset. It is probably a people issue ultimately. Obviously, we've got to fix technology. You're going to need a company-wide buying from it. There are SE different sets of tools that can aid you, but there's not necessarily one that you've found so far that can do everything. But on that culture piece, that sort of breaks us into that next point, which was what poto. Yeah, so the uh, and I call it a follow-up culture. It's uh, I don't know if that's a name that's gonna catch on, but um, just the idea. We'll make it happen. Well, we'll repost this. We'll make this a TikTok, and it's gonna go viral with j or whatever um uh, so just the idea of following up on things that you know need to be fixed. So, we've all been there. We need to make a manual change directly on the kuber cluster. We need to like update something live because you know it's an Inc um an urgent

incident, and we need to get it resolved now. We know that the right, the proper solution is to put it somewhere in code, but we don't have time to do that. That's okay, but what I'm saying is that there are follow-ups to patch the shortcuts that we take. Sometimes it's not an urgent shortcut because we need to do something right now; sometimes it's a, Hey, maybe, uh, I don't know, I'm taking this, I'm using this deprecated API because it's what we need today. We have a couple of months to update it, but then forget, we forget, or the person that took the decision leaves the company, they didn't leave that documentation. Maybe the person that did that created a certificate that expires 365 days later, you know what I mean? Yeah, so just the idea of being able to follow up on the things that we know, it's a little bit of, um, was like the Scout, Boy Scout, Girl Scout mentality of like, you just make sure that everything's clean, and it's better than

than when you found it. Yeah, so you know again a human, a human kind of aspect to the technology side of things, right. And it's what's interesting to me about that is that that kind of mindset, it almost runs to an extent counter towards that like Scrappy move fast and Break Stuff thing, which like ironically is generally like we move fast, forget about that thing, just leave it in the Wake that we're creating and just keep going forward, right. And that becomes the culture of we're not even going to look backwards ever, we're never going to look backwards, and that creates these Road bumps um in the future um, right. Yeah, so what else, what else did you have on this point here? I mean yeah, it's the the other problem with that is that often uh in organizations that are not that, it's one of the downsides of remote work or or like distributed companies that um especially if you have like you know one office, there's you know few people, other office, there's few. So like some of the

conversations that happen kind of organically get lost in that distance. Not saying that distributed work is bad or for is bad, not at all. I'm more of a, sorry, I'm touching my left mic. I'm more of a like in-person, in-office person. I would never force anybody to go there, but I'm saying like one of the things that we lost during the pandemic was like just missing out on those overing conversations. So, this all goes to sometimes a person that took the shortcut or made the decision of doing something, maybe communicated this decision or this action to the person next to them, or just told one person, or only one person was what of this because we were in like the incident war room or whatever. Yeah, but the rest of the team is not aware of what was changed, and it's something that's easy to fix, and very often this is like an online change somewhere, but because the rest of the team is not aware, we don't know that.

Something needs to be followed up on, so we're kind of relying on that person's ability to follow up. And that's not something you don't want to just depend on that. You want to make sure that this is something that's cultural and that's sort of established. And it doesn't mean that you only hire people that are good at following up and are very organized. And like, uh, you know, it means that this is something that's kind of trainable. And S, and it's so easy to do. Yeah, to-do list, jur ticket, whatever, right? So, you know, that's not a complicated thing to do. It's sort of like a um, uh, you know, quite easy to start doing. And you know, we'll have a payoff at some point in the future. Yeah, so, so sort of like to bring that one back around. It generally starts as we're just going to put this in place now. We put it in place and it's working. And generally a small number of people

may or may not know about it, and that has its own issues. And you're saying that like, Oh hey, you know, this maybe wasn't as, didn't seem as prevalent years back because since you are in office, it's a little bit easier to have those conversations. But yeah, remote-wise, it's totally easy to just do it and like not tell anybody. So it becomes another process you have to think about, like you said. And yeah, you do need to, you need to throw it down somewhere in a Jer ticket or some sort of dock. And I think you talked about like kind of like keeping a dock or something like that, but though that may have be in then in the next principle. Yeah, it's um, forget when we talked about this, but this is so um, I think you've heard the idea of um, the broken windows Theory. It has been debunked or you know um, you know disagreed with, but the idea of the broken windows is that the broken windows Theory refers to the fact that when you have neighborhoods in bad shape.

you know, you know, windows are broken, uh, some buildings are abandoned, or cars have, like, you know, flat tires, and they've been there forever. The idea that as long as you keep your windows fixed, then that's gonna kind of, um, you know, like, attract or, what's the word, motivate the rest of the neighbors to keep their house, you know, tidy up, and then maybe, like, clean up the yard or whatever. So, but these are tiny, tiny tasks or something that's, you know, doesn't take a long. It's not like you're building a new school or something. It's like tiny tasks. So, I try to, um, to the extent possible, and, like, I've done it on and off in different throughout the years in different organizations, but keep a document called broken windows issues. And these are things that are, you know, quite simple to do, but because they either, you know, update this variable name or, you know, remove this code or, you know, deprecate this API that nobody uses, but it's, you know, has failing tests, whatever. So, it's tiny things like that.

that um because they are small enough. They usually don't have visibility. Again, it's one of the things that maybe it's not. It wasn't a shortcut that was taken. Maybe it's something that you noticed but only you noticed and didn't communicate it because you didn't think it was important enough because they're not important enough. They tend to be deprioritized when you have like your priority meetings or backlog uh planning or whatever um but once we have disability and this like a team kind of thing um it's easier to get those things resolved and don't because they become like you know like the junk that you have in a drawer or whatever like you all have like that condiment drawer in the kitchen. Yeah, so things, small things like that end up you know piling up and the cleaner your sort of Baseline is, the better because like you don't want to have a bunch of small things in the future when you have like you know your startup is taking on and you do have like other things to do to survive.

don't to have to deal with this other thing. Yeah, yeah. I mean, I've totally seen that too, especially like if you're an early-stage startup. You come in, it's like, Well, everything else is already a mess, so I'll just, you know, why not? Let's... Right, yeah, exactly. Yeah, so yeah, and that also gets kind of transmitted to new people. Like, new people come in, maybe they come from a different culture, different background, and they, you know, it feels like they have to do this the way that someone is not totally that they join a company that they want to learn like how things work, but we're doing things the wrong way. So, you know, you don't want to like keep multiplying that effect. Oh, yeah, yeah. I remember there was one, well, I came into one startup, and we had some function that was both like a problem, but the person had decided to use all single letters for like the variables, and just being like, Yo, what does this do? And he's just very transparent. He's like, I have no idea, man.

he's like, I wrote that so long ago. I don't know what's going on. That um, so yeah, so dealing with that your sort of like proposed approach to just to wrap this point up right what what is what can people do to sort of like to to prevent this whole hey we have something that just sort of has become part of it we're not following up with stuff we just got patches everywhere it looks like duct tape right right so I mean this is one of the the things that out the things that we want to talk about this is one of the things that it's least technical the solution for it is least technical in nature and the one that I always do is or that I will suggest people is just call people out be polite make sure that you know it's within the company's culture um but like ask people hey why did you do this are you sure that this is the right thing to do does this belong in code somewhere let's create a ticket let's take the like one hour of the day to just go make

that one line change. Don't postpone it. It tends to happen more with, uh, in my experience at least, with more junior engineers because, a, the solution they want to, like, show value and that they found a solution, they knew what it was, and they fixed it, and they saved the day, um, for the incident, which is a good thing. Like, we want to have people that's capable of performing under pressure, but, uh, it's harder. Like, this comes from experience that it's not just that patch there. It's like following up with the next thing, and, um, so just maybe calling out is a wrong word, but just remind people like, Hey, there's this other, you know, follow-up steps that we have to take, or ask like, Hey, what was the solution that he added here? Are you sure that, um, we created a ticket for putting it in code? Is this not something that's going to break another test that we GNA have to, like, f up and refactor the test as well? So just

be proactive in that and that being proactive hopefully is going to be transmitted to the rest of the team and like you're G to you're going to kind of start that snowball effect of like everybody's calling everybody out and you want the point when it comes around and and people call you out for the same thing that's when you know that that's getting traction yeah and I've been I've been called out by Junior Engineers as well and like it feels great that's again that's why you talked about it as a culture thing it is it is a people aspect of it and probably the longer you get past day one through two right the more that that's going to play importance but speaking of which this is going to bring us into our final big idea here on this episode right um so let's say that we don't have the follow-up culture in place as much or maybe we do and we've got sort of a tight deadline I'm like hey PTO like

let's just do it like this for now, right? It's good enough. This third point, you had a really fun spin on it. You took a very well-known concept and sort of escalated it. I thought it was very entertaining. So, tell us about it, right? So, we've all heard about tech debt, right? And it's generally defined as when you're consciously taking some shortcut or some easier, like lower-hanging fruit option for fixing a problem that might buy you some time, but you know that the right solution is something else. It's going to take higher effort. And debt in general is not a bad thing. I'm saying like that as long as it's responsible and you pay your Disney vacations with your credit card as long as you're paying on time. That's fine because you don't want to, you know, you want to have that vacation. Same in tech. You want to be able to, you know, maybe defer some

work at the expense of it being higher effort later, but you are moving forward. You're unblocking people, or you're, I don't know, you get to do that demo for that big client because you need it today, right? But then there's a point in time where that Tech debt becomes Tech mortgage, right? And again, mortgages are not bad, but if you're not responsible in the management of, you know, the risk of time, you're assuming that you're going to have time later, but maybe later you're going to have less people because you had to lay people off, or you're going to have a bigger client, or your PR are going to be different. So, um, doing a series of it works now, so we'll fix it later to me, in my experience, again, my company and other companies that I've been with, and this, I don't think this is something bad, but the for now means forever because, and I'm saying this empirically because I don't have a, you know, concrete proof, but every time

people say that I say like, No, I'm going to stop you right there. You're saying this for now unless you show me some script that's gonna delete your code or it's going to deprecate that thing and it's gonna like, It's in the blockchain, it's going to happen this moment. Like, There's no way that you can um uh commit to this for now or whatever, right? So just the um uh yeah the idea that that Tech debt is good or bad. That doesn't mean that you it's not that Tech debt or even Tech mortgage is good or bad. Is that it should be done responsibly but it's you know to an uncontrollable level. That's that's when it becomes problematic. Yeah yeah and so your approaches for these like right. I kind of had like two other points that you want to talk about around like how do you deal with this right? Yeah so um I kind of said it jokingly earlier but one of the one of the ways that that

you can like you can write some automation for for uh you know forcing yourself to do the right thing. One of the examples I was thinking about it uh yesterday a couple days ago is for example you can write in a in HTP tform. That's a new name right HTP terraform no. So in htb ter form you can write uh a sentinel policy or an Opa policy to block any applies after this date if certain change hasn't been made. So that forces you to you you you are not letting you make you're not letting yourself make changes if you haven't fixed this other of thing yeah. So there there's some some automation options that we have there right um. So to the extent possible that's that that's one thing you could do um also again this is harder because it's more of a like organizational human kind of kind of thing but just establishing accountability mechanisms of um you know maybe like what's the name thing I've heard it from

um uh psychologists. It's essentially having like a body system of like, Hey, I'm doing this thing, I'm taking this shortcut, but it's on you to remind me this time later that I'm gonna have to do it. If not, I'm gonna have to pay you $10 or whatever, or $10 goes to the end-of-year party fund or whatever, right? Yeah, yeah um. So, or I'll owe you whatever cover on call for you next time, whatever. So, you know, to the extent that you can do establishing those accountability mechanisms, that's another way to approach it. Yeah, yeah, nice. And then in terms of like kind of like solving the whole thing, you actually already brought up, I think, I think I accidentally brought up your solution sort of a little bit early. No, I, I, I also got ahead of it because it applies to both of these concepts, and a lot of these concepts apply, like, they kind of, you know, mix together because it's like, Yep, we're all humans.

in Tech we're all one big problem, right? Like, it's, it's, it's, it's, there's the, the, yeah, lines are kind of blurred in, like, where we're the problem, where we're the solution. But, um, talking about the, the, um, broken Windows document again, that broken with it is part of, uh, it is an accountability system, and that maybe works in a different way, but it's an accountability mechanism in that you are recognizing this thing that you like, this shortcut that you took, or this thing, that this flaw that you found, or maybe something you find that somebody else did, and you're kind of calling it out there again, going back to the calling, calling out, um, but also the, the other aspect of it is, again, it's okay to have debt, but we have to know what we're trading off, so document those, those trade-offs. But taking a little bit step further, one thing that I can say that I have done, I wish I try to convince people to kind of like, uh, have this practice, but, but

practice of doing like long-term like longitude know whatever postmortem like okay a year from now let's review this trade-off document and let's see what we did right, what we did wrong. Did we have like what-if scenarios that did they turn out correct or not? Would we, if we had, we could go back in time, we would take a different decision. So, sort of like learning from those things, yeah, so a technical reflection to an extent. So, before we wrap up this point, I have one more sort of thought for you. What? So, I was thinking about this when we after we talked about this specific point. It's like why are we so prone to take technical debt? Because obviously there is, well, the shortcut, but I also feel like that the difference here is that I can also just not be responsible for that debt. Like, I can make the decision now and be like, well, I'm not going to be here in six months, so then po to deal with it, and I don't have any debt at all. And I mean, and

that's kind of natural. One of the things that's hard in software, that so you know in software we take a lot of um, a lot of prac is from the manufacturing industry, but U, one of the things that it's hard to do is the um, uh, cost center versus profit center and be able to have checks and balances of hey, you are taking on this debt, but because it's hard to put in numbers and calculate, and also when that debt has to be paid off by somebody else, right? Yeah, I'm uh, running Vault 1.1 because I don't want to, you know, I don't want to upgrade so the other teams have to write like different code or different uh, um, integrations because so the idea of being able to have those checks and balances and like know who took on the debt and who's paying off for it, that's in an ideal world where we have a way to do that and we are closer to the way the manufacturing industry works, um, that would be great. I

don't like. I don't know, like, no joke. One of the things that we've talked about once, and it's you GNA laugh because it has to do with blockchain, but like, if we could have like an SRE coin kind of thing that like people Mine by whatever bribing um s the s team, and then we, you know, give people priority to people that like pay with s coin back, and I don't know, so that way of that way of like having some some checks and balances, right? Yeah, I'm not saying I'm not saying be corrupt in your organization. I'm saying no, definitely not suting that how an S coin would uh would be used. Yeah, but yeah, I mean, just in general, the uh be mindful of, you know, either you're going to have to pay your on Tech debt later, so be honest about, you know, like the work that you plan on taking on, um, or be open about it like, Hey, we take this shortcut, we're gonna be fine, but this other team is gonna is gonna um, you know

pay the price or we are taking this shortcut. We're doing this low-hanging fruit option, but we have to be clear with marketing or sales or whatever that this is not going to be sustainable. So, just like consider the cross-team and even cross-tech and non-tech communication of Italy. So, that wraps up sort of the whole idea of tech that Tech Mor, you know, and that all sort of plays into the original theme, right? This whole theme was the elephant and the infrastructure, is that there's stuff beyond day one and two, and there's a lot of things that happen with that, and a lot of it does have to do with the people, the developer side, right? The type of culture that you set into place, whether or not you're thinking about improving stuff, not just putting it down in one place, and then being smart about the trade-offs that you put into the place, but in general, like moving your in general sort of like

solution seems to be. We have to start with us as the developers. We are both the problem and the solution. What's sort of your general advice to solving this day 366 operations and addressing the lhant in the infrastructure, right? So, um, one of the things that I found that is effective, um, because I've been an IC, I guess, most of my career, so I like being on the technical side of things. I like being Hands-On. Um, you and I have talked about I have my, uh, Raspberry Pi Kubernetes cluster. It's actually behind that shelf, and it's kind of my home lab, and it's like I've run experiments there. Um, one of the things I wanted to mention earlier too is that I have, because I'm a big Big Nerd, I have the, like, sometimes just for fun, I go and read the change log of projects that I, uh, experimented with in the past but didn't quite like, so kind of like look at what's new. Um, and often that's

because I have my cluster setup a little bit ahead of what we have at work, so sometimes I've found issues at home. So like by the time that we upgrade, that's been useful. But yeah, so I was saying that I've been more on the technical IC IND individual contributor side. So for the longest time, I was frustrated because I knew what the right solution was, and I knew that I had the technical skills that maybe managers or people above me in the chain may not have, either the visibility or the knowledge or the skills or whatever, to know how to fix. So I was frustrated that I wasn't able to convince my manager, like go up the chain, but I found that the way to get traction is to actually get one of your peers to have to have an ally to convince one of your PE, especially another more technically oriented IC, so that you get that person's buy-in or support.

then once you have it, doesn't have to be one, but you start with one. Once there's some traction there, you can get the manager, you can go to the project manager or whatever. So like, hey, we are in agreement that this is the right thing to do, so we should do this. We have all these opinions. It's not one person that has one idea and they want to go rogue and do the thing. It's like, we all need this. So that's kind of the main advice, and that this works at any stage in your career. Like junior people that are not maybe don't have the voice or the seat at the table, it's easier to get an ally and a peer. But at the same time, the 15-year career architect that cannot get traction because the management is not technically oriented. So like, get an ally and somebody that understands the solutions that you're trying to

pitch yeah right nice that and the other advice is just drink water. Oh, by the way, I have a hash Corp. Oh, nice, nice. I got this at some hash, which is hard to transport because this is glass. It's hard to transport back to be careful with it. Yeah, all right. So, what else can people check out? Poto, I know you had some stuff you wanted to show as we wrap up here, right? So, I have here. So, like instagram.com papis.io. That's I need to be more active, but I post some tech content there, some like explainers and whatnot. I think my last post this live video when I was at it Summit in New York. Nice. So, I'm local in New York. If you know of any meetups or one at like chat, have coffee, whatever, you can contact me here and be happy to talk. Other two things is U. There's a book called Making Work Visible. Author is Domin de Grandis. I hope I'm pring while you talk. Yeah.

Yeah, so um, this is actually more for like project managers, but there's a lot of um, uh, again, talking about the human aspect of like workflows and work management and prioritization, all that that um, goes to the to like what we talked about earlier in terms of planning, taking shortcuts, uh, understanding that documenting that. There's a lot of like, she talks about a lot of things that, you know, the tiny, like tiny shortcut I had to take and to modify something manually, but that's not captured in like a plan. But it's not just the five minutes that it took to do that, it's the three hours it's going to take you to run tests and do the code, etc. But like, that's not captured, and that takes up half your day that you were supposed to do to use to work on a feature, whatever, right? So, you know, it's uh, I don't, I got this directly from her at a DevOps space once, but I'm pretty sure um, you can find it somewhere. The other thing, just against kind of a

plug so Brian Carlson. He's another um um HashiCorp Ambassador, right? The best-kept secrets of Hash Vault um. I don't know if you're going to be interview interviewing him. You should, but uh. I did contribute um. Where am I? There you are. I did contribute a chapter. So uh. I I don't get royalties. I don't get anything. This is more of like you know me bragging. One of the my mom has one of these. My mom knows actually. My mom doesn't even speak English and like she knows zero technology. But she has one of these because you know right here right nice right nice it man. Yeah yeah so. And also if you all are going to be at HashiConf 2024 in Bost. Poto is going to be on one of our panels there and he'll also be there live as well. So you'll be able to meet him right um. But I think that's it right. I think that's all we've got. So um. Thank you so much. October 14th to 16th right? Yeah.

October 14th through 16th. You think that the Hashi Corp dude would have said that, but that's why you're an ambassador and I am not. All right, thank you so much, PTO, for taking the time to share your thoughts and ideas, and thanks to everyone who tuned in again to watch, whether you were here with us live or after the fact. We look forward to more episodes of the Hashi Corp Ambassador Thought Leader series, and until next time, hey folks.

Day 366 operations: The elephant in the infrastructure

HashiCorp

P, a HashiCorp Ambassador, discusses the often-overlooked challenges that arise after the initial setup of a project, emphasizing the significance of Day 366.

He highlights the common issues, such as expiring certificates and the need for evolving systems, that can occur after the first year.

P shares personal experiences, including a failed certificate and database credential challenges, to illustrate the importance of long-term planning and decision-making in operational management.

The Importance of Long-Term Planning in Software

The speaker discusses the consequences of taking shortcuts in software development, using the example of an expired certificate.

They highlight the common mindset of maintaining the status quo instead of proactively addressing potential issues.

The talk emphasizes the importance of long-term planning, growth mindset, and the necessity of enforcing updates in software systems to avoid operational challenges.

The Importance of Regular Updates and Follow-Up Culture

Regularly checking for updates and security patches is often overlooked, leading to potential issues in an organization's systems.

It's crucial to establish a concrete organizational policy for updates and not rely on a single person for the entire ecosystem.

Additionally, fostering a follow-up culture to address shortcuts and urgent changes is essential for maintaining system integrity and preventing future problems.

The Impact of Remote Work on Communication and Team Culture

Remote work can lead to a loss of organic conversations, making it difficult for team members to stay informed about important decisions and changes.

This lack of communication can create roadblocks and dependencies on individual follow-ups, which is not ideal for team dynamics.

Implementing a culture of documentation and addressing small tasks, often overlooked, can help maintain team alignment and prevent issues from piling up.

The Impact of Startup Culture on Problem-Solving

Startup environments often have a chaotic culture, leading new employees to adopt inefficient practices.

Junior engineers may feel pressured to provide quick fixes, but this can result in a cycle of unresolved issues.

Proactively addressing problems and creating a follow-up culture is essential to break the cycle and promote accountability within the team.

Technical debt can be good or bad, but it should be managed responsibly to avoid problems.

Automation, such as using policies to block changes, can help enforce accountability.

Establishing accountability mechanisms and conducting long-term reviews of trade-offs can also aid in managing technical debt effectively.

Challenges of Managing Tech Debt in Software

Managing tech debt in software is complicated by the difficulty of assigning costs and responsibilities.

The speaker suggests implementing a system, like an SRE coin, to create checks and balances for taking on debt.

They emphasize the importance of clear communication and collaboration across teams to address the impact of shortcuts and tech debt.

Building Traction and Networking in Your Career

Starting with one ally can help build traction for your ideas, regardless of your career stage.

It's important to gather support from peers and managers to drive initiatives forward.

The speaker also shares resources, including tech content on Instagram and a book recommendation on work management, and mentions their upcoming panel at HashiConf 2024.