Episode 077: Fireside Chat with Kelsey Hightower | CloudSkills.fm

In this episode I sit down with Kelsey Hightower, Dan Wahlin, and Anthony Nocentino for a fireside chat at the Kubernetes Day One event. Tons of awesome takeaways for anyone interested in Kubernetes.

Don’t forget to subscribe to our mailing list at cloudskills.io/subscribe for weekly updates, exclusive training, and advice on how to amplify your career.

Full Transcript:

Mike Pfeiffer:
Hey, what’s up everybody. It’s Mike Pfeiffer and you’re listening to the CloudSkills.FM podcast.

Mike Pfeiffer:
Kelsey, thanks so much for joining us today.

Kelsey Hightower:
Awesome. Glad to be here.

Mike Pfeiffer:
So, to kick things off, everybody here is a beginner to Kubernetes. From your perspective, and have seen so much over the years, what is the main thing a beginner completely starting from scratch needs to start thinking about?

Kelsey Hightower:
So, I’m very much a pragmatist. Kubernetes is great. I love this community. There’s a lot to be excited about. And when you look at Kubernetes, the things that it does, I think if you’re a new beginner, you have to look at Kubernetes as just a most recent checkpoint in the way we think about managing and deploying applications, right? It’s an infrastructure layer component solution project. If you understand where that line is drawn, you might understand that Kubernetes is probably not the missing piece to your application level problems, right? For example, lots of people have just maybe code that they’ve written 15 years ago that just hasn’t been maintained. They haven’t updated the libraries, they haven’t figured out things like tracing or structured logging. Unfortunately, Kubernetes is not the missing piece to that, right? That particular problem is a little bit different than how we package and deploy those apps.

Kelsey Hightower:
The nice thing though, is Kubernetes does have a few [inaudible 00:00:01:31]. So, there is some logging infrastructure usually installed in most distributions of Kubernetes by default. There’s a way of thinking about health checks by default, but it doesn’t mean that your app is magically going to have health check-in points. So, I think if you’re a new beginner, you just have to look at this and say, we haven’t changed how the computing world works, right? It’s still x86. A lot of people are going to be deployed through maybe a Linux kernel and that hasn’t changed. How we do it has just now been formalized in this checkpoint we call Kubernetes.

Mike Pfeiffer:
I’d love to go around the horn here. Dan, from your perspective, I know that you’ve been working with developers for a long time, and I know that in the last few years you’ve been really diving into Kubernetes, and I know that you’ve seen lots of stuff in the field. I’d love to hear your thoughts for beginners starting from scratch, and also maybe touching on some of the points from Kelsey there.

Dan Wahlin:
Yeah. Well, Kelsey nailed it there because I think the number one thing not to do is assume that it’s the magic bullet. I think it can actually be the opposite and shoot yourself in the foot if you don’t have, I specifically like how he mentioned tracing. And if you’re not already doing some pretty serious logging in your apps and tracing throughout processes and things like that, from a developer perspective, it’s really hard to track down where that bug may be. Especially if you’re doing things like microservices, where you have a lot of different services out there as what we’ll learn about is pods in containers. And I think the challenge there is when something does come up first [crosstalk 00:03:07] commands today that will help troubleshoot, but I think having that solid diagnostic capability already going before you make the jump is going to be a big deal.

Dan Wahlin:
And then, I think after you make the jump, this will be something I, as more from a developer background, don’t deal with as much. But sometimes you need, whether it’s [inaudible 00:03:31] or some other dashboard type approach to get insight into what’s going on, because I think that’s the missing link. A lot of people see all these benefits we’re going to talk about today, but then we don’t talk about the, “Well, what if you don’t actually monitor?” And hopefully everybody’s going, “Oh yeah, we monitor already at work.” And if that’s the case, great. You should be good, but I did want to ask Kelsey. So, is there anything, because I’m definitely… Kelsey, by the way, good to meet you. I’ve never met you in person, but I’ve seen all your talks and stuff. Great stuff. From a developer perspective, is there any major… Because we’re going to talk a little about Docker Compose today, for example, and the jump between. Any major insights that you’ve come across that have really helped? Because I think the challenge is you’ve got to bridge, you’ve got dev ops, you got developer, and in the old days we didn’t talk, but now all of a sudden we should be. Any thoughts on that at all?

Kelsey Hightower:
Yeah. So, I went through this evolution quite a bit. I’d probably say I started my developer days back in the JBoss days, right? So, Java was going to take over the world, there was this grand vision that you could take someone off the street and teach them Java through bicycle applications and examples, bike shops. And then, they would just not know anything about computers and they could just target the JVM or something like JBoss and application server. So, the problem with that is everyone’s mind was like, “Oh, I have to have an IDE. Everything’s going to be a war file, and there’s going to be some magical thing that’s going to run my apps.” And I think that was an overcorrection in developer responsibility. Meaning a lot of developers are like, “Oh 16 gigs of RAM. Mine.”

Kelsey Hightower:
And then, you started having this explosion in larger servers and it just got out of control. We saw the same thing with the web, right? A lot of websites are over bloated, and then the workflow problem. So, in that world, I would probably say that professional developer world, where you have the luxury of having someone else do your deployments. Someone sitting in the change window, doing everything for you and if there was an error, just giving you the logs back, right? And that actually does work in a lot of scenarios, that isn’t actually terrible. The problem though, is when you’re facing a situation like your dev environment. So, let’s talk about that type loop that you just talked about. I’ve seen people go from, “I can develop on my machine,” to be using something like Vagrant because VDMs came out.

Kelsey Hightower:
They figured, “Oh, okay. Maybe developing on my laptop shouldn’t be done that way. Maybe I’ll stand up 60 VMs on my laptop and need 128 gigs of RAM on my laptop to pretend I’m mimicking production.” And then, we see things like Docker show up where people’s like, “Oh, if we’re doing things with containers, I guess I got to change everything again so that my app runs in a container.” That’s where I think we’ve gone off the rails. I really think we’re overdoing this stuff. Your laptop will never ever look like production, ever. There’s nothing you can do, sorry. It’s unfortunate. The best you can hope for is that your interfaces assume certain inputs, you guard defensively against those bad inputs, and then that’s as far as you’re going to go on your laptop. And then, when you get into an integration environment, you’re going to be able to exercise some of those things that you’re never going to be like production, even in an integration environment.

Kelsey Hightower:
So, when you start talking about Docker Compose, I see how it overcomes some of our framework limitations. If you’re writing in Python or Ruby, RVM and all of this, I can’t have multiple Ruby versions or I guess dependency clashes. You’re trying to overcome that problem. That isn’t the same as how do I deploy my apps at scale? So, I think for the developer workflow, one thing I try to preserve, if you’re new, I preserved this whole time the ability to build my applications locally. I don’t assume Docker would be there. I don’t assume Kubernetes would be there. I look at those as just last mile targets because in 10 years we’re going to be talking about something totally different and it’s going to be the new cool thing, and we’ll probably have a job converting legacy Kubernetes installs to the new thing.

Mike Pfeiffer:
Totally agree. Dan?

Dan Wahlin:
Real quick, short story on that and then we can go to Anthony. There’s a company here, I live in Arizona, in Scottsdale and they had the whole microservices thing going and they didn’t have that many. It was a lot, but not that many. I think they were up to 23 or 25 microservices, and they hit exactly what Kelsey just said, where it’s like, “We’re our of RAM, we need bigger laptops.” And it’s like, no, that’s not maybe the answer to that problem. Maybe you should just have a dev environment they can call into that has those services? So, yeah. And we’ll when we get more into Docker Compose and moving to Kubernetes. We’ll talk about that.

Kelsey Hightower:
I see a couple of questions in the Q and A, you mind if we take those really quick, because I know a lot of people show up to get their questions answered, but there’s a good one in here. How would you explain Kubernetes to a 15 year old? And there’s two approaches here. Learning is the best thing ever. I don’t care if there’s a better, fully managed solution out there, there is no substitute for learning. Even if it’s a useless attempt at doing something, you should just put your hands on it and learn. So, if you’re a 15 year old and you just want to learn the ins and outs of this kind of stuff, I will go all in and I would show them things like Kubernetes the hard way. I would explain to them that before you can respect you understand Kubernetes, I got to take you through the system administrator journey, Bash, Linux.

Kelsey Hightower:
I got to show you the pain of SCP, trying to figure out what server to put things on because those are the fundamentals. So, as a 15 year old, you’re going to need those fundamentals at some point anyway if you’re going to be trying to troubleshoot this stuff. And also, the whole world isn’t running on Kubernetes, so you’ve got to understand the sweet spot. Once they got that base layer, then I would just take those concepts like, “Look, son or daughter, how did you pick that machine?” Well, that was an act of scheduling. We have a automated piece that can then just take your inputs. Instead of you eyeballing all of the available systems, this thing can do it for you. So, I actually use Tetris for that analogy to show making decisions based on inputs, the act of scheduling.

Kelsey Hightower:
But also the 15 year old, let’s say you’re just really interested in just running your app. I’ll give you a quick, short story. My daughter, I was teaching her how to program, with starting with HTML. And she’d built the thing, it was looking like geo cities in my house for a couple of days and she got it running. And I remember she tried to send her friend the URL to view. She sent her friend 127.1.0.0.1, and she said her friend can’t pull it up. And I was like, “This is the moment.” This is when I do what I was called for. And I started thinking, do I teach my daughter Docker and Kubernetes and Influence and all of that. And I was like, hell no, that’s just not the right thing to do. So, we ignored Kubernetes, we just use Firebase. We did Firebase push. The damn thing came up on a URL. She gave it to her friends, and now all the friends are engaged in 1990s web design, and it’s beautiful.

Anthony:
That’s awesome. So, I actually had to do this recently, not Kubernetes specifically, but I had to explain computing and resource management to 12 year olds. And so, I went to the career day at my daughter’s school and they’re like, “What do you do?” And I’m like, “Well, my job is to make systems faster because I come at this from a performance and operations background. So, you guys still got log in and things like that, I care about memory management, consumption and resources and things like that.” And so, for me, the best way to explain the concepts is I started a Q and A, and we started going through what the things that they do with computers. And it goes straight to Xbox and gaming. They’re like, “Well, what’s lag?” And it gave me the opportunity to explain latency, right? And try to talk about how data moves between systems and things like that.

Anthony:
And I saw a lot of heads nodding. I also saw a lot of heads falling asleep, but the cool thing for me was being able to make that connection to a thing that’s relevant to them at that exact point in time. So Kelsey, you were talking about web development and sharing content with your daughter and her friends, so I think the best way is making it relevant and consumable to younger folks, that they can conceptualize what’s really going on. Because we have 15, 20, 30 years of experience in what we do, and we just take some things for granted when we’re instructing sometimes. And that’s the big one when you reach into a different audience, is are you connecting to that audience in a way that they can comprehend and understand and consume?

Mike Pfeiffer:
Yeah. I think another interesting angle of this, I love the looking at it from a younger person’s perspective, but also the executive perspective. There’s lot of business executives, they hear about this technology and the promises, what’s the best way to explain Kubernetes, Kelsey, to the C levels at these companies so they know maybe is this the magic bullet or what should they really be thinking about?

Kelsey Hightower:
There’s only one explanation that has ever worked with me at that level and it’s results. No one cares about Kubernetes at all. No one does, sorry. It’s unfortunate. The thing people want to see is can you ship the ideas and solutions that our customers need? If Kubernetes is the thing that once you show me some results and you say, “Hey, we got to go out and purchase something like OpenShift, there’s a question here about OpenShift.” Those are districts of Kubernetes, so you have to make that call. Do I spend a bunch of time installing Kubernetes, ramping up my team of folks being great Kubernetes engineers and replicating the GKE team at Google. That might be a valuable use of your time if you can prove it out. But for a lot of people, if you’re talking about the executive, just think about the day of someone that’s supposed to be focused on the big picture, right?

Kelsey Hightower:
I got to hire people. I got customers, I got revenue to think about, jobs to preserve, COVID-19. Have all this stuff to worry about and you come in, you show me Kubernetes and containers. And I say, “Okay, don’t know what that is. What do I get for that?” And you say, “Well, in about 18 months we’re going to redo everything. It’s going to look roughly the same as before, but we’ll be using containers and Kubernetes.” “Get the hell out here, man. I need to understand what the benefit is.” So, I think there’s some value here. So, what I would do as a beginner, I would see if you understand the basics of your current problems, maybe you have a hard time scheduling, maybe your bill is a bit too high because of resource utilization, maybe you don’t have a common standard to package up applications and deploy them where you want and when you want. Or maybe you just don’t even have a workflow at all.

Kelsey Hightower:
Installing Kubernetes may check some of those boxes, and then the second thing you got to do is leverage it for results. So, if I go back to the exec team and tell them, “Yo, we can now do a deployment in 30 minutes,” right? I can do it in 30 minutes. Maybe there’s another way of doing that with other tools, but we chose Kubernetes and we’re at that 30 minute boundary, this means this for the business. An executive might say, “Oh great, how’d you do it?” And then you can say Kubernetes.

Mike Pfeiffer:
I love that. There was another really good Q and A question. Kelsey, we talked about this last year when we got to catch up on the CloudSkills.FM podcast, but it was a question about certification. I’m a big believer in certification, and what are your thoughts on the CNCF certification and the value of that for everybody on the call?

Kelsey Hightower:
I want to address a follow up in the Q and A. I’m at the CXO level and I care. So, the thing about that is I probably believe this person, right? Because there’s different types of CXO roles at most companies, right? And some people do get into the lower level details, but I remember a time when people used to care about Linux a lot. I remember when people used to care about the TCP stack and TCP versus Net Buoy a lot and printer drivers a lot. And then, over time you just stop caring about some of that stuff because the thing on top becomes the value point. So, I think the reason why a lot of people may care is because it’s so new, they see a lot of potential and they are like, “Wow, I hear a lot how people are changing how they do things because of Kubernetes, I want in on that.” So, it’s very interesting until it gets boring. And then, it’ll be like virtual machines. I haven’t seen a virtual machine conference in a long time. I haven’t heard about a virtual machine strategy in a long time. So, I just see this as infrastructure, so I just wanted to follow up with that. So, it’s okay to care about it, but it isn’t the end game, and that’s the thing that’s important. And repeat your question again, sorry. I just wanted to make sure I followed up.

Mike Pfeiffer:
I love the extra context that, it matters, it helps, but the question, the other one that came up was about certification and the value and really doubling down on that. And because it is a significant investment, to go through and do all that?

Kelsey Hightower:
I started my IT career with certifications, right? So, my last job in high school, I went from McDonald’s to Pizza Hut and I was like, “What am I going to do? College looks okay, but it looks a bit boring for what I want to do.” And I remember getting my A+ certification, right? A motherboard. What is a PCI slot? And I remember taking those little Q and A sessions in my browser trying to learn and remember all the answers, but I did learn a lot about computers, just how to build them from scratch. And it was a sense of an achievement for me, it was very accessible, right? Where everything else is a multiyear investment, four years for college, could be expensive for a lot of people, but certification, there was just a book I could buy for 30 bucks, go through it and then have a test that I can take and actually got my first tech job because of that certification.

Kelsey Hightower:
So, I like the fact that for a lot of people, we have a structured path towards achieving something that will give you a bit of confidence. Now, is it a replacement for experience? No, right? I remember going to take my Red Hat certification after the A+ and network+. You had to have hands on skills and there was a bit of a difference than I think the CNCF has been trying to go in that direction, to make sure people really know their stuff. So, I applaud the effort and I think it’s an accessible way and I think it’s also part of the success of Cube that people have a place to learn and certify their skills.

Mike Pfeiffer:
I agree. I had a similar experience. My first certification was the A+, and I couldn’t agree with that more. I really agree with that. Anthony, what are your thoughts on that? Do you think certification is important or is it more tactical hands on from your perspective?

Anthony:
So, I’m going to add to the A+ certification list also. Almost all of us started in that route. Exactly, right? I had a challenging time through college to say the least, and it was a way for me to start my career. I loved computing. I knew I wanted to work on systems and I was able to set up my own content path to get to where I wanted to be from a technology standpoint. Got my first job because of that first certification, and I continue to support that community with the education and stuff that I do. And it’s the baseline, right? It’s where you can get into and get started and then figure out where you want to go and continue to invest, right? So, it’s a valuable way to get started. And then continue, honestly, never ever stop learning, right? Books and blogs and all that stuff. All that content you can consume as much as you can and continue to learn over time, and if there’s a certification along that way, that’s awesome.

Kelsey Hightower:
Yeah. There’s a question here about basic knowledge required for beginners. If it was me starting now, I don’t think I would skip the whole learning about Linux system administration part, we talked about that a little earlier. The way to think about this as if I bought you a car and you had nowhere to go, and you didn’t even have a driver’s license, but you have this car and you have the keys to it. And that’s what Kubernetes looks like. You can get in there and fumble around. Hopefully you’re in an empty parking lot where there are no other cars, so you don’t break anything, AKA production. And you just don’t have anywhere to go, so you’re not going to have a lot of desire to get in that car and jump on the highway because where are you going?

Kelsey Hightower:
So, a lot of times I think the basic knowledge you want for this is make sure you have a reason and it could be a toy app that you’ve got running on a VM, or you got running on your laptop and you want to just know what it will take to make that thing running Kubernetes. And that’s going to force you to learn about creating config objects and getting an IP and being able to hit it. So, I think Linux is still a requirement and having something to do it should be a requirement. A problem to solve, I think that is part of the basic knowledge here.

Mike Pfeiffer:
I bet Anthony agrees with that. Anthony is a big time Linux expert, and it’s been interesting to see even in the Microsoft world, Microsoft products that were traditionally Windows only are now running on Linux. It’s a interesting time.

Dan Wahlin:
Well, and I’m like you, Mike, I come from more of a Windows background and I knew what [inaudible 00:19:35] was, maybe. And that’s about it, back in the day with CGI is about the only time I ever used it. But I think it’s a fun thing though, because from a developer standpoint, I pretty much run everything now in Linux containers. And it’s one of those things that you hear, “What do you mean don’t run as root user?” And you’re like, “Well, it’s root user,” if you’re new to this, you know? And so, there’s a lot to learn there, but I think at the same time, it’s absolutely fun to learn and it’s worth the effort because I’d say the next step up from that is, I totally agree with Kelsey, you need those fundamentals.

Dan Wahlin:
And I love how we’re focusing this morning guys on do you really need this? Because I guarantee there’s plenty of line of business apps that have 10 users out there, internal apps. No, you probably don’t need Kubernetes or even containers for that. It’s just easier just to deploy the app, but then there’s the other category of apps that have a lot of moving parts. Maybe not like Google or Netflix level, because that’s extreme, but where containers has a huge advantage. And I’ll talk about that in my little section a little later, but I think the next step up from that is then, what are the container fundamentals and do you understand why are you using a container? Because if you’re doing it because everyone says, you’re probably just going to fall off the cliff, like those that did the same thing and not have great success there.

Anthony:
I’m seeing our participant count go down as we can make this point.

Dan Wahlin:
Right.

Kelsey Hightower:
I want to tackle this question about stateful services. So, there’s a way you think about putting global directory services into Kubernetes for easy scale resources. I think people really have to stop for a second. Kubernetes is just a thin layer on top of Linux, right? Hey Kubernetes, put this binary on this server. If Beck’s server has a file system on it, you can read from that file system just like before. If you want to mount some storage, you can mount storage and Kubernetes can maybe automate that storage for you. The thing Kubernetes doesn’t do is it doesn’t deal with things like replication. It’s not going to turn your application to this smart thing where it’s like, “Oh, I was writing to the local file system on server A. Server A goes away.” Now, Kubernetes is really great, right? It’s going to put that app on server B and keep running, but it has no concept of what to do with that file system. It has no idea what you wrote on that file system.

Kelsey Hightower:
So, if you still want NFS, great. NFS, “Hey Google, make sure you mount that share on this server while the app is running there and then mount it on this other server.” The fundamentals have not changed, right? If you’re still doing positive stuff where you’re mounting in a directory from a server, maybe you have a large SSD only on one of those servers. That’s where your data is going to have to live, on that one server. So, 20, 30 years ago, we tried to solve this with NFS, there’s iSCSI in the cloud that can attach these volumes and detach them, but nothing’s changed. Kubernetes just automates and allows you to articulate that’s what you need. So, can you put stateful stuff on Kubernetes? Sure. You may have to pin it to a single server instead of letting things automatically move around because your data may not be there when you do. So, it’s not really about can Kubernetes run stateful stuff. Of course it can. It’s just that Kubernetes is not going to help stateful stuff be any better than it was before.

Anthony:
It’s important, my background is in the SQL server world and when SQL server came to Linux, then next thing you know, SQL server comes to Kubernetes. And that’s the key thing that we have to deal with, right? A relational database system is basically the definition of a stateful application, but it winds up being more liberating when you’re deploying applications like database systems in Kubernetes. In the sense that you can decouple configuration, computation and state, and you have these three different things that really changes how you can deploy your application. As long as I have access to the data, just like Kelsey described, and I have the configuration I want to assert on that particular system, as long as I can get that up and running somewhere in my data center, that’s a really powerful construct and breaks that traditional dependency. Especially in database systems where I had to keep PRD SQL 001 online all the time, right? Now, I have the ability for this thing to move within my data center provided I have the ability to move storage around, and that where that state of my application really is. And that, for us as database professionals has really changed the way that we deploy apps or deploy databases in data centers. So, that’s one way that we’re deploying safe applications in Kubernetes.

Kelsey Hightower:
I know we’re low on time here. There’s one question about, can we describe Kubernetes in simpler terms? The problem with Kubernetes, it’s like eight different systems in one. It’s not a one thing. The LS command is one thing. The Linus kernel could be seen as one thing. Your network card can be seen as one thing. Kubernetes is trying to automate all of those things. That’s what it’s trying to do. There’s a component in Kubernetes that picks what server to run. There’s a component in Kubernetes that says, once you picked a server to run, I’m going to run it and babysit it and if it crashes, I’m going to restart it. I’m going to take its logs and push it somewhere. I’m going to health check it. It’s all of these systems bundled into one platform. So, if you want to think about Kubernetes, it is an automation platform. That’s what it is. It’s an opinionated automation platform.

Kelsey Hightower:
A lot of the integrations, like when you say give me a load balancer and somehow a load bouncer appears, it’s only because there’s a little control loop there saying I know how to create load balancers in Azure, AWS, and GCP. So, I’m going to take your configuration and create a load bouncer. If you’re really struggling to understand this, think about Puppet, Chef and Ansible. Those are also just configuration platforms. The only difference to me really is that Kubernetes adds a scheduler, picking what machine instead of you coming with the inventory file. And second, it provides application lifecycle management in places where those other tools didn’t, and it gives us a new config language. This is the key here. There’s a new config language that’s a little bit higher level, most people talk about it in the form of Yammel. It’s just an automation platform, so you’re not going to get it in one term. Unless you know the problems that it’s trying to automate, it’s always going to seem like this magical black box. And in that case, you might want to look at things like Kuroku or a cloud Foundry, Cloud Run or Lambda because those things have a lot of that stuff underneath and hidden because all you may care about is just running an app.

Mike Pfeiffer:
Yeah. I think the big takeaway for me in this discussion is really focusing on value and solving problems. And I think today’s going to be a great way for everybody to know, when do I choose Kubernetes, when do I don’t? And when I do choose it, how does it work and how do I get started? Kelsey, we know you’re super busy and we really appreciate you coming on and doing this. It means a lot. Thank you.

Kelsey Hightower:
Awesome. Thanks for having me. Good luck to all the new people out there running Kubernetes. Stick with it, you’ll get it.

Subscribe to the CloudSkills Weekly Newletter

Get exclusive access to special trainings, updates on industry trends, and tips on how to advance your career in the tech industry.