Episode 070: Securing DevOps in the Cloud | CloudSkills.fm

Mike Pfeiffer on April, 09, 2020

In this episode I catch up with Julien Vehent who leads the Firefox Operations Security team at Mozilla. Julien is a security architect, DevOps advocate, and is responsible for the security of Firefox’s high-traffic cloud services and public websites.

Don’t forget to subscribe to our mailing list at cloudskills.io/subscribe for weekly updates, exclusive training, and advice on how to amplify your career.

Full Transcript:

Mike Pfeiffer:
Hey, what’s up everybody? It’s Mike Pfeiffer and you’re listening to the Cloud Skills FM podcast.

Mike Pfeiffer:
What’s up everybody? It’s Mike Pfeiffer. Welcome back to another episode of the Cloud Skills FM podcast, and in today’s episode, I’m super excited to be talking about security when it comes to doing things in a DevOps space, in the cloud space. In this episode, I’ve got Julien Vehent with me. He leads the Firefox operations security team at Mozilla. Julien, how’s it going?

Julien Vehent:
Hey Mike. Hey, it’s good to be here. Thank you.

Mike Pfeiffer:
Awesome. Yeah, so really excited to have you here. I know that you’ve got a lot of practical experience, and for anybody that doesn’t know you already, maybe you could share with us kind of who you are, what you’re working on today, maybe how you got to this moment in time in your career, all that fun stuff.

Julien Vehent:
Sure, yeah. Sure, so I’m a security engineer at heart and that’s what was in my training. I got a master’s degree in information security about 15 years ago and I’ve been in the industry since. I’ve been on the operational side of things for the best part of two decades now, building and running services on the internet, and I try to apply security to everything that I do, and that’s what led me to work mostly on security nowadays. I’ve been at Mozilla for almost seven years now, working in cloud services and helping developers and operators run highly secure cloud services for Firefox users. So I lead a team of eight people now that is entirely focused on operation security in the world of Firefox services.

Mike Pfeiffer:
Nice. I guess you guys are probably busy right now with the surge of internet activity, I would assume?

Julien Vehent:
We’ve certainly seen a number of spikes, yes. People that are using the internet more than ever and it’s more than ever important to give them secure tools and secure infrastructure to navigate a fully remote world safely.

Mike Pfeiffer:
Right. Yeah, totally agree, and I love the book that you put outs through Manning, so Securing DevOps: Security in the Cloud, and I know that security is a space a lot of people are getting into for the first time, but a lot of people already have lots of IT experience. We’re seeing lots of practitioners kind of moving to this space. I’d love to talk about the book though, just basically what it’s all about. I know that you kind of break it down from beginning to end, but maybe we could talk a little bit about the book and how somebody listening might be able to work through it, because we are giving away copies of the book through Manning as part of this episode for everybody listening, and so this is an awesome book, but yeah, Julien, maybe you could take us through the book and explain what it’s all about.

Julien Vehent:
Sure. So perhaps a little bit of history would help. A few years ago, I think it was probably five or six years ago, I was a working, like most people, on traditional data center infrastructure with the LANs and everything, but I had been running services in AWS for several years by then, and at Mozilla, a group of people started working on what we called the cloud services and how we would run those cloud services in the cloud and take advantage of all of the new operational techniques that had been developed in the early 2010s, mid 2010s. So that’s immutable deployments and CI/CD and all of that stuff, and they started asking security questions. How do we secure high-scale services in AWS and in the cloud in general? What kind of problems do we have to deal with? How different is it from traditional infrastructure? And I started working with a group of engineers in trying to figure out exactly how we’re going to implement a security strategy for a cloud environment.

Julien Vehent:
And almost at the same time as I was doing that work, many reached out and said, "Hey, we’re looking for someone to write a book about DevOps security. So as I was hashing out the details of security strategy for Mozilla and the Firefox services world, I also was writing the book at the same time and kind of mapping what we were doing at work with what I was writing for in the book. So the book is designed to be a progression from … we’re starting with something very, very simple, to we have a number of large scale services that have been running for several years, and how do we go from this very simple environment that has very simple risks to this massively complex organization that needs to deal with massively complex problems? And it goes through the technical aspects of building security in the cloud, building security specifically in AWS, but there are also these concepts applied to other cloud providers as well, whether it’s GCP or Azure or others, and some of these concepts are more engineering concepts and cultural concepts that security teams need to adopt in order to work efficiently with DevOps teams. So I tried to kind of mix cultural and the practical with the technical in the book and give something that is very tangible to the reader.

Mike Pfeiffer:
Yeah, I love that, because we really need that. It’s been obvious to me over the last couple of years working with so many people, it’s a lot of people new to just the concept of some of the basics, like you mentioned, CI/CD, immutable infrastructure is a big stretch if you’ve been used to doing enterprise IT for awhile. Lots of people in the enterprise space deploy servers for three years and you don’t have to think about it again until the next refresh, and so now everybody’s getting into DevOps and learning that, and then I think to your point, once you see a pipeline and finally understand how the components fit together, now you’re like, “Oh, how do we secure it?” And so I love the first kind of section of your book is all about taking a bare bones pipeline and then looking at every layer pretty much of the stack and where could we apply security everywhere along the way. So web tier, infrastructure tier, stuff like that?

Julien Vehent:
That’s right, and what’s interesting is the first chapter of the book is the one that really took a long time to write, and we’re talking maybe three or four months to write that chapter, because I worked with my editor a lot on refining that first chapter, and it’s available for free, so anyone can go download it and read it, and kind of map what the book was going to look like based on this first chapter. In my initial approach, having the experience that I had in traditional infrastructure, was to compare what we were doing in traditional infrastructure with what we would be doing in the cloud, and that turned out to be inefficient in the most part, and we cannot scrap that out completely after the first months of thinking about it, because in fact, it’s much, much harder to try to transfer existing security knowledge to the cloud than it is to just scrap it out and go look at a fresh clean slate and figure out how we’re going to secure it from the get go. A lot of techniques that we took for granted in traditional data centers … you control the network, you can go maybe plug something on the core switch and dump all the traffic and do network analysis are completely impossible to do in the cloud.

Julien Vehent:
And many people have tried to do deep network security monitoring in AWS, and I’m sure you can find ways to do it, but that’s not necessarily how you should approach the security problems. So instead of trying to map the old to the new world, we took the approach of starting from a clean slate completely and looking at why we’re securing these things and what layers do we have to worry about and how do we map our security requirements into those various layers. So taking advantage of completely programmable infrastructure, for example, or the fact that we had … anytime you’re running something in AWS or GCP or Azure or any other, you have a perfect inventory of your assets. That doesn’t exist in a data center and it’s a very powerful tool for security and for access control and everything else. So really the goal of Securing DevOps was to explain to people how they could think about security in a brand new world and kind of abandon the older tools that don’t really map well into this current world.

Mike Pfeiffer:
That is a massive point that I’d love to underline, because I see that a lot. I see a lot of over engineering because of the fact of what you said is they take stuff that has been on prem and they try to smash it into the cloud using the old pattern and practices. I love that you took that approach, because I think more people need to do that. I think over-engineering is a side effect of trying to stick with what you know and not really sitting back and being like, “Hey, this is a new way of thinking,” and so that’s awesome. Was there any major challenges along the way that had to … maybe something unexpected that you faced or anything like that? Or was it pretty much just trial and error and slowly chunking your way through it?

Julien Vehent:
I think the biggest really hurdle for people to adopting the cloud is figuring out exactly how much you trust it in the first place, and what does that mean in technical terms? Any organization that goes from on prem to cloud starts with a very low trust approach. They don’t want to trust AWS or any cloud provider, but over time you realize that yeah, you have to trust it, and when you start trusting it for certain things you don’t necessarily need to over complexify your security. Like you said, that keeps things simple so you can keep them secure as well. An example of that would be do you need encryption inside of your infrastructure between your application node and your database, for example, when it entirely depends on how you run your environment?

Julien Vehent:
If you’re on prem and you have a completely locked down data center, no one can get to the hardware. Do you really need your encrypt what’s going between the application server and the switch into database server? What does that buy you on top of everything else that you have? Perhaps not. Perhaps you do. It entirely depends on your environment. In the cloud, it’s the same thing, and at first people are don’t want to trust the cloud because it’s something that they don’t directly own, but over time they get to learn, all right these controls I’m on edge by the cloud provider and we can trust them to do a good job and we can focus on other things, and little by little, people go further up the stack and they start thinking about problems more from a business perspective and a functionality perspective and less from a hardware perspective.

Julien Vehent:
And that’s the hardest problem for anyone really approaching DevSecOps from the beginning, and it’s to detach yourself from the hardware and stop looking at those problems as functionalities and business risks and this from really what’s going on on the wire. That was the biggest hurdle. From a technical perspective, perhaps the most complex technical challenge of adopting any of these technologies is that they are very, very complex and it’s very hard to ramp up, whether you look at GCP or Azure or AWS or you start talking about [inaudible 00:11:11] or even Puppet, Ansible, all of that stuff, Terraform, whatnot is super hard to do. It’s going to take weeks and weeks and weeks for any engineer, whether they’re junior or experienced, to figure out how to use them at the most basic levels. It’s going to take them years to become experts at it, and what I’ve seen … and I’ve heard you talk about this with a previous guest, what a lot of people try to do it to go straight for the most complex possible solution from day one.

Julien Vehent:
“I want to go to the cloud. I’m going to do [inaudible 00:11:43] myself in a super complex AWS environment,” and you’re going to completely fail at it, your services are going to be unreliable, and you’re going to just go crazy with the of complexity you have to deal with. A better approach, I think, is to gradually add complexity as you need it. So one I personally love, if you’re a single dev building stuff is to just use something that Heroku, which is really just a couple of lines of configuration. You can fully automate your deployments in a GitHub repository, and there’s almost no complexity to deal with, but that’s only going to take you so far. There’s going to be a point where if your company grows and it starts having specific needs around data analysis, maybe you want to access more databases, et cetera, et cetera, and you add more engineers, then you’re going to add complexity. You’re going to adopt technologies that are more difficult to manage, more difficult to understand, and that’s really the complexity of the cloud that becomes apparent to you is that those things are really fricking hard to use. Don’t try to use all of them from day one.

Mike Pfeiffer:
Yeah. It’s like that don’t try to boil the ocean, or try to hit a home run your first at bat in the major leagues. So true, and I see that. I see that a fair amount, and yeah, we have talked about that in the past. I love that you kind of touched on the fact that there is an element of re-learning or unlearning going into this world, because a lot of people do drag in their baggage of thinking of things in terms of hardware or the way they did it on prem. It’s good stuff, man. It’s really good stuff. So in terms of different things inside a pipeline, there’s different ways to do security depending on the different phases of the pipeline. There’s concepts like static code analysis and then there’s a [inaudible 00:13:27] response further down the pipeline, and there’s the whole concept of analyzing your logs and intrusion detection, all these kinds of things, and that’s kind of what you step people through in the book, kind of procedurally one step at a time and uncover those concepts, is that right?

Julien Vehent:
Yes. I think there’s a fairly reasonable approach to building a security program in an organization. I’ve seen teams that have terrible infrastructure security invest it all in threat hunting. I’m like, you’re trying to solve a third year problem when you don’t even know how to do the basics of year one, and I think that’s generically a mistake. What I recommend and the approach we take in the book is start with the basics of running a service in a cloud environment. The security groups, the permissions on the users, the making sure your databases aren’t exposed to the internet with weak username and password, the very basic things, and you might think that this is too basic, but in fact to this day it is still what we spend most of our time auditing in very complex professional environments, because it’s very easy to build a database and let it run for a few years and add to it over time and overlook a very simple security controller and leave it open on the internet. Simple cases where I’ve seen this happen is a database was being migrated, so someone opened it up to the internet because they needed to open for a day and forgot to close it.

Julien Vehent:
And these mistakes happen. You need to audit for them, you need to make sure that at any time your infrastructure is absolutely secure, and then you can go up to application security and look at, again, some of the basic controls, make sure you use safe web framework and make sure you can test those frameworks in CI/CD so you can leverage automated testing tools. We use ZAP, the Zed attack proxy heavily in the team and we automate scans in ZAP using Jenkins to the applications all the time, and we do things like baseline. I talk about it in the book and we use it quite a bit in our services. We call that the baseline scan, which is really a set of 20 or 30 basic tests that all applications must be able to pass, and it’s the basic stuff. There’s nothing super fancy in there, but covering the basics is really what should keep a security team busy on their first year. Make sure you understand how the infrastructure and the applications work and you have good basic security and then ramp it up and go to more complex topics over time.

Mike Pfeiffer:
Yeah, I totally agree. I think a lot of the black eye news stories that some of the platforms have gotten, it’s usually a user issue, to your point of not doing the basics, not applying proper fundamental security controls, like the data leak issues in Amazon S3, usually Amazon gets the black eye, but it’s typically the customer that didn’t configure their security rules the right way or their permissions, the access lists for their buckets and stuff, or -

Julien Vehent:
Yeah, it’s super easy to create a mystery bucket and just leave it like this, and one day someone would need to have access to the data, you create an ACL and leave it there. What’s really hard to do, in fact, is to regularly audit your cloud infrastructure for these kinds of issues, and I talk about this a lot. There’s this idea of test-driven security in the book that’s free. Make sure that even before you start building a service, you have good tests for it, because none of your services are going to look the same. You’re going to have 20 applications that are all just one database server and one application server, and you cannot use the same tests across all of them, and so make sure you have good tests and you run them every day and you can detect those very basic failures, because it’s easy for anyone to make just a quick mistake and for that quick mistake to become a huge, huge problem.

Mike Pfeiffer:
Right. So when it comes to doing all of this work, one of the things that I think will happen at scale for most companies is a very locked down pipeline and making people do all of their changes through code. Is that something that you guys do, where nobody actually goes in and touches the infrastructure, it’s all running through a pipeline? So you’re validating the infrastructure, you’re validating the code itself or the application, all that stuff. Is that kind of how you guys handle it?

Julien Vehent:
Yeah, and in fact that’s how we’ve been operating for I want to say four years now, four or five years. At first it wasn’t necessarily straight forward. It is still difficult to do that for legacy applications that weren’t really built to be deployed this way, but yeah, a typical application nowadays will be running inside of a Docker container. So we don’t have to worry about deploying dependencies directly on systems, and so all the pipeline has to do really is to create an application server that’s going to run a Docker container, is going to open … maybe provision a database and migrate database versions, and when you re-deploy a new version of the application, you essentially create a separate stack, qualify it, and when it’s ready, you move your load balancer to that new stack.

Julien Vehent:
So blue-green deployments where you have those two stacks ready and you can even send a little bit of traffic to the new stack to make sure it behaves properly before you send all the traffic to it, et cetera, and all of that is entirely automated. Now entirely automated doesn’t mean you don’t have operations teams at the wheels, because we still have dedicated, highly experienced, highly skilled ops people who are essentially driving those deployments, but they work like developers. They commit their code, they wrote out features through automations, they wrote back through automation, et cetera, et cetera, but they’re really focused [inaudible 00:19:20]. They’re really focused on the reliability of those running services.

Mike Pfeiffer:
Yeah, and that’s one of the reasons I’ve been talking about that a lot is because I think there’s a lot of ops and sys admin types out there that don’t realize that eventually that’s going to be their reality, of being an more of an SRE and doing things through code than anything else, especially at the Microsoft side of the world.

Julien Vehent:
And I think they’re going to start … and that’s also true for ops security folks who don’t necessarily have those programming and engineering skills, but effectively with the cloud environments maturing, with businesses realizing that they really gain a lot of value from using those cloud environments … because that was still not the case five years ago. Businesses were very reluctant to adopt cloud infrastructure. They wanted on-prem. They wanted to control the hardware. They’re starting to come to terms with that, and so they’re asking of their sys admins and of their developers and of their security people to be trained to be capable of managing those environments, and what that means in practice is that everybody codes, everybody adopts the same programming principle. You end up having sys admins doing sprints on their infrastructure, which is something only developers were doing some time ago. Everyone in my team writes code and it’s a focused security team, but everybody’s a developer.

Julien Vehent:
In fact, when we hire people, whether it’s on the operations side or the security side, we would ask them to a program, to run through a little programming challenge, because we know they’re going to have to write tools. They’re going to have to write AWS API clients to go audit the configuration, et cetera, et cetera. So that really is becoming a reality. There is a change in the type of skills we’re asking. I don’t know if it’s a change. Maybe we’re going back to the 90s where everybody was a programmer, and we kind of moved away from that in the 2000s and early 2010s, but for sure in the 2020s we’re going to go back to a world where everybody codes.

Mike Pfeiffer:
I agree 100%, and I really see that being a big thing in the future, what you’re talking about interviews and maybe asking you to do some white boarding or some pseudo code or some kind of code challenge, which isn’t very common for ops right now, but I believe, to your point, that’s where this is going longterm.

Julien Vehent:
Absolutely.

Mike Pfeiffer:
That’s really interesting to hear, and it’s also interesting to hear from you guys, because … like yourself, Julien, with the book, but also what you guys do day to day at Mozilla, because you were early in cloud compared to almost everybody else. There was several bleeding edge kind of organizations that started there, and you’re obviously one of them, and it’s a good pre-frame to understand where are other organizations going to end up going? Because you guys were a lot earlier than most people, especially in enterprise.

Julien Vehent:
That’s right, and that’s actually interesting to see when I talk to people about security and how we’re adopting security at Mozilla and trying to compare that to where they’re at right now, and this isn’t … I’m not trying to brag or anything, but I do see that we’re a few years ahead, that what we adopted three or four years ago is what most organizations are dealing with now, in particularly for large technical corporations that have a massive on-prem presence and are starting to think about ways to reduce their costs or at least better manage their resources, and so they’re going through this transformation right now and they are asking their engineers to retrain and unlearn, re-learn, transform, to really adopt these new environments. So we’re seeing it, and I suspect in the next … it’s now at the stage where most of the Bay Area tech companies are 100% cloud and all of their people that are retrained. We’re seeing it in major corporations throughout the world, mostly the financial companies that historically had major tech presence.

Julien Vehent:
They are transforming now, retails, et cetera, et cetera, all of these big organizations, manufacturers, and the small businesses, in fact, are starting to think about it. They’re starting to pay attention to it. So even the sys admins, who work in a small 50 person company somewhere in Alabama and only has to manage 10 windows servers in a closet will soon have to think about the impact of cloud infrastructure in their own environment. So it’s coming. They have a few more years to go, but eventually, before 2025, or I think around that time, they will be in the cloud fully.

Mike Pfeiffer:
Yeah, that’s true because the ecosystem changes, and so when the vast majority of everybody is doing things in the cloud, yeah there will still be some on-prem stuff, but you’re going to be left with what’s available in the ecosystem where there won’t be as much on-prem. That’s a really good point. In your book itself, is it platform agnostic or is that more AWS centered? Or is it just kind of like high level patterns and practices? Because obviously the patterns are important.

Julien Vehent:
Right. The practice aspect of the book, particularly part one, is very AWS centric, and there are some things in there that do not really a transfer over to GCP or Azure, in particular when you start talking about access controls inside of AWS, the permission model, et cetera, but the concepts are very, very similar. You may not have a security group, but you have a network [inaudible 00:25:06], et cetera, et cetera. You have ways to convert that to a different environment, and I’d been wondering if I should have a GCP edition of the book, for example, but I don’t think at this point it’s necessarily needed. I think most people … first of all, most organizations, when they think about cloud, they first think about the AWS. That’s the environment they go to. That’s the number one on the market, and then both Azure and GCP are catching up to this quite quickly, but it’s still going to be a couple of years before they really can match to the size, just the impact that everybody AWS has on the crowd world. So they’re challenging it, but it’s going to be a couple of years before they get there I think.

Mike Pfeiffer:
Yeah.

Julien Vehent:
So the book is still -

Mike Pfeiffer:
Yeah. Sorry to cut you off there. I was just going to say AWS is massive. Microsoft and even Google has done an amazing job of closing the gap, but really the one thing that really helps you understand how big AWS really is is to go to re:Invent, their annual conference, especially over the last couple of years, because they literally take up the entire Vegas strip with casinos. They’re renting all the casinos out. I don’t know how many tens of thousands of people are at re:Invent, but it’s kind of obvious when you go to an AWS conference, you’re like, “All right, this thing is way bigger than I probably thought it was,” just because they were first, and so you’re right, it is massive, but what I’ve noticed working with enterprise customers is Azure has gotten a lot of interest over the last 18 months, and now people that are Microsoft shops are going to that first because there’s a tie in with all of the active directory stuff.

Mike Pfeiffer:
So it’s getting interesting, and when I left Amazon in 2016, all I did was AWS projects for the first couple of years. So it is starting to … the pendulum is definitely swinging. So it’s an interesting time, but I think that there’s so many similarities that you could draw a parallel from AWS to Azure and to GCP, and so I don’t think … like you said, I don’t think you need a specific chapter on it. I think the patterns and practices that you cover in a book are really good. I know that you had a case study in there on incident response, so I’d love to hear about that because I think that’s a place where people don’t really have a ton of visibility if they’re new to this.

Julien Vehent:
Yeah. This was a way for me to share. It gets kind of boring to write 400 pages of technical advice. So I wanted just something a little bit different halfway through the book, like a little story, something a little more entertaining, and also I’ve read a lot of forensics and incident response books, and it’s always the same thing. Here is 200 pages of tools and we’re going to run you through how to use them and you feel like you’re reading manual pages for seven hours and I can’t stand it. I can’t. It’s just so boring, and so I wanted to put it in context. So the idea was to take a couple bits and pieces from incidents we’ve had to deal with over the years and turn that into a funny little story of an organization getting compromised, and to show that a typical organization would respond to a compromise. So it’s actually fairly close to how we respond to incidents and security breaches. Not that it happens. We never get attacked. We’re so secure nobody ever tries to attack us.

Mike Pfeiffer:
Right. You’re not a big target or anything.

Julien Vehent:
Not a big target, and so I just essentially tried to map how any organization would efficiently respond to a security incident to not only stop the attack, but also puts themselves in a position to recover as quickly as possible, and the interesting thing is that hopefully the readers would get a few interesting data points out of this chapter, but at the end of the day, you got to run through it yourself. You get to get hacked. You’re going to get hacked at some point if you run something on the internet. No matter how secure you are, you’re going to get hacked, and it’s not so much the attack or the bridge that’s going to take you down, it’s the way you respond to it, and that’s really what I’m trying to show in that chapter is you’re going to get attacked. That’s fine. It’s going to happen to you, just make sure you’re handling it properly, you’re communicating properly, that you’re not burning bridges across your organization, and you’re putting yourself in a position to recover and learn from it and become more secure out of it. That’s really the goal of this chapter.

Mike Pfeiffer:
Yeah, that’s awesome because you need those reference points to really crystallize it in your mind, I think, and to your point, you can have all the binary information, but seeing and hearing about an example really helps solidify it, and so I thought that was a really cool aspect of the book. So I agree, to mix it up a little bit and show something real world is really interesting.

Julien Vehent:
I was just going to say, I like this Mike Tyson quote, “Everybody has a plan until they get punched in the face.”

Mike Pfeiffer:
Yeah. Yep.

Julien Vehent:
That’s basically incident response.

Mike Pfeiffer:
I love that one, man. That’s so true too, and that’s true with performance and availability and security. Nothing is 100% in this world. Stuff’s going to fail.

Julien Vehent:
And the things you’re going to prepare for are probably not going to be the things that are going to hit you, and that’s always … when people would talk about scalability, for example, it is something that I find so silly, because you’re going to deal with … you need to make proper engineering decisions from the get go, but the scalability problems you’re going to have are probably not the problems you’re identifying now. So the point is more need to be prepared. The same for security. You need to make sure that you have a good engineering organization, that you have good communication, that people work well together so when problems happen you can deal with them efficiently, and that’s more important than trying to map out every single scenario from day one.

Mike Pfeiffer:
Yeah. With some of those services as well, especially AWS, and Microsoft starting to do this now. I’m not sure … GCP probably the same way, but in certain places they’re starting to use machine learning to predict performance issues and predict other things. Is that something that’s happening in the security space and do you see that being a big focus maybe in the future?

Julien Vehent:
We’re playing with this a bit. I mean there are tons of startups that would tell you that they apply to official intelligence and machine learning to predict where you’re going to get hacked and most of it is snake oil. We’re investigating what we can do with it internally, and I’ll tell you, it’s hard. It’s really hard, because before you can even get to the machine learning part, you need to have really, really good data on what’s happening inside of your environments. You need to have that data standardized in formats you can manipulate easily. So that part … we get into the world of fraud detection and it’s really where most organizations break down is that they would have some logs but they won’t have the visibility across everything. It would be a very small part of it and they can’t even have visibility across everything because the volume is so high that they can’t collect it all, and even if you manage to get all of that data, categorizing it is incredibly hard.

Julien Vehent:
If you have a thousand hits per second on a web service, categorizing the 10 requests per second that are malicious is very, very hard to do. So I think we’re still at the stages where, again, simplicity is the best approach here, going back to the basics of flagging, using known bad actor and indicators of bad behavior without getting into the statistical artificial intelligence world will get you very, very far, and there is a point where if you can do the basics really, really well in fraud detection, by all means, hire a data scientist and have them look at your data, but also have a budget for it because that’s expensive to do and the results are uncertain. So I would say this is an area where as an industry we need to grow. I haven’t seen any out of the box compelling solutions yet. Everything that I’ve seen that’s mature and efficient comes from major corporations like Google and Facebook and Twitter that can afford to throw 10 or 20 engineers at a problem and have major resources, but smaller shops that have typically between three and five security engineers, they can’t afford to look at machine learning and artificial intelligence for these sort of things.

Mike Pfeiffer:
Yeah, it’s really good food for thought. I think the recurring message of stick to the fundamentals or the simplicity, embrace that, because to your point … and I agree 100% even in architecture projects, just not even thinking security only, sticking with the fundamentals and simplicity first, there’s a lot of value in that and I think that it gets skipped, and we’ve touched on that quite a bit, but for anybody that’s listening that’s an AWS type of focus person, they’re probably curious about specific technologies, and in the conversation of DevOps, we often talk about it’s the patterns and the practices and the people that are the important thing, but there are those of us that like to geek out on the tech. Any major services you’d like to call out in AWS that you think might be interesting for a security practitioner?

Julien Vehent:
There are a few open source projects that are interesting. So inside of AWS you can already spend a lot of time using things like [inaudible 00:34:37]. there’s a new service that they just launched that’s a detective that’s kind of like the machine learning aspect of things and it will grab all of your VPC flow logs and all of your [inaudible 00:34:49] data and try to find out bad patterns. So all of these things would already keep you busy for a while. Then there are a few open source projects that help. There is Pensar, and it used to be called something else … I forget. It came out of Airbnb a few years ago, I think it was Treme Alerts back then. It was initially designed to be an SIM, so security incident management system that was designed to help you detect malicious activity in AWS. We’ve built our own from scratch after several years of testing different things using Apache Beam, and we actually run that in GCP. We use data flow to execute that, and it’s a massive log event consumer that applies security heuristics to streams of data, but again, this is something that’s keeping two to three people busy full time, so not something I would recommend for a team that isn’t quite at that level of maturity yet.

Mike Pfeiffer:
Yeah. Not everybody’s Mozilla, so we’ve got to think about that a little bit too.

Julien Vehent:
And even us, we’re small. We still have to think about things in the smaller side of things. There’s definitely a train of people looking at what Google is doing in being like, “We can do that too.” No, you can’t. You absolutely can’t.

Mike Pfeiffer:
Knowing who you are as a team and an organization is super important, man. I agree. There’s so many of those situations, but as we’re kind of wrapping up this episode, obviously the book is a big focus. That’s going to be in the show notes and there’ll be a giveaway. So multiple lucky people will get a free copy of Julien’s book, but Julien, kind of as we’re wrapping it up, anywhere else we should send people outside of Manning and the book and all that kind of stuff?

Julien Vehent:
There are some excellent resources at OASP that always try to help with application security and they’re adding more and more cloud security content these days. So it’s always a good place to start. Some excellent conferences out there besides San Francisco has a lot of really good DevSecOps content. So it’s a good one to follow as well, and I personally like … I have a little DevSecOps community on Twitter that I like to follow, and everybody posts a link every once in a while, but also I’ll say some of the best security content I get just comes from my operations teams, because they’re security people are too. Their main job is to keep things secure and running well and et cetera, and they often come up with a new techniques that are very useful to security or new tooling that I wouldn’t look at because they’re not marked as security, necessarily, that really help us do our job. So stay close to your ops people or stay close to your devs. They are good resources too.

Mike Pfeiffer:
Great tip, talk to each other, actually, and share. Yeah, I love it. Awesome. The book is Securing DevOps: Security in the Cloud from Manning Publication. Julien Vehent, the author. Thank you so much. Appreciate it.

Julien Vehent:
Thanks for having me.

Mike Pfeiffer:
Hey everybody. If you want to keep up with what’s going on in the cloud, we have a weekly email newsletter called Cloud Skills Weekly, and you can subscribe for free by going to cloudskills.io/subscribe. Every single week I’ll send out my five best tips and resources that cover what’s going on in the cloud. We’ll focus on Microsoft Azure, Amazon Web Services, Google Cloud, and more. Topics will include things like cloud architecture, application developments, containerized applications, DevOps and automation, certification strategy, career tips, and more. So if that sounds awesome to you, head over to cloudskills.io/subscribe and join the Cloud Skills Weekly newsletter today.

Subscribe to the CloudSkills Weekly Newletter

Get exclusive access to special trainings, updates on industry trends, and tips on how to advance your career in the tech industry.