Episode 051: Azure Arc and Hybrid Cloud Management

Posted by Mike Pfeiffer on November, 27, 2019

In this episode I’ll chat with Michael Greene from Microsoft about Azure Arc, hybrid cloud, governance, end-to-end management, and more!

Don’t forget to subscribe to my mailing list at askmike.io/subscribe for weekly updates. If you want me to answer your question on the show, just reply to my emails and ask away!

Full Transcript:

Mike:
What’s up everybody, it’s Mike Pfeiffer, welcome back to another livestream. I’m here with Michael Greene from Microsoft. Michael, how’s it going?

Michael:
Hey, good afternoon. Things are going really well, how about you?

Mike:
Doing good. We’ve got a bunch of people on this livestream already, and for the folks that are out there listening, let us know in the comments if you can hear us okay. Make sure the audio is working. We had a little bit of a fiasco with the audio a couple of weeks back, so it was like the check in and see if that sounded good for folks.

Mike:
You sound great Michael. Michael, how’s it going?

Michael:
Things are going well, we just got back from Ignite last week, it was crazy. That’s probably one the of the best events that I’ve been to, maybe like back to the MMS days, the Microsoft Management Summit, and that feeling of tight-knit community and everybody trying to solve the same problems. It kind of had that feel to it.

Mike:
That one is Minnesota, right?

Michael:
There’s the Mid-West Management Summit that’s now community run, but I think the last Microsoft management summit may have been somewhere between '05 and '07, I [crosstalk 00:01:18].

Mike:
Okay.

Michael:
'04, '05, that was the last one, somewhere around that timeframe.

Mike:
Awesome, cool. So there’s a lot of people here, I just want to say hello to a few folks that they’re in the chat or in the comments [inaudible 00:01:29]. But John Carlos, it’s good to see you, Anna, good to see you. Philip, good to see you. My friend Jerry, what’s up, Joyce, thank you for being here. Duncan, good to see you, Abdul Carlton is back, [Ohits 00:01:42], thanks for letting us know about the audio everybody.

Mike:
All right, so let’s get into this. I’ve got some questions for Michael about Azure Arc and the whole hybrid management story. I know you guys have some questions too, so go ahead and add your questions into the comments. But probably the most natural place to start Michael is what the heck is this thing? What is Azure Arc, it’s brand new, right?

Michael:
Yeah. Arc has been, so Arc is A-R-C, and it’s not an acronym, so capital A, lower case R-C. And the whole concept is what if you could take things that are not Azure and project them into Azure. And maybe even to open this up a little bit more, probably most of your audience already knows what Azure is, but some might not. So this is the public cloud the Microsoft hosts, where you can publish your applications, run servers, store things, build virtual networks, things like that.

Michael:
And, for all of these different services, there’s, okay, I’m going to trust a public provide, in this case Microsoft, to run them for me. Then those patterns evolve and management techniques gets improved upon and there’s always this edge where new things have come out in the cloud that just aren’t yet available outside of that cloud environment.

Michael:
Across the last, oh gosh, five, six, seven years, we’ve seen these services come available and many of them had a hybrid capability. So you could say, I want to run something, I want to configure something, I want to inventory something. And they each had their own way where you could take a machine, put an agent on it, things like that, and connect it to that service, that one service.

Michael:
The difference for Arc is, let’s put one agent on your box, whether it’s Windows or Linux, and we actually don’t care where it’s at. It could be in your private data center, it could be AWS, it could be in Google Cloud, it could be Rackspace, it could be in a data center down the street that you got a relationship with a local provider, a [COLO 00:03:45] or something. It doesn’t matter. And it also by the way, doesn’t matter if it’s a virtual machine, it doesn’t matter if it’s physical. So anywhere you’ve got Windows or Linux, you can put this agent on it, and then now it can talk to the cloud.

Michael:
Now, when you go look in the cloud, you’re going to see this device show up in a subscription, it’s going to be in a resource script. So now you can sort of group these things in a logical manner. It’s going to have RBAC, you can control who can do things with it. It has tags that are free form, so if you want to create relationships and say these things are related to a data service and these things are related to identity management or whatever, it has all these different metadata, and all of the services that are native in these cloud environments, just work.

Michael:
So for us, the hero story of this going into Ignite was a solution that our team runs called Azure Policy Guest Configuration. We’ll go into more depth in that, but that’s where we’re getting started with Arc. And actually, I’ve been around this space for a long time and you have as well, if you think back to the system center days, there was always, the question is, think back to the system center days, because there’s billions of machines that are being managed by system center right now. But for the entire duration of system center, people have said, “Isn’t there some way we can just have an agent of agents? Can we just have one thing that we install?”

Mike:
One agent to rule them all.

Michael:
Exactly. And that’s the kind of, one of the hypothesis here is it works that way in the cloud, so why can’t we do that outside the cloud too.

Mike:
Yeah, it’s exciting because it’s one of those things that seems like everybody, to your point there, people have always been trying to figure out management and distributed systems and how are we going to do this. One of the questions or comments that came up in the chat was, I’m interested to see how I’ll be able to monitor AWS with this, and it sounds like it just doesn’t matter. If it’s an [inaudible 00:05:46] machine, you onboard it, just you would a physical server or anything else, right?

Michael:
That’s right, it could be anywhere as long as it can talk to the services that are hosted in Azure. One of our hard requirements going into this was support for HTTP proxy. So, whether you’re in a private network in AWS, or if you’re in a private data center and your security team says, our servers absolutely cannot have direct outbound internet access. I could talk all day about this type of [crosstalk 00:06:20], but that is a common scenario. So we can route through an HTTP proxy, you can expect the traffic, you can control …

Michael:
We actually publish a smaller range of IP addresses that you would need to allow traffic too. So it’s not even saying all of Azure, it’s saying specifically to our service, which is pretty neat. That’s a big change from the past.

Mike:
That’s awesome. So basically, big takeaway there is, you can, there’s nothing stopping you as long as you’ve got an outbound proxy. So internet access of some kind, it doesn’t have to be direct outbound.

Michael:
Yeah-

Mike:
… Other services that worked that way too, right? Like you guys have an Azure 80 application proxy, which is kind of similar and different service, but that’s … I didn’t mean to cut you off there.

Michael:
No, no, I was going to say it’s insightful that they mention monitoring. The first service we brought up was Guest Config, which primarily, and we’ll talk more about this, but primarily has been focused on governance and auditing and things like that. But our next engineering deliverable as a team is to bring extension management for Arc.

Michael:
So when you’re in Azure and you say, “I want to perform monitoring operations inside this VM,” and that loads up an extension for monitoring, for log analytics for that machine, and Azure Monitor becomes available and you can start digesting the data and looking at it, et cetera. We’re going to follow exactly that same model. So, whenever you have a machine, let’s say a non-Azure server or Azure Arc server showing up in the portal, you’ll be able to enable these different solutions and it will work identically to how it works for cloud hosted services. Except for once those extensions show up in ARM, that sort of master agent if you want to think about it that way on that machine, will take care of downloading what it needs to be successful, getting the services started up and managing the whole thing.

Mike:
Awesome.

Michael:
It’s really neat, I’m pretty excited about the whole situation.

Mike:
I was talking to some other of the MVPs when were at Ignite last week, and I’m starting to get that old feeling again when it comes to Microsoft, like there’s a tipping point. I remember back in the old days when all my buddies were, the guys that were ahead of me in IT that were like, hey come work on these network servers with us and do all this stuff. And I was like I don’t know man, I think the Microsoft thing is looking cool. So I got into Active Directory and obviously that was a game changer because of how deep you could go with the policy, it was similar but you had much more capabilities with policies. And the management experience was the game changer and I’m starting to feel like that’s going to be history repeating itself one more time.

Michael:
Do you mind if we dig into that one, the example that you just gave?

Mike:
Yeah, absolutely.

Michael:
All right, so let’s rewind. 1999 NT4 is popular, people are running NetWare, we’re thinking about NT5. This is before I worked at Microsoft. I actually presented on this topic last week, and I tried, legitimately tried to go find on YouTube original video contents from when Active Directory and Group Policy were introduced. The timeframe you’re talking about. Spoiler alert, there was no YouTube in 1999. Whatever the leadership from Windows server went to COMDEX and presented. It wasn’t captured on video and put on YouTube.

Michael:
There’s a lot you can go and look at, but I did find Group Policy as part of Active Directory, released to manufacturing, which meant they’re done with the work. Remember at that point in time, it was like we’re going to stamp the CDs and hand them off to people who make servers, so that they can start giving this to customers six months from now, or three months from now. That was next month, 20 years ago. December 1999, that’s how long we’ve been at this. [crosstalk 00:10:19]. So the marketing bullet point was, NT5 is going to internet enable your business.

Mike:
I remember that.

Michael:
Check. That one got done. Business has been internet enabled. So the reason I bring this up, we’ve been working on Guest Configuration, which is, if you’ve been following the evolution over the past few years, there’s been PowerShell DSC, and the whole notion of deploying, configuration as code to Windows servers. We made that a richer experience through Azure. And then we’ve gone through this transition of okay, how else can we use this same technology and how do we take this big list of UserVoice items around this tool and how are we going to address them? A lot of them require us to basically reboot the whole thing and take some different ideas.

Michael:
So we have gone through several cycles now, but we’ve completely re-written that whole DSC concept in native code with the idea that being something that was designed for cloud scenarios. Originally the goal was to bring Windows server to the cloud and Windows server has all these different APIs, whether you’re calling the registry, or file system or WMI or Active Directory or IS, they’re all very different. So DSC standardized all of that.

Michael:
One of the first scenarios that we brought this new DSC platform into is governance. Can we, as servers are deployed, can we just do a read-only operation from a security team or an operations team, who’s in the admins group? What’s your password policy? What applications are installed? And we’re all using DSC stuff under the hood. That’s been one of our first scenarios for Arc.

Michael:
We’ve been working the last three months at least, on trying to solve a problem that relates to what you’ve just brought up. All of these tools and configuration as code, [inaudible 00:12:21] CF Engine, DSC, you name it, if you’re trying to manage those original Group Policy settings, it’s not as simple as it looks. You can go set the value, but you’re not really integrated with Group Policy, and one of the original presentations on DSC, Jeffry Snover was presenting. And somebody raised their hand and said, “Okay, if I apply the same setting with DSC that’s being managed by Group Policy, what’s going to happen?” And his response was, “Well, it’s going to fight like two raccoons in a bag.”

Michael:
And so, that’s been at the back of my mind because this is a common thing. If you look at the industry standards for security, CIS dig, NIST ISO, TCI, all of those different things, they’re all published as Group Policy examples. So one of the things that we’re working on and this will plug into Arc for hybrid, this will plug into Azure, is can we natively integrate with Group Policy? And so we’re going to take that 20 year technology, which is a skillset that probably everybody whose listening has, and breathe new life into it, and make it like a first class experience in Azure.

Mike:
That makes a lot of sense because there’s so much investment out there already and when you’re talking hybrid, we’re really talking about maybe we can’t go to the cloud for five years, and that’s okay, for certain things. So we’ve got some servers over here that we can’t lift and shift. There’s been some comments in this post here, kind of like around, what about the container and I know there’s data services support and I think that people weren’t even sure we were even talking about Arc there just for a second. So, policy is actually part of this, right and being able to see pretty much single pane of glass for all your servers and know whether they’re compliant with your policies or not. That’s part of thing.

Mike:
But you guys do, are going to support Kubernetes clusters anywhere, as well as have some data services as well, right?

Michael:
Exactly. So the other side of Arc, it’s sort of Arc for servers and then Arc data services, and that’s where you’re plugging in a Kubernetes cluster that’s hosted outside of Azure and kind of the same principles we’ve been talking about just now for servers, would apply there. In fact, they go one step further where you can even do provisioning from Azure, once you’ve got that Kubernetes cluster enabled. So if you need a new SQL cluster and it’s going to be on-prem instead of in Azure, then you’ll be able to do that for ARM, which is pretty neat.

Michael:
Next week I’m going out to the Kube conference and the deeper focus there would be on the Kubernetes side.

Mike:
Oh wow. So that’s something I know everything is in preview and you’ve got to register the providers manually, and kind of get going. Has the Kubernetes stuff lite up for customers at this point of the preview?

Michael:
No, that’s a preview as well, yeah.

Mike:
That’s awesome. So Kube Con next week or Kube Con, that will be really fun, too bad I can’t make it to that.

Michael:
I’m excited. Linux Foundation puts on great content. They put on great conferences, so I’m looking forward to this a lot.

Mike:
So, now we kind of understand what the hybrid approach with Azure Arc, some of that. And obviously one of the big things that people are struggling with a lot of times is governance. Figuring out how to manage stuff, how to make sure it’s secure. So you mentioned, we can use RBAC with these hybrid resources that we’re projecting to Azure Arc that we already use with our native Azure resources, but maybe we could dive a little bit deeper into the governance story, there’s been a couple of comments about that.

Michael:
Yeah -

Mike:
… Something we should be kind of aware of around that concept.

Michael:
And by the way, feel free to jump in if I get into a rumble and there’s good questions from the audience. I want to make sure that we’re taking them as they come.

Michael:
This is a re-occurring pattern that I’ve seen. It actually doesn’t matter how big the company is either, this is the same concern whether it’s a company of tens of thousands of people, or five people, it’s when I put my stuff in the cloud, then what? How do I know it can be trusted? There’s almost this feeling that if I can walk over to a server and put my hand on it, then worst case scenario, I can pull the network cable or something and how am I even going to know what’s going on.

Michael:
To reinforce this, whenever the concept of creating servers in the cloud was new, I’m just as guilty of this as anybody else. You go provision a Windows box, what’s the first thing you do? You remote desktop into it, and you click the start menu. Holy cow, this is real, this is a server, this is a thing. If you take this, fast forward from that point in time to now, and think about what these large organizations are actually doing with the cloud, they’re not just creating one subscription and one virtual machine and standing up services on that, they’re dealing with the scale units of hundreds of application teams, and if each of those applications teams is getting their own subscription and they’re each standing tens or hundreds of servers. Somewhere, there is a security team, probably a manager of that team who is on the hook to say, our organization is meeting the regulatory requirements that are required for our business.

Michael:
If out of 1000 application teams, let’s say 600 of them are dealing with conducting credit card transactions. Are they all meeting CIS requirements for their servers and things like that. And if not, he’s sort of on the hook, or he or she is on the hook to make that real.

Michael:
What we need was a way that as servers are being provisioned, we automatically could have someone look across that huge span and just do rational checks. This isn’t trying to impose things, or slow down application developers, it’s actually the opposite. It’s saying, let’s just be able to do a quick scan as part of that application release cycle that says are the requirements met or not?

Michael:
This actually plugs into Azure devops, so if you’re an application … From the security and compliance team, it’s like data, so we just need a way to conduct the scan, get the information, bring it back centrally, look for the outliers and then we go have the hard conversations. For the application owners it’s, things like that Group Policy thing we were talking about a second ago, that whole environment, not particularly suited for moving fast. You deploy the machine and you’re waiting for domain things to happen and that kind of stuff. So this idea was, let’s build something that as soon as the machine is deployed, you can conduct and audit, see the results.

Michael:
And in Azure devops, that’s just one of the release gates, so you’ve kind of got that compliance view centrally, and then from the individual team’s perspective, I’m going to be working on an application, I’ll use the tools of my choice to make this machine look the way it’s supposed to, and then as a release gate, I’ll deploy to a resource group for test, I’ll run all my unit test, integration test, that all looks good. And if I’ve checked that extra box, it’s going to say what was the result from Azure policy, does everything look good? In other words, maybe I needed to install an update to .NET Core as part of this change to my policy. Maybe I didn’t realize that with that update, there’s some special registry key that’s supposed to be set, that actually disables something, make sure I’m not exposed to a security vulnerability, we’ve all kind of been through this process. That would just be part of your release gate, that, that is checked.

Michael:
Normally what happens is you release to production and then you find out later that you screwed up and the security team has to come and talk to you and say, how soon can fix this, and it’s like, with or without an outage. So the whole ideas here is to shift left and find it earlier.

Mike:
Yeah, and that’s a really cool feature, and folks who are listening or watching haven’t gotten into Azure devops and building on pipelines and stuff, that concept of a release gate is a really cool thing, because it’s like you could check the policy compliance to your point or if you even look at Azure monitor alerts now. As you guys are servicing more from these hybrid resources, I’m sure we’ll be able to look at that stuff too. So, a really interesting time to be working on all this stuff.

Mike:
One of the comments that came in, which is a good, and I want to circle back to extensions, virtual machine extensions, I’ve got to remind myself to do that, I have a question there. But one of the questions in here is, what’s the difference and advantages between Azure Stack and Arc, or are they related, or what’s the story there?

Michael:
That’s a really, really good question. Not everybody is going to know what Azure for Stack is, so I’ll explain what that is first, and then contrast the two.

Michael:
Azure Stack is the idea of buying hardware that has been tested and is specifically built for this scenario of Azure Stack. And within that, you’re getting a local instance of ARM. So just like you may have used the Azure PowerShell commands, or ACC [inaudible 00:21:24], maybe you’ve gone to the portal.azure.com, if you’ve got an Arc instance, you’re going to your local instance of portal.azure.com, and clicking, I want to deploy a VM and that’s getting deployed on your hardware.

Michael:
So the main difference is, that control plain as we call it, which is the STK level of I make my request and then there’s an engine that goes and spin things up for me and then does all the CRUD operations. That’s being hosted locally in your data center, and it can be completely offline. There’s the submarine scenario that we talk about where it’s not internet connected and things like that, and depending on your identity situation, you may have different flavors of whether it’s connected or not. But that’s the idea, it’s your instance of that cloud.

Michael:
Arc by contrast is taking servers from your private data center and projecting them into the public cloud. So even though they’re still running locally, you’re not hosting ARM locally within your data center. You’re working with that public endpoint and you’re just surfacing into that public environment, records that are associated with things happening in your data center.

Michael:
And the way I like to think about this playing out longterm, if you really wanted to fast forward this and think about what happens, of course now you can audit, but as we get into you can monitor, you can inventory, you can configure, you can deploy things like that. You’re really getting to the point where right now if you go into Azure, there is locations like West US and East US and North Europe and all these different locations. Your data center kind of becomes an Azure location, where you’ve got a bunch of servers and the location string is whatever you want to set it to. So if it’s my data center, Chicago East, whatever, that’s your instance that you’re managing, which is following a lot of the same capabilities as an Azure data center, which is pretty neat.

Mike:
That is really cool. I’m super excited to spend more time on it. To circle back to the thing that came up earlier, I’m a huge fan of all the virtual machine extensions in Azure, like DCS extension, custom script extension where you could do all kinds of cool automation, especially good when you’re bringing servers up and, but even just for ad hoc administration and all that kind of stuff, do we get that with those hybrid kind of resources as well?

Michael:
That is the next thing that we plan to deliver in this space. Actually it will work exactly the same way. If you look at this in a deployment template where you’re working with a JSON file and creating a server, and you’re just adding a section underneath the server that says, I want DSC extension or I want log analytics or shelf extension, third party and things like that, it will work the same way. So you’ll be able to from ARM say, I want to light up these extensions and then the agent will just take care of it, you won’t have to go do anything special on the servers that you’re managing.

Mike:
Nice. And then Peter was saying in the comments, and I think we covered this, but just to make sure he’s saying, “Hey Michael and Mike, reading a lot about policy management, compliance in Arc,” and he’s like, “Can I assume that the Arc agent will also integrate seamlessly into Azure Monitor,” or Azure Monitor and you mentioned that you guys are going bring that next, right? Kind of the next big thing?

Michael:
But he brings up a really good point. All of this stuff goes with the activity log as well, when you’re talking about monitoring. So, from one place you’ll be able to see, I’m in and out of compliance or monitoring picked up on something, and all these different capabilities from one stream in an activity log, which is pretty neat.

Mike:
That’s what I like about Azure policy is, not only seeing stuff out of compliance, but I’ve got the option to remediate, and I’m not sure if that’s every single policy that you guys do. I know that there’s still a lot of moving parts here and you guys are changing stuff and adding features and stuff-

Michael:
I’m excited that you brought that up, that was … If you look at the timeline leading up to Ignite, the Wednesday before, we published our first policy where Guest Configuration can push to a machine. So we’ve been doing all this work around audit, and there’s always been this promise that we would bring DSC forward and it would just be first class citizen in ARM. It wouldn’t require an automation account, extensions are great, but they’re limited in reporting. It’s great for execution but limited, there’s not a central place that you can go at scale and say, across these 5000 machines, what’s the status.

Michael:
So Guest Configuration provides that capability and we can talk about the nitty gritty of that if you’d like to. But our latest policy is named Configure The Timezone on Windows Machines. And we picked that for a specific reason. Setting the timezone on a server is not particularly impactful and we needed something we could bring out as our first one to say, we’re going to give the opportunity to make changes at scale using policy, let’s get a lot of feedback to make sure to make sure we got this right before we go big with, I’m going to stop and start services or add and remove files, you name it.

Michael:
Our first one is really focused on just getting feedback on the experience, and so far its been really good. We’ve got some work to do, but-

Mike:
You kind of got to do that, right? You kind of have to get some points on the board. It’s like our customers that try to do the migration on a first try. You’re trying to hit a home run in your first hit bat in the major league, like slow down a little bit.

Michael:
It’s actually part of moving fast. People think to move fast is, let me go introduce 50 of these things and it’s like, get that first one out, get feedback, because if you’re going to make a mistake, make a small mistake that you can recover from within a few days. If you make a big mistake and it’s times 50, you’ve got a lot of work to do some unwinding before you can go on to phase two. That’s part of the cloud cadence.

Mike:
It’s a good way to go. I’m excited to see what happens next. One of the things that I was thinking when you were talking about that, the remediation and stuff. I started thinking, it’s a lot in a way, onboarding nodes into Azure automation and having kind of the whole config management solution in place, where the nodes are always checking in, am I compliant, am I compliant, no, fix it. Are we going to see that pattern move over into this stuff?

Michael:
We’ve got a really good approach here that I think we’ve got right, and I’m happy about it. I think when people see this [inaudible 00:28:07], in this case, Guest Configuration has an overlapping feature set with Azure automation DSC. The similar thing is happening for run books. People are seeing that Azure functions is building an overlapping feature set with Azure automation run books.

Michael:
And then the question comes, are you going to pull the rug out from underneath me next week when I log in and find out that I just can’t use that service anymore? And the answer is, absolutely not. At least that’s not the plan right now. But, we’ve made a very conscious decision, we’ve actually spun up a team that’s focused on continuing that capability forward. So Azure automation will live on, will continue with run books and DSC within that capability.

Michael:
Traditionally whenever we say like something goes into some sort of maintenance mode, that’s not what we’re doing in this case. We’re having a team take it on, just looking at the UserVoice stuff, and saying, more than just keeping the lights on, what are the top requests? What naturally happens is you’ve got somebody like our team that’s been working on DSC, and guest config comes in and you take on governance, and you still have the same number of people, so nobody puts on more hours in the day. So you need that additional teamwork coming in and saying, if you really want to continue to address what your customers want, you’ve got to throw some more people at it, there’s no way to create more time, somebody start going and addressing these UserVoice concerns and things like that. So that’s what we’re doing.

Michael:
You’re going to end up with this nice situation where new and better things are coming out all the time and when you’re ready, you can make the decision on your own, do I keep using, can I stay on the Azure automation track, maybe there’s some things there that keep me interested. Or is there a business requirement that I need to meet for my organization that is interesting over on Guest Configuration as part of ARM, or in functions, and if so, then when you’re ready, you can take a look at those new features when it makes sense for you and bring that into a new solution.

Michael:
So I like that a lot better than us setting a deadline and saying like, January 15th, you have to do … No, we’re not going to do that.

Mike:
I love deadlines, and then passing them by [crosstalk 00:30:29] and kind of circling back later. Ethan in the comments was saying, “The publishing example mentioned, was it talking about public cloud? Is there a difference when publishing to private cloud?” So it kind of takes us back into Azure Stack, right?

Michael:
That’s interesting. We haven’t yet brought Arc services or even Azure policy and Guest Configuration into Azure Stack. It’s not to say that we won’t, it’s not live right now. So I guess theoretically speaking, if Arc services were available for Azure Stack, what that would look like is the ability to take that agent and put it on VMs that are not part of your Azure Stack environment and have them show up there.

Michael:
Technically, that seems like something that we could enable, but it hasn’t happened as part of this initial preview. What happens for stuff like this by the way is, it’s up to customers. So we listen, as being a program manager at Microsoft, that’s a huge part of your job. Just keep track of what’s going on, kind of go get the pulse of what people are looking at, and what their reaction is and then help steer that next planning cycle.

Mike:
I actually wanted to highlight UserVoice because we threw that out a couple of times and there maybe a lot of people that never even heard of that. Obviously that’s a big part of how you navigate all this. So people are like, okay, I want to suggest something, that’s what UserVoice is about, right?

Michael:
Yes, that’s the best place because in UserVoice, we’ll see that accumulation of interest. My Twitter is migreene, you can tweet at me and say, I don’t like this or I love that. And that goes into my own anecdotal data collection and whenever the planning cycles come up, I may even refer to those tweets and things like that.

Michael:
Similarly, for people that I’m working with, they’ll email comments to me and then throughout the year we’ll go onsite with customers and things like that. But if you think about it, all of that individual interaction, you can only reach X number of people and there’s thousands and thousands of opinions out there and many people probably don’t even have time to directly engage with this if the opportunity was there.

Michael:
So if you search for Azure UserVoice, honestly that would be the easiest way to find it, and under governance, you’ll find Azure policy and you’ll see a list there of all the things that people have kind of looked at as gaps in the service and there might be some things where go, oh, I didn’t even think about yet, but this request would be pretty neat and it hasn’t even occurred to me yet and you can vote for it. And then we’ll do our best to go through, like for Azure automation, I’ve done this for a couple of years now, and we have internal processes that hold us accountable for this. But we’ll go through and as we go on each planning cycle, we’ll make sure that we’ve included this in the data we use to make decisions or try to respond to them and at least let people know that they’ve been heard and whether or not that feature is planned or is going on the backlog for now and so forth.

Michael:
It’s just a way to organize, and if anybody who’s listening works on a software project, UserVoice is actually available, I’ll say it’s not strictly a Microsoft solution. Anybody can go and create a UserVoice for their project.

Mike:
Nice, all right cool. We had a couple of questions about the Kubernetes support in Azure Arc. Honestly I haven’t looked at it either-

Michael:
We’re going to stretch my skillset a little bit. Let’s go for it.

Mike:
I guess my biggest question is, am I just using Arc to manage my Kubernetes infrastructure, it’s not like a service mash, where I’ve got, or is it? Maybe you could [crosstalk 00:34:14] a little bit.

Michael:
Yeah, I understand some pieces of this. So you’re right, it’s looking at, and this is actually a huge, my understanding is that this is sort of an evolving story or gap, if you will, in that whole space, which is, let’s say you go spin up Kubernetes environment and you’re managing it and you’re happy with it, and that’s great. And then you decide for your business, it makes sense to have a Kubernetes cluster at each location and maybe you’re a retail business and you’ve got 500 stores around the globe. And then the question becomes, how am I going to manage this at scale? Probably the person at the retail location isn’t going to be a Kubernetes expert, so how do I deal with that?

Michael:
What we’re trying to answer here is, provide options, like how do I deploy across those, how do I keep it up to date, how do I manage underlying infrastructure, how do I govern the compliance of that hosting environment? It’s all running on Linux, and if an update comes out for [inaudible 00:35:16] or something, where it needs to be updated, how are you going to manage that at those 500 stores if you’ve got a three node cluster at each of them and on and on and on. That’s kind of what we’re thinking about in this space.

Mike:
Got it. So it’s more management, nothing application specific, really like inside, like connecting services. That makes a lot of sense. Ahmed was asking, with Azure Arc, are you able to deploy Kubernetes, and it sounds like you’re just onboarding existing clusters that you’ve already got, right?

Michael:
Right. And then be able to deploy data services at that point. So if you want to spin up a container out of these remote locations, be able to manage that centrally.

Mike:
Did we get much, and I don’t know if we’ve talked a lot about that, the onboarding process for bringing servers into Azure Arc. It’s not super complicated because you just didn’t [crosstalk 00:36:03]-

Michael:
… Complicated. This is one of the things I actually love about Azure Arc, is that it’s so easy to onboard. I’ve done some customer POCs, and actually last night I was talking to a customer that I keep in close contact with, and I said, “Do you guys want to spend a few hours tomorrow and we’ll stand up Arc?” And they’re like, “Yeah, we looked at the docs, this couldn’t be easier. We don’t need to spin up a team’s meeting to go do this.” So it’s literally for Windows it’s an MSI, and Linux, it’s an after [inaudible 00:36:33] package, so you just add the package source and install it.

Michael:
The quickest path here is go into the Azure portal, navigate all services, search for Arc, it will bubble the top, you click on it, then just click the plus sign for add. And it’s going to give you a wizard that generates a PowerShell script for windows, or a [inaudible 00:36:56] script for Linux. And that PowerShell script is three lines long and it’s in [inaudible 00:37:02] web request, where it goes and gets the agent and then install the agents and then there’s actually a binary, it’s part of the install that you run and pass the command line argument so that it could be registered.

Michael:
Now, the most interesting part of this, especially for service providers is the identity management side of this, like who can register, what are the permissions that are needed, things like that. If you just run the commands in a test VM because you’re just experimenting, then it does the whole thing that you see in PowerShell core and other [ACCLI 00:37:35] environments where it says, go to Microsoft.com/device login and type in this code. And then we’ll get the token and then you’re good. But if you’re onboarding, what is you want to put in your image and spin up automatically connects whenever I create the image?

Michael:
In that case, you go create new app ID, just as like you were going to register an application in Azure. And so it’s almost a stored username and password that’s not associated to a user, it’s associated with an application. And in this case the application is just registering Arc servers, and we have a security rule that has no other capabilities than registering servers to Arc. So yes, you should still protect that username and password, but if it does get stored as a variable somewhere within your process for server deployment. Even if somebody gets it, you’re pretty well protected, and even if somebody gets access to it, they’re not going to get very far.

Michael:
So now you can just take this agent, put it in your image or put it as part of your automation process if you’re building new machines, you take that app ID in secret, in a way that it can be provided to that command line at runtime and that’s it, you’re good.

Michael:
And then I’ll tell you the number one thing that I heard at the Ignite booth for Arc is, once that’s done, if I go to the Azure portal, can I create servers in my data center from the Azure portal? And then answer is, not right now. So, to be honest, the amount of feedback we got from the Ignite booth, we’re going into, we kind of run six month planning cycles and then many sprints within those. Going into this planning cycle, that feedback from Ignite, just hits hot and heavy. So it comes back and says, if that’s what people want, then we should figure out if that’s something we can do.

Mike:
[crosstalk 00:39:32] so in tune with what people are looking for, that makes a lot of sense, that’s cool that you guys are listening. And so for everybody that’s out there, that’s listening to this, make sure you’re submitting your feedback to UserVoice and uploading the stuff you think Microsoft should be working on.

Mike:
Greg had a good comment, or Gregory had a good comment in the livestream here. He was saying, “What is the time that I shouldn’t use Arc? What’s a not good time?” Is there any kind of, what you’ve seen talking to customers, miscategorization or people thinking it’s going to be used for one thing and it’s totally not, or is it just so early and it’s the wild west right now?

Michael:
The inside patterns. First of all, I haven’t had any hard no, answers yet. Just hard no, don’t do that. One would be total isolation. So if you are on an Uber high security network, where you’re air gaped and people have to sign in and out before they can get to a terminal to reach these servers, things like that, then you can’t talk to Azure, and it’s just not an Arc scenario. And that’s where things like Azure Stack would probably make more sense.

Michael:
Otherwise, I haven’t run into any scenarios yet where Arc is not a great fit. I’ll tell you, from my own environment, whenever I’ve been testing just iterations of the Arc agents, something that you might want to just sort of put on your radar as you’re thinking about deploying at scale and this is a feature request that we’re looking at. So right now I use Vagrant, I can go into details about this too, but I’ve been carrying a MacBook so I’m keeping our team honest and that we’re being cross platform, so everything we do, I test it on our machine to make sure … So as I’m testing in Linux and Windows, I’ve been using Vagrant to spin up machines on virtual box. Every time I run a test, it creates a new registered machine showing up in Azure.

Michael:
And so, after a while I sort of get this scroll of stale records of machines, because these machines lasted for a second. A test round and then it deleted it and move on. So I’ve just written a quick Azure function that goes and looks at the last modified timestamp of that resource and if it’s more than X number of days, then I tomb stone it out. That’s probably something that we’ll take on before too long, so that’s sort of a natural evolution, most likely before we even get out of preview, we’ll have something like that, like a tombstone process. I would say that’s going to be something to look out for, otherwise, I really haven’t hit major hick-ups yet.

Mike:
A couple of other comments came in as well, Peter had a good one here. He was saying in a scenario where you’re managing Kubernetes clusters with Arc, you still manage all the low level stuff like deployment and all that kind of stuff through Kubernetes, or can you do it through Arc?

Michael:
My understanding-

Mike:
… All that kind of stuff.

Michael:
My understanding, and I’m still building up my own expertise on my Kubernetes side, is that you should be able to do both, but I should probably reserve that to the experts on the Kubernetes side. My understanding based on the booth conversations that I overheard, is that you could do it from Azure, and sort of like a management tier for that Kubernetes environment. So if you’re doing it locally, then hopefully it’s recognized there. But I should probably reserve that to the real experts.

Michael:
It’s a great question. Hopefully next week, I’ll find out.

Mike:
Yeah, I know. I wish I was going to San Diego next week, but [crosstalk 00:43:08] warm place, I’ll be in Vegas next week, so I can’t complain too much.

Mike:
Cool. So obviously you guys are working on a ton of stuff with Azure policy, that’s one thing that I think a lot of people, especially our customers should be paying more attention to. What are a couple of things that you would recommend for people to start diving into that, because once you understand that, then it’s like I understand policy for everything I have, whether its virtual machines Azure, virtual machines over here on my own data center or physical servers, all that kind of stuff.

Michael:
Let’s explain how to get started, and then let’s paint the picture a year from now into how this could all evolve to sort of light the fire and maybe people will be interested in that getting started phase.

Michael:
If I was looking at this for the first time, I would just go into policy first and I would, and if you look at, click on definitions and then you’re going to get a flood of available built in definitions. But there’s filters across the top. So I would change my type to initiative, and then I would click on the category filter, uncheck select all and check Guest Configuration, and that’s going to give you this nice clean list of things that I can go audit, plus our one configure policy.

Michael:
I would pick sort of an easy to follow audit path, and the reason I say that, you’re going to see right away that one of these initiatives has 58 definitions, and that is aligned with, we call it Azure best practices for Windows, but it’s aligned with what security center looks for, for secure Windows, machine. And it’s basically all Group Policy setting. And it’s a pretty slick audit, you can actually get some good flexibility through parameters and we could go on and on with that one.

Michael:
There’s also just dead simple scenarios. Most of our policy content has been created based on what our customers are asking us for, and there’s some pretty neat ones. There’s the obvious stuff like pending reboot, like I applied patches, but the machine didn’t get rebooted, just go check for that. And if it shows [inaudible 00:45:16], and people think about that because if they hear governance and they think security, like a security scanner. There’s tons of operational governance opportunity here. We have one that’s based on customer feedback, that will go look for certificates on a box that are going to expire within a time range, because they would have websites that were hosted IIS, the SSL certificate expires and they have an outage. Or if it’s not an outage, it just shows up like, you shouldn’t trust this site, that kind of stuff [crosstalk 00:45:48]-

Mike:
… Operate.

Michael:
Yeah. It’s not really something that you monitor for, it’s not an application error, so how the heck do you keep on the look out? So we’ve got stuff like that. We have one that is, go check for machines that have not been rebooted within a timeline. And customers asked for that, and I said, why do you want to look for this? And they said, chances are, that’s a rogue server, somebody created this VM, they ran some tests or something, and they forgot about. They did it on Friday, they came back on Monday, and they didn’t think about it. Now its been out there running for three months, nobody has patched it, nobody has even checked on it, we’re paying for this thing, how do we find them? That’s a good example of policy.

Michael:
That’s where I would get started. Some of these things don’t even require parameters, you just assign it, and then kind of look at your compliance results.

Mike:
What’s funny about what you’ve just said is, that’s like one of the first things I always check for when we go to a customer, is nine times out of 10, they’ve got stuff running and they don’t even realize that it’s there. Keep on asking around and you get to the end of the day and you realize that it’s something somebody spun up and forgot about.

Michael:
I think there’s a fair amount of that. This is actually an area we’re thinking about digging into. There’s old horror stories about this, remember SCCM had network discovery capability and there was all kinds of cases of people with servers, it’s under somebody’s desk or, I heard one story where maintenance had come and taken a door off a closet and they just dry walled it shut, and they didn’t even care what was in there, they just considered it to be lost and garbage, and the server was still running and nobody could find it. There’s all these weird stories.

Mike:
Uptime is [crosstalk 00:47:26], I used to get that a lot back in the very early days of Powershell, we were running scripts and stuff, and people are, hey, how do we query the uptime in the server and all that kind of stuff. I love that all that stuff showing up in policy because it reminds of, going back to the beginning of our conversation, it reminds me of when I first got into IT and realized, whoa, look at all the stuff we could do with Group Policy.

Mike:
So I think that with Arc, what you guys are doing with hybrid and governance and all that is going to be interesting. I can’t wait to see how you guys continue to innovate and I’m glad to hear that there’s so much connectivity to what people are asking for and all that kind of stuff. I just wanted to mention a couple of comments that came in real quick. There was a question or a comment earlier about managing Kubernetes and can you work with open shift and kind of deal with that. And it sounds like it needs to be Kubernetes, right? As long as it’s Kubernetes proper, whether it’s managed or not, I don’t know the back story about it-

Michael:
It’s my understanding as well.

Mike:
Yeah. And then one of the other things that came up was, AJ was asking, when it comes to a policy, it gets a list of all the apps and the certification. So, I guess he’s asking, what can I do with policy. So the one you were talking about was certificates being expired, that would be one example of a policy. Then maybe what’s installed, would be something else-

Michael:
And for a lot of these we can do that positive and the negative case. So you could say, go look for an application that I don’t want in my data center like Mimikatz or something. And it’s going to check for a Linux package or, it’s not scanning for binaries, it’s just checking to see what Windows is aware of. Then we have the reverse of that, which is required, I expect these applications to be there, and typically for that scenario we find its agents. So people have said, for every machine I spin up, I expect maybe Splunk or McAfee anti-virus or, name your favorite tool that you want to be there on all your servers.

Michael:
You could take either approach, you could say, red flag this machine that it’s gone live but it doesn’t have our expected footprint for applications. Or you could start tying this to things like activity log, where it says I actually want to connect this to some sort of an ITSM solution and go create a service ticket at my help desk and create a work item for the server owner and things like that. So your integration is pretty deep.

Mike:
Cool. So a couple of the comments that have come through is like, let’s get a demo. We’ll definitely do demos for you guys at some point. One of the things that I will do with livestream here later on down the road, is we will do more tactical stuff. We’re kind of circling back to Ignite, what happened last week in introducing these and then we’ll dive in, especially as Microsoft continues to innovate.

Mike:
I know that there was a couple of sessions last week at Ignite, there was replays. I also talked last week to Thomas [Mowder 00:50:20] from Microsoft on the cloud advocate team. On his blog he’s got a cool video there and a demo. Michael, what other, I’m sure there are some sessions that you were probably connected to last week, that [crosstalk 00:50:32] watch and see demos-

Michael:
There were a couple of sessions on Arc, it sounded like the response to, there’s a session called PowerShell Seven, but that was Geoffrey and Joey and Jason, just going through all the new capabilities of PowerShell. So my understanding is that was a full room with the weight, but it’s recorded and that is one of the better ones to go watch. Of course the [inaudible 00:50:59] sessions are always super popular as well, and my session was called The Evolution of Group Policy. We have probably three sessions just generally went across server management from the cloud. So we went through, if you want to inventory servers, if you want to just go, do the full monitory garment, if you want to do patch management and all that kind of stuff. What are all the capabilities that are now available. If you look across server management, you’d be pretty happy with those two.

Mike:
Awesome. All right, so I think [Marcey 00:51:33], hopefully I’m saying your name right, I think he had a very good comment to kind of end on and wrap up here. I appreciate your time Michael. So in terms of cost, that’s probably one that people are asking, how do we deal with cost in Azure Arc?

Michael:
We don’t know yet. This was like our number one booth question. From us to customers, as opposed to the other way around. For preview, it doesn’t cost anything, but it is supported, meaning you can test this on your machines and if you have a problem, we’re on call. We take the support case and we’ll escalate it to our engineering team and it’s not just, we won’t dodge the issue just because it’s in preview.

Michael:
We don’t know yet what the licensing model is going to look like, so as soon as we do obviously, we’ll have blog posts and major announcements on the Azure blog to explain what that looks like. But this is an area where we want to get feedback. I know what I’m advocating for, but there’s a whole bunch of things that influence this process.

Mike:
Awesome, great.

Michael:
Let me throw, to really leave people with something that they could chew on as we keep going. I’ll just throw one last thing on here. So this whole concept of Azure policy, in like we’re introducing configuration, and these policies pivot on any known property of this machine and it could be in Azure or not in Azure. So let’s project this out and think about how this starts to work. Let’s say it’s a year from now, we’ve got all these different policies that not only audit, they push configurations to servers, both Windows and Linux.

Michael:
In theory, if this all works out right, you’ll be able to say, for all the machines in my subscription, here’s the security stuff that has to be there no matter what. And then here’s the requirements that have to be there operationally, like these tools and stuff like that. Now you could start getting more interesting, almost like SCCM collections, now Microsoft Endpoint Manager, where you could say, all right, that’s neat. But if I tag this machine’s SQL, it means that I put a SQL server on it. Now I also go and push my best practice configuration for SQL server. And maybe I could even get dynamic and say, if it’s in a European location and the server name is Web number, number. That tells me it’s probably a web server, it’s got to comply with GDPR, so I’ve got some additional layers of configuration to happen there, and build on it and build on it, and build on it.

Michael:
It’s conceivable that we will reach the points where you can just deploy servers based on where they’re at, what resource group they’re in, associates that are with an application, maybe what name, what tag, we’re going to get to the point where you can very reliably reproduce these outcomes with very little work going into it. Once you’ve made the investment and figured out this model and how these configurations are going to flow, and now you can have these application owners just go, go, go, because they’ll be able to say, I can spin up database instances on Windows or Linux, and I don’t have my operations team on my back telling me that I didn’t set a two gig memory limit for my SQL instances and stuff like that. Or two terabit or whatever the case may be. It just happens for me automatically as long as I’ve got that Arc agent on it, or if I deploy it in Azure.

Michael:
And if we continue down this path, it’s going to get really interesting. I can imagine, you stand up a Hyper-V Box and it knows that it needs to create these VMs, based on where it’s at in the world. So I’m pretty excited where this could all head.

Mike:
Yeah, I’m pumped. I’m really pumped with everything that happened at Ignite, what you guys are working on, I’m excited. I appreciate you taking the time out Michael because I know you’re incredibly busy and just really appreciate you coming and talking to everybody. Maybe we can get you back on this show sometime-

Michael:
Sounds great.

Mike:
Thanks so much sir, and thanks everybody for joining in livestream. We’ll see you guys in the next episode. Thanks everybody.

Mike:
Want to keep up with what’s going on in cloud computing? If so, subscribe to my weekly newsletter and get my top five tips every week first thing on top of Azure, AWS and Google Cloud. Just go to askmike.io/subscribe to join today. Every week I’ll send out information about cloud architecture and developments, containerized applications with Docker and Kubernetes, devops and automation and strategies for getting the latest cloud computing certifications.

Mike:
If that sounds awesome to you, go to askmike.io/subscribe to join the list today.

Weekly Updates Delivered to Your Inbox

Subscribe to CloudSkills Weekly and get access to exclusive training, updates on industry trends, and advice to amplify your cloud career.