Cloud Zone is brought to you in partnership with:

Troy Hunt is a Software Architect and Microsoft MVP for Developer Security. He blogs regularly about security principles in software development at troyhunt.com and is the author of the OWASP Top 10 for .NET developers series and free eBook of the same name. Troy is also the creator of the recently released Automated Security Analyser for ASP.NET Websites at asafaweb.com. Troy is a DZone MVB and is not an employee of DZone and has posted 58 posts at DZone. You can read more from them at their website. View Full User Profile

With Great Azure VM Comes Great Responsibility (Which is Why You Really Want an Azure Web Site)

03.11.2014
| 3478 views |
  • submit to reddit

I’ve had a recurring discussion with a number of well-meaning people (WMPs) recently which has gone kind of like this:

WMP: We’re going to build you a web site and we’re going to use Azure.

Me: Awesome! So you’d use an Azure Web Site service then?

WMP: No, even better, we’re going to use an Azure VM!

Me: Ok, so why’s that?

WMP: Because it gives us more power and control. Power and control are good.

Me: Righto, so what are you going to do with that power and control that you can’t do with a web site?

WMP: Because it gives us more power can control!

Me: Sorry, maybe you misunderstood – what’s the benefit you gain in having an operating system at your disposal?

WMP: Because, uh, reasons…

In short, they want a VM because that’s just what you do when you stand up a web site – you get the biggest damn thing you have the most control over as possible and crank it up to 11 from day one. You can then RDP in and move files around and look at event logs and do a whole bunch of stuff that you think you want to do, but that you really don’t.

In near on every single case I see, a virtual machine is exactly what you don’t want. It’s not just as I imply from the title insofar as great power giving you great responsibility (although I will come back to that), in fact there are many, many reasons why a web site should be your default preference and in fact you lose many, many things by having a VM. Let me walk you through it.

How to: Deploying a web site on a VM

Let’s start with the first problem – see if you can spot the virtual machine running IIS in the wizard for adding an Azure resource:

Azure wizard

See it? No, of course you don’t because it doesn’t exist. You’re on your own here which means that you’re going to be building up an instance of IIS all on by yourself. You’d better know what you’re doing on that front too; Microsoft’s web server has plenty of nuances in how it should be optimally configured and if you’re the guy building the web site, chances are you aren’t the guy who deeply understands IIS.

But let’s persevere with this anyway so I’ll create a small VM called, oh I don’t know, let’s say “YouDontNeedAVM” and I’ll use a Windows Server 2008 R2 SP1 image. And now we wait. And wait. And wait. In the scheme of creating an entire operating system, somewhere in the order of 10 minutes is actually very good but an Azure Web Site is near instantaneous. Of course this isn’t really an issue in the scheme of a resource that will be around for years to come, but it’s reflective of the scale of the resources you’re creating here – you’re getting a whole freaking operating system for that one little web site!

So machine created, now we can remote in:

RDP'ing into the VM

Ah, feel the power! I’m revelling in the control I have over this machine that is there purely to do my bidding, let’s start exercising that control and creating all the things you need to make the web site work. The first and most crucial thing is IIS and you are 100% responsible for installing, configuring and managing it so you’d better know what you’re doing and be competent in an infrastructure operations role.

Let’s proceed anyway and I’ll begin by adding a server role which will be IIS:

Adding an IIS role

Moving on all we need to do now is select from a simple list of options:

The options list for installing IIS

Of course what you see here is wrong, at least for your needs. You’re probably going to want ASP.NET support but then you really don’t want directory browsing and dynamic content compression is always nice but then you’re missing the management service for remote admin (because you’re hardcore and really just want to do everything over RDP) and so on and so forth. The point I’m trying to make is that there are a lot of moving parts in IIS and you want to be very confident that you know how to manage them all.

Let’s just run with the defaults for now and that gets IIS up and running in a basic configuration. The next thing you’ll need is an endpoint so that people can actually access it over port 80. You see by default, a VM can only be accessed via PowerShell or RDP which is why there are only two endpoints configured in the management portal:

Default endpoints for a VM

Let’s add an endpoint:

Add an endpoint wizard

End of course we want to enable HTTP over port 80:

Specifying port 80

And eventually…

Default IIS website running on the VM

It’s alive! Now for all the things that are already wrong with this installation: You can’t FTP to it because we have to install that service (although we could have selected it from above) but then you still can’t FTP to it because there’s no endpoint configured for port 21 so you need to configure that but then you shouldn’t for FTP’ing in the clear so you really need port 22 for SFTP but of course you really shouldn’t be deploying your web site by FTP anyway. Ok, so you can configure Web Deploy which is the correct approach then you also need to add an endpoint for port 8172. Oh – if you want SSL you’ll need an endpoint for 443 as well plus you’ll be doing all the certificate installation and configuration by hand as well because there’s no portal to automate it for you. Then you want to think about whether you really want everything just dumped on the C drive as space is limited and you typically want to think about dropping it on a data drive which you can do, but it means heading back to the Azure portal and creating a new disk after which you’ll need to head back into the VM (because you love RDP) and mount is appropriately then reconfigure IIS to point to a path on that disk. Don’t forget to get all those permissions spot on as well – too strict and nothing works, too loose and you’re going to have bigger problems than just Web Sites versus VMs.

To some extent, when you stand up a web site on an Azure VM, you become the hosting provider. What I mean by that is that it’s you that needs to take responsibility for the virtualised infrastructure. You need to be a proficient server administrator to get all of this right and you really do have to ask the question: Is this my expertise? Is it the best use of my time to add this responsibility? Almost always, no, it’s not but we’re only just getting warmed up, let’s talk about patches.

Because everybody loves managing server patches, right?

When you take a new VM from the wizard I showed earlier, you get a fully patched and up to date image which is nice. One month from now it is no longer nice – it’s unpatched and vulnerable as another Patch Tuesday has passed. Of course it may be that the second Tuesday of the month has passed without much to talk about, but it may also be that immediately after you have Exploit Wednesday.

Having a VM is like having a kid – you can’t just stand one up and forget about it. Actually it’s not like having a kid because kids grow up and become responsible for themselves but you’ll always be patching and maintaining your VM and that’s an activity you’ll have to actively perform at least once every month. This is not one of those “yeah but it’s academic and not that important” issues, this is one of those “do it or expect serious problems” issues.

Let me give you two cases in point: End of the year 2011 and Hash DoS hits ASP.NET apps. This was very nasty, easily exploitable and Microsoft quickly released critical patch MS-100 out of band of the usual Patch Tuesday schedule. In fact they released it on December 29 which means you, dear server administrator (because that’s what you are now), needed to take time out of your Xmas holidays and patch vulnerable machines which would include any Azure VMs.

Another similar case was the Padding Oracle exploit in 2010. This was a very similar story insofar as it was a critical risk within the ASP.NET framework that was readily exploitable and quickly patched – out of band again. This is what all that power some people lust after in their own VM gets you – a huge amount of responsibility. And these are only a couple of ASP.NET examples, there’s all the various RDP patches and others that plugged the sort of serious holes you expect to find in a large scale operating system over time.

Of course the other problem this gets you is how are you going to test these patches? Remember, when you have your own bespoke server configuration that inevitably begins to deviate from the off-the-shelf installation that Microsoft gave you, you’re increasing the risk of future updates going wrong. In all of the cases I know of where you have professional teams managing infrastructure, they’re installing patches in dedicated testing environments, making sure things play nice then pushing them out. In other words, are you happy to maintain a mirror copy of the VM just to make sure patches aren’t breaking your things? Or is this your testing strategy:

I don't always test my code, but when I do I do it in production

I don’t want to berate Azure VMs per se because they’re excellent for certain purposes. For example, I made great use of one for some very short term resource intensive data processing recently. I also use one for the import of data breaches on my “Have I been pwned?” project (HIBP) and it makes enormous sense in both these cases. They’re also both things I simply can’t do any other way and in both cases I start the VM, get in there and do what I need to do then get out again. I certainly don’t want the responsibility of maintaining a VM environment the public hits 24x7.

Now you may wonder – “Don’t I still need to test my web apps deployed to an Azure Web Site after server patches anyway?” – and that’s a fair question. Let me draw on an old cloud paradigm analogy to try and explain this: we all have access to electricity which we readily consume by plugging our things into the wall. The electricity is the service and we get that from the likes of Energex in Australia or other providers in other locations. How they deliver that service to my wall socket is not my concern so long as I get a constant flow of the stuff (tree-hugging hippies may disagree but you can see my point). Their power stations will undergo maintenance and their wires will be replaced and all sorts of bits required to deliver the service will change, but the service itself remains the same.

When you have an Azure Web Site, you have a service guarantee from Microsoft that your little slice of their infrastructure will remain stable and indeed they have SLAs in place to very clearly set expectations. You don’t need to proverbially test that you kettle still works when Energex updates facets of their power station operations any more than you need to test that your web sites still work when Microsoft applies updates to the underlying infrastructure. New features will be added to Azure as the platform evolves and inevitably we’ll see support for things like new language versions, but they’re not about to say, yoink PHP support or break your auth implementation and if that did happen on the Azure Web Site platform, there’s going to be a whole heap of other people jumping up and down very quickly because they’re all sitting on exactly the same service. When your mind is back in the old paradigm of managing your own server and being responsible for everything as you are with an Azure VM, this can be difficult to get your head around but it shouldn’t be – it’s actually all very simple now in that it just works!

Incidentally, there are facilities within the Azure scale out model (which I’ll talk about shortly) which provide high-availability options by deploying assets across multiple “Update Domains” such that when the underlying service is updated it happens domain by domain and load can be shuffled from one to the other without the service needing to go down during updates. Have a read of this piece by Mark Russinovich (emphasis mine):

A key attribute of Windows Azure is its PaaS scale-out compute model. When you use one of the stateless virtual machine types in your Cloud Service, whether Web or Worker, you can easily scale-up and scale-down the role just by updating the instance count of the role in your Cloud Service’s configuration. The FC does all the work automatically to create new virtual machines when you scale out and to shut down virtual machines and remove when you scale down.

What makes Windows Azure’s scale-out model unique, though, is the fact that it makes high-availability a core part of the model. The FC defines a concept called Update Domains (UDs) that it uses to ensure a role is available throughout planned updates that cause instances to restart, whether they are updates to the role applied by the owner of the Cloud Service, like a role code update, or updates to the host that involve a server reboot, like a host OS update. The FC’s guarantee is that no planned update will cause instances from different UDs to be offline at the same time.

Mark talks about PaaS – Platform as a Server (such as Azure Web Sites) – which is fundamentally different to IaaS – Infrastructure as a Service (such as a dedicated Azure VM). When you’re getting a platform such as an IIS web site, there are all sorts of tricks that can be done underneath such as the one quoted above. When all you have is a single virtual machine, many of these options are completely off the cards.

A VM is not just for Xmas

When you create a new standalone VM as we did earlier on, you have a state of the art, entirely modern operating system. When you create a new Azure Web Site, you have a state of the art, entirely modern web hosting platform.

Fast forward three years and what do you have? If it’s an Azure Web Site, well, you still have a state of the art, modern web hosting platform that has had three years’ worth of new features and automatic upgrades from Microsoft. The platform – the service – has evolved over time and you’ve benefitted from all the new shiny bits.

However, if it’s an Azure VM you manage yourself then you still have exactly what you created three years ago except now you’re on a 5 year old OS (this is the 2012 image, of course, albeit the R2 edition). Any new features or fancy bits will only be the things that you have added yourself as part of a general server maintenance and upgrade regime (which, of course, you’ve carefully tested in a non-production environment). It will also be at least one generation out of date, maybe two given the rate at which operating systems are revving these days.

Perhaps three years is too short – maybe think along the lines of 5 and in this era with the rate of change of modern software, that’s a lot. At some point you’re going to have to move off that platform which, if you really are wedded to the idea of managing your own OS, means installing an all new one and setting it up from scratch. Again. Trust me, I know how little people think about events years in advance but I also know how debilitating it can be when you’re on an old OS on which you’ve created all these dependencies but just can’t find the time or justify the effort to move forward onto something modern.

The point I’m making is that using a dedicated VM and managing your own OS is a long-term affair, certainly compared to an Azure Web Site. The friction of change is high (again, that’s a relative term) and that VM is going to be your very own needy child until you get to the point where you’re ready to retire it.

You risk creating machine dependencies

Let’s move onto the web app itself and talk about creating dependencies on the machine. I see this one happen all the time and it’s by no means specific to Azure’s offerings. Let me give you the most classic, common and scary scenario – Microsoft Office. Yes, it’s true, some people really do create web apps that simply won’t work unless Microsoft Office is installed. I’ve seen all sorts of nastiness related to COM dependencies and Office prompts appearing on unintended server screens and it’s just a whole world of pain. People create web apps with dependencies on Office “because they can”.

One of the big problems this creates is that the app is no longer transportable; you can’t just pick it up and whack it on any old IIS web site. Now you may say “but I don’t want to just whack it on any old IIS web site” and that may be true – today. The problem, however, is when you have an entire machine at your disposal the temptation is often to do things that by any measure – VM or no VM – are very poor coding practices. Managing your own VM give you the ability to do this.

Of course what you really want is a bin-deployable solution like, say OpenXML if what you really wanted to do with Office on the server is just create Excel documents. Or maybe iTextSharp if you were installing that PDF app locally just to generate some docs. There are usually free tools that are publishable with the app and run anywhere. If you’re not using libraries like this, take a good hard look at your app because there’s a good chance you’re doing it wrong.

Of course if you’re not creating these dependencies, then you don’t need a VM for installing things at will now, do you?

There’s no management portal for web sites on a self-managed VM

There are many, many things you get with the Azure Web Site portal that otherwise make life hard when you just run IIS in a VM. For example, how are you going to manage your connection strings? Please don’t tell me you’re just going to whack your plain text credentials into source control! Sensitive data doesn’t belong in source control, it’s up to the build and release process to manage things like connections strings or API keys or anything else you want to keep private.

Now of course you can still do this with a VM, just use a build server like TeamCity and parameterise the builds to apply the correct settings. Or you can just do this in an Azure Web Site:

Managing connection strings in an Azure Web Site

On deployment, Azure will automatically transform each of the connection strings in my HIBP web site and apply the correct server name, username and password. I don’t need to store them in source control and indeed the ones in my web.config point to local instances of each of these and they’re just automatically transformed on release. You get that for free in an Azure Web Site.

Actually, the whole connection string management thing becomes even easier when you have a linked resource such as a database:

Database linked resource

This creates an association between the two and automatically created the connection string you see above. Neat.

We’re only just getting started, what about things like managing all your domain names for the site:

Managing domain names

Sure, you can do that inside IIS and manage your own bindings manually, but nothing is easier than just doing it via the browser.

Or how about managing your site and app diagnostics all via the browser:

Site diagnostic settings in the portal Application diagnostic settings in the portal

Again, it’s doable by RDP’ing into the machine but it’s a whole lot easier via the Management Portal where everything is in one place.

That’s just scratching the surface though, let’s move into the powerful stuff.

Published at DZone with permission of Troy Hunt, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)