Web Application Gateway v2 | Aidan Finn, IT Pro

Securing A Storage Account Static Website Using VNet Web Application Firewall

Alternative title: Using the Azure Application Gateway to do content redirection with a storage account static website in a secure way.

I was looking at a scenario where I needed to find a platform method of setting up a website that would:

Be cost-effective
Be able to easily receive content directly from Azure virtual machines
Be secure

This post will describe the solution.

The Storage Account

A resilient storage account is set up with a static website. The content can be uploaded to the $web container. The firewall is enabled and only traffic from the virtual machine subnet and an Azure Application Gateway or Web Application Firewall subnet is allowed. This means that you get a 404 error when you try to access the website from any other address space.

The WAF

A WAFv2 is set up. The WAF subnet is protected by an NSG. The WAF is controlled by a WAF policy. And certificates for custom domains are stored in a Key Vault – the WAF uses a user-managed identity to get Get/List rights to secrets/certificates in the Key Vault’s access policy. A multi-site HTTPS Listener is set up for the static website using a custom domain name:

The HTTP setting will handle the name translation from the custom domain name to the default storage account URI.
The Key Vault will store the certificate for the custom domain name.
There is full end-to-end encryption thanks to the storage account using a Microsoft-supplied certificate for the default storage account URI.

The HTTP setting in the WAF will be set up as follows:

HTTPS
Use Well-Known CA Certificate (Yes)
Override with a new hostname: the default URI of the static website

Solution 1 – Service Endpoint

In this case, the WAF subnet has a Microsoft.Storage service endpoint enabled. This will mean that traffic from the WAF to the storage account hosting the static website will fall through a routing “trap door” across the Azure private backbone to reach the storage account. This keeps the traffic relatively private and reduces latency.

The backend pool of the WAF is the FQDN of the static website.

Pros:

Easy to set up.

Cons:

Service Endpoints appear to be a dead-end technology
It will require the Microsoft.Storage Service Endpoint to be configured in every subnet that needs to interact with the website/storage account.

Solution 2 – Private Link/Private Endpoint

In this design, Service Endpoint is dropped and replaced with a Private Endpoint associated with the Web API of the storage account. This Private Endpoint can be in the same VNet as the WAF or even a different (peered) VNet to the WAF.

The only change to the WAF configuration is that the backend pool must now be the private IP address of the Private Endpoint. Now traffic will route from the WAF subnet to the storage account subnet, even across peering connections.

Pros:

Private Link/Private Endpoint is the future for this kind of connectivity.
There is no need to configure subnets with anything – they just need to route to the storage account (to modify content) or the WAF (access content).

Cons:

A little more complex to set up, but the effort is returned in the long-run with less configuration required.

There is no support for inbound NSG rules for the Private Endpoint but:

That is coming in the future
The storage account firewall is rejecting unwanted direct traffic
The NSG in front of the WAF provides Layer-4 security and the WAF provides Layer-7 security

Want to Learn More?

Why not join me for an ONLINE 1-day training course on securing Azure IaaS and PaaS services. Securing Azure Services & Data Through Azure Networking is my newest Azure training course, designed to give Level 400 training to those who have been using Azure for a while. It dives deep on topics that most people misunderstand and covers many topics similar to the above content.

Understanding How Azure Application Gateway Works

In this post, I will explain how things such as frontend configurations, listeners, HTTP settings, probes, backend pools, and rules work together to enable service publication in the Azure Web Application Gateway (WAG)/Web Application Firewall (WAF).

Introduction

The WAF/WAG is a scary beast at first. When you open one up there are just so many settings to be tweaked. If you are publishing just a simple test HTTP server, it’s easy: you populate the default backend pool and things just start to work. But if you want HTTPS, or to service many pools/sites, then things get complicated. And frustratingly slow 🙂 – Things have improved in v1 and v2 is significantly faster to configure, although it has architectural limitations (force public IP address and lack of support for route tables) that prevent me from using v2 in my large network deployments. Hopefully, the above map and following text will simplify things by explaining what all the pieces do and how they work together.

The below is not feature complete, and things will change in the future. But for 99% of you, this should (hopefully) be helpful.

Backend Pool

The backend pool describes a set of machines/services that will work together. The members of a backend pool must be all of the same type from one of these types:

IP address/hostname: a common choice in large Azure deployments – you can span peering connections to other VNets
Virtual machine: Select a machine from the same VNet as the WAG/WAF
VMSS: Virtual machine scale sets in the same VNet as the WAG/WAF
App Services: In the same subscription as the WAG/WAF

From here on out, I’ll be using the term “web server” to describe the above.

Note that this are the machines that host your website/service. They will all run the same website/service. And you can configure an optional custom probe to test the availability of the service on these machines.

(Optional) Health Probe

You can create a HTTP/HTTPS probe to do deeper probe tests of a service running on a backend pool. The probe is configured for HTTP or HTTPS and tests a hostname on the web server. You specify a path on the website, a frequency, timeout and allowed number of retries before designating a web site on a web server as being unhealthy and no longer a candidate for load balancing.

HTTP Setting

The HTTP setting configures how the WAG/WAF will talk to the members of the backend pool. It does not configure how clients talk to the site (Listener). So anything you see below here is for configuring WAG/WAF to web server communications (see HTTPS).

Control cookie-based affinity for load balancing
Configure connection draining when a machine is removed from a backend pool
Specify if this is for a HTTP or a HTTPS connection to the webserver. This is for end-to-end encryption.
- For HTTPS, you will upload a certificate that will match the web servers’ certificate.
The port that the web server is listening on.
Override the path
Override the hostname
Use a custom probe

Remember that the above HTTPS setting is not required for website to be published as SSL. It is only required to ensure that encryption continues from the WAG/WAF to the web servers.

Frontend IP Configuration

A WAG/WAF can have public or private frontend IP addresses – the variation depends on if you are using V1 (you have a choice on the mix) or V2 (you must use public and private). The public front end is a single public IP address used for publishing services publicly. The private frontend is a single virtual network address used for internal service publication, requiring virtual network connectivity (virtual network, VPN, ExpressRoute, etc).

The DNS records for your sites will point at the frontend IP address of the WAG/WAF. You can use third-party or Azure DNS – Azure DNS has the benefit of being hosted in every Azure region and in edge sites around the world so it is faster to resolve names than some DNS hoster with 3 servers in a single continent.

A single frontend can be shared by many sites. http://www.aidanfinn.com, http://www.cloudmechanix.com and http://www.joeeleway.com can all point to the same IP address. The hostname configuration that you have in the Listener will determine what happens to the incoming traffic afterwards.

Listener

A Listener is configured to listen for traffic destined to a particular hostname and port number and forward it, eventually, to the correct backend pool. There are two kinds of listener:

Basic: For very simple configurations where a site has exclusive ownership over a port number on one of the frontends. Typically this is for point solutions where a WAG/WAF is dedicated to a service.
Multi-Site: A listener shares a frontend configuration with other listeners, and is looking for traffic destined to a specific hostname/port/protocol.

Note that the Listner is where you place the certificate to secure client > WAG/WAF communications. This is known as SSL offloading. If you enable HTTPS you will place the “site certificate” on the WAG/WAF via the Listener. You can optionally re-encrypt traffic to the webserver from the WAG/WAF using the previously discussed HTTP Setting. WAGv2/WAFv2 have a no-support preview to use certs that are securely stored in Key Vault.

The configuration of a basic listener is:

Frontend
Frontend port
HTTP or HTTPS protocol
- The certificate for securing client > WAG/WAF traffic
Optional custom error pages

The multi-site listener is adds an extra configuration: hostname. This is because now the listener is sharing the frontend and is only catching traffic for its website. So if I want 3 websites on my WAG/WAF sharing a frontend, I will have 3 x HTTPS listeners and maybe 3 x HTTP listeners.

Rules

A rule glues together the configuration. A basic rule is pretty easy:

Traffic comes into a Listener
The HTTP Setting determines how to forward that traffic to the backend pool
The Backend Pool lists the web servers that host the site

A path-based rule allows you to extend your site across many backend pools. You might have a set of content for /media on pool1. Therefore all http://www.aidanfinn.com/media content is pulled from that pool1. All video content might be on http://www.aidanfinn.com/video, so you’ll redirect /video to pool2. And so on. And you can have individual HTTP settings for each redirection.

My Tips

There’s nothing like actually setting this up at scale to try this out. You will need a few DNS names to be able to work with.
Remember to enable the protection mode of WAF. I have audited deployments and found situations where people thought they had Layer-7 security but only had the default “alert-only” configuration of WAFv1.
In large environments, don’t forget to ensure that the NSGs protecting any webservers allow traffic in from the WAG/WAF’s subnet into the web servers on the port(s) specified in the HTTP Setting(s). Also ensure that any guest OS firewall is similarly configured.
Possibly the biggest issue you will have is with devs not assigning hostnames to websites in their webservers. If you’re using shared WAGs/WAFs you must use multi-site listeners and the websites should be configured with the hostname.
And the biggest tip I can give is to work out a naming standard for each of the above components so you know what piece is associated with what site. I can’t share what we’re using at work, but we have some big configurations and they are very easy to troubleshoot because of how we have named things.

Azure Availability Zones in the Real World

I will discuss Azure’s availability zones feature in this post, sharing what they can offer for you and some of the things to be aware of.

Uptime Versus SLA

Noobs to hosting and cloud focus on three magic letters: S, L, A or service level agreement. This is a contractual promise that something will be running for a certain percentage of time in the billing period or the hosting/cloud vendor will credit or compensate the customer.

You’ll hear phrases like “three nines”, or “four nines” to express the measure of uptime. The first is a 99.9% measure, and the second is a 99.99% measure. Either is quite a high level of uptime. Azure does have SLAs for all sorts of things. For example, a service deployed in a valid virtual machine availability set has a connectivity (uptime) SLA of 99.9%.

Why did I talk about noobs? Promises are easy to make. I once worked for a hosting company that offers a ridiculous 100% SLA for everything, including cheap-ass generic Pentium “servers” from eBay with single IDE disks. 100% is an unachievable target because … let’s be real here … things break. Even systems with redundant components have downtime. I prefer to see realistic SLAs and honest statements on what you must do to get that guarantee.

Azure gives us those sorts of SLAs. For virtual machines we have:

5% for machines with just Premium SSD disks
9% for services running in a valid availability set
99% for services running in multiple availability zones

Ah… let’s talk about that last one!

Availability Sets

First, we must discuss availability sets and what they are before we move one step higher. An availability set is anti-affinity, a feature of vSphere and in Hyper-V Failover Clustering (PowerShell or SCVMM); this is a label on a virtual machine that instructs the compute cluster to spread the virtual machines across different parts of the cluster. In Azure, virtual machines in the same availability set are placed into different:

Update domains: Avoiding downtime caused by (rare) host reboots for updates.
Fault domains: Enable services to remain operational despite hardware/software failure in a single rack.

The above solution spreads your machines around a single compute (Hyper-V) cluster, in a single room, in a single building. That’s amazing for on-premises, but there can still be an issue. Last summer, a faulty humidity sensor brought down one such room and affected a “small subset” of customers. “Small subset” is OK, unless you are included and some mission critical system was down for several hours. At that point, SLAs are meaningless – a refund for the lost runtime cost of a pair of Linux VMs running network appliance software won’t compensate for thousands or millions of Euros of lost business!

Availability Zones

We can go one step further by instructing Azure to deploy virtual machines into different availability zones. A single region can be made up of different physical locations with independent power and networking. These locations might be close together, as is typically the case in North Europe or West Europe. Or they might be on the other side of a city from each other, as is the case in some in North America. There is a low level of latency between the buildings, but this is still higher than that of a LAN connection.

A region that supports availability zones is split into 4 zones. You see three zones (round robin between customers), labeled as 1, 2, and 3. You can deploy many services across availability zones – this is improving:

VNet: Is software-defined so can cross all zones in a single region.
Virtual machines: Can connect to the same subnet/address space but be in different zones. They are not in availability sets but Azure still maintains service uptime during host patching/reboots.
Public IP Addresses: Standard IP supports anycast and can be used to NAT/load balance across zones in a single region.

Other network resources can work with availability zones in one of two ways:

Zonal: Instances are deployed to a specific zone, giving optimal latency performance within that zone, but can connect to all zones in the region.
Zone Redundant: Instances are spread across the zone for an active/active configuration.

Examples of the above are:

The zone-aware VNet gateways for VPN/ExpressRoute
Standard load balancer
WAGv2 / WAFv2

Considerations

There are some things to consider when looking at availability zones.

Regions: The list of regions that supports availability zones is increasing slowly but it is far from complete. Some regions will not offer this highest level of availability.
Catchup: Not every service in Azure is aware of availability zones, but this is changing.

Let me give you two examples. The first is VM Boot Diagnostics, a service that I consider critical for seeing the console of the VM and getting serial console access without a network connection to the virtual machine. Boot Diagnostics uses an agent in the VM to write to a storage account. That storage account can be:

LRS: 3 replicas reside in a single compute cluster, in a single room, in a single building (availability zone).
GRS: LRS plus 3 asynchronous replicas in the paired region, that are not available for write unless Microsoft declares a total disaster for the primary region.

So, if I have a VM in zone 1 and a VM in zone 2, and both write to a storage account that happens to be in zone 1 (I have no control over the storage account location), and zone 1 goes down, there will be issues with the VM in zone 2. The solution would be to use ZRS GPv2 storage for Boot Diagnostics, however, the agent will not support this type of storage configuration. Gotcha!

Azure Advisor will also be a pain in the ass. Noobs are told to rely on Advisor (it is several questions in the new Azure infrastructure exams) for configuration and deployment advice. Advisor will see the above two VMs as being not highly available because they are not (and cannot) be in a common availability set, so you are advised to degrade their SLA by migrating them to a single zone for an availability set configuration – ignore that advice and be prepared to defend the decision from Azure noobs, such as management, auditors, and ill-informed consultants.

Opinion

Availability zones are important – I use them in an architecture pattern that I am working on with several customers. But you need to be aware of what they offer and how certain things do not understand them yet or do not support them yet.