Long origin custom headers seem to be leading to 403 errors when using them to control access to a static website in S3. Are there implicit restrictions in place that would prune or drop them?

When the Move Goes Wrong

As you may have noticed, this blog has moved to a new platform. All the reasons why we exited Kubernetes and where we moved to can be found in our new “Goodbye K8s!” series.

Unfortunately, you may have also noticed the transition in the most unanticipated — and even worse — unpleasant way: By getting one of these pesky 403: Forbidden error pages shown below. Instead of what you were really after.

Not good. Even worse, the error pages were displayed intermittently, seemingly having a mind of their own.

After some initial analysis and a lengthier post-mortem:

1. We’re very sorry for any inconvenience this may have caused. We trust that Google’s cache may have helped you bridge the gap in the meantime.
2. The problem has been identified to reside within CloudFront and the way we leverage origin custom headers
3. With CloudFront being a black box, we rely on the observations we can make. But we could really use someone to give us some feedback or point us to what may have caused the problem in the first place. There’s a least one pint in it…
4. We’re taking remediating actions to prevent similar situations

The New Stack

On a technical level, we moved this blog to a much simplified stack, now comprised of

1. A static website hosted in S3
2. Route 53 entries for the domain
3. An ACM certificate for the domain
4. A CloudFront distribution providing a globally distributed CDN
5. A dedicated and least-privileged IAM user for basic CRUD operations on the S3 website bucket contents as well as CloudFront distribution invalidations

Everything is built via Terraform for automation, repeatability, and auditability. Moreover, in order to limit direct access to the static S3 website bucket, all requests for s3:GetObject and s3:GetObjectVersion actions need to contain an HTML header with key Referer and a secret value.

The entire setup — and especially the Referer header solution — isn’t original or special in any case. It follows the general setup described in section “Using a website endpoint as the origin, with access restricted by a Referer header” on the corresponding AWS’s premium support website.

CloudFront and Custom Origin Headers

The point where it gets interesting is how CloudFront distributions created at different points in time seem to be handling custom origin headers.

A Working Staging Environment

When originally deploying the entire stack for the staging environment on 2020-10-29, the following 256 character value worked in the corresponding CloudFormation distribution without any problem

Referer: jdYffHSMl9JWZlFC0yoV3ga7Yb8oxnrVh1bkmNBRNQH9WkGDywtQhAzACQretFSJnZo5f41QLF1CohGvcAojpSAdL9QBSFgCSFa1M9SEkH1bsCL6LBpGkRyaFDRkjpaNSYnMQsF8YoA8SLRrxhqGQX8Mlb0Quo5LmsKn8BAEXEWsjI8CVgHAArmb500Y70zLEZWzyBdrlkNtu8Ej4h2F5rP1yPXw15PayqUXciZd9gHsLyOBcDPTOuDMDh4aSXZT

The matching policy (excerpt) for the corresponding S3 bucket was

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowCloudFormationOriginAccessObjects",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObjectVersion",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::staging-how-hard-can-it-be---contents/*",
            "Condition": {
                "StringEquals": {
                    "aws:Referer": "jdYffHSMl9JWZlFC0yoV3ga7Yb8oxnrVh1bkmNBRNQH9WkGDywtQhAzACQretFSJnZo5f41QLF1CohGvcAojpSAdL9QBSFgCSFa1M9SEkH1bsCL6LBpGkRyaFDRkjpaNSYnMQsF8YoA8SLRrxhqGQX8Mlb0Quo5LmsKn8BAEXEWsjI8CVgHAArmb500Y70zLEZWzyBdrlkNtu8Ej4h2F5rP1yPXw15PayqUXciZd9gHsLyOBcDPTOuDMDh4aSXZT"
                }
            }
        }
    ]
}

Access from the CloudFormation distribution to the S3 website endpoint was working. No 403s. Just websites.

A Wobbly Production Environment

When deploying the entire stack again for the production environment (you’re looking at it) on 2020-11-15, the following 256 character was used in the corresponding CloudFront distribution

Referer: ZV7xSRECNgeaqUvyLoJWuwaD86FgBeV35dYAEUiMgjgRJlQc8Hpx9HASunyziQqIl0aF9r1MGJPejKWE7iEEAFxN4M28a1njdXUKJwDOlNxqsg3TZOJU2leqJItuQzwJNkwk8Z2F9g9bEmOtPjQgx1Nshrgbv4PfTvjoGYl3rgL2Bd8PiHsQDGYMclW4bdpyJn6bXJRIlQUTSnZHTVmUBnMYEqtZ4XHXOyVG3oooCi5du0NmNkS6DldDRaXQ1SCn

The matching policy (excerpt) for the corresponding S3 bucket was

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowCloudFormationOriginAccessObjects",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObjectVersion",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::production-how-hard-can-it-be---contents/*",
            "Condition": {
                "StringEquals": {
                    "aws:Referer": "ZV7xSRECNgeaqUvyLoJWuwaD86FgBeV35dYAEUiMgjgRJlQc8Hpx9HASunyziQqIl0aF9r1MGJPejKWE7iEEAFxN4M28a1njdXUKJwDOlNxqsg3TZOJU2leqJItuQzwJNkwk8Z2F9g9bEmOtPjQgx1Nshrgbv4PfTvjoGYl3rgL2Bd8PiHsQDGYMclW4bdpyJn6bXJRIlQUTSnZHTVmUBnMYEqtZ4XHXOyVG3oooCi5du0NmNkS6DldDRaXQ1SCn"
                }
            }
        }
    ]
}

Despite manual GET requests to the S3 bucket website endpoint — using the Postman app and the above custom Referer header — succeeding, the CloudFront distribution kept on forwarding 403s.

Debugging this problem wasn’t really helped by the fact that CloudFront caches responses (as it should) and only exposes errors after the cache times out. And then there’s always DNS as well. Invalidations and tests to the rescue!

After following all steps on another Premium Support website without success, a working solution under the given time constraints was to disable the Referer header constraint on the S3 bucket.

After some more experimentation, it turned out that reducing the Referer value down to 32 characters did the trick. However, this is by no means a boundary as it was discovered by accident without any binary search to determine its precise limits.

Who Would Be Able to Clarify?!

We are unable to explain the observed behaviour in the CloudFront distribution. And it might be absolutely possible that I did something funky (or stupid) on our end as well to deserve that behaviour.

Worst of all, we were able to reproduce the error with the identical Terraform stack on 2020-11-15 but not since. It’s superb to have things working again but the root problem remains an unsolved mystery that could come back to haunt us at any point in time.

We’re on a basic support plan and it seem that technical support is not included by default. Hence, we are asking the AWS community and all experts:

Would anyone reading this be able to help us better understand what we did wrong or if something changed in regards to CloudFormation distributions in the meantime?

You could argue that 32 characters are plenty enough for such a primitive mechanism to prevent direct access. If needed at all, given it is a public website at the end of the day.

But then the documentation also states that 1,783 characters are allowed for each single custom header up to a total of 10,240 characters. Any we only use one header value.

Who would be able to help?! There’s at least one pint in it!

Remediating Actions

Until we get a definitive answer and a suitable solution, the best we can do for now is to put health checks in place which monitor the page and raise alerts when things go wrong.

So, How to do Custom Origin Headers?!

While the above sort-of Worx for Me!™ for now but still leaves me partially puzzled, you may have an alternative or better way.

Think this is all rubbish, incomplete, or massively overcomplicated?! Feel free to reach out to me on LinkedIn and teach me something new!

As always, prove me wrong and I’ll buy you a pint!