This article describes how to create an AWS IAM User (including access keys and secret access keys) with a minimum set of permissions to work with the QNAP Glacier app.

I prefer my functional accounts to be restricted to the minimum set of permissions required to do the job. Not only is it a recommended best practice but it also limits the blast radius in case something goes wrong. As I couldn’t find any documentation on the minimum set of IAM permission required by the QNAP Glacier app, I started my own investigation. This article summarises my findings that are also available as a Terraform module on https://github.com/dumrauf/qnap_glacier_iam_user.

The Importance of Pictures

I love taking pictures. When on holidays or in everyday life. Pictures transport me back to the place they were taken. They bring back memories of that one picturesque sunset over some of the most spectacular scenery on earth. Plus, all the anxiety on the steep three hour uphill slog in the middle of the night that followed shortly after. I’ll never forget those seven river crossings that were a mere side note in the hiking guide. But then, look at that view!

Pictures are my everyday teleporter. My time machine. My short escape. Naturally, they mean a lot to me.

The Blessing of High Resolution Cameras

I’ve owned a camera as long as I can remember and I was there when the transition from analogue to digital happened. Digital photography has really come a long way in the last decades. From grainy pictures best viewed at fingernail size in the dark to modern 35+ megapixels (MP in short) miracles that paint a stunning high resolution image of the world with silicone and photons.

And these are just your standard full frame cameras available on the high street. If it has to be the cutting edge of what’s possible, then there are always medium format megapixel monsters from the likes of Hasselblad and Co. which easily crack the 50+ MP barrier and go up to 100 MP.

The Curse of High Resolution Cameras

However, this increased resolution comes at a price - storage. While you can play with formats and settings to reduce the resulting file size, given that you’ve spent all that hard earned money on a high resolution camera in the first place, you probably end up shooting in something like JPEG Fine mode. RAW if don’t fancy compression. TIFF if storage hardly means anything to you.

A Lot Of MBs for a Snap

Not too long ago (and still available if you can get hold of one), a 12 MP camera used to produce on average 6 MB images on half way decently lit subjects. A modern 35+ MP camera easily churns out images of an average size of around 30 MB. You could significantly lower the size of the images by just shooting in pitch dark environments but what’s the point of 35+ MP series of pitch dark images? Oh, and if you really want to burn through storage, start filming in 4K with a high bitrate.

36 Snaps on a Roll

Back in the film days, an image had a real instant cost. You had to buy the film before you could start and those things cost real money. Plus, take up space in your luggage. Oh, and there was no delete option in case you didn’t like the picture (there was no option to check, remember?!). Now, all of that has changed with digital photography. No more need for films. Delete, if you don’t like the snap. Pop in a new storage medium to continue, most likely an SD card these days. Get them by the dozen (or rather not, as speed and reliability can matter and drive up the price, depending on your use case). It’s not uncommon to see people set off on extended trips with half a Terabyte of available storage. That’s 500 GB. Imagine that as a stack of printed sheets of paper…

Securely Storing Those Precious Photos

While it’s perfectly reasonable to store the precious pictures on an SD card inside or outside the camera while on the road, it’s not really an ideal long term archiving solution in my opinion. Plus, it obviously limits the number of pictures you can take in the future, unless you’re invested in SD cards. So, they have to come off. But where to?

A Bunch of Drives on My Local

A local machine might be a good option. But its hard drive could fail. So, we need a backup to another drive. This means backing up the main drive regularly which can be a bit tedious and more often than enough I ended up forgetting about it. Brilliant. The backup was out of sync. Not really ideal in a disaster situation. So, is there a good tool for this job?

A Bunch of Drives in my Network

It turns out that a half-way decent NAS with at least two bays is well suited for the job. Put it into some RAID mode, depending on the number of bays and personal preference, and you’re good. As usual, buy drives of identical size from different manufacturers to lower the risk of outages due to manufacturing errors.

A Bunch of Drives in the Cloud

However, at this point, all your precious pictures are most likely stored in a single physical location. A power surge or a natural disaster could destroy your precious photos in an instance. We need more replication. So, buy another NAS? This could be an option but it’s only a tiny bit better than what we have so far.

If we securely store the pictures in the cloud, then they are backed up off-premise and can be retrieved in the event of a disaster. Plus, we get an SLA around how securely our precious pictures are stored and how long it takes to retrieve them again. All we need is an affordable cloud storage provider and an application that performs regular backups to the long term cloud storage of choice.

AWS Glacier

An answer to the first problem can be AWS Glacier. It’s AWS’s long term archiving solution trusted by numerous organisations around the world. It comes with a rich suite of features, including access controls via IAM policies and vault access policies as well as immutable data retention policies. The latter can be used to enforce regulatory requirements via their apply, test within 24 hours, and apply for good mechanism. Note that once applied, vault lock policies cannot be changed. It does what it says on the tin. Locks the vault policy.

And all of that at a pretty competitive price point. Storing about 1.5 TB of data in AWS Glacier sets me back around USD 6 per month. On top of that goes a one-off cost of around USD 20 for the initial upload. The only catch is that retrieving the data can take up to several hours. Oh, and cost around USD 150. However, if my NAS just went up in flames, then USD 150 seem to be quite reasonable in comparison to kissing all my precious pictures goodbye forever.

The QNAP Glacier App

The answer to the second question depends on your NAS. On my QNAP, there is an app called QNAP Glacier which pretty much does what it says on the tin. It expects a pair of access and secret access keys in the case of AWS. From there on, you just point it to the AWS Glacier vault you want the app to use, select the folders to be synchronised, a (periodic) execution time, and lean back and see the bits’ digital clones wander off to AWS Glacier.

Creating an IAM User for the QNAP Glacier App

In order to arrive at a set of access and secret access keys, we need

1. An IAM user
2. An access key and a secret access keys for the IAM user
3. A policy granting minimum permissions to the IAM user

While we’re at it, we may as well create

4. An SNS topic
5. The AWS Glacier vault and have it send all ArchiveRetrievalCompleted and InventoryRetrievalCompleted events to the SNS topic created in 4.

A First Attempt at a Policy

While the first two resources are fairly straight forward to create, the policy turned out to be a tricky one. The original idea was to allow the IAM user to perform all CRUD operations on archives but limit the operations on the underlying vault to ready only. Following the excellent Amazon Glacier API Permissions: Actions, Resources, and Conditions Reference documentation, the first attempt at a policy turned out to be

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowGlacierGlobal",
            "Effect": "Allow",
            "Action": [
                "glacier:ListVaults",
                "glacier:GetDataRetrievalPolicy"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowGlacierVault",
            "Effect": "Allow",
            "Action": [
                "glacier:UploadMultipartPart",
                "glacier:UploadArchive",
                "glacier:ListTagsForVault",
                "glacier:ListParts",
                "glacier:ListMultipartUploads",
                "glacier:ListJobs",
                "glacier:InitiateMultipartUpload",
                "glacier:InitiateJob",
                "glacier:GetVaultNotifications",
                "glacier:GetVaultLock",
                "glacier:GetVaultAccessPolicy",
                "glacier:GetJobOutput",
                "glacier:DescribeVault",
                "glacier:DescribeJob",
                "glacier:DeleteArchive",
                "glacier:CompleteMultipartUpload",
                "glacier:AbortMultipartUpload",
                "glacier:RemoveTagsFromVault",
                "glacier:AddTagsToVault"
            ],
            "Resource": "arn:aws:glacier:<your-region>:<your-account>:vaults/<your-vault>"
        }
    ]
}

Here, the value of Resource is tailored to the specific region, account, and vault being used.

A Second Attempt at a Policy

The infrastructure required for the QNAP Glacier app is a nice example of what can be built in AWS in a short span on time. The parts work really well inside the QNAP app as well, as it happily accepted my pair of access key and secret access key. However, when testing access to the actual AWS Glacier vault, the QNAP Glacier complained that

Because the vault inventory is updated once a day, you cannot use the exist vault for a backup job. Please create a new vault.

This makes sense when considering that AWS Glacier vault inventories are generated approximately once a day. From the QNAP Glacier app’s perspective, it’s simply the safest way to create a brand new AWS Glacier vault and use that as that’s more likely to be empty.

The above limitation implies that the glacier:CreateVault permission is required by the QNAP Glacier IAM user. That’s not really ideal, as the Amazon Glacier API Permissions: Actions, Resources, and Conditions Reference documentation suggests that the glacier:CreateVault permission can only be granted on a global scope. This then implies that the QNAP Glacier app could potentially create lots of AWS Glacier vaults.

With a little experimentation, it turns out that the glacier:CreateVault permission can be restricted to a specific AWS Glacier vault. The final IAM policy being used for the QNAP Glacier IAM User is hence

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowGlacierGlobal",
            "Effect": "Allow",
            "Action": [
                "glacier:ListVaults",
                "glacier:GetDataRetrievalPolicy"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowGlacierVault",
            "Effect": "Allow",
            "Action": [
                "glacier:UploadMultipartPart",
                "glacier:UploadArchive",
                "glacier:ListTagsForVault",
                "glacier:ListParts",
                "glacier:ListMultipartUploads",
                "glacier:ListJobs",
                "glacier:InitiateMultipartUpload",
                "glacier:InitiateJob",
                "glacier:GetVaultNotifications",
                "glacier:GetVaultLock",
                "glacier:GetVaultAccessPolicy",
                "glacier:GetJobOutput",
                "glacier:DescribeVault",
                "glacier:DescribeJob",
                "glacier:DeleteArchive",
                "glacier:CreateVault",
                "glacier:CompleteMultipartUpload",
                "glacier:AbortMultipartUpload",
                "glacier:RemoveTagsFromVault",
                "glacier:AddTagsToVault"
            ],
            "Resource": "arn:aws:glacier:<your-region>:<your-account>:vaults/<your-vault>"
        }
    ]
}

Again, the value of Resource is tailored to the specific region, account, and vault being used.

The above policy eventually gets the job done and allows to supply access and secret access keys to the QNAP glacier app. However, the assumptions made by the app somehow limit the ability to leverage other functionalities of AWS Glacier vaults. I have yet to find the option that allows me to configure notifications, set vault access policies, set vault Locks, or tag vaults that are being created via the QNAP Glacier app.

Creating a QNAP Glacier IAM User

There are many ways to create access and secret access keys for a QNAP Glacier IAM User. The AWS console provides an option to create and configure a corresponding IAM user and CloudFormation can be used for the job as well. In general, Terraform seems to be well suited for the job of programmatic infrastructure deployment. This is what we are going to use here.

The qnap_glacier_iam_user Terraform Module

The GitHub repository https://github.com/dumrauf/qnap_glacier_iam_user contains a Terraform module which creates an IAM user, including access keys, that can then be used in the QNAP Glacier app.

You Have

Before you can use the Terraform module in this repository out of the box, you need

• an AWS account
• a Terraform CLI

You Want

After applying the Terraform module in the repository, you get an IAM user with corresponding access key and secret access key that can be used in the QNAP Glacier app. Here, the IAM user is only permitted to work with archives in an AWS Glacier vault specified in the input variables. Modifying or deleting the vault is not permitted.

Setup

The input variables for the module are defined in https://github.com/dumrauf/qnap_glacier_iam_user/settings/example.tfvars as

region = "<your-region>"

shared_credentials_file = "/path/to/.aws/credentials"

profile = "<your-profile>"

qnap_vault_name = "<your-QNAP-Glacier-vault>"

Here, you need to replace the example values with your settings. Note that you also need to update the qnap_vault_name to the name of an AWS Glacier vault that does not exist. Moreover, the current value is not a valid input

Make sure that the qnap_vault_name refers to an AWS Glacier vault which does not exist. The QNAP Glacier app assumes that it can create the given AWS Glacier vault.

Execution

Initialise Terraform by running

terraform init

As a best practice, create a new workspace by running

terraform workspace new example

The QNAP Glacier IAM user can be planned by running

terraform plan -var-file=settings/example.tfvars

and created by running

terraform apply -var-file=settings/example.tfvars

Outputs

The module has two outputs, namely qnap_glacier_user_access_key and qnap_glacier_user_secret_access_key which are the access key and secret access key of the newly created IAM user.

The access key and secret access key are intended to be used in the QNAP Glacier app as is.

Deletion

The QNAP Glacier IAM user can be deleted by running

terraform destroy -var-file=settings/example.tfvars

Note that this also removes all access to the AWS Glacier vault for the QNAP Glacier app.

Using Terraform to Tweak the Vault Configuration

While Terraform has the option to pull resources back under its control, it seems to be widely accepted that it’s better to start out with Terraform creating them in the first place. So, it’s possible to tweak the configuration of the AWS Glacier vault created by the QNAP Glacier app but, at least for me, the side effects have yet to be unearthed.

Conclusion

The qnap_glacier_iam_user Terraform module introduced above makes it easy for me to create IAM users tailored to the QNAP Glacier app. This allows me to back up my precious pictures to AWS Glacier with relative peace of mind. While this works for me, it may be different in your case. Feel free to leave a comment or contribute to the Git repository, so we can make things better for everyone. Thanks!