Sometimes you just need a logging bucket.

Chances are, you are working with one of the following AWS services and want to store their logs in an AWS S3 Bucket

• EC2
• VPC
• S3
• ELB
• CloudTrail
• CloudFront
• CloudWatch
• ECS
• SNS
• OpsWorks

That’s what logging buckets are for.

The Nature of a Logging Bucket

A logging bucket is an AWS S3 Bucket which is accessible by the “special” AWS Log Delivery Group. For this, it needs to at least grant the log-delivery-write permission in the ACL of the S3 Bucket.

As a best practice, it should also have server side encryption enabled and be tagged.

# Creating a Logging Bucket

There are many ways to create logging buckets. The AWS console provides an option to create and configure logging buckets and CloudFormation can be used for the job as well. In general, Terraform seems to be well suited for the job of programmatic infrastructure deployment. This is what we are going to use here.

The aws_log_bucket Terraform Module

The GitHub repository https://github.com/dumrauf/aws_log_bucket contains a Terraform module which creates logging buckets.

You Have

Before you can use the Terraform module out of the box, you need

• an AWS account
• a Terraform CLI

You Want

After applying the Terraform module you get an S3 bucket which can be used to store AWS service logs.

Setup

The input variables for the module are defined in https://github.com/dumrauf/aws_log_bucket/settings/example.tfvars as

region = "us-east-1"

shared_credentials_file = "/path/to/.aws/credentials"

profile = "<your-profile>"

log_bucket_prefix = "<your-prefix>-"

Here, you need to replace the example values with your settings. Note that you also need to update the log_bucket_prefix as the current value is not a valid input.

Execution

Initialise Terraform by running

terraform init

As a best practice, create a new workspace by running

terraform workspace new example

The logging bucket can then be planned by running

terraform plan -var-file=settings/example.tfvars

and created by running

terraform apply -var-file=settings/example.tfvars

Outputs

The module has two outputs, namely log_bucket_id and bucket_domain_name which are the corresponding Terraform attributes of the newly created logging bucket.

Deletion

The logging bucket can be deleted by running

terraform destroy -var-file=settings/example.tfvars

Note that the actual logs in the logging bucket are not deleted by default. Hence, if the logging bucket still contains logs, deletion will fail. This is on purpose, as it requires you to remove the log files before being able to delete the logging Bucket.

AWS Logging Overview

For a detailed overview of AWS logging, see the excellent article on https://logmatic.io/blog/everything-you-need-to-know-about-aws-logging/.