AWS S3

Ein paar Kniffe und Tricks für AWS S3

Buckets listen

aws s3 ls
aws s3api list-buckets # json output

File upload

aws s3 cp <local> s3://<bucketname>/path/to/file

File download

aws s3 cp s3://<bucketname>/path/to/file <local>

Files auflisten nach Name filtern

aws s3api list-objects --bucket <bucket> --query "Contents[?contains(Key, `path/to/FILENAME`)]"
aws s3api list-objects --bucket <bucket> --query "Contents[?contains(Key, `FILENAME`)]"

Files suchen und sortieren nach letzter Bearbeitung

aws s3api list-objects --bucket <bucket> \
  --query "Contents[?contains(Key, `MyScan`)] | sort_by(@, &LastModified)[].Key"

Alle bestehenden Objekte mit KMS Verschlüsseln

aws s3 cp --recursive --sse aws:kms \
  --sse-kms-key-id arn:aws:kms:eu-central-1:... s3://<bucket>/ s3://<bucket>/

Download eines ganzen Ordners

aws s3 cp --recursive \
  s3://<bucket>/AWSLogs/elasticloadbalancing/eu-central-1/2020/08/12/ logs/

Terraform

Dinge auf die man achten möchte:

Lifecycle Rule
Serverside Encryption (KMS vs AES)
Versioning
Public Access (Block)
Access Logging

module "s3bucket-bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "1.6.0"

  bucket        = "bucket-${local.environment}-${local.region}"
  acl           = "private"
  force_destroy = false

  logging = {
    target_bucket = local.log_bucket
    target_prefix = "s3/bucket-${local.environment}-${local.region}"
  }

  block_public_acls   = true
  block_public_policy = true

  lifecycle_rule = [
    {
      abort_incomplete_multipart_upload_days = 7
      enabled                                = true
      id                                     = "moveToIntelligentTieringAfter7Days"
      transition = [
        {
          days          = 7
          storage_class = "INTELLIGENT_TIERING"
        }
      ]
    },
    {
      abort_incomplete_multipart_upload_days = 0
      enabled                                = true
      id                                     = "removePreviousVersionsAfter90Days "
      noncurrent_version_expiration = {
        days = 90
      }
    }
  ]

  versioning = {
    enabled = true
  }

  server_side_encryption_configuration = {
    rule = {
      apply_server_side_encryption_by_default = {
        sse_algorithm = "AES256"
      }
    }
  }

  # Bucket Name
  tags = merge(
    local.common_tags,
    {
      "Name" = "bucket-${local.environment}-${local.region}"
    },
  )
}

S3 Cloudwatch Delivery

Das ist eine Bucket Policy, die es erlaubt einen Cloudwatch Stream in den gewählten Bucket zu exportieren.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"Service": "logs.eu-central-1.amazonaws.com"
			},
			"Action": "s3:GetBucketAcl",
			"Resource": "arn:aws:s3:::<bucket>"
		},
		{
			"Effect": "Allow",
			"Principal": {
				"Service": "logs.eu-central-1.amazonaws.com"
			},
			"Action": "s3:PutObject",
			"Resource": "arn:aws:s3:::<bucket>/*",
			"Condition": {
				"StringEquals": {
					"s3:x-amz-acl": "bucket-owner-full-control"
				}
			}
		}
	]
}