Amazon S3 Sources

An S3 source watches an Amazon S3 bucket for newly-delivered log objects and streams their lines into LogRaker. It's built for the logs that land in S3 as files - ALB / ELB and CloudFront access logs, CloudTrail, S3 server access logs, and anything shipped via Kinesis Firehose. Add one from the sidebar's + menu ▸ New Amazon S3 Source….

How it works

S3 has no "tail" API - objects are immutable and appear whole. So LogRaker polls: it lists the bucket/prefix on an interval, fetches any new objects in key order, decompresses gzip automatically (detected by the file's content, not its extension), and appends the lines. Because delivery to S3 is itself batched, expect the view to update minutes behind live - typically 1-5 minutes, set by the producer (ALB flushes about every 5 minutes; Firehose buffers 1-5). That's a delivery-side reality, not a LogRaker delay.

Region & credentials

S3 uses the same AWS credentials as an AWS CloudWatch source - the same account, signed the same way, unlocked by the same Keychain secret. So this editor's Region and Auth controls work exactly as they do there:

Access Key - an IAM access-key ID and secret. The secret is stored in your Keychain, never in the source's saved data or a configuration backup.
Shared Profile - a named profile from your ~/.aws/credentials file. Only the selected profile's keys are stored, never the whole file.

Both methods sync via iCloud Keychain so a source set up once connects on your other Macs. See Authentication and iCloud Sync for the full details - they're identical here.

The Region is where the bucket lives. If you get it wrong, LogRaker detects the redirect S3 returns and retries in the correct region automatically, so a mismatch self-corrects.

Permissions (IAM)

The identity you authenticate as needs two S3 actions, ideally scoped to the bucket and prefix you'll read:

s3:ListBucket - list objects under the prefix (the poll).
s3:GetObject - fetch each object's contents.

A minimal, prefix-scoped policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::my-logs-bucket",
      "Condition": { "StringLike": { "s3:prefix": ["AWSLogs/*"] } }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::my-logs-bucket/AWSLogs/*"
    }
  ]
}

s3:ListAllMyBuckets is optional - it's used only by Test Connection below. Streaming works without it.

Test Connection

Click Test Connection. LogRaker signs a real request against the chosen region. Three outcomes:

✓ Connected - the credentials are valid and the region is reachable.
Credentials valid - limited access (yellow) - the keys authenticated, but this identity can't list buckets (s3:ListAllMyBuckets). Streaming still works as long as it can read the bucket (s3:ListBucket + s3:GetObject on the prefix).
✗ Test failed - wrong keys, wrong region, no network, or another error. The result carries S3's own message.

Streams under an S3 source

A stream under an S3 source watches one bucket + prefix - see Amazon S3 Streams for the bucket/prefix fields, the poll interval, and how seeding on connect works.