S3 event notifications: run code when a file is uploaded to Amazon S3

Four destinations, and how to choose

S3 can send an event to an Amazon SNS topic, an Amazon SQS queue, an AWS Lambda function, or Amazon EventBridge. The first three are configured per rule on the bucket. Each rule has exactly one destination, the destination must be in the same Region as the bucket, and SQS FIFO queues and SNS FIFO topics are not supported. Directory buckets do not support event notifications at all.

EventBridge works differently. It is a single switch per bucket. Turn it on and S3 sends every event type to the default event bus, where you write rules that match on the event and route it to one or more targets. AWS says changes to that switch take about five minutes to apply.

The practical choice is usually between a direct Lambda trigger and a queue. Lambda is the shortest path for small, fast work. A queue in front of the function absorbs bursts, lets you retry failed files and gives you a dead-letter queue to inspect. EventBridge is the right call when several teams want the same events, when you need a FIFO queue, or when a prefix and suffix are not enough to describe which files you care about.

• Lambda: quickest for small jobs, one function per rule.
• SQS: buffer bursts and retry; standard queues only, at least 64 KB maximum message size.
• SNS: fan out to several subscribers.
• EventBridge: one switch per bucket, all event types, rules route to many targets.

Which events to listen for

Uploads fire s3:ObjectCreated events. There are four kinds, named after the API that created the object: Put, Post (browser form uploads), Copy, and CompleteMultipartUpload. Large files uploaded by the CLI or an SDK use multipart upload, so a rule that only listens for Put will miss them. Use s3:ObjectCreated:* unless you have a reason not to.

Other event families cover deletes (s3:ObjectRemoved, which does not fire for lifecycle deletes), Glacier restores (s3:ObjectRestore:Completed tells you a restored copy is ready), replication failures, lifecycle expirations and transitions, Intelligent-Tiering archive moves, tag changes and ACL changes. In EventBridge the same events have readable names such as Object Created and Object Deleted, and the detail.reason field says which API created the object.

How to upload files to Amazon S3 from the console, the CLI or an SDKHow to restore files from S3 Glacier Flexible Retrieval and Deep Archive

• s3:ObjectCreated:* catches Put, Post, Copy and CompleteMultipartUpload.
• s3:ObjectRemoved:* covers deletes and delete markers, not lifecycle deletes.
• s3:ObjectRestore:Completed fires when a Glacier restore finishes.
• s3:LifecycleExpiration:* is the event for lifecycle deletes.

Filters, and the overlap rule

A native rule can filter only on the start and end of the object key: a prefix such as incoming/ and a suffix such as .csv. Wildcards are not allowed, so incoming/*/report.csv cannot be expressed. Anything finer, such as file size or a specific uploader, means filtering in your code or using EventBridge rules.

S3 also refuses overlapping rules for the same event type. Two ObjectCreated rules with prefixes incoming/ and incoming/csv/ overlap, and so does a rule with no filter against any other rule. Overlapping prefixes are allowed when the suffixes don't overlap, so incoming/ with .csv and incoming/ with .pdf can go to different functions. Different event types may overlap freely.

• Prefix and suffix only, no wildcards.
• A space in a filter value is written as +, and other special characters are URL-encoded.
• Same event type, overlapping filters: rejected when you save.
• A rule with no filter overlaps everything for that event type.

Example: trigger a Lambda function for new CSV files

Two calls wire a bucket to a function. First give S3 permission to invoke the function, scoped to your bucket and account. Then write the notification configuration. The console does the first step for you; the CLI does not.

The second call replaces the bucket's entire notification configuration. If the bucket already sends events somewhere, read the current configuration with get-bucket-notification-configuration and include it, or you will silently remove it. When you save, S3 checks every destination and saves nothing if one of them fails, which is also how a rule pointing at a deleted function can block new ones.

aws lambda add-permission \
  --function-name process-incoming-csv \
  --statement-id s3-invoke-incoming \
  --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::amzn-s3-demo-bucket \
  --source-account 123456789012

cat > notification.json <<'EOF'
{
  "LambdaFunctionConfigurations": [
    {
      "Id": "incoming-csv",
      "LambdaFunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:process-incoming-csv",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            { "Name": "prefix", "Value": "incoming/" },
            { "Name": "suffix", "Value": ".csv" }
          ]
        }
      }
    }
  ]
}
EOF

aws s3api get-bucket-notification-configuration --bucket amzn-s3-demo-bucket
aws s3api put-bucket-notification-configuration \
  --bucket amzn-s3-demo-bucket \
  --notification-configuration file://notification.json

What the function receives

The event is JSON with a Records array. Each record carries the event name without the s3: prefix (ObjectCreated:Put), the time, the bucket name and ARN, and the object's key, size, ETag, a sequencer, and the version ID if the bucket is versioned.

The key is URL-encoded. A file called Q3 report.pdf arrives as Q3+report.pdf, so code that passes the raw key to GetObject gets a NoSuchKey error on any file name with a space. Decode it first: urllib.parse.unquote_plus in Python, or replace + with a space and call decodeURIComponent in JavaScript.

cat > handler.py <<'EOF'
import urllib.parse
import boto3

s3 = boto3.client("s3")

def handler(event, context):
    for record in event["Records"]:
        bucket = record["s3"]["bucket"]["name"]
        key = urllib.parse.unquote_plus(record["s3"]["object"]["key"])
        size = record["s3"]["object"].get("size", 0)
        obj = s3.get_object(Bucket=bucket, Key=key)
        # Write results under a different prefix than the one this rule watches.
        s3.put_object(Bucket=bucket, Key="processed/" + key.split("/", 1)[-1] + ".done", Body=b"")
        print(f"processed s3://{bucket}/{key} ({size} bytes)")
EOF

Automate S3 file workflows with Python and boto3

Duplicates, ordering and delay

S3 delivers events at least once. Most arrive within seconds, AWS says some take a minute or longer, the order is not guaranteed, and a retry can occasionally deliver the same event twice. Code that sends an email or charges a customer on every event will eventually do it twice.

Make the handler idempotent. Record the bucket, key and version ID (or ETag) you have processed, and skip repeats. When two events for the same key arrive out of order, the sequencer field tells you which is later: left-pad the shorter value with zeros and compare the strings. Sequencers are only comparable for the same key.

• At least once, not exactly once.
• Usually seconds, sometimes a minute or more.
• No ordering guarantee; use sequencer per key.
• Key off bucket, key and version ID to skip repeats.

The loop that writes back into the same bucket

The classic mistake is a function that listens for ObjectCreated on a bucket and writes its output into the same bucket and prefix. Each output file triggers the function again. AWS's advice is to use a separate output bucket, or to scope the trigger to an incoming-only prefix and write results elsewhere, as the example above does.

Lambda now has recursive loop detection for S3. After roughly 16 invocations in the same chain it stops the next call and notifies you, and the RecursiveInvocationsDropped metric counts what it stopped. It needs a recent SDK in the function (boto3 1.24.46 or later, or the JavaScript SDK v3 3.105.0 or later). Treat it as a safety net, not the design.

Automation, agents and the approval boundary

• Watch incoming/, write to processed/ or another bucket.
• Lambda loop detection stops a chain after about 16 calls.
• It only works with supported SDK versions.

What it costs

S3 does not charge for event notifications. You pay for what receives them: Lambda invocations and duration, SQS requests, SNS deliveries. AWS service events on the EventBridge default bus are free to publish; rules that deliver to another account, and custom events, are billed. Check the EventBridge pricing page for the current terms before you route high volumes through it.

The bigger cost is usually the work the function does. A function that downloads every uploaded file to inspect it pays for the GET and the transfer if it runs in another Region, so filter on prefix and suffix before the function starts rather than inside it.

Amazon S3 pricing explained: what a business file archive really costs

Where BucketDesk fits

Event notifications are for code. The people uploading the files usually never see them; they see a folder. BucketDesk is the browser workspace on top of that folder: a team opens the bucket, previews what arrived, shares it, and asks a document a question with Document AI, all through a scoped IAM role in your own AWS account.

Automations that react to uploads are on the BucketDesk roadmap and not in any plan yet. Until they ship, the pattern above is how you would trigger processing, and BucketDesk is where people look at the input and the processed/ output side by side.

Access Amazon S3 files without the AWS console