Automating S3 file workflows with Python and boto3

Paginate every listing, from the first line of code

A list keys response returns a page of up to 1,000 keys with an indicator of whether the response was truncated. A script that reads Contents from a single list_objects_v2 call is therefore correct on a test bucket and silently wrong on a real one, processing the first thousand objects and reporting success. boto3’s paginator handles the continuation tokens for you, and there is no reason to write the loop by hand.

Pass Delimiter="/" when you want folder-style navigation rather than a flat walk. S3 then rolls keys sharing a prefix up into CommonPrefixes, so you can browse one level at a time instead of enumerating everything beneath it. AWS notes that list performance is not substantially affected by the total number of keys in the bucket, nor by the presence of the prefix, marker, maxkeys or delimiter arguments, so scoping a listing is about getting less data back, not about making S3 work harder.

import boto3

s3 = boto3.client("s3")
paginator = s3.get_paginator("list_objects_v2")

# Flat walk of a prefix, correct past 1,000 objects.
for page in paginator.paginate(
    Bucket="amzn-s3-demo-bucket", Prefix="contracts/2026/"
):
    for obj in page.get("Contents", []):
        print(obj["Key"], obj["Size"], obj["LastModified"])

# One level only: sub-prefixes arrive as CommonPrefixes.
for page in paginator.paginate(
    Bucket="amzn-s3-demo-bucket", Prefix="contracts/", Delimiter="/"
):
    for folder in page.get("CommonPrefixes", []):
        print("prefix:", folder["Prefix"])

Use the managed transfers instead of rebuilding multipart

upload_file and download_file are managed transfers. Anything above the multipart_threshold becomes a multipart operation automatically, split into multipart_chunksize parts and moved concurrently, with the defaults set to an 8,388,608-byte (8 MB) threshold and chunk size, max_concurrency of 10, and use_threads enabled. download_file parallelises the same way.

The low-level create_multipart_upload, upload_part and complete_multipart_upload calls are worth reaching for only when you need what multipart adds beyond throughput: uploading parts over time and resuming later, or beginning an upload before the final object size is known. Otherwise you are reimplementing retry, part accounting and completion logic that already exists, and inheriting the bug where a failed run leaves billable parts behind.

Set metadata through ExtraArgs at transfer time. Content-Type and Content-Disposition decide whether the object later opens in a browser or lands in a Downloads folder, and correcting them afterwards means copying the object over itself.

The upload limits and CLI equivalents in full

Credentials belong to the environment, not the script

boto3 resolves credentials through a provider chain, which is the feature that lets the same code run on a laptop, in a container and in a scheduled task without an if statement. Create clients with no explicit keys and let the chain find the task role, the instance profile or the local profile. An access key pasted into a script is a key in your shell history, your git history and your logs.

For anything touching another account’s bucket, assume a role scoped to that work rather than holding long-lived credentials. STS returns temporary credentials with an expiration, and a static credentials provider will not refresh them, so production code needs a lifecycle that re-assumes before expiry rather than one that discovers the problem at 3am.

Set retries explicitly instead of inheriting whatever the default retry mode gives you. botocore’s Config takes a max_attempts and a mode, and writing them down means the behaviour is reviewable.

import boto3
from botocore.config import Config

# No keys in code: the provider chain finds the role or profile.
s3 = boto3.client(
    "s3",
    config=Config(
        retries={"max_attempts": 5, "mode": "standard"},
        # Fail fast on a dead network rather than hanging a worker.
        connect_timeout=5,
        read_timeout=60,
    ),
)

How cross-account access works without keys

Presigned URLs from boto3, and their two hard limits

generate_presigned_url produces a time-limited URL for one object and one operation, usable without an AWS sign-in. It is the right tool for handing a single file to a single recipient, and you can override the stored response headers for that one request, which is how a mislabelled PDF is made to open inline without modifying the object.

Two limits decide whether it fits. The URL carries the permissions of whoever signed it and works until its expiration or those credentials expire, whichever comes first, so a URL signed with a short-lived task role can die well before the lifetime you asked for. And it is a bearer token: anyone who obtains it can use it while valid, and it does not record which person opened the file. If your requirement is an audit trail with a name on it, a presigned URL is the wrong primitive no matter how short you make it.

url = s3.generate_presigned_url(
    "get_object",
    Params={
        "Bucket": "amzn-s3-demo-bucket",
        "Key": "contracts/2026/acme-msa",
        # Override the stored headers for this response only.
        "ResponseContentType": "application/pdf",
        "ResponseContentDisposition": "inline",
    },
    ExpiresIn=300,
)

Previewing S3 files without downloading them

About S3 Select: closed to new customers

S3 Select let you run a SQL expression against a single object and get back only the matching rows. It still turns up in tutorials, so the current status is worth stating plainly. The AWS documentation says Amazon S3 Select is no longer available to new customers, and that existing customers can continue to use the feature as usual. If your account is not already using it, it is not something to design around.

Existing users should also know where it stops: one object per request, CSV, JSON or Parquet input, CSV or JSON output, a SQL expression up to 256 KB, records up to 1 MB, no objects in the Glacier storage classes or the Intelligent-Tiering archive tiers, and a 40 MB cap on what the console will return.

For the common cases there are ordinary answers. To query across many objects, use a query service over the data in place. To transform an object on the way out of S3, that is what S3 Object Lambda is for. And when all you actually need is a slice of one large file, a Range GET fetches the bytes you want without any query engine at all.

Let lifecycle rules do the deleting

A nightly script that deletes objects older than N days is a script with s3:DeleteObject permission, a cron schedule and a prefix it computes itself. A lifecycle configuration expresses the same intent declaratively, runs without your code, and is reviewable in the console by someone who does not read Python.

Three rules are worth having almost everywhere: transition objects to a colder class on a schedule, expire what genuinely should not be kept, and abort incomplete multipart uploads after a few days so cancelled transfers stop accruing storage charges. Moving that work out of your script also shrinks the IAM policy your automation needs, which is the part that matters when something is compromised.

What happens when you need an archived file back

Draw the approval boundary before you automate a change

Reading, listing, copying and reporting are safe to automate freely. Deleting, overwriting and sharing are not, and the difference is not about how good the code is. A script that computes which objects to remove will eventually compute the wrong set, and the blast radius is decided by the permissions you gave it, not by the bug.

The guards are unglamorous and effective: enable versioning so an overwrite is recoverable, use conditional writes when a key must not be clobbered, give every destructive job a dry-run mode that is the default, and keep the policy that decides what may happen to customer files separate from the code that proposes it. BucketDesk applies the same split deliberately, letting automation read and recommend while a deterministic policy governs what can actually change.

Automation, agents and the approval boundarySee the workspace side of this

• Automate: listing, inventory, copying, metadata repair, reporting, notification.
• Automate with a dry run and versioning behind it: transitions, re-keying, bulk metadata changes.
• Keep behind an explicit human approval: deletion, overwriting a current version, sharing outside the organisation.