From PHP to Go: Evolving Archive Processing on AWS Batch

Start with the extra I/O

The current Batch worker checks the source object, downloads the full archive, and opens it with PHP’s ZipArchive. For each entry, it copies the entry stream into a seekable temporary file before uploading. Large entries already use multipart upload, but the extracted file is still staged first.

That staging gives the upload path bytes it can rewind and retry. It also means the worker needs space for the archive plus a staged entry. Illustratively, a 10 GB archive containing a 2 GB extracted file can require roughly 12 GB of temporary space at that point, before overhead. This is a capacity example, not a benchmark.

Changing languages alone does not remove this I/O. The useful change is how the reader, decompressor, retry buffers, and uploader fit together.

Current path
S3 ZIP → full local archive → extracted temporary file → S3

First Go phase
S3 ZIP → full local archive → decompressor → bounded upload buffers → S3

Later ZIP phase
S3 ranges → ZIP reader → decompressor → bounded upload buffers → S3

PHP coordinates; Go moves archive bytes

PHP/Laravel remains responsible for authentication, workspace authorization, job creation, audit records, AWS Batch submission, job state, cancellation requests, notifications, and the UI/API. Go executes an approved inspection or extraction job; it does not become a second application control plane.

The target worker runs on AWS Batch with Fargate in the customer’s AWS account and Region. Archive input, extracted output, and temporary processing stay within that customer-account boundary. Only the minimum authorized job metadata and sanitized status callbacks return to the control plane. This boundary describes archive processing, not every other BucketDesk feature.

A versioned archive-worker-contract-v1 should capture the job and attempt IDs, operation, approved source and destination, object version or ETag and size, selection, limits, collision policy, and callback protocol. The same contract allows PHP and Go workers to coexist during rollout. Credentials must come from workload identity, not a manifest containing long-lived keys.

PHP/Laravel control plane
  authorize → create job → submit Batch → track status
                         ↓
Customer account / customer Region
  Go Batch worker → ZIP reader → decompressor → multipart upload
                         ↕
                S3 Gateway VPC Endpoint
                         ↕
                    Customer S3

Stream into multipart uploads with backpressure

The first Go milestone can retain the local source ZIP while removing full extracted-entry staging. The decompressed entry reader feeds an uploader that retains bounded, replayable parts. When upload capacity is full, backpressure stops further reads instead of accumulating the whole extracted file.

Streaming does not mean zero buffering. A failed part needs the same bytes again. Keep its buffer until the request succeeds or the job fails, and release it promptly afterward. Preserve the entry reader’s final checksum and size errors; an object must not be treated as successful before decompression and validation finish.

Choose part sizes against the allowed entry size and S3’s part-count limit. Non-final parts must be at least 5 MiB, and a multipart upload supports at most 10,000 parts. A fixed small part size is therefore not suitable for every allowed file. The final part may be smaller.

Complete an upload only after all required parts and validations succeed. On failure or cancellation, stop producers, close readers, and abort outstanding multipart uploads. A lifecycle rule for incomplete uploads supplies cleanup if the worker dies before it can abort.

Bound concurrency across the whole worker

Go makes concurrent work convenient; a production worker still needs an explicit budget. Limit active entries, concurrent range requests, queued work, and multipart uploads together. An uploader’s concurrency setting applies per upload call, so several entries can multiply the number of active requests.

For planning, estimate upload buffers as active entries × parts in flight per entry × part size. Two entries with four 16 MiB parts each suggest about 128 MiB for those buffers alone. Add ZIP metadata, range caches, decompression state, SDK overhead, and runtime headroom. This is an illustrative budget, not a memory guarantee.

Use bounded queues and a shared request budget. Apply deadlines and capped retries with backoff; cancel the full pipeline when the job loses authorization, exceeds limits, or receives a cancellation request. Tune concurrency against observed CPU, memory, S3 throttling, and request cost.

Use the S3 Gateway VPC Endpoint for the data path

The network phase associates an S3 Gateway VPC Endpoint with the worker subnet route tables in the same Region as the buckets. Eligible S3 traffic can then reach S3 without traversing a NAT gateway or internet gateway. AWS does not add an endpoint charge for gateway endpoints; normal S3 and other applicable charges remain.

This does not automatically give the entire job a network path to every dependency. Container image pulls, CloudWatch Logs, STS calls where used, and the application callback need their own reachable paths. Review those dependencies before removing outbound internet access.

aws:SourceVpce is intentionally not part of the current implementation. This migration does not add or enforce an aws:SourceVpce restriction in customer bucket policies. Endpoint routing and bucket authorization are separate decisions: the endpoint provides the S3 path, while scoped IAM and existing bucket policies continue to govern access. An endpoint policy can further restrict traffic through that endpoint.

Consequently, the endpoint is not a claim that all access from other authorized paths is denied. Customers retain their existing access arrangements; endpoint-only enforcement is outside the current scope.

Inspect ZIP metadata with S3 Range GETs

The next phase replaces the local archive with an S3-backed random-access reader. ZIP stores a central directory describing its entries near the end of the object. Inspection can fetch a bounded tail, locate and validate the end-of-central-directory record, then fetch the central directory rather than download every compressed entry.

The reader must handle variable-length comments and ZIP64 records, validate offsets and lengths against the object size, and cap directory bytes and entry counts before expensive allocation. A large central directory is still a resource cost. Listing metadata also does not prove that every entry’s compressed data is valid.

Go’s archive/zip.NewReader accepts an io.ReaderAt plus the archive size. An S3 adapter can satisfy that interface with validated byte ranges, bounded caching, and concurrency-safe reads. It must handle short responses and EOF correctly and avoid turning tiny reads into an uncontrolled request storm.

Pin the source identity for the entire job: use the same non-null VersionId where available, or require the captured ETag with If-Match on every read and validate size. ETag is an object validator, not a universal content hash. If the source changes, fail the attempt rather than combine bytes from different objects.

Select entries without reading the entire archive

After inspection and authorization, use the central-directory entry to locate its local file header. Validate that header and calculate the compressed payload offset, including variable filename and extra-field lengths. Read the selected entry’s compressed bytes, decompress them, and stream the result into the approved destination.

Range selection works at ZIP-entry boundaries. It does not generally let a worker jump to the middle of a deflated entry and resume decompression there. Extracting one large compressed entry may still require reading that entire entry from its beginning.

S3 accepts one range per GetObject request, not multiple disjoint ranges in a single request. Coalescing nearby reads can reduce requests, but caches and read-ahead must stay bounded. Full extraction may read most of the archive anyway; selective reads are most useful when only a subset is needed.

Unsupported compression, encryption, split archives, and malformed metadata need explicit outcomes. Use a documented, bounded fallback only for supported cases that pass the same controls; otherwise return a clear unsupported-archive error. Never bypass validation to force an archive through.

Checkpoint completed entries, not just bytes sent

Checkpointing makes retries useful only when a checkpoint proves that the intended output is complete. The current PHP flow already checks for verified completed outputs. Preserve that behavior in Go and bind durable records to the job, source identity, entry identity, destination, output size, and integrity evidence.

Write a completed-entry checkpoint after the object is committed and verified. On restart, reconcile S3 with the record before skipping an entry. A crash after S3 completion but before checkpoint persistence must not cause an unsafe overwrite or an unexplained permanent collision.

For the first Go phase, retry an interrupted entry from its beginning while skipping previously verified entries. Retaining an upload ID and part list can support later multipart recovery, but it does not restore a deflate decoder’s state. Without a validated restart strategy, decompression must replay from the entry start.

Define collision handling at commit time. A destination HEAD check alone has a race with another writer. Use supported conditional writes, including conditional multipart completion where applicable, or an explicitly designed staging and publication protocol. Make status callbacks idempotent and reject stale attempts so retries cannot overwrite a terminal job state.

Keep security controls on both sides of the contract

The control plane approves scope; the worker independently enforces it. Treat archive metadata and entry names as untrusted input, even when S3 access was authorized. Validate the destination prefix after normalization and reject traversal, absolute paths, ambiguous separators, duplicate destinations, and unsupported links or special entries.

Keep limits on entry count, per-entry output, total expanded bytes, expansion ratio, runtime, and request volume. Enforce actual bytes while decompressing, not just sizes declared in the central directory. A small archive can otherwise expand into an unexpectedly large workload.

Use a least-privilege job role for approved S3 reads, destination writes, required multipart cleanup, and applicable KMS operations. Keep the container execution role distinct from the job role. Preserve TLS, encryption requirements, and cross-account trust controls.

Callbacks need authentication, job/attempt binding, replay handling, and a fixed approved destination. Logs must exclude credentials, callback tokens, signed URLs, archive contents, and sensitive raw object names. Cleanup should remove only artifacts owned by the job, never unrelated destination objects.

Measure each phase before widening the rollout

Sanitized structured CloudWatch events should identify the job, attempt, worker version, phase, duration, and error code. Track compressed bytes fetched, bytes expanded and uploaded, range request counts, retries, peak memory, temporary disk use, multipart aborts, checkpoint skips, and callback failures. Keep detailed archive metadata inside the approved customer boundary.

Compare equivalent jobs by archive shape, selected-entry count, compression method, encryption, and Region. Separate Batch queue and startup time from processing time. Reduced local disk use can be valuable even when CPU-bound decompression limits throughput; more range requests can trade transfer savings for request cost.

Roll out through explicit gates rather than switching every customer at once:

• 1. Freeze the worker contract and document existing authorization, callback, retry, collision, and cleanup behavior.
• 2. Add Go behind a worker-selection flag, initially using a local source archive with streaming output. Compare bytes and terminal states against PHP using isolated test destinations.
• 3. Validate endpoint routing and all remaining network dependencies in a customer test stack.
• 4. Enable range-based inspection for supported ZIPs, then selective extraction as a separate gate.
• 5. Exercise source replacement, corrupt ZIPs, oversized expansion, destination races, cancellation, transient S3 failures, and crashes around upload completion and checkpoint writes.
• 6. Expand from opted-in canaries only after correctness and resource budgets hold. Retain the PHP path for eligible new jobs while Go stabilizes; resolve running attempts and output ownership before retrying across implementations.