Handling Large File Uploads: R2 Multipart Upload Strategies

Previously in this course, we explored working with KV storage for fast, global state. While KV is perfect for metadata, it isn't designed for large binary objects. In this lesson, we move from small key-value pairs to heavy lifting: uploading large files directly to R2 using multipart uploads to ensure stability and performance.

The Challenge of Large Uploads

When you upload a file via a standard HTTP POST request, the entire payload must pass through the Worker. This approach has two hard limits: the maximum request size and the execution time limit. If your user tries to upload a 500MB video, the Worker will time out or hit memory limits long before the upload finishes.

Multipart uploads solve this by splitting a single file into smaller, independent parts (usually 5MB to 50MB each). You upload these parts individually to R2 and then trigger a "complete" command to assemble them into the final object. This makes your uploads resilient; if one chunk fails, you only need to retry that specific chunk, not the entire file.

Implementing Multipart Upload Logic

To handle large files, we coordinate the process in three phases:

1. Initiation: Request an uploadId from R2.
2. Chunking: Upload individual parts to the R2 bucket.
3. Completion: Send the list of part numbers and their ETag headers to R2 to finalize the object.

Here is a simplified example of how you might initiate and upload a part from a client-side perspective, interacting with your R2-bound Worker:

JAVASCRIPT
// Worker logic: Initiating an upload
export async function initiateUpload(bucket, key) {
  const upload = await bucket.createMultipartUpload(key);
  return upload.uploadId;
}

// Worker logic: Handling a specific chunk
export async function uploadPart(bucket, key, uploadId, partNumber, body) {
  const upload = bucket.resumeMultipartUpload(key, uploadId);
  return await upload.uploadPart(partNumber, body);
}

Managing Timeouts and Progress Tracking

Because R2 R2 handles the heavy lifting, your Worker acts primarily as a traffic controller. To keep the user informed, you should track progress on the client side.

Since HTTP doesn't natively "stream" progress percentages for a single request easily, we use the XMLHttpRequest or fetch API with a ReadableStream.

For a robust implementation, generate a "Presigned URL" for each chunk. This allows the client to upload directly to R2, bypassing the Worker's CPU time entirely, which is the most effective way to manage timeout constraints.

Hands-on Exercise: Chunking a File

1. In your project, create a new endpoint /upload-init that calls bucket.createMultipartUpload().
2. On your frontend, use the Blob.slice() method to split a user-selected file into 5MB chunks.
3. For each chunk, call your Worker to get a signed URL (or proxy the upload if you are just starting).
4. Use Promise.all to manage the uploads and log the progress to the console as each chunk finishes.

Common Pitfalls

• Part Size Errors: R2 requires most parts to be at least 5MB. If you try to upload a 1MB chunk (except for the final part), the API will return a 400 error.
• Missing ETags: Every time you upload a part, R2 returns an ETag. You must store these ETags in your database (perhaps alongside the metadata you learned to store in managing database connections) to complete the multipart upload successfully.
• Orphaned Uploads: If a user cancels an upload halfway through, the "in-progress" multipart upload remains in R2. Use Lifecycle Policies to automatically clean up these incomplete uploads after a few days to avoid unnecessary storage costs.

FAQ

Q: Can I upload a 10GB file? A: Yes, R2 supports very large files, but you must use the multipart upload API. Standard single-request POSTs will fail for files of that size.

Q: Do I need to reassemble the file myself? A: No. R2 handles the assembly process on their servers once you provide the final list of part ETags.

Q: Does this count against my Worker CPU limit? A: If you stream the data through the Worker, yes. If you generate signed URLs and let the client upload directly to R2, your Worker CPU usage remains near zero.

Recap

We’ve learned that large file handling is less about raw throughput and more about reliable orchestration. By breaking files into chunks and using multipart APIs, we move from brittle, timeout-prone processes to resilient, scalable uploads. Your application can now handle high-fidelity assets without crashing under the weight of large payloads.

Up next: We will discuss Global State Management and how to keep your data consistent across different geographic regions.