Handling Large File Uploads: Streaming to S3 & Async Processing

Previously in this course, we explored testing DDD components to ensure our business logic remains predictable and isolated. This lesson shifts our focus to infrastructure concerns, specifically handling high-volume, large-file I/O without blocking our PHP workers or exhausting server memory.

When building a high-traffic SaaS platform, treating file uploads as simple request-response cycles is a recipe for disaster. If you've previously followed our beginner's guide to file uploads, you know the basics; now, we're moving into production-grade architecture where Storage, S3, and Performance are the primary constraints.

The Architecture of Scalable Uploads

The biggest bottleneck in file handling isn't the storage medium itself—it's the PHP execution lifecycle. When you upload a 500MB file through a standard controller, you are effectively holding a worker process hostage while the binary data travels from the user's browser, through your load balancer, into PHP memory, and finally to your storage driver.

To scale, we must decouple the transport from the processing.

1. Direct-to-S3 Streaming

Instead of routing binary data through your application server, we leverage S3 Pre-signed URLs. This allows the client to upload directly to your S3 bucket. Your Laravel application merely orchestrates the permission, keeping the web server's I/O overhead near zero.

2. Asynchronous Processing

Once the file is in S3, we trigger an asynchronous event. Whether it's virus scanning, image optimization, or metadata extraction, this work must happen in a background queue. This keeps the user's request cycle responsive, even for massive files.

Worked Example: Secure, Streamed Uploads

We'll implement a secure upload flow using Pre-signed URLs.

Step 1: Generate the Pre-signed URL

In your UploadService, generate a URL that allows the client to PUT the file directly to your bucket.

PHP
namespace App\Modules\Storage\Services;

use Illuminate\Support\Facades\Storage;

class UploadService
{
    public function getPresignedUrl(string $path): string
    {
        #6A9955">// We define a 10-minute window for the upload
        return Storage::disk('s3')->temporaryUploadUrl(
            $path, 
            now()->addMinutes(10)
        );
    }
}

Step 2: Client-side Upload

The frontend receives this URL and uses the browser's fetch API to push the file. Because the request goes straight to AWS, your Laravel server isn't waiting for the stream to finish.

Step 3: Trigger Background Processing

Once the upload completes, the client hits an endpoint to notify your system. Your controller shouldn't process the file; it should only dispatch a job.

PHP
#6A9955">// In your Controller
public function finalize(Request $request)
{
    $request->validate(['file_path' => 'required|string']);
    
    #6A9955">// Dispatch to a high-priority queue specifically for I/O
    ProcessUploadedFile::dispatch($request->file_path)
        ->onQueue('file-processing');
        
    return response()->json(['message' => 'Processing started']);
}

Hands-on Exercise

1. Infrastructure: Configure your filesystems.php to use the s3 driver with the league/flysystem-aws-s3-v3 package.
2. Implementation: Create an Action class GenerateUploadUrlAction that returns a signed URL for a specific user-owned directory.
3. Queueing: Create a ProcessUploadedFile job that uses Storage::disk('s3')->get($path) to pull the file into a temporary local path for processing (e.g., using Intervention Image or FFmpeg).

Common Pitfalls

• Memory Exhaustion: Never attempt to read a large file into a variable using file_get_contents(). Always use streams (fopen and stream_copy_to_stream) to process files in chunks.
• Security: Never trust the user-provided filename. Always generate a UUID or a hash-based path for storage to prevent directory traversal or file overwriting attacks.
• Queue Bottlenecks: If you process thousands of files, don't use your default queue. Create a dedicated queue (file-processing) and assign it a specific worker pool so that long-running file tasks don't delay transactional emails or API responses.

Recap

Handling large files effectively requires moving away from the synchronous request-response model. By using S3 pre-signed URLs, you remove your application from the data path, and by offloading file logic to background queues, you maintain sub-millisecond response times for your API.

Next, we will look at how to optimize the delivery of these assets to the end user by integrating robust caching and versioning strategies.

Up next: Optimizing Asset Pipelines