Managing Long Running Tasks

The Solution

Async Worker Pool Architecture

Async Worker Pool Scale

Trade-offs

What you gain

• Fast user response times - API calls return in milliseconds instead of timing out after 30 seconds. Users get immediate acknowledgment that their request was received.
• Independent scaling - Web servers and workers scale separately. Add more workers during peak processing times without paying for idle web servers.
• Fault isolation - A worker crash processing one video doesn't bring down your entire API. Failed jobs can be retried without affecting user-facing services.
• Better resource utilization - CPU-intensive workers run on compute-optimized instances. Memory-heavy tasks get high-memory machines. Web servers use cheap, general-purpose instances.

What you lose

• System complexity - You now have queues, workers, and job status tracking to manage. More moving parts means more things that can break.
• Eventual consistency - The work isn't done when the API returns. Users might see stale data until background processing completes.
• Job status tracking - You need infrastructure to store job states, handle retries, and expose status endpoints. This adds database load and API complexity.
• Monitoring overhead - Queue depth, worker health, job failure rates, processing latency - you're monitoring a distributed system instead of simple request/response cycles.

How to Implement

1. Message Queue : A message queue is a system that allows you to send and receive messages between different parts of your system. It's a way to decouple the sender and receiver of a message, allowing them to operate independently.
2. Pool of Workers : A pool of workers is a group of processes that are dedicated to processing messages from a queue. They are responsible for pulling messages from the queue, processing them, and updating the job status.

Message Queue

Async Worker Pool Message Queue

Workers

# Simplified worker loop
while True:
    job = queue.pop()  # Blocks until job available
    if job:
        process_job(job)
        mark_complete(job.id)

Putting It Together

1. Web server validates the request and creates a job record in your database with status "pending"
2. Web server pushes a message to the queue containing the job ID (not the full job data)
3. Web server returns the job ID to the client immediately
4. Worker pulls message from queue, fetches job details from database
5. Worker updates job status to "processing"
6. Worker performs the actual work
7. Worker stores results (in S3 for files, database for metadata, etc.)
8. Worker updates job status to "completed" or "failed"

Async Worker Pool Putting It Together

When to Use in Interviews

1. When they mention specific slow operations - The moment you hear "video transcoding", "image processing", "PDF generation", "sending bulk emails", or "data exports" that's your cue. These operations take seconds to minutes. Jump in immediately: "Video transcoding will take several minutes, so I'll return a job ID right away and process it asynchronously."
2. When the math doesn't work - If they say "we process 1 million images per day" and you know image processing takes 10 seconds, do the quick calculation out loud: "That's about 12 images per second, which means 120 seconds of processing time per second. We'd need 120+ servers just for image processing. I'll use async workers instead."
3. When different operations need different hardware - If the problem involves both simple API requests and GPU-heavy work (like ML inference or video processing), that's a clear async signal. "We shouldn't run GPU workloads on the same servers handling login requests. I'll separate these with async workers on GPU instances."
4. When they ask about scale or failures - Questions like "what if a server crashes during processing?" or "how do you handle 10x traffic?" are perfect openings to introduce async workers. "With async workers, if one crashes mid-job, another worker picks it up from the queue. No user requests are lost."

For Example

Common Deep Dives

Handling Failures

Handling Repeated Failures

Async Worker Pool Dead Letter Queue

Preventing Duplicate Work

def submit_job(user_id, job_type, job_data, idempotency_key):
    # Check if job already exists
    existing_job = db.get_job_by_key(idempotency_key)
    if existing_job:
        return existing_job.id
    
    # Create new job
    job_id = create_job(user_id, job_type, job_data)
    db.store_idempotency_key(idempotency_key, job_id)
    queue.push(job_id)
    return job_id

Async Worker Pool Idempotency

Managing Queue Backpressure

Handling Mixed Workloads

queues:
  fast:
    max_duration: 60s
    worker_count: 50
    instance_type: t3.medium
    
  slow:
    max_duration: 6h
    worker_count: 10
    instance_type: c5.xlarge

Async Worker Pool Mixed Workloads

Orchestrating Job Dependencies

{
  "workflow_id": "report_123",
  "step": "generate_pdf",
  "previous_steps": ["fetch_data"],
  "context": {
    "user_id": 456,
    "data_s3_url": "s3://bucket/data.json"
  }
}

Conclusion

Flow Chart