AI Image Throughput Planning

Definition. Sizing the compute and concurrency needed to process an image workload within its required time, accounting for per-image processing time, batch efficiency and GPU memory limits. Throughput planning determines whether a pipeline can meet its deadlines and what it costs.

How it works. Measure per-image time for each step on representative hardware. Account for batching efficiency, since GPU throughput improves with batch size up to a memory limit. Compute required concurrency as arrival rate multiplied by processing time. Add headroom for bursts, and test with a load test rather than arithmetic alone.

Example. A pipeline must process 500,000 images overnight in 8 hours, requiring 17.4 images per second. Measured throughput is 6 images per second per GPU at batch size 8, so three GPUs are needed, and four are provisioned for headroom. A load test confirms 19 images per second, and the overnight window completes in 7.3 hours with margin for retries.

Use case. Throughput planning is required for any batch image workload with a deadline, and it is the analysis that determines hardware cost. It is also relevant to interactive pipelines with latency commitments.

Warnings. Planning without measuring on representative hardware produces estimates that are wrong by a factor of two or more. Never ignore batching efficiency. And never plan without headroom for retries and bursts.

Limitations. Throughput varies with image size and content, so a single figure is an average that may not hold for a mixed workload. GPU availability and pricing also change, affecting the cost calculation.

← Browse all glossary terms