Performance is at the heart of every successful AI/ML project, and if you’ve ever fought with slow training loops or underutilised GPUs, you know the pain. In this article, we take a closer look at CUDA streams—a smart way to overlap GPU tasks so you can run operations concurrently. Here, we explore this approach using examples from a series on profiling and optimisation in PyTorch.
Most training workloads are implemented as single computation graphs, but sometimes breaking them into subgraphs means parts can run in parallel. This technique, often called pipelining, lets you execute segments of your model simultaneously, giving efficiency a solid boost.
CUDA streams shine when portions of your workload aren’t fully taxing your GPU. Consider two common scenarios: training parts of a model independently, and offloading data preprocessing to the GPU. For instance, if you’re freezing a network’s backbone and just training the head, you can run the two components in tandem. Similarly, heavy data augmentation can exhaust your CPU and leave your GPU waiting—by transferring those tasks to the GPU and running them concurrently, you can see significant throughput improvements.
To test these ideas, two toy training scripts were run on an Amazon EC2 instance equipped with an NVIDIA A10G GPU. The results were promising. In one test case, a CNN-based image segmentation model had its encoder and decoder trained concurrently. This tweaking of the training loop led to a 9.6% speedup, showing that even a modest parallelisation can pay off.
The gains may vary. If you’re already maxing out your GPU with large batches, the benefit from pipelining might be less dramatic. However, when your GPU is underutilised, CUDA streams can dramatically cut idle times and improve overall performance.
Another example is data augmentation in image classification. When heavy augmentations deplete CPU resources, the GPU might end up waiting, which creates a bottleneck. Offloading these resource-intensive augmentations to the GPU, paired with CUDA streams, resulted in more than a 72% improvement in throughput compared to running them on the CPU.
If you’re considering this strategy, the key takeaway is to test CUDA streams with your specific workload. This technique can be a valuable addition to your optimisation toolkit, but its performance boost will depend on your particular data and system configuration.
By adapting these methods, you can fine-tune your training process and get more out of your hardware. Whether you’re working on image segmentation or classification, leveraging CUDA streams offers a practical pathway to higher efficiency and smoother performance.
