Selecting CUDA Devices

I observed that when I run a Pytorch program on a system with GPUs, the Pytorch runner dispatches the computational tasks to both GPUs. Since the program is not optimized for using multiple GPUs, the performance using the two GPU is worse than just using one. A simple method to address this turns out to be that we inform Pytorch to use a designated GPU via environmental variable CUDA_VISIBLE_DEVICES.


For instance, to run a task run_task.sh, we can 

 CUDA_VISIBLE_DEVICES=0 ./run_task.sh SEED=1234

which results in running the task on a single GPU. 

For the non-optimized program, I got much better computational efficiency by doing than letting each run on two GPUs:

 CUDA_VISIBLE_DEVICES=0 ./run_task.sh SEED=1234

 CUDA_VISIBLE_DEVICES=1 ./run_task.sh SEED=4321