[NLPL Task Force (A)] GPU usage patterns

[Vinit Ravishankar]

Hi all,

I had three jobs queued up for GPU time on Saga yesterday, and I'd been waiting quite a while (~22 hours) before investigating - turns out a single user has been allocated virtually all the free GPUs for 2 days. This is obviously extremely inconvenient - I submitted my jobs on a Tuesday, which means I’d have to wait till Thursday for their jobs to terminate, which would mean I’d get my results over the weekend. 

Now this obviously isn’t the user’s fault, because it is quite convenient to queue a lot of jobs and leave them to run - I often do this myself. However, given how people’s work patterns on GPUs have drastically changed over the past few years, and given that Saga has a relatively limited number of GPUs - would it be possible to have a fairer queuing and scheduling system that wouldn’t instantly allocate all free resources for long stretches of time? Thanks!

Best

– Vinit