[NLPL Task Force (A)] very inefficient GPU utilization on Saga by one user
Andrei Kutuzov
andreku at ifi.uio.no
Fri Dec 20 17:46:46 UTC 2019
Hi,
This issue has already been raised before, but the situation did not change.
Right now, again, one particular user (josece,
https://www.nhm.uio.no/english/about/organization/research-collections/people/josece/index.html)
is occupying all Saga GPUs. There are now 24 active GPU jobs running
under this user (some for several days already) and 9 more jobs pending.
Even worse, it seems that these jobs do not actually make use of GPUs.
The GPU utilization on all the nodes used by josece is 0% (in fact, the
CPU utilization is not much higher). But this still effectively
prohibits other users from getting access to Saga GPUs.
I attach the list of GPU nodes queue (produced by squeue
--partition=accel) and the overview of actual GPU usage by josece jobs.
It was produced by running the following commands:
for i in $(squeue -p accel | egrep 'c[0-9]-[0-9]$' | sort -u | awk
'{print $NF}')
do
ssh $i nvidia-smi | grep Default
done
I do not think it is a good way of using the precious GPU resource of Saga.
Can it be the case that josece jobs do not actually need GPUs and can be
run on the CPU nodes with the same speed? Is it possible to give josece
some feedback on that?
Also, all Saga users will probably benefit if some limit on the amount
of GPU jobs run by one user is imposed (at least when there are other
pending GPU jobs from other users).
Thanks in advance!
--
Andrei
PhD Candidate at Language Technology Group (LTG)
University of Oslo
-------------- next part --------------
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
186936 accel pacasus josece PD 0:00 1 (Priority)
186935 accel pacasus josece PD 0:00 1 (Priority)
186934 accel pacasus josece PD 0:00 1 (Priority)
186933 accel pacasus josece PD 0:00 1 (Resources)
186937 accel pacasus josece PD 0:00 1 (Priority)
186938 accel pacasus josece PD 0:00 1 (Priority)
186939 accel pacasus josece PD 0:00 1 (Priority)
186940 accel pacasus josece PD 0:00 1 (Priority)
186941 accel pacasus josece PD 0:00 1 (Priority)
188924 accel test andreku PD 0:00 1 (Priority)
173990 accel pacasus josece R 2-05:50:07 1 c7-2
173991 accel pacasus josece R 2-05:50:07 1 c7-2
173989 accel pacasus josece R 2-08:48:26 1 c7-7
173988 accel pacasus josece R 2-12:29:08 1 c7-4
173987 accel pacasus josece R 2-18:08:19 1 c7-3
173986 accel pacasus josece R 3-11:50:50 1 c7-3
186771 accel pacasus josece R 4:06:42 1 c7-8
186770 accel pacasus josece R 4:58:23 1 c7-1
186769 accel pacasus josece R 11:02:15 1 c7-8
186768 accel pacasus josece R 18:06:27 1 c7-6
186767 accel pacasus josece R 22:12:39 1 c7-6
186766 accel pacasus josece R 23:04:19 1 c7-1
186764 accel pacasus josece R 1-00:19:05 1 c7-2
186765 accel pacasus josece R 1-00:19:05 1 c7-5
186762 accel pacasus josece R 1-02:36:24 1 c7-7
186763 accel pacasus josece R 1-02:36:24 1 c7-8
186752 accel pacasus josece R 1-02:37:45 1 c7-1
186753 accel pacasus josece R 1-02:37:45 1 c7-3
186754 accel pacasus josece R 1-02:37:45 1 c7-4
186755 accel pacasus josece R 1-02:37:45 1 c7-4
186756 accel pacasus josece R 1-02:37:45 1 c7-5
186757 accel pacasus josece R 1-02:37:45 1 c7-5
186758 accel pacasus josece R 1-02:37:45 1 c7-6
186759 accel pacasus josece R 1-02:37:45 1 c7-7
-------------- next part --------------
| N/A 27C P0 30W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 27C P0 30W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 28C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 31C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 32C P0 37W / 250W | 325MiB / 16280MiB | 0% Default |
| N/A 31C P0 32W / 250W | 267MiB / 16280MiB | 1% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 25W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 28C P0 41W / 250W | 317MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 29C P0 29W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 29C P0 29W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 31C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 34W / 250W | 281MiB / 16280MiB | 0% Default |
| N/A 30C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 27C P0 30W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 28C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 31C P0 25W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 28C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 31C P0 25W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 29C P0 32W / 250W | 507MiB / 16280MiB | 0% Default |
| N/A 27C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 27C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 32C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 29C P0 36W / 250W | 327MiB / 16280MiB | 0% Default |
| N/A 28C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 27C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 32C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 31C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 31C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 32C P0 38W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 31C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 25W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 32C P0 38W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 31C P0 36W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 25W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 31C P0 36W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 33C P0 33W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 32C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 29C P0 29W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 29C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 26C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 32C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 31C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 31C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 29C P0 34W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 31C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 29C P0 35W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 31W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 31C P0 36W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 33C P0 33W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 32C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 31C P0 34W / 250W | 321MiB / 16280MiB | 0% Default |
| N/A 28C P0 41W / 250W | 425MiB / 16280MiB | 0% Default |
| N/A 30C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 28C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
| N/A 31C P0 36W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 30C P0 32W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 33C P0 34W / 250W | 267MiB / 16280MiB | 0% Default |
| N/A 32C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
More information about the infrastructure
mailing list