[NLPL Task Force (A)] Fwd: [in5550-help] Broke Saga storage again, plz help :(

Sun May 10 13:02:25 UTC 2020

dear colleagues,

every now and again, users under the NLPL project umbrella (NN9447k)
run into disk quote issues on Saga.  and, trying to advice in a
current case, i realize i do not understand the set-up and constraints
very well myself (despite reasonably careful reading of
'https://documentation.sigma2.no/files_storage/clusters.html' :-).

i would be grateful if someone could explain in more detail how to
read the various lines output by dusage(1):

===============================================================================
[oe at login-1.SAGA ~]$ groups
oe dgi xa9910k nlpl ns9008k ns9052k nn9106k nn9447k oe_g
[oe at login-1.SAGA ~]$ dusage -a
===============================================================================
Block quota usage on: SAGA
===============================================================================
File system    User/Group          Usage          Soft Limit     Hard Limit
-------------------------------------------------------------------------------
oe_g           $HOME               680.9 GiB      0 Bytes        0 Bytes
oe             oe (u)              3.8 TiB        0 Bytes        0 Bytes
oe             oe (g)              1.2 MiB        0 Bytes        0 Bytes
nn9106k        nn9106k (g)         0 Bytes        0 Bytes        0 Bytes
ns9008k        ns9008k (g)         0 Bytes        0 Bytes        0 Bytes
ns9052k        ns9052k (g)         0 Bytes        0 Bytes        0 Bytes
xa9910k        xa9910k (g)         0 Bytes        0 Bytes        0 Bytes
nlpl           nlpl (g)            3.6 TiB        0 Bytes        0 Bytes
dgi            dgi (g)             0 Bytes        0 Bytes        0 Bytes
nn9447k        nn9447k (g)         308.7 GiB      1.0 TiB        1.0 TiB
===============================================================================

i am guessing that BeeGFS quotas are organized around user and group
ownership rather than around actual path locations on the '/cluster/'
filesystem?  group ownership appears to be automatically adjusted
periodically on Saga, e.g. to 'oe_g' below my $HOME directory, to
'nn9447k' below '/cluster/projects/nn9447k/', and to 'nlpl' below
'/cluster/shared/nlpl/' (on this view, 'oe_g' means something like
'oe_home')?  but no such automatic adjustment applies below $USERWORK?
 that would mean that moving data from $HOME to $USERWORK will not
affect the quota system, until i also run something like 'chgrp -R oe'
below $USERWORK, right?

related to the above, is what is reported by dusage(1) (i.e. the
BeeGFS quotas) real-time information, or is there some delay in how
frequent this data is updated?  i am asking because folk wisdom in our
group has it that there can be an hour or two delay when one has
overrun the quota on $HOME and after freeing up space, before one can
again write to $HOME.  now i am wondering whether the delay actually
reflects that people move files into, say, the project area, but until
the group ownership is automatically adjusted this data is still
counted against their $HOME quota (because it is owned by the '_g'
group)?

with thanks in advance, oe

---------- Forwarded message ---------
From: Maria Singstad Paulsen <marispau at ifi.uio.no>
Date: Sun, May 10, 2020 at 1:49 PM
Subject: SV: [in5550-help] Broke Saga storage again, plz help :(
To: Stephan Oepen <oe at ifi.uio.no>
Cc: in5550-help at ifi.uio.no <in5550-help at ifi.uio.no>

Either I'm doing something wrong, or there is something wrong
somewhere else. I have not uploaded or run my program since last
night, so the quota should be updated by now. I'll just try to work
around this, but it might be worth looking into if this behaviour is
not what's intended, I reckon.

What I don't really understand is why these numbers don't even seem to
add up. dusage shows 18.0 GiB for $HOME, 18.7 GiB (u) and 798.7 MiB
(g). Shouldn't files in $USERWORK count towards (g) here? By the looks
of it, some of my files in $USERWORK must've been counted towards (g),
but seemingly not the rest? If I'm reading this correctly, that is.

[marispau at login-1.SAGA ~]$ du -h -s $HOME/*
16M   /cluster/home/marispau/fastText
0     /cluster/home/marispau/loadingscript.sh
1,0K  /cluster/home/marispau/remote-py-interpreter.sh
1,0K  /cluster/home/marispau/sanity.py
1,5K  /cluster/home/marispau/syspaths.txt

[marispau at login-1.SAGA ~]$ du -h -s $HOME/.??*
30K   /cluster/home/marispau/.bash_history
512   /cluster/home/marispau/.bash_profile
512   /cluster/home/marispau/.bashrc
20M   /cluster/home/marispau/.cache
512   /cluster/home/marispau/.config
92K   /cluster/home/marispau/.ipython
1,0K  /cluster/home/marispau/.keras
512   /cluster/home/marispau/.lesshst
21K   /cluster/home/marispau/.lmod.d
512   /cluster/home/marispau/.pki
66M   /cluster/home/marispau/.pycharm_helpers
1,0K  /cluster/home/marispau/.python_history
7,5K  /cluster/home/marispau/.ssh
1,0K  /cluster/home/marispau/.viminfo

[marispau at login-1.SAGA ~]$ du -h -s $USERWORK/*
11M   /cluster/work/users/marispau/dev
146M  /cluster/work/users/marispau/exam
19G   /cluster/work/users/marispau/in5550
69M   /cluster/work/users/marispau/train

[marispau at login-1.SAGA ~]$ dusage
===============================================================================
Block quota usage on: SAGA
===============================================================================
File system    User/Group          Usage          Soft Limit     Hard Limit
-------------------------------------------------------------------------------
marispau_g     $HOME               18.0 GiB       20.0 GiB       20.0 GiB
marispau       marispau (u)        18.7 GiB       0 Bytes        0 Bytes
marispau       marispau (g)        798.7 MiB      0 Bytes        0 Bytes
nn9447k        nn9447k (g)         308.7 GiB      1.0 TiB        1.0 TiB
dgi            dgi (g)             0 Bytes        0 Bytes        0 Bytes
===============================================================================

________________________________________
Fra: Stephan Oepen <oe at ifi.uio.no>
Sendt: 10. mai 2020 00:31
Til: Maria Singstad Paulsen
Kopi: in5550-help at ifi.uio.no
Emne: Re: [in5550-help] Broke Saga storage again, plz help :(

i don't think soft links are relevant here, they just create 'aliases'
(another way of referencing a file).  the file itself is either in
your home directory or below your user work directory, and the
location of the actual file determines where it is counted against
your quota (no matter how many soft links may point to it from other
directories).

could it be the case that hidden files or directories in your $HOME
account for what seems like a discrepancy between du and dusage?  try
the following

du -h -s $HOME/.??*

files or directories whose names start with a period are not match by
standard globbing (expansion of the '*' wildcard).  the above will
match all files or directories below $HOME that start with a period,
are followed by another two arbitrary characters (the '?' in shell
wildcarding), and then have an abitrary, possibly empty suffix (the
'*' wildcard).  this complicated pattern is necessary to avoid
matching '..' (which would be included by the simpler patterns '.*' or
'.?*'), which would refer to the parent directory of your $HOME, i.e.
the huge filesystem with all user home directories ... running du(1)
on all of those would probably take a long time ...

hth, oe