[NLPL Task Force (A)] reducing nightly email: file permissions

[Stephan Oepen]

at the core of those scripts is just an rsync(1) invocation; i am not
sure that an except pattern would actually appease it; and i would be
a tad nervous about accidentally dropping sub-directories elsewhere in
the tree that should actually be replicated.

how about a clean-up script as part of the installation instead,
removing directories (at the top level, maybe) that start with a dot?

oe


oe

On Wed, Sep 18, 2019 at 2:34 PM Tiedemann, Jörg
<jorg.tiedemann at helsinki.fi> wrote:
>
>
> Could we skip directories that start with ‘.’
> That would solve all (most?) of the issues below.
>
> Jörg
>
> ********************************************************************************************
> Jörg Tiedemann
> Language Technology https://blogs.helsinki.fi/language-technology/
> University of Helsinki
>
> On 18 Sep 2019, at 14:58, Stephan Oepen <oe at ifi.uio.no> wrote:
>
> dear all,
>
> truth be told, i have barely looked at the emails that are
> auto-generated to the infrastructure task force each night over the
> past several months, but as we are setting up new instances of the
> laboratory on Puhti and Saga i would like to go back to actually
> paying attention to what happens.  among other things, the replication
> and back-up notifications are good indicators of someone making
> changes to the shared data and software directories ... and in
> principle, i think, we should keep abreast with these developments
> :-).
>
> the scripts are written to only send email when something actually
> happened, or when there is an indication of problems.  for the past
> several months, we have had some directories and files on both Abel
> Taito that were not readable by the back-up scripts (which run in user
> land, as 'oe'), which led to nightly 'problem' emails.
>
> joerg and yves, could i ask that you both try fixing file permissions
> on both systems.  current trouble makers appear to be:
>
> [oe at login-nird-0 nlpl]$ grep -i 'permission denied'
> mirror/*/current/operation/log/*/nird.19-09-17-*
> mirror/abel/current/operation/log/abel/nird.19-09-17-02:00:rsync:
> opendir "/projects/nlpl/software/nlpl-common/.cache" failed:
> Permission denied (13)
> mirror/abel/current/operation/log/abel/nird.19-09-17-02:00:rsync:
> opendir "/projects/nlpl/software/nlpl-common/.emacs.d/auto-save-list"
> failed: Permission denied (13)
> mirror/abel/current/operation/log/abel/nird.19-09-17-02:00:rsync:
> send_files failed to open
> "/projects/nlpl/software/nlpl-common/.lesshst": Permission denied (13)
> mirror/taito/current/operation/log/taito/nird.19-09-17-02:00:rsync:
> opendir "/proj/nlpl/software/nlpl-common/.cache/pip/http/8" failed:
> Permission denied (13)
> mirror/taito/current/operation/log/taito/nird.19-09-17-02:00:rsync:
> opendir "/proj/nlpl/software/nlpl-common/.cache/pip/http/9/c" failed:
> Permission denied (13)
> mirror/taito/current/operation/log/taito/nird.19-09-17-02:00:rsync:
> opendir "/proj/nlpl/software/nlpl-common/.cache/pip/http/b/6" failed:
> Permission denied (13)
> mirror/taito/current/operation/log/taito/nird.19-09-17-02:00:rsync:
> send_files failed to open
> "/proj/nlpl/software/nlpl-common/.cache/pip/http/6/3/c/9/7/63c973d02997ec099a5c771c5ca2ac4b2e125e001a171c85b274b610":
> Permission denied (13)
> mirror/taito/current/operation/log/taito/nird.19-09-17-02:00:rsync:
> send_files failed to open
> "/proj/nlpl/software/nlpl-common/.cache/pip/http/b/b/8/7/6/bb876b2e6a22c2739002e9ef3388ae978b7a6192a19fe67d8b0030f2":
> Permission denied (13)
>
> from the above, i am guessing that the installation of the 'common'
> module temporarily treats the module directory as a home directory,
> leading to stray configuration and cache files there.  minimally, i
> would recommend you set a more generous umask(1) for these steps, so
> that the files do not end up with restrictive permissions?
>
> cheers, oe
>
>