[NLPL Task Force (A)] Storage alternatives

[Jeremy Claude Barnes]

I still have to try the $USERWORK option, but I do agree with Vinit that the current workflow is sometimes problematic. I've run into version compatibility issues with python trying to run other peoples code on Saga (python 2.7 with keras 2.2.4 and tensorflow 1.4.1) which would take about 5 seconds given the virtual environment option.

Jeremy Barnes
Language Technology Group
University of Oslo
Office 7645
jeremycb at ifi.uio.no

________________________________________
From: Vinit Ravishankar
Sent: 18 November 2020 15:07
To: Stephan Oepen
Cc: Andrei Kutuzov; Jeremy Claude Barnes; Tiedemann, Jörg; infrastructure
Subject: Re: [NLPL Task Force (A)] Storage alternatives

The issue with using the module system is that when you’re running someone else’s code, it’s very difficult to select the perfect combination of modules that both a) works and b) does not result in version mismatches with the very unique version combination that whoever wrote that code may have chosen to use. Conda or pip make this easy because you normally have a requirements.txt that you can just install from.

@jeremy has also had issues with this - Jeremy, could you weigh in?

$USERWORK is an option I suppose, with backups elsewhere - I’ll look into it.

– Vinit

> On 18 Nov 2020, at 15:02, Stephan Oepen <oe at ifi.uio.no> wrote:
>
> why should a user have a private installation of basic software like python or pytorch?
>
> ideally, the pre-installed NLPL modules should provide all common software (and data, where appropriate).  why are you not using those?
>
> alternatively, there is the $USERWORK area.  if you fully automate environment creation, e.g. from conda, then why not put your software stack on that more transient storage area?