[NLPL Task Force (A)] emerging NLPL approach to parallel Python environments

[Martin Matthiesen]

Hello Stephan,

I only had a quick look through all the pages, but I think they look really good. I fully agree with concentrating on Python 3.5, those who need something else are free to compile/pip install themselves. I would like to forward the links to our admins, I think they appreciate the information as well.

Martin

-- 
Martin Matthiesen
CSC - Tieteen tietotekniikan keskus
CSC - IT Center for Science
PL 405, 02101 Espoo, Finland
+358 9 457 2376, martin.matthiesen at csc.fi
Public key : https://pgp.mit.edu/pks/lookup?op=get&search=0x74B12876FD890704
Fingerprint: AA25 6F56 5C9A 8B42 009F  BA70 74B1 2876 FD89 0704

----- Original Message -----
> From: "Stephan Oepen" <oe at ifi.uio.no>
> To: "infrastructure" <infrastructure at nlpl.eu>, "Andrei Kutuzov" <andreku at ifi.uio.no>, "Eivind Alexander Bergem"
> <eivinabe at student.matnat.uio.no>, "Yves Scherrer" <yves.scherrer at helsinki.fi>
> Sent: Tuesday, 2 October, 2018 17:33:39
> Subject: [NLPL Task Force (A)] emerging NLPL approach to parallel Python	environments

> colleagues,
> 
> over the past week or so, i have worked towards a set-up on Abel and
> Taito that aims for a parallel user experience, as much as possible.
> to try out this philosophy i have installed PyTorch and TensorFlow,
> OpenNMT-py, and a couple of more ‘basic’ toolkits (e.g. NLTK and
> SpaCy) on both systems, in each case building on ‘stacked’ Python 3.5
> virtual environments.  for all i know, all of the above work on both
> cpu and gpu nodes on Abel and Taito.
> 
> in doing so, i realized there are some non-trivial design decisions to
> be made here, and i plan to email the NLPL team before too long and
> ask for feedback and suggestions for how to move forward.  but before
> that, i was hoping at least some of you could take a look at the
> current state of affairs and share your thoughts?
> 
> http://wiki.nlpl.eu/index.php/Infrastructure/software/python
> http://wiki.nlpl.eu/index.php/Infrastructure/software/tensorflow
> http://wiki.nlpl.eu/index.php/Infrastructure/software/pytorch
> http://wiki.nlpl.eu/index.php/Infrastructure/software/catalogue
> 
> over the next several months, i expect NLPL will install a broad range
> of additional pieces of software.  for maximum long-term benefit, i am
> thinking it would be good to give the various team members who will
> produce these installations as much as possible of a general
> framework, including for example a script to automate creation of the
> initial virtual environment (and maybe corresponding module
> definition), recommendations for how to document the installation, and
> a template for the minimum level of user documentation?
> 
> with thanks in advance, oe