[NLPL Task Force (A)] emerging NLPL approach to parallel Python environments

[Stephan Oepen]

colleagues,

over the past week or so, i have worked towards a set-up on Abel and
Taito that aims for a parallel user experience, as much as possible.
to try out this philosophy i have installed PyTorch and TensorFlow,
OpenNMT-py, and a couple of more ‘basic’ toolkits (e.g. NLTK and
SpaCy) on both systems, in each case building on ‘stacked’ Python 3.5
virtual environments.  for all i know, all of the above work on both
cpu and gpu nodes on Abel and Taito.

in doing so, i realized there are some non-trivial design decisions to
be made here, and i plan to email the NLPL team before too long and
ask for feedback and suggestions for how to move forward.  but before
that, i was hoping at least some of you could take a look at the
current state of affairs and share your thoughts?

http://wiki.nlpl.eu/index.php/Infrastructure/software/python
http://wiki.nlpl.eu/index.php/Infrastructure/software/tensorflow
http://wiki.nlpl.eu/index.php/Infrastructure/software/pytorch
http://wiki.nlpl.eu/index.php/Infrastructure/software/catalogue

over the next several months, i expect NLPL will install a broad range
of additional pieces of software.  for maximum long-term benefit, i am
thinking it would be good to give the various team members who will
produce these installations as much as possible of a general
framework, including for example a script to automate creation of the
initial virtual environment (and maybe corresponding module
definition), recommendations for how to document the installation, and
a template for the minimum level of user documentation?

with thanks in advance, oe