[NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)

Wed Sep 19 11:49:19 UTC 2018

Hi Yves, 

> From: "Yves Scherrer" <yves.scherrer at helsinki.fi>
> To: "Martin Matthiesen" <martin.matthiesen at csc.fi>, "Stephan Oepen"
> <oe at ifi.uio.no>
> Cc: "infrastructure" <infrastructure at nlpl.eu>
> Sent: Wednesday, 19 September, 2018 14:12:21
> Subject: RE: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)

> Hi Stephan, Martin,

> I’m catching up on this thread… A few questions from my side:

> Regarding Martin’s latest suggestion: that seems indeed to work fine, although
> with the exact same commands I get a different version of PyTorch:

> >>> import torch

> >>> torch.__file__

> '/appl/opt/python/intelpython36-2018.3/intelpython3/lib/python3.6/site-packages/torch/__init__.py'

> >>> torch.__version__

> '0.4.0a0+3749c58'

My bad, I had the newest pytorch locally installed and forgot about it: 
pip install -v --user /appl/opt/pytorch/0.4.1/cu90/torch-0.4.1-cp36-cp36m-linux_x86_64.whl 

> In any case, if PyTorch is already installed in some Python distribution, that
> would make setting up a specific OpenNMT module rather easy. If not, virtual
> environments should work as well (the tricky thing is mainly to figure out
> which python versions play well with CUDA…)

> Regarding Stephan’s suggestion of virtual environments: do you know if virtual
> environments can be “stacked”, i.e. whether I could create an OpenNMT virtual
> environment that lies on top of your PyTorch environment? Or would I have to
> re-install another instance of PyTorch in the OpenNMT virtualenv?

> I’ll be travelling for the rest of the week, but will try to have a closer look
> at these options next week.

My colleague Jarmo said the other week that we likely need a full time python expert for these rather complex installations... It seems to me that machine learning tools are really a moving target for a sysadmin. 

Martin 

> Best,

> Yves

> From: Martin Matthiesen <martin.matthiesen at csc.fi>
> Sent: Wednesday, September 19, 2018 1:29:35 PM
> To: Stephan Oepen
> Cc: infrastructure; Scherrer, Yves
> Subject: Re: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)
> Hello Stephan,

> ----- Original Message -----
> > From: "Stephan Oepen" <oe at ifi.uio.no>
> > To: "Martin Matthiesen" <martin.matthiesen at csc.fi>
>> Cc: "infrastructure" <infrastructure at nlpl.eu>, "Yves Scherrer"
> > <yves.scherrer at helsinki.fi>
> > Sent: Tuesday, 18 September, 2018 14:13:53
> > Subject: Re: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)

> > sorry, i was the one who had introduced the confusion about mailing
> > lists. there is no ‘translation at nlpl.eu’ currently, and upon
> > consultation with joerg there appears not to be a great need for it
> > either (once i get around to documenting the task force structure on
> > the project wiki, i might want to create that list nevertheless).

> > i am adding yves to thread now, so he at least has a chance of knowing
> > what we are talking about :-).

> Ok!

> > martin, i doubt that an installation of OpenNMT that requires everyone
> > to ‘pip install --user’ into their home directory will be a good
> > solution. that way, the getting started instructions will be more
> > complex, and we lack control over which version of PyTorch gets
> > installed at the time the user actually runs the command. my
> > immediate reaction at least is that NLPL-supported software should be
> > ‘self-contained’, in the sense of not depending on software components
> > maintained by the user.

> Ok, I understand.

> > what i am doing increasingly on abel is deriving virtual environments;
> > e.g. my PyTorch installation (for NLPL) straightforwardly builds on
> > the USIT-maintained python 3.5. i suppose we should be able to do the
> > same thing on taito, i.e. create ‘nlpl-pytorch’ as a virtual
> > environment that includes the precompiled PyTorch wheel from your CSC
> > colleagues?

> Yes, I guess that is the only sensible solution to not lose track completely. In
> the meantime, how would this work for you all:

> [GPU-Env ~]$ module load python-env/intelpython3.6-2018.3
> Loading application Intel Distribution for Python 2018 update 3
> [GPU-Env ~]$ module list

> Currently Loaded Modules:
> 1) gcc/4.9.3 2) cuda/7.5 3) StdEnv 4) git/2.17.1 5)
> python-env/intelpython3.6-2018.3

> [GPU-Env ~]$ python3
> Python 3.6.3 |Intel Corporation| (default, May 4 2018, 04:22:28)
> [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> Intel(R) Distribution for Python is brought to you by Intel Corporation.
> Please check out: [ https://software.intel.com/en-us/python-distribution |
> https://software.intel.com/en-us/python-distribution ]
> >>> import torch
> >>> torch.__version__
> '0.4.1'

> Kudos to my colleagues Markus and Jarmo here.

> Martin

> > oe

> > On Mon, Sep 17, 2018 at 5:06 PM, Martin Matthiesen
> > <martin.matthiesen at csc.fi> wrote:
> >> Hello,

> >> We already have a way to use pytorch 0.4.1 on Taito-GPU:

> >> module load python-env/intelpython3.6-2018.3
> >> [GPU-Env ~]$ pip install -v --user
> >> /appl/opt/pytorch/0.4.1/cu90/torch-0.4.1-cp36-cp36m-linux_x86_64.whl

> >> One of my colleagues has compiled the module. Note that the module needs python
> >> 3.6 to work, the highest available on Taito-GPU.

> >> Before I investigate CPU-support or support for other compilers, would this
> >> pip-approach work for you?

> >> Regards,
> >> Martin

> >> ----- Original Message -----
> >>> From: "Stephan Oepen" <oe at ifi.uio.no>
> >>> To: translation at nlpl.eu
> >>> Cc: "infrastructure" <infrastructure at nlpl.eu>
> >>> Sent: Saturday, 15 September, 2018 18:59:29
> >>> Subject: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)

> >>> colleagues,

> >>> joerg, martin, and i talked about getting the new release version of
> >>> OpenNMT installed for NLPL. it appears it requires the most recent
> >>> version of PyTorch, which currently is not available on Taito. martin
> >>> will ask for it to be installed by CSC.

> >>> in parallel, i believe i managed to put an NLPL-owned installation of
> >>> the right PyTorch version onto Abel, please see:

>>>> [ http://wiki.nlpl.eu/index.php/Infrastructure/software/pytorch |
> >>> http://wiki.nlpl.eu/index.php/Infrastructure/software/pytorch ]

> >>> before announcing this more widely, i would be grateful for some
> >>> testing, in particular for both cpu and gpu usage. would anyone we
> >>> readily set up to give this a shot on Abel?

> >>> assuming our PyTorch is healthy, would someone from the helsinki team
> >>> have the time to try and install OpenNMT onto Abel, e.g. as

> >>> /projects/nlpl/software/opennmt-py/0.2.1

> >>> there have been two relatively recent requests for OpenNMT in oslo
> >>> (one of them for seq2seq dependency parsing :-), so i believe it would
> >>> now be warranted to provide it on both systems.

> > >> best wishes, oe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nlpl.eu/archives/infrastructure/attachments/20180919/a1c9c961/attachment.htm>