[NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)

[Martin Matthiesen]

Hello Stephan,

----- Original Message -----
> From: "Stephan Oepen" <oe at ifi.uio.no>
> To: "Martin Matthiesen" <martin.matthiesen at csc.fi>
> Cc: "infrastructure" <infrastructure at nlpl.eu>, "Yves Scherrer" <yves.scherrer at helsinki.fi>
> Sent: Tuesday, 18 September, 2018 14:13:53
> Subject: Re: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)

> sorry, i was the one who had introduced the confusion about mailing
> lists.  there is no ‘translation at nlpl.eu’ currently, and upon
> consultation with joerg there appears not to be a great need for it
> either (once i get around to documenting the task force structure on
> the project wiki, i might want to create that list nevertheless).
> 
> i am adding yves to thread now, so he at least has a chance of knowing
> what we are talking about :-).

Ok!
> 
> martin, i doubt that an installation of OpenNMT that requires everyone
> to ‘pip install --user’ into their home directory will be a good
> solution.  that way, the getting started instructions will be more
> complex, and we lack control over which version of PyTorch gets
> installed at the time the user actually runs the command.  my
> immediate reaction at least is that NLPL-supported software should be
> ‘self-contained’, in the sense of not depending on software components
> maintained by the user.

Ok, I understand. 
> 
> what i am doing increasingly on abel is deriving virtual environments;
> e.g. my PyTorch installation (for NLPL) straightforwardly builds on
> the USIT-maintained python 3.5.  i suppose we should be able to do the
> same thing on taito, i.e. create ‘nlpl-pytorch’ as a virtual
> environment that includes the precompiled PyTorch wheel from your CSC
> colleagues?

Yes, I guess that is the only sensible solution to not lose track completely. In the meantime, how would this work for you all:

[GPU-Env ~]$ module load python-env/intelpython3.6-2018.3
Loading application Intel Distribution for Python 2018 update 3 
[GPU-Env ~]$ module list

Currently Loaded Modules:
  1) gcc/4.9.3   2) cuda/7.5   3) StdEnv   4) git/2.17.1   5) python-env/intelpython3.6-2018.3

[GPU-Env ~]$ python3
Python 3.6.3 |Intel Corporation| (default, May  4 2018, 04:22:28) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distribution
>>> import torch
>>> torch.__version__
'0.4.1'

Kudos to my colleagues Markus and Jarmo here.

Martin

> 
> oe
> 
> 
> 
> 
> On Mon, Sep 17, 2018 at 5:06 PM, Martin Matthiesen
> <martin.matthiesen at csc.fi> wrote:
>> Hello,
>>
>> We already have a way to use pytorch 0.4.1 on Taito-GPU:
>>
>> module load python-env/intelpython3.6-2018.3
>> [GPU-Env ~]$ pip install -v --user
>> /appl/opt/pytorch/0.4.1/cu90/torch-0.4.1-cp36-cp36m-linux_x86_64.whl
>>
>> One of my colleagues has compiled the module. Note that the module needs python
>> 3.6 to work, the highest available on Taito-GPU.
>>
>> Before I investigate CPU-support or support for other compilers, would this
>> pip-approach work for you?
>>
>> Regards,
>> Martin
>>
>> ----- Original Message -----
>>> From: "Stephan Oepen" <oe at ifi.uio.no>
>>> To: translation at nlpl.eu
>>> Cc: "infrastructure" <infrastructure at nlpl.eu>
>>> Sent: Saturday, 15 September, 2018 18:59:29
>>> Subject: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)
>>
>>> colleagues,
>>>
>>> joerg, martin, and i talked about getting the new release version of
>>> OpenNMT installed for NLPL.  it appears it requires the most recent
>>> version of PyTorch, which currently is not available on Taito.  martin
>>> will ask for it to be installed by CSC.
>>>
>>> in parallel, i believe i managed to put an NLPL-owned installation of
>>> the right PyTorch version onto Abel, please see:
>>>
>>> http://wiki.nlpl.eu/index.php/Infrastructure/software/pytorch
>>>
>>> before announcing this more widely, i would be grateful for some
>>> testing, in particular for both cpu and gpu usage.  would anyone we
>>> readily set up to give this a shot on Abel?
>>>
>>> assuming our PyTorch is healthy, would someone from the helsinki team
>>> have the time to try and install OpenNMT onto Abel, e.g. as
>>>
>>> /projects/nlpl/software/opennmt-py/0.2.1
>>>
>>> there have been two relatively recent requests for OpenNMT in oslo
>>> (one of them for seq2seq dependency parsing :-), so i believe it would
>>> now be warranted to provide it on both systems.
>>>
> >> best wishes, oe