[NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)

[Stephan Oepen]

sorry, i was the one who had introduced the confusion about mailing
lists.  there is no ‘translation at nlpl.eu’ currently, and upon
consultation with joerg there appears not to be a great need for it
either (once i get around to documenting the task force structure on
the project wiki, i might want to create that list nevertheless).

i am adding yves to thread now, so he at least has a chance of knowing
what we are talking about :-).

martin, i doubt that an installation of OpenNMT that requires everyone
to ‘pip install --user’ into their home directory will be a good
solution.  that way, the getting started instructions will be more
complex, and we lack control over which version of PyTorch gets
installed at the time the user actually runs the command.  my
immediate reaction at least is that NLPL-supported software should be
‘self-contained’, in the sense of not depending on software components
maintained by the user.

what i am doing increasingly on abel is deriving virtual environments;
e.g. my PyTorch installation (for NLPL) straightforwardly builds on
the USIT-maintained python 3.5.  i suppose we should be able to do the
same thing on taito, i.e. create ‘nlpl-pytorch’ as a virtual
environment that includes the precompiled PyTorch wheel from your CSC
colleagues?

oe




On Mon, Sep 17, 2018 at 5:06 PM, Martin Matthiesen
<martin.matthiesen at csc.fi> wrote:
> Hello,
>
> We already have a way to use pytorch 0.4.1 on Taito-GPU:
>
> module load python-env/intelpython3.6-2018.3
> [GPU-Env ~]$ pip install -v --user /appl/opt/pytorch/0.4.1/cu90/torch-0.4.1-cp36-cp36m-linux_x86_64.whl
>
> One of my colleagues has compiled the module. Note that the module needs python 3.6 to work, the highest available on Taito-GPU.
>
> Before I investigate CPU-support or support for other compilers, would this pip-approach work for you?
>
> Regards,
> Martin
>
> ----- Original Message -----
>> From: "Stephan Oepen" <oe at ifi.uio.no>
>> To: translation at nlpl.eu
>> Cc: "infrastructure" <infrastructure at nlpl.eu>
>> Sent: Saturday, 15 September, 2018 18:59:29
>> Subject: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)
>
>> colleagues,
>>
>> joerg, martin, and i talked about getting the new release version of
>> OpenNMT installed for NLPL.  it appears it requires the most recent
>> version of PyTorch, which currently is not available on Taito.  martin
>> will ask for it to be installed by CSC.
>>
>> in parallel, i believe i managed to put an NLPL-owned installation of
>> the right PyTorch version onto Abel, please see:
>>
>> http://wiki.nlpl.eu/index.php/Infrastructure/software/pytorch
>>
>> before announcing this more widely, i would be grateful for some
>> testing, in particular for both cpu and gpu usage.  would anyone we
>> readily set up to give this a shot on Abel?
>>
>> assuming our PyTorch is healthy, would someone from the helsinki team
>> have the time to try and install OpenNMT onto Abel, e.g. as
>>
>> /projects/nlpl/software/opennmt-py/0.2.1
>>
>> there have been two relatively recent requests for OpenNMT in oslo
>> (one of them for seq2seq dependency parsing :-), so i believe it would
>> now be warranted to provide it on both systems.
>>
>> best wishes, oe