[NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)

[Stephan Oepen]

thanks for adjusting those permissions, yves!

roughle how long into the job would you expect the error to occur?

i have been running for around six minutes so far, and training
appears to get going:

2018-12-18 14:47:43,683 INFO] encoder: 14116000
[2018-12-18 14:47:43,683 INFO] decoder: 25862084
[2018-12-18 14:47:43,683 INFO] * number of parameters: 39978084
/proj/nlpl/software/opennmt-py/0.2.1/lib/python3.5/site-packages/torch/nn/_reduction.py:49:
UserWarning: size_average and reduce args will be deprecated, please
use reduction='sum' instead.
  warnings.warn(warning.format(ret))
[2018-12-18 14:47:43,685 INFO] Start training...
[2018-12-18 14:47:43,707 INFO] Loading train dataset from
data.train.1.pt, number of examples: 1030
/proj/nlpl/software/opennmt-py/0.2.1/lib/python3.5/site-packages/torch/nn/functional.py:1320:
UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
  warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")
[2018-12-18 14:49:15,649 INFO] Loading train dataset from
data.train.10.pt, number of examples: 1162
[2018-12-18 14:50:55,474 INFO] Loading train dataset from
data.train.100.pt, number of examples: 1199
[2018-12-18 14:52:13,191 INFO] Step 50/100000; acc:   5.83; ppl:
5884.51; xent: 8.68; lr: 1.00000; 272/262 tok/s;    269 sec
[2018-12-18 14:52:38,496 INFO] Loading train dataset from
data.train.1000.pt, number of examples: 1216

but earlier you had sent a traceback involving a function called
training_opt_postprocessing() ... so maybe the error ony occurs
towards the end of training?  which would seem pretty weird, seeing as
i suppose PyTorch has been used extensively up to that point already?

oe



On Tue, Dec 18, 2018 at 10:11 AM Martin Matthiesen
<martin.matthiesen at csc.fi> wrote:
>
> Hi,
>
> I did try for an hour and a bit yesterday to pinpoint the problem, but could not make head or tail of it. Did I understand correctly that Stephan, you got this working on Taito?
>
> Martin
>
> P.S.: Should we keep infrastructure out of this or is this interesting to Jörg and Björn?
>
> --
> Martin Matthiesen
> CSC - Tieteen tietotekniikan keskus
> CSC - IT Center for Science
> PL 405, 02101 Espoo, Finland
> +358 9 457 2376, martin.matthiesen at csc.fi
> Public key : https://pgp.mit.edu/pks/lookup?op=get&search=0x74B12876FD890704
> Fingerprint: AA25 6F56 5C9A 8B42 009F  BA70 74B1 2876 FD89 0704
>
> ________________________________
>
> From: "Yves Scherrer" <yves.scherrer at helsinki.fi>
> To: "Stephan Oepen" <oe at ifi.uio.no>
> Cc: "Martin Matthiesen" <martin.matthiesen at csc.fi>, "infrastructure" <infrastructure at nlpl.eu>
> Sent: Tuesday, 18 December, 2018 10:35:25
> Subject: RE: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)
>
> Hi,
>
>
>
> > could you make the complete data directory group- or world-readable,
> > so i can try running the ‘train.py’ script without creating my own
> > copy of the data?
>
>
>
> That should work now.
>
>
>
> > thinking (possbily over-)optimistically, maybe the problem has
> > magically disappeared already?
>
>
>
> Unfortunately, it hasn’t. Or was I supposed to reinstall the OpenNMT-py module locally?
>
>
>
> Yves
>
>
>
>