<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"> </head> <body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> <div class="WordSection1"> <p class="MsoNormal">Hi,</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Just a quick update on this issue – I’m unfortunately running into it time and again… It all seems very inconsistent – the same scripts that worked in December don’t work anymore with another dataset now.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">The OpenNMT pipeline consists of three steps, data preprocessing, model training and translating. Sometimes the CUDA version error appears at the beginning of training, sometimes at the beginning of translating. It looks so far that putting everything into the same script (and thus forcing the three steps to be executed on the same GPU node) relieves the issue somewhat, but this kind of defeats the purpose of pretrained models…</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">I’ll have my scripts tested by a colleague here, as it could be that my CSC account is somehow corrupted (I remember that Jörg could run some scripts fine while I got errors with them). I’ve also tried to clean my home directory from hidden configuration settings, but I might give that another shot when Taito is back running…</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">@Stephan: Thanks for the update on winter school activities – I will prepare a quick walk-through of the MT activities.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Best,</p> <p class="MsoNormal">Yves</p> <p class="MsoNormal"><o:p> </o:p></p> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Scherrer, Yves<br> <b>Sent:</b> Friday, December 21, 2018 12:49:39 PM<br> <b>To:</b> Martin Matthiesen<br> <b>Cc:</b> Stephan Oepen; infrastructure<br> <b>Subject:</b> Re: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)</font> <div> </div> </div> <div>Hi, <div class=""><br class=""> </div> <div class="">In my tests, removing srun indeed resolves the issue. I will update my scripts accordingly, and from my point of view we can “close” this discussion, although the underlying reasons for this changing behavior of srun are still a bit unclear…</div> <div class=""><br class=""> </div> <div class="">Thanks for your help anyway!</div> <div class="">Yves</div> <div class=""> <div><br class=""> <blockquote type="cite" class=""> <div class="">On 20 Dec 2018, at 10:05, Martin Matthiesen <<a href="mailto:martin.matthiesen@csc.fi" class="">martin.matthiesen@csc.fi</a>> wrote:</div> <br class="Apple-interchange-newline"> <div class=""> <div style="caret-color: rgb(0, 0, 0); font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: arial, helvetica, sans-serif; font-size: 10pt;" class=""> <div class=""></div> <div class="">Hi again,<br class=""> </div> <div class=""><br data-mce-bogus="1" class=""> </div> <div class="">Sorry, I accidentally hit send too early.<br data-mce-bogus="1" class=""> </div> <div class=""><br data-mce-bogus="1" class=""> </div> <div class="">So my suspicion is that some environment setting is set slightly differently now than it used to be and this affects srun. Is removing srun from the script resolving the issue or is it only a workaround?<br data-mce-bogus="1" class=""> </div> <div class=""><br data-mce-bogus="1" class=""> </div> <div class="">Martin<br data-mce-bogus="1" class=""> </div> <div data-marker="__SIG_PRE__" class=""><br data-mce-bogus="1" class=""> </div> <div class=""><br class=""> </div> <hr id="zwchr" data-marker="__DIVIDER__" class=""> <div data-marker="__HEADERS__" class=""> <blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> <b class="">From:<span class="Apple-converted-space"> </span></b>"Martin Matthiesen" <<a href="mailto:martin.matthiesen@csc.fi" class="">martin.matthiesen@csc.fi</a>><br class=""> <b class="">To:<span class="Apple-converted-space"> </span></b>"Yves Scherrer" <<a href="mailto:yves.scherrer@helsinki.fi" class="">yves.scherrer@helsinki.fi</a>><br class=""> <b class="">Cc:<span class="Apple-converted-space"> </span></b>"Stephan Oepen" <<a href="mailto:oe@ifi.uio.no" class="">oe@ifi.uio.no</a>>, "infrastructure" <<a href="mailto:infrastructure@nlpl.eu" class="">infrastructure@nlpl.eu</a>><br class=""> <b class="">Sent:<span class="Apple-converted-space"> </span></b>Thursday, 20 December, 2018 11:02:00<br class=""> <b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)<br class=""> </blockquote> </div> <div data-marker="__QUOTED_TEXT__" class=""> <blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> <div style="font-family: arial, helvetica, sans-serif; font-size: 10pt;" class=""> <div class=""></div> <div class="">Hello Yves and all,<br class=""> </div> <br class=""> <div class="">Here's a summary of the basic differences between srun and sbatch:<br class=""> </div> <br class=""> <div class=""><a href="https://stackoverflow.com/questions/43767866/slurm-srun-vs-sbatch-and-their-parameters" class="">https://stackoverflow.com/questions/43767866/slurm-srun-vs-sbatch-and-their-parameters</a><br class=""> </div> <br class=""> <div class="">--<span class="Apple-converted-space"> </span><br class=""> Martin Matthiesen<br class=""> CSC - Tieteen tietotekniikan keskus<br class=""> CSC - IT Center for Science<br class=""> PL 405, 02101 Espoo, Finland<br class=""> +358 9 457 2376, <a href="mailto:martin.matthiesen@csc.fi" class="">martin.matthiesen@csc.fi</a><br class=""> Public key : <a href="https://pgp.mit.edu/pks/lookup?op=get&search=0x74B12876FD890704" class=""> https://pgp.mit.edu/pks/lookup?op=get&search=0x74B12876FD890704</a><br class=""> Fingerprint: AA25 6F56 5C9A 8B42 009F BA70 74B1 2876 FD89 0704</div> <br class=""> <hr id="zwchr" class=""> <div class=""> <blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> <b class="">From:<span class="Apple-converted-space"> </span></b>"Yves Scherrer" <<a href="mailto:yves.scherrer@helsinki.fi" class="">yves.scherrer@helsinki.fi</a>><br class=""> <b class="">To:<span class="Apple-converted-space"> </span></b>"Stephan Oepen" <<a href="mailto:oe@ifi.uio.no" class="">oe@ifi.uio.no</a>><br class=""> <b class="">Cc:<span class="Apple-converted-space"> </span></b>"Martin Matthiesen" <<a href="mailto:martin.matthiesen@csc.fi" class="">martin.matthiesen@csc.fi</a>>, "infrastructure" <<a href="mailto:infrastructure@nlpl.eu" class="">infrastructure@nlpl.eu</a>><br class=""> <b class="">Sent:<span class="Apple-converted-space"> </span></b>Wednesday, 19 December, 2018 17:47:49<br class=""> <b class="">Subject:<span class="Apple-converted-space"> </span></b>RE: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)<br class=""> </blockquote> </div> <div class=""> <blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> <div lang="EN-US" class=""> <div class="x_WordSection1"> <div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> It looks like the “srun” (present in my script, absent in Stephan’s) was the culprit. I still have to say that I haven’t completely grasped its use – back in Theano times, it was compulsory (at least for me, but Jörg was able to run the same jobs without it), now it seems that it must be avoided…</div> <p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;"> </p> <div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Yves</div> <p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;"> </p> </div> <hr style="display: inline-block; width: 564.46875px;" class=""> <div id="x_divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Stephan Oepen <<a href="mailto:oe@ifi.uio.no" class="">oe@ifi.uio.no</a>><br class=""> <b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, December 19, 2018 3:48:05 PM<br class=""> <b class="">To:</b><span class="Apple-converted-space"> </span>Scherrer, Yves<br class=""> <b class="">Cc:</b><span class="Apple-converted-space"> </span>Martin Matthiesen; infrastructure<br class=""> <b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)</font> <div class=""> </div> </div> </div> <font size="2" class=""><span style="font-size: 11pt;" class=""> <div class="PlainText">this is weird: as if i did not get the error, yves? i had stripped<br class=""> down your script to just the training; see<br class=""> <br class=""> /homeappl/home/oe/onmt.sh<br class=""> <br class=""> to confirm, i turned on that job once more earlier today (‘sbatch<br class=""> onmt.sh’), and it appears to be training happily for now. standard<br class=""> output and error from that job should be visible to you in my home<br class=""> directory.<br class=""> <br class=""> for ultimate comparability, could you also run that job<br class=""> <br class=""> sbatch ~oe/onmt.sh<br class=""> <br class=""> oe<br class=""> <br class=""> On Tue, Dec 18, 2018 at 3:15 PM Scherrer, Yves<br class=""> <<a href="mailto:yves.scherrer@helsinki.fi" class="">yves.scherrer@helsinki.fi</a>> wrote:<br class=""> ><br class=""> > Hi,<br class=""> ><br class=""> ><br class=""> ><br class=""> > My error occurs right away, I don’t even get these INFO messages… This is the full content of the training.*.err file:<br class=""> ><br class=""> ><br class=""> ><br class=""> > Loading application python-3.5.3 environment with needed modules<br class=""> ><br class=""> > THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=34 error=35 : CUDA driver version is insufficient for CUDA runtime version<br class=""> ><br class=""> > Traceback (most recent call last):<br class=""> ><br class=""> > File "/proj/nlpl/software/opennmt-py/0.2.1/scripts/train.py", line 40, in <module><br class=""> ><br class=""> > main(opt)<br class=""> ><br class=""> > File "/proj/nlpl/software/opennmt-py/0.2.1/scripts/train.py", line 27, in main<br class=""> ><br class=""> > single_main(opt)<br class=""> ><br class=""> > File "/proj/nlpl/software/opennmt-py/0.2.1/lib/python3.5/site-packages/OpenNMT_py-0.2.1-py3.5.egg/onmt/train_single.py", line 73, in main<br class=""> ><br class=""> > opt = training_opt_postprocessing(opt)<br class=""> ><br class=""> > File "/proj/nlpl/software/opennmt-py/0.2.1/lib/python3.5/site-packages/OpenNMT_py-0.2.1-py3.5.egg/onmt/train_single.py", line 60, in training_opt_postprocessing<br class=""> ><br class=""> > torch.cuda.set_device(opt.device_id)<br class=""> ><br class=""> > File "/proj/nlpl/software/opennmt-py/0.2.1/lib/python3.5/site-packages/torch/cuda/__init__.py", line 264, in set_device<br class=""> ><br class=""> > torch._C._cuda_setDevice(device)<br class=""> ><br class=""> > RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at /pytorch/torch/csrc/cuda/Module.cpp:34<br class=""> ><br class=""> > Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7ff93231b400><br class=""> ><br class=""> > Traceback (most recent call last):<br class=""> ><br class=""> > File "/wrk/project_nlpl/software/opennmt-py/0.2.1/lib/python3.5/weakref.py", line 117, in remove<br class=""> ><br class=""> > TypeError: 'NoneType' object is not callable<br class=""> ><br class=""> > srun: error: g110: task 0: Exited with exit code 1<br class=""> ><br class=""> > srun: Terminating job step 33310480.0<br class=""> ><br class=""> ><br class=""> ><br class=""> ><br class=""> ><br class=""> > ________________________________<br class=""> > From: Stephan Oepen <<a href="mailto:oe@ifi.uio.no" class="">oe@ifi.uio.no</a>><br class=""> > Sent: Tuesday, December 18, 2018 2:56:49 PM<br class=""> > To: Martin Matthiesen<br class=""> > Cc: Scherrer, Yves; infrastructure<br class=""> > Subject: Re: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)<br class=""> ><br class=""> > thanks for adjusting those permissions, yves!<br class=""> ><br class=""> > roughle how long into the job would you expect the error to occur?<br class=""> ><br class=""> > i have been running for around six minutes so far, and training<br class=""> > appears to get going:<br class=""> ><br class=""> > 2018-12-18 14:47:43,683 INFO] encoder: 14116000<br class=""> > [2018-12-18 14:47:43,683 INFO] decoder: 25862084<br class=""> > [2018-12-18 14:47:43,683 INFO] * number of parameters: 39978084<br class=""> > /proj/nlpl/software/opennmt-py/0.2.1/lib/python3.5/site-packages/torch/nn/_reduction.py:49:<br class=""> > UserWarning: size_average and reduce args will be deprecated, please<br class=""> > use reduction='sum' instead.<br class=""> > warnings.warn(warning.format(ret))<br class=""> > [2018-12-18 14:47:43,685 INFO] Start training...<br class=""> > [2018-12-18 14:47:43,707 INFO] Loading train dataset from<br class=""> > <a href="http://data.train.1.pt" class="">data.train.1.pt</a>, number of examples: 1030<br class=""> > /proj/nlpl/software/opennmt-py/0.2.1/lib/python3.5/site-packages/torch/nn/functional.py:1320:<br class=""> > UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.<br class=""> > warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")<br class=""> > [2018-12-18 14:49:15,649 INFO] Loading train dataset from<br class=""> > <a href="http://data.train.10.pt" class="">data.train.10.pt</a>, number of examples: 1162<br class=""> > [2018-12-18 14:50:55,474 INFO] Loading train dataset from<br class=""> > <a href="http://data.train.100.pt" class="">data.train.100.pt</a>, number of examples: 1199<br class=""> > [2018-12-18 14:52:13,191 INFO] Step 50/100000; acc: 5.83; ppl:<br class=""> > 5884.51; xent: 8.68; lr: 1.00000; 272/262 tok/s; 269 sec<br class=""> > [2018-12-18 14:52:38,496 INFO] Loading train dataset from<br class=""> > <a href="http://data.train.1000.pt" class="">data.train.1000.pt</a>, number of examples: 1216<br class=""> ><br class=""> > but earlier you had sent a traceback involving a function called<br class=""> > training_opt_postprocessing() ... so maybe the error ony occurs<br class=""> > towards the end of training? which would seem pretty weird, seeing as<br class=""> > i suppose PyTorch has been used extensively up to that point already?<br class=""> ><br class=""> > oe<br class=""> ><br class=""> ><br class=""> ><br class=""> > On Tue, Dec 18, 2018 at 10:11 AM Martin Matthiesen<br class=""> > <<a href="mailto:martin.matthiesen@csc.fi" class="">martin.matthiesen@csc.fi</a>> wrote:<br class=""> > ><br class=""> > > Hi,<br class=""> > ><br class=""> > > I did try for an hour and a bit yesterday to pinpoint the problem, but could not make head or tail of it. Did I understand correctly that Stephan, you got this working on Taito?<br class=""> > ><br class=""> > > Martin<br class=""> > ><br class=""> > > P.S.: Should we keep infrastructure out of this or is this interesting to Jörg and Björn?<br class=""> > ><br class=""> > > --<br class=""> > > Martin Matthiesen<br class=""> > > CSC - Tieteen tietotekniikan keskus<br class=""> > > CSC - IT Center for Science<br class=""> > > PL 405, 02101 Espoo, Finland<br class=""> > > +358 9 457 2376, <a href="mailto:martin.matthiesen@csc.fi" class="">martin.matthiesen@csc.fi</a><br class=""> > > Public key :<span class="Apple-converted-space"> </span><a href="https://pgp.mit.edu/pks/lookup?op=get&search=0x74B12876FD890704" target="_blank" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">https://pgp.mit.edu/pks/lookup?op=get&search=0x74B12876FD890704</a><br class=""> > > Fingerprint: AA25 6F56 5C9A 8B42 009F BA70 74B1 2876 FD89 0704<br class=""> > ><br class=""> > > ________________________________<br class=""> > ><br class=""> > > From: "Yves Scherrer" <<a href="mailto:yves.scherrer@helsinki.fi" class="">yves.scherrer@helsinki.fi</a>><br class=""> > > To: "Stephan Oepen" <<a href="mailto:oe@ifi.uio.no" class="">oe@ifi.uio.no</a>><br class=""> > > Cc: "Martin Matthiesen" <<a href="mailto:martin.matthiesen@csc.fi" class="">martin.matthiesen@csc.fi</a>>, "infrastructure" <<a href="mailto:infrastructure@nlpl.eu" class="">infrastructure@nlpl.eu</a>><br class=""> > > Sent: Tuesday, 18 December, 2018 10:35:25<br class=""> > > Subject: RE: [NLPL Task Force (A)] OpenNMT installation for NLPL (on Abel)<br class=""> > ><br class=""> > > Hi,<br class=""> > ><br class=""> > ><br class=""> > ><br class=""> > > > could you make the complete data directory group- or world-readable,<br class=""> > > > so i can try running the ‘train.py’ script without creating my own<br class=""> > > > copy of the data?<br class=""> > ><br class=""> > ><br class=""> > ><br class=""> > > That should work now.<br class=""> > ><br class=""> > ><br class=""> > ><br class=""> > > > thinking (possbily over-)optimistically, maybe the problem has<br class=""> > > > magically disappeared already?<br class=""> > ><br class=""> > ><br class=""> > ><br class=""> > > Unfortunately, it hasn’t. Or was I supposed to reinstall the OpenNMT-py module locally?<br class=""> > ><br class=""> > ><br class=""> > ><br class=""> > > Yves<br class=""> > ><br class=""> > ><br class=""> > ><br class=""> > ></div> </span></font></blockquote> </div> </div> </blockquote> </div> </div> </div> </blockquote> </div> <br class=""> </div> </div> </body> </html>