<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <meta name="Generator" content="Microsoft Exchange Server"> <style></style> </head> <body> <meta content="text/html; charset=UTF-8"> <style type="text/css" style="">  </style> <div dir="ltr"> <div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif"> Hi Stephan, Thanks for your quick help. Yes, I mean the model files generated by OpenNMT. Actually, the job has finished and the slurm log file has all the log information. I just tried to run the job on the log-node (cpu), the OpenNMT could generate models successfully. I think the problem happens when using GPUs. The log file should have a log like this: "[2019-09-27 21:24:57,888 INFO] Saving checkpoint /usit/abel/u1/gtang/model_word_step_200.pt" However, there is no such log in the slurm file. I guess that OpenNMT does not save models at all. (have no access?) Best, Gongbo </div> <hr tabindex="-1" style="display:inline-block; width:98%"> <div id="x_divRplyFwdMsg" dir="ltr">From: Stephan Oepen <oe@ifi.uio.no> Sent: Friday, September 27, 2019 10:21:29 PM To: Gongbo Tang Cc: hpc@usit.uio.no; infrastructure Subject: Re: [NLPL Task Force (A)] Jobs' output <div> </div> </div> </div> <div class="PlainText">hi gonbo, i cannot say that i have used OpenNMT much myself, but more generally: unless i run something that is very i/o-intensive, i do not take the trouble of copying input and output data back and forth between the $SCRATCH filesystem, i.e. i doubt you need to worry about chkfile and friends. i would just work out of your home directory, i.e. read and write data there. the SLURM log file you sent does not look as if the job actually has completed? i assume by 'output files' you mean files generated during the OpenNMT run, i.e. the actual model file? i might guess that the model is only serialized to disk upon completion of the training, so could it be the case that your job actually had not gotten to that point? a general piece of advice: to debug it might help to reduce the problem to a tiny training file, possibly even something that can complete in a matter of a few minutes on a cpu node. that should allow you to find out where the output file(s) end up, and once you have a working set-up, you can submit larger jobs (to the gpu nodes). best wishes, oe On Fri, Sep 27, 2019 at 10:09 PM Gongbo Tang <gongbo.tang@lingfil.uu.se> wrote: > > Hi, > > > I met a problem. I cannot find any output files/models after running a job. Or the job did not generate any models during running. > > > I am using Open-NMT 0.2.1, maintained by NLPL. I did not find any "Saving checkpoint ..." information from the log file which should be found. I attached the slurm file and the job script. > > > I tried to use "chkfile" or "cleanup" command to save the outputs, following the guide here (<a href="https://www.uio.no/english/services/it/research/hpc/abel/help/user-guide/job-scripts.html#Work_Directory">https://www.uio.no/english/services/it/research/hpc/abel/help/user-guide/job-scripts.html#Work_Directory</a>), but I was told that "chkfile" and "cleanup" are not found. > > > I also tried to set the output directory as the home directory(~, /usit/abel/u1/gtang). I still got nothing. > > > Could you please tell me how can I get the job's outputs? Thanks a lot! > > > Best, > > Gongbo > > > > > > > > > > När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här: <a href="http://www.uu.se/om-uu/dataskydd-personuppgifter/">http://www.uu.se/om-uu/dataskydd-personuppgifter/</a> > > E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: <a href="http://www.uu.se/en/about-uu/data-protection-policy">http://www.uu.se/en/about-uu/data-protection-policy</a> </div> </body> </html>