<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Stephan,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for this.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I hadn't module loaded Cuda in my scripts. I'm going back and looking through some of the instructions now. So I guess when using CUDA, it is required to module load cuda/[version]? I've done as follows:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: Consolas, Courier, monospace;">module load cuda/8.0</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: Consolas, Courier, monospace;">module load nlpl-cupy</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
When I try <span style="font-family: Consolas, Courier, monospace;">module load nlpl-cupy</span> I get the following message:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>
<p style="color: rgb(0, 0, 0); font-family: Menlo; font-size: 11px; margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures">ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'nlpl-cupy'</span></p>
<div style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
When I use <span style="font-family: Consolas, Courier, monospace;">module -h avail</span><span style="font-family: Arial, Helvetica, sans-serif;"> I also don't see nlpl-cupy there.</span></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br>
</span></div>
<div><font color="#000000" face="Arial, Helvetica, sans-serif" size="3">However, despite that error message it seems (going by my </font><font color="#000000" face="Arial, Helvetica, sans-serif">slurm output) that the problem is no longer cupy failing to connect
to the CUDA:</font></div>
</div>
<div><font color="#000000" face="Arial, Helvetica, sans-serif"><br>
</font></div>
<div><font color="#000000" face="Arial, Helvetica, sans-serif">
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000"><span style="font-variant-ligatures: no-common-ligatures">-bash-4.1$ cat slurm-26960146.out</span></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000"><span style="font-variant-ligatures: no-common-ligatures">ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'nlpl-cupy'</span></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000"><span style="font-variant-ligatures: no-common-ligatures">Traceback (most recent call last):</span></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000"><span style="font-variant-ligatures: no-common-ligatures"><span>
</span>File "/usit/abel/u1/andidyer/vecmap/map_embeddings.py", line 422, in <module></span></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000"><span style="font-variant-ligatures: no-common-ligatures"><span>
</span>main()</span></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000"><span style="font-variant-ligatures: no-common-ligatures"><span>
</span>File "/usit/abel/u1/andidyer/vecmap/map_embeddings.py", line 148, in main</span></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000"><span style="font-variant-ligatures: no-common-ligatures"><span>
</span>trg_words, z = embeddings.read(trgfile, dtype=dtype)</span></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000"><span style="font-variant-ligatures: no-common-ligatures"><span>
</span>File "/cluster/home/andidyer/vecmap/embeddings.py", line 35, in read</span></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000"><span style="font-variant-ligatures: no-common-ligatures"><span>
</span>matrix[i] = np.fromstring(vec, sep=' ', dtype=dtype)</span></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000"><span style="font-variant-ligatures: no-common-ligatures">ValueError: could not broadcast input array from shape (125) into shape (300)</span></p>
<br>
</font></div>
<div><font color="#000000" face="Arial, Helvetica, sans-serif">So something is obviously going right!</font></div>
<div><font color="#000000" face="Arial, Helvetica, sans-serif"><br>
</font></div>
<div><font color="#000000" face="Arial, Helvetica, sans-serif">Again, many thanks for your assistance.</font></div>
<div><font color="#000000" face="Arial, Helvetica, sans-serif"><br>
</font></div>
<div><font color="#000000" face="Arial, Helvetica, sans-serif"><span style="caret-color: rgb(0, 0, 0);">Best wishes,</span></font></div>
<div><font color="#000000" face="Arial, Helvetica, sans-serif"><span style="caret-color: rgb(0, 0, 0);"><br>
</span></font></div>
<div><font color="#000000" face="Arial, Helvetica, sans-serif"><span style="caret-color: rgb(0, 0, 0);">Andrew</span></font></div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Stephan Oepen <oe@ifi.uio.no><br>
<b>Sent:</b> 09 May 2019 18:17<br>
<b>To:</b> Andrew Dyer<br>
<b>Cc:</b> infrastructure@nlpl.eu<br>
<b>Subject:</b> Re: [NLPL Task Force (A)] Trouble with using GPU in Cupy package</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">hi andrew,<br>
<br>
it appears your local cupy installation is somewhat unable to find its<br>
external dependencies. have you 'module load'ed the right CUDA<br>
version? are you sure it ended up running on a gpu node?<br>
<br>
this things can be tricky to sort out, what with the many different<br>
(and mutually incompatible) module versions available on a large and<br>
old system like Abel.<br>
<br>
CuPy looks like a relevant tool for the NLPL software inventory, so i<br>
installed it as an NLPL module. the following (when running on a gpu<br>
node) appears to work:<br>
<br>
[oe@compute-19-1 ~]$ module purge; module load nlpl-cupy<br>
module list<br>
[oe@compute-19-1 ~]$ module list<br>
Currently Loaded Modulefiles:<br>
1) intel/2019.0 4) gcc/4.9.2 7)<br>
nlpl-cython/0.29.3/3.7<br>
2) openssl.intel/1_1_1 5) cuda/9.0 8)<br>
nlpl-scipy/201901/3.7<br>
3) python3/3.7.0 6) nlpl-numpy/1.16.0/3.7 9) nlpl-cupy/5.4.0/3.7<br>
[oe@compute-19-1 ~]$ python3 -c "import cupy; print(cupy.__version__);"<br>
5.4.0<br>
<br>
in general, i would suggest testing things interactive first, before<br>
you invest the time in putting a job in the queue. these past few<br>
days, it appears there can be fairly long wait times for gpu nodes on<br>
Abel (we are really looking foward to transitioning to the new system<br>
after the summer). but in principle, one can create an interactive<br>
session on a gpu node as follows:<br>
<br>
qlogin --account=nn9447k --time=00:30:00 --mem-per-cpu=2048M<br>
--partition=accel --gres=gpu:1<br>
<br>
please see whether the new NLPL version of CuPy works for you (but<br>
please make sure there are no unwanted interactions with your local<br>
virtualenv)?<br>
<br>
best wishes, oe<br>
<br>
<br>
On Mon, May 6, 2019 at 2:27 PM Andrew Dyer<br>
<Andrew.Dyer.6854@student.uu.se> wrote:<br>
><br>
> Hi,<br>
><br>
> Apologies for the bother. I'm currently trying to run an experiment using GPU nodes in Abel. The Python program that I am using uses Cupy, which I have installed in my venv with pip. On my sbatch script, I set the GPU request as instructed on the Job Scripts
page:<br>
><br>
> #SBATCH --partition=accel --gres=gpu:1<br>
><br>
> However, the program that I'm using seems to be having trouble connecting to the CUDA software. I've checked that the versions match (8.0). I'm at a loss for what else to do though, so any help you can provide would be appreciated.<br>
><br>
> For reference, see attached my script and the error message in the slurm output.<br>
><br>
> Many thanks,<br>
><br>
> Andrew<br>
</div>
</span></font></div>
</body>
</html>