<div dir="ltr"><div><div dir="auto">hi sabry,</div><div dir="auto"> </div><div dir="auto">i was planning to give your new 3.7 a shot, i.e. start rebuilding our derived modules for NLPL (rather than try to reatroactively fix based XZ support based on 3.5). however, it seems ‘virtualenv’ is not available in python3/<a href="http://3.7.0.">3.7.0.</a> is that something you could still add for us?</div><div dir="auto"> </div><div>oe</div><div> </div></div></div><div> <div class="gmail_quote"><div dir="ltr">On Tue, 15 Jan 2019 at 09:56 Sabry Razick via RT <<a href="mailto:hpc-drift@usit.uio.no" target="_blank">hpc-drift@usit.uio.no</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello, Will the module "module load Python/3.5.2-foss-2016b" work for you all you want to recompile a new one ? Regards, Sabry. On 2019-01-08 14:34:40, sabryr wrote: > Hi, > > On 2019-01-08 13:35:25, oe wrote: > > many thanks for the quick follow-up, sabry! > > You are welcome > > > would there be a relatively easy way of retroactively fixing the > > 3.5.0 > > module? > > No, due to the policy that installed modules will not change. As I > would have to recompile,can not guarantee the change will be only the > lzima,thus potentially effect others. But I can install another 3.5. > module for you, if miner version is not a problem. > > For example is following module usable (a secret module we have) > > module load Python/3.5.2-foss-2016b > > If not, we can install similar semi-secret module. > > i am asking because in the NLPL context we have produced at > > least half a dozen python modules that are derived from the base > > ‘python3/3.5.0’ module. here, derivative means creating a virtual > > environment, installing one or more add-on packages, and tweaking > > PYTHONPATH such that several such derivative modules can be combined. > > we have adopted this path for NLPL, for better modularity, > > interoperability, and uniformity between Abel and Taito. some > > additional background on this approach is on the NLPL wiki: > > > > <a href="http://wiki.nlpl.eu/index.php/Infrastructure/software/python" rel="noreferrer" target="_blank">http://wiki.nlpl.eu/index.php/Infrastructure/software/python</a> > > As the workload will be less on our end, I will try to help on this. > > > > switching to a different base module (e.g. from 3.5.0 to 3.7.0) would > > require rebuilding all of these derivative modules (hence is not > > something i would not do light-heartedly :-). but see below ... > > > > —more generally, we have been meaning to start a fresh batch of > > derivative modules based on 3.6 or even 3.7. to start down this > > route, it would be important to agree on which characteristics the > > base python module should have. (unlike the older 3.5.0 > > installation) > > i see that your new 3.7.0 is minimal, i.e. does not (yet) include any > > add-on components. > > That is correct. The future plan is to provide basic python and > another module called "python-modules". There is no PYTHONPATH set in > this module either and highly customizable. > > in principle, i think this is right, and all > > add-ons should arguably be provisioned as separate modules. what > > about NumPy with MKL support, however? that was compiled into the > > 3.5.0 installation and is something i suspect many users might want. > > or maybe the default OpenBLAS back-end for NumPy is just as good (on > > average), and we can make do with standard, pre-compiled NumPy > > wheels? > > if so, is there a particular reason you opted to build the new 3.7.0 > > with the Intel tool chain? > > Latest MKL, older OpenBLAS, (latest blas and Atlas soon) available > when you load the " module load python3/3.7.0", but you must compile > numpy (not from wheel) to use it. > > $ module load python3/3.7.0 > $ echo $CPATH | grep mkl --color=always > > /cluster/software/VERSIONS/intel- > 2019.0/compilers_and_libraries_2019.0.117/linux/mkl/include/ > /cluster/software/VERSIONS/intel- > 2019.0/compilers_and_libraries_2019.0.117/linux/mkl/include/fftw: > > > > in the NLPL context, we have been tossing around some of these > > high-level design questions for a little while now, and i am hoping > > to > > tap into your broader experience with maintaining modules on Abel > > here. would it make sense to discuss some of the more general > > questions over a cup of coffee sometime this week? anyone else in > > your group with strong opinions about python modules? > > Yes we could discuss this, I do not think there are strong opinions, > but I could ask permission from Jon if needed . Only thing is once > installed modules are (almost) newer changed. > > > Regards, > Sabry > > > > with thanks in advance, oe > > > > On Tue, Jan 8, 2019 at 11:31 AM Sabry Razick via RT > > <<a href="mailto:hpc-drift@usit.uio.no" target="_blank">hpc-drift@usit.uio.no</a>> wrote: > > > > > > On 2019-01-07 23:20:24, oe wrote: > > > > colleagues, > > > > > > > > using the default ‘python3’ module on Abel, i am running into the > > > > following import error (required by a third-party component): > > > > > > > > [oe@login-0-0 ~]$ module purge; module load python3/3.5.0 > > > > [oe@login-0-0 ~]$ type -all python3 > > > > python3 is /cluster/software/VERSIONS/python3-3.5.0/bin/python3 > > > > [oe@login-0-0 ~]$ python3 -c "import lzma;" > > > > Traceback (most recent call last): > > > > File "<string>", line 1, in <module> > > > > File "/cluster/software/VERSIONS/python3- > > > > 3.5.0/lib/python3.5/lzma.py", > > > > line 26, in <module> > > > > from _lzma import * > > > > ImportError: No module named '_lzma' > > > > > > > > for all i can tell, there are no RHEL lzma packaged installed, > > > > but > > > > there is a separate module ‘xz/5.2.2’. i wonder whether my > > > > import > > > > error could just mean that no lzma (aka xz) headers and libraries > > > > were > > > > visible when the python3 installation was compiled? > > > > > > > > with thanks in advance, oe > > > > > > > > > > Hi, > > > Your observation is correct. I have fixed this in the latest > > > version (did not fix the old). > > > > > > -bash-4.1$ module purge > > > -bash-4.1$ module load python3/3.7.0 > > > -bash-4.1$ python3 -c "import lzma;" > > > -bash-4.1$ python3 -c "import lzma;print(lzma.__file__)" > > > /cluster/software/VERSIONS/python3-3.7.0/lib/python3.7/lzma.py > > > > > > > > > Regards, > > > Sabry > > > </blockquote></div></div>