<div><div dir="auto">very much so; should we meet sometime this week to chat? i am free after 14:00 today for example.</div></div><div dir="auto"> </div><div dir="auto">oe</div><div dir="auto"> </div><div> <div class="gmail_quote"><div dir="ltr">On Mon, 21 Jan 2019 at 10:24 Sabry Razick via RT <<a href="mailto:hpc-drift@usit.uio.no">hpc-drift@usit.uio.no</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 2019-01-21 10:10:54, oe wrote: > please go ahead and close this ticket! XZ support works fine in the > new 3.7.0 installation, and i am now well underway to creating a new > batch of NLPL python modules based on that base python. > Hi, Would you be interested in evaluating whether NLPL modules could be installed in a more portable way ? Regards, Sabry > > On Mon, Jan 21, 2019 at 7:35 AM Sabry Razick via RT > <<a href="mailto:hpc-drift@usit.uio.no" target="_blank">hpc-drift@usit.uio.no</a>> wrote: > > > > Hi, > > Do you need further assistance with this or should I close the > > ticket ? > > > > Regards, > > Sabry. > > > > > > On 2019-01-17 10:08:18, sabryr wrote: > > > On 2019-01-17 07:42:18, sabryr wrote: > > > > On 2019-01-16 23:17:48, oe wrote: > > > > > hi sabry, > > > > > > > > > > i was planning to give your new 3.7 a shot, i.e. start > > > > > rebuilding our > > > > > derived modules for NLPL (rather than try to reatroactively fix > > > > > based > > > > > XZ > > > > > support based on 3.5). however, it seems ‘virtualenv’ is not > > > > > available in > > > > > python3/<a href="http://3.7.0." rel="noreferrer" target="_blank">3.7.0.</a> is that something you could still add for us? > > > > > > > > > > oe > > > > > > Hi > > > Both, > > > python3/3.7.0.gnu > > > and > > > python3/3.7.0 (the intel version) > > > > > > Have virtualenv in pythonhome (no PYTHONPATH used) > > > > > > -bash-4.1$ virtualenv --version > > > 16.2.0 > > > > > > Regards, > > > Sabry > > > > > > > > > > Hi, > > > > I will check this today. There has been some discussions about > > > > the > > > > portability of NLPL modules, I could assist you to make it a > > > > regular > > > > module, if you think that is useful. > > > > Regards > > > > Sabry > > > > > On Tue, 15 Jan 2019 at 09:56 Sabry Razick via RT <hpc- > > > > > <a href="mailto:drift@usit.uio.no" target="_blank">drift@usit.uio.no</a>> > > > > > wrote: > > > > > > > > > > > Hello, > > > > > > Will the module "module load Python/3.5.2-foss-2016b" work > > > > > > for > > > > > > you all > > > > > > you want to recompile a new one ? > > > > > > > > > > > > Regards, > > > > > > Sabry. > > > > > > > > > > > > On 2019-01-08 14:34:40, sabryr wrote: > > > > > > > Hi, > > > > > > > > > > > > > > On 2019-01-08 13:35:25, oe wrote: > > > > > > > > many thanks for the quick follow-up, sabry! > > > > > > > > > > > > > > You are welcome > > > > > > > > > > > > > > > would there be a relatively easy way of retroactively > > > > > > > > fixing > > > > > > > > the > > > > > > > > 3.5.0 > > > > > > > > module? > > > > > > > > > > > > > > No, due to the policy that installed modules will not > > > > > > > change. As > > > > > > > I > > > > > > > would have to recompile,can not guarantee the change will > > > > > > > be only > > > > > > > the > > > > > > > lzima,thus potentially effect others. But I can install > > > > > > > another > > > > > > > 3.5. > > > > > > > module for you, if miner version is not a problem. > > > > > > > > > > > > > > For example is following module usable (a secret module we > > > > > > > have) > > > > > > > > > > > > > > module load Python/3.5.2-foss-2016b > > > > > > > > > > > > > > If not, we can install similar semi-secret module. > > > > > > > > > > > > > > i am asking because in the NLPL context we have produced at > > > > > > > > least half a dozen python modules that are derived from > > > > > > > > the > > > > > > > > base > > > > > > > > ‘python3/3.5.0’ module. here, derivative means creating > > > > > > > > a > > > > > > > > virtual > > > > > > > > environment, installing one or more add-on packages, and > > > > > > > > tweaking > > > > > > > > PYTHONPATH such that several such derivative modules can > > > > > > > > be > > > > > > > > combined. > > > > > > > > we have adopted this path for NLPL, for better > > > > > > > > modularity, > > > > > > > > interoperability, and uniformity between Abel and Taito. > > > > > > > > some > > > > > > > > additional background on this approach is on the NLPL > > > > > > > > wiki: > > > > > > > > > > > > > > > > <a href="http://wiki.nlpl.eu/index.php/Infrastructure/software/python" rel="noreferrer" target="_blank">http://wiki.nlpl.eu/index.php/Infrastructure/software/python</a> > > > > > > > > > > > > > > As the workload will be less on our end, I will try to help > > > > > > > on > > > > > > > this. > > > > > > > > > > > > > > > > > > > > > > switching to a different base module (e.g. from 3.5.0 to > > > > > > > > 3.7.0) > > > > > > > > would > > > > > > > > require rebuilding all of these derivative modules (hence > > > > > > > > is > > > > > > > > not > > > > > > > > something i would not do light-heartedly :-). but see > > > > > > > > below > > > > > > > > ... > > > > > > > > > > > > > > > > —more generally, we have been meaning to start a fresh > > > > > > > > batch of > > > > > > > > derivative modules based on 3.6 or even 3.7. to start > > > > > > > > down > > > > > > > > this > > > > > > > > route, it would be important to agree on which > > > > > > > > characteristics > > > > > > > > the > > > > > > > > base python module should have. (unlike the older 3.5.0 > > > > > > > > installation) > > > > > > > > i see that your new 3.7.0 is minimal, i.e. does not (yet) > > > > > > > > include any > > > > > > > > add-on components. > > > > > > > > > > > > > > That is correct. The future plan is to provide basic python > > > > > > > and > > > > > > > another module called "python-modules". There is no > > > > > > > PYTHONPATH > > > > > > > set in > > > > > > > this module either and highly customizable. > > > > > > > > > > > > > > in principle, i think this is right, and all > > > > > > > > add-ons should arguably be provisioned as separate > > > > > > > > modules. > > > > > > > > what > > > > > > > > about NumPy with MKL support, however? that was compiled > > > > > > > > into > > > > > > > > the > > > > > > > > 3.5.0 installation and is something i suspect many users > > > > > > > > might > > > > > > > > want. > > > > > > > > or maybe the default OpenBLAS back-end for NumPy is just > > > > > > > > as > > > > > > > > good (on > > > > > > > > average), and we can make do with standard, pre-compiled > > > > > > > > NumPy > > > > > > > > wheels? > > > > > > > > if so, is there a particular reason you opted to build > > > > > > > > the new > > > > > > > > 3.7.0 > > > > > > > > with the Intel tool chain? > > > > > > > > > > > > > > Latest MKL, older OpenBLAS, (latest blas and Atlas soon) > > > > > > > available > > > > > > > when you load the " module load python3/3.7.0", but you > > > > > > > must > > > > > > > compile > > > > > > > numpy (not from wheel) to use it. > > > > > > > > > > > > > > $ module load python3/3.7.0 > > > > > > > $ echo $CPATH | grep mkl --color=always > > > > > > > > > > > > > > /cluster/software/VERSIONS/intel- > > > > > > > 2019.0/compilers_and_libraries_2019.0.117/linux/mkl/include/ > > > > > > > /cluster/software/VERSIONS/intel- > > > > > > > 2019.0/compilers_and_libraries_2019.0.117/linux/mkl/include/fftw: > > > > > > > > > > > > > > > > > > > > > > in the NLPL context, we have been tossing around some of > > > > > > > > these > > > > > > > > high-level design questions for a little while now, and i > > > > > > > > am > > > > > > > > hoping > > > > > > > > to > > > > > > > > tap into your broader experience with maintaining modules > > > > > > > > on > > > > > > > > Abel > > > > > > > > here. would it make sense to discuss some of the more > > > > > > > > general > > > > > > > > questions over a cup of coffee sometime this week? > > > > > > > > anyone else > > > > > > > > in > > > > > > > > your group with strong opinions about python modules? > > > > > > > > > > > > > > Yes we could discuss this, I do not think there are strong > > > > > > > opinions, > > > > > > > but I could ask permission from Jon if needed . Only thing > > > > > > > is > > > > > > > once > > > > > > > installed modules are (almost) newer changed. > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > Sabry > > > > > > > > > > > > > > > > with thanks in advance, oe > > > > > > > > > > > > > > > > On Tue, Jan 8, 2019 at 11:31 AM Sabry Razick via RT > > > > > > > > <<a href="mailto:hpc-drift@usit.uio.no" target="_blank">hpc-drift@usit.uio.no</a>> wrote: > > > > > > > > > > > > > > > > > > On 2019-01-07 23:20:24, oe wrote: > > > > > > > > > > colleagues, > > > > > > > > > > > > > > > > > > > > using the default ‘python3’ module on Abel, i am > > > > > > > > > > running > > > > > > > > > > into the > > > > > > > > > > following import error (required by a third-party > > > > > > > > > > component): > > > > > > > > > > > > > > > > > > > > [oe@login-0-0 ~]$ module purge; module load > > > > > > > > > > python3/3.5.0 > > > > > > > > > > [oe@login-0-0 ~]$ type -all python3 > > > > > > > > > > python3 is /cluster/software/VERSIONS/python3- > > > > > > > > > > 3.5.0/bin/python3 > > > > > > > > > > [oe@login-0-0 ~]$ python3 -c "import lzma;" > > > > > > > > > > Traceback (most recent call last): > > > > > > > > > > File "<string>", line 1, in <module> > > > > > > > > > > File "/cluster/software/VERSIONS/python3- > > > > > > > > > > 3.5.0/lib/python3.5/lzma.py", > > > > > > > > > > line 26, in <module> > > > > > > > > > > from _lzma import * > > > > > > > > > > ImportError: No module named '_lzma' > > > > > > > > > > > > > > > > > > > > for all i can tell, there are no RHEL lzma packaged > > > > > > > > > > installed, > > > > > > > > > > but > > > > > > > > > > there is a separate module ‘xz/5.2.2’. i wonder > > > > > > > > > > whether my > > > > > > > > > > import > > > > > > > > > > error could just mean that no lzma (aka xz) headers > > > > > > > > > > and > > > > > > > > > > libraries > > > > > > > > > > were > > > > > > > > > > visible when the python3 installation was compiled? > > > > > > > > > > > > > > > > > > > > with thanks in advance, oe > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > Your observation is correct. I have fixed this in the > > > > > > > > > latest > > > > > > > > > version (did not fix the old). > > > > > > > > > > > > > > > > > > -bash-4.1$ module purge > > > > > > > > > -bash-4.1$ module load python3/3.7.0 > > > > > > > > > -bash-4.1$ python3 -c "import lzma;" > > > > > > > > > -bash-4.1$ python3 -c "import > > > > > > > > > lzma;print(lzma.__file__)" > > > > > > > > > /cluster/software/VERSIONS/python3- > > > > > > > > > 3.7.0/lib/python3.7/lzma.py > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Sabry > > > > > > > > > > > > > > > > > > > > > > > </blockquote></div></div>