[NLPL Task Force (A)] [rt.uio.no #3243738] XY support in Python3

[Sabry Razick via RT]

Hi,

On 2019-01-08 13:35:25, oe wrote:
> many thanks for the quick follow-up, sabry!

You are welcome
 
> would there be a relatively easy way of retroactively fixing the 3.5.0
> module? 

No, due to the policy that installed modules will not change. As I would have to recompile,can not guarantee the change will be only the lzima,thus potentially effect others. But I can install another 3.5. module for you, if miner version is not a problem. 

For example is following module usable (a secret module we have)

module load Python/3.5.2-foss-2016b

If not, we can install similar semi-secret module. 

 i am asking because in the NLPL context we have produced at
> least half a dozen python modules that are derived from the base
> ‘python3/3.5.0’ module.  here, derivative means creating a virtual
> environment, installing one or more add-on packages, and tweaking
> PYTHONPATH such that several such derivative modules can be combined.
> we have adopted this path for NLPL, for better modularity,
> interoperability, and uniformity between Abel and Taito.  some
> additional background on this approach is on the NLPL wiki:
> 
> http://wiki.nlpl.eu/index.php/Infrastructure/software/python

As the workload will be less on our end, I will try to help on this. 


> switching to a different base module (e.g. from 3.5.0 to 3.7.0) would
> require rebuilding all of these derivative modules (hence is not
> something i would not do light-heartedly :-).  but see below ...
> 
> —more generally, we have been meaning to start a fresh batch of
> derivative modules based on 3.6 or even 3.7.  to start down this
> route, it would be important to agree on which characteristics the
> base python module should have.  (unlike the older 3.5.0 installation)
> i see that your new 3.7.0 is minimal, i.e. does not (yet) include any
> add-on components. 

That is correct. The future plan is to provide basic python and another module called "python-modules". There is no PYTHONPATH set in this module either and highly customizable.  

 in principle, i think this is right, and all
> add-ons should arguably be provisioned as separate modules.  what
> about NumPy with MKL support, however?  that was compiled into the
> 3.5.0 installation and is something i suspect many users might want.
> or maybe the default OpenBLAS back-end for NumPy is just as good (on
> average), and we can make do with standard, pre-compiled NumPy wheels?
>  if so, is there a particular reason you opted to build the new 3.7.0
> with the Intel tool chain?

Latest MKL, older OpenBLAS, (latest blas and Atlas soon) available when you load the " module load python3/3.7.0", but you must compile numpy (not from wheel) to use it.

$ module load python3/3.7.0
$ echo $CPATH | grep mkl --color=always

/cluster/software/VERSIONS/intel-2019.0/compilers_and_libraries_2019.0.117/linux/mkl/include/
/cluster/software/VERSIONS/intel-2019.0/compilers_and_libraries_2019.0.117/linux/mkl/include/fftw:


> in the NLPL context, we have been tossing around some of these
> high-level design questions for a little while now, and i am hoping to
> tap into your broader experience with maintaining modules on Abel
> here.  would it make sense to discuss some of the more general
> questions over a cup of coffee sometime this week?  anyone else in
> your group with strong opinions about python modules?

Yes we could discuss this, I do not think there are strong opinions, but I could ask permission from Jon if needed .  Only thing is once installed modules are (almost) newer changed.


Regards,
Sabry
> 
> with thanks in advance, oe
> 
> On Tue, Jan 8, 2019 at 11:31 AM Sabry Razick via RT
> <hpc-drift at usit.uio.no> wrote:
> >
> > On 2019-01-07 23:20:24, oe wrote:
> > > colleagues,
> > >
> > > using the default ‘python3’ module on Abel, i am running into the
> > > following import error (required by a third-party component):
> > >
> > > [oe at login-0-0 ~]$ module purge; module load python3/3.5.0
> > > [oe at login-0-0 ~]$ type -all python3
> > > python3 is /cluster/software/VERSIONS/python3-3.5.0/bin/python3
> > > [oe at login-0-0 ~]$ python3 -c "import lzma;"
> > > Traceback (most recent call last):
> > >   File "<string>", line 1, in <module>
> > >   File "/cluster/software/VERSIONS/python3-
> > > 3.5.0/lib/python3.5/lzma.py",
> > > line 26, in <module>
> > >     from _lzma import *
> > > ImportError: No module named '_lzma'
> > >
> > > for all i can tell, there are no RHEL lzma packaged installed, but
> > > there is a separate module ‘xz/5.2.2’.  i wonder whether my import
> > > error could just mean that no lzma (aka xz) headers and libraries
> > > were
> > > visible when the python3 installation was compiled?
> > >
> > > with thanks in advance, oe
> > >
> >
> > Hi,
> >   Your observation is correct. I have fixed this in the latest
> > version (did not fix the old).
> >
> > -bash-4.1$ module purge
> > -bash-4.1$ module load python3/3.7.0
> > -bash-4.1$ python3 -c "import lzma;"
> > -bash-4.1$ python3 -c "import lzma;print(lzma.__file__)"
> > /cluster/software/VERSIONS/python3-3.7.0/lib/python3.7/lzma.py
> >
> >
> > Regards,
> > Sabry
> >