[NLPL Task Force (A)] [rt.uio.no #3243738] XY support in Python3

[Stephan Oepen via RT]

many thanks for the quick follow-up, sabry!

would there be a relatively easy way of retroactively fixing the 3.5.0
module?  i am asking because in the NLPL context we have produced at
least half a dozen python modules that are derived from the base
‘python3/3.5.0’ module.  here, derivative means creating a virtual
environment, installing one or more add-on packages, and tweaking
PYTHONPATH such that several such derivative modules can be combined.
we have adopted this path for NLPL, for better modularity,
interoperability, and uniformity between Abel and Taito.  some
additional background on this approach is on the NLPL wiki:

http://wiki.nlpl.eu/index.php/Infrastructure/software/python

switching to a different base module (e.g. from 3.5.0 to 3.7.0) would
require rebuilding all of these derivative modules (hence is not
something i would not do light-heartedly :-).  but see below ...

—more generally, we have been meaning to start a fresh batch of
derivative modules based on 3.6 or even 3.7.  to start down this
route, it would be important to agree on which characteristics the
base python module should have.  (unlike the older 3.5.0 installation)
i see that your new 3.7.0 is minimal, i.e. does not (yet) include any
add-on components.  in principle, i think this is right, and all
add-ons should arguably be provisioned as separate modules.  what
about NumPy with MKL support, however?  that was compiled into the
3.5.0 installation and is something i suspect many users might want.
or maybe the default OpenBLAS back-end for NumPy is just as good (on
average), and we can make do with standard, pre-compiled NumPy wheels?
 if so, is there a particular reason you opted to build the new 3.7.0
with the Intel tool chain?

in the NLPL context, we have been tossing around some of these
high-level design questions for a little while now, and i am hoping to
tap into your broader experience with maintaining modules on Abel
here.  would it make sense to discuss some of the more general
questions over a cup of coffee sometime this week?  anyone else in
your group with strong opinions about python modules?

with thanks in advance, oe

On Tue, Jan 8, 2019 at 11:31 AM Sabry Razick via RT
<hpc-drift at usit.uio.no> wrote:
>
> On 2019-01-07 23:20:24, oe wrote:
> > colleagues,
> >
> > using the default ‘python3’ module on Abel, i am running into the
> > following import error (required by a third-party component):
> >
> > [oe at login-0-0 ~]$ module purge; module load python3/3.5.0
> > [oe at login-0-0 ~]$ type -all python3
> > python3 is /cluster/software/VERSIONS/python3-3.5.0/bin/python3
> > [oe at login-0-0 ~]$ python3 -c "import lzma;"
> > Traceback (most recent call last):
> >   File "<string>", line 1, in <module>
> >   File "/cluster/software/VERSIONS/python3-3.5.0/lib/python3.5/lzma.py",
> > line 26, in <module>
> >     from _lzma import *
> > ImportError: No module named '_lzma'
> >
> > for all i can tell, there are no RHEL lzma packaged installed, but
> > there is a separate module ‘xz/5.2.2’.  i wonder whether my import
> > error could just mean that no lzma (aka xz) headers and libraries were
> > visible when the python3 installation was compiled?
> >
> > with thanks in advance, oe
> >
>
> Hi,
>   Your observation is correct. I have fixed this in the latest version (did not fix the old).
>
> -bash-4.1$ module purge
> -bash-4.1$ module load python3/3.7.0
> -bash-4.1$ python3 -c "import lzma;"
> -bash-4.1$ python3 -c "import lzma;print(lzma.__file__)"
> /cluster/software/VERSIONS/python3-3.7.0/lib/python3.7/lzma.py
>
>
> Regards,
> Sabry
>