[NLPL Task Force (A)] [rt.uio.no #3243738] XY support in Python3

Mon Jan 21 09:44:13 UTC 2019

7th floor, right by the door to your left as you get out of the elevator,
i.e. face towards the t-bane tracks.

thanks for making the trip :-)!  oe

On Mon, 21 Jan 2019 at 10:39 Sabry Razick via RT <hpc-drift at usit.uio.no>
wrote:

> On 2019-01-21 10:35:01, oe wrote:
> > very much so; should we meet sometime this week to chat?  i am free after
> > 14:00 today for example.
> >
> > oe
>
> OK, I will come to you office at 14:00. Is your room correct here:
> https://www.mn.uio.no/ifi/personer/vit/oe/index.html
>
>
> Regards,
> Sabry
>
> >
> > On Mon, 21 Jan 2019 at 10:24 Sabry Razick via RT <hpc-drift at usit.uio.no>
> > wrote:
> >
> > > On 2019-01-21 10:10:54, oe wrote:
> > > > please go ahead and close this ticket!  XZ support works fine in the
> > > > new 3.7.0 installation, and i am now well underway to creating a new
> > > > batch of NLPL python modules based on that base python.
> > > >
> > >
> > > Hi,
> > >   Would you be interested in evaluating whether NLPL modules could be
> > > installed in a more portable way ?
> > >
> > > Regards,
> > > Sabry
> > >
> > > >
> > > > On Mon, Jan 21, 2019 at 7:35 AM Sabry Razick via RT
> > > > <hpc-drift at usit.uio.no> wrote:
> > > > >
> > > > > Hi,
> > > > >   Do you need further assistance with this or should I close the
> > > > > ticket ?
> > > > >
> > > > > Regards,
> > > > > Sabry.
> > > > >
> > > > >
> > > > > On 2019-01-17 10:08:18, sabryr wrote:
> > > > > > On 2019-01-17 07:42:18, sabryr wrote:
> > > > > > > On 2019-01-16 23:17:48, oe wrote:
> > > > > > > > hi sabry,
> > > > > > > >
> > > > > > > > i was planning to give your new 3.7 a shot, i.e. start
> > > > > > > > rebuilding our
> > > > > > > > derived modules for NLPL (rather than try to reatroactively
> fix
> > > > > > > > based
> > > > > > > > XZ
> > > > > > > > support based on 3.5).  however, it seems ‘virtualenv’ is not
> > > > > > > > available in
> > > > > > > > python3/3.7.0.  is that something you could still add for
> us?
> > > > > > > >
> > > > > > > > oe
> > > > > >
> > > > > > Hi
> > > > > >   Both,
> > > > > > python3/3.7.0.gnu
> > > > > > and
> > > > > > python3/3.7.0   (the intel version)
> > > > > >
> > > > > > Have virtualenv in pythonhome (no PYTHONPATH used)
> > > > > >
> > > > > > -bash-4.1$ virtualenv --version
> > > > > > 16.2.0
> > > > > >
> > > > > > Regards,
> > > > > > Sabry
> > > > > >
> > > > > >
> > > > > > > Hi,
> > > > > > >   I will check this today. There has been some discussions
> about
> > > > > > > the
> > > > > > > portability of NLPL modules, I could assist you to make it a
> > > > > > > regular
> > > > > > > module, if you think that is useful.
> > > > > > > Regards
> > > > > > > Sabry
> > > > > > > > On Tue, 15 Jan 2019 at 09:56 Sabry Razick via RT <hpc-
> > > > > > > > drift at usit.uio.no>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >   Will the module "module load Python/3.5.2-foss-2016b"
> work
> > > > > > > > > for
> > > > > > > > > you all
> > > > > > > > > you want to recompile a new one ?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Sabry.
> > > > > > > > >
> > > > > > > > > On 2019-01-08 14:34:40, sabryr wrote:
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > On 2019-01-08 13:35:25, oe wrote:
> > > > > > > > > > > many thanks for the quick follow-up, sabry!
> > > > > > > > > >
> > > > > > > > > > You are welcome
> > > > > > > > > >
> > > > > > > > > > > would there be a relatively easy way of retroactively
> > > > > > > > > > > fixing
> > > > > > > > > > > the
> > > > > > > > > > > 3.5.0
> > > > > > > > > > >  module?
> > > > > > > > > >
> > > > > > > > > > No, due to the policy that installed modules will not
> > > > > > > > > > change. As
> > > > > > > > > > I
> > > > > > > > > > would have to recompile,can not guarantee the change will
> > > > > > > > > > be only
> > > > > > > > > > the
> > > > > > > > > > lzima,thus potentially effect others. But I can install
> > > > > > > > > > another
> > > > > > > > > > 3.5.
> > > > > > > > > > module for you, if miner version is not a problem.
> > > > > > > > > >
> > > > > > > > > > For example is following module usable (a secret module
> we
> > > > > > > > > > have)
> > > > > > > > > >
> > > > > > > > > > module load Python/3.5.2-foss-2016b
> > > > > > > > > >
> > > > > > > > > > If not, we can install similar semi-secret module.
> > > > > > > > > >
> > > > > > > > > > i am asking because in the NLPL context we have produced
> at
> > > > > > > > > > > least half a dozen python modules that are derived from
> > > > > > > > > > > the
> > > > > > > > > > > base
> > > > > > > > > > > ‘python3/3.5.0’ module.  here, derivative means
> creating
> > > > > > > > > > > a
> > > > > > > > > > > virtual
> > > > > > > > > > > environment, installing one or more add-on packages,
> and
> > > > > > > > > > > tweaking
> > > > > > > > > > > PYTHONPATH such that several such derivative modules
> can
> > > > > > > > > > > be
> > > > > > > > > > > combined.
> > > > > > > > > > > we have adopted this path for NLPL, for better
> > > > > > > > > > > modularity,
> > > > > > > > > > > interoperability, and uniformity between Abel and
> Taito.
> > > > > > > > > > > some
> > > > > > > > > > > additional background on this approach is on the NLPL
> > > > > > > > > > > wiki:
> > > > > > > > > > >
> > > > > > > > > > >
> > > http://wiki.nlpl.eu/index.php/Infrastructure/software/python
> > > > > > > > > >
> > > > > > > > > > As the workload will be less on our end, I will try to
> help
> > > > > > > > > > on
> > > > > > > > > > this.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > switching to a different base module (e.g. from 3.5.0
> to
> > > > > > > > > > > 3.7.0)
> > > > > > > > > > > would
> > > > > > > > > > > require rebuilding all of these derivative modules
> (hence
> > > > > > > > > > > is
> > > > > > > > > > > not
> > > > > > > > > > > something i would not do light-heartedly :-).  but see
> > > > > > > > > > > below
> > > > > > > > > > > ...
> > > > > > > > > > >
> > > > > > > > > > > —more generally, we have been meaning to start a fresh
> > > > > > > > > > > batch of
> > > > > > > > > > > derivative modules based on 3.6 or even 3.7.  to start
> > > > > > > > > > > down
> > > > > > > > > > > this
> > > > > > > > > > > route, it would be important to agree on which
> > > > > > > > > > > characteristics
> > > > > > > > > > > the
> > > > > > > > > > > base python module should have.  (unlike the older
> 3.5.0
> > > > > > > > > > > installation)
> > > > > > > > > > > i see that your new 3.7.0 is minimal, i.e. does not
> (yet)
> > > > > > > > > > > include any
> > > > > > > > > > >  add-on components.
> > > > > > > > > >
> > > > > > > > > > That is correct. The future plan is to provide basic
> python
> > > > > > > > > > and
> > > > > > > > > > another module called "python-modules". There is no
> > > > > > > > > > PYTHONPATH
> > > > > > > > > > set in
> > > > > > > > > > this module either and highly customizable.
> > > > > > > > > >
> > > > > > > > > > in principle, i think this is right, and all
> > > > > > > > > > > add-ons should arguably be provisioned as separate
> > > > > > > > > > > modules.
> > > > > > > > > > > what
> > > > > > > > > > > about NumPy with MKL support, however?  that was
> compiled
> > > > > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > 3.5.0 installation and is something i suspect many
> users
> > > > > > > > > > > might
> > > > > > > > > > > want.
> > > > > > > > > > > or maybe the default OpenBLAS back-end for NumPy is
> just
> > > > > > > > > > > as
> > > > > > > > > > > good (on
> > > > > > > > > > > average), and we can make do with standard,
> pre-compiled
> > > > > > > > > > > NumPy
> > > > > > > > > > > wheels?
> > > > > > > > > > >  if so, is there a particular reason you opted to build
> > > > > > > > > > > the new
> > > > > > > > > > > 3.7.0
> > > > > > > > > > > with the Intel tool chain?
> > > > > > > > > >
> > > > > > > > > > Latest MKL, older OpenBLAS, (latest blas and Atlas soon)
> > > > > > > > > > available
> > > > > > > > > > when you load the " module load python3/3.7.0", but you
> > > > > > > > > > must
> > > > > > > > > > compile
> > > > > > > > > > numpy (not from wheel) to use it.
> > > > > > > > > >
> > > > > > > > > > $ module load python3/3.7.0
> > > > > > > > > > $ echo $CPATH | grep mkl --color=always
> > > > > > > > > >
> > > > > > > > > > /cluster/software/VERSIONS/intel-
> > > > > > > > > >
> 2019.0/compilers_and_libraries_2019.0.117/linux/mkl/include/
> > > > > > > > > > /cluster/software/VERSIONS/intel-
> > > > > > > > > >
> > > 2019.0/compilers_and_libraries_2019.0.117/linux/mkl/include/fftw:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > in the NLPL context, we have been tossing around some
> of
> > > > > > > > > > > these
> > > > > > > > > > > high-level design questions for a little while now,
> and i
> > > > > > > > > > > am
> > > > > > > > > > > hoping
> > > > > > > > > > > to
> > > > > > > > > > > tap into your broader experience with maintaining
> modules
> > > > > > > > > > > on
> > > > > > > > > > > Abel
> > > > > > > > > > > here.  would it make sense to discuss some of the more
> > > > > > > > > > > general
> > > > > > > > > > > questions over a cup of coffee sometime this week?
> > > > > > > > > > > anyone else
> > > > > > > > > > > in
> > > > > > > > > > > your group with strong opinions about python modules?
> > > > > > > > > >
> > > > > > > > > > Yes we could discuss this, I do not think there are
> strong
> > > > > > > > > > opinions,
> > > > > > > > > > but I could ask permission from Jon if needed .  Only
> thing
> > > > > > > > > > is
> > > > > > > > > > once
> > > > > > > > > > installed modules are (almost) newer changed.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Sabry
> > > > > > > > > > >
> > > > > > > > > > > with thanks in advance, oe
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 8, 2019 at 11:31 AM Sabry Razick via RT
> > > > > > > > > > > <hpc-drift at usit.uio.no> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On 2019-01-07 23:20:24, oe wrote:
> > > > > > > > > > > > > colleagues,
> > > > > > > > > > > > >
> > > > > > > > > > > > > using the default ‘python3’ module on Abel, i am
> > > > > > > > > > > > > running
> > > > > > > > > > > > > into the
> > > > > > > > > > > > > following import error (required by a third-party
> > > > > > > > > > > > > component):
> > > > > > > > > > > > >
> > > > > > > > > > > > > [oe at login-0-0 ~]$ module purge; module load
> > > > > > > > > > > > > python3/3.5.0
> > > > > > > > > > > > > [oe at login-0-0 ~]$ type -all python3
> > > > > > > > > > > > > python3 is /cluster/software/VERSIONS/python3-
> > > > > > > > > > > > > 3.5.0/bin/python3
> > > > > > > > > > > > > [oe at login-0-0 ~]$ python3 -c "import lzma;"
> > > > > > > > > > > > > Traceback (most recent call last):
> > > > > > > > > > > > >   File "<string>", line 1, in <module>
> > > > > > > > > > > > >   File "/cluster/software/VERSIONS/python3-
> > > > > > > > > > > > > 3.5.0/lib/python3.5/lzma.py",
> > > > > > > > > > > > > line 26, in <module>
> > > > > > > > > > > > >     from _lzma import *
> > > > > > > > > > > > > ImportError: No module named '_lzma'
> > > > > > > > > > > > >
> > > > > > > > > > > > > for all i can tell, there are no RHEL lzma packaged
> > > > > > > > > > > > > installed,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > there is a separate module ‘xz/5.2.2’.  i wonder
> > > > > > > > > > > > > whether my
> > > > > > > > > > > > > import
> > > > > > > > > > > > > error could just mean that no lzma (aka xz) headers
> > > > > > > > > > > > > and
> > > > > > > > > > > > > libraries
> > > > > > > > > > > > > were
> > > > > > > > > > > > > visible when the python3 installation was compiled?
> > > > > > > > > > > > >
> > > > > > > > > > > > > with thanks in advance, oe
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >   Your observation is correct. I have fixed this in
> the
> > > > > > > > > > > > latest
> > > > > > > > > > > > version (did not fix the old).
> > > > > > > > > > > >
> > > > > > > > > > > > -bash-4.1$ module purge
> > > > > > > > > > > > -bash-4.1$ module load python3/3.7.0
> > > > > > > > > > > > -bash-4.1$ python3 -c "import lzma;"
> > > > > > > > > > > > -bash-4.1$ python3 -c "import
> > > > > > > > > > > > lzma;print(lzma.__file__)"
> > > > > > > > > > > > /cluster/software/VERSIONS/python3-
> > > > > > > > > > > > 3.7.0/lib/python3.7/lzma.py
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Sabry
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > >
> > >
> > >
>
>