[NLPL Task Force (A)] Storage alternatives

[Stephan Oepen]

why should a user have a private installation of basic software like python
or pytorch?

ideally, the pre-installed NLPL modules should provide all common software
(and data, where appropriate).  why are you not using those?

alternatively, there is the $USERWORK area.  if you fully automate
environment creation, e.g. from conda, then why not put your software stack
on that more transient storage area?

oe


On Wed, 18 Nov 2020 at 14:44 Vinit Ravishankar <vinitr at ifi.uio.no> wrote:

> My bad, I assumed it was some sort of binary because it was called
> python3.7
>
> – Vinit
>
> > On 18 Nov 2020, at 14:20, Andrey Kutuzov <andreku at ifi.uio.no> wrote:
> >
> > The Python binary is usually located in the 'bin/' directory, not 'lib/'.
> >
> > 'lib/' contains installed Python packages, and for this I indeed can
> > easily believe in 5 GiB and more. But not the Python executable/binary
> > itself: it rarely exceeds a few MiB in size, whatever are the Cython
> > optimizations or anything.
> >
> > But this is just to clear up the details. Of course the storage space
> > problem still persists, even if it is because of packages, not Python
> > binary.
> >
> > 18.11.2020 14:10, Vinit Ravishankar wrote:
> >> I am sure, I followed du -h and for eg.
> .miniconda3/envs/nlp/lib/python3.7 is 5 GiB. A quick Google search hints
> that this is because of Cython optimisation but I have no idea honestly.
> >>
> >> The size of the binary is different for different environments, so it
> seems to depend on the packages installed.
> >>
> >> – Vinit
> >>
> >>> On 18 Nov 2020, at 14:08, Andrey Kutuzov <andreku at ifi.uio.no> wrote:
> >>>
> >>>
> >>> 18.11.2020 13:47, Vinit Ravishankar пишет:
> >>>> The space issues aren’t just the huggingface models (though those
> >>>> are obviously an issue too) - a single virtual environment is multiple
> >>>> gigabytes worth of libraries, where just the python3.7 binary is often
> >>>> ~5 gigabytes.
> >>> Are you sure about that? What can I do to make the Python binary 5 GB
> worth?
> >>>
> >>> Just interested, may be I don't understand something.
> >>>
> >>> --
> >>> Andrey
> >>> Language Technology Group (LTG)
> >>> University of Oslo
> >>
> >
> >
> > --
> > Andrey
> > Language Technology Group (LTG)
> > University of Oslo
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nlpl.eu/archives/infrastructure/attachments/20201118/b0492a89/attachment.htm>