<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class="">ok - done</div>
<div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="" style="orphans: 2; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
Jörg</div>
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<span class="" style="orphans: 2; widows: 2;"><br class="">
</span></div>
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<span class="" style="orphans: 2; widows: 2;">********************************************************************************************</span><br class="" style="orphans: 2; widows: 2;">
<span class="" style="orphans: 2; widows: 2;">Jörg Tiedemann</span></div>
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<span class="" style="orphans: 2; widows: 2;">Language Technology<span class="Apple-tab-span" style="white-space: pre;">
</span></span><a href="https://blogs.helsinki.fi/language-technology/" class="">https://blogs.helsinki.fi/language-technology/</a></div>
<div class=""><span style="orphans: 2; widows: 2;" class="">University of Helsinki</span></div>
</div>
</div>
</div>
</div>
<br class="">
<div style="">
<blockquote type="cite" class="">
<div class="">On 13 Feb 2019, at 19:29, Stephan Oepen <<a href="mailto:oe@ifi.uio.no" class="">oe@ifi.uio.no</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<div dir="auto" class="">okay! could you run something like ‘chmod -R g+w /projects/nlpl/data/OPUS’ on Abel, so i get to selectively delete those files?</div>
</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">oe</div>
<div dir="auto" class=""><br class="">
</div>
<div class=""><br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 13 Feb 2019 at 18:04 Tiedemann, Jörg <<a href="mailto:jorg.tiedemann@helsinki.fi" class="">jorg.tiedemann@helsinki.fi</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Hi,
<div class=""><br class="">
</div>
<div class="">I would leave xml and raw as the primary data files and you can delete “moses” and “mono”, which are both derived plain text data versions. That should save enough space I guess as especially the OpenSubtitles corpus takes most of the space and
the moses directory already occupies 264G. mono is another 31G. Would that be sufficient.</div>
<div class=""><br class="">
</div>
<div class="">The reason for keeping raw is because this is the non-tokenized XML, which is probably more important than xml, which is the tokenized version. In many cases, people would like to apply their own tokenization/preprocessing pipeline to be consistent
with any downstream task later on.</div>
<div class=""><br class="">
</div>
<div class="">The problem is that xml contains the sentence alignment files that you need to keep. It’s a bit mixed and therefore not easy to separate in cronjobs without lots of specific rules for excluding files etc. Can some of the syads change the owenership
of the files on abel so that you can run your cronjobs?</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="word-wrap:break-word" class=""><span class="">All the best,</span></div>
<div style="word-wrap:break-word" class="">Jörg</div>
<div style="word-wrap:break-word" class=""><span class=""><br class="">
</span></div>
<div style="word-wrap:break-word" class=""><span class="">********************************************************************************************</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="word-wrap:break-word" class="">
<div class="">
<div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="word-wrap:break-word" class=""><br class="">
<span class="">Jörg Tiedemann</span></div>
<div style="word-wrap:break-word" class=""><span class="">Language Technology<span class="m_9165083973474320842Apple-tab-span" style="white-space:pre-wrap">
</span></span><a href="https://blogs.helsinki.fi/language-technology/" target="_blank" class="">https://blogs.helsinki.fi/language-technology/</a></div>
<div class=""><span class="">University of Helsinki</span></div>
</div>
</div>
</div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On 11 Feb 2019, at 20:08, Stephan Oepen <<a href="mailto:oe@ifi.uio.no" target="_blank" class="">oe@ifi.uio.no</a>> wrote:</div>
<br class="m_9165083973474320842Apple-interchange-newline">
<div class="">hi joerg,<br class="">
<br class="">
our NLPL partition on Abel has hit the disk quota limit (two<br class="">
terabytes), which means we cannot install software updates. i am<br class="">
afraid i would like to propose that we further restrict the OPUS<br class="">
mirror on Abel, as it accounts for by far the biggest ‘chunk’ of NLPL<br class="">
data (715 gigabytes currently on Abel). would it make sense to just<br class="">
keep the XML variants of the data (i am guessing the fairly bulky<br class="">
‘moses’ and ‘raw’ variants are derived)?<br class="">
<br class="">
more generally, i was planning to suggest that we move to automated<br class="">
mirroring of the most important parts of OPUS from Taito to Abel, as<br class="">
we do for most of the other data sub-directories now. could you (a)<br class="">
suggest an rsync(1) command to selective copy from Taito to Abel and<br class="">
(b) temporarilty ‘\rm -rf /projects/nlpl/data/OPUS’ on Abel?<br class="">
<br class="">
i could then include the rsync(1) in my nightly cron(5) job on Taito,<br class="">
such that for the selected parts at least the two copies would remain<br class="">
synchronized (because the cron(5) jobs runs in my account, i will have<br class="">
to be the owner of the rsync(1) target directory on Abel).<br class="">
<br class="">
best wishes, oe<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>