<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: #000000"><div>Hi,</div><div><br data-mce-bogus="1"></div><div>I hope I am not spamming "infrastructure" here, but this exchange is precisely the reason why I said in Oslo I benefit from NLPL. It is very difficult to get to conclusions below by just thinking about them. I made some measurements in terms of compression vs. uncompressed and found uncompressed significantly faster. But of course, it is all relative, as you point out. As discussed below compression solves some integrity issues as well. So I am all for it. And thanks for the trust, we do try not to lose bits, but in this case we first need to know we have the right kind of stuff :)</div><div><br data-mce-bogus="1"></div><div>Martin</div><div><br></div><div data-marker="__SIG_PRE__">-- <br>Martin Matthiesen<br>CSC - Tieteen tietotekniikan keskus<br>CSC - IT Center for Science<br>PL 405, 02101 Espoo, Finland<br>+358 9 457 2376, martin.matthiesen@csc.fi<br>Public key : https://pgp.mit.edu/pks/lookup?op=get&search=0x74B12876FD890704<br>Fingerprint: AA25 6F56 5C9A 8B42 009F BA70 74B1 2876 FD89 0704</div><br><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Filip Ginter" <figint@utu.fi><br><b>To: </b>"Martin Matthiesen" <martin.matthiesen@csc.fi><br><b>Cc: </b>"infrastructure" <infrastructure@nlpl.eu><br><b>Sent: </b>Friday, 1 December, 2017 11:17:34<br><b>Subject: </b>Re: [NLPL Task Force (A)] CoNLL-2017 raw data on taito<br></blockquote></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div><div><div>Good points, Martin. Making sure we don't lose them bits is what I trust CSC with. I guess I can be non-precision because I can rely you precision people running the systems. :) <br></div><br><div>The original page with the downloads has MD5 hashes <a href="https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-1989" target="_blank">https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-1989</a> so I suppose we can check those. I'll take care of it.<br><br></div>I am used to compress. My rationale here is that the processing applied to the data is typically heavy, the relative proportion of time spent decompressing is tiny. For example right now, when you look at "squeue -u ginter", you see that very much is happening with these .xz files as I am parsing them. The decompression is maybe 1% of the work, if even that. And I think decompressed these are easily 1.5TB, so this saves 1TB of space on taito. So I figured that is the way to go. :)<br><br></div>Cheers<br><br>F<br></div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 1, 2017 at 11:08 AM, Martin Matthiesen <span dir="ltr"><<a href="mailto:martin.matthiesen@csc.fi" target="_blank">martin.matthiesen@csc.fi</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div> <div style="font-family:arial,helvetica,sans-serif;font-size:10pt;color:#000000"> <div>Hi Filip!</div> <div><br> </div> <hr id="m_-566189095556308183zwchr"> <div> <blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"> <b>From: </b>"Filip Ginter" <<a href="mailto:figint@utu.fi" target="_blank">figint@utu.fi</a>><br> <b>To: </b>"Martin Matthiesen" <<a href="mailto:martin.matthiesen@csc.fi" target="_blank">martin.matthiesen@csc.fi</a>><br> <b>Cc: </b>"infrastructure" <<a href="mailto:infrastructure@nlpl.eu" target="_blank">infrastructure@nlpl.eu</a>><br> <b>Sent: </b>Thursday, 30 November, 2017 21:38:11<br> <b>Subject: </b>Re: [NLPL Task Force (A)] CoNLL-2017 raw data on taito<br> </blockquote> </div> <div><span class=""> <blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"> <div dir="ltr"> <div> <div> <div> <div>Hi Martin<br> <br> </div> Not sure about the hashes. If you think it's useful, then go for it. :)</div> </div> </div> </div> </blockquote> </span><div>That is the thing, I am not sure myself, I don't want to over-engineer things.</div><span class=""> <blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"> <div dir="ltr"> <div> <div> <div>Myself, I'm thinking this is not Space Shuttle plans nor bank records and if something goes haywire then xz will crash and we will redownload. :D I guess I'm not much of a precision guy with these things. :)</div> </div> </div> </div> </blockquote> </span><div>Well, we just got bitten with data getting lost in a conversion process and us not noticing, I guess it is part of my job to run after those pesky bits. I would like to get to some kind of process where you indeed would not need to be a precision guy concerning data storage, but you can just rely it works ok. As soon as we want to have one data set in two places (be it for performance reasons in both places) I think we need to think about this. The other thing is that you personally can always re-download, since you have full control, but someone using your dataset cannot and unless we have a super fast process in place to reliably fix such missing/corrupted data errors this might slow down someone else's research.</div><span class=""> <blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"> <div dir="ltr"> <div> <div> <div><br> </div> The tars are built the way they are in Prague. They actually contain a bunch of .xz files, so compression of the whole tar wouldn't anymore make any difference. I will unpack the tars, but of course not decompress the xz files.</div> </div> </div> </blockquote> </span><div>Ok, I thought there was a reason for just tar. And indeed compressed files have some inbuild integrity checking. Crashing xz/zip files are indeed less of a problem than silently missing ones. I am curious to hear your experiences with compression in the first place, I some time ago decided against it for data on Taito, since you pay in processing time. For compressed data I think making shure that the xz files are all there and work would be enough. My concern is accidental data loss, not a deliberate one.</div> <div><br> </div> <div>Maybe this an item for our infrastructure meeting. This is indeed not rocket science, but even there rather trivial problems can have severe consequences:</div> <div><a href="https://edition.cnn.com/TECH/space/9909/30/mars.metric.02/" target="_blank">https://edition.cnn.com/TECH/space/9909/30/mars.metric.02/</a><br> </div> <div><br> </div> <div>Regards,</div> <div>Martin</div><div><div class="h5"> <blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"> <div dir="ltr"> <div> <div><br> <br> </div> Cheers<br> <br> </div> F<br> <br> </div> <div class="gmail_extra"><br> <div class="gmail_quote">On Thu, Nov 30, 2017 at 6:24 PM, Martin Matthiesen <span dir="ltr"> <<a href="mailto:martin.matthiesen@csc.fi" target="_blank">martin.matthiesen@csc.fi</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div> <div style="font-family:arial,helvetica,sans-serif;font-size:10pt;color:#000000"> <div>Hi Filip,</div> <div><br> </div> <div>This sounds good to me, This raises some interesting infra questions (to me at least):</div> <div><br> </div> <div>Could we compute a grand total hash that ensures that the whole thing is correctly in place (eg [1])?</div> <div>Would we want that on a per-tar file basis (to be able to use only a partial corpus)?</div> <div>And here I do not mean to hash the tar-file itself, but to make sure that the extracted tar is in place correctly.</div> <div><br> </div> <div>I am curious: Why did you not compress the tar files? To slow?</div> <div><br> </div> <div>Cheers,</div> <div>Martin</div> <div><br> </div> <div><br> </div> <div>[1] <a href="https://stackoverflow.com/questions/4830089/how-to-checksum-an-entire-folder-structure" target="_blank"> https://stackoverflow.com/questions/4830089/how-to-checksum-an-entire-folder-structure</a><br> </div> <div><br> </div> <div>-- <br> Martin Matthiesen<br> CSC - Tieteen tietotekniikan keskus<br> CSC - IT Center for Science<br> PL 405, 02101 Espoo, Finland<br> <a href="tel:+358%209%204572376" target="_blank">+358 9 457 2376</a>, <a href="mailto:martin.matthiesen@csc.fi" target="_blank"> martin.matthiesen@csc.fi</a><br> Public key : <a href="https://pgp.mit.edu/pks/lookup?op=get&search=0x74B12876FD890704" target="_blank"> https://pgp.mit.edu/pks/lookup?op=get&search=0x74B12876FD890704</a><br> Fingerprint: AA25 6F56 5C9A 8B42 009F BA70 74B1 2876 FD89 0704</div> <br> <hr id="m_-566189095556308183m_-4254096139939864232zwchr"> <div> <blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"> <b>From: </b>"Filip Ginter" <<a href="mailto:ginter@cs.utu.fi" target="_blank">ginter@cs.utu.fi</a>><br> <b>To: </b>"infrastructure" <<a href="mailto:infrastructure@nlpl.eu" target="_blank">infrastructure@nlpl.eu</a>><br> <b>Sent: </b>Thursday, 30 November, 2017 10:15:06<br> <b>Subject: </b>[NLPL Task Force (A)] CoNLL-2017 raw data on taito<br> </blockquote> </div> <span> <div> <blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"> <div dir="ltr"> <div> <div>Hi guys<br> <br> </div> Is it okay for me to stick this data <a href="https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-1989" target="_blank"> https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-1989</a> to the nlpl directory on taito? We actually have this data in one of our researcher's work directory on taito, so the total space usage on taito stays. 522GB. Thiis is a useful dataset for parser training etc. <br> <br> </div> - Filip</div> <br> </blockquote> </div> </span></div> </div> </blockquote> </div> </div> <br> </blockquote> </div></div></div> </div> </div> </blockquote></div></div><br></blockquote></div></div></body></html>