<div dir="ltr">Dear EPE co-organizers,<div> </div><div>Could you please provide the .bib file (or title) of the EPE 2018 overview paper?</div><div> </div><div>Thank you very much.</div><div> </div><div>Best regards,</div><div>Dat Quoc Nguyen.</div> <div class="gmail_quote"><div dir="ltr">On Mon, Jul 30, 2018 at 4:10 AM Stephan Oepen <<a href="mailto:oe@ifi.uio.no">oe@ifi.uio.no</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">dear colleagues, we have just put on-line a preliminary summary table of results from the 2018 Extrinsic Parser Evaluation (EPE) campaign: <a href="https://goo.gl/3Fmjke" rel="noreferrer" target="_blank">https://goo.gl/3Fmjke</a> from the participants of the CoNLL Shared Task on UD Parsing, seventeen teams have submitted valid parser outputs for the EPE data. with just a couple of exceptions, we have been able to compute end-to-end scores for the three EPE downstream applications: event extraction, negation resolution, and opinion analysis. for general background and comparison to the EPE 2017 results, please see <a href="http://epe.nlpl.eu" rel="noreferrer" target="_blank">http://epe.nlpl.eu</a> we are still debugging a few end-to-end runs and have a little more consistency checking that we want to apply. thus, the above results might still change, in principle. we will issue a separate announcement once we declare the EPE 2018 campaign complete and final :-). in case you notice anything surprising, please either use the comment functionality in the on-line spreadsheet, or get in touch with us at the ‘epe-organizers’ address! the most interesting work is yet to be done: seeking to correlate EPE results to the various intrinsic measures. our spreadsheet already contains some of the official measures of the UD Parsing task. from a first glance at the EPE results, we see some correspondences and some seemingly surprising results. besides dependency trees, all three EPE applications make heavy use of PoS tags and lemmas—we will work with the core task organizers (hi, dan) to try and isolate intrinsic accuracy measures for these layers of analysis as well. more to come! oe (for the EPE co-organizers) On Fri, Jul 6, 2018 at 11:54 AM, Stephan Oepen <<a href="mailto:oe@ifi.uio.no" target="_blank">oe@ifi.uio.no</a>> wrote: > dear participants in the 2018 UD parsing shared task: > > close to half the teams who participated in the core task last week > have already submitted parsing runs on the optional extrinsic parser > evaluation (EPE) data set. however, many of you are running into > time-out and memory issues, possibly related to the comparatively > large EPE document sizes. > > seeing as we are in no big hurry here and want everyone to be able to > participate, we have just extended the EPE submission deadline by one > week, to friday, july 13, 2018. for general background, please see: > > <a href="http://epe.nlpl.eu" rel="noreferrer" target="_blank">http://epe.nlpl.eu</a> > > the EPE data set is comprised of three large files, one each for > event, negation, and opinion analysis. these files count 343,353, > 99,043, and 393,043 tokens, respectively (using the UDPipe baseline > segmentation from). it will be necessary that your parser can > successfully work through all three files in a single run on TIRA. > > furthermore, it appears that some teams struggle with the unusual > ‘tcode’ values in the EPE data set (which does not correspond to one > of the english UD treebanks, meaning that teams are free to decide > freely on which of the available treebanks to train), or maybe also > stumble over the missing ‘goldfile’ entries in our ‘metadata.json’. > in case any of these throw up your scripts from the core parsing task, > please use the public EPE trial data (available on TIRA) to debug, so > that you can easily inspect system behavior. > > —the main goal of the EPE add-on option to the 2018 UD parsing task > is, of course, to investigate to what degree different intrinsic > evaluation measures correlate with end-to-end performance in > syntax-aware downstream applications. this correlation exercise will > be scientifically most interesting if all or at least the vast > majority of task participants also submit parser outputs to EPE. even > though EPE 2018 sadly remains limited to english, in our view it is > after all downstream utility that motivates continued research in > syntactic parsing :-). > > please do not hesitate to contact us for further assistance! for all > EPE queries, please always email both organizer contact lists, i.e. > ‘<a href="mailto:epe-organizers@nlpl.eu" target="_blank">epe-organizers@nlpl.eu</a>’ and ‘<a href="mailto:udst-orgs@googlegroups.com" target="_blank">udst-orgs@googlegroups.com</a>’. > > with thanks in advance, oe (for the EPE co-organizers) -- Contact <a href="mailto:zeman@ufal.mff.cuni.cz" target="_blank">zeman@ufal.mff.cuni.cz</a> in case of problems with this group --- You received this message because you are subscribed to the Google Groups "UD CoNLL Shared Task" group. To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:ud-conll-shared-task%2Bunsubscribe@googlegroups.com" target="_blank">ud-conll-shared-task+unsubscribe@googlegroups.com</a>. To post to this group, send email to <a href="mailto:ud-conll-shared-task@googlegroups.com" target="_blank">ud-conll-shared-task@googlegroups.com</a>. For more options, visit <a href="https://groups.google.com/d/optout" rel="noreferrer" target="_blank">https://groups.google.com/d/optout</a>. </blockquote></div></div>