[epe-users] [ud-conll-shared-task] Re: deadline extension for extrinsic parser evaluation runs

[Dat Quoc Nguyen]

Dear EPE co-organizers,

Could you please provide the .bib file (or title) of the EPE 2018 overview
paper?

Thank you very much.

Best regards,
Dat Quoc Nguyen.

On Mon, Jul 30, 2018 at 4:10 AM Stephan Oepen <oe at ifi.uio.no> wrote:

> dear colleagues,
>
> we have just put on-line a preliminary summary table of results from
> the 2018 Extrinsic Parser Evaluation (EPE) campaign:
>
> https://goo.gl/3Fmjke
>
> from the participants of the CoNLL Shared Task on UD Parsing,
> seventeen teams have submitted valid parser outputs for the EPE data.
> with just a couple of exceptions, we have been able to compute
> end-to-end scores for the three EPE downstream applications: event
> extraction, negation resolution, and opinion analysis.
>
> for general background and comparison to the EPE 2017 results, please see
>
> http://epe.nlpl.eu
>
> we are still debugging a few end-to-end runs and have a little more
> consistency checking that we want to apply.  thus, the above results
> might still change, in principle.  we will issue a separate
> announcement once we declare the EPE 2018 campaign complete and final
> :-).  in case you notice anything surprising, please either use the
> comment functionality in the on-line spreadsheet, or get in touch with
> us at the ‘epe-organizers’ address!
>
> the most interesting work is yet to be done: seeking to correlate EPE
> results to the various intrinsic measures.  our spreadsheet already
> contains some of the official measures of the UD Parsing task.  from a
> first glance at the EPE results, we see some correspondences and some
> seemingly surprising results.  besides dependency trees, all three EPE
> applications make heavy use of PoS tags and lemmas—we will work with
> the core task organizers (hi, dan) to try and isolate intrinsic
> accuracy measures for these layers of analysis as well.
>
> more to come!  oe (for the EPE co-organizers)
>
>
>
> On Fri, Jul 6, 2018 at 11:54 AM, Stephan Oepen <oe at ifi.uio.no> wrote:
> > dear participants in the 2018 UD parsing shared task:
> >
> > close to half the teams who participated in the core task last week
> > have already submitted parsing runs on the optional extrinsic parser
> > evaluation (EPE) data set.  however, many of you are running into
> > time-out and memory issues, possibly related to the comparatively
> > large EPE document sizes.
> >
> > seeing as we are in no big hurry here and want everyone to be able to
> > participate, we have just extended the EPE submission deadline by one
> > week, to friday, july 13, 2018.  for general background, please see:
> >
> >   http://epe.nlpl.eu
> >
> > the EPE data set is comprised of three large files, one each for
> > event, negation, and opinion analysis.  these files count 343,353,
> > 99,043, and 393,043 tokens, respectively (using the UDPipe baseline
> > segmentation from).  it will be necessary that your parser can
> > successfully work through all three files in a single run on TIRA.
> >
> > furthermore, it appears that some teams struggle with the unusual
> > ‘tcode’ values in the EPE data set (which does not correspond to one
> > of the english UD treebanks, meaning that teams are free to decide
> > freely on which of the available treebanks to train), or maybe also
> > stumble over the missing ‘goldfile’ entries in our ‘metadata.json’.
> > in case any of these throw up your scripts from the core parsing task,
> > please use the public EPE trial data (available on TIRA) to debug, so
> > that you can easily inspect system behavior.
> >
> > —the main goal of the EPE add-on option to the 2018 UD parsing task
> > is, of course, to investigate to what degree different intrinsic
> > evaluation measures correlate with end-to-end performance in
> > syntax-aware downstream applications.  this correlation exercise will
> > be scientifically most interesting if all or at least the vast
> > majority of task participants also submit parser outputs to EPE.  even
> > though EPE 2018 sadly remains limited to english, in our view it is
> > after all downstream utility that motivates continued research in
> > syntactic parsing :-).
> >
> > please do not hesitate to contact us for further assistance!  for all
> > EPE queries, please always email both organizer contact lists, i.e.
> > ‘epe-organizers at nlpl.eu’ and ‘udst-orgs at googlegroups.com’.
> >
> > with thanks in advance, oe (for the EPE co-organizers)
>
> --
> Contact zeman at ufal.mff.cuni.cz in case of problems with this group
> ---
> You received this message because you are subscribed to the Google Groups
> "UD CoNLL Shared Task" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ud-conll-shared-task+unsubscribe at googlegroups.com.
> To post to this group, send email to ud-conll-shared-task at googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nlpl.eu/archives/epe-users/attachments/20180730/ce37d666/attachment.html>