[epe-users] deadline extension for extrinsic parser evaluation runs

[Stephan Oepen]

dear colleagues,

we have just put on-line a preliminary summary table of results from
the 2018 Extrinsic Parser Evaluation (EPE) campaign:

https://goo.gl/3Fmjke

from the participants of the CoNLL Shared Task on UD Parsing,
seventeen teams have submitted valid parser outputs for the EPE data.
with just a couple of exceptions, we have been able to compute
end-to-end scores for the three EPE downstream applications: event
extraction, negation resolution, and opinion analysis.

for general background and comparison to the EPE 2017 results, please see

http://epe.nlpl.eu

we are still debugging a few end-to-end runs and have a little more
consistency checking that we want to apply.  thus, the above results
might still change, in principle.  we will issue a separate
announcement once we declare the EPE 2018 campaign complete and final
:-).  in case you notice anything surprising, please either use the
comment functionality in the on-line spreadsheet, or get in touch with
us at the ‘epe-organizers’ address!

the most interesting work is yet to be done: seeking to correlate EPE
results to the various intrinsic measures.  our spreadsheet already
contains some of the official measures of the UD Parsing task.  from a
first glance at the EPE results, we see some correspondences and some
seemingly surprising results.  besides dependency trees, all three EPE
applications make heavy use of PoS tags and lemmas—we will work with
the core task organizers (hi, dan) to try and isolate intrinsic
accuracy measures for these layers of analysis as well.

more to come!  oe (for the EPE co-organizers)



On Fri, Jul 6, 2018 at 11:54 AM, Stephan Oepen <oe at ifi.uio.no> wrote:
> dear participants in the 2018 UD parsing shared task:
>
> close to half the teams who participated in the core task last week
> have already submitted parsing runs on the optional extrinsic parser
> evaluation (EPE) data set.  however, many of you are running into
> time-out and memory issues, possibly related to the comparatively
> large EPE document sizes.
>
> seeing as we are in no big hurry here and want everyone to be able to
> participate, we have just extended the EPE submission deadline by one
> week, to friday, july 13, 2018.  for general background, please see:
>
>   http://epe.nlpl.eu
>
> the EPE data set is comprised of three large files, one each for
> event, negation, and opinion analysis.  these files count 343,353,
> 99,043, and 393,043 tokens, respectively (using the UDPipe baseline
> segmentation from).  it will be necessary that your parser can
> successfully work through all three files in a single run on TIRA.
>
> furthermore, it appears that some teams struggle with the unusual
> ‘tcode’ values in the EPE data set (which does not correspond to one
> of the english UD treebanks, meaning that teams are free to decide
> freely on which of the available treebanks to train), or maybe also
> stumble over the missing ‘goldfile’ entries in our ‘metadata.json’.
> in case any of these throw up your scripts from the core parsing task,
> please use the public EPE trial data (available on TIRA) to debug, so
> that you can easily inspect system behavior.
>
> —the main goal of the EPE add-on option to the 2018 UD parsing task
> is, of course, to investigate to what degree different intrinsic
> evaluation measures correlate with end-to-end performance in
> syntax-aware downstream applications.  this correlation exercise will
> be scientifically most interesting if all or at least the vast
> majority of task participants also submit parser outputs to EPE.  even
> though EPE 2018 sadly remains limited to english, in our view it is
> after all downstream utility that motivates continued research in
> syntactic parsing :-).
>
> please do not hesitate to contact us for further assistance!  for all
> EPE queries, please always email both organizer contact lists, i.e.
> ‘epe-organizers at nlpl.eu’ and ‘udst-orgs at googlegroups.com’.
>
> with thanks in advance, oe (for the EPE co-organizers)