[epe-users] preliminary EPE 2017 results

Stephan Oepen oe at ifi.uio.no
Thu Jun 29 16:25:06 CEST 2017


dear all,

i know at least some of you are impatiently waiting to hear from us
about end-to-end results.  even though we do not have negation
analysis scores yet and are still double-checking a few individual
results, please see the following link for a preliminary summary:

  https://goo.gl/ZTZxXW

we can only apologize for the delay in the negation evaluation, but it
may be a couple more weeks before emanuele returns to the office; i
know it for a fact that the father wanted the baby to respect our EPE
schedule, but little erik appears to have his own independent mind.

there are, of course, many parameters involved in the EPE 2017 set-up,
including for example which node properties to access for part of
speech and lemma values.  the defaults are ‘pos’ and ‘lemma’,
respectively, but several submissions provide multiple parts of speech
(UPOS and XPOS, for example, in UD graphs) and some do not (seem to)
contain lemmas.

in case you have specific suggestions for how to map properties on
your graphs to the ‘pos’ and ‘lemma’ fields expected by all three
downstream systems, please feel free to email us at
‘epe-organizers at nlpl.eu’.  several of the core developers are about to
start their summer vacations, so we cannot promise all too many
additional evaluation runs before august, though.

we have yet to publish more specific instructions for system
descriptions, but in a nutshell please note that each team is invited
to contribute a paper of up to six pages (excluding references) to the
EPE proceedings, which will be published by the ACL.  submissions will
be due by monday, august 7, and need to follow the ACL 2017 formatting
instructions linked from the following page:

  http://acl2017.org/calls/papers/

unlike for the ACL conference, however, system descriptions need not
be anonymous.

as we now start to try and make sense of the end-to-end results, we
expect it will be important to characterize each submission (i.e. each
‘run’, in our terminology) as precisely as possible, e.g. with regard
to the target dependency representations, training data used, general
parsing approach, pre-processing (starting from segmentation), and
such.

we will seek to assemble some of the high-level system characteristics
into the above summary spreadsheet, starting from the README files
that you have provided with your submissions.  we also plan on making
all submissions available publicly and hope you will agree to this
policy.  if not, please email ‘epe-organizers at nlpl.eu’ before
wednesday next week (july 5).

so much for now :-); more soon!  oe



More information about the epe-users mailing list