[epe-users] wrapping up the EPE 2017 evaluation phase

Stephan Oepen oe at ifi.uio.no
Thu Jun 15 10:33:24 CEST 2017


dear all,

today is the closing date we had set for the evaluation phase in our
EPE 2017 shared task.  a few teams have already submitted their runs;
we are very grateful for those!

please note the instructions on the task web site on exactly how to
prepare your parser outputs and the directory structure and file
naming conventions that we expect:

  http://epe.nlpl.eu/index.php?page=5#evaluation

in particular, for each run, we need parser outputs for the complete
document collection (training, development, and evaluation), because
the downstream systems are automatically re-trained for each
individual submission.

to make automation feasible, it is important that you preserve
directory and file names from the ‘raw’ data package exactly, but
replace all ‘.txt’ (or ‘.tt’ suffixes) on input files with ‘.epe’ (or
‘.sdp’, ‘.conll’, or something else—in case you have agreed with us
beforehand that we convert from one of the supported formats on our
end).

finally, a few of you have been asking whether we can allow a little
more time.  to get end-to-end feedback to you in a timely fashion, we
depend on a large group of people, notably the developers of the three
downstream systems.  one of us just had a baby (last night :-), and
others are preparing for summer vacation starting in the next couple
of weeks.  so there is not a lot of slack in our schedule.

the final deadline extension we can offer is noon in oslo (12:00 CEST)
this coming sunday, june 18.  i will then register and convert
submissions for end-to-end evaluation later on sunday, and starting on
monday the system developers will try to run everything through.  on
this schedule, we still plan on sending out end-to-end results on
monday, june 26.

please keep those dependency graphs coming :-)!  oe



More information about the epe-users mailing list