[mrp-users] preliminary MRP 2019 results

[Stephan Oepen]

dear colleagues,

while we hope many of you are enjoying ACL in florence this week, we
have now completed scoring of all submissions (that were received
within the deadline) and applied a first round of plausibility
checking.  we would now like to expose scoring results to you and
invite you to participate in quality control.  please consider these
scores as preliminary (and, thus, kind of confidential), as we might
still want to apply corrections in case we have messed something up!
we will make a final, public announcement on monday, august 12.

in particular, please check whether the CodaLab submission identifier,
time stamp, and graph counts per framework correspond to the most
recent submission you had uploaded; this is on the first sheet of the
on-line summary table.  also, please see whether results on the
evaluation data correspond with your expectations from system
development and tuning.  in case you notice anything surprising,
please do get in touch with us as soon as possible!

the sheet labeled ‘MRP’ provides a summary of the official task
results: macro-averages of the MRP metric over the five frameworks,
computed for each of the distinct tuple types, and for all of them
together.  furthermore, there are two rows of results per submission,
one for the complete evaluation data (white background), and another
one for the 100-item LPPS sub-set that is annotated in all frameworks
(yellow background).  column AD shows overall F1 scores, and column AE
the corresponding ranking.

the macro-averages in this main MRP result table are computed from the
next five sheets in the on-line table, labeled ‘MRP: DM’ through ‘MRP:
AMR’, which show scores of the MRP metric on the sub-sets of graphs
for each of the frameworks.  finally, there are another five sheets
with framework-specific metrics on these same sub-sets, i.e. using the
pre-existing SDP, EDM, UCCA, and SMATCH metrics.  the per-framework
rankings are summarized in the main ‘MRP’ sheet in columns AF through
AJ, for the MRP metric, and in columns AK through AO, for the
framework-specific metrics.

finally, to gain access to these preliminary results, we kindly ask
that you work through a very short questionnaire.  we are admittedly
curious about our task participants (including those who did not make
a submission in the end) and about your feedback regarding the task
design and execution.  the access link for the on-line results
spreadsheet will be available upon completion of our participant
survey:

https://nettskjema.no/a/121656

with thanks in advance, and more soon!

stephan oepen, omri abend, jan hajič, daniel hershcovich,
marco kuhlmann, tim o'gorman, and nianwen xue