[mrp-users] updates from the CoNLL 2019 shared task

[Stephan Oepen]

dear colleagues,

warmest thanks, again, for your interest in our shared task on Meaning
Representation Parsing (MRP 2019)!  as of today, some 110 individuals
have subscribed to this mailing list, and a little more than 30 teams
have obtained the data from the LDC already.

we have posted several updates to the task web site today, notably (a)
an update to the UCCA training data, with additional graphs and
improved consistency; and (b) premium-quality morpho-syntactic
analyses for the sentences underlying the MRP training data.  we are
grateful to sebastian schuster and milan straka for their assistance
in preparing these parses.

please see the task web site for download links and instructions for
how to obtain the full training data (at no charge) through the LDC.
for the time being at least, we do not foresee further revisions of
the available data ... until the release of the evaluation data on
july 8, of course.

as we see it, the task is quickly moving towards its ‘hot’ phase: nine
weeks remain until the closing date of the evaluation phase.  if you
are still on the fence about participation, now may be a good time to
obtain the training data and start developing your cross-framework
parsing system :-).

some of you have asked about using additional resources, beyond the
data provided by the task organizers.  we have just tried to clarify
the language regarding the rules of participation on the web site, but
truth be told, we are still in the process of finding out what makes
most sense, in terms of balancing reproducibility and some notion of
fairness vs. ease of participation and best possible results.  in a
nutshell, common resources like large text collections, pre-trained
embeddings, taggers, or syntactic parsers will be legitimate to use.

we are having a conference call among ourselves tomorrow to clarify
our thinking regarding resources.  in parallel, we would like to hear
from prospective participants about what additional, third-party data
or tools, if any, you would like to put to use in your MRP parsers.
please email us about this at ‘mrp-organizers at nlpl.eu’ by the
beginning of june!

our next scheduled deliverable for task participants will be the MRP
support software in early june, which will enable format conversion
and validation and, most importantly, scoring.  as announced earlier,
the tool will provide reference implementations (compatible with the
uniform MRP graph model) of common framework-specific metrics, as well
as a generalization of extant evaluation metrics that is
parameterizable and applicable across all five frameworks.

again, we hope to see as many of you as possible submitting your
parsing results for evaluation in late july!  please do not hesitate
to get in touch with us (at the above email address), for example
about open questions regarding the task or suggestions for
improvement!

best wishes,

stephan oepen, omri abend, jan hajič, daniel hershcovich,
marco kuhlmann, tim o'gorman, and nianwen xue