[mrp-users] updates: companion ‘alignments’; graph validation; file format

[Stephan Oepen]

dear colleagues,

(0) two weeks remain until the start of the evaluation period for our
2019 CoNLL Shared Task on Cross-Framework Meaning Representation
Parsing (MRP 2019).  between monday, july 8, and monday, july 22,
registered teams will have access to the MRP evaluation data and must
make sure to submit their parser outputs by the end of the evaluation
period.  we will provide more technical detail on the format of the
evaluation data and requirements for system submissions in the next
few days.

(1) with the kind assistance of jayeol chun (of brandeis university),
we have been able to augment the MRP companion package with reference
(if not gold-standard) ‘alignments’ (i.e. anchorings, in MRP terms)
for the AMR training graphs; please see:

  http://svn.nlpl.eu/mrp/2019/public/companion.tgz

(2) the MRP software package (mtool) continues to evolve.  we
recommend that you follow development on its Microsoft GitHub
repository and regularly upgrade to the latest version.  the
cross-framework MRP scorer is in a relatively stable state now
(although we continue to search for efficiency improvements), so that
system development can be guided by the generalized MRP scores instead
of the individual per-framework metrics (where, of course, there
should be strong correlations).  emerging validation support for
parser outputs in MRP format is now available:

  https://github.com/cfmrp/mtool#validation

(3) we have posted to the task web site some additional details on how
different types of values are compared in MRP scoring:

  http://mrp.nlpl.eu/index.php?page=5#software

with the available information on that page, we would like it to be
the case that there is no uncertainty whatsoever about the exact
nature of the official MRP metric.  if you remain unsure about
scoring, please be in touch!

(4) for terminological clarity, we have made two minor revisions to
the MRP serialization format: (a) dropping hours and minutes from the
‘time’ top-level property on graphs and (b) renaming the ‘properties’
fields on edges to ‘attributes’.  this way, we hope to avoid confusion
between node properties and edge attributes, where the latter are in
fact only present in UCCA graphs.  the revised serialization format
can be recognized by a top-level ‘version’ value of 1.0 (compared to
0.9 earlier), but until the completion of the shared task, mtool will
transparently support both versions.  to exemplify the new
serialization, we have re-release the shared sample of WSJ graphs in
all frameworks:

  http://svn.nlpl.eu/mrp/2019/public/sample.tgz

(5) as a reminder, for example in case you joined the mailing list
only recently: there has been an update to the UCCA training graphs in
mid-may, improving consistency of segmentation and annotations and
adding some 1,500 more training graphs.  this UCCA re-release has been
made available for public download and is meant to supersede the
contents of the ‘training/ucca/’ sub-directory in the original release
of the MRP training data, as it is distributed via the LDC:

  http://svn.nlpl.eu/mrp/2019/public/ucca.tgz

as always, please do not hesitate to contact us about any questions or
suggestions regarding the task that you might have!

stephan oepen, omri abend, jan hajič, daniel hershcovich,
marco kuhlmann, tim o'gorman, and nianwen xue