[mrp-users] parser evaluation and other updates

[Stephan Oepen]

dear colleagues,

(0) we are writing with a few updates from the MRP 2019 parsing task.
in recent weeks, we receive about one question a day from
participants, so there appears to be some development activity :-).
we will regularly try to share relevant information and clarifications
with everyone, either by emailing this list, updating the task web
pages, or posting to the mtool issue tracker on Microsoft GitHub.  we
would recommend you watch the mtool repository:

  https://github.com/cfmrp/mtool

(1) we consider the ‘white-list’ for valid third-party data and models
to use in system development closed as of earlier this month.  the
last resource we added was the UIUC named entity tagger.  please make
sure to only use additional resources (beyond the MRP training and
‘companion’ packages) that are included on the list:

  http://svn.nlpl.eu/mrp/2019/public/resources.txt

(2) several of you have inquired about the specifics of the ‘official’
cross-framework evaluation metric.  we have just posted a more
technical description to the task web pages and provided an initial
implementation.  please note, however, that further testing and
refinement of the task scorer will continue for at least another week.

  http://mrp.nlpl.eu/index.php?page=5

(3) for the unanchored AMR graphs, some parsers at least will want to
make explicit the correspondence from graph nodes to sub-strings of
the parser input (often called ‘alignments’ in AMR parsing).  various
third-party tools exist for this purpose (e.g. the JAMR or ISI
aligners), and their use is legitimate because they do not include
substantive data or models.  for ease of reference, we will make
available alignments against our mropho-syntactic ‘companion’ trees
within the next week (the same alignments will then be provided with
the evaluation data).

(4) last we heard from the LDC, some 40 teams had obtained the MRP
training data.  this mailing list currently has close to 130
subscribers, so we are wondering about how many of you are still ‘on
the fence’ regarding active involvement in the MRP 2019 task?  a
little more than five weeks remain until the closing date of the
evaluation period.  to lower the threshold to participation, the task
does allow partial submissions, i.e. parser outputs for only a sub-set
of the five frameworks.  thus, if you have available an AMR, EDS, SDP
or UCCA parser that you would like to see evaluated within the MRP
context, we believe that even single-framework submissions can be
informative, e.g. to determine how MRP 2019 results compare to
previous work.

(5) once more, please do not hesitate to contact us (at the
‘mrp-organizers’ email address, or though the Microsoft GitHub issue
tracker for mtool) about any questions or suggestions you might have!

best wishes,

stephan oepen, omri abend, jan hajič, daniel hershcovich,
marco kuhlmann, tim o'gorman, and nianwen xue