[mrp-users] improvements in MRP scorer and validator

[Stephan Oepen]

hi again, parser developers,

i apologize for sending yet another message today, but when we have to
make changes to the mtool software during the evaluation phase, it
only seems right to alert everyone to such updates.

we have fixed two bugs in the scorer today, which in principle can
affect your scoring results (if in fairly minor ways):

+ an empty ‘anchors’ list was not treated as equivalent to no value at
all (or a value of ‘null’); the official graphs for the task will
never contain "anchors": [], but if your parser outputs do, this fix
can lead to slightly improved scores.

+ multi-part anchors were considered separately in scoring: something
like "anchors": [{"from": 0, "to": 5}, {"from": 5, "to": 8}] could
have been counted as either wholly incorrect, partly correct (one
point), or doubly correct (two points) ... depending on the
corresponding value in the gold standard.  in the revised code, each
(non-empty) ‘anchors’ value is considered as a whole, i.e. counts for
exactly one point.

also today, we have made the validator more robust to ill-formed MRP
serializations and added a few more diagnostic tests.  as always, we
recommend you run ‘--validate all’ regularly over your parser outputs
:-).

these improvements are available in both the ‘master’ and ‘dev’
branches of mtool.

best wishes, oe (for the MRP co-organizers)