[epe-users] [ud-conll-shared-task] Regarding UniMelb with illegitimate training data at EPE 2018 ?

[Stephan Oepen]

hi dat quoc,

i am sorry about the delay in getting back to you about your scores!

yes, indeed, by ‘available treebanks’ in the email you noticed, we had
in mind the set of english treebanks in the official training data for
the CoNLL 2018 shared task.  i am very sorry you were misled by that
wording, but i believe the EPE 2018 web site did provide an
unambiguous task description:

> The ‘wildcard’ treebank code indicates that participants can choose
> freely which English parsing model to use; however, only the training
> data (and other materials) provided from the core UD parsing task can
> be used.

anway, in a sense it will be useful that your submission helps compare
scores to some of the EPE 2017 submissions (where participants were
free to use whatever training data they could get their hands on).
so, we will of course include your end-to-end results in our summary
paper, but for considerations of fairness we will not consider your
system in the EPE 2018 ranking.  i briefly glanced of the EPE-related
section in your draft system description, and personally i would think
that the comparisons to 2017 results are informative; but you should
of course adjust what you say about your ranking relative to the EPE
2018 field :-).

—everyone, we are still missing a few replies to our survey of EPE
2018 participants; if you have not responded yet, please go there now:

https://goo.gl/forms/MRRMzZqf8imkQT5b2

from responses so far, at least one team is still a bit unclear about
publication plans around EPE 2018.  there will not be a separate
proceedings volume for this task, and there is no expectation that
anyone prepare an additional system description.  EPE 2018 is
essentially a mere add-on to the CoNLL 2018 Shared Task on UD Parsing.

we encourage participants to make reference to end-to-end results in
their system descriptions, and maybe speculate about correlations you
see between the intrinsic scores for your system (in the CoNLL 2018
task) and the EPE 2018 results.  a more high-level summary of EPE 2018
results will be published as an additional overview paper in the CoNLL
2018 Shared Task proceedings.  we plan to circulate a draft version of
the EPE 2018 overview paper later this months, i.e. at least a week or
so before the camera-ready deadline for system descriptions.

best wishes, oe (for the EPE 2018 co-organizers)


On Thu, Aug 2, 2018 at 6:36 PM, Dat Quoc Nguyen <dqnguyen at unimelb.edu.au> wrote:
> Dear EPE 2018 organisers,
>
> I have just learned that our UniMelb submission for EPE 2018 used
> illegitimate training data.
>
> I was not aware of the restriction of using only English UD treebanks.
>
> From your (previous) email, I initially thought that participating teams
> could use any treebank available for English (i.e., similar to EPE 2017):
>
>> On Fri, Jul 6, 2018 at 11:54 AM, Stephan Oepen <oe at ifi.uio.no> wrote:
>>
>> >
>> > furthermore, it appears that some teams struggle with the unusual
>> > ‘tcode’ values in the EPE data set (which does not correspond to one
>> > of the english UD treebanks, meaning that teams are free to decide
>> > freely on which of the available treebanks to train), or maybe also
>> > stumble over the missing ‘goldfile’ entries in our ‘metadata.json’.
>> > in case any of these throw up your scripts from the core parsing task,
>> > please use the public EPE trial data (available on TIRA) to debug, so
>> > that you can easily inspect system behavior.
>> >
>>
>
> It was my mistake for not carefully double-checking the EPE website.
>
> I understand that our UniMelb submission scores are likely not officially
> ranked. So in this email I would like to ask you that whether our scores
> will be mentioned in the EPE overview paper ?
> (i.e. will there be any option for unofficial rank in the overview paper?)
>
> I completed camera ready version for our CoNLL 2018 shared task paper (as in
> the attachment file), including our EPE scores with a comparison analysis to
> the EPE 2017 Stanford-Paris system (for which I followed the data used by
> this system).
>
> Your response will help me to decide: either removing the EPE 2018 section
> in our shared task paper in case our UniMelb scores not mentioned in the
> overview paper, or modifying this section to mention an unofficial rank.
>
> Thank you very much.
>
> Best regards,
>
> Dat Quoc Nguyen
> Research Fellow
> School of Computing and Information Systems
> The University of Melbourne
> https://people.eng.unimelb.edu.au/dqnguyen
>
> --
> Contact zeman at ufal.mff.cuni.cz in case of problems with this group
> ---
> You received this message because you are subscribed to the Google Groups
> "UD CoNLL Shared Task" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ud-conll-shared-task+unsubscribe at googlegroups.com.
> To post to this group, send email to ud-conll-shared-task at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.