The idea for a shared task on web parsing is really cool. But I don’t get this one:
They’re explicitly banning
- Manually annotating in-domain (web) sentences
- Creating new word clusters, or anything, from as much text data as possible
… instead restricting participants to the data sets they release.
Isn’t a cycle of annotation, error analysis, and new annotations (a self-training + active-learning loop, with smarter decisions through error analysis) the hands-down best way to make an NLP tool for a new domain? Are people scared of this reality? Am I off-base?
I am, of course, just advocating for our Twitter POS tagger approach, where we annotated some data, made a supervised tagger, and iterated on features. The biggest weakness in that paper is we didn’t have additional iterations of error analysis. Our lack of semi-supervised learning was not a weakness.