Automating the download would be fine as long as we cache it, as Richard
suggested. Maybe it could be done by a script to prepare the environment,
and not be part of the unit test itself.
Anyway, it would be a good idea to save the data somewhere because we never
know if some of the websites will become unavailable in the future.
2015-04-15 5:31 GMT-03:00 Richard Eckart de Castilho
richard.eck...@gmail.com:
On 15.04.2015, at 10:23, Joern Kottmann kottm...@gmail.com wrote:
With publicly accessible data I mean a corpus you can somehow acquire,
opposed to the data you create on your own for a project.
All the corpora we support in the formats package are publicly
accessible.
Maybe
some you have to buy and for others you just have to sign some agreement.
A very interesting corpus for testing (and training models on) is
OntoNotes.
Here is a link to the LDC entry:
https://catalog.ldc.upenn.edu/LDC2011T03
You can get it for free (or for a small distribution fee) but you can't
just download it.
It would be great if the ASF could acquire this data set so we can share
it
among the committers.
Is that what you mean with proprietary data?
Yes, that is what I mean.
E.g. the TIGER corpus requires clicking through some pages and forms to
reach a download page, but in principle, it appears as if the corpus was
simply downloadable by a deep-link URL. The license terms state, that the
corpus must not be redistributed.
Some tools are also publicly accessible and downloadable but not
redistributable. For example anybody can download TreeTagger and its
models, but only from the original homepage. It is not permitted to
redistribute it, i.e. to publish it to a repository or offer it on an
alternative homepage.
So there is a (small) class of resources between being redistributable and
proprietary (for fee), namely being in principle publicly accessible (for
free) but not redistributable.
Cheers,
-- Richard
