[
https://issues.apache.org/jira/browse/OPENNLP-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15544995#comment-15544995
]
Joern Kottmann commented on OPENNLP-862:
----------------------------------------
OpenNLP has to tokenize its input text. Brat can avoid this by just letting the
user decide how he wants to mark things. In the end you will need a tokenizer,
the Whitespace Tokenizer has the isue you mentioned, the SimpleTokenizer splits
on character class and will probably work better for you.
Anyway, I think it makes sense to add an option to let the Brat parser assume
that annotation boundaries are also always token boundaries. It would be very
nice if you could send us a patch to add this option.
> BRAT format packages do not handle punctuation correctly when training NER
> model
> --------------------------------------------------------------------------------
>
> Key: OPENNLP-862
> URL: https://issues.apache.org/jira/browse/OPENNLP-862
> Project: OpenNLP
> Issue Type: Bug
> Components: Formats
> Affects Versions: 1.6.0
> Reporter: Gregory Werner
>
> BRAT does not require preprocessing of text files in order to add annotations
> to text documents. And this is great because I can feed documents from
> corpora I am given directly into BRAT. If I have a line such as:
> Residence: Athens, Georgia
> I would provide 2 annotations in BRAT, Athens and Georgia, and BRAT would
> generate the offset and everything would be fine.
> It appears though that I only get 1 entity correctly processed (and the other
> dropped) in OpenNLP with TokenNameFinderTrainer.brat, Georgia, because the
> comma is not separated from Athens. I have 789 annotated raw, non
> pre-processed text documents from past efforts. I believe that OpenNLP should
> be able to handle lines like the above in the case of the BRAT format code.
> It appears that BratNameSampleStream uses the WhitespaceTokenizer and that is
> what creates Athens, as a token. I find that the SimpleTokenizer might
> perform better with BRAT through my limited testing of raw documents if the
> current general approach is held.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
