[
https://issues.apache.org/jira/browse/STANBOL-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096705#comment-13096705
]
Joern Kottmann commented on STANBOL-321:
----------------------------------------
We also have this issue when we analyze scanned news paper articles. In my
opinion there should be a dedicated component to remove the hyphen and merge
the tokens.
> Named Entity detection engine should better deal with hyphenated text
> ---------------------------------------------------------------------
>
> Key: STANBOL-321
> URL: https://issues.apache.org/jira/browse/STANBOL-321
> Project: Stanbol
> Issue Type: Bug
> Reporter: Olivier Grisel
> Assignee: Olivier Grisel
>
> We need some pre-processing to make it easier for OpenNLP to deal with
> hyphens, for instance this is an example of a real PDF:
> Sparse RBMs and sparse auto-encoders
> (RBM, SAE): In some of our experiments, we
> train sparse RBMs (Hinton et al., 2006) and
> sparse auto-encoders (Ranzato et al., 2007; Ben-
> gio et al., 2006), both using a logistic sigmoid non-
> linearity g(W x + b). These algorithms yield a set
> of weights W and biases b. To obtain the dictio-
> nary, D, we simply discard the biases and take
> D = W ⊤ , then normalize the columns of D.
> The current implementation return a TextAnnotation with entity-type = Person
> for the mention "Ben -" (which is then matched to "Ben Stiller" :P).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
