[
https://issues.apache.org/jira/browse/CTAKES-371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637554#comment-14637554
]
Sean Finan commented on CTAKES-371:
-----------------------------------
To be clear, this isn't really a bug in dictionary lookup; there is
inconsistency in ctakes ptb tokenization. Type the following into ctakes and
check out the tokenization differences for “.2”:
“22q11.2”
22q11 | . | 2
“22q11.2 deletion”
22q11 | .2 | deletion
“22q11.22 deletion”
22q11 | . | 22 | deletion
> update PTB tokenization logic in fast dictionary module
> -------------------------------------------------------
>
> Key: CTAKES-371
> URL: https://issues.apache.org/jira/browse/CTAKES-371
> Project: cTAKES
> Issue Type: Bug
> Components: ctakes-dictionary-lookup
> Affects Versions: 3.2.2
> Reporter: britt fitch
> Assignee: Sean Finan
> Fix For: 3.2.3
>
>
> PTB tokenization logic is used in places like the tokenizer & dictionary
> building code.
> For example, given “22q11.2 deletion syndrome”:
> PTB tokenizer: [22q11, .2, deletion, syndrome]
> Dictionary module: [22q11, ., 2, deletion, syndrome]
> (RareWordTermMapCreator.getTokens)
> Dictionary module should be updated to match PTB tokenization logic used
> elsewhere in ctakes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
