Hi opennlp,
I started a thread on ctakes-dev about training the sentence detector to
allow newlines in the middle of sentences, Jorn said it was possible,
now I have a question about how to proceed.
I've replaced all newlines with a special character (ß) and built a
small training file of sentences. I have a question about opennlp
training file format that I couldn't find in the documentation. At the
end of a section, there might be a period, multiple newlines, and some
miscellaneous whitespace:
This concludes the recording of family history.ß ß ßHISTORY OF
PRESENT ILLNESS:
Now, for downstream processing we probably want one sentence ending with
"..history." and the next beginning "HISTORY..."
But what does that mean for the file format? Should it include all the
newlines and other whitespace between the period (which "officially"
ends the sentence) and the start of the next sentence? If so, does it go
at the end of the first line or the beginning of the second? Does the
algorithm even use this info?
Sorry about the barrage of questions and thanks for your help with this.
It's already coming along nicely but just want to make sure I'm doing it
optimally.
Tim
On 05/23/2013 01:52 PM, Tim Miller wrote:
OK I've started doing this, was able to get training working on a very
small example, will try doing slightly bigger.
Tim
On 05/22/2013 08:03 AM, Jörn Kottmann wrote:
On 05/22/2013 01:17 PM, Miller, Timothy wrote:
That's awesome! It might be worth trying at least. How does the
training
process change? Previously the training data would be one sentence per
line, but with newlines as possible mid-sentence characters that could
be trouble, is there a new representation for training data? Or
would we
have to use the training api?
Good point, yes that will be a problem with the default training
format, but it shouldn't be hard
to solve. In the format itself we could define a new line tag e.g.
<NEWLINE> to mark new lines.
as a hack to make it work with 1.5.3 you could instead use a special
char as a replacement
for the new line char.
When you pass the text down to the sentence detector a simple string
replace could be used to
convert all new line chars to the special new line marker char.
If things work out for you performance wise as well we will just
integrate it properly into OpenNLP
for the next release.
Could you produce a sentence detector training file with a new line
marker char?
You should try to pick a char you can also pass in on a terminal
otherwise you have to use the
API to train the model. The build in cross validation could be used
to evaluate the performance.
Jörn
