Kristian Rickert created OPENNLP-1921:
-----------------------------------------
Summary: Paragraph Normalizer
Key: OPENNLP-1921
URL: https://issues.apache.org/jira/browse/OPENNLP-1921
Project: OpenNLP
Issue Type: Improvement
Reporter: Kristian Rickert
Assignee: Kristian Rickert
I'm working a lot of gutenberg and markdown text. Both have a habit of putting
CRs and paragraph separation is to CR/LF (or variations thereof). To better
parse that text without splitting sentences, I made the paragraph normalizer.
I ran the real class against the Alice in Wonderland opening (Project Gutenberg
hard-wrap). Usage is one builder call:
{code:java}
CharSequenceNormalizer normalizer = TextNormalizer.builder()
.whitespacePreservingParagraphs()
.build();
String unwrapped = normalizer.normalize(text).toString();
{code}
Real output from the run:
{noformat}
=== INPUT ===
Alice was beginning to get very tired of sitting by her sister on the
bank, and of having nothing to do: once or twice she had peeped into
the book her sister was reading, but it had no pictures or
conversations in it, "and what is the use of a book," thought Alice
"without pictures or conversations?"
So she was considering in her own mind (as well as she could, for the
=== NORMALIZED ===
Alice was beginning to get very tired of sitting by her sister on the bank, and
of having nothing to do: once or twice she had peeped into the book her sister
was reading, but it had no pictures or conversations in it, "and what is the
use of a book," thought Alice "without pictures or conversations?"
So she was considering in her own mind (as well as she could, for the hot day
made her feel very sleepy and stupid), ...
{noformat}
The rule: a whitespace run with at most one line break becomes a single space
(intra-paragraph wrap is layout, not meaning), a run with two or more becomes
one newline (the blank line is the paragraph break), and CRLF counts as one
break. Because it implements OffsetAwareNormalizer, token offsets after
unwrapping still map back into the original wrapped text, which is what makes
it safe to put in front of a sentence detector.
Markdown: it works for prose markdown (paragraphs stay intact, blank-line
separated headers survive), but it will flatten anything
single-newline-separated that you care about structurally, tight lists, code
blocks. For markdown you'd typically want it only on extracted prose, not the
raw file. I'm thinking if I should make an option to normalize wiki/markdown
style around this as an extension, but off the table for now.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
