[
https://issues.apache.org/jira/browse/OPENNLP-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner reassigned OPENNLP-1921:
---------------------------------------
Assignee: Kristian Rickert (was: Martin Wiesner)
> Add Paragraph Normalizer for Gutenberg-style and Markdown text
> --------------------------------------------------------------
>
> Key: OPENNLP-1921
> URL: https://issues.apache.org/jira/browse/OPENNLP-1921
> Project: OpenNLP
> Issue Type: Improvement
> Affects Versions: 3.0.0-M5
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M6
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> I'm working a lot of gutenberg and markdown text. Both have a habit of
> putting CRs and paragraph separation is to CR/LF (or variations thereof). To
> better parse that text without splitting sentences, I made the paragraph
> normalizer.
> I ran the real class against the Alice in Wonderland opening (Project
> Gutenberg hard-wrap). Usage is one builder call:
> {code:java}
> CharSequenceNormalizer normalizer = TextNormalizer.builder()
> .whitespacePreservingParagraphs()
> .build();
> String unwrapped = normalizer.normalize(text).toString();
> {code}
> Real output from the run:
> {noformat}
> === INPUT ===
> Alice was beginning to get very tired of sitting by her sister on the
> bank, and of having nothing to do: once or twice she had peeped into
> the book her sister was reading, but it had no pictures or
> conversations in it, "and what is the use of a book," thought Alice
> "without pictures or conversations?"
> So she was considering in her own mind (as well as she could, for the
> === NORMALIZED ===
> Alice was beginning to get very tired of sitting by her sister on the bank,
> and of having nothing to do: once or twice she had peeped into the book her
> sister was reading, but it had no pictures or conversations in it, "and what
> is the use of a book," thought Alice "without pictures or conversations?"
> So she was considering in her own mind (as well as she could, for the hot day
> made her feel very sleepy and stupid), ...
> {noformat}
> The rule: a whitespace run with at most one line break becomes a single space
> (intra-paragraph wrap is layout, not meaning), a run with two or more becomes
> one newline (the blank line is the paragraph break), and CRLF counts as one
> break. Because it implements OffsetAwareNormalizer, token offsets after
> unwrapping still map back into the original wrapped text, which is what makes
> it safe to put in front of a sentence detector.
> Markdown: it works for prose markdown (paragraphs stay intact, blank-line
> separated headers survive), but it will flatten anything
> single-newline-separated that you care about structurally, tight lists, code
> blocks. For markdown you'd typically want it only on extracted prose, not the
> raw file. I'm thinking if I should make an option to normalize wiki/markdown
> style around this as an extension, but off the table for now.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
