A while back I had a similar problem while extracting text from HTML using
Tika. What I did was to hack the Tika HTML parser to extract the text as I
needed. I can't remember exactly how it was, but as far as I remember Tika
raises events when it finds a markup (at least a HTML markup), that is not
handled by default. If you know the structure of the document you are
reading, you can decide what to do with the markup and maybe change the
output (adding a space, a line break etc).
2014-07-15 5:00 GMT-03:00 Jörn Kottmann kottm...@gmail.com:
Text extracted from PDFs must often be cleaned up first, e.g.
fix tokenization, remove page header/footer, fix hyphenation, detect
headlines/titles, etc.
If there are fundamental issues with the plain text the OpenNLP components
trained on cleaned text will not work very well.
Jörn
On 07/15/2014 05:38 AM, Carlos Scheidecker wrote:
Hello all,
I have an interesting problem here. More of a challenge.
I have been doing text cleansing for bad characters and all.
Then I have another interesting problem.
Extracted a public PDF with Tika does not necessary mean you will get
clean
text because the original PDF might have different fonts within a section
that will cause weird behaviors.
If you then divide it into Senteces via OpenNLP you will then get some
interesting sentences.
Trying to parse those sentences then it gets worse.
I am showing an example bellow and I would like to ask about solutions to
it, considering the text can be noisy.
I do not think that it will be easy to fix the Sentence Parser. Here is
what I think on approaching it:
Instead, the best way to do is to look at the sentences poorly parsed,
parse them and extract the inner (S) from the parse as separate sentences.
What would you suggest?
Here is an example of a piece of text extracted with Tika from a public
pdf. This part is what OpenNLP considered to be a sentence:
related research DocumentsBrief: Your Next Portal should Be An Engagement
WorkplaceFebruary 3, 2014Microsoft Aims sharePoint To The CloudJanuary 27,
2014setting The Technology Foundation For Your social Business And
Collaboration strategyJuly 29, 2013The Forrester wave : enterprise social
Platforms, Q2 2014The 13 Providers That Matter Most And How They stack
Upby
rob Koplowitzwith Peter Burris and Nancy Wang2257913JUNE 5, 2014For CIos
The Forrester Wave : Enterprise social Platforms, Q2 2014 2 2014,
Forrester Research, Inc. Reproduction Prohibited June 5, 2014 eNTeRPRIse
sOCIaL PLaTFORM MaRKeT MaTuRes aMID CONsOLIDaTION aND INTegRaTIONThe
enterprise social platform is no longer in its infancy as offerings become
increasingly functional.
It is now parsed as follows:
(S (S (S (NP (VBN related) (NN research) (NNP DocumentsBrief:) (NNP Your)
(NNP Next) (NNP Portal)) (VP (MD should) (VP (VB Be) (NP (NP (DT An) (NNP
Engagement) (NNP WorkplaceFebruary) (CD 3,) (JJ 2014Microsoft) (NNP Aims)
(NN sharePoint)) (PP (TO To) (NP (DT The) (NNP CloudJanuary))) (VP
(VBD
27,) (S (VP (VBG 2014setting) (NP (NP (NP (DT The) (NNP Technology) (NNP
Foundation)) (PP (IN For) (NP (PRP$ Your) (JJ social) (NNP Business
(CC
And) (NP (NNP Collaboration))) (PP (RB strategyJuly) (NP (CD 29,) (CD
2013The) (NNP Forrester) (NN wave))) (: :) (S (VP (VB enterprise) (NP
(JJ social) (NN Platforms,)) (PP (IN Q2) (NP (NP (DT 2014The) (CD 13) (NNS
Providers)) (NP (NP (DT That) (NNP Matter) (JJS Most)) (SBAR (S (CC And)
(SBAR (WHADVP (WRB How)) (S (NP (PRP They)) (VP (VBP stack) (PP (IN Upby)
(NP (NP (NN rob)) (PP (IN Koplowitzwith) (NP (NP (NNP Peter) (NNP Burris))
(CC and) (NP (NP (NNP Nancy) (NNP Wang2257913JUNE) (CD 5,)) (PP (IN
2014For) (NP (NP (NNP CIos)) (NP (DT The) (NNP Forrester) (NNP Wave)) (:
:)
(S (NP (NP (NP (NP (NN Enterprise) (JJ social) (NN Platforms,)) (PP (IN
Q2)
(NP (CD 2014) (CD 2) (JJ 2014,) (NNP Forrester) (NNP Research,) (NNP Inc.)
(NNP Reproduction) (NNP Prohibited) (NNP June) (CD 5,) (CD 2014) (NN
eNTeRPRIse) (NN sOCIaL (NP (NNP PLaTFORM) (NNP MaRKeT) (NNP MaTuRes)
(NN aMID) (NN CONsOLIDaTION))) (PP (IN aND) (NP (DT INTegRaTIONThe) (NN
enterprise) (JJ social) (NN platform (VP (VP (VBZ is) (ADVP (RB no)
(RB
longer)) (PP (IN in) (NP (PRP$ its) (NN infancy))) (SBAR (IN as) (S (NP
(NNS offerings)) (VP (VBP become) (ADVP (RB increasingly)) (VBG
functional.)
Notice that I have more than one (S (S (S
And then I have the first correct structure as (S (NP . (VP.
What is the best way to deal with it?
