[jira] [Updated] (TIKA-1599) Switch from TagSoup to JSoup
[ https://issues.apache.org/jira/browse/TIKA-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Allison updated TIKA-1599: -- Attachment: tagsoup_vs_jsoup_reports.zip I ran a comparison on our 80k html docs. It looks like we gain about 5% in common English words if we move to JSoup at the cost of 4 new exceptions. I'm currently getting far fewer metadata items with JSoup, but that's almost certain to be the fault of my initial implementation. If anyone has a chance to look through contents/content_diffs.xlsx, I'd appreciate all feedback. > Switch from TagSoup to JSoup > > > Key: TIKA-1599 > URL: https://issues.apache.org/jira/browse/TIKA-1599 > Project: Tika > Issue Type: Improvement > Components: parser >Affects Versions: 1.7, 1.8 >Reporter: Ken Krugler >Assignee: Ken Krugler >Priority: Minor > Attachments: tagsoup_vs_jsoup_reports.zip > > > There are several Tika issues related to how TagSoup cleans up HTML > ([TIKA-381], [TIKA-985], maybe [TIKA-715]), but TagSoup doesn't seem to be > under active development. > On the other hand I know of several projects that are now using > [JSoup|https://github.com/jhy/jsoup], which is an active project (albeit only > one main contributor) under the MIT license. > I haven't looked into how hard it would be to switch this dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TIKA-1599) Switch from TagSoup to JSoup
[ https://issues.apache.org/jira/browse/TIKA-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated TIKA-1599: Attachment: TIKA-1599-crazy-files.tar.gz Here are some crazy files from our tests : * maszol - no title reported in characters() method * newyorker - incorrectly closes a DIV tag and therefore misses the closing NAV tag. See line ~249 * rtlnieuws-emptystackexception - our content handler trips over something, causing ESE in endElement() * techhive - some date block is not reported in SAX events * aljazeera2 - there is no title because the div in the head * assemblydems - moved the title tag from the head section to the bottom of the page * events - title text not reported in the content handler's characters() method * greenenergy4us - page actually manages to have two bodies AND and DOCTYPE on top of every article > Switch from TagSoup to JSoup > > > Key: TIKA-1599 > URL: https://issues.apache.org/jira/browse/TIKA-1599 > Project: Tika > Issue Type: Improvement > Components: parser >Affects Versions: 1.7, 1.8 >Reporter: Ken Krugler >Assignee: Ken Krugler >Priority: Minor > Attachments: TIKA-1599-crazy-files.tar.gz, > tagsoup_vs_jsoup_reports.zip > > > There are several Tika issues related to how TagSoup cleans up HTML > ([TIKA-381], [TIKA-985], maybe [TIKA-715]), but TagSoup doesn't seem to be > under active development. > On the other hand I know of several projects that are now using > [JSoup|https://github.com/jhy/jsoup], which is an active project (albeit only > one main contributor) under the MIT license. > I haven't looked into how hard it would be to switch this dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TIKA-1599) Switch from TagSoup to JSoup
[ https://issues.apache.org/jira/browse/TIKA-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated TIKA-1599: Attachment: consumentenbond.html > Switch from TagSoup to JSoup > > > Key: TIKA-1599 > URL: https://issues.apache.org/jira/browse/TIKA-1599 > Project: Tika > Issue Type: Improvement > Components: parser >Affects Versions: 1.7, 1.8 >Reporter: Ken Krugler >Assignee: Ken Krugler >Priority: Minor > Attachments: TIKA-1599-crazy-files.tar.gz, consumentenbond.html, > tagsoup_vs_jsoup_reports.zip > > > There are several Tika issues related to how TagSoup cleans up HTML > ([TIKA-381], [TIKA-985], maybe [TIKA-715]), but TagSoup doesn't seem to be > under active development. > On the other hand I know of several projects that are now using > [JSoup|https://github.com/jhy/jsoup], which is an active project (albeit only > one main contributor) under the MIT license. > I haven't looked into how hard it would be to switch this dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TIKA-1599) Switch from TagSoup to JSoup
[ https://issues.apache.org/jira/browse/TIKA-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ken Krugler updated TIKA-1599: -- Priority: Major (was: Minor) > Switch from TagSoup to JSoup > > > Key: TIKA-1599 > URL: https://issues.apache.org/jira/browse/TIKA-1599 > Project: Tika > Issue Type: Improvement > Components: parser >Affects Versions: 1.7, 1.8 >Reporter: Ken Krugler >Assignee: Ken Krugler >Priority: Major > Attachments: TIKA-1599-crazy-files.tar.gz, consumentenbond.html, > tagsoup_vs_jsoup_reports.zip > > > There are several Tika issues related to how TagSoup cleans up HTML > ([TIKA-381], [TIKA-985], maybe [TIKA-715]), but TagSoup doesn't seem to be > under active development. > On the other hand I know of several projects that are now using > [JSoup|https://github.com/jhy/jsoup], which is an active project (albeit only > one main contributor) under the MIT license. > I haven't looked into how hard it would be to switch this dependency. -- This message was sent by Atlassian Jira (v8.3.2#803003)
