[ https://issues.apache.org/jira/browse/TIKA-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418755#comment-15418755 ]
Tim Allison commented on TIKA-2054: ----------------------------------- I don't think we want to modify our SafeContentHandler to stop converting control characters. This is difficult. If I understand correctly, PDFBox complains that the ligatures aren't correctly encoded: {noformat} Aug 12, 2016 8:03:21 AM org.apache.pdfbox.pdmodel.font.PDSimpleFont toUnicode WARNING: No Unicode mapping for f_i (31) in font XOILAG+MyriadPro-Bold Aug 12, 2016 8:03:21 AM org.apache.pdfbox.pdmodel.font.PDSimpleFont toUnicode WARNING: No Unicode mapping for f_i (31) in font XOILAG+MyriadPro-Regular Aug 12, 2016 8:03:21 AM org.apache.pdfbox.pdmodel.font.PDSimpleFont toUnicode WARNING: No Unicode mapping for f_f (30) in font XOILAG+MyriadPro-Regular Aug 12, 2016 8:03:21 AM org.apache.pdfbox.pdmodel.font.PDSimpleFont toUnicode WARNING: No Unicode mapping for f_l (29) in font XOILAG+MyriadPro-Regular Aug 12, 2016 8:03:22 AM org.apache.pdfbox.pdmodel.font.PDSimpleFont toUnicode WARNING: No Unicode mapping for f_f_i (28) in font XOILAG+MyriadPro-Regular {noformat} So "fi" is being mapped to "0x1f" (31), "ff" to "0x1e" (30), and, as you point out, you can recover these by a custom mapping in the output of PDFBox. Tika via its SafeContentHandler converts most chars < 0x20 to '\ufffd'. Adobe Reader seems to do the same thing that PDFBox does, but Microsoft Edge is able to correctly extract e.g. "confidentiality"...not sure how that is happening?! > Problem with ligatures converting from PDF to HTML with Tika > ------------------------------------------------------------ > > Key: TIKA-2054 > URL: https://issues.apache.org/jira/browse/TIKA-2054 > Project: Tika > Issue Type: Bug > Affects Versions: 1.11, 1.13 > Reporter: Angela O > Attachments: 2482_2014_DAVIDE+CAMPARI-MILANO+SPA_SUSTY-AR.pdf > > > When converting certain PDFs from PDF to HTML I am having trouble with > ligature characters being displayed as U+FFFD � REPLACEMENT CHARACTER > I have tried using Apache Tika 1.11 and 1.13, converting on the command line > using the .jar and get the same results. > If I use pdfbox-app-2.0.1.jar and 'ExtractText' with the icu4j-57_1.jar in > the path and I convert to text rather than HTML then I am able to at least > preserve information about what each ligature was originally, even if they > are still represented as unprintable characters. > I.e. if I run the following from the command line: > java -jar pdfbox-app-1.8.12.jar ExtractText 'test.pdf' 'test.txt' > Then the resulting test.txt when viewed in Sublime2 has "fi" represented as > the US (unit separator character), "ff" represented as RS, "fl" represented > as GS and "ffl" reperesented as FS, which I could then replace with the > appropriate characters. > I was under the impression Tika uses icu4j, is there a way to get the same > behaviour I see with PDFBox with Tika when converting from PDF to HTML? -- This message was sent by Atlassian JIRA (v6.3.4#6332)