Hi all,

we extract text from newspaper PDFs with Tika 4.2.0 / PDFBox 3.0.8 and lose
every word that contains an ff, fi, fl or ffi ligature when the font names
these glyphs in the AGL underscore form (f_f, f_i, f_l, f_f_i) and has no
/ToUnicode. PDFBox logs "No Unicode mapping for f_i (31) in font ..." and
drops the glyph; Tika writes U+FFFD in its place. "Schaffhauser" becomes
"Scha?hauser", "flexible" becomes "?exible", and the words no longer match
in search.

The cause is GlyphList.toUnicode(): "fi" maps to U+FB01, "five.lin" maps via
the dot-suffix rule, but "f_i" and "f_f_i" return null. PDFBox ships
Adobe's glyph list and already applies the AGL specification's [1] suffix
rule and uniXXXX/uXXXX forms; the underscore rule from the same section
(f_f_i -> f f i) is the only part missing. trunk and 3.0 have the same
code.

This has come up before as a warning (dev@ 2015 with HelveticaNeueLTStd,
PDFBOX-3333 with MyriadPro-Bold, users@ 2019 with Gotham), and the answer
then was that such PDFs are at fault. Strictly, that's right: ToUnicode is
optional, and ISO 32000-1 9.10.2 only guarantees the glyph-name fallback
for names from the Adobe standard Latin character set, which f_i is not.
But PDFBox already goes beyond 9.10.2 with uniXXXX/uXXXX names and suffix
stripping, and the underscore form is what section 6 of the AGL
specification recommends for ligatures ("the 'o f f i' ligature should be
named 'o_f_f_i'"). These are mainstream commercial fonts, and poppler
(pdftotext) and pdf.js extract the same files correctly.

The change is small and only touches names that map to null today:

    else if (name.indexOf('_') >= 0)
    {
        // AGL specification: a ligature name maps component by component
        unicode = toUnicodeComponents(name.split("_", -1));
    }

    private String toUnicodeComponents(String[] components)
    {
        StringBuilder sb = new StringBuilder();
        for (String component : components)
        {
            String part = component.isEmpty() ? null : toUnicode(component);
            if (part == null)
            {
                return null;
            }
            sb.append(part);
        }
        return sb.toString();
    }

With it: f_i -> "fi", f_f_i -> "ffi", f_f_i.alt -> "ffi", T_h -> "Th",
uni0066_uni0069 -> "fi"; f__i, _, f_ and f_bogus stay null; fi and A are
unchanged. On a 12-page newspaper supplement it repaired 45 words and
changed nothing else.

One choice: the AGL spec maps an unknown component to an empty string and
keeps the rest. I return null for the whole name instead, so no output
changes for names PDFBox can't resolve today. Happy to do it either way.

If this is welcome, I'll open a Jira issue with a minimal synthetic
reproducer PDF (non-embedded Helvetica, /Differences with f_i, f_f_i, f_l,
f_f, no ToUnicode) and a PR against trunk with a regression test, for
backport to 3.0.

[1] https://github.com/adobe-type-tools/agl-specification

Thanks,
Michael

Reply via email to