[jira] [Created] (TIKA-4098) Detection fail on PDF with garbage before header
Thierry Guérin created TIKA-4098: Summary: Detection fail on PDF with garbage before header Key: TIKA-4098 URL: https://issues.apache.org/jira/browse/TIKA-4098 Project: Tika Issue Type: Bug Components: core Affects Versions: 2.8.0 Reporter: Thierry Guérin Attachments: garbageBeforeHeader.pdf PDF detection fails on files that contain too much garbage before the header 'PDF%-'. Those PDFs do not respect the specification, but are nonetheless correctly handled by PDF viewers. The joined PDF is an example on the garbage found in a real-life PDF (looks like email headers that 'leaked' onto the PDF file). The PDF itself is one that I generated so that the exemple si small. The current magic for PDFs limits the search for the '%PDF-%' header to 512 bytes, and in the joined PDF it's located after 702 garbage bytes. I looked at the sources of PdfBox and Ghostscript to see how they handle this case and: * Ghostscript searches through the entire file (see [https://github.com/ArtifexSoftware/ghostpdl/blob/master/pdf/ghostpdf.c] lines 1323-1339) * PdfBox reads the file line by line, and stops looking for the header when it encounters a line that starts with a digit (see [https://github.com/apache/pdfbox/blob/trunk/pdfbox/src/main/java/org/apache/pdfbox/pdfparser/COSParser.java] lines 1561-) >From the doc in tika-mimetypes.xml for the application/pdf MIME type, I >understand that increasing the maximum offset can trigger false positives. I >increased it to 768, and the unit tests pass, but I didn't find any PDF that >tests this particular case, so either it doesn't exist or there are >integration tests that aren't part of this repo ? How can I go about testing for regressions ? I can provide a pull request for this change, but where do I put the test PDF and a unit test? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (TIKA-4098) Detection fails on PDF with garbage before header
[ https://issues.apache.org/jira/browse/TIKA-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thierry Guérin updated TIKA-4098: - Summary: Detection fails on PDF with garbage before header (was: Detection fail on PDF with garbage before header) > Detection fails on PDF with garbage before header > - > > Key: TIKA-4098 > URL: https://issues.apache.org/jira/browse/TIKA-4098 > Project: Tika > Issue Type: Bug > Components: core >Affects Versions: 2.8.0 >Reporter: Thierry Guérin >Priority: Minor > Attachments: garbageBeforeHeader.pdf > > > PDF detection fails on files that contain too much garbage before the header > 'PDF%-'. > Those PDFs do not respect the specification, but are nonetheless correctly > handled by PDF viewers. > The joined PDF is an example on the garbage found in a real-life PDF (looks > like email headers that 'leaked' onto the PDF file). The PDF itself is one > that I generated so that the exemple si small. > The current magic for PDFs limits the search for the '%PDF-%' header to 512 > bytes, and in the joined PDF it's located after 702 garbage bytes. > I looked at the sources of PdfBox and Ghostscript to see how they handle this > case and: > * Ghostscript searches through the entire file (see > [https://github.com/ArtifexSoftware/ghostpdl/blob/master/pdf/ghostpdf.c] > lines 1323-1339) > * PdfBox reads the file line by line, and stops looking for the header when > it encounters a line that starts with a digit (see > [https://github.com/apache/pdfbox/blob/trunk/pdfbox/src/main/java/org/apache/pdfbox/pdfparser/COSParser.java] > lines 1561-) > From the doc in tika-mimetypes.xml for the application/pdf MIME type, I > understand that increasing the maximum offset can trigger false positives. I > increased it to 768, and the unit tests pass, but I didn't find any PDF that > tests this particular case, so either it doesn't exist or there are > integration tests that aren't part of this repo ? > How can I go about testing for regressions ? I can provide a pull request for > this change, but where do I put the test PDF and a unit test? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [tika] SchwingSK opened a new pull request, #1231: [TIKA-4098] Detection fails on PDF with garbage before header
SchwingSK opened a new pull request, #1231: URL: https://github.com/apache/tika/pull/1231 Thanks for your contribution to [Apache Tika](https://tika.apache.org/)! Your help is appreciated! Before opening the pull request, please verify that * there is an open issue on the [Tika issue tracker](https://issues.apache.org/jira/projects/TIKA) which describes the problem or the improvement. We cannot accept pull requests without an issue because the change wouldn't be listed in the release notes. * the issue ID (`TIKA-`) - is referenced in the title of the pull request - and placed in front of your commit messages surrounded by square brackets (`[TIKA-] Issue or pull request title`) * commits are squashed into a single one (or few commits for larger changes) * Tika is successfully built and unit tests pass by running `mvn clean test` * there should be no conflicts when merging the pull request branch into the *recent* `main` branch. If there are conflicts, please try to rebase the pull request branch on top of a freshly pulled `main` branch * if you add new module that downstream users will depend upon add it to relevant group in `tika-bom/pom.xml`. We will be able to faster integrate your pull request if these conditions are met. If you have any questions how to fix your problem or about using Tika in general, please sign up for the [Tika mailing list](http://tika.apache.org/mail-lists.html). Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@tika.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (TIKA-4098) Detection fails on PDF with garbage before header
[ https://issues.apache.org/jira/browse/TIKA-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17741557#comment-17741557 ] ASF GitHub Bot commented on TIKA-4098: -- SchwingSK opened a new pull request, #1231: URL: https://github.com/apache/tika/pull/1231 Thanks for your contribution to [Apache Tika](https://tika.apache.org/)! Your help is appreciated! Before opening the pull request, please verify that * there is an open issue on the [Tika issue tracker](https://issues.apache.org/jira/projects/TIKA) which describes the problem or the improvement. We cannot accept pull requests without an issue because the change wouldn't be listed in the release notes. * the issue ID (`TIKA-`) - is referenced in the title of the pull request - and placed in front of your commit messages surrounded by square brackets (`[TIKA-] Issue or pull request title`) * commits are squashed into a single one (or few commits for larger changes) * Tika is successfully built and unit tests pass by running `mvn clean test` * there should be no conflicts when merging the pull request branch into the *recent* `main` branch. If there are conflicts, please try to rebase the pull request branch on top of a freshly pulled `main` branch * if you add new module that downstream users will depend upon add it to relevant group in `tika-bom/pom.xml`. We will be able to faster integrate your pull request if these conditions are met. If you have any questions how to fix your problem or about using Tika in general, please sign up for the [Tika mailing list](http://tika.apache.org/mail-lists.html). Thanks! > Detection fails on PDF with garbage before header > - > > Key: TIKA-4098 > URL: https://issues.apache.org/jira/browse/TIKA-4098 > Project: Tika > Issue Type: Bug > Components: core >Affects Versions: 2.8.0 >Reporter: Thierry Guérin >Priority: Minor > Attachments: garbageBeforeHeader.pdf > > > PDF detection fails on files that contain too much garbage before the header > 'PDF%-'. > Those PDFs do not respect the specification, but are nonetheless correctly > handled by PDF viewers. > The joined PDF is an example on the garbage found in a real-life PDF (looks > like email headers that 'leaked' onto the PDF file). The PDF itself is one > that I generated so that the exemple si small. > The current magic for PDFs limits the search for the '%PDF-%' header to 512 > bytes, and in the joined PDF it's located after 702 garbage bytes. > I looked at the sources of PdfBox and Ghostscript to see how they handle this > case and: > * Ghostscript searches through the entire file (see > [https://github.com/ArtifexSoftware/ghostpdl/blob/master/pdf/ghostpdf.c] > lines 1323-1339) > * PdfBox reads the file line by line, and stops looking for the header when > it encounters a line that starts with a digit (see > [https://github.com/apache/pdfbox/blob/trunk/pdfbox/src/main/java/org/apache/pdfbox/pdfparser/COSParser.java] > lines 1561-) > From the doc in tika-mimetypes.xml for the application/pdf MIME type, I > understand that increasing the maximum offset can trigger false positives. I > increased it to 768, and the unit tests pass, but I didn't find any PDF that > tests this particular case, so either it doesn't exist or there are > integration tests that aren't part of this repo ? > How can I go about testing for regressions ? I can provide a pull request for > this change, but where do I put the test PDF and a unit test? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (TIKA-4098) Detection fails on PDF with garbage before header
[ https://issues.apache.org/jira/browse/TIKA-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17741558#comment-17741558 ] Thierry Guérin commented on TIKA-4098: -- Created pull request with a unit test : https://github.com/apache/tika/pull/1231 > Detection fails on PDF with garbage before header > - > > Key: TIKA-4098 > URL: https://issues.apache.org/jira/browse/TIKA-4098 > Project: Tika > Issue Type: Bug > Components: core >Affects Versions: 2.8.0 >Reporter: Thierry Guérin >Priority: Minor > Attachments: garbageBeforeHeader.pdf > > > PDF detection fails on files that contain too much garbage before the header > 'PDF%-'. > Those PDFs do not respect the specification, but are nonetheless correctly > handled by PDF viewers. > The joined PDF is an example on the garbage found in a real-life PDF (looks > like email headers that 'leaked' onto the PDF file). The PDF itself is one > that I generated so that the exemple si small. > The current magic for PDFs limits the search for the '%PDF-%' header to 512 > bytes, and in the joined PDF it's located after 702 garbage bytes. > I looked at the sources of PdfBox and Ghostscript to see how they handle this > case and: > * Ghostscript searches through the entire file (see > [https://github.com/ArtifexSoftware/ghostpdl/blob/master/pdf/ghostpdf.c] > lines 1323-1339) > * PdfBox reads the file line by line, and stops looking for the header when > it encounters a line that starts with a digit (see > [https://github.com/apache/pdfbox/blob/trunk/pdfbox/src/main/java/org/apache/pdfbox/pdfparser/COSParser.java] > lines 1561-) > From the doc in tika-mimetypes.xml for the application/pdf MIME type, I > understand that increasing the maximum offset can trigger false positives. I > increased it to 768, and the unit tests pass, but I didn't find any PDF that > tests this particular case, so either it doesn't exist or there are > integration tests that aren't part of this repo ? > How can I go about testing for regressions ? I can provide a pull request for > this change, but where do I put the test PDF and a unit test? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (TIKA-4098) Detection fails on PDF with garbage before header
[ https://issues.apache.org/jira/browse/TIKA-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17741578#comment-17741578 ] Nick Burch commented on TIKA-4098: -- The more bytes beyond the start we check for the PDF marker, the more likely we are to mis-identify a different file as a PDF. The %PDF- marker is pretty unique at the start of a file, but progressively less so as the content continues. (Consider a markdown file of a talk on file formats, that could easily have the text "Look for %PDF- at the start" on page 10 and we don't want to mark the whole thing as a PDF!) If you know for sure that a file is a PDF, just skip detection and tell Tika and we'll hand it off to the PDF parser! If your use case has very few text-based formats, you can fairly safely bump the search window up. Out-of-the-box, I'd be very worried to push it much further due to the false positive risk > Detection fails on PDF with garbage before header > - > > Key: TIKA-4098 > URL: https://issues.apache.org/jira/browse/TIKA-4098 > Project: Tika > Issue Type: Bug > Components: core >Affects Versions: 2.8.0 >Reporter: Thierry Guérin >Priority: Minor > Attachments: garbageBeforeHeader.pdf > > > PDF detection fails on files that contain too much garbage before the header > 'PDF%-'. > Those PDFs do not respect the specification, but are nonetheless correctly > handled by PDF viewers. > The joined PDF is an example on the garbage found in a real-life PDF (looks > like email headers that 'leaked' onto the PDF file). The PDF itself is one > that I generated so that the exemple si small. > The current magic for PDFs limits the search for the '%PDF-%' header to 512 > bytes, and in the joined PDF it's located after 702 garbage bytes. > I looked at the sources of PdfBox and Ghostscript to see how they handle this > case and: > * Ghostscript searches through the entire file (see > [https://github.com/ArtifexSoftware/ghostpdl/blob/master/pdf/ghostpdf.c] > lines 1323-1339) > * PdfBox reads the file line by line, and stops looking for the header when > it encounters a line that starts with a digit (see > [https://github.com/apache/pdfbox/blob/trunk/pdfbox/src/main/java/org/apache/pdfbox/pdfparser/COSParser.java] > lines 1561-) > From the doc in tika-mimetypes.xml for the application/pdf MIME type, I > understand that increasing the maximum offset can trigger false positives. I > increased it to 768, and the unit tests pass, but I didn't find any PDF that > tests this particular case, so either it doesn't exist or there are > integration tests that aren't part of this repo ? > How can I go about testing for regressions ? I can provide a pull request for > this change, but where do I put the test PDF and a unit test? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (TIKA-4098) Detection fails on PDF with garbage before header
[ https://issues.apache.org/jira/browse/TIKA-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17741681#comment-17741681 ] Tilman Hausherr commented on TIKA-4098: --- Yeah what if we pass an e-mail that contains a PDF? > Detection fails on PDF with garbage before header > - > > Key: TIKA-4098 > URL: https://issues.apache.org/jira/browse/TIKA-4098 > Project: Tika > Issue Type: Bug > Components: core >Affects Versions: 2.8.0 >Reporter: Thierry Guérin >Priority: Minor > Attachments: garbageBeforeHeader.pdf > > > PDF detection fails on files that contain too much garbage before the header > 'PDF%-'. > Those PDFs do not respect the specification, but are nonetheless correctly > handled by PDF viewers. > The joined PDF is an example on the garbage found in a real-life PDF (looks > like email headers that 'leaked' onto the PDF file). The PDF itself is one > that I generated so that the exemple si small. > The current magic for PDFs limits the search for the '%PDF-%' header to 512 > bytes, and in the joined PDF it's located after 702 garbage bytes. > I looked at the sources of PdfBox and Ghostscript to see how they handle this > case and: > * Ghostscript searches through the entire file (see > [https://github.com/ArtifexSoftware/ghostpdl/blob/master/pdf/ghostpdf.c] > lines 1323-1339) > * PdfBox reads the file line by line, and stops looking for the header when > it encounters a line that starts with a digit (see > [https://github.com/apache/pdfbox/blob/trunk/pdfbox/src/main/java/org/apache/pdfbox/pdfparser/COSParser.java] > lines 1561-) > From the doc in tika-mimetypes.xml for the application/pdf MIME type, I > understand that increasing the maximum offset can trigger false positives. I > increased it to 768, and the unit tests pass, but I didn't find any PDF that > tests this particular case, so either it doesn't exist or there are > integration tests that aren't part of this repo ? > How can I go about testing for regressions ? I can provide a pull request for > this change, but where do I put the test PDF and a unit test? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (TIKA-4064) Update to 2.8.1
[ https://issues.apache.org/jira/browse/TIKA-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17741859#comment-17741859 ] Hudson commented on TIKA-4064: -- SUCCESS: Integrated in Jenkins build Tika » tika-main-jdk11 #1156 (See [https://ci-builds.apache.org/job/Tika/job/tika-main-jdk11/1156/]) TIKA-4064: update jwarc (tilman: [https://github.com/apache/tika/commit/28c8e657517b8bf3d92fd2e0cd774ca05100817d]) * (edit) tika-parent/pom.xml > Update to 2.8.1 > --- > > Key: TIKA-4064 > URL: https://issues.apache.org/jira/browse/TIKA-4064 > Project: Tika > Issue Type: Task > Components: build >Affects Versions: 2.8.0 >Reporter: Tilman Hausherr >Priority: Minor > Fix For: 2.8.1 > > > The latest maven versions plugin finds much more outdated stuff than the > previous one, so I'll do a few updates. -- This message was sent by Atlassian Jira (v8.20.10#820010)
