[jira] [Commented] (TIKA-1224) Adding Source code (Java, Groovy, C) parser
[ https://issues.apache.org/jira/browse/TIKA-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890560#comment-13890560 ] Markus Jelsma commented on TIKA-1224: - A patch seems to be missing here. > Adding Source code (Java, Groovy, C) parser > --- > > Key: TIKA-1224 > URL: https://issues.apache.org/jira/browse/TIKA-1224 > Project: Tika > Issue Type: Improvement > Components: parser >Affects Versions: 1.5 >Reporter: Hong-Thai Nguyen >Priority: Minor > > We can parser some source code file formats: > text/x-java-source > text/x-groovy > text/x-c > for HTML rendering from code, we can use jhightlight: > http://www.ohloh.net/p/jhighlight -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (TIKA-1229) Hyperlink in .doc page header broken
Lutz Theurer created TIKA-1229: -- Summary: Hyperlink in .doc page header broken Key: TIKA-1229 URL: https://issues.apache.org/jira/browse/TIKA-1229 Project: Tika Issue Type: Bug Components: parser Affects Versions: 1.4 Reporter: Lutz Theurer Priority: Minor If you have a hyperlink to a webpage or mailto in the page header (german: Kopfzeile) of your .doc document the import is defaced like this: �HYPERLINK "http://tw-systemhaus.de"; �http://tw-systemhaus.de� It's however not an issue in text. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (TIKA-1229) Hyperlink in .doc page header broken
[ https://issues.apache.org/jira/browse/TIKA-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890564#comment-13890564 ] Nick Burch commented on TIKA-1229: -- Any chance you could upload a file that shows the issue? Bonus marks if you could write a small junit unit test that uses it to show the problem! > Hyperlink in .doc page header broken > > > Key: TIKA-1229 > URL: https://issues.apache.org/jira/browse/TIKA-1229 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.4 >Reporter: Lutz Theurer >Priority: Minor > > If you have a hyperlink to a webpage or mailto in the page header (german: > Kopfzeile) of your .doc document the import is defaced like this: > �HYPERLINK "http://tw-systemhaus.de"; �http://tw-systemhaus.de� > It's however not an issue in text. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1229) Hyperlink in .doc page header broken
[ https://issues.apache.org/jira/browse/TIKA-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lutz Theurer updated TIKA-1229: --- Attachment: mail.doc > Hyperlink in .doc page header broken > > > Key: TIKA-1229 > URL: https://issues.apache.org/jira/browse/TIKA-1229 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.4 >Reporter: Lutz Theurer >Priority: Minor > Attachments: mail.doc > > > If you have a hyperlink to a webpage or mailto in the page header (german: > Kopfzeile) of your .doc document the import is defaced like this: > �HYPERLINK "http://tw-systemhaus.de"; �http://tw-systemhaus.de� > It's however not an issue in text. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (TIKA-1228) Embedded files not extracted properly from PDF
[ https://issues.apache.org/jira/browse/TIKA-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890605#comment-13890605 ] Tim Allison commented on TIKA-1228: --- Not sure I understand. Is this the snippet that you refer to in PDNameTreeNode: {noformat} public Map getNames() throws IOException { COSArray namesArray = (COSArray)node.getDictionaryObject( COSName.NAMES ); {noformat} The above throws a class cast exception, but the code that you show doesn't? Are you getting a class cast exception on the document that you submitted with this issue or is it a different document? Thank you, again. > Embedded files not extracted properly from PDF > -- > > Key: TIKA-1228 > URL: https://issues.apache.org/jira/browse/TIKA-1228 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.4 > Environment: CentOS 6.5 VM >Reporter: Jason Sherman > Labels: easyfix > Fix For: 1.5 > > Attachments: pdf_with_doc_and_text_attached.pdf > > > IAW pdfbox example here: > http://svn.apache.org/repos/asf/pdfbox/trunk/examples/src/main/java/org/apache/pdfbox/examples/pdmodel/ExtractEmbeddedFiles.java > the PDF parser does not check for additional entries under Kids node when > Names node does not exist. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (TIKA-1228) Embedded files not extracted properly from PDF
[ https://issues.apache.org/jira/browse/TIKA-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890607#comment-13890607 ] Jason Sherman commented on TIKA-1228: - Tim, I saw you already added a test and fix to the codebase. Thanks! I'm going to clone it and use it if you don't mind. Jason > Embedded files not extracted properly from PDF > -- > > Key: TIKA-1228 > URL: https://issues.apache.org/jira/browse/TIKA-1228 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.4 > Environment: CentOS 6.5 VM >Reporter: Jason Sherman > Labels: easyfix > Fix For: 1.5 > > Attachments: pdf_with_doc_and_text_attached.pdf > > > IAW pdfbox example here: > http://svn.apache.org/repos/asf/pdfbox/trunk/examples/src/main/java/org/apache/pdfbox/examples/pdmodel/ExtractEmbeddedFiles.java > the PDF parser does not check for additional entries under Kids node when > Names node does not exist. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (TIKA-1228) Embedded files not extracted properly from PDF
[ https://issues.apache.org/jira/browse/TIKA-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890610#comment-13890610 ] Tim Allison commented on TIKA-1228: --- Y. That's the point of open source. :) Enjoy! Now that I'm looking at this issue again, I dragged out some of my pre-Tika code for pdf attachments using a different pdf library. It looks like the pdf files I was coding against could have the file name in a parent node and the actual bytes in a child or more distant descendant node. Will see if I can dig up the triggering files and see if Tika needs any more mods on PDF attachment extraction. {noformat} private MyPDFAttachment lookForByteStream(COSDictionary dict, MyPDFAttachment attach, int recursiveDepth){ COSName fCOSName = COSName.create("F"); COSName efCOSName = COSName.create("EF"); COSObject fObj = dict.get(fCOSName); COSObject efObj = dict.get(efCOSName); if (null != fObj){ if (fObj.getClass() == COSString.class){ attach.setName(fObj.stringValue()); } else if (fObj.getClass() == COSStream.class){ attach.setBytes(((COSStream)fObj).getDecodedBytes()); return attach; } } if (null != efObj && efObj.getClass() == COSDictionary.class){ int tmpI = recursiveDepth; tmpI++; return lookForByteStream((COSDictionary)efObj, attach, tmpI); } return null; } {noformat} > Embedded files not extracted properly from PDF > -- > > Key: TIKA-1228 > URL: https://issues.apache.org/jira/browse/TIKA-1228 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.4 > Environment: CentOS 6.5 VM >Reporter: Jason Sherman > Labels: easyfix > Fix For: 1.5 > > Attachments: pdf_with_doc_and_text_attached.pdf > > > IAW pdfbox example here: > http://svn.apache.org/repos/asf/pdfbox/trunk/examples/src/main/java/org/apache/pdfbox/examples/pdmodel/ExtractEmbeddedFiles.java > the PDF parser does not check for additional entries under Kids node when > Names node does not exist. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (TIKA-1228) Embedded files not extracted properly from PDF
[ https://issues.apache.org/jira/browse/TIKA-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890608#comment-13890608 ] Jason Sherman commented on TIKA-1228: - Tim, Dang. During my troubleshooting, I first updated pdfbox to 1.8.3 and was using that source to step through the code. After the weirdness with the exception in code, but not in my expression evaluator, I reverted to the original tika code, but failed to revert the pdfbox code. I apologize for the confusion. Thanks again for your fast responses. Jason > Embedded files not extracted properly from PDF > -- > > Key: TIKA-1228 > URL: https://issues.apache.org/jira/browse/TIKA-1228 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.4 > Environment: CentOS 6.5 VM >Reporter: Jason Sherman > Labels: easyfix > Fix For: 1.5 > > Attachments: pdf_with_doc_and_text_attached.pdf > > > IAW pdfbox example here: > http://svn.apache.org/repos/asf/pdfbox/trunk/examples/src/main/java/org/apache/pdfbox/examples/pdmodel/ExtractEmbeddedFiles.java > the PDF parser does not check for additional entries under Kids node when > Names node does not exist. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (TIKA-1228) Embedded files not extracted properly from PDF
[ https://issues.apache.org/jira/browse/TIKA-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890613#comment-13890613 ] Tim Allison commented on TIKA-1228: --- Ok, to confirm, the PDNameTreeNode class cast exception is a non-issue? Thanks again. > Embedded files not extracted properly from PDF > -- > > Key: TIKA-1228 > URL: https://issues.apache.org/jira/browse/TIKA-1228 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.4 > Environment: CentOS 6.5 VM >Reporter: Jason Sherman > Labels: easyfix > Fix For: 1.5 > > Attachments: pdf_with_doc_and_text_attached.pdf > > > IAW pdfbox example here: > http://svn.apache.org/repos/asf/pdfbox/trunk/examples/src/main/java/org/apache/pdfbox/examples/pdmodel/ExtractEmbeddedFiles.java > the PDF parser does not check for additional entries under Kids node when > Names node does not exist. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (TIKA-1228) Embedded files not extracted properly from PDF
[ https://issues.apache.org/jira/browse/TIKA-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890639#comment-13890639 ] Jason Sherman edited comment on TIKA-1228 at 2/4/14 1:11 PM: - Correct. PDNameTreeNode class cast exception is a non-issue. was (Author: agi20dla): Correct. PDNameTreeNode clas cast exception is a non-issue. > Embedded files not extracted properly from PDF > -- > > Key: TIKA-1228 > URL: https://issues.apache.org/jira/browse/TIKA-1228 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.4 > Environment: CentOS 6.5 VM >Reporter: Jason Sherman > Labels: easyfix > Fix For: 1.5 > > Attachments: pdf_with_doc_and_text_attached.pdf > > > IAW pdfbox example here: > http://svn.apache.org/repos/asf/pdfbox/trunk/examples/src/main/java/org/apache/pdfbox/examples/pdmodel/ExtractEmbeddedFiles.java > the PDF parser does not check for additional entries under Kids node when > Names node does not exist. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (TIKA-1228) Embedded files not extracted properly from PDF
[ https://issues.apache.org/jira/browse/TIKA-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890639#comment-13890639 ] Jason Sherman commented on TIKA-1228: - Correct. PDNameTreeNode clas cast exception is a non-issue. > Embedded files not extracted properly from PDF > -- > > Key: TIKA-1228 > URL: https://issues.apache.org/jira/browse/TIKA-1228 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.4 > Environment: CentOS 6.5 VM >Reporter: Jason Sherman > Labels: easyfix > Fix For: 1.5 > > Attachments: pdf_with_doc_and_text_attached.pdf > > > IAW pdfbox example here: > http://svn.apache.org/repos/asf/pdfbox/trunk/examples/src/main/java/org/apache/pdfbox/examples/pdmodel/ExtractEmbeddedFiles.java > the PDF parser does not check for additional entries under Kids node when > Names node does not exist. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (TIKA-1230) Update PDFBox to v1.8.4
Tim Allison created TIKA-1230: - Summary: Update PDFBox to v1.8.4 Key: TIKA-1230 URL: https://issues.apache.org/jira/browse/TIKA-1230 Project: Tika Issue Type: Improvement Affects Versions: 1.5 Reporter: Tim Allison Priority: Trivial Fix For: 1.5 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (TIKA-1230) Update PDFBox to v1.8.4
[ https://issues.apache.org/jira/browse/TIKA-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Allison resolved TIKA-1230. --- Resolution: Fixed r1564335 > Update PDFBox to v1.8.4 > --- > > Key: TIKA-1230 > URL: https://issues.apache.org/jira/browse/TIKA-1230 > Project: Tika > Issue Type: Improvement >Affects Versions: 1.5 >Reporter: Tim Allison >Priority: Trivial > Fix For: 1.5 > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (TIKA-1231) Safely handle null embedded files in PDFs
Tim Allison created TIKA-1231: - Summary: Safely handle null embedded files in PDFs Key: TIKA-1231 URL: https://issues.apache.org/jira/browse/TIKA-1231 Project: Tika Issue Type: Bug Components: parser Affects Versions: 1.5 Reporter: Tim Allison Assignee: Tim Allison Priority: Minor Fix For: 1.5 I filed a potential fix, unit test and test doc for this in PDFBOX-1884. We'll need to add one test for null in the Tika PDFParser to handle this change once it is fixed in PDFBox. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Key Revocation
Hello, (CC dev@tika.apache.org for info) I have had to revoke my code signing key due to media failure. Attached is the revocation key and I am following the steps here: http://www.apache.org/dev/release-signing.html#revoke-cert Cheers, Dave -BEGIN PGP PUBLIC KEY BLOCK- Version: GnuPG v1.4.9 (GNU/Linux) Comment: A revocation certificate should follow iQIfBCABAgAJBQJLrjSDAh0CAAoJEGBlJ+auqMarJnsP/j6E+hQ9vkdvMncXbQXX 9auQeI0tRQDvgoKmQYk9T16QyhXANZoJDzuLEmE/8kMNwr0U5ay3lEV0KsJZe+z8 fnsEmfNoV6xACNwa/DT4V4dQnVvy++K9z8CndX/3QNimduvuKzCZhEbMltwdQSYh Lr7JUWBerayMD3XR8Jl+bYnU47FapI4pgDNOKbLsdhEPhlEcZUhqBy/8d/p/NjI6 NLzOpBieSbYhYYh5O8wjR0JJw5gtYf//IO7GQYoAGzbGLa9m/MCsNSNU9M3HYQs0 dnBYxS3yGOk+rIRrfb/MR6ySjfSLiNb6IvSrSQ2oaJfjm3NnBYD4z1QcAq+JmfFm b0Zq7uAEaxrovzkFX3mwNmxeoMTqJP0nr6napU5y7maJYzsLO/tHf+NeIiZIuoK8 Q/GQE1QJW3nQ2/LOWwAyXTLoMR6/IP+HO9Lk6ad9hXPswqPKw6vGTJrMJJS+eyTR AFoxY8nQ24rkE6uT37hZUPeoZdZ6sGDIJBDWUXTez/U/TgCFrbpoMhMC2oLOWdA0 Qr/vgz6wdb/8x/CUe49eKzdXhIacAI7aYXtLOQwVprARDTUqSrfB9ijwz80BkrL1 OWqgI1iHcsQD0e8eqzVQCndehvLMhezRQ2NSQyZNF0AsBP39OtyK5ATUNrrq6g8L raF/l9VwsC/63dtfJyvq0sLE =Q/66 -END PGP PUBLIC KEY BLOCK-
[jira] [Resolved] (TIKA-1229) Hyperlink in .doc page header broken
[ https://issues.apache.org/jira/browse/TIKA-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Burch resolved TIKA-1229. -- Resolution: Fixed Fix Version/s: 1.5 Should be fixed as of r1564540. Ended up being a bit more work than first anticipated, as we were processing headers and footers in a very simplistic way, which has now been replaced with handling them as proper ranges > Hyperlink in .doc page header broken > > > Key: TIKA-1229 > URL: https://issues.apache.org/jira/browse/TIKA-1229 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.4 >Reporter: Lutz Theurer >Priority: Minor > Fix For: 1.5 > > Attachments: mail.doc > > > If you have a hyperlink to a webpage or mailto in the page header (german: > Kopfzeile) of your .doc document the import is defaced like this: > �HYPERLINK "http://tw-systemhaus.de"; �http://tw-systemhaus.de� > It's however not an issue in text. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (TIKA-1086) Tika-bundle 1.3 does not import org.w3c.dom package
[ https://issues.apache.org/jira/browse/TIKA-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle resolved TIKA-1086. --- Resolution: Fixed Patch added in r1553845. > Tika-bundle 1.3 does not import org.w3c.dom package > --- > > Key: TIKA-1086 > URL: https://issues.apache.org/jira/browse/TIKA-1086 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.3 >Reporter: Gaurav > Fix For: 1.5 > > Attachments: TIKA-1086.svn.diff > > > The tika-bundle 1.3 version does not import org.w3c.dom package, as a result > it is not able to parse DOM based documents such as Microsoft Word (docx) > documents. > This issue does not have in version 1.2 as it does import the necessary > package and therefore the parsing of the documents work fine. > Can someone please look into the issue, as Microsoft Word is a very popular > document. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-605) Tika GDAL parser
[ https://issues.apache.org/jira/browse/TIKA-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-605: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Tika GDAL parser > > > Key: TIKA-605 > URL: https://issues.apache.org/jira/browse/TIKA-605 > Project: Tika > Issue Type: New Feature > Components: parser > Environment: indep. of env. >Reporter: Chris A. Mattmann >Assignee: Chris A. Mattmann > Labels: gdal, gsoc2013, integration, mentor, tika > Fix For: 1.6 > > Attachments: 0001-TIKA-605-Tika-GDAL-parser.patch, > TIKA-605.Mattmann.092511.patch.txt > > > Leverage the GDAL toolkit and its Java SWIG bindings to create a Tika parser > around GDAL. See here: > http://trac.osgeo.org/gdal/browser/trunk/gdal/swig/java/apps/gdalinfo.java -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1167) Embedded object not extracted
[ https://issues.apache.org/jira/browse/TIKA-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1167: -- Fix Version/s: (was: 1.5) 1.6 Push to 1.6, preparing for 1.5 RC > Embedded object not extracted > - > > Key: TIKA-1167 > URL: https://issues.apache.org/jira/browse/TIKA-1167 > Project: Tika > Issue Type: Bug >Affects Versions: 1.4 >Reporter: Daniel Bonniot de Ruisselet >Priority: Critical > Fix For: 1.6 > > Attachments: Doc w Structure that wont extract.docx > > > For the attached docx, tika seems to detect the embedded object, as shown by > this tag: > {{}} > However, extraction itself (using -z on the command line, or using the API) > does not seem to work for this object: > {{java -jar tika-app-1.4.jar -z Doc\ w\ Structure\ that\ wont\ extract.docx}} > {{Extracting 'rId9_image1.wmf' (application/x-msmetafile) to > /tmp/tika/rId9_image1.wmf}} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-539) Encoding detection is too biased by encoding in meta tag
[ https://issues.apache.org/jira/browse/TIKA-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-539: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Encoding detection is too biased by encoding in meta tag > > > Key: TIKA-539 > URL: https://issues.apache.org/jira/browse/TIKA-539 > Project: Tika > Issue Type: Bug > Components: metadata, parser >Affects Versions: 0.8, 0.9, 0.10 >Reporter: Reinhard Schwab >Assignee: Ken Krugler > Fix For: 1.6 > > Attachments: TIKA-539.patch, TIKA-539_2.patch > > > if the encoding in the meta tag is wrong, this encoding is detected, > even if there is the right encoding set in metadata before(which can be from > http response header). > test code to reproduce: > static String content = "\n" > + " content=\"application/xhtml+xml; charset=iso-8859-1\" />" > + "Über den Wolken\n"; > /** >* @param args >* @throws IOException >* @throws TikaException >* @throws SAXException >*/ > public static void main(String[] args) throws IOException, SAXException, > TikaException { > Metadata metadata = new Metadata(); > metadata.set(Metadata.CONTENT_TYPE, "text/html"); > metadata.set(Metadata.CONTENT_ENCODING, "UTF-8"); > System.out.println(metadata.get(Metadata.CONTENT_ENCODING)); > InputStream in = new > ByteArrayInputStream(content.getBytes("UTF-8")); > AutoDetectParser parser = new AutoDetectParser(); > BodyContentHandler h = new BodyContentHandler(1); > parser.parse(in, h, metadata, new ParseContext()); > System.out.print(h.toString()); > System.out.println(metadata.get(Metadata.CONTENT_ENCODING)); > } -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-715) Some parsers produce non-well-formed XHTML SAX events
[ https://issues.apache.org/jira/browse/TIKA-715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-715: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Some parsers produce non-well-formed XHTML SAX events > - > > Key: TIKA-715 > URL: https://issues.apache.org/jira/browse/TIKA-715 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 0.10 >Reporter: Michael McCandless > Fix For: 1.6 > > Attachments: TIKA-715.patch > > > With TIKA-683 I committed simple, commented out code to > SafeContentHandler, to verify that the SAX events produced by the > parser have valid (matched) tags. Ie, each startElement("foo") is > matched by the closing endElement("foo"). > I only did basic nesting test, plus checking that is never > embedded inside another ; we could strengthen this further to check > that all tags only appear in valid parents... > I was able to use this to fix issues with the new RTF parser > (TIKA-683), but I was surprised that some other parsers failed the new > asserts. > It could be these are relatively minor offenses (eg closing a table > w/o closing the tr) and we need not do anything here... but I think > it'd be cleaner if all our parsers produced matched, well-formed XHTML > events. > I haven't looked into any of these... it could be they are easy to fix. > Failures: > {noformat} > testOutlookHTMLVersion(org.apache.tika.parser.microsoft.OutlookParserTest) > Time elapsed: 0.032 sec <<< ERROR! > java.lang.AssertionError: end tag=body with no startElement > at > org.apache.tika.sax.SafeContentHandler.verifyEndElement(SafeContentHandler.java:224) > at > org.apache.tika.sax.SafeContentHandler.endElement(SafeContentHandler.java:275) > at > org.apache.tika.sax.XHTMLContentHandler.endDocument(XHTMLContentHandler.java:210) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:242) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) > at > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:129) > at > org.apache.tika.parser.microsoft.OutlookParserTest.testOutlookHTMLVersion(OutlookParserTest.java:158) > testParseKeynote(org.apache.tika.parser.iwork.IWorkParserTest) Time elapsed: > 0.116 sec <<< ERROR! > java.lang.AssertionError: mismatched elements open=tr close=table > at > org.apache.tika.sax.SafeContentHandler.verifyEndElement(SafeContentHandler.java:226) > at > org.apache.tika.sax.SafeContentHandler.endElement(SafeContentHandler.java:275) > at > org.apache.tika.sax.XHTMLContentHandler.endElement(XHTMLContentHandler.java:252) > at > org.apache.tika.sax.XHTMLContentHandler.endElement(XHTMLContentHandler.java:287) > at > org.apache.tika.parser.iwork.KeynoteContentHandler.endElement(KeynoteContentHandler.java:136) > at > org.apache.tika.sax.ContentHandlerDecorator.endElement(ContentHandlerDecorator.java:136) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:601) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2938) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648) > at > com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205) > at > com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522) > at javax.xml.parsers.SAXParser.parse(SAXParser.java:395) > at javax.xml.parsers.SAXParser.parse(SAXParser.java:198) > at > org.apache.tika.parser.iwork.IWorkPackageParser.parse(IWorkPackageParser.java:190) > at > org.apache.tika.parser.iwork.IWorkParserTest.testParseKeynote(IWorkParserTest.java:49) > testMultipart(org.apache.tika.parser.m
[jira] [Updated] (TIKA-776) ExifTool Embedder
[ https://issues.apache.org/jira/browse/TIKA-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-776: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > ExifTool Embedder > - > > Key: TIKA-776 > URL: https://issues.apache.org/jira/browse/TIKA-776 > Project: Tika > Issue Type: New Feature > Components: metadata >Affects Versions: 1.0 > Environment: ExifTool is required > (http://www.sno.phy.queensu.ca/~phil/exiftool/) >Reporter: Ray Gauss II > Labels: embed, exiftool, patch > Fix For: 1.6 > > Attachments: tika-parsers-exiftool-embed-patch.txt > > > This patch adds an ExifTool ExternalEmbedder which builds upon the work in > issue TIKA-774 and TIKA-775. > In the tika-parsers an ExiftoolExternalEmbedder is added which extends > ExternalEmbedder to programmatically create an Embedder which calls the > ExifTool command line to embed tika metadata into a file stream and an > ExiftoolExternalEmbedderTest unit test is added which embeds several IPTC and > XMP fields then parses the resulting file stream to verify the operation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-819) Make Option to Exclude Embedded Files' Text for Text Content
[ https://issues.apache.org/jira/browse/TIKA-819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-819: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Make Option to Exclude Embedded Files' Text for Text Content > > > Key: TIKA-819 > URL: https://issues.apache.org/jira/browse/TIKA-819 > Project: Tika > Issue Type: New Feature > Components: general >Affects Versions: 1.0 > Environment: Windows-7 + JDK 1.6 u26 >Reporter: Albert L. > Fix For: 1.6 > > > It would be nice to be able to disable text content from embedded files. > For example, if I have a DOCX with an embedded PPTX, then I would like the > option to disable text from the PPTX from showing up when asking for the text > content from DOCX. In other words, it would be nice to have the option to > get text content *only* from the DOCX instead of the DOCX+PPTX. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-985) Support for HTML5 elements
[ https://issues.apache.org/jira/browse/TIKA-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-985: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Support for HTML5 elements > -- > > Key: TIKA-985 > URL: https://issues.apache.org/jira/browse/TIKA-985 > Project: Tika > Issue Type: Improvement > Components: parser >Affects Versions: 1.2 >Reporter: Markus Jelsma > Fix For: 1.6 > > Attachments: TIKA-985-1.3-1.patch, TIKA-985-1.3-2.patch, > TIKA-985-1.3-3.patch, TIKA-985-1.5.patch > > > TagSoup's schema.tssl does not include some HTML5 elements (e.g. article, > section). This prevents some custom ContentHandlers from reading expected > elements and/or attributes. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-995) XHTMLContentHandler doesn't pass attributes of body element
[ https://issues.apache.org/jira/browse/TIKA-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-995: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > XHTMLContentHandler doesn't pass attributes of body element > --- > > Key: TIKA-995 > URL: https://issues.apache.org/jira/browse/TIKA-995 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.2 >Reporter: Markus Jelsma > Fix For: 1.6 > > Attachments: TIKA-995-1.3-1.patch, TIKA-995-unit.patch > > > XHTMLContentHandler.startElement() uses lazyHead() for the body element > because it's defined in the AUTO Set. As a consequence, attributes of the > body element are not passed to downstream content handlers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-774) ExifTool Parser
[ https://issues.apache.org/jira/browse/TIKA-774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-774: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > ExifTool Parser > --- > > Key: TIKA-774 > URL: https://issues.apache.org/jira/browse/TIKA-774 > Project: Tika > Issue Type: New Feature > Components: parser >Affects Versions: 1.0 > Environment: Requires be installed > (http://www.sno.phy.queensu.ca/~phil/exiftool/) >Reporter: Ray Gauss II > Labels: features, newbie, patch, > Fix For: 1.6 > > Attachments: testJPEG_IPTC_EXT.jpg, > tika-core-exiftool-parser-patch.txt, tika-parsers-exiftool-parser-patch.txt > > > Adds an external parser that calls ExifTool to extract extended metadata > fields from images and other content types. > In the core project: > An ExifTool interface is added which contains Property objects that define > the metadata fields available. > An additional Property constructor for internalTextBag type. > In the parsers project: > An ExiftoolMetadataExtractor is added which does the work of calling ExifTool > on the command line and mapping the response to tika metadata fields. This > extractor could be called instead of or in addition to the existing > ImageMetadataExtractor and JempboxExtractor under TiffParser and/or > JpegParser but those have not been changed at this time. > An ExiftoolParser is added which calls only the ExiftoolMetadataExtractor. > An ExiftoolTikaMapper is added which is responsible for mapping the ExifTool > metadata fields to existing tika and Drew Noakes metadata fields if enabled. > An ElementRdfBagMetadataHandler is added for extracting multi-valued RDF Bag > implementations in XML files. > An ExifToolParserTest is added which tests several expected XMP and IPTC > metadata values in testJPEG_IPTC_EXT.jpg. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1059) Better Handling of InterruptedException in ExternalParser and ExternalEmbedder
[ https://issues.apache.org/jira/browse/TIKA-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1059: -- Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Better Handling of InterruptedException in ExternalParser and ExternalEmbedder > -- > > Key: TIKA-1059 > URL: https://issues.apache.org/jira/browse/TIKA-1059 > Project: Tika > Issue Type: Improvement > Components: parser >Affects Versions: 1.3 >Reporter: Ray Gauss II > Fix For: 1.6 > > > The {{ExternalParser}} and {{ExternalEmbedder}} classes currently catch > {{InterruptedException}} and ignore it. > The methods should either call {{interrupt()}} on the current thread or > re-throw the exception, possibly wrapped in a {{TikaException}}. > See TIKA-775 for a previous discussion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1079) Word document hits AIOOBE in SummaryExtractor.parseSummaries
[ https://issues.apache.org/jira/browse/TIKA-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1079: -- Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Word document hits AIOOBE in SummaryExtractor.parseSummaries > > > Key: TIKA-1079 > URL: https://issues.apache.org/jira/browse/TIKA-1079 > Project: Tika > Issue Type: Bug > Components: parser >Reporter: Michael McCandless > Fix For: 1.6 > > Attachments: guide_to_daips_(id_3152_ver_1.0.0).doc > > > I'm not yet sure if this is a corrupted document (though, MS Word opens it > just fine) or a bug in POI ... but I hit this exc when running it through > TikaCLI: > {noformat} > java.lang.ArrayIndexOutOfBoundsException: -1 > at org.apache.poi.hpsf.CodePageString.(CodePageString.java:161) > at > org.apache.poi.hpsf.TypedPropertyValue.readValue(TypedPropertyValue.java:158) > at org.apache.poi.hpsf.VariantSupport.read(VariantSupport.java:163) > at org.apache.poi.hpsf.Property.(Property.java:164) > at org.apache.poi.hpsf.Section.(Section.java:277) > at org.apache.poi.hpsf.PropertySet.init(PropertySet.java:451) > at org.apache.poi.hpsf.PropertySet.(PropertySet.java:246) > at > org.apache.tika.parser.microsoft.SummaryExtractor.parseSummaryEntryIfExists(SummaryExtractor.java:78) > at > org.apache.tika.parser.microsoft.SummaryExtractor.parseSummaries(SummaryExtractor.java:69) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:170) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:161) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) > at > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120) > at org.apache.tika.cli.TikaCLI$OutputType.process(TikaCLI.java:139) > at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:415) > at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:109) > {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1072) AIOOBE when handling embedded document in .doc file
[ https://issues.apache.org/jira/browse/TIKA-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1072: -- Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > AIOOBE when handling embedded document in .doc file > --- > > Key: TIKA-1072 > URL: https://issues.apache.org/jira/browse/TIKA-1072 > Project: Tika > Issue Type: Bug > Components: parser >Reporter: Michael McCandless > Fix For: 1.6 > > Attachments: 20-Force-on-a-current-S00.doc, Ole10NativeEntry.bin > > > I have a Word (.doc) document that hits an exception when I run: > {noformat} > java -jar tika-app/target/tika-app-1.4-SNAPSHOT.jar > /x/tmp/20-Force-on-a-current-S00.doc > {noformat} > Here's the exception: > {noformat} > Caused by: java.lang.ArrayIndexOutOfBoundsException: 40 > at org.apache.poi.util.LittleEndian.getShort(LittleEndian.java:225) > at > org.apache.poi.poifs.filesystem.Ole10Native.(Ole10Native.java:139) > at > org.apache.poi.poifs.filesystem.Ole10Native.createFromEmbeddedOleObject(Ole10Native.java:89) > at > org.apache.tika.parser.microsoft.AbstractPOIFSExtractor.handleEmbeddedOfficeDoc(AbstractPOIFSExtractor.java:149) > at > org.apache.tika.parser.microsoft.WordExtractor.parse(WordExtractor.java:135) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:186) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:161) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) > {noformat} > It happens when we try to parse an OLE10 embedded object ... the code > that does this parsing captures and ignores Ole10NativeException and > skips the entry ... so I'm wondering if we should also catch AIOOBE > and skip the entry? Ie, maybe this entry really is not OLE10, and the > Ole10Native code is failing to throw Ole10NativeException for it? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1108) Represent individual slides in pptx
[ https://issues.apache.org/jira/browse/TIKA-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1108: -- Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Represent individual slides in pptx > --- > > Key: TIKA-1108 > URL: https://issues.apache.org/jira/browse/TIKA-1108 > Project: Tika > Issue Type: Improvement > Components: parser >Reporter: Daniel Bonniot de Ruisselet > Fix For: 1.6 > > > When parsing ppt, tika produces for each slide: > > However for pptx these seem to be missing, all the text is directly under > . -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-987) Embedded drawing (SHAPE MERGEFORMAT) sometimes not extracted
[ https://issues.apache.org/jira/browse/TIKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-987: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Embedded drawing (SHAPE MERGEFORMAT) sometimes not extracted > > > Key: TIKA-987 > URL: https://issues.apache.org/jira/browse/TIKA-987 > Project: Tika > Issue Type: Bug > Components: parser >Reporter: Michael McCandless > Fix For: 1.6 > > Attachments: picture.doc, picture_3.doc > > > I have two Word docs, both containing the same drawing, but one has > text added. > In one case (picture.doc) the extraction is correct: it contains only > an embedded image.wmf; when I view the image it's correct. > In the second case (picture_3.doc) the picture is extracted as image > (no extension), and is 0 bytes, and there is an invalid character > (mapped to unicode replacement char) inserted before the image: > {noformat} > > > � > > > vehicle > > {noformat} > (Though, the text "vehicle" is extracted correctly). > I dug a bit, and with the 2nd doc there is an embedded {SHAPE * > MERGEFORMAT} field, which we invoke > WordExtractor.handleSpecialCharacterRuns on, and somehow it extracts > the 0-byte no-extension image as well as the invalid character. With > the first doc there is no field (at least not one that's handle with > handleSpecialCharacterRuns...). Otherwise I'm not sure how to > fix... it could be something is going wrong in how POI parses the > Pictures from PictureSource. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-988) We don't extract a placeholder for a Word document embedded in an Excel document
[ https://issues.apache.org/jira/browse/TIKA-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-988: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > We don't extract a placeholder for a Word document embedded in an Excel > document > > > Key: TIKA-988 > URL: https://issues.apache.org/jira/browse/TIKA-988 > Project: Tika > Issue Type: Improvement > Components: parser >Reporter: Michael McCandless > Fix For: 1.6 > > Attachments: bug31373.xls > > > In TIKA-956 we fixed the Word parser so that at the point where an embedded > document appears, we output a tag. > It would be nice to do this for documents embedded in Excel too. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-980) MicrodataContentHandler for Apache Tika
[ https://issues.apache.org/jira/browse/TIKA-980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-980: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > MicrodataContentHandler for Apache Tika > --- > > Key: TIKA-980 > URL: https://issues.apache.org/jira/browse/TIKA-980 > Project: Tika > Issue Type: New Feature > Components: parser >Reporter: Markus Jelsma >Assignee: Ken Krugler > Fix For: 1.6 > > Attachments: TIKA-980-1.3-1.patch, TIKA-980-1.3-2.patch, > TIKA-980-1.3-3.patch, TIKA-980-1.3-4.patch, TIKA-980-1.3-5.patch > > > ContentHandler for Apache Tika capable of building a data structure > containing Microdata item scopes and item properties. The Item* classes are > borrowed from the Apache Any23 project and are slightly modified to > accomodate this SAX-based extractor vs the original DOM-based extractor. > The provided unit test outputs two item scopes about the Europe and NA > ApacheCon events and each has a nested property. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1106) CLAVIN Integration
[ https://issues.apache.org/jira/browse/TIKA-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1106: -- Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > CLAVIN Integration > -- > > Key: TIKA-1106 > URL: https://issues.apache.org/jira/browse/TIKA-1106 > Project: Tika > Issue Type: Wish > Components: general >Affects Versions: 1.3 > Environment: All >Reporter: Adam Estrada >Priority: Minor > Labels: entity, geospatial > Fix For: 1.6 > > > I've been evaluating CLAVIN as a way to extract location information from > unstructured text. It seems like meshing it with Tika in some way would make > a lot of sense. From CLAVIN website... > {quote} > CLAVIN (*Cartographic Location And Vicinity INdexer*) is an open source > software package for document geotagging and geoparsing that employs > context-based geographic entity resolution. It combines a variety of open > source tools with natural language processing techniques to extract location > names from unstructured text documents and resolve them against gazetteer > records. Importantly, CLAVIN does not simply "look up" location names; > rather, it uses intelligent heuristics in an attempt to identify precisely > which "Springfield" (for example) was intended by the author, based on the > context of the document. CLAVIN also employs fuzzy search to handle > incorrectly-spelled location names, and it recognizes alternative names > (e.g., "Ivory Coast" and "Côte d'Ivoire") as referring to the same geographic > entity. By enriching text documents with structured geo data, CLAVIN enables > hierarchical geospatial search and advanced geospatial analytics on > unstructured data. > {quote} > There was only one other instance of the word "clavin" mentioned in the ASF > jira site so I thought it was definitely worth posting here. > https://github.com/Berico-Technologies/CLAVIN -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1208) Migrate Any23 mime contributions to Tika
[ https://issues.apache.org/jira/browse/TIKA-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1208: -- Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Migrate Any23 mime contributions to Tika > > > Key: TIKA-1208 > URL: https://issues.apache.org/jira/browse/TIKA-1208 > Project: Tika > Issue Type: Sub-task > Components: mime >Reporter: Lewis John McGibbney > Fix For: 1.6 > > Attachments: TIKA-1208.patch > > > We begin with one of the most obvious areas in which there > is overlap. > In short, the appeal of this package is the addition of detection > for the following types: > - text/n3 > - text/rdf+n3 > - application/n3 > - text/x-nquads > - text/rdf+nq > - text/nq > - application/nq > - text/turtle > - application/x-turtle > - application/turtle > - application/trix > > Therefore although both Tika and Any23 execute the task of Mimetype-related > tasks, there is a contribution to be made. This involves the trasferral of > code pertaining to pattern recogition, Mimetype XML defitinions within > tika-mimetypes.xml and a Purifier implementation that removes all > the eventual blank characters at the header of a file that might > prevents its MIME Type detection. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1220) Parser implementration for IFC files
[ https://issues.apache.org/jira/browse/TIKA-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1220: -- Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Parser implementration for IFC files > > > Key: TIKA-1220 > URL: https://issues.apache.org/jira/browse/TIKA-1220 > Project: Tika > Issue Type: New Feature > Components: parser >Reporter: Lewis John McGibbney >Priority: Minor > Fix For: 1.6 > > Attachments: 2012-03-23-Duplex-Programming.ifc > > > The Industry Foundation Classes (IFC) [0] data model is intended to describe > building and construction industry data. For the sake of argument, it can be > considered as a more intelligent successor to the .dwg data models used > within CAD models. > I've tracked down a potential 3rd party library [1] which we maybe able to > wrap and use within Tika however the provided software packages are licensed > under: http://creativecommons.org/licenses/by-nc-sa/3.0/de/ so I am currently > over on legal-discuss@ in an attempt to see if it is possible to wrap some > code and contribute it to tika-parsers. > When I get feedback from legal-discuss, and if this is a go-ahead, I'll need > to help the developers package the code as a Maven artifact(s), then I will > progress with writing the implementation. > [0] http://en.wikipedia.org/wiki/Industry_Foundation_Classes > [1] http://www.ifctoolsproject.com/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-891) Use POST in addition to PUT on method calls in tika-server
[ https://issues.apache.org/jira/browse/TIKA-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-891: - Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Use POST in addition to PUT on method calls in tika-server > -- > > Key: TIKA-891 > URL: https://issues.apache.org/jira/browse/TIKA-891 > Project: Tika > Issue Type: Improvement > Components: general >Reporter: Chris A. Mattmann >Assignee: Chris A. Mattmann >Priority: Trivial > Fix For: 1.6 > > > Per Jukka's email: > http://s.apache.org/uR > It would be a better use of REST/HTTP "verbs" to use POST to put content to a > resource where we don't intend to store that content (which is the > implication of PUT). Max suggested adding: > {code} > @POST > {code} > annotations to the methods we are currently exposing using PUT to take care > of this. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1231) Safely handle null embedded files in PDFs
[ https://issues.apache.org/jira/browse/TIKA-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1231: -- Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Safely handle null embedded files in PDFs > - > > Key: TIKA-1231 > URL: https://issues.apache.org/jira/browse/TIKA-1231 > Project: Tika > Issue Type: Bug > Components: parser >Affects Versions: 1.5 >Reporter: Tim Allison >Assignee: Tim Allison >Priority: Minor > Labels: easyfix > Fix For: 1.6 > > > I filed a potential fix, unit test and test doc for this in PDFBOX-1884. > We'll need to add one test for null in the Tika PDFParser to handle this > change once it is fixed in PDFBox. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (TIKA-973) PDF form data isn't included in extracted content.
[ https://issues.apache.org/jira/browse/TIKA-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle resolved TIKA-973. -- Resolution: Fixed > PDF form data isn't included in extracted content. > -- > > Key: TIKA-973 > URL: https://issues.apache.org/jira/browse/TIKA-973 > Project: Tika > Issue Type: Bug > Components: general >Affects Versions: 1.2 >Reporter: Michael Graessle >Assignee: Tim Allison >Priority: Minor > Fix For: 1.5 > > Attachments: TIKA-973-patch.tar.gz, TIKA-973.patch.tar.gz, > i-9_screenshot.png > > > When extracting content from PDFs, PDF form data isn't extracted. > The following code extracts this data via PDF box, but it seems like > something Tika should be doing. > PDDocumentCatalog docCatalog = load.getDocumentCatalog(); > if (docCatalog != null) { > PDAcroForm acroForm = docCatalog.getAcroForm(); > if (acroForm != null) { > @SuppressWarnings("unchecked") > List fields = acroForm.getFields(); > if (fields != null && fields.size() > 0) { > documentContent.append(" "); > for (PDField field : fields) { > if (field.getValue()!=null) { > documentContent.append(field.getValue()); > documentContent.append(" "); > } > } > } > } > } -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1223) Extract thumbnail of OOXML Office files
[ https://issues.apache.org/jira/browse/TIKA-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1223: -- Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Extract thumbnail of OOXML Office files > --- > > Key: TIKA-1223 > URL: https://issues.apache.org/jira/browse/TIKA-1223 > Project: Tika > Issue Type: Improvement > Components: parser >Affects Versions: 1.4 >Reporter: Hong-Thai Nguyen >Priority: Minor > Fix For: 1.6 > > Attachments: TIKA-1223.patch > > > From Microsoft Office 2007 file formats, thumbnail could be included in > package. We can extract this embedded thumbnail for OOXML files. > As discussed in mailing list, we should extract thumbnail as a attachment, > not as metadata (TIKA-90). > {noformat} > embeddedRelationId format is thumbnail_{i}.{extension}. > {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TIKA-1205) Allow PDFParser to fallback to other parser if there is an exception
[ https://issues.apache.org/jira/browse/TIKA-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Meikle updated TIKA-1205: -- Fix Version/s: (was: 1.5) 1.6 Pushed out to 1.6, preparing for 1.5 RC > Allow PDFParser to fallback to other parser if there is an exception > > > Key: TIKA-1205 > URL: https://issues.apache.org/jira/browse/TIKA-1205 > Project: Tika > Issue Type: Improvement > Components: parser >Reporter: Tim Allison >Assignee: Tim Allison >Priority: Trivial > Fix For: 1.6 > > > With TIKA-1201, there is now an option to use PDFBox's NonSequentialPDFParser > instead of the traditional parser for parsing PDF files. Following the > description in PDFBOX-1199, it would be useful to allow fallback to the > classic parser if NonSequentialPDFParser throws an IOException. For the sake > of symmetry, I propose a boolean useParserFallbackOnException parameter. If > this parameter is true, and if Tika's PDFParser is using the classic parser, > Tika will fallback to the NonSequentialPDFParser if there is an IOException; > if this parameter is true and if Tika's PDFParser is using the > NonSequentialPDFParser it will fallback to the classic parser if there is an > IOException. > Many thanks to Hong-Thai for championing the addition of the added > NonSequentialPDFParser capability in TIKA-1201, and many thanks to Timo for > PDFBox's NonSequentialPDFParser (PDFBOX-1199)! -- This message was sent by Atlassian JIRA (v6.1.5#6160)
buildbot failure in ASF Buildbot on tika-trunk
The Buildbot has detected a new failure on builder tika-trunk while building ASF Buildbot. Full details are available at: http://ci.apache.org/builders/tika-trunk/builds/1151 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: portunus_ubuntu Build Reason: scheduler Build Source Stamp: [branch tika/trunk] 1564580 Blamelist: dmeikle BUILD FAILED: failed compile sincerely, -The Buildbot
buildbot success in ASF Buildbot on tika-trunk
The Buildbot has detected a restored build on builder tika-trunk while building ASF Buildbot. Full details are available at: http://ci.apache.org/builders/tika-trunk/builds/1153 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: portunus_ubuntu Build Reason: scheduler Build Source Stamp: [branch tika/trunk] 1564605 Blamelist: dmeikle Build succeeded! sincerely, -The Buildbot
[VOTE] Apache Tika 1.5 RC1
Hi Guys, A candidate for the Tika 1.5 release is now available at: http://people.apache.org/~dmeikle/tika-1.5-rc1/ The release candidate is a zip archive of the sources in: http://svn.apache.org/repos/asf/tika/tags/1.5-rc1/ The SHA1 checksum of the archive is: 66adb7e73058da73a055a823bd61af48129c1179 A staged M2 repository can also be found on repository.apache.org here: https://repository.apache.org/content/repositories/orgapachetika-1000 Please vote on releasing this package as Apache Tika 1.5. The vote is open for the next 72 hours and passes if a majority of at least three +1 Tika PMC votes are cast. [ ] +1 Release this package as Apache Tika 1.5 [ ] -1 Do not release this package because... Here is my +1 for the release. Cheers, Dave
Unsubscribe?
Can someone unsubscribe or tell me how to do this from this Tika emai list? > Date: Tue, 4 Feb 2014 23:18:18 + > From: j...@apache.org > To: dev@tika.apache.org > Subject: [jira] [Updated] (TIKA-1205) Allow PDFParser to fallback to other > parser if there is an exception > > > [ > https://issues.apache.org/jira/browse/TIKA-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > ] > > Dave Meikle updated TIKA-1205: > -- > > Fix Version/s: (was: 1.5) >1.6 > > Pushed out to 1.6, preparing for 1.5 RC > > > Allow PDFParser to fallback to other parser if there is an exception > > > > > > Key: TIKA-1205 > > URL: https://issues.apache.org/jira/browse/TIKA-1205 > > Project: Tika > > Issue Type: Improvement > > Components: parser > >Reporter: Tim Allison > >Assignee: Tim Allison > >Priority: Trivial > > Fix For: 1.6 > > > > > > With TIKA-1201, there is now an option to use PDFBox's > > NonSequentialPDFParser instead of the traditional parser for parsing PDF > > files. Following the description in PDFBOX-1199, it would be useful to > > allow fallback to the classic parser if NonSequentialPDFParser throws an > > IOException. For the sake of symmetry, I propose a boolean > > useParserFallbackOnException parameter. If this parameter is true, and if > > Tika's PDFParser is using the classic parser, Tika will fallback to the > > NonSequentialPDFParser if there is an IOException; if this parameter is > > true and if Tika's PDFParser is using the NonSequentialPDFParser it will > > fallback to the classic parser if there is an IOException. > > Many thanks to Hong-Thai for championing the addition of the added > > NonSequentialPDFParser capability in TIKA-1201, and many thanks to Timo for > > PDFBox's NonSequentialPDFParser (PDFBOX-1199)! > > > > -- > This message was sent by Atlassian JIRA > (v6.1.5#6160)
Re: [VOTE] Apache Tika 1.5 RC1
Hi Dave, You ROCK! I cannot verify the SIGS though: [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% $HOME/bin/stage_apache_rc tika 1.5-src http://people.apache.org/~dmeikle/tika-1.5-rc1/ % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 41.3M 100 41.3M0 0 450k 0 0:01:33 0:01:33 --:--:-- 609k % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 819 100 8190 0 3854 0 --:--:-- --:--:-- --:--:-- 7247 % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 10033 100330 0147 0 --:--:-- --:--:-- --:--:-- 300 [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% $HOME/bin/verify_gpg_sigs Verifying Signature for file tika-1.5-src.zip.asc gpg: Signature made Tue Feb 4 20:34:06 2014 EST using RSA key ID 0EB30B07 gpg: Can't check signature: No public key [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% curl -O http://people.apache.org/~dmeikle/tika-1.5-rc1/KEYS % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 9258 100 92580 0 3801 0 0:00:02 0:00:02 --:--:-- 4638 [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% gpg --import < KEYS gpg: key A355A63E: "Jukka Zitting " not changed gpg: key B876884A: "Chris Mattmann (CODE SIGNING KEY) " not changed gpg: key 9740DD55: "David Meikle (CODE SIGNING KEY) " not changed gpg: Total number processed: 3 gpg: unchanged: 3 [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% $HOME/bin/verify_gpg_sigs Verifying Signature for file tika-1.5-src.zip.asc gpg: Signature made Tue Feb 4 20:34:06 2014 EST using RSA key ID 0EB30B07 gpg: Can't check signature: No public key [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% $HOME/bin/verify_md5_checksums md5sum: stat '*.tar.gz': No such file or directory md5sum: stat '*.bz2': No such file or directory md5sum: stat '*.tgz': No such file or directory tika-1.5-src.zip: OK [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% Checksums check out. Can you scope the SIGS problem? Cheers, Chris -Original Message- From: David Meikle Reply-To: "dev@tika.apache.org" Date: Tuesday, February 4, 2014 5:59 PM To: "dev@tika.apache.org" Cc: "u...@tika.apache.org" Subject: [VOTE] Apache Tika 1.5 RC1 >Hi Guys, > >A candidate for the Tika 1.5 release is now available at: >http://people.apache.org/~dmeikle/tika-1.5-rc1/ > >The release candidate is a zip archive of the sources in: >http://svn.apache.org/repos/asf/tika/tags/1.5-rc1/ > >The SHA1 checksum of the archive is: >66adb7e73058da73a055a823bd61af48129c1179 > >A staged M2 repository can also be found on repository.apache.org here: >https://repository.apache.org/content/repositories/orgapachetika-1000 > >Please vote on releasing this package as Apache Tika 1.5. >The vote is open for the next 72 hours and passes if a majority of at >least three +1 Tika PMC votes are cast. > > [ ] +1 Release this package as Apache Tika 1.5 > [ ] -1 Do not release this package because... > >Here is my +1 for the release. > >Cheers, >Dave
Re: [VOTE] Apache Tika 1.5 RC1
OK worked this time grabbing tika.asc. Used it to validate GPG and looks good. Should probably either remove the KEYS file and simply point to [1] though. Cheers and +1 from me! Cheers, Chris [1] https://people.apache.org/keys/group/tika.asc [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% curl -O https://people.apache.org/keys/group/tika.asc % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 133k 100 133k0 0 55054 0 0:00:02 0:00:02 --:--:-- 58432 [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% gpg --import < tika.asc gpg: key B876884A: "Chris Mattmann (CODE SIGNING KEY) " not changed gpg: key A355A63E: "Jukka Zitting " 7 new signatures gpg: key 8A26D9A6: public key "Jukka Zitting " imported gpg: key 42CFAE07: public key "Jukka Zitting (CODE SIGNING KEY) " imported gpg: key 0EB30B07: public key "David Meikle (CODE SIGNING KEY) " imported gpg: key D84E41AE: public key "Nick Burch " imported gpg: key E7AC2BA5: public key "Oleg Tikhonov " imported gpg: key 6E68DA61: public key "Michael McCandless (CODE SIGNING KEY) " imported gpg: key 95D21F2E: public key "Ray Gauss II (CODE SIGNING KEY) " imported gpg: key DEDEAB92: public key "Sergey Beryozkin (Release Management) " imported gpg: key 97EDDE66: public key "tallison (apache_distro_keys) " imported gpg: Total number processed: 11 gpg: imported: 9 (RSA: 6) gpg: unchanged: 1 gpg: new signatures: 7 gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 3 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 3u gpg: next trustdb check due at 2015-08-18 [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% $HOME/bin/verify_gpg_sigs Verifying Signature for file tika-1.5-src.zip.asc gpg: Signature made Tue Feb 4 20:34:06 2014 EST using RSA key ID 0EB30B07 gpg: Good signature from "David Meikle (CODE SIGNING KEY) " gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: F3F2 3C1E DB33 8077 254E DEEC 5241 4B0B 0EB3 0B07 [chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% -Original Message- From: Chris Mattmann Reply-To: "dev@tika.apache.org" Date: Tuesday, February 4, 2014 10:48 PM To: "dev@tika.apache.org" Cc: "u...@tika.apache.org" Subject: Re: [VOTE] Apache Tika 1.5 RC1 >Hi Dave, > >You ROCK! > >I cannot verify the SIGS though: > >[chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% $HOME/bin/stage_apache_rc >tika 1.5-src http://people.apache.org/~dmeikle/tika-1.5-rc1/ > % Total% Received % Xferd Average Speed TimeTime Time >Current > Dload Upload Total SpentLeft >Speed >100 41.3M 100 41.3M0 0 450k 0 0:01:33 0:01:33 --:--:-- >609k > % Total% Received % Xferd Average Speed TimeTime Time >Current > Dload Upload Total SpentLeft >Speed >100 819 100 8190 0 3854 0 --:--:-- --:--:-- --:--:-- >7247 > % Total% Received % Xferd Average Speed TimeTime Time >Current > Dload Upload Total SpentLeft >Speed >10033 100330 0147 0 --:--:-- --:--:-- --:--:-- >300 >[chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% $HOME/bin/verify_gpg_sigs >Verifying Signature for file tika-1.5-src.zip.asc >gpg: Signature made Tue Feb 4 20:34:06 2014 EST using RSA key ID 0EB30B07 >gpg: Can't check signature: No public key >[chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% curl -O >http://people.apache.org/~dmeikle/tika-1.5-rc1/KEYS > % Total% Received % Xferd Average Speed TimeTime Time >Current > Dload Upload Total SpentLeft >Speed >100 9258 100 92580 0 3801 0 0:00:02 0:00:02 --:--:-- >4638 >[chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% gpg --import < KEYS >gpg: key A355A63E: "Jukka Zitting " not changed >gpg: key B876884A: "Chris Mattmann (CODE SIGNING KEY) >" not changed >gpg: key 9740DD55: "David Meikle (CODE SIGNING KEY) " >not changed >gpg: Total number processed: 3 >gpg: unchanged: 3 >[chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% $HOME/bin/verify_gpg_sigs >Verifying Signature for file tika-1.5-src.zip.asc >gpg: Signature made Tue Feb 4 20:34:06 2014 EST using RSA key ID 0EB30B07 >gpg: Can't check signature: No public key >[chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% >$HOME/bin/verify_md5_checksums >md5sum: stat '*.tar.gz': No such file or directory >md5sum: stat '*.bz2': No such file or directory >md5sum: stat '*.tgz': No such file or directory >tika-1.5-src.zip: OK >[chipotle:~/tmp/apache-tika-1.5-rc1] mattmann% > > >Checksums check out. > >Can you scope the SIGS problem? > >Cheers, >Chris > > > >-Original Message- >From: David Meikle >Reply-To: "dev@tika.apache.org" >Da
Re: [VOTE] Apache Tika 1.5 RC1
Hi David, [x] +1 Release this package as Apache Tika 1.5 Thanks! BR, Oleg On Wed, Feb 5, 2014 at 3:59 AM, David Meikle wrote: > Hi Guys, > > A candidate for the Tika 1.5 release is now available at: > http://people.apache.org/~dmeikle/tika-1.5-rc1/ > > The release candidate is a zip archive of the sources in: > http://svn.apache.org/repos/asf/tika/tags/1.5-rc1/ > > The SHA1 checksum of the archive is: > 66adb7e73058da73a055a823bd61af48129c1179 > > A staged M2 repository can also be found on repository.apache.org here: > https://repository.apache.org/content/repositories/orgapachetika-1000 > > Please vote on releasing this package as Apache Tika 1.5. > The vote is open for the next 72 hours and passes if a majority of at > least three +1 Tika PMC votes are cast. > >[ ] +1 Release this package as Apache Tika 1.5 >[ ] -1 Do not release this package because... > > Here is my +1 for the release. > > Cheers, > Dave