[ https://issues.apache.org/jira/browse/TIKA-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612321#comment-14612321 ]
Jeremy B. Merrill commented on TIKA-1602: ----------------------------------------- Thank you, [~chrismattmann], [~talli...@mitre.org] et al.! [~talli...@mitre.org] -- got a bunch of normal headers, but also this `Status:` one. The only possible value in my dataset (a bunch of publicly-released emails from Jeb Bush's tenure as FL Gov) is `RO`, so the first lines of the emails who were treated improperly by Tika before this patch was uniformly `Status: RO`. I'm going to check the whole dataset once I manage to download it all back down again from storage to make sure there are no other values than `RO`. My understanding is that some mail servers use this header internally to keep track of read status. When emails are exported, they retain the header, and it sometimes appears first -- even though the server would never send this header over the wire. > Detecting standards-non-compliant emails as message/rfc822 > ---------------------------------------------------------- > > Key: TIKA-1602 > URL: https://issues.apache.org/jira/browse/TIKA-1602 > Project: Tika > Issue Type: New Feature > Components: mime > Reporter: Jeremy B. Merrill > Assignee: Chris A. Mattmann > Priority: Minor > Fix For: 1.10 > > Attachments: 036491.txt.zip > > Original Estimate: 1h > Remaining Estimate: 1h > > Tika does not properly detect certain emails as `message/rfc822` if they're > slightly standards-non-compliant and begin with `Status: ` as the first > header. I've added `Status: ` as a magic detection line in > tika-mimetypes.xml. > This solves my problem and does not appear to cause unit test failures. I > have not yet run the tika-batch tests. > As further information, the emails that are processed incorrectly come from > dumps directly from various US public officials' mailservers. The dumps, I > believe since they're not intended to be transmitted over the wire, sometimes > are slightly non-compliant. > It's important to note that Tika (and the underlying library, James Mime4J) > do properly *parse* these emails, despite the non-compliant header. The > problem is getting Tika to *detect* the file as an email so that Mime4J gets > chosen to parse it. > Pull request on Github at https://github.com/apache/tika/pull/40 -- This message was sent by Atlassian JIRA (v6.3.4#6332)