[jira] [Comment Edited] (TIKA-879) Detection problem: message/rfc822 file is detected as text/plain.

2015-04-21 Thread Tim Allison (JIRA) 

[ 
https://issues.apache.org/jira/browse/TIKA-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505269#comment-14505269
 ] 

Tim Allison edited comment on TIKA-879 at 4/21/15 5:04 PM:
---

Y, will do. Results probably tomorrow.

This?
mime-type type=message/rfc822
magic priority=50
  match value=Relay-Version: type=string offset=0/
  match value=#!\ rnews type=string offset=0/
  match value=N#!\ rnews type=string offset=0/
  match value=Forward\ to type=string offset=0/
  match value=Pipe\ to type=string offset=0/
  match value=Return-Path: type=string offset=0/
  match value=\nReturn-Path: type=string offset=0:1000/
 match value=From: type=string offset=0/
  match value=Received: type=string offset=0/
 match value=\nReceived: type=string offset=0:1000/
 match value=Message-ID: type=string offset=0/
  match value=\nMessage-ID: type=string offset=0:1000/
 match value=Date: type=string offset=0/
/magic
glob pattern=*.eml/
glob pattern=*.mime/
glob pattern=*.mht/
glob pattern=*.mhtml/
sub-class-of type=text/plain/
  /mime-type



was (Author: talli...@mitre.org):
Y, will do. Results probably tomorrow.

 Detection problem: message/rfc822 file is detected as text/plain.
 -

 Key: TIKA-879
 URL: https://issues.apache.org/jira/browse/TIKA-879
 Project: Tika
  Issue Type: Bug
  Components: metadata, mime
Affects Versions: 1.0, 1.1, 1.2
 Environment: linux 3.2.9
 oracle jdk7, openjdk7, sun jdk6
Reporter: Konstantin Gribov
  Labels: new-parser
 Attachments: TIKA-879-thunderbird.eml


 When using {{DefaultDetector}} mime type for {{.eml}} files is different (you 
 can test it on {{testRFC822}} and {{testRFC822_base64}} in 
 {{tika-parsers/src/test/resources/test-documents/}}).
 Main reason for such behavior is that only magic detector is really works for 
 such files. Even if you set {{CONTENT_TYPE}} in metadata or some {{.eml}} 
 file name in {{RESOURCE_NAME_KEY}}.
 As I found {{MediaTypeRegistry.isSpecializationOf(message/rfc822, 
 text/plain)}} returns {{false}}, so detection by {{MimeTypes.detect(...)}} 
 works only by magic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (TIKA-879) Detection problem: message/rfc822 file is detected as text/plain.

2012-11-21 Thread Vladimir L. (JIRA) 

[ 
https://issues.apache.org/jira/browse/TIKA-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502287#comment-13502287
 ] 

Vladimir L. edited comment on TIKA-879 at 11/21/12 8:33 PM:


I've got the same problem: trying to parse Outlook Express file: *.eml and get 
default text/plain Content-Type instead of expected message/rfc822

I'm using {{org.apache.tika.parser.AutoDetectParser}} with default settings and 
during debugging came to the same conclusion as a reporter of this issue: 
MediaTypeRegistry.isSpecializationOf(message/rfc822, text/plain) returns 
false.

If there is a way to vote for this bug to be fixed, or easy work around, please 
share it with us!

  was (Author: vladimir_l):
I've got the same problem: trying to parse Outlook Express file: *.eml and 
get default text/plain Content-Type instead of expected message/rfc822

I'm using preorg.apache.tika.parser.AutoDetectParser/pre with default 
settings and during debugging came to the same conclusion as a reporter of this 
issue: MediaTypeRegistry.isSpecializationOf(message/rfc822, text/plain) 
returns false.

If there is a way to vote for this bug to be fixed, or easy work around, please 
share it with us!
  
 Detection problem: message/rfc822 file is detected as text/plain.
 -

 Key: TIKA-879
 URL: https://issues.apache.org/jira/browse/TIKA-879
 Project: Tika
  Issue Type: Bug
  Components: metadata, mime
Affects Versions: 1.0, 1.1, 1.2
 Environment: linux 3.2.9
 oracle jdk7, openjdk7, sun jdk6
Reporter: Konstantin Gribov

 When using {{DefaultDetector}} mime type for {{.eml}} files is different (you 
 can test it on {{testRFC822}} and {{testRFC822_base64}} in 
 {{tika-parsers/src/test/resources/test-documents/}}).
 Main reason for such behavior is that only magic detector is really works for 
 such files. Even if you set {{CONTENT_TYPE}} in metadata or some {{.eml}} 
 file name in {{RESOURCE_NAME_KEY}}.
 As I found {{MediaTypeRegistry.isSpecializationOf(message/rfc822, 
 text/plain)}} returns {{false}}, so detection by {{MimeTypes.detect(...)}} 
 works only by magic.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (TIKA-879) Detection problem: message/rfc822 file is detected as text/plain.

2012-11-21 Thread Vladimir L. (JIRA) 

[ 
https://issues.apache.org/jira/browse/TIKA-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502393#comment-13502393
 ] 

Vladimir L. edited comment on TIKA-879 at 11/21/12 10:57 PM:
-

I figured out where the root of problem lays on.
The original Tika configuration for message/rfc822 is following:
{code:xml}
  mime-type type=message/rfc822
magic priority=50
  match value=Relay-Version: type=string offset=0/
  match value=#!\ rnews type=string offset=0/
  match value=N#!\ rnews type=string offset=0/
  match value=Forward\ to type=string offset=0/
  match value=Pipe\ to type=string offset=0/
  match value=Return-Path: type=string offset=0/
  match value=From: type=string offset=0/
  match value=Received: type=string offset=0/
  match type=string value=Message-ID: offset=0/
  match type=string value=Date: offset=0/
/magic
glob pattern=*.eml/
glob pattern=*.mime/
glob pattern=*.mht/
glob pattern=*.mhtml/
  /mime-type
{code}

Unfortunately the e-mail message I'm testing with has following header:
{code}
x-store-info:J++/JTCzmObr++wNraA4 .
Authentication-Results: something.com; sender-id= ..
X-SID-PRA: vladimi...@example.com
X-SID-Result: Pass
X-DKIM-Result: None
X-AUTH-Result: PASS
X-Message-Status: n:n
X-Message-Delivery: Vj0xLjE7dXM .
X-Message-Info: aKlYzGSc+Ll01bU5 
Received: from mailout- 
Received: (qmail invoked by alias); 21 Nov 2012 20:11:35 -
Received: from mp017. 
X-Authenticated: #2407 
X-Provags-ID: V01U2FsdGVkX 
Received: (qmail 22194 invoked by uid 0); 21 Nov 2012 20:11:34 -
Received: from 
Content-Type: text/plain; charset=utf-8
Date: Wed, 21 Nov 2012 21:11:32 +0100
From: Vladimir L. vladimi...@example.com
Message-ID: 20121121201132.74...@example.com
MIME-Version: 1.0
Subject: JUnit test message
To: vladimi...@something.com
X-Flags: 0001
X-Mailer: WWW-Mail 6100 (Global Message Exchange)
X-Priority: 3
Content-Transfer-Encoding: 8bit
Return-Path: vladimi...@example.com
X-OriginalArrivalTime: 21 Nov 2012 20:11:36.0285 

Dear Vladimir .
{code}

As you can see none of the mentioned patterns is matching since they are all 
configured with offset=0
However the e-mail header defines the Content-Type: text/plain, which i assume 
influence the initial content type detection.
The {{sub-class-of type=text/plain/}} is not defined in mime-type 
definition, therefore auto-detection via extension *.eml fails for 
aforementioned reason of this issue.

The current workaround for me is following:
1. Create {{custom-mimetypes.xml}} as described here: 
[http://tika.apache.org/1.0/parser_guide.html#Add_your_MIME-Type]
2. Add redefinition for message/rfc822 mime-type as following:
{code:xml}
  mime-type type=message/rfc822
magic priority=50
  match value=Relay-Version: type=string offset=0/
  match value=#!\ rnews type=string offset=0/
  match value=N#!\ rnews type=string offset=0/
  match value=Forward\ to type=string offset=0/
  match value=Pipe\ to type=string offset=0/
  match value=Return-Path: type=string offset=0:2000/
  match value=From: type=string offset=0/
  match value=Received: type=string offset=0:2000/
  match value=Message-ID: type=string offset=0:2000/
  match value=Date: type=string offset=0/
/magic
glob pattern=*.eml/
glob pattern=*.mime/
glob pattern=*.mht/
glob pattern=*.mhtml/
sub-class-of type=text/plain/
  /mime-type
{code}

Note the offset settings for *Message-ID:*, *Return-Path:*, and *Received:*
I decided to leave fall-back to extension detection through definition of 
super-class {{text/plain}}

Hope this will help you to go around this issue too.

Good luck,
vladimir

  was (Author: vladimir_l):
I figured out where the root of problem lays on.
The original Tika configuration for message/rfc822 is following:
{code:xml}
  mime-type type=message/rfc822
magic priority=50
  match value=Relay-Version: type=string offset=0/
  match value=#!\ rnews type=string offset=0/
  match value=N#!\ rnews type=string offset=0/
  match value=Forward\ to type=string offset=0/
  match value=Pipe\ to type=string offset=0/
  match value=Return-Path: type=string offset=0/
  match value=From: type=string offset=0/
  match value=Received: type=string offset=0/
  match type=string value=Message-ID: offset=0/
  match type=string value=Date: offset=0/
/magic
glob pattern=*.eml/
glob pattern=*.mime/
glob pattern=*.mht/
glob pattern=*.mhtml/
  /mime-type
{code}

Unfortunately the e-mail message I'm testing with has following header:
{code}
x-store-info:J++/JTCzmObr++wNraA4 .
Authentication-Results: something.com; sender-id= ..
X-SID-PRA: vladimi...@example.com
X-SID-Result: Pass
X-DKIM-Result: None
X-AUTH-Result: PASS
X-Message-Status: