[
https://issues.apache.org/jira/browse/TIKA-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17385514#comment-17385514
]
Nick Burch commented on TIKA-3489:
----------------------------------
I'm not keen on us throwing away information we can easily return to users
A subtype of {{text/plain}} with a more specific set of information feels the
right way to go to me. I assume that the draft uses {{text/plain}} for
compatibility with webservers which don't properly / at all support more
specific types
Most people who want to handle robots files differently are going to need a way
to know that it is a robots file, so won't be helped by {{text/plain}}. Most
people handling "all texty stuff" will just take any {{text/}} so won't care
about the subtype. Something like {{text/plain; format=robots.txt}} feels
possible but not ideal. {{text/x-robots}} seems a bit more preferable to me
> Robots.txt files frequently identified as message/rfc822
> --------------------------------------------------------
>
> Key: TIKA-3489
> URL: https://issues.apache.org/jira/browse/TIKA-3489
> Project: Tika
> Issue Type: Bug
> Components: mime
> Affects Versions: 2.0.0, 1.25, 1.26, 1.27
> Reporter: Sebastian Nagel
> Priority: Minor
> Attachments: robots.txt
>
>
> The Tika MIME detector recognizes a robots.txt file as "message/rfc822" if
> the file starts with a "User-Agent" rule and contains also a second rule not
> too far away from the beginning, e.g.:
> {noformat}
> User-Agent: goodbot
> Disallow:
> User-Agent: badbot
> Disallow: /
> {noformat}
> The change
> [7769a2b|https://github.com/apache/tika/commit/7769a2b4fba2b4af7127eba0c7694f663fd97a13]
> requires that two different clauses are matched. However, the two
> occurrences of "User-Agent:" (initial and after a new line) are treated as
> different instead of equivalent matches.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
