[ https://issues.apache.org/jira/browse/TIKA-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397511#comment-17397511 ]
Hudson commented on TIKA-3489: ------------------------------ UNSTABLE: Integrated in Jenkins build Tika ยป tika-branch1x-jdk8 #144 (See [https://ci-builds.apache.org/job/Tika/job/tika-branch1x-jdk8/144/]) TIKA-3489 -- detect robots.txt files as text/x-robots (tallison: [https://github.com/apache/tika/commit/27e7eac5fc7c2122076237c191a2bd0aa2748aa4]) * (add) tika-parsers/src/test/resources/test-documents/testRobots.txt * (edit) tika-parsers/src/test/java/org/apache/tika/mime/TestMimeTypes.java * (edit) tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml > Robots.txt files frequently identified as message/rfc822 > -------------------------------------------------------- > > Key: TIKA-3489 > URL: https://issues.apache.org/jira/browse/TIKA-3489 > Project: Tika > Issue Type: Bug > Components: mime > Affects Versions: 2.0.0, 1.25, 1.26, 1.27 > Reporter: Sebastian Nagel > Assignee: Tim Allison > Priority: Minor > Fix For: 2.0.1, 1.27.1 > > Attachments: robots.txt > > > The Tika MIME detector recognizes a robots.txt file as "message/rfc822" if > the file starts with a "User-Agent" rule and contains also a second rule not > too far away from the beginning, e.g.: > {noformat} > User-Agent: goodbot > Disallow: > User-Agent: badbot > Disallow: / > {noformat} > The change > [7769a2b|https://github.com/apache/tika/commit/7769a2b4fba2b4af7127eba0c7694f663fd97a13] > requires that two different clauses are matched. However, the two > occurrences of "User-Agent:" (initial and after a new line) are treated as > different instead of equivalent matches. -- This message was sent by Atlassian Jira (v8.3.4#803005)