[ https://issues.apache.org/jira/browse/NUTCH-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158397#comment-13158397 ]
Julien Nioche commented on NUTCH-1206: -------------------------------------- With trunk and httpclient enabled <property> <name>plugin.includes</name> <value>protocol-httpclient|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value> </property> ./nutch parsechecker -dumpText https://issues.apache.org/jira/secure/attachment/12505323/direct.pdf {noformat} Url --------------- https://issues.apache.org/jira/secure/attachment/12505323/direct.pdf --------- ParseData --------- Version: 5 Status: success(1,0) Title: E-age Banking Channels Outlinks: 0 Content Metadata: Date=Mon, 28 Nov 2011 12:52:52 GMT Content-Length=26013 Content-Disposition=inline; filename*=utf-8''direct.pdf; Set-Cookie=atlassian.xsrf.token=A5KQ-2QAV-T4JA-FDED|ef6e4a4b644ef557205c91bc9b1e392746848171|lout; Path=/jira Connection=close Content-Type=application/pdf;charset=utf-8 X-AUSERNAME=anonymous X-AREQUESTID=772x3514779x2 Server=Apache-Coyote/1.1 Parse Metadata: xmpTPg:NPages=2 Creation-Date=2002-06-09T11:44:37Z created=Sun Jun 09 12:44:37 BST 2002 subject=E-age Banking Channels Author=Shivram producer=Acrobat Distiller 3.0 for Windows Last-Modified=2002-02-17T10:57:57Z Content-Type=application/pdf Keywords= creator=Adobe PageMaker 6.5 --------- ParseText --------- E-age Banking Channels Application form for e-Age Banking Channels (Individuals/Sole Proprietorship)We understand your world SOURCE CODE : NetBanking / WAP Yes, I wish to apply for NetBanking/WAP (Please tick) E-MAIL ID : MobileBanking using SMS Yes, I wish to apply for MobileBanking (Please tick) CELLULAR NO. NAME OF CELLULAR SERVICE PROVIDER City PhoneBanking Yes, I wish to apply for PhoneBanking (Please tick) YOUR MOTHER’S MAIDEN {noformat} i.e there is nothing wrong with the parsing of this document. There must be something wrong in your config or test code. Use the command above instead > tika parser of nutch 1.3 is failing to prcess pdfs > -------------------------------------------------- > > Key: NUTCH-1206 > URL: https://issues.apache.org/jira/browse/NUTCH-1206 > Project: Nutch > Issue Type: Bug > Components: parser > Affects Versions: 1.3 > Environment: Solaris/Linux/Windows > Reporter: dibyendu ghosh > Assignee: Chris A. Mattmann > Attachments: direct.pdf > > > Please refer to this message: > http://www.mail-archive.com/user%40nutch.apache.org/msg04315.html. Old > parse-pdf parser seems to be able to parse old pdfs (checked with nutch 1.2) > though it is not able to parse acrobat 9.0 version of pdfs. nutch 1.3 does > not have parse-pdf plugin and it is not able to parse even older pdfs. > my code (TestParse.java): > ---------------------------- > bash-2.00$ cat TestParse.java > import java.io.File; > import java.io.FileInputStream; > import java.io.FileOutputStream; > import java.io.PrintStream; > import java.util.Iterator; > import java.util.Map; > import java.util.Map.Entry; > import org.apache.hadoop.conf.Configuration; > import org.apache.hadoop.io.Text; > import org.apache.nutch.metadata.Metadata; > import org.apache.nutch.parse.ParseResult; > import org.apache.nutch.parse.Parse; > import org.apache.nutch.parse.ParseStatus; > import org.apache.nutch.parse.ParseUtil; > import org.apache.nutch.parse.ParseData; > import org.apache.nutch.protocol.Content; > import org.apache.nutch.util.NutchConfiguration; > public class TestParse { > private static Configuration conf = NutchConfiguration.create(); > public TestParse() { > } > public static void main(String[] args) { > String filename = args[0]; > convert(filename); > } > public static String convert(String fileName) { > String newName = "abc.html"; > try { > System.out.println("Converting " + fileName + " to html."); > if (convertToHtml(fileName, newName)) > return newName; > } catch (Exception e) { > (new File(newName)).delete(); > System.out.println("General exception " + e.getMessage()); > } > return null; > } > private static boolean convertToHtml(String fileName, String newName) > throws Exception { > // Read the file > FileInputStream in = new FileInputStream(fileName); > byte[] buf = new byte[in.available()]; > in.read(buf); > in.close(); > // Parse the file > Content content = new Content("file:" + fileName, "file:" + > fileName, > buf, "", new Metadata(), conf); > ParseResult parseResult = new ParseUtil(conf).parse(content); > parseResult.filter(); > if (parseResult.isEmpty()) { > System.out.println("All parsing attempts failed"); > return false; > } > Iterator<Map.Entry<Text,Parse>> iterator = > parseResult.iterator(); > if (iterator == null) { > System.out.println("Cannot iterate over successful parse > results"); > return false; > } > Parse parse = null; > ParseData parseData = null; > while (iterator.hasNext()) { > parse = parseResult.get((Text)iterator.next().getKey()); > parseData = parse.getData(); > ParseStatus status = parseData.getStatus(); > // If Parse failed then bail > if (!ParseStatus.STATUS_SUCCESS.equals(status)) { > System.out.println("Could not parse " + fileName + ". " + > status.getMessage()); > return false; > } > } > // Start writing to newName > FileOutputStream fout = new FileOutputStream(newName); > PrintStream out = new PrintStream(fout, true, "UTF-8"); > // Start Document > out.println("<html>"); > // Start Header > out.println("<head>"); > // Write Title > String title = parseData.getTitle(); > if (title != null && title.trim().length() > 0) { > out.println("<title>" + parseData.getTitle() + "</title>"); > } > // Write out Meta tags > Metadata metaData = parseData.getContentMeta(); > String[] names = metaData.names(); > for (String name : names) { > String[] subvalues = metaData.getValues(name); > String values = null; > for (String subvalue : subvalues) { > values += subvalue; > } > if (values.length() > 0) > out.printf("<meta name=\"%s\" content=\"%s\"/>\n", > name, values); > } > out.println("<meta http-equiv=\"Content-Type\" > content=\"text/html;charset=UTF-8\"/>"); > // End Meta tags > out.println("</head>"); // End Header > // Start Body > out.println("<body>"); > out.print(parse.getText()); > out.println("</body>"); // End Body > out.println("</html>"); // End Document > out.close(); // Close the file > return true; > } > } > ---------------------------- > command: > ====== > bash-2.00$ java -classpath > conf:runtime/local/lib/nutch-1.3.jar:runtime/local/lib/hadoop-core-0.20.2.jar:runtime/local/lib/commons-logging-api-1.0.4.jar:runtime/local/lib/tika-core-0.9.jar:runtime/local/lib/log4j-1.2.15.jar:runtime/local/lib/oro-2.0.8.jar:. > TestParse direct.pdf > ====== > output: > _____ > Converting direct.pdf to html. > Oct 19, 2011 5:05:19 PM org.apache.hadoop.conf.Configuration > getConfResourceAsInputStream > INFO: found resource tika-mimetypes.xml at > file:/path/to/nutch/1.3/conf/tika-mimetypes.xml > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginManifestParser > parsePluginFolder > INFO: Plugins: looking in: /path/to/nutch/1.3/runtime/local/plugins > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Plugin Auto-activation mode: [true] > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Registered Plugins: > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: the nutch core extension points (nutch-extensionpoints) > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Tika Parser Plug-in (parse-tika) > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Registered Extension-Points: > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Nutch URL Normalizer > (org.apache.nutch.net.URLNormalizer) > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Nutch Protocol (org.apache.nutch.protocol.Protocol) > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Nutch Segment Merge Filter > (org.apache.nutch.segment.SegmentMergeFilter) > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Nutch URL Filter (org.apache.nutch.net.URLFilter) > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Nutch Indexing Filter > (org.apache.nutch.indexer.IndexingFilter) > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: HTML Parse Filter > (org.apache.nutch.parse.HtmlParseFilter) > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Nutch Content Parser (org.apache.nutch.parse.Parser) > Oct 19, 2011 5:05:20 PM org.apache.nutch.plugin.PluginRepository > displayStatusINFO: Nutch Scoring (org.apache.nutch.scoring.ScoringFilter) > Oct 19, 2011 5:05:20 PM org.apache.hadoop.conf.Configuration > getConfResourceAsInputStream > INFO: found resource parse-plugins.xml at > file:/path/to/nutch/1.3/conf/parse-plugins.xml > Oct 19, 2011 5:05:20 PM org.apache.nutch.parse.ParserFactory matchExtensions > INFO: The parsing plugins: [org.apache.nutch.parse.tika.TikaParser] are > enabled via the plugin.includes system property, and all claim to support > the content type application/pdf, but they are not mapped to it in the > parse-plugins.xml file > Oct 19, 2011 5:05:21 PM org.apache.nutch.parse.ParseUtil parse > WARNING: Unable to successfully parse content file:direct.pdf of type > application/pdf > Oct 19, 2011 5:05:21 PM org.apache.nutch.parse.ParseResult filter > WARNING: file:direct.pdf is not parsed successfully, filtering > All parsing attempts failed > _____ > my customized nutch-site.xml: > ~~~~~~~~~~~~~~~~~~~~ > bash-2.00$ cat conf/nutch-site.xml > <?xml version="1.0"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > <configuration> > <property> > <name>plugin.folders</name> > <value>runtime/local/plugins</value> > <description>Directories where nutch plugins are located. Each > element may be a relative or absolute path. If absolute, it is used > as is. If relative, it is searched for on the classpath.</description> > </property> > <property> > <name>plugin.includes</name> > <value>parse-tika</value> > <description>Regular expression naming plugin directory names to > include. Any plugin not matching this expression is excluded. > </description> > </property> > </configuration> > ~~~~~~~~~~~~~~~~~~~~ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira