On 7/25/07, Robert Young <[EMAIL PROTECTED]> wrote: > The message which was appearing in the logs is pasted below. > > Basically, in org.apache.nutch.crawl.MapWritable#getKeyValueEntry the > Writable is instantiated. It's class is determined by a two byte code > (which is written to crawldb I guess), if there is no entry for the > class it fails to create it, regardless of if it's a Writable. You're > right in that it can potentially handle any Writabl object, but only > if it has a maping for it's class.
If you add a writable that does not have a mapping, MapWritable automatically creates it and then stores the mapping internally. When a MapWritable is written, any new mapping is also written (as a byte and the corresponding class name). So, when you read a MapWritable it first reads all the mappings (that are not already statically defined) then proceeds to reading the <Writable,Writable> map. So I think your problem is caused by something else (perhaps, there is a bug in MapWritable's implementation but that is what that code is trying to do). Also, replying to JIRA generated emails does not add comments to issues (despite what the email is saying). So please use JIRA to reply. > > Cheers > Rob > > 07/07/25 11:52:00 WARN crawl.MapWritable: Unable to load meta data > entry, ignoring.. : java.io.IOException: unable to load class for id: > 36 > > On 7/25/07, Doğacan Güney (JIRA) <[EMAIL PROTECTED]> wrote: > > > > [ > > https://issues.apache.org/jira/browse/NUTCH-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515283 > > ] > > > > Doğacan Güney commented on NUTCH-527: > > ------------------------------------- > > > > What was the error you were having? MapWritable supports reading and > > writing *all* writables. The ones defined at the top of the file are an > > optimization and shouldn't affect correctness (basically mapwritable first > > writes a byte and the associated classname, then writes that byte to > > indicate classname everywhere else. For commonly used types we statically > > define the association so that "first write the byte then the classname" > > phase is not necessary). > > > > > MapWritable doesn't support all hadoops writable types > > > ------------------------------------------------------ > > > > > > Key: NUTCH-527 > > > URL: https://issues.apache.org/jira/browse/NUTCH-527 > > > Project: Nutch > > > Issue Type: Bug > > > Affects Versions: 0.9.0 > > > Environment: Tested on Solaris and Windows with Java 1.5 > > > Reporter: Rob Young > > > Attachments: mapwritable.patch > > > > > > > > > The map of classes which implement org.apache.hadoop.io.Writable is not > > > complete. It does not, for example, include > > > org.apache.hadoop.io.BooleanWritable. I would happily provide a patch if > > > someone would explain what the Byte parameter is. > > > > -- > > This message is automatically generated by JIRA. > > - > > You can reply to this email to add a comment to the issue online. > > > > > -- Doğacan Güney ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Nutch-developers mailing list Nutch-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nutch-developers
