You String as the outputKey (). java.lang.String is not Writable. Change it
to Text just like you did for the Mapper.
Regards,
Shahab
On Wed, Sep 17, 2014 at 10:43 AM, Blanca Hernandez <
blanca.hernan...@willhaben.at> wrote:
> Thanks for answering:
>
>
>
> hadoop jar /tmp/hadoop-test.jar
> at.willhaben.hadoop.AveragePriceCalculationJob
>
>
>
> In the AveragePriceCalculationJob I have my configuration:
>
>
>
>
>
> private static class AveragePriceCalculationJob extends MongoTool {
>
> private AveragePriceCalculationJob(AveragePriceNode currentNode,
> String currentId, int nodeNumber) {
>
> Configuration conf = new Configuration();
>
> MongoConfig config = new MongoConfig(conf);
>
> setConf(conf);
>
> // change for my values
>
> config.setInputFormat(MongoInputFormat.class);
>
> config.setOutputFormat(MongoOutputFormat.class);
>
>
>
> config.setMapperOutputKey(Text.class);
>
> config.setMapperOutputValue(BSONObject.class);
>
> config.setOutputKey(String.class);
>
> config.setOutputValue(BSONWritable.class);
>
>
>
> config.setInputURI("myUrl");
>
> config.setOutputURI("myUrl");
>
> config.setMapper(AveragePriceMapper.class);
>
> config.setReducer(AveragePriceReducer.class);
>
>
>
> }
>
> }
>
>
>
>
>
> And the main method:
>
>
>
>
>
> public static void main(String [] args) throws InterruptedException,
> IOException, ClassNotFoundException {
>
> // … some code
>
>
>
> try {
>
> ToolRunner.run(new AveragePriceCalculationJob(currentNode,
> currentId, nodeNumber), args);
>
> } catch (Exception e) {
>
> e.printStackTrace(); //To change body of catch statement
> use File | Settings | File Templates.
>
> }
>
> }
>
>
>
>
>
> Best regards,
>
>
>
> Blanca
>
>
>
> *Von:* Shahab Yunus [mailto:shahab.yu...@gmail.com]
> *Gesendet:* Mittwoch, 17. September 2014 16:37
> *An:* user@hadoop.apache.org
> *Betreff:* Re: ClassCastException on running map-reduce jobs + tests on
> Windows (mongo-hadoop)
>
>
>
> Can you provide the driver code for this job?
>
>
>
> Regards,
>
> Shahab
>
>
>
> On Wed, Sep 17, 2014 at 10:28 AM, Blanca Hernandez <
> blanca.hernan...@willhaben.at> wrote:
>
> Hi again, I changed the String objects with org.apache.hadoop.io.Text
> objects (why is String not accepted?), and now I get another exception, so
> I don´t really know if I solved something or I broke something:
>
>
>
>
>
> java.lang.Exception: java.lang.NullPointerException
>
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
>
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
>
> Caused by: java.lang.NullPointerException
>
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:988)
>
> at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
>
> at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
>
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
>
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
>
>
> If I could debug it in my IDE, I think I could work faster, but I have the
> problems already exposed. How am I testing now? Building a jar, copying it
> on the server and running a Hadoop jar command (not very performance
> approach…).
>
>
>
> Could you give me a hand on this? Any Windows + IntelliJ IDEa there?
> Maaaany thanks!
>
>
>
>
>
>
>
> *Von:* Blanca Hernandez [mailto:blanca.hernan...@willhaben.at]
> *Gesendet:* Mittwoch, 17. September 2014 15:27
> *An:* user@hadoop.apache.org
> *Betreff:* ClassCastException on running map-reduce jobs + tests on
> Windows (mongo-hadoop)
>
>
>
> Hi!
>
>
>
> I am getting some CCE and don´t really understand why…
>
>
>
> Here my mapper:
>
>
>
> public class AveragePriceMapper extends Mapper<String, BSONObject, String,
> BSONObject>{
>
> @Override
>
> public void map(final String key, final BSONObject val, final Context
> context) throws IOException, InterruptedException {
>
> String id = “result_of_making_some_operations”;
>
> context.write(id, val);
>
> }
>
> }
>
>
>
> And in my configuration:
>
>
>
> config.setMapperOutputKey(String.class);
>
> config.setMapperOutputValue(BSONObject.class);
>
>
>
>
>
> On running my generated jar on the server, seems to work everything ok
> until:
>
>
>
> 14/09/17 15:20:36 INFO mapred.MapTask: Processing split:
> MongoInputSplit{URI=mongodb://user:pass@host:27017/my_db.my_collection,
> authURI=null, min={ "_id" : { "$oid" : "541666d8e4b07265e257a42e"}}, max={
> }, query={ }, sort={ }, fields={ }, notimeout=false}
>
> 14/09/17 15:20:36 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
>
> 14/09/17 15:20:36 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
>
> 14/09/17 15:20:36 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
>
> 14/09/17 15:20:36 INFO mapred.MapTask: soft limit at 83886080
>
> 14/09/17 15:20:36 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
>
> 14/09/17 15:20:36 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
>
> 14/09/17 15:20:36 INFO mapred.LocalJobRunner: map task executor complete.
>
> 14/09/17 15:20:36 WARN mapred.LocalJobRunner:
> job_local1701078621_0001java.lang.Exception: java.lang.ClassCastException:
> class java.lang.String
>
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
>
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
>
> Caused by: java.lang.ClassCastException: class java.lang.String
>
> at java.lang.Class.asSubclass(Class.java:3126)
>
> at
> org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:885)
>
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:981)
>
> at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
>
> at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
>
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
>
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
>
>
>
>
> Did I miss something??
>
>
>
>
>
> Another issue I am worry about: working on a Windows system makes
> everything quite complicated with Hadoop. I have it installed and running,
> the same as my mongoDB database (I am using the connector provided by
> them). Running the same main class I am using in the hadooop jar call on
> the server (in the example before), but from my IDE, I get this exception:
>
>
>
> PriviledgedActionException as:hernanbl cause:java.io.IOException: Failed
> to set permissions of path:
> \tmp\hadoop-hernanbl\mapred\staging\hernanbl1600842219\.staging to 0700
>
>
>
> How could I make it run?
>
>
>
>
>
> Many thanks!!
>
>
>
> Best regards,
>
>
>
> Blanca
>
>
>
