Re: Can't get a streaming job to use a custom partitioner

2013-11-30 Thread Adam Kawa
1) Could you print the output of: $ jar tf ./NumericPartitioner.jar 2) Could you try: $ export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:./NumericPartitioner.jar" $ ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \ -libjars ./NumericPartitioner.jar \ -input /input -output /output/keys

Re: MultipleInputs.addInputPath

2013-11-30 Thread Adam Kawa
Can not you specify such a file to process as Path in MultipleInputs.addInputPath? 1) MultipleInputs.addInputPath(job, new Path(args[0] + "/part-0"), TextInputFormat.class, Data1.class) or 2) MultipleInputs.addInputPath(job, new Path(args[0] + "/part-{1-2,5,8-9}"), TextInputFormat.class, D

Re: Desicion Tree Implementation in Hadoop MapReduce

2013-11-30 Thread Yexi Jiang
I watched the video in it but I cannot access its source code due to permission issue. In my opinion, once the decision tree model is built, the model is small enough to be loaded into memory and can be used directly without another mrjob for prediction. The prediction can be conducted in a streami

Re: How to do aggregate operations with multiple reducers

2013-11-30 Thread Adam Kawa
> I know I could just write a script that examines the output from the > reducers and picks out the value with the highest temperature. > In general, I think that this is an acceptable solution. However, if you have a very large number of reducer (let's say, large hundreds or thousands), then rea

Re: Folder creation using Path in MR java code

2013-11-30 Thread Ted Yu
Take a look at this API (and its variants): http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#create(org.apache.hadoop.fs.Path,%20boolean) On Nov 30, 2013, at 12:54 AM, unmesha sreeveni wrote: > Hi all > How to create a folder in hdfs using java code > > I

Re: Desicion Tree Implementation in Hadoop MapReduce

2013-11-30 Thread unmesha sreeveni
I have gone through a Map Reduce implementation of c4.5 in http://btechfreakz.blogspot.in/2013/04/implementation-of-c45-algorithm-using.html Here a decision tree is build. So my doubt is Can we also include the prediction along with that? On Tue, Nov 26, 2013 at 8:52 AM, Yexi Jiang wrote: > Y

Folder creation using Path in MR java code

2013-11-30 Thread unmesha sreeveni
Hi all How to create a folder in hdfs using java code I tried Path folder = new Path("dir1"); But it is not created in hdfs. And i also tried it in eclipse. There also the folder is not creating. -- *Thanks & Regards* Unmesha Sreeveni U.B *Junior Developer*