Re: Map output files contains headers?

Are you sure the value type in pair output by mapper, is the same with the the value type in pair that accepted by reducer. I use the BytesWritable, and the sizes of the data are same.

Re: how to find the id of each map task?

*org.apache.hadoop.mapreduce.MapContext::*getTaskAttemptID() On Tue, Jul 6, 2010 at 11:43 AM, Denim Live wrote: > Hello > > I want to get the id of each mapper and reducer task because I want to tag > the output of these mappers and reducers according to the mapper and reducer > id. How can

Re: Who creates job.jar?

*job.jar* is used to encapsulate the program code you write by yourself. It's not an automatic produced file, but you self should create it manually. And of course you can use other names to call this jar file. If you don't understand, you can learn to study the hadoop-0.20.2-dev-examples.jar fil

Re: log files not found

Hi, Raghava Try to see the web UI of job tracker. The default address should be : http://master-host:50030/jobtracker.jsp Best Regards Jiamin Lu

Re: Writing Bytes from Map and Reduce functions

Hi, Saliya, If you said the part files, I think you are talking about the results of the reduce function that stored inside the HDFS, right? If so, I think this example in "Hadoop The Definitive Guide" can help you. - Example 3-1. Displaying files from a Hadoop filesystem on standard o

Re: Writing Bytes from Map and Reduce functions

Hi, Saliya, The data transformation in MapReduce is: *map*(k1,v1)-> list(k2,list(v2)) *reduce* (k2, list(v2)) -> (k3, list(v3)) The output from map will be sent to reducer as input directly. In your recude function, you can only get k2, v2 as input type. So, in your case, the type of

Re: How can I get system environment variable in core-site.xml

I solved how to set different data directories on the same NFS $HOME directories shared by all nodes in the cluster. The HADOOP_OPTS value set in conf/hadoop-env.sh is the used to set custom properties. All the values set to this variable will be set to JVM. Just like we set java.net.preferIPv4Stac

Re: Small Files as input, Heap Size and garbage Collector

I am reading the "Hadoop The Definitive Guide", and in the page 71, it said, when there are too many small files, the memory of the NameNode will be eat out since each file need to keep its metadata in NameNode. The book also suggest using Hadoop Archives, or HAR files to pack files into HDFS block

Re: Does it possible that output value type of Map function different from final type?

Dear Aaron, Thanks for your reply! It's very help for you to tell me this, I don't know that the key type of the final output also can be different from the intermediate result. That's great! Best Regards Jiamin Lu

Re: Does it possible that output value type of Map function different from final type?

Hi, all I solved it! By through set: conf.setMapOutputKeyClass(Text.class); conf.setMapOutputValueClass(BytesWritable.class); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(Text.class); So stupid I am! I need to read more documents first! Best Regar

Does it possible that output value type of Map function different from final type?

Hi, all >From the original paper of mapReduce by google, the signature of data transform is that: map: (k1, v1) -> list(k2, v2) reduce: (k2, list(v2)) -> list(v2) Here, the output value type is v2, and the final type is also v2. But, what I want to achieve is that, the final value type should

How can I copy a local disk file, and output it as bytesWritable

Hi, everyone! The problem what I met is that, I want to transform a local disk file into bytesWritable to output. Now, all I found is only can use FileSystem.copyFromLocalFile to copy a file from local disk to HDFS. The first problem is that, will this file be distributed naturally? And the othe