org.mortbay.jetty.EofException

Hi all My hadoop version is 1.1.2 I found the following errors in the logs tasktracker This seems like https://issues.apache.org/jira/browse/MAPREDUCE-5 But, but I found a little different, as Caused by: java.io.IOException: Connection reset by peer Whether it is a network problem caused it?

Re: org.mortbay.jetty.EofException

I also found this issue. Hope answer here. Sent from my iPhone5s On 2014年1月2日, at 16:56, centerqi hu cente...@gmail.com wrote: Hi all My hadoop version is 1.1.2 I found the following errors in the logs tasktracker This seems like https://issues.apache.org/jira/browse/MAPREDUCE-5

What are the methods to share dynamic data among mappers/reducers?

Hi, As I know, the Distributed Cache will copy the shared data to the slaves before starting job, and won't change the shared data after that. So are there any solutions to share dynamic data among mappers/reducers? Thanks!

Re: Setting up Snappy compression in Hadoop

I did everything mentioned in the link Ted mentioned, and the test actually works, but using Snappy for MapReduce map output compression still fails with native snappy library not available. On Wed, Jan 1, 2014 at 6:37 PM, bharath vissapragada bharathvissapragada1...@gmail.com wrote: Did you

Re: Setting up Snappy compression in Hadoop

Your natives should be in LD_LIBRARY_PATH or java.library.path for hadoop to pick them up. You can try adding export HADOOP_OPTS=$HADOOP_OPTS -Djava.library.path=Path to your natives lib to hadoop-env.sh in TTs and clients/gateways and restart TTs and give it another try. The reason its working

Re: MapReduce MIME Input type?

Thanks Harsh! Looks like something that might be useful! I appreciate it! *Devin Suiter* Jr. Data Solutions Software Engineer 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212 Google Voice: 412-256-8556 | www.rdx.com On Tue, Dec 31, 2013 at 1:08 AM, Harsh J ha...@cloudera.com wrote: Hey

RE: Setting up Snappy compression in Hadoop

If you really confirmed that libsnappy.so.1 is in the correct location, and being loaded into java library path, working in your test program, but still didn't work in MR, there is one another possibility which was puzzling me before. How do you get the libhadoop.so in your hadoop environment?

RE: Setting up Snappy compression in Hadoop

Amit You may also need to check whether the native library you are using includes Snappy or not. For example, when you compile from source and the libsnappy.so is not found then snappy support is not included as part of the native library for Hadoop (for force it to fail if libsnappy is not

RE: Unable to change the virtual memory to be more than the default 2.1 GB

A few things you can try a) If you don't care about virtual memory controls at all you can bypass it by doing the following change in the XML and restarting YARN. Only you know if this is OK for the application you are trying (IMO the virtual memory being used is huge!) property

RE: get job data in command line in MRv2

Perhaps some of the data you require can be obtained with the mapred tool (here I’m using 2.2.0) e.g. htf@ivygerman:~/hadoop/bin$ ./mapred job Usage: CLI command args [-submit job-file] [-status job-id] [-counter job-id group-name counter-name]

Re: Unable to change the virtual memory to be more than the default 2.1 GB

You need to change the application configuration itself to tell YARN that each task needs more than the default. I see that this is a mapreduce app, so you have to change the per-application configuration: mapreduce.map.memory.mb and mapreduce.reduce.memory.mb in either mapred-site.xml or via

Re: What are the methods to share dynamic data among mappers/reducers?

There isn't anything natively supported for that in the framework, but you can do that yourselves by using a shared service (for e.g via HDFS files, ZooKeeper nodes) that mappers/reducers all have access to. More details on your usecase? In any case, once you start making mappers and reducers

Re: Map succeeds but reduce hangs

Check the TaskTracker configuration in mapred-site.xml: mapred.task.tracker.report.address. You may be setting it to 127.0.0.1:0 or localhost:0. Change it to 0.0.0.0:0 and restart the daemons. Thanks, +Vinod On Jan 1, 2014, at 2:14 PM, navaz navaz@gmail.com wrote: I dont know y it is

Need YARN - Test job.

Hi Guys, we recently installed CDH5 in our test cluster. we need any test job for YARN framework. we are able to run mapreduce job successfully on YARN framework without code change. But we need to test yarn job functionality. can you please guide me. we tired

Reduce task hang[EMERGENCE]

Hi, Our prod cluster met some issues recently, All map tasks finished successfully, but reduce task hanged. but It's not happened on all TaskTrackers, only sometimes. we used mapred-1.0.4 There is 0.0% reduce copy forever until kill task manually. reduce logs on the TaskTracker:

Re: Reduce task hang[EMERGENCE]

Add addtional: Our MR version is 1.2.1, not 1.0.4 There is no useful information in the JT log. On Fri, Jan 3, 2014 at 12:20 PM, Azuryy Yu azury...@gmail.com wrote: Hi, Our prod cluster met some issues recently, All map tasks finished successfully, but reduce task hanged. but It's not

Re: Need YARN - Test job.

Hi Dhahasekaran, You can use the Hadoop workload to do this testing. The key is to set the HADOOP_CONF_DIR envar to the right location: *Yarn* export HADOOP_CONF_DIR=/etc/hadoop/conf.cloudera.yarn1 hadoop fs -rm -r /terasort/TS_input2 yarn jar

Re: Reduce task hang[EMERGENCE]

Does the Reduce task log (of attempt_201312201200_34795_r_00_0) show any errors in trying to communicate with the various TaskTrackers in trying to obtain the data? On Fri, Jan 3, 2014 at 9:54 AM, Azuryy Yu azury...@gmail.com wrote: Add addtional: Our MR version is 1.2.1, not 1.0.4 There

XML to TEXT

Hi, Need to convert XML into text using mapreduce. I have used DOM and SAX parser. After using SAX Builder in mapper class. the child node act as root Element. While seeing in Sys out i found thar root element is taking the child element and printing. For Eg,

Re: Reduce task hang[EMERGENCE]

Hi Harsh, Thanks. There is no any error logs for attempt_201312201200_34795_r_00_0 in the tasktracker log. only '0.0% reduce copy ' I configured all hosts in all slaves and master. This job has only one reduce. it hanged. but I configured everybody's max job running to '1' in the Fair

Re: Reduce task hang[EMERGENCE]

In detail: 'and these people's job never hanged...' these people's map and reduce tasks never hanged. On Fri, Jan 3, 2014 at 1:46 PM, Azuryy Yu azury...@gmail.com wrote: Hi Harsh, Thanks. There is no any error logs for attempt_201312201200_34795_r_00_0 in the tasktracker log. only

Re: XML to TEXT

Hi, you can use org.apache.hadoop.streaming.StreamInputFormat using map reduce to convert XML to text. such as your xml like this: xml namelll/name /xml you need to specify stream.recordreader.begin and stream.recordreader.end in the Configuration: Configuration conf = new Configuration();

Re: any suggestions on IIS log storage and analysis?

Thanks, Peyman. The problem is that the dependence is not simply a key, instead it's so complicated that without #Fields line in one block, it's not even able to parse any line in another block. 2014/1/1 Peyman Mohajerian mohaj...@gmail.com You can run a series of map-reduce jobs on your data,

Haoop: Container is running beyond physical memory limits

Container [pid=8689,containerID=container_1388722003823_0001_01_62] is running beyond physical memory limits. Current usage: 4.1 GB of 4 GB physical memory used; 4.9 GB of 8.4 GB virtual memory used. Killing container. Dump of the process-tree for container_1388722003823_0001_01_62 : |-

Re: Haoop: Container is running beyond physical memory limits

Container [pid=8689,containerID=container_1388722003823_0001_01_62] is running beyond physical memory limits. Current usage: 4.1 GB of 4 GB physical memory used; 4.9 GB of 8.4 GB virtual memory used. Killing container. Dump of the process-tree for container_1388722003823_0001_01_62 : |-

issue about running hive MR job in hadoop

hi, i submit a MR job through hive ,but when it run stage-2 ,it failed why? it seems permission problem ,but i do not know which dir cause the problem Application application_1388730279827_0035 failed 1 times due to AM Container for appattempt_1388730279827_0035_01 exited with exitCode:

Re: issue about running hive MR job in hadoop

Could you check your yarn-local directory authority? From the diagnosis, error occurs at mkdir in local directory. I guess something wrong with local direcotry which is set as yarn local dir. 2014/1/3 ch huang justlo...@gmail.com hi, i submit a MR job through hive ,but when it run stage-2

Re: Haoop: Container is running beyond physical memory limits

If memory of node manager is enough, please boost up the physical memory, and heap size of reduce task. -Dmapreduce.reduce.memory.mb=6144 -Dmapreduce.reduce.java.opts=-Xmx5120M 2014/1/3 Baron Tsai tsaize...@gmail.com Container [pid=8689,containerID=container_1388722003823_0001_01_62] is