Re: Filesystem closed exception

Thanks, Harsh. Particularly for pointing out HADOOP-7973. On Fri, Jan 25, 2013 at 11:51 AM, Harsh J ha...@cloudera.com wrote: It is pretty much the same in 0.20.x as well, IIRC. Your two points are also correct (for a fix to this). Also see: https://issues.apache.org/jira/browse/HADOOP-7973.

Re: Re: EOF when Combiner works

Hi, Harsh. Sorry, I failed to write to user@hadoop.apache.org (http://mail-archives.apache.org/mod_mbox/hadoop-user/201301.mbox/%3CCAOcnVr2A%2BmDQ%2BVY3iovxuP13VTXk2Wog7ui_o6x%3DW0et5igUSg%40mail.gmail.com%3E) The logic of combiner is the same as reducer's one - summation of some statistical

Re: mappers-node relationship

Mappers and Reducers will run in Task instances mapper/reducer instances also called as mapper/reducer slots. Each node can have multiple slots (I mean multiple mapper instances, each run in a child JVM). And this is configurable with properties like mapred.tasktracker.map.tasks.maximum and

affected by 0.20: Allow block reports to proceed without holding FSDataset lock

hi all, My datanodes all have very big disks, and I found block reports can take multiple minutes to generate, and at this time, datanode will not respond for reading. I'v noticed hadoop 1.0.1 fixed this bug, but I'm using 0.20.2 and I can't upgrade my hadoop version unless 1.0.1 supports

Re: affected by 0.20: Allow block reports to proceed without holding FSDataset lock

Hi, 1.0.1 does support fuse_dfs. You should be able to upgrade to 1.0.x latest from 0.20 without too many issues - but the new version is not backward compatible for older clients and will also require a metadata upgrade. On Fri, Jan 25, 2013 at 4:11 PM, Xibin Liu xibin.liu...@gmail.com wrote:

Re: Problems

Hi Ke, We are still looking at possible complications of the VM environment. I will post whatever we discover. Thanks for your interest, Sean From: ke yuan Sent: Friday, January 25, 2013 2:45 AM To: user@hadoop.apache.org Subject: Re: Problems is there anything done with

Cat HDFS files

Hi, I have been thinking recently about various functions I plan to implement in MapReduce. One common theme is that I have many reducers to increase the parallelism and so the performance. However this means that I potentially end up with many separate files in HDFS, one from each reducer. This

Re: Problems

Can you remove native snappy ? and try! On Fri, Jan 25, 2013 at 5:24 PM, Sean Hudson sean.hud...@ostiasolutions.com wrote: Hi Ke, We are still looking at possible complications of the VM environment. I will post whatever we discover. Thanks for your interest, Sean

TT nodes distributed cache failure

Running hadoop-0.20.2 on a 20 node cluster. When running a Map/Reduce job that uses several .jars loaded into the Distributed cache, several (~4) nodes have their map jobs fails because of ClassNotFoundException. All the other nodes proceed through the job normally and the jobs completes. But

Executing a Python program inside Map Function

Is it possible to run a python script inside a Map function which is in java? I what to to run a python script which is on my local disk and I want to use the output of that script for further processing in Map Function to produce key/Value Pairs. Can some give me some idea how to do it.

Re: mappers-node relationship

This may beof some use, about how maps are decided: http://wiki.apache.org/hadoop/HowManyMapsAndReduces Thanks Hemanth On Friday, January 25, 2013, jamal sasha wrote: Hi. A very very lame question. Does numbers of mapper depends on the number of nodes I have? How I imagine map-reduce is

Re: TT nodes distributed cache failure

Could you post the stack trace from the job logs. Also looking at the task tracker logs on the failed nodes may help. Thanks Hemanth On Friday, January 25, 2013, Terry Healy wrote: Running hadoop-0.20.2 on a 20 node cluster. When running a Map/Reduce job that uses several .jars loaded into