Hi all,
Had some queries on Map task's awareness. From what I understand,
every map task instance is destined to process the data in a specific
Input split (can be across HDFS blocks).
1) Do these map tasks have a unique instance number? If yes, are they
mapped to its specific input splits and
Dear all,
Today I faced a problem while running a map-reduce job in C++. I am not
able to understand to find the reason of the below error :
11/03/30 12:09:02 INFO mapred.JobClient: Task Id :
attempt_201103301130_0011_m_00_0, Status : FAILED
java.io.IOException: pipe child exception
Hey guys, I'm new here, and recently I'm working on configuring a cluster
with 32 nodes.
However, there are some problems, I describe below
The cluster consists of nodes, which I don't have root to configure as I
wish. We only have the space /localhost_name/local space to use.
Thus, we only have
Ok, so if I understand correctly, you want to change the location of
the datastore on individual computers.
I've tested it on my cluster, and it seems to work. Just for the sake
of troubleshooting, you didn't mention the following:
1) Which computer were you editing the files on
2) which file
Thank you modemide for your quick response.
Sorry for not be clear...your understanding is right.
I have a machine, called grande, and the other called pseg. Now i'm using
grande as master (by fill the masters file by grande) and pseg as slave.
the configuration of grande is (core-site.xml)
I'm a little confused as to why you're putting
/pseg/local /...
as the location.
Are you sure that you've been given a folder name at the root of the
drive called /pseg/ ?
Maybe try to ssh to your server and navigate to your datastore folder,
then do pwd.
That should give you the working
Harsh:
I found that jvmManager.getPid(...) returned the pid of
MapTaskRunner, but I want to get the task's pid. For example, I ran the the
example randomwrite, the pid of task which is writing is 8268, but
jvmManager.getPid(...) seemed to be its parent pid. I can not figure out the
Hello,
My map tasks are freezing after 100% .. I'm suspecting my mapper.close()
function which does some sorting. Any better suggestion of where shall I put my
sorting method ? I thought of mapper.close() so that each map task sorts its
own output (which is local) and hence faster.
I haven't used 0.21. You can compare the source codes of the two versions.
I set these in namenode's hdfs-site.xml to 1. I'm not sure you'd want to do it
on a production cluster if its a big one.
On 3/29/11 7:13 PM, Rita rmorgan...@gmail.com wrote:
what about for 0.21 ?
Also, where do you
Hi,
When I click the Browse the filesystem link, I was redirected to
http://localhost.localdomain:50075/browseDirectory.jsp?namenodeInfoPort=50070dir=/,
which is an error URL, I think it should be related to the domain name of my
server.
I am setting up a pseudo cluster environment.
Regards,
It's not the sorting, since the sorted files are produced in output, it's then
mapper not existing well. so can anyone tell me if it's wrong to write
mapper.close() function like this ?
@Override
public void close() throws IOException{
helper.CleanUp();
Thanks for that tidbit, it appears to be the problem... Maybe that's a
well known issue? or perhaps it should be added to the setup WIKI ???
-Bill
On 03/29/2011 09:47 PM, Harsh J wrote:
On Wed, Mar 30, 2011 at 3:59 AM, Bill Brunebbr...@decarta.com wrote:
Hi,
I've been running hadoop
On Thu, Mar 31, 2011 at 12:59 AM, Bill Brune bbr...@decarta.com wrote:
Thanks for that tidbit, it appears to be the problem... Maybe that's a well
known issue? or perhaps it should be added to the setup WIKI ???
It isn't really a Hadoop issue. See here for what defines a valid
hostname (The
It seems like when JVM reuse is enabled map task log data is not getting
written to their corresponding log files; log data from certain map tasks
gets appended to log files corresponding to some other map task.
For example, I have a case here where 8 map JVMs are running simultaneously
and all
Hi Shrinivas,
Yes, this is the behavior of the task logs when using JVM Reuse. You should
notice in the log directories for the other tasks a log index file which
specifies the byte offsets into the log files where the task starts and
stops. When viewing logs through the web UI, it will use these
Dear all,
Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.
I attached the patch.
Please find the attachment. I just follow below steps for Hadoop :
1. Download Hadoop-0.20.2.tar.gz
2. Extract the file.
3. Set Configurations in site.xml files
Thanks best Regards,
Sorry, Just check the attachment now.
Adarsh Sharma wrote:
Dear all,
Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.
I attached the patch.
Please find the attachment. I just follow below steps for Hadoop :
1. Download Hadoop-0.20.2.tar.gz
2. Extract the file.
3. Set
There is a utility available for Unix called 'patch'. You can use that
with a suitable -p(num) argument (man patch, for more info).
On Thu, Mar 31, 2011 at 9:41 AM, Adarsh Sharma adarsh.sha...@orkash.com wrote:
Dear all,
Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.
Any update on the below error.
Please guide.
Thanks best Regards,
Adarsh Sharma
Adarsh Sharma wrote:
Dear all,
Today I faced a problem while running a map-reduce job in C++. I am
not able to understand to find the reason of the below error :
11/03/30 12:09:02 INFO mapred.JobClient:
Dear all,
Today I faced a problem while running a map-reduce job in C++. I am not
able to understand to find the reason of the below error :
11/03/30 12:09:02 INFO mapred.JobClient: Task Id :
attempt_201103301130_0011_m_00_0, Status : FAILED
java.io.IOException: pipe child exception
20 matches
Mail list logo
