Re: Creating files through the hadoop streaming interface

The raw streaming interface has much issues of this manner. The python open(…, 'w') calls won't open files on HDFS, further. Perhaps, since you wish to use Python for its various advantages, check out this detailed comparison guide of various Python-based Hadoop frameworks (including the raw stream

RE: Profiling the Mapper using hprof on Hadoop 0.20.205

I tried various range values. I'm not even seeing the map method and the recordreader methods. On Feb 7, 2013 3:58 AM, "java8964 java8964" wrote: > What range you gave it for mapred.task.profile.maps? And you sure your > mapper will invoke the methods you expect in the traces? > > Yong > > -

Re: What to do/check/debug/root cause analysis when jobtracker hang

I wish it is the case. i have another prod. cluster using cdh3u4 too, but it won't happen. On Wed, Feb 6, 2013 at 6:12 PM, java8964 java8964 wrote: > Our cluster on cdh3u4 has the same problem. I think it is caused by some > bugs in JobTracker. I believe Cloudera knows about this issue. > > After

RE: What to do/check/debug/root cause analysis when jobtracker hang

Our cluster on cdh3u4 has the same problem. I think it is caused by some bugs in JobTracker. I believe Cloudera knows about this issue. After upgrading to cdh3u5, we havn't faced this issue yet, but I am not sure if it is confirmed to fix in the CDH3U5. Yong > Date: Mon, 4 Feb 2013 15:21:18 -08

RE: Profiling the Mapper using hprof on Hadoop 0.20.205

What range you gave it for mapred.task.profile.maps? And you sure your mapper will invoke the methods you expect in the traces? Yong Date: Wed, 6 Feb 2013 23:50:08 +0200 Subject: Profiling the Mapper using hprof on Hadoop 0.20.205 From: yaron.go...@gmail.com To: user@hadoop.apache.org Hi,I wish

Profiling the Mapper using hprof on Hadoop 0.20.205

Hi, I wish to profile my mapper, so I've set the properties mapred.task.profileand mapred.task.profile.maps in mapred-site.xml. At the end of the job I'm getting a profile.out file, however I think it's not the profile of the mapper... none of the methods I use in the mapper appear on any of the tr

Re: TaskStatus Exception using HFileOutputFormat

It is my pleasure. Sean. On Wed, Feb 6, 2013 at 1:46 PM, Sean McNamara wrote: > > > Can you check whether hdfs related config was passed to Job correctly? > > Ahhh, that was it! It wasn't picking up the .xml files. Fixed that and > it seems to be working now. > > Thank you for your help!!!

Re: TaskStatus Exception using HFileOutputFormat

> Can you check whether hdfs related config was passed to Job correctly? Ahhh, that was it! It wasn't picking up the .xml files. Fixed that and it seems to be working now. Thank you for your help!!! Sean From: Ted Yu mailto:yuzhih...@gmail.com>> Reply-To: "user@hadoop.apache.org

RE: Using hadoop streaming with binary data

You can use hadoop's DistCp to copy files via map/reduce. Date: Wed, 6 Feb 2013 16:19:23 -0500 Subject: Using hadoop streaming with binary data From: jayqhac...@gmail.com To: user@hadoop.apache.org Is it possible to pass unmolested binary data through a map-only streaming job from the command li

Using hadoop streaming with binary data

Is it possible to pass unmolested binary data through a map-only streaming job from the command line? I.e., is there a way to avoid extra tabs and newlines in the output? I don't need input splits or key/value pairs, I just want one whole input file fed unmodified into a program, and its output w

Re: TaskStatus Exception using HFileOutputFormat

Thanks for this information. Here is related code: public static void configureIncrementalLoad(Job job, HTable table) throws IOException { Configuration conf = job.getConfiguration(); ... Path partitionsPath = new Path(job.getWorkingDirectory(),

Re: TaskStatus Exception using HFileOutputFormat

Ok, a bit more info- From what I can tell is that the partitions file is being placed into the working dir on the node I launch from, and the task trackers are trying to look for that file, which doesn't exist where they run (since they are on other nodes.) Here is the exception on the TT in

TaskTracker heartbeats not sent at the configured intervals

Hello everyone, I've been encountering the following problem for some time now and it is really slowing down my work. I would appreciate any help you guys can provide. I am using Hadoop 1.0.3. I configured the TaskTrackers to send heartbeats to the JobTracker every 1 second. Most of the time the

Re: How to find HDFS folders file count.

count works perfectly fine. do you need something from the API? Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Wed, Feb 6, 2013 at 9:38 PM, Harsh J wrote: > You have posted the "fs -count" command which is also the answer to > your question - does it not help? > > On We

Re: “hadoop namenode -format” formats wrong directory

Try running the command using "hadoop --config /etc/hadoop/conf" to make sure it is looking at the right conf dir. It would help to understand how you installed hadoop - local build/rpm, etc .. to figure out which config dir is being looked at by default. -- Hitesh On Feb 6, 2013, at 7:25 AM

Re: TaskStatus Exception using HFileOutputFormat

> Using the below construct, do you still get exception ? Correct, I am still getting this exception. Sean From: Ted Yu mailto:yuzhih...@gmail.com>> Reply-To: "user@hadoop.apache.org" mailto:user@hadoop.apache.org>> Date: Tuesday, February 5, 2013 7:50 PM To: "use

Re: “hadoop namenode -format” formats wrong directory

That's worked! Thank you! Can't find, which setting should I change to show hadoop my conf directory? 2013/2/6 Samir Ahmic > Hi, > Can you try to execute format command like this: > > hadoop --config /etc/hadoop/conf namenode -format > > Look like hadoop script is unable to pickup your config

Re: How to find HDFS folders file count.

You have posted the "fs -count" command which is also the answer to your question - does it not help? On Wed, Feb 6, 2013 at 3:56 PM, Dhanasekaran Anbalagan wrote: > Hi Guys, > > We have done moving local file to HDFS > hadoop fs -copyFromLocal > > we have verified some of the file missing in the

Re: “hadoop namenode -format” formats wrong directory

Hi, Can you try to execute format command like this: hadoop --config /etc/hadoop/conf namenode -format Look like hadoop script is unable to pickup your config dir. Cheers On Wed, Feb 6, 2013 at 4:32 PM, Andrey V. Romanchev wrote: > Done. > $ echo $HADOOP_HOME > /usr/lib/hadoop/ > $ ll /usr/l

Re: “hadoop namenode -format” formats wrong directory

Done. $ echo $HADOOP_HOME /usr/lib/hadoop/ $ ll /usr/lib/hadoop/conf lrwxrwxrwx. 1 root root 16 Фев 6 16:09 /usr/lib/hadoop/conf -> /etc/hadoop/conf Sadly, it did not help. 2013/2/6 Nitin Pawar > HADOOP_HOME is the directory where you installed hadoop > > a directory above your conf directory

Cannot use env variables in "hodrc"

Hello again, Considering that I have not received any replies, I was wondering whether this is not the correct list for "hod" questions? Please let me know if I should better direct this question to another list, a hod-specific one perhaps? Thank you! > > On a related note, env-vars is also

Re: “hadoop namenode -format” formats wrong directory

HADOOP_HOME is the directory where you installed hadoop a directory above your conf directory (in other words parent directory of conf) On Wed, Feb 6, 2013 at 8:55 PM, Andrey V. Romanchev < andrey.romanc...@gmail.com> wrote: > Hello Samir > Yes. Here is full section from my hdfs-site.xml > >

Re: “hadoop namenode -format” formats wrong directory

Hello Samir Yes. Here is full section from my hdfs-site.xml dfs.name.dir /mnt/ext/hadoop/hdfs/namenode Determines where on the local filesystem the DFS name node should store the name table. If this is a comma-delimited list of directories then the name tab

Re: “hadoop namenode -format” formats wrong directory

Added to hdfs-site.xml dfs.namenode.name.dir /mnt/ext/hadoop/hdfs/namenode true The same problem exists :( 2013/2/6 Yanbo Liang > you can try to use the new parameter "dfs.namenode.name.dir" to > specify the directory. > > 2013/2/6, Andrey V. Romanchev : > > Hello! > > > > I'

Re: “hadoop namenode -format” formats wrong directory

No. Is it necessary? What directory it should point? Just tested export HADOOP_HOME=/mnt/ext/hadoop the same problem, /tmp/hadoop-hadoop used 2013/2/6 Nitin Pawar > have you set the variable HADOOP_HOME ? > > i just tested it and its working fine on my hdfs > > > On Wed, Feb 6, 2013 at 8:37 PM

Re: “hadoop namenode -format” formats wrong directory

Hi, Andrey Did you try true in your config ? It should be like this: dfs.name.dir /mnt/ext/hadoop/hdfs/namenode true Cheers On Wed, Feb 6, 2013 at 4:07 PM, Andrey V. Romanchev wrote: > > Hello! > > I'm trying to install Hadoop 1.1.2.21 on CentOS 6.3. > > I've configured dfs.name.dir in /etc/h

Re: “hadoop namenode -format” formats wrong directory

you can try to use the new parameter "dfs.namenode.name.dir" to specify the directory. 2013/2/6, Andrey V. Romanchev : > Hello! > > I'm trying to install Hadoop 1.1.2.21 on CentOS 6.3. > > I've configured dfs.name.dir in /etc/hadoop/conf/hdfs-site.xml file > > dfs.name.dir > /mnt/ext/hadoop/hdfs/n

Re: “hadoop namenode -format” formats wrong directory

have you set the variable HADOOP_HOME ? i just tested it and its working fine on my hdfs On Wed, Feb 6, 2013 at 8:37 PM, Andrey V. Romanchev < andrey.romanc...@gmail.com> wrote: > Hello! > > I'm trying to install Hadoop 1.1.2.21 on CentOS 6.3. > > I've configured dfs.name.dir in /etc/hadoop/con

“hadoop namenode -format” formats wrong directory

Hello! I'm trying to install Hadoop 1.1.2.21 on CentOS 6.3. I've configured dfs.name.dir in /etc/hadoop/conf/hdfs-site.xml file dfs.name.dir /mnt/ext/hadoop/hdfs/namenode But when I run "hadoop namenode -format" command, it formats /tmp/hadoop-hadoop/dfs/name directory instead. It seems like de

Re: [Hadoop-Help]About Map-Reduce implementation

thats more than sufficient On Wed, Feb 6, 2013 at 4:56 PM, Mayur Patil wrote: > Thanks to you duo. You solved my problem so easily. I want to > > ask one more question; for reference. I have > > 1. hadoop the definitive guide > 2. Hadoop In Action > > Is it sufficient or do I need some more mate

Re: [Hadoop-Help]About Map-Reduce implementation

Thanks to you duo. You solved my problem so easily. I want to ask one more question; for reference. I have 1. hadoop the definitive guide 2. Hadoop In Action Is it sufficient or do I need some more material to study your suggested implementation?? * -- Cheers, Mayur* Hey Mayur, > > If you are