Re: getAllocatedContainers() is not returning when ran against 2.0.3-alpha

Hi Hitesh, Thanks for the reply. I understand that it may not always allocate all the requested containers in one go. After the actual request is sent, my code repeatedly sends an empty container request and checks the responses until all the requested containers are allocated. But my main wo

Re: are we able to decommission multi nodes at one time?

thanks for all. my strategies for removing from 15 DN to 8 DN. 1. kill two DNs at same time. : NN will detect nodes' down and he will try to retain replication factors of lost blocks. 2. check your NN web UI. there is an info to let you know counts of under-replicated blocks. 3. if it is retur

Re: Error while running MapR program on multinode configuration

Hi Mohammad Thanks for your response . As far as I know,the jars which the client submit to the MP through the net in DistributedCache mode because of the jar localization . The point that I aim at is we need to reduce the size of the objects we submit through network ,which will save

Re: NameNode failure and recovery!

Thats also doable , reducing the checkpoint period would also have have some amount of edit log loss and how short should be the checkpoint interval has to be evaluated.I think the good way to go , in case HA is not doable is SNN and secondary storage NFS. Thanks, Rahul On Thu, Apr 4, 2013 at 12

Re: fsImage & editsLog questions

1. Will fsImage maintain the data/metadata of name node. 2. Will any input files be stored in fsImage. 3. When a namenode goes down will all the data in the name node go down or just the meta data only and what will happen to fsimage & editslog. 5. Is the fsimage file which is also maintained as

Re: Error while running MapR program on multinode configuration

DistributedCache?? Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Thu, Apr 4, 2013 at 5:01 AM, yypvsxf19870706 wrote: > Hi all > >However,I do think the export jar contains external libraries is too > huge to be submitted to the MP by the client. >So It comes to

Re: Error while running MapR program on multinode configuration

Hi all However,I do think the export jar contains external libraries is too huge to be submitted to the MP by the client. So It comes to the conclusion that the jar without the external jars could be suited the situation ,however the runnable jar brings the errors. How to solve the err

Re: Error while running MapR program on multinode configuration

As expected :) Actually this is something related to Eclipse. When it comes to the creation of executable jars containing external jars we need to be a bit careful. One thing which you could probably try is to select the "Package required libraries into generated jar" option while exporting your pr

Re: Error while running MapR program on multinode configuration

Thank you! You are right! I created the jar file using the command line and not eclipse and it worked! On Thu, Apr 4, 2013 at 4:06 AM, Mohammad Tariq wrote: > It seems to be a non-hadoop issue to me. Is the jar which you are finally > creating a runnable jar?We can't reference external class

Re: Error while running MapR program on multinode configuration

Try to run your code directly from eclipse once (without creating the jar). Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Thu, Apr 4, 2013 at 4:06 AM, Mohammad Tariq wrote: > It seems to be a non-hadoop issue to me. Is the jar which you are finally > creating a runnabl

Re: Error while running MapR program on multinode configuration

It seems to be a non-hadoop issue to me. Is the jar which you are finally creating a runnable jar?We can't reference external classes(jars) in a runnable jar file. Everything must be contained inside the jar itself. Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Thu, Apr

Re: Error while running MapR program on multinode configuration

Thanks for replying! version : hadoop 1.1.1 I am creating a jar file called ga_test.jar and then placing it in the build path of another program. Also, I am using eclipse to create another jar file including ga_test.jar as a reference library. I do not know why hadoop is taking the filename as r

Re: Error while running MapR program on multinode configuration

Which version are you using? Could you plz show me your code, if possible for you. Also, how are you running the job? For a detailed explanation you might find this useful. Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Thu,

Re: Error while running MapR program on multinode configuration

But I do not have a ":" in any of my file names. What could the other reasons be? I am not able to debug the error.. Thank you for replying. On Thu, Apr 4, 2013 at 2:02 AM, Mohammad Tariq wrote: > Hello ma'am, > >Please make sure that you don't have ":" in your files. > > Warm Regards, >

Re: Error while running MapR program on multinode configuration

Hello ma'am, Please make sure that you don't have ":" in your files. Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Thu, Apr 4, 2013 at 1:50 AM, Varsha Raveendran < varsha.raveend...@gmail.com> wrote: > Hello, > > I am facing this error while trying to run a jar file

Error while running MapR program on multinode configuration

Hello, I am facing this error while trying to run a jar file on hadoop : 13/04/04 01:48:01 INFO mapred.JobClient: Cleaning up the staging area hdfs://MT2012158:54310/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201304032344_0008 Exception in thread "main" java.lang.reflect.InvocationTargetEx

Re: NameNode failure and recovery!

If you are not in position to go for HA just keep your checkpoint period shorter to have recent data recoverable from SNN. and you always have a option hadoop namenode -recover try this on testing cluster and get versed to it. and take backup of image at some solid state storage. ∞ Shashwat Sh

Re: Linux io scheduler

Thanks Chris for response, I would love to try out. What would be the metric(s) that we will be use to measure if we improve thing.? On Tue, Apr 2, 2013 at 11:24 AM, Chris Embree wrote: > I assume your talking about the I/O scheduler. Based on normal advice, > only change this if you have a "s

Re: NameNode failure and recovery!

There is a 3rd, most excellent way: Use HDFS's own HA, see http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html :) On Wed, Apr 3, 2013 at 8:10 PM, Rahul Bhattacharjee wrote: > Hi all, > > I was reading about Hadoop and got to know that there are two

Re: NameNode failure and recovery!

If it's not possible to restart the NN daemon on the same box, then yes. Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Wed, Apr 3, 2013 at 9:30 PM, Rahul Bhattacharjee wrote: > Thanks to all of you for precise and complete responses. > > S > o in case of failure we hav

Re: NameNode failure and recovery!

Thanks to all of you for precise and complete responses. S ​o in case of failure we have to bring another backup system up with the fsimage and edit logs from the NFS filer. SNN stays as is for the new NN. Thanks, Rahul​ On Wed, Apr 3, 2013 at 8:38 PM, Azuryy Yu wrote: > for Hadoopv2, there i

Re: getAllocatedContainers() is not returning when ran against 2.0.3-alpha

If I understand your question, you are expecting all the containers to be allocated in one go? Or are you seeing your application hang because it asked for 10 containers but it only received a total of 9 even after repeated calls to the RM? There is no guarantee that you will be allocated all

Re: NameNode failure and recovery!

for Hadoopv2, there is HA, so SNN is not necessary. On Apr 3, 2013 10:41 PM, "Rahul Bhattacharjee" wrote: > Hi all, > > I was reading about Hadoop and got to know that there are two ways to > protect against the name node failures. > > 1) To write to a nfs mount along with the usual local disk. >

getAllocatedContainers() is not returning when ran against 2.0.3-alpha

Hi, I am running an application that had been written and working well with the hadoop-2.0.0-alpha but when I am running the same against 2.0.3-alpha, the getAllocatedContainers() method called on AMResponse is not returning all the containers allocated sometimes. For example, I request for 10 co

Re: NameNode failure and recovery!

@Vijay : We seem to be in 100% sync though :) Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Wed, Apr 3, 2013 at 8:27 PM, Mohammad Tariq wrote: > Hello Rahul, > > It's always better to have both 1 and 2 together. One common > misconception is that SNN is a backup

Re: NameNode failure and recovery!

Hello Rahul, It's always better to have both 1 and 2 together. One common misconception is that SNN is a backup of the NN, which is wrong. SNN is a helper node to the NN. In case of any failure SNN is not gonna take up the NN spot. Yes, we can't guarantee that the SNN fsimage replica will a

RE: NameNode failure and recovery!

Hi Rahul, The SNN does not act as a backup / standby NameNode in the event of failure. The sole purpose of the Secondary NameNode (or as it’s otherwise / more correctly known as the Checkpoint Node) is to perform checkpointing of the current state of HDFS: The SNN retrieves the fsima

Re: NameNode failure and recovery!

Or both the options are used together. NFS + SNN ? On Wed, Apr 3, 2013 at 8:10 PM, Rahul Bhattacharjee wrote: > Hi all, > > I was reading about Hadoop and got to know that there are two ways to > protect against the name node failures. > > 1) To write to a nfs mount along with the usual local

NameNode failure and recovery!

Hi all, I was reading about Hadoop and got to know that there are two ways to protect against the name node failures. 1) To write to a nfs mount along with the usual local disk. -or- 2) Use secondary name node. In case of failure of NN , the SNN can take in charge. My questions :- 1) SNN is al

Re: Job log location and retention

Zheyi, The jobtracker doesn't keep a reference to the job to save memory, but you may still find it in the filesystem. For a default CDH3 installation, it will be in the jobtracker's local filesystem, at /var/log/hadoop-0.20/history/done/ Logs from individual tasks are a little trickier to find

Re: MapReduce on Local files

Thank you Azuryy. It was about the files ending with a tilde "~". These files are actually backup files, hidden to the users but my job was able to see them. I am working on Ubuntu(Gnome DE). Nothing serious, just out of curiosity :) Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot

Re: MapReduce on Local files

For FileInputFormat, start with "_" is hidden file by default. you can write a custom PathFilter, and pass it to the InputFormat. On Wed, Apr 3, 2013 at 5:58 PM, Harsh J wrote: > You've been misled by the GUI you use, I'm afraid. Many DEs (Desktop > Environments) consider ~-suffix files as hidd

Re: MapReduce on Local files

I see. Thank you so much the clarification Harsh :) Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Wed, Apr 3, 2013 at 3:28 PM, Harsh J wrote: > You've been misled by the GUI you use, I'm afraid. Many DEs (Desktop > Environments) consider ~-suffix files as hidden but no

Re: MapReduce on Local files

You've been misled by the GUI you use, I'm afraid. Many DEs (Desktop Environments) consider ~-suffix files as hidden but not the general standards (try ls for example, or even shell expansions, it will ignore . prefixes, but not ~ suffixes) :) To answer specifically though, no, the base FileInputF

Re: MapReduce on Local files

Hello Harsh, Thank you for the response. I am sorry for being unclear. Actually I was talking about the backup files which end with "~" I mean these files are not visible normally, but my job is able to see them. Does FileInputFormat behave in the same way for "~" as it does in the case of

Re: are we able to decommission multi nodes at one time?

not at all. so don't worry about that. On Wed, Apr 3, 2013 at 2:04 PM, Yanbo Liang wrote: > It means that may be some replicas will be stay in under replica state? > > > 2013/4/3 Azuryy Yu > >> bq. then namenode start to copy block replicates on DN-2 to another DN, >> supposed DN-2. >> >> sor

Re: hadoop datanode kernel build and HDFS multiplier factor

I have done similar experiment for tuning hadoop performance. Many factors will influence the performance such as hadoop configuration, JVM, OS. For Linux kernel related factors, we have found two main focus of attention: 1, Every read operation of file system will trigger one disk write operation