Hadoop 2.2 Built-in Counters

Hi all, I'm trying to analyze my mapreduce job performance via built-in counters such as physical memory usage, heap memory usage... When the job is running, I can watch these counters via Resource manager website(namenode:8088). However, when the job is done, counter information is not available i

Re: Started learning Hadoop. Which distribution is best for native install in pseudo distributed mode?

He didn’t ask for the best and nobody framed up their answer like that. He asked what people were using. Out of the 10 responses only four of them actually answered his question. I’ve been studying Hadoop for two months straight. Quite frankly, I wish more people would ask for community input a

Re: Started learning Hadoop. Which distribution is best for native install in pseudo distributed mode?

Engough wars on going on which is best. You choose one of it and try to learn and there is nothing that x is better or y is better. It is upto your choice. Thanks, Sam From: Sebastiano Di Paola mailto:sebastiano.dipa...@gmail.com>> Reply-To: "user@hadoop.apache.org

Re: fair scheduler not working as intended

Hi Henry, I think FairScheduler is a better option for your case. As if you used FifoScheduler, the latency of the short jobs will be worst (in case of any of longrun jobs are currently running in your cluster). So I think your queue layout is good to apply fairness between the long and short jobs

RE: Implementing security in hadoop

You just need to click the link, or you can search HDFS-6134 in apache JIRA or google. >>> Are there any other security tool for the same? Currently in Hadoop trunk, there is no such security tool. You may need to rely on OS/FS encryption functionality of your servers or you can find some solut

RE: Setting up hadoop cluster offline

Sure, you can do it. Please follow the quick start: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html Regards, Yi Liu From: harish tangella [mailto:harish.tange...@gmail.com] Sent: Wednesday, August 13, 2014 8:34 PM To: user@hadoop.apache.org Subject: Sett

Re: Started learning Hadoop. Which distribution is best for native install in pseudo distributed mode?

Hi, I'm a newbie too and I'm not using any particular distribution. Just download the component I need / want to try for my deploiment and use them. It's a slow process but allows me to better understand what I'm doing under the hood. Regards, Seba On Tue, Aug 12, 2014 at 10:12 PM, mani kandan w

Setting up hadoop cluster offline

Hi all, Request you to help. Is it possible to set up a hadoop cluster offline using tarball files. Regards, Harish

RE: Implementing security in hadoop

Thanks, but this link is not working it says issuer doesn't exist Are there any other security tool for the same? From: Liu, Yi A [mailto:yi.a@intel.com] Sent: Wednesday, August 13, 2014 1:57 PM To: user@hadoop.apache.org Subject: RE: Implementing security in hadoop OK, it's still under devel

Re: Started learning Hadoop. Which distribution is best for native install in pseudo distributed mode?

Why don't you just use the apache tarball? We even have that automated, if vagrant is your thing: https://github.com/Cascading/vagrant-cascading-hadoop-cluster - André On Tue, Aug 12, 2014 at 10:12 PM, mani kandan wrote: > Which distribution are you people using? Cloudera vs Hortonworks vs > B

Test read caching

Hi, I'm trying to test Read Cache in HDFS on version 2.4.0. I followed this article without success : http://blog.cloudera.com/blog/2014/08/new-in-cdh-5-1-hdfs-read-caching/ I made this cmds : > hadoop-2.4.0/bin/hdfs cacheadmin -addPool testPool > hadoop-2.4.0/bin/hdfs cacheadmin -addDirecti

Re: Negative value given by getVirtualCores() or getAvailableResources()

Hi Wangda, I was actually wondering why should it give me -ve value for vcores when I call getAvailableResources(). Thanks, Kishore On Tue, Aug 12, 2014 at 12:50 PM, Wangda Tan wrote: > By default, vcore = 1 for each resource request. If you don't like this > behavior, you can set yarn.sche

Re: 100% CPU consumption by Resource Manager process

Hi Wangda, Thanks for the reply, here are the details, please see if you could suggest anything. 1) Number of nodes and running app in the cluster 2 nodes, and I am running my own application that keeps asking for containers, a) running something on the containers, b) releasing the containers, c

RE: Implementing security in hadoop

OK, it's still under development (https://issues.apache.org/jira/browse/HDFS-6134) and close to completion, maybe you can use it in near future. Regards, Yi Liu From: Chhaya Vishwakarma [mailto:chhaya.vishwaka...@lntinfotech.com] Sent: Wednesday, August 13, 2014 3:30 PM To: user@hadoop.apache.o

Re: Pseudo -distributed mode

Probably you don't understand meaning of this phrase. There are 3 ways to configure hadoop: 1. local mode - when you install hadoop you have empty configs, you have no daemon processes. In this case your file system would be used instead of HDFS, map-reduce jobs would be processed in he s

Re: MR AppMaster unable to load native libs

hi Susheel, Thanks for your input. I did build the libs for 64 bit but, still the problem was there. Though the problem is resolved now. I had to configure the property: yarn.app.mapreduce.am.env Cheers, Subroto Sanyal On 13 Aug 2014, at 10:39, Susheel Kumar Gadalay wrote: > This message I hav

RE: Implementing security in hadoop

Yea you are right I need encryption for data-at-rest From: Liu, Yi A [mailto:yi.a@intel.com] Sent: Wednesday, August 13, 2014 12:46 PM To: user@hadoop.apache.org Subject: RE: Implementing security in hadoop Hi Chhaya, >>> I have looked into Kerboroes but it doesn't provide encryption for dat

RE: Implementing security in hadoop

Hi Chhaya, >>> I have looked into Kerboroes but it doesn't provide encryption for data >>> already residing in HDFS. For encryption for data, I suppose you mean data at rest encryption (*not* encryption for data transport which is already supported), this feature is still under development and