setLocalResources() on ContainerLaunchContext

Hi, Can someone please tell me what is the use of calling setLocalResources() on ContainerLaunchContext? And, also an example of how to use this will help... I couldn't guess what is the String in the map that is passed to setLocalResources() like below: // Set the local resources

Re: setLocalResources() on ContainerLaunchContext

The string for each LocalResource in the map can be anything that serves as a common identifier name for your application. At execution time, the passed resource filename will be aliased to the name you've mapped it to, so that the application code need not track special names. The behavior is very

Re: about replication

Hello Irfan, You can find all the answers from HDFS architecture guide. See the section Data Organizationin particular for this question. Warm Regards, Tariq cloudfront.

Re: metics v1 in hadoop-2.0.5

There is hadoop-metrics.properties file in etc/hadoop directory. I config the file with below content: dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 dfs.period=10 dfs.servers=dw74:8649 But the configuration does not work. Do I only use metrics v2 in hadoop-2.0.5? 2013/8/5 lei

Re: metics v1 in hadoop-2.0.5

Yes. The daemons all use metrics-v2 in Apache Hadoop 2.x. On Mon, Aug 5, 2013 at 3:56 PM, lei liu wrote: > There is hadoop-metrics.properties file in etc/hadoop directory. > I config the file with below content: > dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 > dfs.period=10 >

Re: metics v1 in hadoop-2.0.5

metrics v1 is deprecated, both in 1.2 and 2.x. the existence of hadoop-metrics.properties is confusing, I think it should be removed. On Mon, Aug 5, 2013 at 6:26 PM, lei liu wrote: > There is hadoop-metrics.properties file in etc/hadoop directory. > I config the file with below content: > df

Re: about replication

thanks mohammad i ran the below command on NameNode $ ./hadoop dfs -mkdir /wksp and the "wksp" dir got created in c:\ ( as i have windows environment) now , when i log in to one of the DataNode , then i am not able to see c:\wksp any issue ? please suggest regards On Mon, Aug 5, 2013 at 3

Re: about replication

You cannot physically see the HDFS files and directories through local FS. Either use HDFS shell or HDFS webUI(namenode_machine:50070). Warm Regards, Tariq cloudfront.blogspot.com On Mon, Aug 5, 2013 at 4:46 PM, Irfan Sayed wrote: > thanks mohammad > i ran the below command on NameNode > > $ .

Re: about replication

thanks. please refer below: Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin $ ./hadoop dfs -ls /wksp Found 1 items drwxr-xr-x - Administrator Domain 0 2013-08-05 16:58 /wksp/New folder Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin $ same command if i r

Re: about replication

You can not physically access the datanode.You have to understand it to logically and it really happens. Type "jps" command to check ur datanode was started or not. when user stores the file into hdfs ,the request is goes to datanode and datanode will divide the file into number of blocks. Each b

Re: setLocalResources() on ContainerLaunchContext

Hi Harsh, Thanks for the quick and detailed reply, it really helps. I am trying to use it and getting this error in node manager's log: 2013-08-05 08:57:28,867 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundExcept

Hadoop Essentials: a Quantitative Approach - PDF book for purchase

Hello Hadoop Community, I am looking to buy a PDF version of the following book. Hadoop Essentials: a Quantitative Approach by Henry Liu I only find a hard copy. Can anyone tell where can I buy the PDF version of this book?   Thanks, Raj

Large-scale collection of logs from multiple Hadoop nodes

Hi... I am facing a large-scale usage scenario of log collection from a Hadoop cluster and examining ways as to how it should be implemented. More specifically, imagine a cluster that has hundreds of nodes, each of which constantly produces Syslog events that need to be gathered an analyzed at an

Re: setLocalResources() on ContainerLaunchContext

The detail is insufficient to answer why. You should also have gotten a trace after it, can you post that? If possible, also the relevant snippets of code. On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri wrote: > Hi Harsh, > Thanks for the quick and detailed reply, it really helps. I am

Re: Large-scale collection of logs from multiple Hadoop nodes

We have been using a flume like system for such usecases at significantly large scale and it has been working quite well. Would like to hear thoughts/challenges around using zeromq alike systems at good enough scale. inder "you are the average of 5 people you spend the most time with" On Aug 5, 2

Re: about replication

thanks. i verified, datanode is up and running i ran the below command: Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin $ ./hadoop dfs -copyFromLocal C:\\Users\\Administrator\\Desktop\\hadoop-1.1.2.tar /wksp copyFromLocal: File C:/Users/Administrator/Desktop/hadoop-1.1.2.tar does n

Re: setLocalResources() on ContainerLaunchContext

Hi Harsh, Please see if this is useful, I got a stack trace after the error has occurred 2013-08-06 00:55:30,559 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to /tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004 = file:/tmp/nm-local

Re: Large-scale collection of logs from multiple Hadoop nodes

We have similar requirements and build our log collection system around RSyslog and Flume. It is not in production yet, but tests so far look pretty well. We rejected idea of using AMQP since it introduces large overhead for log events. Probably you can use Flume interceptors to do real-time proce

Re: about replication

hello, I think u r newbie.You need to work on to learn from scratch. Please do not mind.whatever u try to think hadoop should be like this, ur concept is totally wrong. but,your effort is positive. My suggestion is try to learn on ubuntu os.Windows is not good enough for hadoop. Follow the link

throughput metrics in hadoop-2.0.5

I use hadoop-2.0.5 and config hadoop-metrics2.properties file with below content. *.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31 *.sink.ganglia.period=10 *.sink.ganglia.supportsparse=true namenode.sink.ganglia.servers=10.232.98.74:8649 datanode.sink.ganglia.servers=10.23

Re: about replication

thanks. yes , i am newbie. however, i need windows setup. let me surely refer the doc and link which u sent but i need this to be working ... can you please help regards

Re: Large-scale collection of logs from multiple Hadoop nodes

Andrei's flume interceptor mention reminds me of James Kinley's Top-N example on his flume-interceptor-analytics GH repo at https://github.com/jrkinley/flume-interceptor-analytics#the-streaming-topn-example On Tue, Aug 6, 2013 at 11:41 AM, Andrei wrote: > We have similar requirements and build ou

issue about try lzo in hadoop

i use yarn ,and hadoop version is CDH4.3, lzo installed,but when i run test ,it failed,why? # sudo -u hdfs hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar com.hadoop.compression.lzo.LzoIndexer /test 13/08/06 22:28:03 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 13/08

Re: about replication

hi, i have gone high level through the doc and seems very promising. really nice but can you please help me on this issue . i need to do PoC first and get the demo out regards On Tue, Aug 6, 2013 at 11:49 AM, Irfan Sayed wrote: > thanks. yes , i am newbie. > however, i need windows setup. >

MutableRate metrics in hadoop-2.0.5

There is code in MutableRate class: public synchronized void snapshot(MetricsRecordBuilder builder, boolean all) { if (all || changed()) { numSamples += intervalStat.numSamples(); builder.addCounter(numInfo, numSamples) .addGauge(avgInfo, lastStat().mean()); *i