Re: YarnException: Unauthorized request to start container. This token is expired.

Can you check few things? What is the container expiry interval set to? How many containers are getting allocated? Is there any reservation of the containers happening..? if yes then that was a known problem...I don't remember the jira number though... Underlying problem in case of reservation was

Re: Command line tools for YARN

if it is not present in CLI then writing a simple wrapper around yarn -web services will be a option. I think all your listed requirements will be covered with web services though On Thu, Dec 26, 2013 at 11:28 AM, Jian He j...@hortonworks.com wrote: 1) checking how many nodes are in my cluster

Re: Container exit status for released container

I think this is tricky in the sense if application returns with same exit code then I am not sure if we differentiate between the two. But can be used as defaults. On Thu, Dec 26, 2013 at 11:24 AM, Jian He j...@hortonworks.com wrote: Not true, you can look at ContainerExitStatus.java which

Re: YARN: LocalResources and file distribution

add this file in the files to be localized. (LocalResourceRequest). and then refer it as ./list.ksh .. While adding this to LocalResource specify the path which you have mentioned. On Thu, Dec 5, 2013 at 10:40 PM, Krishna Kishore Bonagiri write2kish...@gmail.com wrote: Hi Arun, I have

Re: In YARN, how does a task tracker knows the address of a job tracker?

thishttp://www.slideshare.net/ovjforu/yarn-way-to-share-cluster-beyond (slide 12) for how job actually gets executed. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Thu, Nov 21, 2013 at 7:52 AM, ricky l rickylee0...@gmail.com wrote: Hi all, I have a question of how a task

Re: Limit on total jobs running using fair scheduler

MRv2 :) Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Thu, Nov 21, 2013 at 1:46 PM, Ivan Tretyakov itretya...@griddynamics.com wrote: Thank you for your replies! We are using MR version 1 and my question is regarding this version. Omkar, are you talking about MR1

Re: Limit on total jobs running using fair scheduler

Not sure about the fair scheduler but in capacity scheduler you can achieve this by controlling number of jobs/applications per queue. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov itretya...@griddynamics.com wrote: Hello

Re: DefaultResourceCalculator ClassNotFoundException

. DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses dominant-resource to compare multi-dimensional resources such as Memory, CPU etc. /description /property inside capacity-scheduler.xml Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com

Re: Cannot start resourcemanager

to compare Resources in the scheduler. The default i.e. DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses dominant-resource to compare multi-dimensional resources such as Memory, CPU etc. /description /property Thanks, Omkar Joshi

Re: Problem: org.apache.hadoop.mapred.ReduceTask: java.net.SocketTimeoutException: connect timed out

what was the problem? oozie problem or machine was getting rebooted where map task ran before? Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Thu, Apr 18, 2013 at 12:47 PM, Som Satpathy somsatpa...@gmail.comwrote: Never mind, got it fixed. Thanks, Som On Tue, Apr

Re: Can container requests be made paralelly from multiple threads

Hi, I suggest you should not do that. After YARN-744 goes in this will be prevented on RM side. May I know why you want to do this? any advantage/ use case? Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Sep 27, 2013 at 8:31 AM, Krishna Kishore Bonagiri write2kish

Re: Can container requests be made paralelly from multiple threads

a request. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Sep 27, 2013 at 11:14 AM, Krishna Kishore Bonagiri write2kish...@gmail.com wrote: Hi Omkar, Thanks for the quick reply. I have a requirement for sets of containers depending on some of my business logic. I

Re: Distributed cache in command line

Hi, I have no idea about RHadoop but in general in YARN we do create symlinks for the files in distributed cache in the current working directory of every container. You may be able to use that somehow. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Mon, Sep 23, 2013 at 6

Re: How to make hadoop use all nodes?

running 2, few running 1 where as few with 0 containers. I hope it clarifies your doubt. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Thu, Sep 19, 2013 at 7:19 AM, Vandecreme, Antoine antoine.vandecr...@nist.gov wrote: Hi all, I am working with Hadoop 2.0.5 (I plan

Re: ContainerLaunchContext in 2.1.x

) User 4) NodeId It is present there by default (irrespective of security). I hope it answers your doubt. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Wed, Sep 4, 2013 at 2:51 AM, Janne Valkealahti janne.valkeala...@gmail.com wrote: With 2.0.x ContainerId was part

Re: Cache file conflict

you should check this https://issues.apache.org/jira/browse/MAPREDUCE-4493?focusedCommentId=13713706page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13713706 Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Thu, Aug 29, 2013 at 5:06 PM, Public

RE: Hadoop - impersonation doubts/issues while accessing from remote machine

Thanks :) Regards, Omkar Joshi -Original Message- From: Harsh J [mailto:ha...@cloudera.com] Sent: Friday, August 23, 2013 3:52 PM To: user@hadoop.apache.org Subject: Re: Hadoop - impersonation doubts/issues while accessing from remote machine I've answered this on the stackoverflow

Hadoop - impersonation doubts/issues while accessing from remote machine

(WordCounter.java:58) Regards, Omkar Joshi The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information

Re: Method Calls

. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Tue, Aug 20, 2013 at 2:58 PM, Rajesh Jain rjai...@gmail.com wrote: I am looking for the various method calls (entry points) in the inter tier communication. When the client submits a job for map-reduce, what is the sequence

Re: setLocalResources() on ContainerLaunchContext

location /kishore/kk.sh? is /kishore and /kishore/kk.sh accessible to the user who is making startContainer call or the one running AM container? Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Tue, Aug 6, 2013 at 10:43 PM, Krishna Kishore Bonagiri write2kish...@gmail.com wrote

Re: setLocalResources() on ContainerLaunchContext

Hi, You need to match the timestamp. Probably get the timestamp locally before adding it. This is explicitly done to ensure that file is not updated after user makes the call to avoid possible errors. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Tue, Aug 6, 2013 at 5

Re: DistributedCache incompatibility issue between 1.0 and 2.0

check https://issues.apache.org/jira/browse/MAPREDUCE-4493 and https://issues.apache.org/jira/browse/YARN-916 Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Jul 19, 2013 at 8:12 AM, Ted Yu yuzhih...@gmail.com wrote: See this thread also: http://search-hadoop.com/m

Re: New Distributed Cache

. context.getLocalCacheFiles .. will give you the actual file path on node manager... (after file is localized). Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Thu, Jul 11, 2013 at 8:19 AM, Botelho, Andrew andrew.bote...@emc.comwrote: So in my driver code, I try to store the file in the cache

Re: New Distributed Cache

did you try JobContext.getCacheFiles() ? Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Wed, Jul 10, 2013 at 10:15 AM, Botelho, Andrew andrew.bote...@emc.comwrote: Hi, ** ** I am trying to store a file in the Distributed Cache during my Hadoop job

Re: Distributed Cache

try JobContext.getCacheFiles() Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Wed, Jul 10, 2013 at 6:31 AM, Botelho, Andrew andrew.bote...@emc.comwrote: Ok using job.addCacheFile() seems to compile correctly. However, how do I then access the cached file in my

Re: can not start yarn

probably you should run jps everytime you start/stop NM/RM. just for you to know whether RM/NM started/stopped successfully or not. devaraj is right.. try checking RM logs.. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Tue, Jul 9, 2013 at 8:20 PM, Devaraj k devara

Re: ConnectionException in container, happens only sometimes

can you post RM/NM logs too.? Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Wed, Jul 10, 2013 at 6:42 AM, Andrei faithlessfri...@gmail.com wrote: If it helps, full log of AM can be found herehttp://pastebin.com/zXTabyvv . On Wed, Jul 10, 2013 at 4:21 PM, Andrei

Re: Distributed Cache

the uri used for localizing files... (original uri used for adding it to cache). getLocalCacheFiles .. will give you the actual file path on node manager. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Wed, Jul 10, 2013 at 2:43 PM, Botelho, Andrew andrew.bote...@emc.comwrote

Re: could only be replicated to 0 nodes instead of minReplication exception during job execution

is as file creation in hdfs is failing it is marking that reduce attempt as failure and restarting it. Keep checking namenode state when it reaches 67%. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Mon, Jun 24, 2013 at 3:01 PM, Yuzhang Han yuzhanghan1...@gmail.comwrote

Re: Compile Just a Subproject

(with updated time stamp) or not in target directory.* Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Jun 21, 2013 at 3:12 PM, Curtis Ullerich curtuller...@gmail.comwrote: I've executed the commands as you've said, and the jar that is run is unchanged. Here's exactly

Reducer not getting called

In the map method, image is mystic.JPG mystic.JPG is taken using K750i Returning from the map method But nothing is reflected in stdout logs of the reducer. What have I missed? Regards, Omkar Joshi The contents of this e-mail and any attachment(s) may contain

RE: Reducer not getting called

Ok but that link is broken - can you provide a working one? Regards, Omkar Joshi -Original Message- From: Harsh J [mailto:ha...@cloudera.com] Sent: Thursday, June 13, 2013 11:01 AM To: user@hadoop.apache.org Subject: Re: Reducer not getting called You're not using the recommended

Re: Warnings?

then you can try rebuilding your hadoop installation mvn clean install -Pnative -Pdist -Dtar -DskipTests Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Mon, Apr 29, 2013 at 11:19 AM, Kevin Burton rkevinbur...@charter.netwrote: If it doesn't work what are my options

Re: M/R Staticstics

Have you enabled security? can you share the output for your hdfs? bin/hadoop fs -ls / and is /tmp/hadoop-yarn/staging/history/done directory present in hdfs ? if so then what permissions? also please share exception stack trace... Thanks, Omkar Joshi Hortonworks Inc On Fri, Apr 26, 2013