setLocalResources() on ContainerLaunchContext
Hi, Can someone please tell me what is the use of calling setLocalResources() on ContainerLaunchContext? And, also an example of how to use this will help... I couldn't guess what is the String in the map that is passed to setLocalResources() like below: // Set the local resources Map localResources = new HashMap(); Thanks, Kishore
Re: setLocalResources() on ContainerLaunchContext
The string for each LocalResource in the map can be anything that serves as a common identifier name for your application. At execution time, the passed resource filename will be aliased to the name you've mapped it to, so that the application code need not track special names. The behavior is very similar to how you can, in MR, define a symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). For an example, checkout the DistributedShell app sources. Over [1], you can see we take a user provided file path to a shell script. This can be named anything as it is user-supplied. Onto [2], we define this as a local resource [2.1] and embed it with a different name (the string you ask about) [2.2], as defined at [3] as an application reference-able constant. Note that in [4], we add to the Container arguments the aliased name we mapped it to (i.e. [3]) and not the original filename we received from the user. The resource is placed on the container with this name instead, so thats what we choose to execute. [1] - https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 [2] - [2.1] https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 and [2.2] https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 [3] - https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 [4] - https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri wrote: > Hi, > > Can someone please tell me what is the use of calling setLocalResources() > on ContainerLaunchContext? > > And, also an example of how to use this will help... > > I couldn't guess what is the String in the map that is passed to > setLocalResources() like below: > > // Set the local resources > Map localResources = new HashMap LocalResource>(); > > Thanks, > Kishore > -- Harsh J
Re: setLocalResources() on ContainerLaunchContext
Hi Harsh, Thanks for the quick and detailed reply, it really helps. I am trying to use it and getting this error in node manager's log: 2013-08-05 08:57:28,867 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://isredeng/kishore/kk.ksh This file is there on the machine with name "isredeng", I could do ls for that file as below: -bash-4.1$ hadoop fs -ls kishore/kk.ksh 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 1 items -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48 kishore/kk.ksh Note: I am using a single node cluster Thanks, Kishore On Mon, Aug 5, 2013 at 3:00 PM, Harsh J wrote: > The string for each LocalResource in the map can be anything that > serves as a common identifier name for your application. At execution > time, the passed resource filename will be aliased to the name you've > mapped it to, so that the application code need not track special > names. The behavior is very similar to how you can, in MR, define a > symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). > > For an example, checkout the DistributedShell app sources. > > Over [1], you can see we take a user provided file path to a shell > script. This can be named anything as it is user-supplied. > Onto [2], we define this as a local resource [2.1] and embed it with a > different name (the string you ask about) [2.2], as defined at [3] as > an application reference-able constant. > Note that in [4], we add to the Container arguments the aliased name > we mapped it to (i.e. [3]) and not the original filename we received > from the user. The resource is placed on the container with this name > instead, so thats what we choose to execute. > > [1] - > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 > > [2] - [2.1] > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 > and [2.2] > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 > > [3] - > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 > > [4] - > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 > > On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri > wrote: > > Hi, > > > > Can someone please tell me what is the use of calling > setLocalResources() > > on ContainerLaunchContext? > > > > And, also an example of how to use this will help... > > > > I couldn't guess what is the String in the map that is passed to > > setLocalResources() like below: > > > > // Set the local resources > > Map localResources = new HashMap > LocalResource>(); > > > > Thanks, > > Kishore > > > > > > -- > Harsh J >
Re: setLocalResources() on ContainerLaunchContext
The detail is insufficient to answer why. You should also have gotten a trace after it, can you post that? If possible, also the relevant snippets of code. On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri wrote: > Hi Harsh, > Thanks for the quick and detailed reply, it really helps. I am trying to > use it and getting this error in node manager's log: > > 2013-08-05 08:57:28,867 ERROR > org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException > as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not > exist: hdfs://isredeng/kishore/kk.ksh > > > This file is there on the machine with name "isredeng", I could do ls for > that file as below: > > -bash-4.1$ hadoop fs -ls kishore/kk.ksh > 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48 kishore/kk.ksh > > Note: I am using a single node cluster > > Thanks, > Kishore > > > > > On Mon, Aug 5, 2013 at 3:00 PM, Harsh J wrote: >> >> The string for each LocalResource in the map can be anything that >> serves as a common identifier name for your application. At execution >> time, the passed resource filename will be aliased to the name you've >> mapped it to, so that the application code need not track special >> names. The behavior is very similar to how you can, in MR, define a >> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). >> >> For an example, checkout the DistributedShell app sources. >> >> Over [1], you can see we take a user provided file path to a shell >> script. This can be named anything as it is user-supplied. >> Onto [2], we define this as a local resource [2.1] and embed it with a >> different name (the string you ask about) [2.2], as defined at [3] as >> an application reference-able constant. >> Note that in [4], we add to the Container arguments the aliased name >> we mapped it to (i.e. [3]) and not the original filename we received >> from the user. The resource is placed on the container with this name >> instead, so thats what we choose to execute. >> >> [1] - >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 >> >> [2] - [2.1] >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 >> and [2.2] >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 >> >> [3] - >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 >> >> [4] - >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 >> >> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri >> wrote: >> > Hi, >> > >> > Can someone please tell me what is the use of calling >> > setLocalResources() >> > on ContainerLaunchContext? >> > >> > And, also an example of how to use this will help... >> > >> > I couldn't guess what is the String in the map that is passed to >> > setLocalResources() like below: >> > >> > // Set the local resources >> > Map localResources = new HashMap> > LocalResource>(); >> > >> > Thanks, >> > Kishore >> > >> >> >> >> -- >> Harsh J > > -- Harsh J
Re: setLocalResources() on ContainerLaunchContext
ith name "isredeng", I could do ls for > > that file as below: > > > > -bash-4.1$ hadoop fs -ls kishore/kk.ksh > > 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load > native-hadoop > > library for your platform... using builtin-java classes where applicable > > Found 1 items > > -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48 > kishore/kk.ksh > > > > Note: I am using a single node cluster > > > > Thanks, > > Kishore > > > > > > > > > > On Mon, Aug 5, 2013 at 3:00 PM, Harsh J wrote: > >> > >> The string for each LocalResource in the map can be anything that > >> serves as a common identifier name for your application. At execution > >> time, the passed resource filename will be aliased to the name you've > >> mapped it to, so that the application code need not track special > >> names. The behavior is very similar to how you can, in MR, define a > >> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). > >> > >> For an example, checkout the DistributedShell app sources. > >> > >> Over [1], you can see we take a user provided file path to a shell > >> script. This can be named anything as it is user-supplied. > >> Onto [2], we define this as a local resource [2.1] and embed it with a > >> different name (the string you ask about) [2.2], as defined at [3] as > >> an application reference-able constant. > >> Note that in [4], we add to the Container arguments the aliased name > >> we mapped it to (i.e. [3]) and not the original filename we received > >> from the user. The resource is placed on the container with this name > >> instead, so thats what we choose to execute. > >> > >> [1] - > >> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 > >> > >> [2] - [2.1] > >> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 > >> and [2.2] > >> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 > >> > >> [3] - > >> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 > >> > >> [4] - > >> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 > >> > >> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri > >> wrote: > >> > Hi, > >> > > >> > Can someone please tell me what is the use of calling > >> > setLocalResources() > >> > on ContainerLaunchContext? > >> > > >> > And, also an example of how to use this will help... > >> > > >> > I couldn't guess what is the String in the map that is passed to > >> > setLocalResources() like below: > >> > > >> > // Set the local resources > >> > Map localResources = new HashMap >> > LocalResource>(); > >> > > >> > Thanks, > >> > Kishore > >> > > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > -- > Harsh J >
Re: setLocalResources() on ContainerLaunchContext
thing else. > > Thanks, > Kishore > > > > On Tue, Aug 6, 2013 at 12:05 AM, Harsh J wrote: >> >> The detail is insufficient to answer why. You should also have gotten >> a trace after it, can you post that? If possible, also the relevant >> snippets of code. >> >> On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri >> wrote: >> > Hi Harsh, >> > Thanks for the quick and detailed reply, it really helps. I am trying >> > to >> > use it and getting this error in node manager's log: >> > >> > 2013-08-05 08:57:28,867 ERROR >> > org.apache.hadoop.security.UserGroupInformation: >> > PriviledgedActionException >> > as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does >> > not >> > exist: hdfs://isredeng/kishore/kk.ksh >> > >> > >> > This file is there on the machine with name "isredeng", I could do ls >> > for >> > that file as below: >> > >> > -bash-4.1$ hadoop fs -ls kishore/kk.ksh >> > 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load >> > native-hadoop >> > library for your platform... using builtin-java classes where applicable >> > Found 1 items >> > -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48 >> > kishore/kk.ksh >> > >> > Note: I am using a single node cluster >> > >> > Thanks, >> > Kishore >> > >> > >> > >> > >> > On Mon, Aug 5, 2013 at 3:00 PM, Harsh J wrote: >> >> >> >> The string for each LocalResource in the map can be anything that >> >> serves as a common identifier name for your application. At execution >> >> time, the passed resource filename will be aliased to the name you've >> >> mapped it to, so that the application code need not track special >> >> names. The behavior is very similar to how you can, in MR, define a >> >> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). >> >> >> >> For an example, checkout the DistributedShell app sources. >> >> >> >> Over [1], you can see we take a user provided file path to a shell >> >> script. This can be named anything as it is user-supplied. >> >> Onto [2], we define this as a local resource [2.1] and embed it with a >> >> different name (the string you ask about) [2.2], as defined at [3] as >> >> an application reference-able constant. >> >> Note that in [4], we add to the Container arguments the aliased name >> >> we mapped it to (i.e. [3]) and not the original filename we received >> >> from the user. The resource is placed on the container with this name >> >> instead, so thats what we choose to execute. >> >> >> >> [1] - >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 >> >> >> >> [2] - [2.1] >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 >> >> and [2.2] >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 >> >> >> >> [3] - >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 >> >> >> >> [4] - >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 >> >> >> >> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri >> >> wrote: >> >> > Hi, >> >> > >> >> > Can someone please tell me what is the use of calling >> >> > setLocalResources() >> >> > on ContainerLaunchContext? >> >> > >> >> > And, also an example of how to use this will help... >> >> > >> >> > I couldn't guess what is the String in the map that is passed to >> >> > setLocalResources() like below: >> >> > >> >> > // Set the local resources >> >> > Map localResources = new HashMap> >> > LocalResource>(); >> >> > >> >> > Thanks, >> >> > Kishore >> >> > >> >> >> >> >> >> >> >> -- >> >> Harsh J >> > >> > >> >> >> >> -- >> Harsh J > > -- Harsh J
Re: setLocalResources() on ContainerLaunchContext
; >> Note that in [4], we add to the Container arguments the aliased name > >> >> we mapped it to (i.e. [3]) and not the original filename we received > >> >> from the user. The resource is placed on the container with this name > >> >> instead, so thats what we choose to execute. > >> >> > >> >> [1] - > >> >> > >> >> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 > >> >> > >> >> [2] - [2.1] > >> >> > >> >> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 > >> >> and [2.2] > >> >> > >> >> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 > >> >> > >> >> [3] - > >> >> > >> >> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 > >> >> > >> >> [4] - > >> >> > >> >> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 > >> >> > >> >> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri > >> >> wrote: > >> >> > Hi, > >> >> > > >> >> > Can someone please tell me what is the use of calling > >> >> > setLocalResources() > >> >> > on ContainerLaunchContext? > >> >> > > >> >> > And, also an example of how to use this will help... > >> >> > > >> >> > I couldn't guess what is the String in the map that is passed to > >> >> > setLocalResources() like below: > >> >> > > >> >> > // Set the local resources > >> >> > Map localResources = new > HashMap >> >> > LocalResource>(); > >> >> > > >> >> > Thanks, > >> >> > Kishore > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Harsh J > >> > > >> > > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > -- > Harsh J >
Re: setLocalResources() on ContainerLaunchContext
a common identifier name for your application. At >> execution >> >> >> time, the passed resource filename will be aliased to the name >> you've >> >> >> mapped it to, so that the application code need not track special >> >> >> names. The behavior is very similar to how you can, in MR, define a >> >> >> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). >> >> >> >> >> >> For an example, checkout the DistributedShell app sources. >> >> >> >> >> >> Over [1], you can see we take a user provided file path to a shell >> >> >> script. This can be named anything as it is user-supplied. >> >> >> Onto [2], we define this as a local resource [2.1] and embed it >> with a >> >> >> different name (the string you ask about) [2.2], as defined at [3] >> as >> >> >> an application reference-able constant. >> >> >> Note that in [4], we add to the Container arguments the aliased name >> >> >> we mapped it to (i.e. [3]) and not the original filename we received >> >> >> from the user. The resource is placed on the container with this >> name >> >> >> instead, so thats what we choose to execute. >> >> >> >> >> >> [1] - >> >> >> >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 >> >> >> >> >> >> [2] - [2.1] >> >> >> >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 >> >> >> and [2.2] >> >> >> >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 >> >> >> >> >> >> [3] - >> >> >> >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 >> >> >> >> >> >> [4] - >> >> >> >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 >> >> >> >> >> >> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri >> >> >> wrote: >> >> >> > Hi, >> >> >> > >> >> >> > Can someone please tell me what is the use of calling >> >> >> > setLocalResources() >> >> >> > on ContainerLaunchContext? >> >> >> > >> >> >> > And, also an example of how to use this will help... >> >> >> > >> >> >> > I couldn't guess what is the String in the map that is passed to >> >> >> > setLocalResources() like below: >> >> >> > >> >> >> > // Set the local resources >> >> >> > Map localResources = new >> HashMap> >> >> > LocalResource>(); >> >> >> > >> >> >> > Thanks, >> >> >> > Kishore >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Harsh J >> >> > >> >> > >> >> >> >> >> >> >> >> -- >> >> Harsh J >> > >> > >> >> >> >> -- >> Harsh J >> > >
Re: setLocalResources() on ContainerLaunchContext
ndException: File >>> does >>> >> > not >>> >> > exist: hdfs://isredeng/kishore/kk.ksh >>> >> > >>> >> > >>> >> > This file is there on the machine with name "isredeng", I could do >>> ls >>> >> > for >>> >> > that file as below: >>> >> > >>> >> > -bash-4.1$ hadoop fs -ls kishore/kk.ksh >>> >> > 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load >>> >> > native-hadoop >>> >> > library for your platform... using builtin-java classes where >>> applicable >>> >> > Found 1 items >>> >> > -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48 >>> >> > kishore/kk.ksh >>> >> > >>> >> > Note: I am using a single node cluster >>> >> > >>> >> > Thanks, >>> >> > Kishore >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > On Mon, Aug 5, 2013 at 3:00 PM, Harsh J wrote: >>> >> >> >>> >> >> The string for each LocalResource in the map can be anything that >>> >> >> serves as a common identifier name for your application. At >>> execution >>> >> >> time, the passed resource filename will be aliased to the name >>> you've >>> >> >> mapped it to, so that the application code need not track special >>> >> >> names. The behavior is very similar to how you can, in MR, define a >>> >> >> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). >>> >> >> >>> >> >> For an example, checkout the DistributedShell app sources. >>> >> >> >>> >> >> Over [1], you can see we take a user provided file path to a shell >>> >> >> script. This can be named anything as it is user-supplied. >>> >> >> Onto [2], we define this as a local resource [2.1] and embed it >>> with a >>> >> >> different name (the string you ask about) [2.2], as defined at [3] >>> as >>> >> >> an application reference-able constant. >>> >> >> Note that in [4], we add to the Container arguments the aliased >>> name >>> >> >> we mapped it to (i.e. [3]) and not the original filename we >>> received >>> >> >> from the user. The resource is placed on the container with this >>> name >>> >> >> instead, so thats what we choose to execute. >>> >> >> >>> >> >> [1] - >>> >> >> >>> >> >> >>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 >>> >> >> >>> >> >> [2] - [2.1] >>> >> >> >>> >> >> >>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 >>> >> >> and [2.2] >>> >> >> >>> >> >> >>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 >>> >> >> >>> >> >> [3] - >>> >> >> >>> >> >> >>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 >>> >> >> >>> >> >> [4] - >>> >> >> >>> >> >> >>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 >>> >> >> >>> >> >> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri >>> >> >> wrote: >>> >> >> > Hi, >>> >> >> > >>> >> >> > Can someone please tell me what is the use of calling >>> >> >> > setLocalResources() >>> >> >> > on ContainerLaunchContext? >>> >> >> > >>> >> >> > And, also an example of how to use this will help... >>> >> >> > >>> >> >> > I couldn't guess what is the String in the map that is passed to >>> >> >> > setLocalResources() like below: >>> >> >> > >>> >> >> > // Set the local resources >>> >> >> > Map localResources = new >>> HashMap>> >> >> > LocalResource>(); >>> >> >> > >>> >> >> > Thanks, >>> >> >> > Kishore >>> >> >> > >>> >> >> >>> >> >> >>> >> >> >>> >> >> -- >>> >> >> Harsh J >>> >> > >>> >> > >>> >> >>> >> >>> >> >>> >> -- >>> >> Harsh J >>> > >>> > >>> >>> >>> >>> -- >>> Harsh J >>> >> >>
Re: setLocalResources() on ContainerLaunchContext
hing else. >>>> > >>>> > Thanks, >>>> > Kishore >>>> > >>>> > >>>> > >>>> > On Tue, Aug 6, 2013 at 12:05 AM, Harsh J wrote: >>>> >> >>>> >> The detail is insufficient to answer why. You should also have gotten >>>> >> a trace after it, can you post that? If possible, also the relevant >>>> >> snippets of code. >>>> >> >>>> >> On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri >>>> >> wrote: >>>> >> > Hi Harsh, >>>> >> > Thanks for the quick and detailed reply, it really helps. I am >>>> trying >>>> >> > to >>>> >> > use it and getting this error in node manager's log: >>>> >> > >>>> >> > 2013-08-05 08:57:28,867 ERROR >>>> >> > org.apache.hadoop.security.UserGroupInformation: >>>> >> > PriviledgedActionException >>>> >> > as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File >>>> does >>>> >> > not >>>> >> > exist: hdfs://isredeng/kishore/kk.ksh >>>> >> > >>>> >> > >>>> >> > This file is there on the machine with name "isredeng", I could do >>>> ls >>>> >> > for >>>> >> > that file as below: >>>> >> > >>>> >> > -bash-4.1$ hadoop fs -ls kishore/kk.ksh >>>> >> > 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load >>>> >> > native-hadoop >>>> >> > library for your platform... using builtin-java classes where >>>> applicable >>>> >> > Found 1 items >>>> >> > -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48 >>>> >> > kishore/kk.ksh >>>> >> > >>>> >> > Note: I am using a single node cluster >>>> >> > >>>> >> > Thanks, >>>> >> > Kishore >>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> >> > On Mon, Aug 5, 2013 at 3:00 PM, Harsh J >>>> wrote: >>>> >> >> >>>> >> >> The string for each LocalResource in the map can be anything that >>>> >> >> serves as a common identifier name for your application. At >>>> execution >>>> >> >> time, the passed resource filename will be aliased to the name >>>> you've >>>> >> >> mapped it to, so that the application code need not track special >>>> >> >> names. The behavior is very similar to how you can, in MR, define >>>> a >>>> >> >> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). >>>> >> >> >>>> >> >> For an example, checkout the DistributedShell app sources. >>>> >> >> >>>> >> >> Over [1], you can see we take a user provided file path to a shell >>>> >> >> script. This can be named anything as it is user-supplied. >>>> >> >> Onto [2], we define this as a local resource [2.1] and embed it >>>> with a >>>> >> >> different name (the string you ask about) [2.2], as defined at >>>> [3] as >>>> >> >> an application reference-able constant. >>>> >> >> Note that in [4], we add to the Container arguments the aliased >>>> name >>>> >> >> we mapped it to (i.e. [3]) and not the original filename we >>>> received >>>> >> >> from the user. The resource is placed on the container with this >>>> name >>>> >> >> instead, so thats what we choose to execute. >>>> >> >> >>>> >> >> [1] - >>>> >> >> >>>> >> >> >>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 >>>> >> >> >>>> >> >> [2] - [2.1] >>>> >> >> >>>> >> >> >>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 >>>> >> >> and [2.2] >>>> >> >> >>>> >> >> >>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 >>>> >> >> >>>> >> >> [3] - >>>> >> >> >>>> >> >> >>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 >>>> >> >> >>>> >> >> [4] - >>>> >> >> >>>> >> >> >>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 >>>> >> >> >>>> >> >> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri >>>> >> >> wrote: >>>> >> >> > Hi, >>>> >> >> > >>>> >> >> > Can someone please tell me what is the use of calling >>>> >> >> > setLocalResources() >>>> >> >> > on ContainerLaunchContext? >>>> >> >> > >>>> >> >> > And, also an example of how to use this will help... >>>> >> >> > >>>> >> >> > I couldn't guess what is the String in the map that is passed >>>> to >>>> >> >> > setLocalResources() like below: >>>> >> >> > >>>> >> >> > // Set the local resources >>>> >> >> > Map localResources = new >>>> HashMap>>> >> >> > LocalResource>(); >>>> >> >> > >>>> >> >> > Thanks, >>>> >> >> > Kishore >>>> >> >> > >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> -- >>>> >> >> Harsh J >>>> >> > >>>> >> > >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Harsh J >>>> > >>>> > >>>> >>>> >>>> >>>> -- >>>> Harsh J >>>> >>> >>> >
Re: setLocalResources() on ContainerLaunchContext
gt;> > This file is there on the machine with name "isredeng", I could do ls > >> > for > >> > that file as below: > >> > > >> > -bash-4.1$ hadoop fs -ls kishore/kk.ksh > >> > 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load > >> > native-hadoop > >> > library for your platform... using builtin-java classes where applicable > >> > Found 1 items > >> > -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48 > >> > kishore/kk.ksh > >> > > >> > Note: I am using a single node cluster > >> > > >> > Thanks, > >> > Kishore > >> > > >> > > >> > > >> > > >> > On Mon, Aug 5, 2013 at 3:00 PM, Harsh J wrote: > >> >> > >> >> The string for each LocalResource in the map can be anything that > >> >> serves as a common identifier name for your application. At execution > >> >> time, the passed resource filename will be aliased to the name you've > >> >> mapped it to, so that the application code need not track special > >> >> names. The behavior is very similar to how you can, in MR, define a > >> >> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). > >> >> > >> >> For an example, checkout the DistributedShell app sources. > >> >> > >> >> Over [1], you can see we take a user provided file path to a shell > >> >> script. This can be named anything as it is user-supplied. > >> >> Onto [2], we define this as a local resource [2.1] and embed it with a > >> >> different name (the string you ask about) [2.2], as defined at [3] as > >> >> an application reference-able constant. > >> >> Note that in [4], we add to the Container arguments the aliased name > >> >> we mapped it to (i.e. [3]) and not the original filename we received > >> >> from the user. The resource is placed on the container with this name > >> >> instead, so thats what we choose to execute. > >> >> > >> >> [1] - > >> >> > >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 > >> >> > >> >> [2] - [2.1] > >> >> > >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 > >> >> and [2.2] > >> >> > >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 > >> >> > >> >> [3] - > >> >> > >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 > >> >> > >> >> [4] - > >> >> > >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 > >> >> > >> >> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri > >> >> wrote: > >> >> > Hi, > >> >> > > >> >> > Can someone please tell me what is the use of calling > >> >> > setLocalResources() > >> >> > on ContainerLaunchContext? > >> >> > > >> >> > And, also an example of how to use this will help... > >> >> > > >> >> > I couldn't guess what is the String in the map that is passed to > >> >> > setLocalResources() like below: > >> >> > > >> >> > // Set the local resources > >> >> > Map localResources = new HashMap >> >> > LocalResource>(); > >> >> > > >> >> > Thanks, > >> >> > Kishore > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Harsh J > >> > > >> > > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > -- > Harsh J > > >
Re: setLocalResources() on ContainerLaunchContext
ctx.setLocalResources(localResources); >> > >> > >> > Please let me know if you need anything else. >> > >> > Thanks, >> > Kishore >> > >> > >> > >> > On Tue, Aug 6, 2013 at 12:05 AM, Harsh J wrote: >> >> >> >> The detail is insufficient to answer why. You should also have gotten >> >> a trace after it, can you post that? If possible, also the relevant >> >> snippets of code. >> >> >> >> On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri >> >> wrote: >> >> > Hi Harsh, >> >> > Thanks for the quick and detailed reply, it really helps. I am trying >> >> > to >> >> > use it and getting this error in node manager's log: >> >> > >> >> > 2013-08-05 08:57:28,867 ERROR >> >> > org.apache.hadoop.security.UserGroupInformation: >> >> > PriviledgedActionException >> >> > as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does >> >> > not >> >> > exist: hdfs://isredeng/kishore/kk.ksh >> >> > >> >> > >> >> > This file is there on the machine with name "isredeng", I could do ls >> >> > for >> >> > that file as below: >> >> > >> >> > -bash-4.1$ hadoop fs -ls kishore/kk.ksh >> >> > 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load >> >> > native-hadoop >> >> > library for your platform... using builtin-java classes where applicable >> >> > Found 1 items >> >> > -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48 >> >> > kishore/kk.ksh >> >> > >> >> > Note: I am using a single node cluster >> >> > >> >> > Thanks, >> >> > Kishore >> >> > >> >> > >> >> > >> >> > >> >> > On Mon, Aug 5, 2013 at 3:00 PM, Harsh J wrote: >> >> >> >> >> >> The string for each LocalResource in the map can be anything that >> >> >> serves as a common identifier name for your application. At execution >> >> >> time, the passed resource filename will be aliased to the name you've >> >> >> mapped it to, so that the application code need not track special >> >> >> names. The behavior is very similar to how you can, in MR, define a >> >> >> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). >> >> >> >> >> >> For an example, checkout the DistributedShell app sources. >> >> >> >> >> >> Over [1], you can see we take a user provided file path to a shell >> >> >> script. This can be named anything as it is user-supplied. >> >> >> Onto [2], we define this as a local resource [2.1] and embed it with a >> >> >> different name (the string you ask about) [2.2], as defined at [3] as >> >> >> an application reference-able constant. >> >> >> Note that in [4], we add to the Container arguments the aliased name >> >> >> we mapped it to (i.e. [3]) and not the original filename we received >> >> >> from the user. The resource is placed on the container with this name >> >> >> instead, so thats what we choose to execute. >> >> >> >> >> >> [1] - >> >> >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 >> >> >> >> >> >> [2] - [2.1] >> >> >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 >> >> >> and [2.2] >> >> >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 >> >> >> >> >> >> [3] - >> >> >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 >> >> >> >> >> >> [4] - >> >> >> >> >> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 >> >> >> >> >> >> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri >> >> >> wrote: >> >> >> > Hi, >> >> >> > >> >> >> > Can someone please tell me what is the use of calling >> >> >> > setLocalResources() >> >> >> > on ContainerLaunchContext? >> >> >> > >> >> >> > And, also an example of how to use this will help... >> >> >> > >> >> >> > I couldn't guess what is the String in the map that is passed to >> >> >> > setLocalResources() like below: >> >> >> > >> >> >> > // Set the local resources >> >> >> > Map localResources = new HashMap> >> >> > LocalResource>(); >> >> >> > >> >> >> > Thanks, >> >> >> > Kishore >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Harsh J >> >> > >> >> > >> >> >> >> >> >> >> >> -- >> >> Harsh J >> > >> > >> >> >> >> -- >> Harsh J >> >> >> > -- Harsh J
Re: setLocalResources() on ContainerLaunchContext
>> Map localResources = new HashMap>>> LocalResource>();
>>>>
>>>> LocalResource shellRsrc = Records.newRecord(LocalResource.class);
>>>> shellRsrc.setType(LocalResourceType.FILE);
>>>> shellRsrc.setVisibility(LocalResourceVisibility.APPLICATION);
>>>> String shellScriptPath = "hdfs://isredeng//kishore/kk.ksh";
>>>> try {
>>>>shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new
>>>> URI(shellScriptPath)));
>>>> } catch (URISyntaxException e) {
>>>>LOG.error("Error when trying to use shell script path specified"
>>>>+ " in env, path=" + shellScriptPath);
>>>>e.printStackTrace();
>>>> }
>>>>
>>>> shellRsrc.setTimestamp(0/*shellScriptPathTimestamp*/);
>>>> shellRsrc.setSize(0/*shellScriptPathLen*/);
>>>> String ExecShellStringPath = "ExecShellScript.sh";
>>>> localResources.put(ExecShellStringPath, shellRsrc);
>>>>
>>>> ctx.setLocalResources(localResources);
>>>>
>>>>
>>>> Please let me know if you need anything else.
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>>
>>>> On Tue, Aug 6, 2013 at 12:05 AM, Harsh J wrote:
>>>>>
>>>>> The detail is insufficient to answer why. You should also have gotten
>>>>> a trace after it, can you post that? If possible, also the relevant
>>>>> snippets of code.
>>>>>
>>>>> On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri
>>>>> wrote:
>>>>>> Hi Harsh,
>>>>>> Thanks for the quick and detailed reply, it really helps. I am trying
>>>>>> to
>>>>>> use it and getting this error in node manager's log:
>>>>>>
>>>>>> 2013-08-05 08:57:28,867 ERROR
>>>>>> org.apache.hadoop.security.UserGroupInformation:
>>>>>> PriviledgedActionException
>>>>>> as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does
>>>>>> not
>>>>>> exist: hdfs://isredeng/kishore/kk.ksh
>>>>>>
>>>>>>
>>>>>> This file is there on the machine with name "isredeng", I could do ls
>>>>>> for
>>>>>> that file as below:
>>>>>>
>>>>>> -bash-4.1$ hadoop fs -ls kishore/kk.ksh
>>>>>> 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load
>>>>>> native-hadoop
>>>>>> library for your platform... using builtin-java classes where applicable
>>>>>> Found 1 items
>>>>>> -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48
>>>>>> kishore/kk.ksh
>>>>>>
>>>>>> Note: I am using a single node cluster
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 5, 2013 at 3:00 PM, Harsh J wrote:
>>>>>>>
>>>>>>> The string for each LocalResource in the map can be anything that
>>>>>>> serves as a common identifier name for your application. At execution
>>>>>>> time, the passed resource filename will be aliased to the name you've
>>>>>>> mapped it to, so that the application code need not track special
>>>>>>> names. The behavior is very similar to how you can, in MR, define a
>>>>>>> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar).
>>>>>>>
>>>>>>> For an example, checkout the DistributedShell app sources.
>>>>>>>
>>>>>>> Over [1], you can see we take a user provided file path to a shell
>>>>>>> script. This can be named anything as it is user-supplied.
>>>>>>> Onto [2], we define this as a local resource [2.1] and embed it with a
>>>>>>> different name (the string you ask about) [2.2], as defined at [3] as
>>>>>>> an application reference-able constant.
>>>>>>> Note that in [4], we add to the Container arguments the aliased name
>>>>>>> we mapped it to (i.e. [3]) and not the original filename we received
>>>>>>> from the user. The resource is placed on the container with this name
>>>>>>> instead, so thats what we choose to execute.
>>>>>>>
>>>>>>> [1] -
>>>>>>>
>>>>>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390
>>>>>>>
>>>>>>> [2] - [2.1]
>>>>>>>
>>>>>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764
>>>>>>> and [2.2]
>>>>>>>
>>>>>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780
>>>>>>>
>>>>>>> [3] -
>>>>>>>
>>>>>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205
>>>>>>>
>>>>>>> [4] -
>>>>>>>
>>>>>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791
>>>>>>>
>>>>>>> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri
>>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Can someone please tell me what is the use of calling
>>>>>>>> setLocalResources()
>>>>>>>> on ContainerLaunchContext?
>>>>>>>>
>>>>>>>> And, also an example of how to use this will help...
>>>>>>>>
>>>>>>>> I couldn't guess what is the String in the map that is passed to
>>>>>>>> setLocalResources() like below:
>>>>>>>>
>>>>>>>> // Set the local resources
>>>>>>>> Map localResources = new HashMap>>>>>>> LocalResource>();
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Kishore
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Harsh J
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>>
>>>
>>
>
>
>
> --
> Harsh J
Re: setLocalResources() on ContainerLaunchContext
>at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:163)
>>>>>at
>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106)
>>>>>at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:979)
>>>>>
>>>>>
>>>>>
>>>>> And here is my code snippet:
>>>>>
>>>>> ContainerLaunchContext ctx =
>>>>> Records.newRecord(ContainerLaunchContext.class);
>>>>>
>>>>> ctx.setEnvironment(oshEnv);
>>>>>
>>>>> // Set the local resources
>>>>> Map localResources = new HashMap>>>> LocalResource>();
>>>>>
>>>>> LocalResource shellRsrc = Records.newRecord(LocalResource.class);
>>>>> shellRsrc.setType(LocalResourceType.FILE);
>>>>> shellRsrc.setVisibility(LocalResourceVisibility.APPLICATION);
>>>>> String shellScriptPath = "hdfs://isredeng//kishore/kk.ksh";
>>>>> try {
>>>>>shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new
>>>>> URI(shellScriptPath)));
>>>>> } catch (URISyntaxException e) {
>>>>>LOG.error("Error when trying to use shell script path specified"
>>>>>+ " in env, path=" + shellScriptPath);
>>>>>e.printStackTrace();
>>>>> }
>>>>>
>>>>> shellRsrc.setTimestamp(0/*shellScriptPathTimestamp*/);
>>>>> shellRsrc.setSize(0/*shellScriptPathLen*/);
>>>>> String ExecShellStringPath = "ExecShellScript.sh";
>>>>> localResources.put(ExecShellStringPath, shellRsrc);
>>>>>
>>>>> ctx.setLocalResources(localResources);
>>>>>
>>>>>
>>>>> Please let me know if you need anything else.
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 6, 2013 at 12:05 AM, Harsh J wrote:
>>>>>>
>>>>>> The detail is insufficient to answer why. You should also have gotten
>>>>>> a trace after it, can you post that? If possible, also the relevant
>>>>>> snippets of code.
>>>>>>
>>>>>> On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri
>>>>>> wrote:
>>>>>>> Hi Harsh,
>>>>>>> Thanks for the quick and detailed reply, it really helps. I am trying
>>>>>>> to
>>>>>>> use it and getting this error in node manager's log:
>>>>>>>
>>>>>>> 2013-08-05 08:57:28,867 ERROR
>>>>>>> org.apache.hadoop.security.UserGroupInformation:
>>>>>>> PriviledgedActionException
>>>>>>> as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does
>>>>>>> not
>>>>>>> exist: hdfs://isredeng/kishore/kk.ksh
>>>>>>>
>>>>>>>
>>>>>>> This file is there on the machine with name "isredeng", I could do ls
>>>>>>> for
>>>>>>> that file as below:
>>>>>>>
>>>>>>> -bash-4.1$ hadoop fs -ls kishore/kk.ksh
>>>>>>> 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load
>>>>>>> native-hadoop
>>>>>>> library for your platform... using builtin-java classes where applicable
>>>>>>> Found 1 items
>>>>>>> -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48
>>>>>>> kishore/kk.ksh
>>>>>>>
>>>>>>> Note: I am using a single node cluster
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kishore
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 5, 2013 at 3:00 PM, Harsh J wrote:
>>>>>>>>
>>>>>>>> The string for each LocalResource in the map can be anything that
>>>>>>>> serves as a common identifier name for your application. At execution
>>>>>>>> time, the passed resource filename will be aliased to the name you've
>>>>>>>> mapped it to, so that the application code need not track special
>>>>>>>> names. The behavior is very similar to how you can, in MR, define a
>>>>>>>> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar).
>>>>>>>>
>>>>>>>> For an example, checkout the DistributedShell app sources.
>>>>>>>>
>>>>>>>> Over [1], you can see we take a user provided file path to a shell
>>>>>>>> script. This can be named anything as it is user-supplied.
>>>>>>>> Onto [2], we define this as a local resource [2.1] and embed it with a
>>>>>>>> different name (the string you ask about) [2.2], as defined at [3] as
>>>>>>>> an application reference-able constant.
>>>>>>>> Note that in [4], we add to the Container arguments the aliased name
>>>>>>>> we mapped it to (i.e. [3]) and not the original filename we received
>>>>>>>> from the user. The resource is placed on the container with this name
>>>>>>>> instead, so thats what we choose to execute.
>>>>>>>>
>>>>>>>> [1] -
>>>>>>>>
>>>>>>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390
>>>>>>>>
>>>>>>>> [2] - [2.1]
>>>>>>>>
>>>>>>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764
>>>>>>>> and [2.2]
>>>>>>>>
>>>>>>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780
>>>>>>>>
>>>>>>>> [3] -
>>>>>>>>
>>>>>>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205
>>>>>>>>
>>>>>>>> [4] -
>>>>>>>>
>>>>>>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791
>>>>>>>>
>>>>>>>> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri
>>>>>>>> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Can someone please tell me what is the use of calling
>>>>>>>>> setLocalResources()
>>>>>>>>> on ContainerLaunchContext?
>>>>>>>>>
>>>>>>>>> And, also an example of how to use this will help...
>>>>>>>>>
>>>>>>>>> I couldn't guess what is the String in the map that is passed to
>>>>>>>>> setLocalResources() like below:
>>>>>>>>>
>>>>>>>>> // Set the local resources
>>>>>>>>> Map localResources = new HashMap>>>>>>>> LocalResource>();
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Kishore
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Harsh J
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Harsh J
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Harsh J
>
--
Harsh J
Re: setLocalResources() on ContainerLaunchContext
t;>>>>>> 2013-08-05 08:57:28,867 ERROR > >>>>>>> org.apache.hadoop.security.UserGroupInformation: > >>>>>>> PriviledgedActionException > >>>>>>> as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File > does > >>>>>>> not > >>>>>>> exist: hdfs://isredeng/kishore/kk.ksh > >>>>>>> > >>>>>>> > >>>>>>> This file is there on the machine with name "isredeng", I could do > ls > >>>>>>> for > >>>>>>> that file as below: > >>>>>>> > >>>>>>> -bash-4.1$ hadoop fs -ls kishore/kk.ksh > >>>>>>> 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load > >>>>>>> native-hadoop > >>>>>>> library for your platform... using builtin-java classes where > applicable > >>>>>>> Found 1 items > >>>>>>> -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48 > >>>>>>> kishore/kk.ksh > >>>>>>> > >>>>>>> Note: I am using a single node cluster > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Kishore > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Mon, Aug 5, 2013 at 3:00 PM, Harsh J > wrote: > >>>>>>>> > >>>>>>>> The string for each LocalResource in the map can be anything that > >>>>>>>> serves as a common identifier name for your application. At > execution > >>>>>>>> time, the passed resource filename will be aliased to the name > you've > >>>>>>>> mapped it to, so that the application code need not track special > >>>>>>>> names. The behavior is very similar to how you can, in MR, define > a > >>>>>>>> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). > >>>>>>>> > >>>>>>>> For an example, checkout the DistributedShell app sources. > >>>>>>>> > >>>>>>>> Over [1], you can see we take a user provided file path to a shell > >>>>>>>> script. This can be named anything as it is user-supplied. > >>>>>>>> Onto [2], we define this as a local resource [2.1] and embed it > with a > >>>>>>>> different name (the string you ask about) [2.2], as defined at > [3] as > >>>>>>>> an application reference-able constant. > >>>>>>>> Note that in [4], we add to the Container arguments the aliased > name > >>>>>>>> we mapped it to (i.e. [3]) and not the original filename we > received > >>>>>>>> from the user. The resource is placed on the container with this > name > >>>>>>>> instead, so thats what we choose to execute. > >>>>>>>> > >>>>>>>> [1] - > >>>>>>>> > >>>>>>>> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 > >>>>>>>> > >>>>>>>> [2] - [2.1] > >>>>>>>> > >>>>>>>> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 > >>>>>>>> and [2.2] > >>>>>>>> > >>>>>>>> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 > >>>>>>>> > >>>>>>>> [3] - > >>>>>>>> > >>>>>>>> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 > >>>>>>>> > >>>>>>>> [4] - > >>>>>>>> > >>>>>>>> > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 > >>>>>>>> > >>>>>>>> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri > >>>>>>>> wrote: > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> Can someone please tell me what is the use of calling > >>>>>>>>> setLocalResources() > >>>>>>>>> on ContainerLaunchContext? > >>>>>>>>> > >>>>>>>>> And, also an example of how to use this will help... > >>>>>>>>> > >>>>>>>>> I couldn't guess what is the String in the map that is passed to > >>>>>>>>> setLocalResources() like below: > >>>>>>>>> > >>>>>>>>> // Set the local resources > >>>>>>>>> Map localResources = new > HashMap >>>>>>>>> LocalResource>(); > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Kishore > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Harsh J > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Harsh J > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Harsh J > >>>> > >>>> > >>>> > >>> > >> > >> > >> > >> -- > >> Harsh J > > > > > > -- > Harsh J >
Re: setLocalResources() on ContainerLaunchContext
gt;e.printStackTrace(); >> >>>>> } >> >>>>> >> >>>>> shellRsrc.setTimestamp(0/*shellScriptPathTimestamp*/); >> >>>>> shellRsrc.setSize(0/*shellScriptPathLen*/); >> >>>>> String ExecShellStringPath = "ExecShellScript.sh"; >> >>>>> localResources.put(ExecShellStringPath, shellRsrc); >> >>>>> >> >>>>> ctx.setLocalResources(localResources); >> >>>>> >> >>>>> >> >>>>> Please let me know if you need anything else. >> >>>>> >> >>>>> Thanks, >> >>>>> Kishore >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Tue, Aug 6, 2013 at 12:05 AM, Harsh J >> wrote: >> >>>>>> >> >>>>>> The detail is insufficient to answer why. You should also have >> gotten >> >>>>>> a trace after it, can you post that? If possible, also the relevant >> >>>>>> snippets of code. >> >>>>>> >> >>>>>> On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri >> >>>>>> wrote: >> >>>>>>> Hi Harsh, >> >>>>>>> Thanks for the quick and detailed reply, it really helps. I am >> trying >> >>>>>>> to >> >>>>>>> use it and getting this error in node manager's log: >> >>>>>>> >> >>>>>>> 2013-08-05 08:57:28,867 ERROR >> >>>>>>> org.apache.hadoop.security.UserGroupInformation: >> >>>>>>> PriviledgedActionException >> >>>>>>> as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File >> does >> >>>>>>> not >> >>>>>>> exist: hdfs://isredeng/kishore/kk.ksh >> >>>>>>> >> >>>>>>> >> >>>>>>> This file is there on the machine with name "isredeng", I could >> do ls >> >>>>>>> for >> >>>>>>> that file as below: >> >>>>>>> >> >>>>>>> -bash-4.1$ hadoop fs -ls kishore/kk.ksh >> >>>>>>> 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load >> >>>>>>> native-hadoop >> >>>>>>> library for your platform... using builtin-java classes where >> applicable >> >>>>>>> Found 1 items >> >>>>>>> -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48 >> >>>>>>> kishore/kk.ksh >> >>>>>>> >> >>>>>>> Note: I am using a single node cluster >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> Kishore >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> On Mon, Aug 5, 2013 at 3:00 PM, Harsh J >> wrote: >> >>>>>>>> >> >>>>>>>> The string for each LocalResource in the map can be anything that >> >>>>>>>> serves as a common identifier name for your application. At >> execution >> >>>>>>>> time, the passed resource filename will be aliased to the name >> you've >> >>>>>>>> mapped it to, so that the application code need not track special >> >>>>>>>> names. The behavior is very similar to how you can, in MR, >> define a >> >>>>>>>> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar). >> >>>>>>>> >> >>>>>>>> For an example, checkout the DistributedShell app sources. >> >>>>>>>> >> >>>>>>>> Over [1], you can see we take a user provided file path to a >> shell >> >>>>>>>> script. This can be named anything as it is user-supplied. >> >>>>>>>> Onto [2], we define this as a local resource [2.1] and embed it >> with a >> >>>>>>>> different name (the string you ask about) [2.2], as defined at >> [3] as >> >>>>>>>> an application reference-able constant. >> >>>>>>>> Note that in [4], we add to the Container arguments the aliased >> name >> >>>>>>>> we mapped it to (i.e. [3]) and not the original filename we >> received >> >>>>>>>> from the user. The resource is placed on the container with this >> name >> >>>>>>>> instead, so thats what we choose to execute. >> >>>>>>>> >> >>>>>>>> [1] - >> >>>>>>>> >> >>>>>>>> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390 >> >>>>>>>> >> >>>>>>>> [2] - [2.1] >> >>>>>>>> >> >>>>>>>> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764 >> >>>>>>>> and [2.2] >> >>>>>>>> >> >>>>>>>> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780 >> >>>>>>>> >> >>>>>>>> [3] - >> >>>>>>>> >> >>>>>>>> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205 >> >>>>>>>> >> >>>>>>>> [4] - >> >>>>>>>> >> >>>>>>>> >> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791 >> >>>>>>>> >> >>>>>>>> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri >> >>>>>>>> wrote: >> >>>>>>>>> Hi, >> >>>>>>>>> >> >>>>>>>>> Can someone please tell me what is the use of calling >> >>>>>>>>> setLocalResources() >> >>>>>>>>> on ContainerLaunchContext? >> >>>>>>>>> >> >>>>>>>>> And, also an example of how to use this will help... >> >>>>>>>>> >> >>>>>>>>> I couldn't guess what is the String in the map that is passed to >> >>>>>>>>> setLocalResources() like below: >> >>>>>>>>> >> >>>>>>>>> // Set the local resources >> >>>>>>>>> Map localResources = new >> HashMap> >>>>>>>>> LocalResource>(); >> >>>>>>>>> >> >>>>>>>>> Thanks, >> >>>>>>>>> Kishore >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> Harsh J >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Harsh J >> >>>>> >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Harsh J >> >>>> >> >>>> >> >>>> >> >>> >> >> >> >> >> >> >> >> -- >> >> Harsh J >> > >> >> >> >> -- >> Harsh J >> > >
Re: setLocalResources() on ContainerLaunchContext
gt;>> >>>>>
>>> >>>>>
>>> >>>>> And here is my code snippet:
>>> >>>>>
>>> >>>>> ContainerLaunchContext ctx =
>>> >>>>> Records.newRecord(ContainerLaunchContext.class);
>>> >>>>>
>>> >>>>> ctx.setEnvironment(oshEnv);
>>> >>>>>
>>> >>>>> // Set the local resources
>>> >>>>> Map localResources = new
>>> HashMap>> >>>>> LocalResource>();
>>> >>>>>
>>> >>>>> LocalResource shellRsrc =
>>> Records.newRecord(LocalResource.class);
>>> >>>>> shellRsrc.setType(LocalResourceType.FILE);
>>> >>>>> shellRsrc.setVisibility(LocalResourceVisibility.APPLICATION);
>>> >>>>> String shellScriptPath = "hdfs://isredeng//kishore/kk.ksh";
>>> >>>>> try {
>>> >>>>>shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new
>>> >>>>> URI(shellScriptPath)));
>>> >>>>> } catch (URISyntaxException e) {
>>> >>>>>LOG.error("Error when trying to use shell script path
>>> specified"
>>> >>>>>+ " in env, path=" + shellScriptPath);
>>> >>>>>e.printStackTrace();
>>> >>>>> }
>>> >>>>>
>>> >>>>> shellRsrc.setTimestamp(0/*shellScriptPathTimestamp*/);
>>> >>>>> shellRsrc.setSize(0/*shellScriptPathLen*/);
>>> >>>>> String ExecShellStringPath = "ExecShellScript.sh";
>>> >>>>> localResources.put(ExecShellStringPath, shellRsrc);
>>> >>>>>
>>> >>>>> ctx.setLocalResources(localResources);
>>> >>>>>
>>> >>>>>
>>> >>>>> Please let me know if you need anything else.
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> Kishore
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, Aug 6, 2013 at 12:05 AM, Harsh J
>>> wrote:
>>> >>>>>>
>>> >>>>>> The detail is insufficient to answer why. You should also have
>>> gotten
>>> >>>>>> a trace after it, can you post that? If possible, also the
>>> relevant
>>> >>>>>> snippets of code.
>>> >>>>>>
>>> >>>>>> On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri
>>> >>>>>> wrote:
>>> >>>>>>> Hi Harsh,
>>> >>>>>>> Thanks for the quick and detailed reply, it really helps. I am
>>> trying
>>> >>>>>>> to
>>> >>>>>>> use it and getting this error in node manager's log:
>>> >>>>>>>
>>> >>>>>>> 2013-08-05 08:57:28,867 ERROR
>>> >>>>>>> org.apache.hadoop.security.UserGroupInformation:
>>> >>>>>>> PriviledgedActionException
>>> >>>>>>> as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File
>>> does
>>> >>>>>>> not
>>> >>>>>>> exist: hdfs://isredeng/kishore/kk.ksh
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> This file is there on the machine with name "isredeng", I could
>>> do ls
>>> >>>>>>> for
>>> >>>>>>> that file as below:
>>> >>>>>>>
>>> >>>>>>> -bash-4.1$ hadoop fs -ls kishore/kk.ksh
>>> >>>>>>> 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load
>>> >>>>>>> native-hadoop
>>> >>>>>>> library for your platform... using builtin-java classes where
>>> applicable
>>> >>>>>>> Found 1 items
>>> >>>>>>> -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48
>>> >>>>>>> kishore/kk.ksh
>>> >>>>>>>
>>> >>>>>>> Note: I am using a single node cluster
>>> >>>>>>>
>>> >>>>>>> Thanks,
>>> >>>>>>> Kishore
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On Mon, Aug 5, 2013 at 3:00 PM, Harsh J
>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>> The string for each LocalResource in the map can be anything
>>> that
>>> >>>>>>>> serves as a common identifier name for your application. At
>>> execution
>>> >>>>>>>> time, the passed resource filename will be aliased to the name
>>> you've
>>> >>>>>>>> mapped it to, so that the application code need not track
>>> special
>>> >>>>>>>> names. The behavior is very similar to how you can, in MR,
>>> define a
>>> >>>>>>>> symlink name for a DistributedCache entry (e.g.
>>> foo.jar#bar.jar).
>>> >>>>>>>>
>>> >>>>>>>> For an example, checkout the DistributedShell app sources.
>>> >>>>>>>>
>>> >>>>>>>> Over [1], you can see we take a user provided file path to a
>>> shell
>>> >>>>>>>> script. This can be named anything as it is user-supplied.
>>> >>>>>>>> Onto [2], we define this as a local resource [2.1] and embed it
>>> with a
>>> >>>>>>>> different name (the string you ask about) [2.2], as defined at
>>> [3] as
>>> >>>>>>>> an application reference-able constant.
>>> >>>>>>>> Note that in [4], we add to the Container arguments the aliased
>>> name
>>> >>>>>>>> we mapped it to (i.e. [3]) and not the original filename we
>>> received
>>> >>>>>>>> from the user. The resource is placed on the container with
>>> this name
>>> >>>>>>>> instead, so thats what we choose to execute.
>>> >>>>>>>>
>>> >>>>>>>> [1] -
>>> >>>>>>>>
>>> >>>>>>>>
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390
>>> >>>>>>>>
>>> >>>>>>>> [2] - [2.1]
>>> >>>>>>>>
>>> >>>>>>>>
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764
>>> >>>>>>>> and [2.2]
>>> >>>>>>>>
>>> >>>>>>>>
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780
>>> >>>>>>>>
>>> >>>>>>>> [3] -
>>> >>>>>>>>
>>> >>>>>>>>
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205
>>> >>>>>>>>
>>> >>>>>>>> [4] -
>>> >>>>>>>>
>>> >>>>>>>>
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791
>>> >>>>>>>>
>>> >>>>>>>> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri
>>> >>>>>>>> wrote:
>>> >>>>>>>>> Hi,
>>> >>>>>>>>>
>>> >>>>>>>>> Can someone please tell me what is the use of calling
>>> >>>>>>>>> setLocalResources()
>>> >>>>>>>>> on ContainerLaunchContext?
>>> >>>>>>>>>
>>> >>>>>>>>> And, also an example of how to use this will help...
>>> >>>>>>>>>
>>> >>>>>>>>> I couldn't guess what is the String in the map that is passed
>>> to
>>> >>>>>>>>> setLocalResources() like below:
>>> >>>>>>>>>
>>> >>>>>>>>> // Set the local resources
>>> >>>>>>>>> Map localResources = new
>>> HashMap>> >>>>>>>>> LocalResource>();
>>> >>>>>>>>>
>>> >>>>>>>>> Thanks,
>>> >>>>>>>>> Kishore
>>> >>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> --
>>> >>>>>>>> Harsh J
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Harsh J
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Harsh J
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Harsh J
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
