Re: Long running Spark job on YARN throws "No AMRMToken"
On 11 Feb 2016, at 15:24, Prabhu Joseph mailto:prabhujose.ga...@gmail.com>> wrote: Steve, When ResourceManager is submitted with an application, AMLauncher creates the token YARN_AM_RM_TOKEN (token used between RM and AM). When ApplicationMaster is launched, it tries to contact RM for registering request, allocate request to receive containers, finish request. In all the requests, yes, see https://github.com/steveloughran/hadoop-trunk/blob/HADOOP-12649-security/YARN-4653-yarn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md ResourceManager does the authorizeRequest, where it checks if the Current User has the token YARN_AM_RM_TOKEN, if not throws the "No AMRMToken". yes; prior to YARN-3103 it did the login user ResourceManager for every yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-sec rolls the master key, before rolling it, it has a period of 1.5 * yarn.am.liveness-monitor.expiry-interval-ms during which if AM contacts RM with allocate request, RM checks if the AM has the YARN_AM_RM_TOKEN prepared using the previous master key, if so, it updates the AM user with YARN_AM_RM_TOKEN prepared using new master key. If AM contacts with an YARN_AM_RM_TOKEN which is neither constructed using current master key nor previous master key, then "Invalid AMRMToken" message is thrown. This error is the one will happen if AM has not been updated with new RM master key. [YARN-3103 and YARN-2212 ] Need your help to find scenario where "No AMRMToken" will happen, an user added with a token but later that token is missing. Is token removed since expired? ...or there's some confusion about the current user I've got a java class to help with credential creation and diagnostics, not yet ported to hadoop core, which can do some listing & dumping of credentials https://github.com/apache/incubator-slider/blob/develop/slider-core/src/main/java/org/apache/slider/core/launch/CredentialUtils.java you may be able to copy that code and use it to print out what tokens the current user has; otherwise I don't know. I've never personally hit the message
Re: Long running Spark job on YARN throws "No AMRMToken"
Steve, When ResourceManager is submitted with an application, AMLauncher creates the token YARN_AM_RM_TOKEN (token used between RM and AM). When ApplicationMaster is launched, it tries to contact RM for registering request, allocate request to receive containers, finish request. In all the requests, ResourceManager does the authorizeRequest, where it checks if the Current User has the token YARN_AM_RM_TOKEN, if not throws the *"No AMRMToken". * ResourceManager for every yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-sec rolls the master key, before rolling it, it has a period of 1.5 * yarn.am.liveness-monitor.expiry-interval-ms during which if AM contacts RM with allocate request, RM checks if the AM has the YARN_AM_RM_TOKEN prepared using the previous master key, if so, it updates the AM user with YARN_AM_RM_TOKEN prepared using new master key. If AM contacts with an YARN_AM_RM_TOKEN which is neither constructed using current master key nor previous master key, then *"Invalid AMRMToken"* message is thrown. This error is the one will happen if AM has not been updated with new RM master key. [YARN-3103 and YARN-2212 ] Need your help to find scenario where "No AMRMToken" will happen, an user added with a token but later that token is missing. Is token removed since expired? Thanks, Prabhu Joseph On Wed, Feb 10, 2016 at 12:59 AM, Hari Shreedharan < hshreedha...@cloudera.com> wrote: > The credentials file approach (using keytab for spark apps) will only > update HDFS tokens. YARN's AMRM tokens should be taken care of by YARN > internally. > > Steve - correct me if I am wrong here: If the AMRM tokens are disappearing > it might be a YARN bug (does the AMRM token have a 7 day limit as well? I > thought that was only for HDFS). > > > Thanks, > Hari > > On Tue, Feb 9, 2016 at 8:44 AM, Steve Loughran > wrote: > >> >> On 9 Feb 2016, at 11:26, Steve Loughran wrote: >> >> >> On 9 Feb 2016, at 05:55, Prabhu Joseph >> wrote: >> >> + Spark-Dev >> >> On Tue, Feb 9, 2016 at 10:04 AM, Prabhu Joseph < >> prabhujose.ga...@gmail.com> wrote: >> >>> Hi All, >>> >>> A long running Spark job on YARN throws below exception after >>> running for few days. >>> >>> yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. >>> org.apache.hadoop.yarn.exceptions.YarnException: *No AMRMToken found* for >>> user prabhu at org.apache.hadoop.yarn.ipc.RPC >>> Util.getRemoteException(RPCUtil.java:45) >>> >>> Do any of the below renew the AMRMToken and solve the issue >>> >>> 1. yarn-resourcemanager.delegation.token.max-lifetime increase from 7 >>> days >>> >>> 2. Configuring Proxy user: >>> >>> hadoop.proxyuser.yarn.hosts * >>> >>> hadoop.proxyuser.yarn.groups * >>> >>> >> >> wouldnt do that: security issues >> >> >>> 3. Can Spark-1.4.0 handle with fix >>> https://issues.apache.org/jira/browse/SPARK-5342 >>> >>> spark.yarn.credentials.file >>> >>> >>> >> I'll say "maybe" there >> >> >> uprated to a no, having looked at the code more >> >> >> How to renew the AMRMToken for a long running job on YARN? >>> >>> >>> >> >> AMRM token renewal should be automatic in AM; Yarn sends a message to the >> AM (actually an allocate() response with no containers but a new token at >> the tail of the message. >> >> i don't see any logging in the Hadoopp code there (AMRMClientImpl); filed >> YARN-4682 to add a log statement >> >> if someone other than me were to supply a patch to that JIRA to add a log >> statement *by the end of the day* I'll review it and get it in to Hadoop 2.8 >> >> >> like I said: I'll get this in to hadoop-2.8 if someone is timely with the >> diff >> >> >
Re: Long running Spark job on YARN throws "No AMRMToken"
The credentials file approach (using keytab for spark apps) will only update HDFS tokens. YARN's AMRM tokens should be taken care of by YARN internally. Steve - correct me if I am wrong here: If the AMRM tokens are disappearing it might be a YARN bug (does the AMRM token have a 7 day limit as well? I thought that was only for HDFS). Thanks, Hari On Tue, Feb 9, 2016 at 8:44 AM, Steve Loughran wrote: > > On 9 Feb 2016, at 11:26, Steve Loughran wrote: > > > On 9 Feb 2016, at 05:55, Prabhu Joseph wrote: > > + Spark-Dev > > On Tue, Feb 9, 2016 at 10:04 AM, Prabhu Joseph > wrote: > >> Hi All, >> >> A long running Spark job on YARN throws below exception after running >> for few days. >> >> yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. >> org.apache.hadoop.yarn.exceptions.YarnException: *No AMRMToken found* for >> user prabhu at org.apache.hadoop.yarn.ipc.RPC >> Util.getRemoteException(RPCUtil.java:45) >> >> Do any of the below renew the AMRMToken and solve the issue >> >> 1. yarn-resourcemanager.delegation.token.max-lifetime increase from 7 days >> >> 2. Configuring Proxy user: >> >> hadoop.proxyuser.yarn.hosts * >> >> hadoop.proxyuser.yarn.groups * >> >> > > wouldnt do that: security issues > > >> 3. Can Spark-1.4.0 handle with fix >> https://issues.apache.org/jira/browse/SPARK-5342 >> >> spark.yarn.credentials.file >> >> >> > I'll say "maybe" there > > > uprated to a no, having looked at the code more > > > How to renew the AMRMToken for a long running job on YARN? >> >> >> > > AMRM token renewal should be automatic in AM; Yarn sends a message to the > AM (actually an allocate() response with no containers but a new token at > the tail of the message. > > i don't see any logging in the Hadoopp code there (AMRMClientImpl); filed > YARN-4682 to add a log statement > > if someone other than me were to supply a patch to that JIRA to add a log > statement *by the end of the day* I'll review it and get it in to Hadoop 2.8 > > > like I said: I'll get this in to hadoop-2.8 if someone is timely with the > diff > >
Re: Long running Spark job on YARN throws "No AMRMToken"
On 9 Feb 2016, at 11:26, Steve Loughran mailto:ste...@hortonworks.com>> wrote: On 9 Feb 2016, at 05:55, Prabhu Joseph mailto:prabhujose.ga...@gmail.com>> wrote: + Spark-Dev On Tue, Feb 9, 2016 at 10:04 AM, Prabhu Joseph mailto:prabhujose.ga...@gmail.com>> wrote: Hi All, A long running Spark job on YARN throws below exception after running for few days. yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. org.apache.hadoop.yarn.exceptions.YarnException: No AMRMToken found for user prabhu at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:45) Do any of the below renew the AMRMToken and solve the issue 1. yarn-resourcemanager.delegation.token.max-lifetime increase from 7 days 2. Configuring Proxy user: hadoop.proxyuser.yarn.hosts * hadoop.proxyuser.yarn.groups * wouldnt do that: security issues 3. Can Spark-1.4.0 handle with fix https://issues.apache.org/jira/browse/SPARK-5342 spark.yarn.credentials.file I'll say "maybe" there uprated to a no, having looked at the code more How to renew the AMRMToken for a long running job on YARN? AMRM token renewal should be automatic in AM; Yarn sends a message to the AM (actually an allocate() response with no containers but a new token at the tail of the message. i don't see any logging in the Hadoopp code there (AMRMClientImpl); filed YARN-4682 to add a log statement if someone other than me were to supply a patch to that JIRA to add a log statement *by the end of the day* I'll review it and get it in to Hadoop 2.8 like I said: I'll get this in to hadoop-2.8 if someone is timely with the diff
Re: Long running Spark job on YARN throws "No AMRMToken"
On 9 Feb 2016, at 05:55, Prabhu Joseph mailto:prabhujose.ga...@gmail.com>> wrote: + Spark-Dev On Tue, Feb 9, 2016 at 10:04 AM, Prabhu Joseph mailto:prabhujose.ga...@gmail.com>> wrote: Hi All, A long running Spark job on YARN throws below exception after running for few days. yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. org.apache.hadoop.yarn.exceptions.YarnException: No AMRMToken found for user prabhu at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:45) Do any of the below renew the AMRMToken and solve the issue 1. yarn-resourcemanager.delegation.token.max-lifetime increase from 7 days 2. Configuring Proxy user: hadoop.proxyuser.yarn.hosts * hadoop.proxyuser.yarn.groups * wouldnt do that: security issues 3. Can Spark-1.4.0 handle with fix https://issues.apache.org/jira/browse/SPARK-5342 spark.yarn.credentials.file I'll say "maybe" there How to renew the AMRMToken for a long running job on YARN? AMRM token renewal should be automatic in AM; Yarn sends a message to the AM (actually an allocate() response with no containers but a new token at the tail of the message. i don't see any logging in the Hadoopp code there (AMRMClientImpl); filed YARN-4682 to add a log statement if someone other than me were to supply a patch to that JIRA to add a log statement *by the end of the day* I'll review it and get it in to Hadoop 2.8 -Steve
Re: Long running Spark job on YARN throws "No AMRMToken"
+ Spark-Dev On Tue, Feb 9, 2016 at 10:04 AM, Prabhu Joseph wrote: > Hi All, > > A long running Spark job on YARN throws below exception after running > for few days. > > yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. > org.apache.hadoop.yarn.exceptions.YarnException: *No AMRMToken found* for > user prabhu at org.apache.hadoop.yarn.ipc.RPC > Util.getRemoteException(RPCUtil.java:45) > > Do any of the below renew the AMRMToken and solve the issue > > 1. yarn-resourcemanager.delegation.token.max-lifetime increase from 7 days > > 2. Configuring Proxy user: > > hadoop.proxyuser.yarn.hosts * > > hadoop.proxyuser.yarn.groups * > > > 3. Can Spark-1.4.0 handle with fix > https://issues.apache.org/jira/browse/SPARK-5342 > > spark.yarn.credentials.file > > > How to renew the AMRMToken for a long running job on YARN? > > > Thanks, > Prabhu Joseph > > > > >
