hi Vinod, hi Bikas
Thanks for your inputs.
Though I am able to spawn the DAGAppMaster with proxy user (I can see the
DAGAppMaster running with proxy user in resource manager UI) but, the
call: TezClient.getAppMasterStatus
fails with security issues.
I have raised a ticket with my client code and more details:
https://issues.apache.org/jira/browse/TEZ-1640
(UserGroupInformation.java:1551) - PriviledgedActionException as:qa
(auth:SIMPLE) cause:java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
Failed to retrieve AM Status via proxy
com.google.protobuf.ServiceException: java.io.IOException: Failed on
local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]; Host Details : local host is:
"ip-10-178-144-254/10.178.144.254"; destination host is:
"ip-10-187-33-206":56660;
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:216)
at com.sun.proxy.$Proxy111.getAMStatus(Unknown Source)
at
org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:522)
On Tue, Oct 7, 2014 at 4:25 AM, Bikas Saha <[email protected]> wrote:
> Thanks Vinod.
>
>
>
> Subroto, looks like there are 2 issues here.
>
>
>
> One is running as use foo but submitting the job as user bar. Vinod’s code
> snippet is relevant to that (Please see my answer with createProxyUser
> earlier and Vinod’s exact code below). The real user is foo but the
> effective user is bar. So when the processes connects to YARN then it YARN
> will use the effective user bar as the job user instead of foo.
>
>
>
> Second is how to obtain credentials for bar while running as foo. That is
> covered by the tez-credentials-file-path solution mentioned earlier. This
> allows user foo to collect credentials from HDFS for user bar (foo is
> expected to be a trusted proxy user in HDFS to be allowed to do this. Like
> Oozie). Then the credentials are passed to the client via the file.
>
>
>
> Hope both these clarify your situation and move you forward.
>
>
>
> Bikas
>
>
>
> *From:* Vinod Kumar Vavilapalli [mailto:[email protected]]
> *Sent:* Monday, October 06, 2014 11:39 AM
>
> *To:* [email protected]
> *Subject:* Re: Tez with secured hadoop
>
>
>
> That is not the way to do impersonation. Please see the following:
>
> UserGroupInformation ugi =
> UserGroupInformation.createProxyUser("qa",
> UserGroupInformation.getLoginUser()); // <-- login user is subroto, the
> realUser. "qa" is the expected effective-user
> ugi.doAs(new PrivilegedExceptionAction<Void>() {
> public Void run() throws Exception {
> ..
> }
> }
>
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> On Thu, Oct 2, 2014 at 6:22 AM, Subroto Sanyal <[email protected]>
> wrote:
>
> hi Bikas,
>
>
>
> My code snippet to create TezClient looks like (TEZ-0.5):
>
> new PrivilegedExceptionAction<TezClient>() {
>
>
>
> @Override
>
> public TezClient run() throws Exception {
>
> UserGroupInformation currentUser =
> UserGroupInformation.getCurrentUser();
>
> LOG.info("Current User:" + currentUser);
>
> File tokenFile = new
> File(System.getProperty("java.io.tmpdir"),
> tezSessionName.replaceAll("[^a-zA-Z0-9]", ""));
>
> LOG.info("Token File:" + tokenFile.getAbsolutePath());
>
>
> currentUser.getCredentials().writeTokenStorageFile(UriUtil.toPath(tokenFile.getAbsoluteFile()),
> conf);
>
> tezConf.set(TezConfiguration.TEZ_CREDENTIALS_PATH,
> tokenFile.getAbsolutePath());
>
> return TezClient.create(tezSessionName, tezConf,
> createSession, localResourceMap, credentials);
>
> }
>
> }
>
>
>
> The logs generated from this piece of code during execution looks like:
>
> (TezClientFacade.java:142) - Current User:qa (auth:PROXY) via
> [email protected] (auth:KERBEROS)
>
> (TezClientFacade.java:144) - Token File:/home/subroto/tmp/testTezJob
>
>
>
> When this runs on cluster the job runs as "subroto" but, what I expect is
> to run it as "qa".
>
>
>
> Please let me know if there is something missing or wrong in the code.
>
>
>
> On Sat, Sep 13, 2014 at 3:33 AM, Bikas Saha <[email protected]> wrote:
>
> If by impersonation you mean what Oozie does where Oozie runs as Oozie but
> get delegation tokens for user FOO then you will need to follow the
> mechanism that Oozie uses. Oozie writes the delegation tokens into a file
> and put that file in a specific path which is picked up by the application
> (in this case TezClient) and the application loads credentials from that
> file. In case of Tez the location of the credentials file is the value of
> config "tez.credentials.path"
>
>
>
> Bikas
>
>
>
> *From:* Bikas Saha [mailto:[email protected]]
> *Sent:* Monday, September 01, 2014 5:34 PM
> *To:* [email protected]
> *Subject:* RE: Tez with secured hadoop
>
>
>
> They way this is supposed to work is the following in a secure cluster.
>
> 1) The user that is running TezClient/DAGClient needs to be Kerberos
> authenticated. This allows the process running DAGClient/TezClient to
> contact the RM and get tokens to communicate with the AM.
>
> 2) The TezClient/DAGClient uses the tokens obtained from the RM and
> populates it into the current user UGI (ie the use who is running
> TezClient/DAGClient). The RPC to the AM will try to authenticate the
> current user using the tokens just added to the current users UGI.
>
>
>
> In a non-secure environment, no tokens are needed. So I am guessing that
> your are running in a secure env.
>
>
>
> Given the above info, what is happening in your case. Whichever user the
> client is running under, it looks like it can authenticate to the RM to get
> the app report. So it should have gotten tokens to access the AM. Its not
> clear what you mean by user “subroto” being privileged and the real user
> not considered by Tez. It looks like you are running the client as user
> “subroto”. Who is “subroto” and who is the real user?
>
>
>
> Does this happen always or occasionally. There is a known race condition
> in YARN where the client gets tokens before the AM gets the key to validate
> the tokens.
>
>
>
> You can turn on debug logging and see the SASL negotiation logs to get
> more info on whats happening. You may add a debug log in getAMProxy() to
> verify that token were obtained from the RM and added to the UGI.
>
>
>
> It may help if you describe your scenario. What are you trying to achieve
> by impersonation and how are you trying to do that. We recently added ACLs
> in case that works for your scenario.
>
>
>
> *From:* Subroto Sanyal [mailto:[email protected]]
> *Sent:* Sunday, August 31, 2014 8:59 PM
> *To:* [email protected]
> *Subject:* Re: Tez with secured hadoop
>
>
>
> Hi Bikas,
>
>
>
> In the method:
>
> org.apache.tez.client.TezClientUtils.getAMProxy(Configuration, String,
> int, Token) a UGI is getting created with name of the current user. I think
> in this process it ignores all the security things and making the
> authentication mode as "SIMPLE". I have piece of code which tries to create
> a TezClient and it keeps throwing the exception:
>
>
>
> [anonymous] WARN [2014-08-28 03:37:50.181] [MrPlanRunnerV2]
> (UserGroupInformation.java:1551) - PriviledgedActionException as:subroto
> (auth:SIMPLE) cause:java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot
> authenticate via:[TOKEN, KERBEROS]
>
> [anonymous] INFO [2014-08-28 03:37:50.182] [MrPlanRunnerV2]
> (TezClient.java:539) - Failed to retrieve AM Status via proxy
>
> com.google.protobuf.ServiceException: java.io.IOException: Failed on local
> exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot
> authenticate via:[TOKEN, KERBEROS]; Host Details : local host is:
> "domU-12-31-39-0F-74-32/10.193.119.192"; destination host is:
> "domU-12-31-39-0C-7D-37":59431;
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:216)
>
> at com.sun.proxy.$Proxy111.getAMStatus(Unknown Source)
>
> at org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:532)
>
> at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:607)
>
> at subroto.tez.TezClusterSession$2.run(TezClusterSession.java:180)
>
>
>
> I m trying to achieve impersonation. Here user "subroto" is privileged
> user and the real user is not at all considered by the Tez Code.
>
>
>
> Request some suggestion on this.
>
>
>
> On Tue, Aug 19, 2014 at 11:18 PM, Bikas Saha <[email protected]>
> wrote:
>
> There is nothing special that you need to do if you are already running
> secure Map Reduce jobs. The client needs to run in a Kerberized
> authenticated context. After that if you are using the built-in library of
> inputs/outputs etc then they should be taking care of all the access
> credentials for you when using the 0.5 API. I
>
>
>
> If you are using 0.4 API to write your job then you may need to use
> additional APIs for passing credentials to the application. Look for
> credentials in
> https://github.com/apache/tez/blob/branch-0.4.0-incubating/tez-mapreduce-examples/src/main/java/org/apache/tez/mapreduce/examples/FilterLinesByWord.java
> and also *public* *synchronized* DAG *addURIsForCredentials(*Collection*<*
> URI*>* uris*)*
>
>
>
> The second method is a shortcut if you are using HDFS files for input. It
> obtains credentials for you from a collection of HDFS input URIs.
>
>
>
> Bikas
>
>
>
> *From:* Subroto Sanyal [mailto:[email protected]]
> *Sent:* Tuesday, August 19, 2014 3:30 AM
> *To:* [email protected]
> *Subject:* Tez with secured hadoop
>
>
>
> hi
>
>
>
> Tez works on secure hadoop cluster since tez-0.3.
>
> Is there any documentation available about configuring TezClient to make
> it work?
>
>
>
> --
> Cheers,
> *Subroto Sanyal*
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
>
>
>
> --
> Cheers,
> *Subroto Sanyal*
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
>
>
>
> --
> Cheers,
> *Subroto Sanyal*
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Cheers,
*Subroto Sanyal*
