[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.8.0 Target Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks [~sershe] for reporting the issue, [~xiaochen] and [~jnp] for the reviews and discussions. I've commit the v03 patch with an extra space in log output as Xiao suggested (Good catch, [~xiaochen]!) to trunk, branch-2 and branch-2.8. > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10757.00.patch, HDFS-10757.01.patch, > HDFS-10757.02.patch, HDFS-10757.03.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HDFS-10757.03.patch Thanks [~xiaochen] for the feedback. Attach a new patch that removes the checking based on UserGroupInformation.AuthenticationMethod as it is not a reliable way for non-server scenario usage of KMSClientProvider. Add logic to use loginUser only if the currentUGI does not have either kerberos credential or kms delegation token. If the currentUGI has Kerberos credential but not kms delegation token, we should go through the SPNEGO authentication rather than using loginUGI directly. > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Attachments: HDFS-10757.00.patch, HDFS-10757.01.patch, > HDFS-10757.02.patch, HDFS-10757.03.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: (was: HADOOP-10757.02.patch) > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Attachments: HDFS-10757.00.patch, HDFS-10757.01.patch, > HDFS-10757.02.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HDFS-10757.02.patch > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Attachments: HDFS-10757.00.patch, HDFS-10757.01.patch, > HDFS-10757.02.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HADOOP-10757.02.patch Thanks [~xiaochen] for the review. Attached a new patch to address the comments. > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Attachments: HADOOP-10757.02.patch, HDFS-10757.00.patch, > HDFS-10757.01.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HDFS-10757.01.patch Thanks [~jnp] for the feedback. bq. However, in a client context, I am not sure it is set correctly in all cases. How about just doing following in getActualUGI. Good point. Attach a new patch as suggested. Also simplify some logic added from HADOOP-13381. [~asuresh]/[~xiaochen], thoughts? > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Attachments: HDFS-10757.00.patch, HDFS-10757.01.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Status: Patch Available (was: Reopened) > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Attachments: HDFS-10757.00.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HDFS-10757.00.patch Attach a low risk patch to make KMSCP stateless for discussion. > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Attachments: HDFS-10757.00.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HDFS-10757: Summary: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used (was: KMSClientProvider combined with KeyProviderCache results in wrong UGI being used) > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Critical > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
