[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond
[
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374807#comment-15374807
]
Harsh J commented on OOZIE-2413:
Note that this issue can happen even in cases of a responsive KDC. The basic
flaw is the second point of the description, in that except for MR1, HDFS,
YARN, HBase clients, the rest (such as Hive HMS client or HS2 JDBC client) do
not have mechanisms to ensure a valid TGT before making connection calls. With
this change a presence of valid TGT in the memory gets ensured (with a new
login where necessary) regardless of what form of client the credential system
builds up.
> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
> Issue Type: Bug
> Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch, OOZIE-2413.002.patch,
> OOZIE-2413.003.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos
> credentials have expired when it tries to get the delegation token. The
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient
> for MR, etc) is because they call
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to
> connect.
> We should do a similar fix by calling
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a
> Credentials implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond
[
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044306#comment-15044306
]
Rohini Palaniswamy commented on OOZIE-2413:
---
+1. Can you rename ensureKerberos() to ensureKerberosLogin() before checking in.
> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
> Issue Type: Bug
> Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch, OOZIE-2413.002.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos
> credentials have expired when it tries to get the delegation token. The
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient
> for MR, etc) is because they call
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to
> connect.
> We should do a similar fix by calling
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a
> Credentials implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond
[
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042722#comment-15042722
]
Hadoop QA commented on OOZIE-2413:
--
Testing JIRA OOZIE-2413
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2619/
> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
> Issue Type: Bug
> Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch, OOZIE-2413.002.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos
> credentials have expired when it tries to get the delegation token. The
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient
> for MR, etc) is because they call
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to
> connect.
> We should do a similar fix by calling
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a
> Credentials implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond
[
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042417#comment-15042417
]
Rohini Palaniswamy commented on OOZIE-2413:
---
You can put code in a new CredentialProvider method too to maintain the
abstraction, but need to call before this loop to avoid checking TGT multiple
times.
> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
> Issue Type: Bug
> Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos
> credentials have expired when it tries to get the delegation token. The
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient
> for MR, etc) is because they call
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to
> connect.
> We should do a similar fix by calling
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a
> Credentials implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond
[
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042382#comment-15042382
]
Rohini Palaniswamy commented on OOZIE-2413:
---
This should be done in JavaActionExecutor.setCredentialTokens() so that the
check is done only once for any action even if there are multiple credentials.
{code}
if (context != null && action != null && credPropertiesMap != null) {
+LOG.debug("About to relogin from keytab");
+UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab();
+LOG.debug("Relogin from keytab successful");
for (Entry entry :
credPropertiesMap.entrySet()) {
{code}
> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
> Issue Type: Bug
> Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos
> credentials have expired when it tries to get the delegation token. The
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient
> for MR, etc) is because they call
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to
> connect.
> We should do a similar fix by calling
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a
> Credentials implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond
[
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042172#comment-15042172
]
Robert Kanter commented on OOZIE-2413:
--
Test failures unrelated.
> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
> Issue Type: Bug
> Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos
> credentials have expired when it tries to get the delegation token. The
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient
> for MR, etc) is because they call
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to
> connect.
> We should do a similar fix by calling
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a
> Credentials implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2413) Kerberos credentials can expire if the KDC is slow to respond
[
https://issues.apache.org/jira/browse/OOZIE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039721#comment-15039721
]
Hadoop QA commented on OOZIE-2413:
--
Testing JIRA OOZIE-2413
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1702
.Tests failed: 4
.Tests errors: 0
.The patch failed the following testcases:
. testForNoDuplicates(org.apache.oozie.event.TestEventGeneration)
. testSamplers(org.apache.oozie.util.TestMetricsInstrumentation)
.
testbulkWfKillSuccess(org.apache.oozie.command.wf.TestBulkWorkflowXCommand)
. testUpdateSLA(org.apache.oozie.sla.TestSLAService)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2617/
> Kerberos credentials can expire if the KDC is slow to respond
> -
>
> Key: OOZIE-2413
> URL: https://issues.apache.org/jira/browse/OOZIE-2413
> Project: Oozie
> Issue Type: Bug
> Components: security
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2413.001.patch
>
>
> We've seen some very rare cases where Oozie gets a Kerberos error when trying
> to get delegation tokens via the {{Credentials}} mechanism (e.g. getting HS2
> delegation tokens).
> We finally narrowed it down to slow KDC responses, so Oozie's Kerberos
> credentials have expired when it tries to get the delegation token. The
> reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient
> for MR, etc) is because they call
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before trying to
> connect.
> We should do a similar fix by calling
> {{UserGroupInformation#checkTGTAndReloginFromKeytab()}} before using a
> Credentials implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
