[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[venn wu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985905#comment-16985905
 ] 

venn wu commented on FLINK-8275:


flink on yarn deploy on simple authentication hadoop cluster can read hbase in 
another hadoop cluster with kerberos authentication ?

> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / YARN
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.4.1, 1.5.0
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356589#comment-16356589
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user suez1224 commented on the issue:

https://github.com/apache/flink/pull/5172
  
It's already merged to master, and should be available in Flink 1.4.1.

On Wed, Feb 7, 2018 at 11:11 PM, toggm  wrote:

> What happended with this PR? Why was it closed?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



-- 
"So you have to trust that the dots will somehow connect in your future."



> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356577#comment-16356577
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user toggm commented on the issue:

https://github.com/apache/flink/pull/5172
  
What happended with this PR? Why was it closed?


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354391#comment-16354391
 ] 

Tzu-Li (Gordon) Tai commented on FLINK-8275:


Merged.

1.5 - 97f0cac2af3a1140fa68090d94d83e009ad1e684
1.4 - 82f3957811d2e5bcefabaa42326d3d4476e45df0

> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354362#comment-16354362
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5172


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352146#comment-16352146
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user suez1224 commented on the issue:

https://github.com/apache/flink/pull/5172
  
@tzulitai thanks a lot for the refactoring, the commit looks good to me.


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350431#comment-16350431
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/5172#discussion_r165635517
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationMasterRunner.java 
---
@@ -170,9 +160,10 @@ protected int run(String[] args) {
 
final Configuration flinkConfig = 
createConfiguration(currDir, dynamicProperties);
 
-   // set keytab principal and replace path with the local 
path of the shipped keytab file in NodeManager
-   if (keytabPath != null && remoteKeytabPrincipal != 
null) {
-   
flinkConfig.setString(SecurityOptions.KERBEROS_LOGIN_KEYTAB, keytabPath);
+   File f = new File(currDir, Utils.KEYTAB_FILE_NAME);
+   if (remoteKeytabPrincipal != null && f.exists()) {
--- End diff --

Can we re-add the debug log `LOG.debug("keytabPath: {}", keytabPath);` to 
provide more visibility to this?


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350438#comment-16350438
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/5172
  
@suez1224 Thanks a lot for the contribution!
I've had a look and the changes LGTM. I did have a comment regarding 
injecting a dependency for the runner, which I've added a commit for.

Can you take a look at 2ffa659 before I actually merge this, and let me 
know what you think? Thanks!
Will make sure this gets in for Flink 1.4.1 ..


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350430#comment-16350430
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/5172#discussion_r165636080
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java ---
@@ -142,19 +153,10 @@ public static void runYarnTaskManager(String[] args, 
final Class() {
-   @Override
-   public Integer call() {
-   try {
-   
TaskManager.selectNetworkInterfaceAndRunTaskManager(configuration, resourceId, 
taskManager);
-   }
-   catch (Throwable t) {
-   LOG.error("Error while starting 
the TaskManager", t);
-   
System.exit(TaskManager.STARTUP_FAILURE_RETURN_CODE());
-   }
-   return null;
-   }
-   });
+   if (mainRunner == null) {
--- End diff --

Not sure about this.
This is basically adding a non-production code relevant path in for testing 
purposes (i.e., it is only ever non-null in the `YarnTaskManagerRunnerTest`).

I think it would be better if we have a `protected createMainRunner(...)` 
method that can be overriden to inject the mock runner dependency for testing.


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350432#comment-16350432
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/5172#discussion_r165663546
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java ---
@@ -142,19 +153,10 @@ public static void runYarnTaskManager(String[] args, 
final Class() {
-   @Override
-   public Integer call() {
-   try {
-   
TaskManager.selectNetworkInterfaceAndRunTaskManager(configuration, resourceId, 
taskManager);
-   }
-   catch (Throwable t) {
-   LOG.error("Error while starting 
the TaskManager", t);
-   
System.exit(TaskManager.STARTUP_FAILURE_RETURN_CODE());
-   }
-   return null;
-   }
-   });
+   if (mainRunner == null) {
--- End diff --

I don't think this is a good idea, to add a code path in production code 
that exists only for the purpose of injecting a mock runner dependency in tests.

I've added another commit upon @suez1224's changes that makes 
`YarnTaskManagerRunner` a factory-like class, that creates a `Runner` 
containing all the final configurations. The unit test can then test against 
that configuration.


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16337049#comment-16337049
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user toggm commented on the issue:

https://github.com/apache/flink/pull/5172
  
We try to use apache flink in Kerberos secured environment and had to 
backport to flink 1.3.2 because of that issue. Would be good to have that PR to 
be able to migrate to the latest flink version.


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329931#comment-16329931
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user suez1224 commented on the issue:

https://github.com/apache/flink/pull/5172
  
Thanks a lot for the review, @EronWright. Could you please take another 
look?


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329929#comment-16329929
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user suez1224 commented on a diff in the pull request:

https://github.com/apache/flink/pull/5172#discussion_r162239390
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java ---
@@ -126,12 +124,6 @@ public static void runYarnTaskManager(String[] args, 
final Classhttps://issues.apache.org/jira/browse/FLINK-8390] to address the 
integration test code issue. Could you please take a look at that as well?


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329655#comment-16329655
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user suez1224 commented on a diff in the pull request:

https://github.com/apache/flink/pull/5172#discussion_r162204277
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java ---
@@ -142,19 +134,22 @@ public static void runYarnTaskManager(String[] args, 
final Class() {
-   @Override
-   public Integer call() {
-   try {
-   
TaskManager.selectNetworkInterfaceAndRunTaskManager(configuration, resourceId, 
taskManager);
-   }
-   catch (Throwable t) {
-   LOG.error("Error while starting 
the TaskManager", t);
-   
System.exit(TaskManager.STARTUP_FAILURE_RETURN_CODE());
+   if (mainRunner == null) {
--- End diff --

Refactored. Since the method is only used in this class and it need access 
to variable in its outer scope, I just put it in the same method.


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329529#comment-16329529
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user suez1224 commented on a diff in the pull request:

https://github.com/apache/flink/pull/5172#discussion_r162190016
  
--- Diff: flink-yarn/src/test/resources/flink-conf.yaml ---
@@ -0,0 +1,23 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+#
+# This is a test configuration for validation of YarnTaskManagerRunner.
+#
+
+taskmanager.tmp.dirs: /tmp
--- End diff --

This is needed for YarnTaskManagerRunnerTest.


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327389#comment-16327389
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user EronWright commented on a diff in the pull request:

https://github.com/apache/flink/pull/5172#discussion_r161822214
  
--- Diff: flink-yarn/src/test/resources/flink-conf.yaml ---
@@ -0,0 +1,23 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+#
+# This is a test configuration for validation of YarnTaskManagerRunner.
+#
+
+taskmanager.tmp.dirs: /tmp
--- End diff --

This seems unrelated.


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327388#comment-16327388
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user EronWright commented on a diff in the pull request:

https://github.com/apache/flink/pull/5172#discussion_r161822100
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java ---
@@ -142,19 +134,22 @@ public static void runYarnTaskManager(String[] args, 
final Class() {
-   @Override
-   public Integer call() {
-   try {
-   
TaskManager.selectNetworkInterfaceAndRunTaskManager(configuration, resourceId, 
taskManager);
-   }
-   catch (Throwable t) {
-   LOG.error("Error while starting 
the TaskManager", t);
-   
System.exit(TaskManager.STARTUP_FAILURE_RETURN_CODE());
+   if (mainRunner == null) {
--- End diff --

If we need to separate out the 'secured' code block, can we do that in all 
scenarios instead of the conditional logic here?   For example, move the block 
on lines 138-147 to a new method, and pass a method reference to 
`runTaskManager` as the callable.  


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327379#comment-16327379
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user EronWright commented on a diff in the pull request:

https://github.com/apache/flink/pull/5172#discussion_r161821092
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java ---
@@ -126,12 +124,6 @@ public static void runYarnTaskManager(String[] args, 
final Class Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327369#comment-16327369
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user EronWright commented on a diff in the pull request:

https://github.com/apache/flink/pull/5172#discussion_r161818655
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityUtils.java
 ---
@@ -48,7 +48,7 @@ public static SecurityContext getInstalledContext() {
}
 
@VisibleForTesting
--- End diff --

(minor) Please remove the `@VisibleForTesting` annotation.


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[Shuyi Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317067#comment-16317067
 ] 

Shuyi Chen commented on FLINK-8275:
---

[~tzulitai], [~eronwright],
I've added unittest to capture the securityconfiguration keytab related 
settings. Also, I've refactored the code to use the existing of krb5.keytab 
file instead of `_KEYTAB_PATH`. 

For the integration test code, I will create a different JIRA to follow, 
otherwise, it's better to a smaller PR rather than fixing everything in one PR.

Could you please take another look at the PR?

> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0, 1.4.1
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295515#comment-16295515
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user EronWright commented on the issue:

https://github.com/apache/flink/pull/5172
  
This PR probably fixes the problem, but it would be good to address the 
deeper problem that the code is confusing.   At least we could add some 
commentary to the code.  The specific problems, in my view, are:
1. A filename is transmitted from client -> AM -> TM in the env variable 
`_KEYTAB_PATH` but the value doesn't appear to be used.   In effect it is a 
flag asserting that a keytab named `krb5.keytab` is available.  Alternatives:
  a. Use `krb5.keytab` as the value.
  b. Eliminate the env check and simply look for the file; if present, use 
it.
2. The existence of the "integration test code" has an unclear purpose.   
It mutates the Hadoop configuration, why?   Is the code active in any 
production scenario?

Note that `YarnTaskExecutorRunner` implements this in a slightly different 
way, and should be re-tested for 1.5.0 (since I don't think it is in use yet).



> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295493#comment-16295493
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/5172
  
@suez1224 keep in mind, that contribution PRs should initially have one 
commit with the commit message appropriately set (the title of the PR would be 
a good commit message for your case).


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16294656#comment-16294656
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/5172
  
cc @EronWright, gentle ping since you mentioned to include you in the 
review :)


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16294630#comment-16294630
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/5172
  
Thanks for the PR @suez1224.
There is a duplicate JIRA for this issue: 
https://issues.apache.org/jira/browse/FLINK-8270.
Can you take a look at the suggestions explained in that JIRA, and include 
that in your solution too?


> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16294629#comment-16294629
 ] 

Tzu-Li (Gordon) Tai commented on FLINK-8275:


This is a duplicate of FLINK-8270.
[~suez1224] can you also incorporate the suggestions explained in FLINK-8270 in 
your solution?

> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path. This causes jobs to fail because the 
> TaskManager can't read the keytab.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8275) Flink YARN deployment with Kerberos enabled not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16294621#comment-16294621
 ] 

ASF GitHub Bot commented on FLINK-8275:
---

GitHub user suez1224 opened a pull request:

https://github.com/apache/flink/pull/5172

[FLINK-8275] [Security] fix keytab local path in YarnTaskManagerRunner

## Brief change log

  - Set the local keytab path in YarnTaskManagerRunner to the correct local 
path.

## Verifying this change
  - Manually verified the change by running in a production secure cluser 
with 1 JobManager and 2 TaskManagers. Both the JobManager and the Taskmanagers 
can start, and verify through kerberos metrics.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suez1224/flink keytab-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5172.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5172


commit e40bef03f18d80f423150c8b94f875d9edb5ef24
Author: Shuyi Chen 
Date:   2017-12-18T07:50:07Z

fix keytab local path in YarnTaskManagerRunner




> Flink YARN deployment with Kerberos enabled not working 
> 
>
> Key: FLINK-8275
> URL: https://issues.apache.org/jira/browse/FLINK-8275
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The local keytab path in YarnTaskManagerRunner is incorrectly set to the 
> ApplicationMaster's local keytab path.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)