[GitHub] flink pull request: [FLINK-592] Add support for Kerberos secured Y...
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/358#issuecomment-73013486 @warneke Thank you for your help! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-592] Add support for Kerberos secured Y...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/358#issuecomment-72937500 Thank you. Thats good to hear. > On 04.02.2015, at 21:56, Daniel Warneke wrote: > > Tested the code and everything works as expected now. Great job! > > â > Reply to this email directly or view it on GitHub. > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-592] Add support for Kerberos secured Y...
Github user warneke commented on the pull request: https://github.com/apache/flink/pull/358#issuecomment-72937077 Tested the code and everything works as expected now. Great job! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-592] Add support for Kerberos secured Y...
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/358#issuecomment-72870536 @warneke Thanks for reporting. If the above issues have been resolved, I suggest to merge the changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-592] Add support for Kerberos secured Y...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/358 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-592] Add support for Kerberos secured Y...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/358#issuecomment-72856331 Thank you for the good feedback! @mxm and I updated the pull request and addressed your concerns. I'm now running the tests on Travis. If they pass I'm going to merge the changes. @warneke: It would be nice if you could test the code again to see if we really fixed the issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-592] Add support for Kerberos secured Y...
Github user warneke commented on the pull request: https://github.com/apache/flink/pull/358#issuecomment-72739181 Hi, I tried the code and found the following three problems: __Flink launch script (bin/flink) points to the wrong log4j configuration file__ log4j:ERROR Could not read configuration file from URL [file:/home/warneke/workspace/flink/flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-yarn-0.9-SNAPSHOT/bin/../conf/log4j-cli.properties]. java.io.FileNotFoundException: /home/warneke/workspace/flink/flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-yarn-0.9-SNAPSHOT/bin/../conf/log4j-cli.properties (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.(FileInputStream.java:146) at java.io.FileInputStream.(FileInputStream.java:101) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.(LogManager.java:127) at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:288) at org.apache.flink.client.FlinkYarnSessionCli.(FlinkYarnSessionCli.java:53) at org.apache.flink.client.CliFrontend.(CliFrontend.java:81) __Flink YARN client hangs indefinitely when user has no Kerberos ticket__ When the user launches Flink without a Kerberos ticket, the client loops indefinitely in the following function call instead of throwing an exception: "main" prio=10 tid=0x7febe800a000 nid=0x1770 waiting on condition [0x7febedf82000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:151) at com.sun.proxy.$Proxy12.getNewApplication(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:191) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:199) at org.apache.flink.yarn.FlinkYarnClient.deployInternal(FlinkYarnClient.java:303) at org.apache.flink.yarn.FlinkYarnClient$1.run(FlinkYarnClient.java:283) at org.apache.flink.yarn.FlinkYarnClient$1.run(FlinkYarnClient.java:280) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.flink.yarn.FlinkYarnClient.deploy(FlinkYarnClient.java:280) at org.apache.flink.client.CliFrontend.getClient(CliFrontend.java:921) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1067) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1091) Interestingly, the code passes the ugi.doAs call even without a valid ticket. In my environment (CDH5.2.0), UserGroupInformation.getCurrentUser() produces the following output inside the doAs run function: With valid ticket: warneke@WARNEKE.LOCAL (auth:KERBEROS) Without valid ticket: warneke (auth:KERBEROS) __Problem with hard-coded default queue name__ Even with a valid Kerberos ticket, the YARN deployment fails with the following error message on CDH5.2.0 java.lang.RuntimeException: Error deploying the YARN cluster at org.apache.flink.client.CliFrontend.getClient(CliFrontend.java:923) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1066) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1090) Caused by: org.apache.flink.yarn.FlinkYarnClient$YarnDeploymentException: The specified queue 'default' does not exist. Available queues: root.default, at org.apache.flink.yarn.FlinkYarnClient.deployInternal(FlinkYarnClient.java:325) at org.apache.flink.yarn.FlinkYarnClient$1.run(FlinkYarnClient.java:286) at org.apache.flink.yarn.FlinkYarnClient$1.run(FlinkYarnClient.java:280) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) a
[GitHub] flink pull request: [FLINK-592] Add support for Kerberos secured Y...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/358 [FLINK-592] Add support for Kerberos secured YARN setups to Flink. This pull request is basically a port of @warneke's branch (https://github.com/warneke/flink/tree/security) to the latest `master` of Flink. The port has been done mostly by @mxm. We tested the change on google compute engine (non-secure setup, to ensure that everything is working as before) and a local secure YARN setup with Kerberos. Open issues: - Test token renewal Once the open issues have been resolved, I would like to merge this asap because a user was asking for this on the mailing list. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink592 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/358.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #358 commit 3fc8d47f3f7322285539454c7a80a8cec4ba043f Author: Max Date: 2015-02-02T15:09:18Z [FLINK-592] Add support for Kerberos secured YARN setups to Flink. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
