[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-gearpump/pull/231 ---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/231 merging. Thanks. ---
[jira] [Commented] (GEARPUMP-355) AppMasterResolver fails to run against a kerberized Hadoop cluster
[ https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16213475#comment-16213475 ] ASF GitHub Bot commented on GEARPUMP-355: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-gearpump/pull/231 > AppMasterResolver fails to run against a kerberized Hadoop cluster > -- > > Key: GEARPUMP-355 > URL: https://issues.apache.org/jira/browse/GEARPUMP-355 > Project: Apache Gearpump > Issue Type: Bug > Components: security, yarn >Affects Versions: 0.8.4 >Reporter: Timea Magyar >Assignee: Timea Magyar > Fix For: 0.8.5 > > > When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn > environment, after the Application Master address has been resolved as a > prerequisite, the YarnAppMaster (responsible for starting GearPump masters, > workers, UI servers as Yarn containers) address (actor reference) must be > obtained via Kerberos/Spnego. (Kerberos over http) > The current implementation for this resides in the AppMasterResolver class > and is using an apache http client (version 3.x) for establishing a > connection to the Application Master and obtain the above YarnAppMaster actor > reference. Since the apache http client does not support the negotiate > authentication scheme in version 3.x (required for a connection over > kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop > cluster set-up. > I tested this in a secured/kerberized CDH 5.7.5 environment. I would like to > provide a patch for this by adapting the SPNEGO-enabled Hadoop web > connection code from WebHDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GEARPUMP-355) AppMasterResolver fails to run against a kerberized Hadoop cluster
[ https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang resolved GEARPUMP-355. - Resolution: Fixed Fix Version/s: (was: 0.8.4) 0.8.5 Issue resolved by pull request 231 [https://github.com/apache/incubator-gearpump/pull/231] > AppMasterResolver fails to run against a kerberized Hadoop cluster > -- > > Key: GEARPUMP-355 > URL: https://issues.apache.org/jira/browse/GEARPUMP-355 > Project: Apache Gearpump > Issue Type: Bug > Components: security, yarn >Affects Versions: 0.8.4 >Reporter: Timea Magyar >Assignee: Timea Magyar > Fix For: 0.8.5 > > > When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn > environment, after the Application Master address has been resolved as a > prerequisite, the YarnAppMaster (responsible for starting GearPump masters, > workers, UI servers as Yarn containers) address (actor reference) must be > obtained via Kerberos/Spnego. (Kerberos over http) > The current implementation for this resides in the AppMasterResolver class > and is using an apache http client (version 3.x) for establishing a > connection to the Application Master and obtain the above YarnAppMaster actor > reference. Since the apache http client does not support the negotiate > authentication scheme in version 3.x (required for a connection over > kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop > cluster set-up. > I tested this in a secured/kerberized CDH 5.7.5 environment. I would like to > provide a patch for this by adapting the SPNEGO-enabled Hadoop web > connection code from WebHDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (GEARPUMP-355) AppMasterResolver fails to run against a kerberized Hadoop cluster
[ https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16213474#comment-16213474 ] ASF GitHub Bot commented on GEARPUMP-355: - Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/231 merging. Thanks. > AppMasterResolver fails to run against a kerberized Hadoop cluster > -- > > Key: GEARPUMP-355 > URL: https://issues.apache.org/jira/browse/GEARPUMP-355 > Project: Apache Gearpump > Issue Type: Bug > Components: security, yarn >Affects Versions: 0.8.4 >Reporter: Timea Magyar >Assignee: Timea Magyar > Fix For: 0.8.4 > > > When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn > environment, after the Application Master address has been resolved as a > prerequisite, the YarnAppMaster (responsible for starting GearPump masters, > workers, UI servers as Yarn containers) address (actor reference) must be > obtained via Kerberos/Spnego. (Kerberos over http) > The current implementation for this resides in the AppMasterResolver class > and is using an apache http client (version 3.x) for establishing a > connection to the Application Master and obtain the above YarnAppMaster actor > reference. Since the apache http client does not support the negotiate > authentication scheme in version 3.x (required for a connection over > kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop > cluster set-up. > I tested this in a secured/kerberized CDH 5.7.5 environment. I would like to > provide a patch for this by adapting the SPNEGO-enabled Hadoop web > connection code from WebHDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Github user titikakatoo commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r146019076
--- Diff:
experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala
---
@@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system:
ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system:
ActorSystem): ActorRef = {
+val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
+LOG.info(s"appMasterPath=$appMasterPath")
+
+val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+val url: URL = new URL(appMasterPath)
+val connection: HttpURLConnection =
connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+connection.setInstanceFollowRedirects(true)
+
+try {
+ connection.connect()
+} catch {
+ case e: IOException =>
+LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
+}
+
+val status = connection.getResponseCode
+if (status == 200) {
+ val stream: java.io.InputStream = connection.getInputStream
+ val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
+ LOG.info("Successfully resolved AppMaster address: " + response)
+ connection.disconnect()
+ AkkaHelper.actorFor(system, response)
+} else {
+ connection.disconnect()
+ throw new IOException("Fail to resolve AppMaster address, please
make sure " +
+s"${report.getOriginalTrackingUrl} is accessible...")
--- End diff --
Done.
---
[jira] [Commented] (GEARPUMP-355) AppMasterResolver fails to run against a kerberized Hadoop cluster
[
https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212907#comment-16212907
]
ASF GitHub Bot commented on GEARPUMP-355:
-
Github user titikakatoo commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r146019076
--- Diff:
experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala
---
@@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system:
ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system:
ActorSystem): ActorRef = {
+val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
+LOG.info(s"appMasterPath=$appMasterPath")
+
+val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+val url: URL = new URL(appMasterPath)
+val connection: HttpURLConnection =
connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+connection.setInstanceFollowRedirects(true)
+
+try {
+ connection.connect()
+} catch {
+ case e: IOException =>
+LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
+}
+
+val status = connection.getResponseCode
+if (status == 200) {
+ val stream: java.io.InputStream = connection.getInputStream
+ val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
+ LOG.info("Successfully resolved AppMaster address: " + response)
+ connection.disconnect()
+ AkkaHelper.actorFor(system, response)
+} else {
+ connection.disconnect()
+ throw new IOException("Fail to resolve AppMaster address, please
make sure " +
+s"${report.getOriginalTrackingUrl} is accessible...")
--- End diff --
Done.
> AppMasterResolver fails to run against a kerberized Hadoop cluster
> --
>
> Key: GEARPUMP-355
> URL: https://issues.apache.org/jira/browse/GEARPUMP-355
> Project: Apache Gearpump
> Issue Type: Bug
> Components: security, yarn
>Affects Versions: 0.8.4
>Reporter: Timea Magyar
>Assignee: Timea Magyar
> Fix For: 0.8.4
>
>
> When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn
> environment, after the Application Master address has been resolved as a
> prerequisite, the YarnAppMaster (responsible for starting GearPump masters,
> workers, UI servers as Yarn containers) address (actor reference) must be
> obtained via Kerberos/Spnego. (Kerberos over http)
> The current implementation for this resides in the AppMasterResolver class
> and is using an apache http client (version 3.x) for establishing a
> connection to the Application Master and obtain the above YarnAppMaster actor
> reference. Since the apache http client does not support the negotiate
> authentication scheme in version 3.x (required for a connection over
> kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop
> cluster set-up.
> I tested this in a secured/kerberized CDH 5.7.5 environment. I would like to
> provide a patch for this by adapting the SPNEGO-enabled Hadoop web
> connection code from WebHDFS.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Github user titikakatoo commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r146003668
--- Diff:
experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala
---
@@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system:
ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system:
ActorSystem): ActorRef = {
+val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
+LOG.info(s"appMasterPath=$appMasterPath")
+
+val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+val url: URL = new URL(appMasterPath)
+val connection: HttpURLConnection =
connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+connection.setInstanceFollowRedirects(true)
+
+try {
+ connection.connect()
+} catch {
+ case e: IOException =>
+LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
+}
+
+val status = connection.getResponseCode
+if (status == 200) {
+ val stream: java.io.InputStream = connection.getInputStream
+ val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
+ LOG.info("Successfully resolved AppMaster address: " + response)
+ connection.disconnect()
+ AkkaHelper.actorFor(system, response)
+} else {
+ connection.disconnect()
+ throw new IOException("Fail to resolve AppMaster address, please
make sure " +
+s"${report.getOriginalTrackingUrl} is accessible...")
--- End diff --
@manuzhang getTrackingURL was merged already in the master branch.
---
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r146002316
--- Diff:
experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala
---
@@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system:
ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system:
ActorSystem): ActorRef = {
+val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
+LOG.info(s"appMasterPath=$appMasterPath")
+
+val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+val url: URL = new URL(appMasterPath)
+val connection: HttpURLConnection =
connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+connection.setInstanceFollowRedirects(true)
+
+try {
+ connection.connect()
+} catch {
+ case e: IOException =>
+LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
+}
+
+val status = connection.getResponseCode
+if (status == 200) {
+ val stream: java.io.InputStream = connection.getInputStream
+ val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
+ LOG.info("Successfully resolved AppMaster address: " + response)
+ connection.disconnect()
+ AkkaHelper.actorFor(system, response)
+} else {
+ connection.disconnect()
+ throw new IOException("Fail to resolve AppMaster address, please
make sure " +
+s"${report.getOriginalTrackingUrl} is accessible...")
--- End diff --
should this be `getTrackingUrl` now ?
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Github user titikakatoo commented on the issue: https://github.com/apache/incubator-gearpump/pull/231 @manuzhang , @huafengw removed unused imports. ---
[GitHub] incubator-gearpump pull request #230: [GEARPUMP-354] Fix the tracking url re...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-gearpump/pull/230 ---
[GitHub] incubator-gearpump issue #230: [GEARPUMP-354] Fix the tracking url resolutio...
Github user huafengw commented on the issue: https://github.com/apache/incubator-gearpump/pull/230 LGTM +1 ---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Github user huafengw commented on the issue: https://github.com/apache/incubator-gearpump/pull/231 I've tested the pr and no regression was found. But it seems this change introduces some unused imports to the affected files. We'll merge it once they were resolved. @titikakatoo ---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/231 LGTM. @huafengw more comments ? ---
[jira] [Commented] (GEARPUMP-355) AppMasterResolver fails to run against a kerberized Hadoop cluster
[
https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212823#comment-16212823
]
ASF GitHub Bot commented on GEARPUMP-355:
-
Github user titikakatoo commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r146003668
--- Diff:
experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala
---
@@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system:
ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system:
ActorSystem): ActorRef = {
+val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
+LOG.info(s"appMasterPath=$appMasterPath")
+
+val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+val url: URL = new URL(appMasterPath)
+val connection: HttpURLConnection =
connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+connection.setInstanceFollowRedirects(true)
+
+try {
+ connection.connect()
+} catch {
+ case e: IOException =>
+LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
+}
+
+val status = connection.getResponseCode
+if (status == 200) {
+ val stream: java.io.InputStream = connection.getInputStream
+ val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
+ LOG.info("Successfully resolved AppMaster address: " + response)
+ connection.disconnect()
+ AkkaHelper.actorFor(system, response)
+} else {
+ connection.disconnect()
+ throw new IOException("Fail to resolve AppMaster address, please
make sure " +
+s"${report.getOriginalTrackingUrl} is accessible...")
--- End diff --
@manuzhang getTrackingURL was merged already in the master branch.
> AppMasterResolver fails to run against a kerberized Hadoop cluster
> --
>
> Key: GEARPUMP-355
> URL: https://issues.apache.org/jira/browse/GEARPUMP-355
> Project: Apache Gearpump
> Issue Type: Bug
> Components: security, yarn
>Affects Versions: 0.8.4
>Reporter: Timea Magyar
>Assignee: Timea Magyar
> Fix For: 0.8.4
>
>
> When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn
> environment, after the Application Master address has been resolved as a
> prerequisite, the YarnAppMaster (responsible for starting GearPump masters,
> workers, UI servers as Yarn containers) address (actor reference) must be
> obtained via Kerberos/Spnego. (Kerberos over http)
> The current implementation for this resides in the AppMasterResolver class
> and is using an apache http client (version 3.x) for establishing a
> connection to the Application Master and obtain the above YarnAppMaster actor
> reference. Since the apache http client does not support the negotiate
> authentication scheme in version 3.x (required for a connection over
> kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop
> cluster set-up.
> I tested this in a secured/kerberized CDH 5.7.5 environment. I would like to
> provide a patch for this by adapting the SPNEGO-enabled Hadoop web
> connection code from WebHDFS.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (GEARPUMP-355) AppMasterResolver fails to run against a kerberized Hadoop cluster
[
https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212820#comment-16212820
]
ASF GitHub Bot commented on GEARPUMP-355:
-
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r146002316
--- Diff:
experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala
---
@@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system:
ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system:
ActorSystem): ActorRef = {
+val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
+LOG.info(s"appMasterPath=$appMasterPath")
+
+val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+val url: URL = new URL(appMasterPath)
+val connection: HttpURLConnection =
connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+connection.setInstanceFollowRedirects(true)
+
+try {
+ connection.connect()
+} catch {
+ case e: IOException =>
+LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
+}
+
+val status = connection.getResponseCode
+if (status == 200) {
+ val stream: java.io.InputStream = connection.getInputStream
+ val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
+ LOG.info("Successfully resolved AppMaster address: " + response)
+ connection.disconnect()
+ AkkaHelper.actorFor(system, response)
+} else {
+ connection.disconnect()
+ throw new IOException("Fail to resolve AppMaster address, please
make sure " +
+s"${report.getOriginalTrackingUrl} is accessible...")
--- End diff --
should this be `getTrackingUrl` now ?
> AppMasterResolver fails to run against a kerberized Hadoop cluster
> --
>
> Key: GEARPUMP-355
> URL: https://issues.apache.org/jira/browse/GEARPUMP-355
> Project: Apache Gearpump
> Issue Type: Bug
> Components: security, yarn
>Affects Versions: 0.8.4
>Reporter: Timea Magyar
>Assignee: Timea Magyar
> Fix For: 0.8.4
>
>
> When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn
> environment, after the Application Master address has been resolved as a
> prerequisite, the YarnAppMaster (responsible for starting GearPump masters,
> workers, UI servers as Yarn containers) address (actor reference) must be
> obtained via Kerberos/Spnego. (Kerberos over http)
> The current implementation for this resides in the AppMasterResolver class
> and is using an apache http client (version 3.x) for establishing a
> connection to the Application Master and obtain the above YarnAppMaster actor
> reference. Since the apache http client does not support the negotiate
> authentication scheme in version 3.x (required for a connection over
> kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop
> cluster set-up.
> I tested this in a secured/kerberized CDH 5.7.5 environment. I would like to
> provide a patch for this by adapting the SPNEGO-enabled Hadoop web
> connection code from WebHDFS.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (GEARPUMP-355) AppMasterResolver fails to run against a kerberized Hadoop cluster
[ https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212402#comment-16212402 ] ASF GitHub Bot commented on GEARPUMP-355: - Github user titikakatoo commented on the issue: https://github.com/apache/incubator-gearpump/pull/231 @manuzhang , @huafengw removed unused imports. > AppMasterResolver fails to run against a kerberized Hadoop cluster > -- > > Key: GEARPUMP-355 > URL: https://issues.apache.org/jira/browse/GEARPUMP-355 > Project: Apache Gearpump > Issue Type: Bug > Components: security, yarn >Affects Versions: 0.8.4 >Reporter: Timea Magyar >Assignee: Timea Magyar > Fix For: 0.8.4 > > > When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn > environment, after the Application Master address has been resolved as a > prerequisite, the YarnAppMaster (responsible for starting GearPump masters, > workers, UI servers as Yarn containers) address (actor reference) must be > obtained via Kerberos/Spnego. (Kerberos over http) > The current implementation for this resides in the AppMasterResolver class > and is using an apache http client (version 3.x) for establishing a > connection to the Application Master and obtain the above YarnAppMaster actor > reference. Since the apache http client does not support the negotiate > authentication scheme in version 3.x (required for a connection over > kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop > cluster set-up. > I tested this in a secured/kerberized CDH 5.7.5 environment. I would like to > provide a patch for this by adapting the SPNEGO-enabled Hadoop web > connection code from WebHDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GEARPUMP-354) The tracking URL of a proxied Application Master can't be resolved when launching Gearpump on Yarn
[ https://issues.apache.org/jira/browse/GEARPUMP-354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huafeng Wang resolved GEARPUMP-354. --- Resolution: Fixed > The tracking URL of a proxied Application Master can't be resolved when > launching Gearpump on Yarn > -- > > Key: GEARPUMP-354 > URL: https://issues.apache.org/jira/browse/GEARPUMP-354 > Project: Apache Gearpump > Issue Type: Bug > Components: yarn >Affects Versions: 0.8.4 >Reporter: Timea Magyar >Assignee: Timea Magyar >Priority: Critical > Fix For: 0.8.4 > > > When I was testing the launch of a GEARPUMP cluster on secured Yarn, the > proxied application master path could not be resolved properly. Looking at > the GEARPUMP source code it turned out that instead of invoking the tracking > url on the application report the original tracking url was used instead. > This will never work in a proxied AM set up. In contrary invoking the > tracking url will always work in a proxied and non-proxied environment alike. > I created a small patch for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (GEARPUMP-355) AppMasterResolver fails to run against a kerberized Hadoop cluster
[ https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huafeng Wang reassigned GEARPUMP-355: - Assignee: Timea Magyar > AppMasterResolver fails to run against a kerberized Hadoop cluster > -- > > Key: GEARPUMP-355 > URL: https://issues.apache.org/jira/browse/GEARPUMP-355 > Project: Apache Gearpump > Issue Type: Bug > Components: security, yarn >Affects Versions: 0.8.4 >Reporter: Timea Magyar >Assignee: Timea Magyar > Fix For: 0.8.4 > > > When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn > environment, after the Application Master address has been resolved as a > prerequisite, the YarnAppMaster (responsible for starting GearPump masters, > workers, UI servers as Yarn containers) address (actor reference) must be > obtained via Kerberos/Spnego. (Kerberos over http) > The current implementation for this resides in the AppMasterResolver class > and is using an apache http client (version 3.x) for establishing a > connection to the Application Master and obtain the above YarnAppMaster actor > reference. Since the apache http client does not support the negotiate > authentication scheme in version 3.x (required for a connection over > kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop > cluster set-up. > I tested this in a secured/kerberized CDH 5.7.5 environment. I would like to > provide a patch for this by adapting the SPNEGO-enabled Hadoop web > connection code from WebHDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (GEARPUMP-354) The tracking URL of a proxied Application Master can't be resolved when launching Gearpump on Yarn
[ https://issues.apache.org/jira/browse/GEARPUMP-354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huafeng Wang reassigned GEARPUMP-354: - Assignee: Timea Magyar > The tracking URL of a proxied Application Master can't be resolved when > launching Gearpump on Yarn > -- > > Key: GEARPUMP-354 > URL: https://issues.apache.org/jira/browse/GEARPUMP-354 > Project: Apache Gearpump > Issue Type: Bug > Components: yarn >Affects Versions: 0.8.4 >Reporter: Timea Magyar >Assignee: Timea Magyar >Priority: Critical > Fix For: 0.8.4 > > > When I was testing the launch of a GEARPUMP cluster on secured Yarn, the > proxied application master path could not be resolved properly. Looking at > the GEARPUMP source code it turned out that instead of invoking the tracking > url on the application report the original tracking url was used instead. > This will never work in a proxied AM set up. In contrary invoking the > tracking url will always work in a proxied and non-proxied environment alike. > I created a small patch for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (GEARPUMP-354) The tracking URL of a proxied Application Master can't be resolved when launching Gearpump on Yarn
[ https://issues.apache.org/jira/browse/GEARPUMP-354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212341#comment-16212341 ] ASF GitHub Bot commented on GEARPUMP-354: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-gearpump/pull/230 > The tracking URL of a proxied Application Master can't be resolved when > launching Gearpump on Yarn > -- > > Key: GEARPUMP-354 > URL: https://issues.apache.org/jira/browse/GEARPUMP-354 > Project: Apache Gearpump > Issue Type: Bug > Components: yarn >Affects Versions: 0.8.4 >Reporter: Timea Magyar >Priority: Critical > Fix For: 0.8.4 > > > When I was testing the launch of a GEARPUMP cluster on secured Yarn, the > proxied application master path could not be resolved properly. Looking at > the GEARPUMP source code it turned out that instead of invoking the tracking > url on the application report the original tracking url was used instead. > This will never work in a proxied AM set up. In contrary invoking the > tracking url will always work in a proxied and non-proxied environment alike. > I created a small patch for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (GEARPUMP-354) The tracking URL of a proxied Application Master can't be resolved when launching Gearpump on Yarn
[ https://issues.apache.org/jira/browse/GEARPUMP-354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212340#comment-16212340 ] ASF GitHub Bot commented on GEARPUMP-354: - Github user huafengw commented on the issue: https://github.com/apache/incubator-gearpump/pull/230 LGTM +1 > The tracking URL of a proxied Application Master can't be resolved when > launching Gearpump on Yarn > -- > > Key: GEARPUMP-354 > URL: https://issues.apache.org/jira/browse/GEARPUMP-354 > Project: Apache Gearpump > Issue Type: Bug > Components: yarn >Affects Versions: 0.8.4 >Reporter: Timea Magyar >Priority: Critical > Fix For: 0.8.4 > > > When I was testing the launch of a GEARPUMP cluster on secured Yarn, the > proxied application master path could not be resolved properly. Looking at > the GEARPUMP source code it turned out that instead of invoking the tracking > url on the application report the original tracking url was used instead. > This will never work in a proxied AM set up. In contrary invoking the > tracking url will always work in a proxied and non-proxied environment alike. > I created a small patch for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (GEARPUMP-355) AppMasterResolver fails to run against a kerberized Hadoop cluster
[ https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212339#comment-16212339 ] ASF GitHub Bot commented on GEARPUMP-355: - Github user huafengw commented on the issue: https://github.com/apache/incubator-gearpump/pull/231 I've tested the pr and no regression was found. But it seems this change introduces some unused imports to the affected files. We'll merge it once they were resolved. @titikakatoo > AppMasterResolver fails to run against a kerberized Hadoop cluster > -- > > Key: GEARPUMP-355 > URL: https://issues.apache.org/jira/browse/GEARPUMP-355 > Project: Apache Gearpump > Issue Type: Bug > Components: security, yarn >Affects Versions: 0.8.4 >Reporter: Timea Magyar > Fix For: 0.8.4 > > > When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn > environment, after the Application Master address has been resolved as a > prerequisite, the YarnAppMaster (responsible for starting GearPump masters, > workers, UI servers as Yarn containers) address (actor reference) must be > obtained via Kerberos/Spnego. (Kerberos over http) > The current implementation for this resides in the AppMasterResolver class > and is using an apache http client (version 3.x) for establishing a > connection to the Application Master and obtain the above YarnAppMaster actor > reference. Since the apache http client does not support the negotiate > authentication scheme in version 3.x (required for a connection over > kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop > cluster set-up. > I tested this in a secured/kerberized CDH 5.7.5 environment. I would like to > provide a patch for this by adapting the SPNEGO-enabled Hadoop web > connection code from WebHDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
