[
https://issues.apache.org/jira/browse/GOBBLIN-1500?focusedWorklogId=632447&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-632447
]
ASF GitHub Bot logged work on GOBBLIN-1500:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 02/Aug/21 15:56
Start Date: 02/Aug/21 15:56
Worklog Time Spent: 10m
Work Description: sv2000 commented on a change in pull request #3345:
URL: https://github.com/apache/gobblin/pull/3345#discussion_r681079489
##########
File path:
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnConfigurationKeys.java
##########
@@ -37,6 +37,8 @@
public static final int DEFAULT_RELEASED_CONTAINERS_CACHE_EXPIRY_SECS = 300;
public static final String APP_VIEW_ACL = GOBBLIN_YARN_PREFIX + "appViewAcl";
public static final String DEFAULT_APP_VIEW_ACL = "*";
+ public static final String YARN_RESOURCEMANAGER_ADDRESS=
"yarn.resourcemanager.address";
+ public static final String POTENTIAL_YARN_RESOURCEMANAGER_ADDRESSES=
"potential.yarn.resourcemanager.addresses";
Review comment:
OTHER_YARN_RESOURCE_MANAGER_ADDRESSES instead of POTENTIAL_
##########
File path:
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java
##########
@@ -574,19 +592,21 @@ private String sanitizeApplicationId(String
applicationId) {
@VisibleForTesting
Optional<ApplicationId> getReconnectableApplicationId() throws
YarnException, IOException {
- List<ApplicationReport> applicationReports =
- this.yarnClient.getApplications(APPLICATION_TYPES,
RECONNECTABLE_APPLICATION_STATES);
- if (applicationReports == null || applicationReports.isEmpty()) {
- return Optional.absent();
- }
+ for (YarnClient yarnClient: potentialYarnClients.values()) {
+ List<ApplicationReport> applicationReports =
yarnClient.getApplications(APPLICATION_TYPES, RECONNECTABLE_APPLICATION_STATES);
+ if (applicationReports == null || applicationReports.isEmpty()) {
+ continue;
+ }
- // Try to find an application with a matching application name
- for (ApplicationReport applicationReport : applicationReports) {
- if (this.applicationName.equals(applicationReport.getName())) {
- String applicationId =
sanitizeApplicationId(applicationReport.getApplicationId().toString());
- LOGGER.info("Found reconnectable application with application ID: " +
applicationId);
- LOGGER.info("Application Tracking URL: " +
applicationReport.getTrackingUrl());
- return Optional.of(applicationReport.getApplicationId());
+ // Try to find an application with a matching application name
+ for (ApplicationReport applicationReport : applicationReports) {
Review comment:
What happens if applicationReports is null? We won't enter this loop.
Shouldn't we return Optional.absent() for that case?
##########
File path:
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java
##########
@@ -835,14 +855,22 @@ private void setupSecurityTokens(ContainerLaunchContext
containerLaunchContext)
TokenUtils.getAllFSTokens(new Configuration(), credentials, renewerName,
Optional.absent(), ConfigUtils.getStringList(this.config,
TokenUtils.OTHER_NAMENODES));
-
+ // Only pass token here and no secrets. (since there is no simple way to
remove single token/ get secrets)
+ // For RM token, only pass the RM token for the current RM, or the RM will
fail to update the token
+ Credentials finalCredentials = new Credentials();
+ for ( Token<? extends TokenIdentifier> token: credentials.getAllTokens()) {
Review comment:
Nit: Remove unnecessary whitespaces after "for" and "(".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 632447)
Time Spent: 50m (was: 40m)
> Support gobblin on yarn to be able to run on clusters with robin enabled
> ------------------------------------------------------------------------
>
> Key: GOBBLIN-1500
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1500
> Project: Apache Gobblin
> Issue Type: New Feature
> Reporter: Zihan Li
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Previously we always check with one default yarn client, but if the cluster
> is enabled with Robin and can connect to different RM, it will ends up we
> cannot find existing application and submit duplicate tasks. So now we need
> to check against different potential yarn cluster, and also make sure when
> submitting yarn application, we only have one RM token.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
