[jira] [Commented] (YARN-9052) Replace all MockRM submit method definitions with a builder
[ https://issues.apache.org/jira/browse/YARN-9052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988605#comment-16988605 ] Hudson commented on YARN-9052: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17724 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17724/]) YARN-9052. Replace all MockRM submit method definitions with a builder. (sunilg: rev 682e6fdeda68b7244e92d32cd35fe317c9b32ede) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesAppsModification.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServiceAppsNodelabel.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRMAppSubmissionData.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoQueueCreation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSignalContainer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/TestPlacementProcessor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resource/TestResourceProfiles.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesSchedulerActivities.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHAForAsyncScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulingWithAllocationRequestId.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerDynamicBehavior.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestWorkPreservingRMRestartForNodeLabel.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHA.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesContainers.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerSchedulingRequestUpdate.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterServiceFair.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop
[jira] [Updated] (YARN-10006) IOException used in place of YARNException in CapaitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated YARN-10006:
--
Description:
IOException used in place of YARNException in CapacityScheduler. As per
YARNException Doc,
{code:java}
/**
* YarnException indicates exceptions from yarn servers. On the other hand,
* IOExceptions indicates exceptions from RPC layer.
*/
{code}
Below methods throws IOException but it is suppose to throw YarnException.
CapaityShedulerQueueManager#parseQueue <- initializeQueues <-
CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
was:
IOException used in place of YARNException in CapaityScheduler. As per
YARNException Doc,
{code:java}
/**
* YarnException indicates exceptions from yarn servers. On the other hand,
* IOExceptions indicates exceptions from RPC layer.
*/
{code}
Below methods throws IOException but it is suppose to throw YarnException.
CapaityShedulerQueueManager#parseQueue <- initializeQueues <-
CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
> IOException used in place of YARNException in CapaitySheduler
> -
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapaityShedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10006) IOException used in place of YARNException in CapaitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated YARN-10006:
--
Description:
IOException used in place of YARNException in CapacityScheduler. As per
YARNException Doc,
{code:java}
/**
* YarnException indicates exceptions from yarn servers. On the other hand,
* IOExceptions indicates exceptions from RPC layer.
*/
{code}
Below methods throws IOException but it is suppose to throw YarnException.
CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
was:
IOException used in place of YARNException in CapacityScheduler. As per
YARNException Doc,
{code:java}
/**
* YarnException indicates exceptions from yarn servers. On the other hand,
* IOExceptions indicates exceptions from RPC layer.
*/
{code}
Below methods throws IOException but it is suppose to throw YarnException.
CapacityShedulerQueueManager#parseQueue <- initializeQueues <-
CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
> IOException used in place of YARNException in CapaitySheduler
> -
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10006) IOException used in place of YARNException in CapaitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated YARN-10006:
--
Description:
IOException used in place of YARNException in CapacityScheduler. As per
YARNException Doc,
{code:java}
/**
* YarnException indicates exceptions from yarn servers. On the other hand,
* IOExceptions indicates exceptions from RPC layer.
*/
{code}
Below methods throws IOException but it is suppose to throw YarnException.
CapacityShedulerQueueManager#parseQueue <- initializeQueues <-
CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
was:
IOException used in place of YARNException in CapacityScheduler. As per
YARNException Doc,
{code:java}
/**
* YarnException indicates exceptions from yarn servers. On the other hand,
* IOExceptions indicates exceptions from RPC layer.
*/
{code}
Below methods throws IOException but it is suppose to throw YarnException.
CapaityShedulerQueueManager#parseQueue <- initializeQueues <-
CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
> IOException used in place of YARNException in CapaitySheduler
> -
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacityShedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10006) IOException used in place of YARNException in CapaitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988708#comment-16988708
]
Adam Antal commented on YARN-10006:
---
Uploaded patch v1.
{{CapacitySchedulerQueueManager#parseQueue}} has multiple sources where
IOException can be thrown (e.g. in the constructor of {{PlanQueue}},
{{ReservationQueue}}, {{ManagedParentQueue}}, {{LeafQueue}}). Also in
{{CapacityScheduler#initializeQueues}} there are other calls that throw
IOException (like {{#updatePlacementRules}}), so I think it's better to catch
any arbitrary type of Exceptions in {{initializeQueues}}, and emit a
{{YarnException}} there.
Also I think it's logically correct that in {{CapacityScheduler#initScheduler}}
we can throw IOException too (from {{CSConfigurationProvider#init}} and
{{#loadConfiguration}}), because it refers to failed IO operation.
> IOException used in place of YARNException in CapaitySheduler
> -
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-10006.001.patch
>
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10006) IOException used in place of YARNException in CapaitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated YARN-10006:
--
Attachment: YARN-10006.001.patch
> IOException used in place of YARNException in CapaitySheduler
> -
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-10006.001.patch
>
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[
https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988719#comment-16988719
]
Adam Antal commented on YARN-9128:
--
The patch has been in trunk for a few months now. I think we can skip the
backport, since it affects the {{Device}} object which is not yet committed in
branch-3.2 and branch-3.1.
> Use SerializationUtils from apache commons to serialize / deserialize
> ResourceMappings
> --
>
> Key: YARN-9128
> URL: https://issues.apache.org/jira/browse/YARN-9128
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9128.001.patch, YARN-9128.002.patch,
> YARN-9128.003.patch, YARN-9128.branch-3.2.001.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[ https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal resolved YARN-9128. -- Resolution: Fixed > Use SerializationUtils from apache commons to serialize / deserialize > ResourceMappings > -- > > Key: YARN-9128 > URL: https://issues.apache.org/jira/browse/YARN-9128 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9128.001.patch, YARN-9128.002.patch, > YARN-9128.003.patch, YARN-9128.branch-3.2.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10006) IOException used in place of YARNException in CapacitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-10006:
--
Summary: IOException used in place of YARNException in CapacitySheduler
(was: IOException used in place of YARNException in CapaitySheduler)
> IOException used in place of YARNException in CapacitySheduler
> --
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-10006.001.patch
>
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[ https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988756#comment-16988756 ] Szilard Nemeth commented on YARN-9128: -- Sure [~adam.antal]! > Use SerializationUtils from apache commons to serialize / deserialize > ResourceMappings > -- > > Key: YARN-9128 > URL: https://issues.apache.org/jira/browse/YARN-9128 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9128.001.patch, YARN-9128.002.patch, > YARN-9128.003.patch, YARN-9128.branch-3.2.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4029) Update LogAggregationStatus to store on finish
[ https://issues.apache.org/jira/browse/YARN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated YARN-4029: - Attachment: YARN-4029.005.patch > Update LogAggregationStatus to store on finish > -- > > Key: YARN-4029 > URL: https://issues.apache.org/jira/browse/YARN-4029 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation >Reporter: Bibin Chundatt >Assignee: Adam Antal >Priority: Major > Labels: oct16-easy > Attachments: 0001-YARN-4029.patch, 0002-YARN-4029.patch, > 0003-YARN-4029.patch, 0004-YARN-4029.patch, Image.jpg, YARN-4029.005.patch > > > Currently the log aggregation status is not getting updated to Store. When RM > is restarted will show NOT_START. > Steps to reproduce > > 1.Submit mapreduce application > 2.Wait for completion > 3.Once application is completed switch RM > *Log Aggregation Status* are changing > *Log Aggregation Status* from SUCCESS to NOT_START -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4029) Update LogAggregationStatus to store on finish
[
https://issues.apache.org/jira/browse/YARN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988809#comment-16988809
]
Adam Antal commented on YARN-4029:
--
Uploaded patch v5.
Added a configuration to enable this feature (disabled by default). Did a bit
of cleanup in {{RMAppImpl}}: now some app-related information is stored, and
have a better solution for sending application data to the state store.
Will add some tests later whether the resolution makes sense.
Turning this feature on may resolve the recovery related issue mentioned in
YARN-4946.
> Update LogAggregationStatus to store on finish
> --
>
> Key: YARN-4029
> URL: https://issues.apache.org/jira/browse/YARN-4029
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Reporter: Bibin Chundatt
>Assignee: Adam Antal
>Priority: Major
> Labels: oct16-easy
> Attachments: 0001-YARN-4029.patch, 0002-YARN-4029.patch,
> 0003-YARN-4029.patch, 0004-YARN-4029.patch, Image.jpg, YARN-4029.005.patch
>
>
> Currently the log aggregation status is not getting updated to Store. When RM
> is restarted will show NOT_START.
> Steps to reproduce
>
> 1.Submit mapreduce application
> 2.Wait for completion
> 3.Once application is completed switch RM
> *Log Aggregation Status* are changing
> *Log Aggregation Status* from SUCCESS to NOT_START
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9052) Replace all MockRM submit method definitions with a builder
[ https://issues.apache.org/jira/browse/YARN-9052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988833#comment-16988833 ] Szilard Nemeth commented on YARN-9052: -- Hi [~sunilg]! Thanks a lot for the commit! Will monitor jenkins and if I see something in relation, will get back to this refactor. thanks! > Replace all MockRM submit method definitions with a builder > --- > > Key: YARN-9052 > URL: https://issues.apache.org/jira/browse/YARN-9052 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Fix For: 3.3.0 > > Attachments: > YARN-9052-004withlogs-patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt, > YARN-9052-testlogs003-justfailed.txt, > YARN-9052-testlogs003-patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt, > YARN-9052-testlogs004-justfailed.txt, YARN-9052.001.patch, > YARN-9052.002.patch, YARN-9052.003.patch, YARN-9052.004.patch, > YARN-9052.004.withlogs.patch, YARN-9052.005.patch, YARN-9052.006.patch, > YARN-9052.007.patch, YARN-9052.008.patch, YARN-9052.009.patch, > YARN-9052.009.patch, YARN-9052.testlogs.002.patch, > YARN-9052.testlogs.002.patch, YARN-9052.testlogs.003.patch, > YARN-9052.testlogs.patch > > > MockRM has 31 definitions of submitApp, most of them having more than > acceptable number of parameters, ranging from 2 to even 22 parameters, which > makes the code completely unreadable. > On top of unreadability, it's very hard to follow what RmApp will be produced > for tests as they often pass a lot of empty / null values as parameters. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10006) IOException used in place of YARNException in CapacitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988894#comment-16988894
]
Hadoop QA commented on YARN-10006:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
59s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 101 unchanged - 1 fixed = 102 total (was 102)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 14s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.activities.TestActivitiesManager |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-10006 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12987583/YARN-10006.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4637529ed97d 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 682e6fd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checksty
[jira] [Commented] (YARN-9992) Max allocation per queue is zero for custom resource types on RM startup
[
https://issues.apache.org/jira/browse/YARN-9992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988937#comment-16988937
]
Eric Payne commented on YARN-9992:
--
Thanks [~jhung] , I verified that backporting YARN-9205 fixed the problem.
> Max allocation per queue is zero for custom resource types on RM startup
>
>
> Key: YARN-9992
> URL: https://issues.apache.org/jira/browse/YARN-9992
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9992.001.patch
>
>
> Found an issue where trying to request GPUs on a newly booted RM cannot
> schedule. It throws the exception in
> SchedulerUtils#throwInvalidResourceException:
> {noformat}
> throw new InvalidResourceRequestException(
> "Invalid resource request, requested resource type=[" + reqResourceName
> + "] < 0 or greater than maximum allowed allocation. Requested "
> + "resource=" + reqResource + ", maximum allowed allocation="
> + availableResource
> + ", please note that maximum allowed allocation is calculated "
> + "by scheduler based on maximum resource of registered "
> + "NodeManagers, which might be less than configured "
> + "maximum allocation="
> + ResourceUtils.getResourceTypesMaximumAllocation());{noformat}
> Upon refreshing scheduler (e.g. via refreshQueues), GPU scheduling works
> again.
> I think the RC is that upon scheduler refresh, resource-types.xml is loaded
> in CapacitySchedulerConfiguration (as part of YARN-7738), so when we call
> ResourceUtils#fetchMaximumAllocationFromConfig in
> CapacitySchedulerConfiguration#getMaximumAllocationPerQueue, it's able to
> fetch the {{yarn.resource-types}} config. But resource-types.xml is not
> loaded into the conf in CapacityScheduler#initScheduler, so it doesn't find
> the custom resource when computing max allocations, and the custom resource
> max allocation is 0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988962#comment-16988962 ] Szilard Nemeth commented on YARN-5106: -- Hi [~adam.antal]! Latest patch LGTM, committed to trunk. Thanks [~pbacsko] for the review. [~adam.antal]: Would you please check feasability of backporting the latest patch to branch-3.2/3.1? Thanks! > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Adam Antal >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.002.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.002.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch, YARN-5106.010.patch, > YARN-5106.011.patch, YARN-5106.012.patch, YARN-5106.013.patch, > YARN-5106.014.patch, YARN-5106.015.patch, YARN-5106.016.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-5106: - Fix Version/s: 3.3.0 > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Adam Antal >Priority: Major > Labels: newbie++ > Fix For: 3.3.0 > > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.002.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.002.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch, YARN-5106.010.patch, > YARN-5106.011.patch, YARN-5106.012.patch, YARN-5106.013.patch, > YARN-5106.014.patch, YARN-5106.015.patch, YARN-5106.016.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988976#comment-16988976 ] Hudson commented on YARN-5106: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17726 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17726/]) YARN-5106. Provide a builder interface for FairScheduler allocations for (snemeth: rev 520fe2c99b328a32c62eb5ff2b3d3562fa91f0fe) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerQueueACLs.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocationfile/AllocationFileQueueProperties.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocationfile/UserSettings.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocationfile/AllocationFileQueuePlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManagerWithFairScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesReservation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ParameterizedSchedulerTestBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAppRunnability.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppStarvation.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocationfile/AllocationFileQueuePlacementPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocationfile/AllocationFileWriter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManagerRealScheduler.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocationfile/AllocationFileSimpleQueueBuilder.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClientWithReservation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ReservationACLsTestBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigConverterTestCommons.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocationfile/AllocationFileQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-ya
[jira] [Commented] (YARN-10006) IOException used in place of YARNException in CapacitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988977#comment-16988977
]
Szilard Nemeth commented on YARN-10006:
---
[~adam.antal] Could you please check the unit test failures and checkstyle
issues? Thanks!
> IOException used in place of YARNException in CapacitySheduler
> --
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-10006.001.patch
>
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10009) In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined
[
https://issues.apache.org/jira/browse/YARN-10009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-10009:
--
Attachment: YARN-10009.002.patch
> In Capacity Scheduler, DRC can treat minimum user limit percent as a max when
> custom resource is defined
>
>
> Key: YARN-10009
> URL: https://issues.apache.org/jira/browse/YARN-10009
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.3, 2.11.0
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Critical
> Attachments: YARN-10009.001.patch, YARN-10009.002.patch,
> YARN-10009.UT.patch
>
>
> | |Memory|Vcores|res_1|
> |Queue1 Totals|20GB|100|80|
> |Resources requested by App1 in Queue1|8GB (40% of total)|8 (8% of total)|80
> (100% of total)|
> In the previous use case:
> - Queue1 has a value of 25 for {{miminum-user-limit-percent}}
> - User1 has requested 8 containers with {{}}
> each
> - {{res_1}} will be the dominant resource this case.
> All 8 containers should be assigned by the capacity scheduler, but with min
> user limit pct set to 25, only 2 containers are assigned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10009) In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined
[
https://issues.apache.org/jira/browse/YARN-10009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989002#comment-16989002
]
Eric Payne commented on YARN-10009:
---
Thank you very much for the review, [~leftnoteasy]. I uploaded a new patch for
trunk that fixes the above-mentioned comment (and a couple of other comments),
and I fixed a checkstyle warning while I was at it.
The 2.10 patch needs a little adjustment for the unit test. I will upload that
once the pre-commit build has completed on YARN-10009.002.patch.
> In Capacity Scheduler, DRC can treat minimum user limit percent as a max when
> custom resource is defined
>
>
> Key: YARN-10009
> URL: https://issues.apache.org/jira/browse/YARN-10009
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.3, 2.11.0
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Critical
> Attachments: YARN-10009.001.patch, YARN-10009.002.patch,
> YARN-10009.UT.patch
>
>
> | |Memory|Vcores|res_1|
> |Queue1 Totals|20GB|100|80|
> |Resources requested by App1 in Queue1|8GB (40% of total)|8 (8% of total)|80
> (100% of total)|
> In the previous use case:
> - Queue1 has a value of 25 for {{miminum-user-limit-percent}}
> - User1 has requested 8 containers with {{}}
> each
> - {{res_1}} will be the dominant resource this case.
> All 8 containers should be assigned by the capacity scheduler, but with min
> user limit pct set to 25, only 2 containers are assigned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10009) In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined
[
https://issues.apache.org/jira/browse/YARN-10009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989003#comment-16989003
]
Hadoop QA commented on YARN-10009:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} YARN-10009 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10009 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12987628/YARN-10009.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25257/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> In Capacity Scheduler, DRC can treat minimum user limit percent as a max when
> custom resource is defined
>
>
> Key: YARN-10009
> URL: https://issues.apache.org/jira/browse/YARN-10009
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.3, 2.11.0
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Critical
> Attachments: YARN-10009.001.patch, YARN-10009.002.patch,
> YARN-10009.UT.patch
>
>
> | |Memory|Vcores|res_1|
> |Queue1 Totals|20GB|100|80|
> |Resources requested by App1 in Queue1|8GB (40% of total)|8 (8% of total)|80
> (100% of total)|
> In the previous use case:
> - Queue1 has a value of 25 for {{miminum-user-limit-percent}}
> - User1 has requested 8 containers with {{}}
> each
> - {{res_1}} will be the dominant resource this case.
> All 8 containers should be assigned by the capacity scheduler, but with min
> user limit pct set to 25, only 2 containers are assigned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6356) Allow different values of yarn.log-aggregation.retain-seconds for succeeded and failed jobs
[
https://issues.apache.org/jira/browse/YARN-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated YARN-6356:
-
Attachment: YARN-6356.001.patch
> Allow different values of yarn.log-aggregation.retain-seconds for succeeded
> and failed jobs
> ---
>
> Key: YARN-6356
> URL: https://issues.apache.org/jira/browse/YARN-6356
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
>Reporter: Robert Kanter
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-6356.001.patch
>
>
> It would be useful to have a value of {{yarn.log-aggregation.retain-seconds}}
> for succeeded jobs and a different value for failed/killed jobs. For jobs
> that succeeded, you typically don't care about the logs, so a shorter
> retention time is fine (and saves space/blocks in HDFS). For jobs that
> failed or were killed, the logs are much more important, and it's likely to
> want to keep them around for longer so you have time to look at them.
> For instance, you could set it to keep logs for succeeded jobs for 1 day and
> logs for failed/killed jobs for 1 week.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989008#comment-16989008 ] Adam Antal commented on YARN-5106: -- Thanks [~snemeth]! I'll upload those patches soon. > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Adam Antal >Priority: Major > Labels: newbie++ > Fix For: 3.3.0 > > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.002.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.002.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch, YARN-5106.010.patch, > YARN-5106.011.patch, YARN-5106.012.patch, YARN-5106.013.patch, > YARN-5106.014.patch, YARN-5106.015.patch, YARN-5106.016.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6356) Allow different values of yarn.log-aggregation.retain-seconds for succeeded and failed jobs
[
https://issues.apache.org/jira/browse/YARN-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989009#comment-16989009
]
Adam Antal commented on YARN-6356:
--
Uploaded patchset v1. Introduced two configuration options (for failed/killed
and succeeded (finished)) applications. I'm still a bit uncertain about the
exact mechanism (what should be default, what if failed/succeeded configs are
given, but the globals are not). Will try to cover edge cases in the next patch.
> Allow different values of yarn.log-aggregation.retain-seconds for succeeded
> and failed jobs
> ---
>
> Key: YARN-6356
> URL: https://issues.apache.org/jira/browse/YARN-6356
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
>Reporter: Robert Kanter
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-6356.001.patch
>
>
> It would be useful to have a value of {{yarn.log-aggregation.retain-seconds}}
> for succeeded jobs and a different value for failed/killed jobs. For jobs
> that succeeded, you typically don't care about the logs, so a shorter
> retention time is fine (and saves space/blocks in HDFS). For jobs that
> failed or were killed, the logs are much more important, and it's likely to
> want to keep them around for longer so you have time to look at them.
> For instance, you could set it to keep logs for succeeded jobs for 1 day and
> logs for failed/killed jobs for 1 week.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10012) Guaranteed and max capacity queue metrics for custom resources
[ https://issues.apache.org/jira/browse/YARN-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989022#comment-16989022 ] Manikandan R commented on YARN-10012: - [~jhung] Attached .001.patch. Can you please review? > Guaranteed and max capacity queue metrics for custom resources > -- > > Key: YARN-10012 > URL: https://issues.apache.org/jira/browse/YARN-10012 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Manikandan R >Priority: Major > > YARN-9085 adds support for guaranteed/maxcapacity MB/vcores. We should add > the same for custom resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10012) Guaranteed and max capacity queue metrics for custom resources
[ https://issues.apache.org/jira/browse/YARN-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-10012: Attachment: YARN-10012.001.patch > Guaranteed and max capacity queue metrics for custom resources > -- > > Key: YARN-10012 > URL: https://issues.apache.org/jira/browse/YARN-10012 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Manikandan R >Priority: Major > Attachments: YARN-10012.001.patch > > > YARN-9085 adds support for guaranteed/maxcapacity MB/vcores. We should add > the same for custom resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989030#comment-16989030 ] Adam Antal commented on YARN-5106: -- I tried backporting this to branch-3.2. There were mainly minor/trivial conflicts (like YARN-9323 or YARN-9025), but it seems that YARN-8967 has changed allocation xml's substantially. The backported patch should be overwritten dozens of places and the build should be added new fields as well. I am not sure what to do, but I suggest to skip pushing this to branch-3.2. Also, I have to admit not backporting this would also influence any FS related new patches in the future, as the definition of allocation xml's will not be compatible of the pre-3.3.0 versions. What is your opinion [~snemeth]? > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Adam Antal >Priority: Major > Labels: newbie++ > Fix For: 3.3.0 > > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.002.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.002.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch, YARN-5106.010.patch, > YARN-5106.011.patch, YARN-5106.012.patch, YARN-5106.013.patch, > YARN-5106.014.patch, YARN-5106.015.patch, YARN-5106.016.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10006) IOException used in place of YARNException in CapacitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated YARN-10006:
--
Attachment: YARN-10006.002.patch
> IOException used in place of YARNException in CapacitySheduler
> --
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-10006.001.patch, YARN-10006.002.patch
>
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10006) IOException used in place of YARNException in CapacitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989045#comment-16989045
]
Adam Antal commented on YARN-10006:
---
Fixed the test cases that my patch broke. The problem was the test was
expecting IllegalArgumentException, but now it receives {{ServiceException}}
since the {{YarnException}} was thrown in the
{{CapacityScheduler#serviceInit()}} function, and the {{CompositeService}}
catches it in its super function.
> IOException used in place of YARNException in CapacitySheduler
> --
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-10006.001.patch, YARN-10006.002.patch
>
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10012) Guaranteed and max capacity queue metrics for custom resources
[
https://issues.apache.org/jira/browse/YARN-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989055#comment-16989055
]
Hadoop QA commented on YARN-10012:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 26 new + 304 unchanged - 8 fixed = 330 total (was 312)
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m
22s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-10012 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12987630/YARN-10012.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 14b1685942d4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 520fe2c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/25260/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/25260/artifact/out/patch-
[jira] [Commented] (YARN-6356) Allow different values of yarn.log-aggregation.retain-seconds for succeeded and failed jobs
[
https://issues.apache.org/jira/browse/YARN-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989057#comment-16989057
]
Hadoop QA commented on YARN-6356:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
48s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 48s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 11s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 10 new + 236 unchanged - 4 fixed = 246 total (was 240) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
28s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
24s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 48s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 32s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-6356 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12987629/YARN-6356.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9c0dbb38187d 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 520fe2c |
| maven | version: Apache
[jira] [Commented] (YARN-9923) Introduce HealthReporter interface to support multiple health checker files
[
https://issues.apache.org/jira/browse/YARN-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989065#comment-16989065
]
Szilard Nemeth commented on YARN-9923:
--
Hi [~adam.antal], [~eyang]!
Nice discussion here, guys!
Adam, please check my comments to the latest patch:
In overall, I like your patch, good job!
Reflecting back to my comment:
https://issues.apache.org/jira/browse/YARN-9923?focusedCommentId=16970862&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16970862
{quote}
7. If NodeHealthScriptRunner#shouldRun returns false, can you log the same
message but on warn or error level? I think this is an error case, as the
script is null or empty. Isn't it?
{quote}
Maybe you could also log the script's file name on debug level, but maybe not
in this method.
*More comments:*
1. In NodeHealthCheckerService#addHealthReporter
{code:java}
LOG.debug("Omitting duplicate service.");
{code}
Please add service.getName() to the log statement as well.
2. In yarn-default.xml: Description of
yarn.nodemanager.health-checker.interval-ms has gone, completely removed. The
original description was: "Frequency of running node health script."
Moreover, there are 2 definitions of:
yarn.nodemanager.health-checker.interval-ms
Please fix this!
3. In NodeManager.md:
There's a changed line:
{code:java}
Users can specify up to 4 scripts to run individually with the
`yarn.nodemanager.health-checker.script` configuration
{code}
I think you need to refer to the new config:
yarn.nodemanager.health-checker.scripts (ending with "scripts", not "script").
4. More on this part, I don't get what this line mean:
{code:java}
|`yarn.nodemanager.health-checker.script`| String | The keywords for the health
checker scripts separated by a comma. The default is "script". |
{code}
5. There's a typo in this line:
{code:java}
| `yarn.nodemanager.health-checker.timeout-ms` | Postive integer
{code}
Please find results for "Postive" as I can see many typos with this in
NodeManager.md.
6. The following code is coming from NodeStatusUpdaterImpl:
{code:java}
@Override
public void reportException(Exception ex) {
healthChecker.getExceptionReporter().reportException(ex);
sendOutofBandHeartBeat();
}
{code}
I think NodeHealthCheckerService should have a method that receives an
exception and propagates it to the ExceptionReporter, in other words, I would
not expose ExceptionReporter to other classes, so clients of
NodeHealthCheckerService should not know about it as it is an implementation
detail for clients. The name of the method can be the same (reportException) or
anything else you can think of.
7. Nit: NodeHealthCheckerService#MAX_SCRIPTS could be private.
8. What happens if the user provides an invalid value (e.g. negative) for
health-checker interval? Following the code path of this value from
org.apache.hadoop.yarn.server.nodemanager.health.NodeHealthScriptRunner#newInstance
does not show any special handling of it.
If the negative value reaches
org.apache.hadoop.yarn.server.nodemanager.health.TimedHealthReporterService#serviceStart,
java.util.Timer#scheduleAtFixedRate(java.util.TimerTask, long, long) will be
invoked with this value, which could have an undesired effect.
Same goes with the timeout value. I can only see checks against value of 0 in
org.apache.hadoop.yarn.server.nodemanager.health.NodeHealthScriptRunner#newInstance.
9.
org.apache.hadoop.yarn.server.nodemanager.health.TimedHealthReporterService#setHealthStatus
is always receiving an empty string if the status is true (successful). If the
status is false (unsuccessful), you always have some message. I think it would
be more cleaner if you introduced a separate class for wrapping the status and
the message. It would be more explicit, since the class can reflect that it
only has a detailed message if the status is not successful. I would be happy
with the name "HealthStatus" or something similar.
10. Nit:
org.apache.hadoop.yarn.server.nodemanager.health.NodeHealthScriptRunner#shexec
could be renamed to commandExecutor.
11. Is it enough that
org.apache.hadoop.yarn.server.nodemanager.health.NodeHealthScriptRunner.NodeHealthMonitorExecutor#hasErrors
only checks for uppercase "ERROR" in the shell output? Where is this
documented that health checker scripts should print ERROR in their output?
12. Nit:
org.apache.hadoop.yarn.server.nodemanager.health.TestNodeHealthCheckerService#nodeHealthscriptFile:
This should renamed to nodeHealthScriptFile.
13. Duplicate code in:
org.apache.hadoop.yarn.server.nodemanager.health.TestNodeHealthScriptRunner#writeNodeHealthScriptFile
AND writeNodeHealthScriptFile. This code seems to be a simple one and does:
Set the file readable, writable and push some contents to it. Isn't a helper
method like this somewhere already implemented (Apache commons, lang3, YARN
helper
[jira] [Commented] (YARN-9290) Invalid SchedulingRequest not rejected in Scheduler PlacementConstraintsHandler
[
https://issues.apache.org/jira/browse/YARN-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989067#comment-16989067
]
Szilard Nemeth commented on YARN-9290:
--
Thanks [~prabhujoseph]!
> Invalid SchedulingRequest not rejected in Scheduler
> PlacementConstraintsHandler
>
>
> Key: YARN-9290
> URL: https://issues.apache.org/jira/browse/YARN-9290
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9290-001.patch, YARN-9290-002.patch,
> YARN-9290-003.patch, YARN-9290-004.patch, YARN-9290-005.patch,
> YARN-9290-006.patch, YARN-9290-007.patch, YARN-9290-008.patch,
> YARN-9290-009.patch
>
>
> SchedulingRequest with Invalid namespace is not rejected in Scheduler
> PlacementConstraintsHandler. RM keeps on trying to allocateOnNode with
> logging the exception. This is rejected in case of placement-processor
> handler.
> {code}
> 2019-02-08 16:51:27,548 WARN
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator:
> Failed to query node cardinality:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.InvalidAllocationTagsQueryException:
> Invalid namespace prefix: notselfi, valid values are:
> all,not-self,app-id,app-tag,self
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.TargetApplicationsNamespace.fromString(TargetApplicationsNamespace.java:277)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.TargetApplicationsNamespace.parse(TargetApplicationsNamespace.java:234)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.AllocationTags.createAllocationTags(AllocationTags.java:93)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfySingleConstraintExpression(PlacementConstraintsUtil.java:78)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfySingleConstraint(PlacementConstraintsUtil.java:240)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyConstraints(PlacementConstraintsUtil.java:321)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyAndConstraint(PlacementConstraintsUtil.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyConstraints(PlacementConstraintsUtil.java:324)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyConstraints(PlacementConstraintsUtil.java:365)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator.checkCardinalityAndPending(SingleConstraintAppPlacementAllocator.java:355)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator.precheckNode(SingleConstraintAppPlacementAllocator.java:395)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.precheckNode(AppSchedulingInfo.java:779)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator.preCheckForNodeCandidateSet(RegularContainerAllocator.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator.allocate(RegularContainerAllocator.java:837)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator.assignContainers(RegularContainerAllocator.java:890)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.ContainerAllocator.assignContainers(ContainerAllocator.java:54)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.assignContainers(FiCaSchedulerApp.java:977)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:1173)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:795)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1630)
> at
> org.apache.hadoop.yarn.server.resourcemanager.schedul
[jira] [Commented] (YARN-8148) Update decimal values for queue capacities shown on queue status CLI
[
https://issues.apache.org/jira/browse/YARN-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989068#comment-16989068
]
Szilard Nemeth commented on YARN-8148:
--
Hi [~prabhujoseph]!
According to [~sunilg], can you please add 3.2 / 3.1 patches so we can have a
green jenkins result and freedom to commit to the respective branches?
Thanks!
> Update decimal values for queue capacities shown on queue status CLI
>
>
> Key: YARN-8148
> URL: https://issues.apache.org/jira/browse/YARN-8148
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8148-002.patch, YARN-8148.1.patch
>
>
> Capacities are shown with two decimal values in RM UI as part of YARN-6182.
> The queue status cli are still showing one decimal value.
> {code}
> [root@bigdata3 yarn]# yarn queue -status default
> Queue Information :
> Queue Name : default
> State : RUNNING
> Capacity : 69.9%
> Current Capacity : .0%
> Maximum Capacity : 70.0%
> Default Node Label expression :
> Accessible Node Labels : *
> Preemption : enabled
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9781) SchedConfCli to get current stored scheduler configuration
[ https://issues.apache.org/jira/browse/YARN-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989090#comment-16989090 ] Szilard Nemeth commented on YARN-9781: -- Hi [~prabhujoseph]! Referring back to my [comment|https://issues.apache.org/jira/browse/YARN-9781?focusedCommentId=16963916&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16963916] Have you had a chance to file this refactor jira to eliminate the boolean-flag based approach? Anyway, latest patch LGTM, committed to trunk. Thanks for your contribution [~prabhujoseph]! > SchedConfCli to get current stored scheduler configuration > -- > > Key: YARN-9781 > URL: https://issues.apache.org/jira/browse/YARN-9781 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9781-001.patch, YARN-9781-002.patch, > YARN-9781-003.patch, YARN-9781-004.patch, YARN-9781-005.patch, > YARN-9781-006.patch, YARN-9781-007.patch, YARN-9781-008.patch > > > SchedConfCLI currently allows to add / remove / remove queue. It does not > support get configuration which RMWebServices provides as part of YARN-8559. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9780) SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single call
[
https://issues.apache.org/jira/browse/YARN-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989093#comment-16989093
]
Szilard Nemeth commented on YARN-9780:
--
Makes sense [~prabhujoseph]!
Latest patch LGTM, then.
Committed to trunk. Thanks [~prabhujoseph] for your contribution!
Thanks [~pbacsko] for your review!
> SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single
> call
>
>
> Key: YARN-9780
> URL: https://issues.apache.org/jira/browse/YARN-9780
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9780-001.patch, YARN-9780-002.patch,
> YARN-9780-003.patch, YARN-9780-004.patch
>
>
> SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single
> call. The queue has to be stopped before removing and so it is useful to
> allow both Stop and remove queue in a single call.
> *Repro:*
> {code:java}
> Capacity-Scheduler.xml:
> yarn.scheduler.capacity.root.queues = new, default, dummy
> yarn.scheduler.capacity.root.default.capacity = 60
> yarn.scheduler.capacity.root.dummy.capacity = 30
> yarn.scheduler.capacity.root.new.capacity = 10
> curl -v -X PUT -d @abc.xml -H "Content-type: application/xml"
> 'http://:8088/ws/v1/cluster/scheduler-conf'
> abc.xml
>
>
> root.default
>
>
> capacity
> 70
>
>
>
>
> root.new
>
>
> state
> STOPPED
>
>
>
> root.new
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9789) Disable Option for Write Ahead Logs of LogMutation
[ https://issues.apache.org/jira/browse/YARN-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9789: - Hadoop Flags: Reviewed > Disable Option for Write Ahead Logs of LogMutation > -- > > Key: YARN-9789 > URL: https://issues.apache.org/jira/browse/YARN-9789 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9789-001.patch, YARN-9789-002.patch > > > When yarn.scheduler.configuration.store.max-logs is set to zero, the > YARNConfigurationStore (ZK, LevelDB) reads the write ahead logs from the > backend which is not needed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9789) Disable Option for Write Ahead Logs of LogMutation
[ https://issues.apache.org/jira/browse/YARN-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989099#comment-16989099 ] Szilard Nemeth commented on YARN-9789: -- Thanks [~prabhujoseph] for this patch! LGTM, commited to trunk! Thanks [~pbacsko] for the review. > Disable Option for Write Ahead Logs of LogMutation > -- > > Key: YARN-9789 > URL: https://issues.apache.org/jira/browse/YARN-9789 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9789-001.patch, YARN-9789-002.patch > > > When yarn.scheduler.configuration.store.max-logs is set to zero, the > YARNConfigurationStore (ZK, LevelDB) reads the write ahead logs from the > backend which is not needed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9788) Queue Management API does not support parallel updates
[ https://issues.apache.org/jira/browse/YARN-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989103#comment-16989103 ] Szilard Nemeth commented on YARN-9788: -- Hi [~prabhujoseph]! As I just committed some of your patches related to this (YARN-9789, YARN-9780, YARN-9781), the latest patch does not apply anymore. Could you please rebase it? Thanks! > Queue Management API does not support parallel updates > -- > > Key: YARN-9788 > URL: https://issues.apache.org/jira/browse/YARN-9788 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9788-001.patch, YARN-9788-002.patch, > YARN-9788-003.patch, YARN-9788-004.patch, YARN-9788-005.patch, > YARN-9788-006.patch, YARN-9788-007.patch, YARN-9788-008.patch, > YARN-9788-009.patch, YARN-9788-010.patch > > > Queue Management API - does not support parallel updates. When there are two > parallel schedule conf updates (logAndApplyMutation), the first update is > overwritten by the second one. > Currently the logAndApplyMutation creates LogMutation and stores it in a > variable pendingMutation. This way at any given time there will be only one > LogMutation. And so the two parallel logAndApplyMutation will override the > pendingMutation and the later one only will be present. > The fix is to return LogMutation object by logAndApplyMutation which can be > passed during confirmMutation. This fixes the parallel updates. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9865) Capacity scheduler: add support for combined %user + %secondary_group mapping
[ https://issues.apache.org/jira/browse/YARN-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16971482#comment-16971482 ] Szilard Nemeth edited comment on YARN-9865 at 12/5/19 7:50 PM: --- Hi [~maniraj...@gmail.com]! Patch looks good, +1, committed to trunk! Some comments: 1. Can you please file a follow-up jira to fix the bloatedness of yarn.scheduler.capacity.queue-mappings in CapacityScheduler.md? There are too many values in the "value" tag. I propose to put the examples + their description as separate lines into the "description" tag instead. 2. Another follow-up jira candidate: In TestUserGroupMappingPlacementRule, verifyQueueMapping contains many parameters. For clarity, we could refactor it as a builder-style invocation, so we could see the named parameters. Within the scope of this jira, one could also refactor QueueMapping and introduce a builder for this class. was (Author: snemeth): Hi [~maniraj...@gmail.com]! Patch looks good, +1, committed to trunk! Some comments: 1. Can you please file a follow-up jira to fix the bloatedness of yarn.scheduler.capacity.queue-mappings in CapacityScheduler.md? There are too many values in the "value" tag. I propose to put the examples + their description as separate lines into the "description" tag instead. 2. Another follow-up jira candite: In TestUserGroupMappingPlacementRule, verifyQueueMapping contains many parameters. For clarity, we could refactor it as a builder-style invocation, so we could see the named parameters. Within the scope of this jira, one could also refactor QueueMapping and introduce a builder for this class. > Capacity scheduler: add support for combined %user + %secondary_group mapping > - > > Key: YARN-9865 > URL: https://issues.apache.org/jira/browse/YARN-9865 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9865-005.patch, YARN-9865.001.patch, > YARN-9865.002.patch, YARN-9865.003.patch, YARN-9865.004.patch > > > Similiar to YARN-9841, but for secondary group. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9969) Improve yarn.scheduler.capacity.queue-mappings documentation
[ https://issues.apache.org/jira/browse/YARN-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989108#comment-16989108 ] Szilard Nemeth commented on YARN-9969: -- Hi [~maniraj...@gmail.com]! Latest patch LGTM, committed to trunk! Thanks for your contribution! Thanks [~pbacsko] for your review! > Improve yarn.scheduler.capacity.queue-mappings documentation > > > Key: YARN-9969 > URL: https://issues.apache.org/jira/browse/YARN-9969 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Attachments: YARN-9969.001.patch, YARN-9969.002.patch, > YARN-9969.003.patch, YARN-9969.004.patch > > > As discussed in > https://issues.apache.org/jira/browse/YARN-9865?focusedCommentId=16971482&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16971482, > scope of this Jira is to improve the yarn.scheduler.capacity.queue-mappings > in CapacityScheduler.md. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9938) Validate Parent Queue for QueueMapping contains dynamic group as parent queue
[
https://issues.apache.org/jira/browse/YARN-9938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989120#comment-16989120
]
Szilard Nemeth commented on YARN-9938:
--
Hi [~maniraj...@gmail.com]!
Latest patch LGTM, committed to trunk, thanks for your contribution!
Thanks [~pbacsko] for your review!
> Validate Parent Queue for QueueMapping contains dynamic group as parent queue
> -
>
> Key: YARN-9938
> URL: https://issues.apache.org/jira/browse/YARN-9938
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9938.001.patch, YARN-9938.002.patch,
> YARN-9938.003.patch, YARN-9938.004.patch, YARN-9938.005.patch,
> YARN-9938.006.patch
>
>
> Currently \{{UserGroupMappingPlacementRule#validateParentQueue}} validates
> the parent queue using queue path. With dynamic group using %primary_group
> and %secondary_group in place (Refer YARN-9841 and YARN-9865) , parent queue
> validation should also happen for these above 2 queue mappings after
> resolving the above wildcard pattern to corresponding groups at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9956) Improve connection error message for YARN ApiServerClient
[
https://issues.apache.org/jira/browse/YARN-9956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989140#comment-16989140
]
Szilard Nemeth commented on YARN-9956:
--
Hi [~prabhujoseph]!
LGTM +1
[~eyang]: Do you want to take a look?
> Improve connection error message for YARN ApiServerClient
> -
>
> Key: YARN-9956
> URL: https://issues.apache.org/jira/browse/YARN-9956
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Yang
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9956-001.patch, YARN-9956-002.patch
>
>
> In HA environment, yarn.resourcemanager.webapp.address configuration is
> optional. ApiServiceClient may produce confusing error message like this:
> {code}
> 19/10/30 20:13:42 INFO client.ApiServiceClient: Fail to connect to:
> host1.example.com:8090
> 19/10/30 20:13:42 INFO client.ApiServiceClient: Fail to connect to:
> host2.example.com:8090
> 19/10/30 20:13:42 INFO util.log: Logging initialized @2301ms
> 19/10/30 20:13:42 ERROR client.ApiServiceClient: Error: {}
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - LOOKING_UP_SERVER)
> at
> java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:771)
> at
> java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:266)
> at
> java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:196)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient$1.run(ApiServiceClient.java:125)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient$1.run(ApiServiceClient.java:105)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.generateToken(ApiServiceClient.java:105)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:290)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:271)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.actionLaunch(ApiServiceClient.java:416)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:125)
> Caused by: KrbException: Server not found in Kerberos database (7) -
> LOOKING_UP_SERVER
> at
> java.security.jgss/sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:73)
> at
> java.security.jgss/sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
> at
> java.security.jgss/sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
> at
> java.security.jgss/sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
> at
> java.security.jgss/sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
> at
> java.security.jgss/sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
> at
> java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
> ... 15 more
> Caused by: KrbException: Identifier doesn't match expected value (906)
> at
> java.security.jgss/sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
> at
> java.security.jgss/sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
> at
> java.security.jgss/sun.security.krb5.internal.TGSRep.(TGSRep.java:60)
> at
> java.security.jgss/sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:55)
> ... 21 more
> 19/10/30 20:13:42 ERROR client.ApiServiceClient: Fail to launch application:
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:293)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:271)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.actionLaunch(ApiServiceClient.java:416)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:125)
> Caused by: java.lan
[jira] [Commented] (YARN-9866) u:user2:%primary_group is not working as expected
[ https://issues.apache.org/jira/browse/YARN-9866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989150#comment-16989150 ] Szilard Nemeth commented on YARN-9866: -- Hi [~maniraj...@gmail.com]! Patch does not apply anymore to trunk. Can you please rebase? > u:user2:%primary_group is not working as expected > - > > Key: YARN-9866 > URL: https://issues.apache.org/jira/browse/YARN-9866 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Attachments: YARN-9866.001.patch, YARN-9866.002.patch, > YARN-9866.003.patch > > > Please refer #1 in > https://issues.apache.org/jira/browse/YARN-9841?focusedCommentId=16937024&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16937024 > for more details -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9969) Improve yarn.scheduler.capacity.queue-mappings documentation
[ https://issues.apache.org/jira/browse/YARN-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989155#comment-16989155 ] Hudson commented on YARN-9969: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17727 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17727/]) YARN-9969. Improve yarn.scheduler.capacity.queue-mappings documentation. (snemeth: rev 0ab9c0398d8270f053dd0b1cee5dbb00346a7a2a) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md > Improve yarn.scheduler.capacity.queue-mappings documentation > > > Key: YARN-9969 > URL: https://issues.apache.org/jira/browse/YARN-9969 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9969.001.patch, YARN-9969.002.patch, > YARN-9969.003.patch, YARN-9969.004.patch > > > As discussed in > https://issues.apache.org/jira/browse/YARN-9865?focusedCommentId=16971482&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16971482, > scope of this Jira is to improve the yarn.scheduler.capacity.queue-mappings > in CapacityScheduler.md. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9780) SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single call
[
https://issues.apache.org/jira/browse/YARN-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989153#comment-16989153
]
Hudson commented on YARN-9780:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17727 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17727/])
YARN-9780. SchedulerConf Mutation API does not Allow Stop and Remove (snemeth:
rev 4627dd670800b2191ebdd41c6971ccd53e718e18)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesConfigurationMutation.java
> SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single
> call
>
>
> Key: YARN-9780
> URL: https://issues.apache.org/jira/browse/YARN-9780
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9780-001.patch, YARN-9780-002.patch,
> YARN-9780-003.patch, YARN-9780-004.patch
>
>
> SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single
> call. The queue has to be stopped before removing and so it is useful to
> allow both Stop and remove queue in a single call.
> *Repro:*
> {code:java}
> Capacity-Scheduler.xml:
> yarn.scheduler.capacity.root.queues = new, default, dummy
> yarn.scheduler.capacity.root.default.capacity = 60
> yarn.scheduler.capacity.root.dummy.capacity = 30
> yarn.scheduler.capacity.root.new.capacity = 10
> curl -v -X PUT -d @abc.xml -H "Content-type: application/xml"
> 'http://:8088/ws/v1/cluster/scheduler-conf'
> abc.xml
>
>
> root.default
>
>
> capacity
> 70
>
>
>
>
> root.new
>
>
> state
> STOPPED
>
>
>
> root.new
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9781) SchedConfCli to get current stored scheduler configuration
[ https://issues.apache.org/jira/browse/YARN-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989152#comment-16989152 ] Hudson commented on YARN-9781: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17727 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17727/]) YARN-9781. SchedConfCli to get current stored scheduler configuration. (snemeth: rev c71befaf8fec1e5a962359c9cc131fb47adfd9d1) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/dao/ConfInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/SchedConfCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestSchedConfCLI.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ConfInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java > SchedConfCli to get current stored scheduler configuration > -- > > Key: YARN-9781 > URL: https://issues.apache.org/jira/browse/YARN-9781 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9781-001.patch, YARN-9781-002.patch, > YARN-9781-003.patch, YARN-9781-004.patch, YARN-9781-005.patch, > YARN-9781-006.patch, YARN-9781-007.patch, YARN-9781-008.patch > > > SchedConfCLI currently allows to add / remove / remove queue. It does not > support get configuration which RMWebServices provides as part of YARN-8559. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9789) Disable Option for Write Ahead Logs of LogMutation
[ https://issues.apache.org/jira/browse/YARN-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989154#comment-16989154 ] Hudson commented on YARN-9789: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17727 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17727/]) YARN-9789. Disable Option for Write Ahead Logs of LogMutation. (snemeth: rev 5cc6f945da648a48441b00aca89f3cc1a9777143) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestLeveldbConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java > Disable Option for Write Ahead Logs of LogMutation > -- > > Key: YARN-9789 > URL: https://issues.apache.org/jira/browse/YARN-9789 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9789-001.patch, YARN-9789-002.patch > > > When yarn.scheduler.configuration.store.max-logs is set to zero, the > YARNConfigurationStore (ZK, LevelDB) reads the write ahead logs from the > backend which is not needed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9938) Validate Parent Queue for QueueMapping contains dynamic group as parent queue
[
https://issues.apache.org/jira/browse/YARN-9938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989156#comment-16989156
]
Hudson commented on YARN-9938:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17727 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17727/])
YARN-9938. Validate Parent Queue for QueueMapping contains dynamic group
(snemeth: rev cf68857631fe12d3e3cabb53ab3029beab9457b7)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerQueueMappingFactory.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
> Validate Parent Queue for QueueMapping contains dynamic group as parent queue
> -
>
> Key: YARN-9938
> URL: https://issues.apache.org/jira/browse/YARN-9938
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9938.001.patch, YARN-9938.002.patch,
> YARN-9938.003.patch, YARN-9938.004.patch, YARN-9938.005.patch,
> YARN-9938.006.patch
>
>
> Currently \{{UserGroupMappingPlacementRule#validateParentQueue}} validates
> the parent queue using queue path. With dynamic group using %primary_group
> and %secondary_group in place (Refer YARN-9841 and YARN-9865) , parent queue
> validation should also happen for these above 2 queue mappings after
> resolving the above wildcard pattern to corresponding groups at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10006) IOException used in place of YARNException in CapacitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989165#comment-16989165
]
Hadoop QA commented on YARN-10006:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 414 unchanged - 2 fixed = 415 total (was 416)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m
58s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-10006 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12987632/YARN-10006.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a2a6bd808691 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 520fe2c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/25261/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25261/testReport/ |
| Max. process+thread count | 860 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-se
[jira] [Comment Edited] (YARN-9607) Auto-configuring rollover-size of IFile format for non-appendable filesystems
[ https://issues.apache.org/jira/browse/YARN-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989168#comment-16989168 ] Szilard Nemeth edited comment on YARN-9607 at 12/5/19 8:51 PM: --- Hi [~adam.antal]! Patch LGTM, committed to trunk. Thanks [~adam.antal] for this contribution. I can see that the target version is 3.2. Do we need backports to branch-3.2, then? Quick question: Do you want to file a follow-up jira to fix your raised concern about the hardcoded local file system appendable status (raised in this [comment of yours|https://issues.apache.org/jira/browse/YARN-9607?focusedCommentId=16973161&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16973161] was (Author: snemeth): Hi [~adam.antal]! Patch LGTM, committed to trunk. Thanks [~adam.antal] for this contribution. Quick question: Do you want to file a follow-up jira to fix your raised concern about the hardcoded local file system appendable status (raised in this [comment of yours|https://issues.apache.org/jira/browse/YARN-9607?focusedCommentId=16973161&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16973161] > Auto-configuring rollover-size of IFile format for non-appendable filesystems > - > > Key: YARN-9607 > URL: https://issues.apache.org/jira/browse/YARN-9607 > Project: Hadoop YARN > Issue Type: Sub-task > Components: log-aggregation, yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9607.001.patch, YARN-9607.002.patch, > YARN-9607.003.patch, YARN-9607.004.patch > > > In YARN-9525, we made IFile format compatible with remote folders with s3a > scheme. In rolling fashioned log-aggregation IFile still fails with the > "append is not supported" error message, which is a known limitation of the > format by design. > There is a workaround though: setting the rollover size in the configuration > of the IFile format, in each rolling cycle a new aggregated log file will be > created, thus we eliminated the append from the process. Setting this config > globally would cause performance problems in the regular log-aggregation, so > I'm suggesting to enforcing this config to zero, if the scheme of the URI is > s3a (or any other non-appendable filesystem). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9607) Auto-configuring rollover-size of IFile format for non-appendable filesystems
[ https://issues.apache.org/jira/browse/YARN-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989168#comment-16989168 ] Szilard Nemeth commented on YARN-9607: -- Hi [~adam.antal]! Patch LGTM, committed to trunk. Thanks [~adam.antal] for this contribution. Quick question: Do you want to file a follow-up jira to fix your raised concern about the hardcoded local file system appendable status (raised in this [comment of yours|https://issues.apache.org/jira/browse/YARN-9607?focusedCommentId=16973161&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16973161] > Auto-configuring rollover-size of IFile format for non-appendable filesystems > - > > Key: YARN-9607 > URL: https://issues.apache.org/jira/browse/YARN-9607 > Project: Hadoop YARN > Issue Type: Sub-task > Components: log-aggregation, yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9607.001.patch, YARN-9607.002.patch, > YARN-9607.003.patch, YARN-9607.004.patch > > > In YARN-9525, we made IFile format compatible with remote folders with s3a > scheme. In rolling fashioned log-aggregation IFile still fails with the > "append is not supported" error message, which is a known limitation of the > format by design. > There is a workaround though: setting the rollover size in the configuration > of the IFile format, in each rolling cycle a new aggregated log file will be > created, thus we eliminated the append from the process. Setting this config > globally would cause performance problems in the regular log-aggregation, so > I'm suggesting to enforcing this config to zero, if the scheme of the URI is > s3a (or any other non-appendable filesystem). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9607) Auto-configuring rollover-size of IFile format for non-appendable filesystems
[ https://issues.apache.org/jira/browse/YARN-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989168#comment-16989168 ] Szilard Nemeth edited comment on YARN-9607 at 12/5/19 8:52 PM: --- Hi [~adam.antal]! Patch LGTM, committed to trunk. Thanks [~adam.antal] for this contribution. I can see that the target version is 3.2. Do we need a backport patch to branch-3.2, then? Quick question: Do you want to file a follow-up jira to fix your raised concern about the hardcoded local file system appendable status (raised in this [comment of yours|https://issues.apache.org/jira/browse/YARN-9607?focusedCommentId=16973161&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16973161] was (Author: snemeth): Hi [~adam.antal]! Patch LGTM, committed to trunk. Thanks [~adam.antal] for this contribution. I can see that the target version is 3.2. Do we need backports to branch-3.2, then? Quick question: Do you want to file a follow-up jira to fix your raised concern about the hardcoded local file system appendable status (raised in this [comment of yours|https://issues.apache.org/jira/browse/YARN-9607?focusedCommentId=16973161&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16973161] > Auto-configuring rollover-size of IFile format for non-appendable filesystems > - > > Key: YARN-9607 > URL: https://issues.apache.org/jira/browse/YARN-9607 > Project: Hadoop YARN > Issue Type: Sub-task > Components: log-aggregation, yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9607.001.patch, YARN-9607.002.patch, > YARN-9607.003.patch, YARN-9607.004.patch > > > In YARN-9525, we made IFile format compatible with remote folders with s3a > scheme. In rolling fashioned log-aggregation IFile still fails with the > "append is not supported" error message, which is a known limitation of the > format by design. > There is a workaround though: setting the rollover size in the configuration > of the IFile format, in each rolling cycle a new aggregated log file will be > created, thus we eliminated the append from the process. Setting this config > globally would cause performance problems in the regular log-aggregation, so > I'm suggesting to enforcing this config to zero, if the scheme of the URI is > s3a (or any other non-appendable filesystem). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10006) IOException used in place of YARNException in CapacitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989177#comment-16989177
]
Szilard Nemeth commented on YARN-10006:
---
Hi [~adam.antal]!
LGTM for latest patch.
[~prabhujoseph]: I think this is ready to commit. Thanks!
> IOException used in place of YARNException in CapacitySheduler
> --
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-10006.001.patch, YARN-10006.002.patch
>
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4029) Update LogAggregationStatus to store on finish
[
https://issues.apache.org/jira/browse/YARN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989193#comment-16989193
]
Hadoop QA commented on YARN-4029:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 21s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 334 unchanged - 0 fixed = 335 total (was 334) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 51s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}152m 17s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}238m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| | hadoop.yarn.server.resourcemanager.TestApplicationACLs |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
| | hadoop.yarn.server.
[jira] [Commented] (YARN-9607) Auto-configuring rollover-size of IFile format for non-appendable filesystems
[ https://issues.apache.org/jira/browse/YARN-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989194#comment-16989194 ] Hudson commented on YARN-9607: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17728 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17728/]) YARN-9607. Auto-configuring rollover-size of IFile format for (snemeth: rev 4f758dd4c682bacbb110c51a96079a6c5d103c95) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/LogAggregationIndexedFileController.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/TestLogAggregationIndexedFileController.java > Auto-configuring rollover-size of IFile format for non-appendable filesystems > - > > Key: YARN-9607 > URL: https://issues.apache.org/jira/browse/YARN-9607 > Project: Hadoop YARN > Issue Type: Sub-task > Components: log-aggregation, yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9607.001.patch, YARN-9607.002.patch, > YARN-9607.003.patch, YARN-9607.004.patch > > > In YARN-9525, we made IFile format compatible with remote folders with s3a > scheme. In rolling fashioned log-aggregation IFile still fails with the > "append is not supported" error message, which is a known limitation of the > format by design. > There is a workaround though: setting the rollover size in the configuration > of the IFile format, in each rolling cycle a new aggregated log file will be > created, thus we eliminated the append from the process. Setting this config > globally would cause performance problems in the regular log-aggregation, so > I'm suggesting to enforcing this config to zero, if the scheme of the URI is > s3a (or any other non-appendable filesystem). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10012) Guaranteed and max capacity queue metrics for custom resources
[
https://issues.apache.org/jira/browse/YARN-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989202#comment-16989202
]
Jonathan Hung commented on YARN-10012:
--
Thanks [~maniraj...@gmail.com]. A few comments:
* We should move the additions in QueueMetricsForCustomResources to a separate
class, perhaps CSQueueMetricsForCustomResources (which inherits from
QueueMetricsForCustomResources). We'll have to change this bit in the
QueueMetrics constructor:
{noformat}
if (ResourceUtils.getNumberOfKnownResourceTypes() > 2) {
this.queueMetricsForCustomResources =
new QueueMetricsForCustomResources();
registerCustomResources();
} {noformat}
Perhaps we can just create a {{createQueueMetricsForCustomResources}} method in
QueueMetrics which returns {{new QueueMetricsForCustomResources();}}, and have
CSQueueMetrics override this metric to return {{new
CSQueueMetricsForCustomResources();}}
* Let's rename QueueMetrics#getQueueMetricsForCustomResource to
QueueMetrics#getQueueMetricsForCustomResources
* Should we rename MAXIMUM_CAPACITY_METRIC_PREFIX to MaxCapacity instead of
MaximumCapacity? To match the existing MaxCapacityMB metric.
* For the test case:
{noformat}
RMNode n1 = MockNodes.newNodeInfo(0,
MockNodes.newResource(50 * GB, 50,
ImmutableMap. builder()
.put(TestQueueMetricsForCustomResources.CUSTOM_RES_1,
String.valueOf(10 * GB))
.build()),
1, "n1");
RMNode n2 = MockNodes.newNodeInfo(0,
MockNodes.newResource(50 * GB, 50,
ImmutableMap. builder()
.put(TestQueueMetricsForCustomResources.CUSTOM_RES_1,
String.valueOf(20 * GB))
.build()),
2, "n2");
{noformat}
> Guaranteed and max capacity queue metrics for custom resources
> --
>
> Key: YARN-10012
> URL: https://issues.apache.org/jira/browse/YARN-10012
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-10012.001.patch
>
>
> YARN-9085 adds support for guaranteed/maxcapacity MB/vcores. We should add
> the same for custom resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10012) Guaranteed and max capacity queue metrics for custom resources
[
https://issues.apache.org/jira/browse/YARN-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989202#comment-16989202
]
Jonathan Hung edited comment on YARN-10012 at 12/5/19 10:11 PM:
Thanks [~maniraj...@gmail.com]. A few comments:
* We should move the additions in QueueMetricsForCustomResources to a separate
class, perhaps CSQueueMetricsForCustomResources (which inherits from
QueueMetricsForCustomResources). We'll have to change this bit in the
QueueMetrics constructor:
{noformat}
if (ResourceUtils.getNumberOfKnownResourceTypes() > 2) {
this.queueMetricsForCustomResources =
new QueueMetricsForCustomResources();
registerCustomResources();
} {noformat}
Perhaps we can just create a {{createQueueMetricsForCustomResources}} method in
QueueMetrics which returns {{new QueueMetricsForCustomResources();}}, and have
CSQueueMetrics override this metric to return {{new
CSQueueMetricsForCustomResources();}}
* Let's rename QueueMetrics#getQueueMetricsForCustomResource to
QueueMetrics#getQueueMetricsForCustomResources
* Should we rename MAXIMUM_CAPACITY_METRIC_PREFIX to MaxCapacity instead of
MaximumCapacity? To match the existing MaxCapacityMB metric.
* For the test case:
{noformat}
RMNode n1 = MockNodes.newNodeInfo(0,
MockNodes.newResource(50 * GB, 50,
ImmutableMap. builder()
.put(TestQueueMetricsForCustomResources.CUSTOM_RES_1,
String.valueOf(10 * GB))
.build()),
1, "n1");
RMNode n2 = MockNodes.newNodeInfo(0,
MockNodes.newResource(50 * GB, 50,
ImmutableMap. builder()
.put(TestQueueMetricsForCustomResources.CUSTOM_RES_1,
String.valueOf(20 * GB))
.build()),
2, "n2");{noformat}
any reason the custom resource uses GB as a multiplier? It's a bit confusing.
* Not sure I understand this assert.
{noformat}
Map guaranteedCap_A11 =
((CSQueueMetrics) cs.getQueue("a1").getMetrics())
.getQueueMetricsForCustomResource().getGuaranteedCapacity();
assertEquals(0 * GB, guaranteedCap_A11
.get(TestQueueMetricsForCustomResources.CUSTOM_RES_1).longValue());
{noformat}
I guess it should be (10*GB + 20*GB)* 0.105 (A capacity) * 0.30 (A1 capacity).
But since we're setting min allocation for this custom resource to 1024, it's
rounding to 0. I think for all of the asserts which check for CUSTOM_RES_1
metrics, we should remove the GB multiplier (related to the above comment).
Perhaps we can set the node capability for the custom resource to something
like 1000 (for n1) and 2000 (for n2), and set the min/max allocations for the
custom resource to something much smaller - min can be 1 (so that we get exact
numbers for the guaranteed/max-capacity metrics), max can be anything > 1 I
think.
* An extra {{assertEquals(10240,
((CSQueueMetrics)cs.getQueue("a").getMetrics()).getGuaranteedMB());}} line was
added in the test case, we don't need this
was (Author: jhung):
Thanks [~maniraj...@gmail.com]. A few comments:
* We should move the additions in QueueMetricsForCustomResources to a separate
class, perhaps CSQueueMetricsForCustomResources (which inherits from
QueueMetricsForCustomResources). We'll have to change this bit in the
QueueMetrics constructor:
{noformat}
if (ResourceUtils.getNumberOfKnownResourceTypes() > 2) {
this.queueMetricsForCustomResources =
new QueueMetricsForCustomResources();
registerCustomResources();
} {noformat}
Perhaps we can just create a {{createQueueMetricsForCustomResources}} method in
QueueMetrics which returns {{new QueueMetricsForCustomResources();}}, and have
CSQueueMetrics override this metric to return {{new
CSQueueMetricsForCustomResources();}}
* Let's rename QueueMetrics#getQueueMetricsForCustomResource to
QueueMetrics#getQueueMetricsForCustomResources
* Should we rename MAXIMUM_CAPACITY_METRIC_PREFIX to MaxCapacity instead of
MaximumCapacity? To match the existing MaxCapacityMB metric.
* For the test case:
{noformat}
RMNode n1 = MockNodes.newNodeInfo(0,
MockNodes.newResource(50 * GB, 50,
ImmutableMap. builder()
.put(TestQueueMetricsForCustomResources.CUSTOM_RES_1,
String.valueOf(10 * GB))
.build()),
1, "n1");
RMNode n2 = MockNodes.newNodeInfo(0,
MockNodes.newResource(50 * GB, 50,
ImmutableMap. builder()
.put(TestQueueMetricsForCustomResources.CUSTOM_RES_1,
String.valueOf(20 * GB))
.build()),
2, "n2");
{noformat}
> Guaranteed and max capacity queue metrics for custom resources
> --
>
> Key: YARN-10012
> URL: https://issues.apache.org/jira/browse/YARN-10012
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Manikandan R
>
[jira] [Comment Edited] (YARN-10012) Guaranteed and max capacity queue metrics for custom resources
[
https://issues.apache.org/jira/browse/YARN-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989202#comment-16989202
]
Jonathan Hung edited comment on YARN-10012 at 12/5/19 10:14 PM:
Thanks [~maniraj...@gmail.com]. A few comments:
* We should move the additions in QueueMetricsForCustomResources to a separate
class, perhaps CSQueueMetricsForCustomResources (which inherits from
QueueMetricsForCustomResources). We'll have to change this bit in the
QueueMetrics constructor:
{noformat}
if (ResourceUtils.getNumberOfKnownResourceTypes() > 2) {
this.queueMetricsForCustomResources =
new QueueMetricsForCustomResources();
registerCustomResources();
} {noformat}
Perhaps we can just create a {{createQueueMetricsForCustomResources}} method in
QueueMetrics which returns {{new QueueMetricsForCustomResources();}}, and have
CSQueueMetrics override this metric to return {{new
CSQueueMetricsForCustomResources();}}
* Let's rename QueueMetrics#getQueueMetricsForCustomResource to
QueueMetrics#getQueueMetricsForCustomResources
* Should we rename MAXIMUM_CAPACITY_METRIC_PREFIX to MaxCapacity instead of
MaximumCapacity? To match the existing MaxCapacityMB metric.
* For the test case:
{noformat}
RMNode n1 = MockNodes.newNodeInfo(0,
MockNodes.newResource(50 * GB, 50,
ImmutableMap. builder()
.put(TestQueueMetricsForCustomResources.CUSTOM_RES_1,
String.valueOf(10 * GB))
.build()),
1, "n1");
RMNode n2 = MockNodes.newNodeInfo(0,
MockNodes.newResource(50 * GB, 50,
ImmutableMap. builder()
.put(TestQueueMetricsForCustomResources.CUSTOM_RES_1,
String.valueOf(20 * GB))
.build()),
2, "n2");{noformat}
any reason the custom resource uses GB as a multiplier? It's a bit confusing.
* Not sure I understand this assert.
{noformat}
Map guaranteedCap_A11 =
((CSQueueMetrics) cs.getQueue("a1").getMetrics())
.getQueueMetricsForCustomResource().getGuaranteedCapacity();
assertEquals(0 * GB, guaranteedCap_A11
.get(TestQueueMetricsForCustomResources.CUSTOM_RES_1).longValue());
{noformat}
I guess it should be (10*GB + 20*GB)* 0.105 (A capacity) * 0.30 (A1 capacity).
But since we're setting min allocation for this custom resource to 1024, it's
rounding to 0. I think for all of the asserts which check for CUSTOM_RES_1
metrics, we should remove the GB multiplier (related to the above comment).
Perhaps we can set the node capability for the custom resource to something
like 1000 (for n1) and 2000 (for n2), and set the min/max allocations for the
custom resource to something much smaller - min can be 1 (so that we get exact
numbers for the guaranteed/max-capacity metrics), max can be anything > 1 I
think.
* An extra {{assertEquals(10240,
((CSQueueMetrics)cs.getQueue("a").getMetrics()).getGuaranteedMB());}} line was
added in the test case, we don't need this
* There's a compilation error, seems FSQueueMetrics has a method
{{getCustomResources}} which conflicts with the one added in QueueMetrics.
* There's a few checkstyle errors, let's fix as many as possible.
was (Author: jhung):
Thanks [~maniraj...@gmail.com]. A few comments:
* We should move the additions in QueueMetricsForCustomResources to a separate
class, perhaps CSQueueMetricsForCustomResources (which inherits from
QueueMetricsForCustomResources). We'll have to change this bit in the
QueueMetrics constructor:
{noformat}
if (ResourceUtils.getNumberOfKnownResourceTypes() > 2) {
this.queueMetricsForCustomResources =
new QueueMetricsForCustomResources();
registerCustomResources();
} {noformat}
Perhaps we can just create a {{createQueueMetricsForCustomResources}} method in
QueueMetrics which returns {{new QueueMetricsForCustomResources();}}, and have
CSQueueMetrics override this metric to return {{new
CSQueueMetricsForCustomResources();}}
* Let's rename QueueMetrics#getQueueMetricsForCustomResource to
QueueMetrics#getQueueMetricsForCustomResources
* Should we rename MAXIMUM_CAPACITY_METRIC_PREFIX to MaxCapacity instead of
MaximumCapacity? To match the existing MaxCapacityMB metric.
* For the test case:
{noformat}
RMNode n1 = MockNodes.newNodeInfo(0,
MockNodes.newResource(50 * GB, 50,
ImmutableMap. builder()
.put(TestQueueMetricsForCustomResources.CUSTOM_RES_1,
String.valueOf(10 * GB))
.build()),
1, "n1");
RMNode n2 = MockNodes.newNodeInfo(0,
MockNodes.newResource(50 * GB, 50,
ImmutableMap. builder()
.put(TestQueueMetricsForCustomResources.CUSTOM_RES_1,
String.valueOf(20 * GB))
.build()),
2, "n2");{noformat}
any reason the custom resource uses GB as a multiplier? It's a bit confusing.
* Not sure I understand this assert.
{noformat}
Map guaranteedCap_A11 =
((CSQueueMetrics) cs.getQueu
[jira] [Commented] (YARN-9956) Improve connection error message for YARN ApiServerClient
[
https://issues.apache.org/jira/browse/YARN-9956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989378#comment-16989378
]
Eric Yang commented on YARN-9956:
-
Thank you for the patch, [~prabhujoseph]. Overall, the patch looks good.
The failed unit test looks a bit concerning. I am not sure how it is related
to the changes. Can you confirm this is not an issue?
> Improve connection error message for YARN ApiServerClient
> -
>
> Key: YARN-9956
> URL: https://issues.apache.org/jira/browse/YARN-9956
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Yang
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9956-001.patch, YARN-9956-002.patch
>
>
> In HA environment, yarn.resourcemanager.webapp.address configuration is
> optional. ApiServiceClient may produce confusing error message like this:
> {code}
> 19/10/30 20:13:42 INFO client.ApiServiceClient: Fail to connect to:
> host1.example.com:8090
> 19/10/30 20:13:42 INFO client.ApiServiceClient: Fail to connect to:
> host2.example.com:8090
> 19/10/30 20:13:42 INFO util.log: Logging initialized @2301ms
> 19/10/30 20:13:42 ERROR client.ApiServiceClient: Error: {}
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - LOOKING_UP_SERVER)
> at
> java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:771)
> at
> java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:266)
> at
> java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:196)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient$1.run(ApiServiceClient.java:125)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient$1.run(ApiServiceClient.java:105)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.generateToken(ApiServiceClient.java:105)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:290)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:271)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.actionLaunch(ApiServiceClient.java:416)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:125)
> Caused by: KrbException: Server not found in Kerberos database (7) -
> LOOKING_UP_SERVER
> at
> java.security.jgss/sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:73)
> at
> java.security.jgss/sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
> at
> java.security.jgss/sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
> at
> java.security.jgss/sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
> at
> java.security.jgss/sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
> at
> java.security.jgss/sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
> at
> java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
> ... 15 more
> Caused by: KrbException: Identifier doesn't match expected value (906)
> at
> java.security.jgss/sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
> at
> java.security.jgss/sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
> at
> java.security.jgss/sun.security.krb5.internal.TGSRep.(TGSRep.java:60)
> at
> java.security.jgss/sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:55)
> ... 21 more
> 19/10/30 20:13:42 ERROR client.ApiServiceClient: Fail to launch application:
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:293)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:271)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.actionLaunch(ApiServiceClient.java:416)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(T
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989383#comment-16989383 ] Eric Yang commented on YARN-9561: - +1 for patch 15. Thank you [~ebadger]. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch, > YARN-9561.015.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9956) Improve connection error message for YARN ApiServerClient
[
https://issues.apache.org/jira/browse/YARN-9956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989426#comment-16989426
]
Prabhu Joseph commented on YARN-9956:
-
[~eyang] Testcase failure is not related, have fixed it as part of YARN-9990.
Will resubmit the patch to make sure the testcase runs fine now.
> Improve connection error message for YARN ApiServerClient
> -
>
> Key: YARN-9956
> URL: https://issues.apache.org/jira/browse/YARN-9956
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Yang
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9956-001.patch, YARN-9956-002.patch
>
>
> In HA environment, yarn.resourcemanager.webapp.address configuration is
> optional. ApiServiceClient may produce confusing error message like this:
> {code}
> 19/10/30 20:13:42 INFO client.ApiServiceClient: Fail to connect to:
> host1.example.com:8090
> 19/10/30 20:13:42 INFO client.ApiServiceClient: Fail to connect to:
> host2.example.com:8090
> 19/10/30 20:13:42 INFO util.log: Logging initialized @2301ms
> 19/10/30 20:13:42 ERROR client.ApiServiceClient: Error: {}
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - LOOKING_UP_SERVER)
> at
> java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:771)
> at
> java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:266)
> at
> java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:196)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient$1.run(ApiServiceClient.java:125)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient$1.run(ApiServiceClient.java:105)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.generateToken(ApiServiceClient.java:105)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:290)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:271)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.actionLaunch(ApiServiceClient.java:416)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:125)
> Caused by: KrbException: Server not found in Kerberos database (7) -
> LOOKING_UP_SERVER
> at
> java.security.jgss/sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:73)
> at
> java.security.jgss/sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
> at
> java.security.jgss/sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
> at
> java.security.jgss/sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
> at
> java.security.jgss/sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
> at
> java.security.jgss/sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
> at
> java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
> ... 15 more
> Caused by: KrbException: Identifier doesn't match expected value (906)
> at
> java.security.jgss/sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
> at
> java.security.jgss/sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
> at
> java.security.jgss/sun.security.krb5.internal.TGSRep.(TGSRep.java:60)
> at
> java.security.jgss/sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:55)
> ... 21 more
> 19/10/30 20:13:42 ERROR client.ApiServiceClient: Fail to launch application:
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:293)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:271)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.actionLaunch(ApiServiceClient.java:416)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.cl
[jira] [Commented] (YARN-9780) SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single call
[
https://issues.apache.org/jira/browse/YARN-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989429#comment-16989429
]
Prabhu Joseph commented on YARN-9780:
-
Thanks [~snemeth].
> SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single
> call
>
>
> Key: YARN-9780
> URL: https://issues.apache.org/jira/browse/YARN-9780
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9780-001.patch, YARN-9780-002.patch,
> YARN-9780-003.patch, YARN-9780-004.patch
>
>
> SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single
> call. The queue has to be stopped before removing and so it is useful to
> allow both Stop and remove queue in a single call.
> *Repro:*
> {code:java}
> Capacity-Scheduler.xml:
> yarn.scheduler.capacity.root.queues = new, default, dummy
> yarn.scheduler.capacity.root.default.capacity = 60
> yarn.scheduler.capacity.root.dummy.capacity = 30
> yarn.scheduler.capacity.root.new.capacity = 10
> curl -v -X PUT -d @abc.xml -H "Content-type: application/xml"
> 'http://:8088/ws/v1/cluster/scheduler-conf'
> abc.xml
>
>
> root.default
>
>
> capacity
> 70
>
>
>
>
> root.new
>
>
> state
> STOPPED
>
>
>
> root.new
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9789) Disable Option for Write Ahead Logs of LogMutation
[ https://issues.apache.org/jira/browse/YARN-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989432#comment-16989432 ] Prabhu Joseph commented on YARN-9789: - Thanks [~snemeth]. > Disable Option for Write Ahead Logs of LogMutation > -- > > Key: YARN-9789 > URL: https://issues.apache.org/jira/browse/YARN-9789 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9789-001.patch, YARN-9789-002.patch > > > When yarn.scheduler.configuration.store.max-logs is set to zero, the > YARNConfigurationStore (ZK, LevelDB) reads the write ahead logs from the > backend which is not needed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10014) Refactor boolean flag based approach in SchedConfCLI#run
Prabhu Joseph created YARN-10014:
Summary: Refactor boolean flag based approach in SchedConfCLI#run
Key: YARN-10014
URL: https://issues.apache.org/jira/browse/YARN-10014
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Prabhu Joseph
Boolean-flag based approach in
org.apache.hadoop.yarn.client.cli.SchedConfCLI#run:
Everything is controlled with boolean flags here.
The flag hasOption is set to true in each of the if-clauses just to make the
condition below the hasOption-conditions happy. The flag is set to true even
for parameter that don't have an option (like 'getConf') at all, this is very
misleading and hard to understand for the first read.
Need below refactoring:
a. Eliminates the hasOption boolean flag
b. Where an option is misused, fail-fast: Have a method that contains this code
and call it for every option, in-place:
{code}
if (!hasOption) {
System.err.println("Invalid Command Usage: ");
printUsage();
return -1;
}
{code}
c. Remove the boolean flags: format and getConf as well. These are unnecessary.
cc [~snemeth]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9781) SchedConfCli to get current stored scheduler configuration
[ https://issues.apache.org/jira/browse/YARN-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989439#comment-16989439 ] Prabhu Joseph commented on YARN-9781: - Thanks [~snemeth]. Have reported YARN-10014 to fix the same. > SchedConfCli to get current stored scheduler configuration > -- > > Key: YARN-9781 > URL: https://issues.apache.org/jira/browse/YARN-9781 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9781-001.patch, YARN-9781-002.patch, > YARN-9781-003.patch, YARN-9781-004.patch, YARN-9781-005.patch, > YARN-9781-006.patch, YARN-9781-007.patch, YARN-9781-008.patch > > > SchedConfCLI currently allows to add / remove / remove queue. It does not > support get configuration which RMWebServices provides as part of YARN-8559. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9956) Improve connection error message for YARN ApiServerClient
[
https://issues.apache.org/jira/browse/YARN-9956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9956:
Attachment: YARN-9956-003.patch
> Improve connection error message for YARN ApiServerClient
> -
>
> Key: YARN-9956
> URL: https://issues.apache.org/jira/browse/YARN-9956
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Yang
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9956-001.patch, YARN-9956-002.patch,
> YARN-9956-003.patch
>
>
> In HA environment, yarn.resourcemanager.webapp.address configuration is
> optional. ApiServiceClient may produce confusing error message like this:
> {code}
> 19/10/30 20:13:42 INFO client.ApiServiceClient: Fail to connect to:
> host1.example.com:8090
> 19/10/30 20:13:42 INFO client.ApiServiceClient: Fail to connect to:
> host2.example.com:8090
> 19/10/30 20:13:42 INFO util.log: Logging initialized @2301ms
> 19/10/30 20:13:42 ERROR client.ApiServiceClient: Error: {}
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - LOOKING_UP_SERVER)
> at
> java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:771)
> at
> java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:266)
> at
> java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:196)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient$1.run(ApiServiceClient.java:125)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient$1.run(ApiServiceClient.java:105)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.generateToken(ApiServiceClient.java:105)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:290)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:271)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.actionLaunch(ApiServiceClient.java:416)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:125)
> Caused by: KrbException: Server not found in Kerberos database (7) -
> LOOKING_UP_SERVER
> at
> java.security.jgss/sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:73)
> at
> java.security.jgss/sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
> at
> java.security.jgss/sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
> at
> java.security.jgss/sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
> at
> java.security.jgss/sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
> at
> java.security.jgss/sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
> at
> java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
> ... 15 more
> Caused by: KrbException: Identifier doesn't match expected value (906)
> at
> java.security.jgss/sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
> at
> java.security.jgss/sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
> at
> java.security.jgss/sun.security.krb5.internal.TGSRep.(TGSRep.java:60)
> at
> java.security.jgss/sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:55)
> ... 21 more
> 19/10/30 20:13:42 ERROR client.ApiServiceClient: Fail to launch application:
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:293)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.getApiClient(ApiServiceClient.java:271)
> at
> org.apache.hadoop.yarn.service.client.ApiServiceClient.actionLaunch(ApiServiceClient.java:416)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:125)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.had
[jira] [Commented] (YARN-9462) TestResourceTrackerService.testNodeRemovalGracefully fails sporadically
[
https://issues.apache.org/jira/browse/YARN-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989446#comment-16989446
]
Prabhu Joseph commented on YARN-9462:
-
[~snemeth] Issue still persists. Dec 3 Trunk Build had this failure -
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1339/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
> TestResourceTrackerService.testNodeRemovalGracefully fails sporadically
> ---
>
> Key: YARN-9462
> URL: https://issues.apache.org/jira/browse/YARN-9462
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, test
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments:
> TestResourceTrackerService.testNodeRemovalGracefully.txt, YARN-9462-001.patch
>
>
> TestResourceTrackerService.testNodeRemovalGracefully fails sporadically
> {code}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService.testNodeRemovalUtilDecomToUntracked(TestResourceTrackerService.java:2318)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService.testNodeRemovalUtil(TestResourceTrackerService.java:2280)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService.testNodeRemovalGracefully(TestResourceTrackerService.java:2133)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9788) Queue Management API does not support parallel updates
[ https://issues.apache.org/jira/browse/YARN-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9788: Attachment: YARN-9788-011.patch > Queue Management API does not support parallel updates > -- > > Key: YARN-9788 > URL: https://issues.apache.org/jira/browse/YARN-9788 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9788-001.patch, YARN-9788-002.patch, > YARN-9788-003.patch, YARN-9788-004.patch, YARN-9788-005.patch, > YARN-9788-006.patch, YARN-9788-007.patch, YARN-9788-008.patch, > YARN-9788-009.patch, YARN-9788-010.patch, YARN-9788-011.patch > > > Queue Management API - does not support parallel updates. When there are two > parallel schedule conf updates (logAndApplyMutation), the first update is > overwritten by the second one. > Currently the logAndApplyMutation creates LogMutation and stores it in a > variable pendingMutation. This way at any given time there will be only one > LogMutation. And so the two parallel logAndApplyMutation will override the > pendingMutation and the later one only will be present. > The fix is to return LogMutation object by logAndApplyMutation which can be > passed during confirmMutation. This fixes the parallel updates. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9788) Queue Management API does not support parallel updates
[ https://issues.apache.org/jira/browse/YARN-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989462#comment-16989462 ] Prabhu Joseph commented on YARN-9788: - Yes Sure [~snemeth], Have attached [^YARN-9788-011.patch] after rebasing. > Queue Management API does not support parallel updates > -- > > Key: YARN-9788 > URL: https://issues.apache.org/jira/browse/YARN-9788 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9788-001.patch, YARN-9788-002.patch, > YARN-9788-003.patch, YARN-9788-004.patch, YARN-9788-005.patch, > YARN-9788-006.patch, YARN-9788-007.patch, YARN-9788-008.patch, > YARN-9788-009.patch, YARN-9788-010.patch, YARN-9788-011.patch > > > Queue Management API - does not support parallel updates. When there are two > parallel schedule conf updates (logAndApplyMutation), the first update is > overwritten by the second one. > Currently the logAndApplyMutation creates LogMutation and stores it in a > variable pendingMutation. This way at any given time there will be only one > LogMutation. And so the two parallel logAndApplyMutation will override the > pendingMutation and the later one only will be present. > The fix is to return LogMutation object by logAndApplyMutation which can be > passed during confirmMutation. This fixes the parallel updates. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10006) IOException used in place of YARNException in CapacitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-10006:
-
Fix Version/s: 3.3.0
> IOException used in place of YARNException in CapacitySheduler
> --
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-10006.001.patch, YARN-10006.002.patch
>
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10006) IOException used in place of YARNException in CapacitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989471#comment-16989471
]
Prabhu Joseph commented on YARN-10006:
--
Thanks [~adam.antal] for the patch, [~snemeth] for the review.
The patch looks good. Have committed to trunk. Will resolve the Jira.
> IOException used in place of YARNException in CapacitySheduler
> --
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-10006.001.patch, YARN-10006.002.patch
>
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8148) Update decimal values for queue capacities shown on queue status CLI
[
https://issues.apache.org/jira/browse/YARN-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-8148:
Attachment: YARN-8148-branch-3.2.001.patch
> Update decimal values for queue capacities shown on queue status CLI
>
>
> Key: YARN-8148
> URL: https://issues.apache.org/jira/browse/YARN-8148
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8148-002.patch, YARN-8148-branch-3.2.001.patch,
> YARN-8148.1.patch
>
>
> Capacities are shown with two decimal values in RM UI as part of YARN-6182.
> The queue status cli are still showing one decimal value.
> {code}
> [root@bigdata3 yarn]# yarn queue -status default
> Queue Information :
> Queue Name : default
> State : RUNNING
> Capacity : 69.9%
> Current Capacity : .0%
> Maximum Capacity : 70.0%
> Default Node Label expression :
> Accessible Node Labels : *
> Preemption : enabled
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8148) Update decimal values for queue capacities shown on queue status CLI
[
https://issues.apache.org/jira/browse/YARN-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-8148:
Attachment: YARN-8148-branch-3.1.001.patch
> Update decimal values for queue capacities shown on queue status CLI
>
>
> Key: YARN-8148
> URL: https://issues.apache.org/jira/browse/YARN-8148
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8148-002.patch, YARN-8148-branch-3.1.001.patch,
> YARN-8148-branch-3.2.001.patch, YARN-8148.1.patch
>
>
> Capacities are shown with two decimal values in RM UI as part of YARN-6182.
> The queue status cli are still showing one decimal value.
> {code}
> [root@bigdata3 yarn]# yarn queue -status default
> Queue Information :
> Queue Name : default
> State : RUNNING
> Capacity : 69.9%
> Current Capacity : .0%
> Maximum Capacity : 70.0%
> Default Node Label expression :
> Accessible Node Labels : *
> Preemption : enabled
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8148) Update decimal values for queue capacities shown on queue status CLI
[
https://issues.apache.org/jira/browse/YARN-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989476#comment-16989476
]
Prabhu Joseph commented on YARN-8148:
-
Yes [~snemeth], have attached branch-3.2 and branch-3.1 patches.
> Update decimal values for queue capacities shown on queue status CLI
>
>
> Key: YARN-8148
> URL: https://issues.apache.org/jira/browse/YARN-8148
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8148-002.patch, YARN-8148-branch-3.1.001.patch,
> YARN-8148-branch-3.2.001.patch, YARN-8148.1.patch
>
>
> Capacities are shown with two decimal values in RM UI as part of YARN-6182.
> The queue status cli are still showing one decimal value.
> {code}
> [root@bigdata3 yarn]# yarn queue -status default
> Queue Information :
> Queue Name : default
> State : RUNNING
> Capacity : 69.9%
> Current Capacity : .0%
> Maximum Capacity : 70.0%
> Default Node Label expression :
> Accessible Node Labels : *
> Preemption : enabled
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10006) IOException used in place of YARNException in CapacitySheduler
[
https://issues.apache.org/jira/browse/YARN-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989477#comment-16989477
]
Hudson commented on YARN-10006:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17731 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17731/])
YARN-10006. IOException used in place of YARNException in (pjoseph: rev
e575df5ca6f9630199b5c1696136184e88af3160)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
> IOException used in place of YARNException in CapacitySheduler
> --
>
> Key: YARN-10006
> URL: https://issues.apache.org/jira/browse/YARN-10006
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Adam Antal
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-10006.001.patch, YARN-10006.002.patch
>
>
> IOException used in place of YARNException in CapacityScheduler. As per
> YARNException Doc,
> {code:java}
> /**
> * YarnException indicates exceptions from yarn servers. On the other hand,
> * IOExceptions indicates exceptions from RPC layer.
> */
> {code}
> Below methods throws IOException but it is suppose to throw YarnException.
> CapacitySchedulerQueueManager#parseQueue <- initializeQueues <-
> CapacityScheduler#initializeQueues <- initScheduler <- serviceInit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
