[jira] [Commented] (YARN-11133) YarnClient gets the wrong EffectiveMinCapacity value
[
https://issues.apache.org/jira/browse/YARN-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537643#comment-17537643
]
Hemanth Boyina commented on YARN-11133:
---
committed to trunk, thanks for the contribution [~zilong zhu]
> YarnClient gets the wrong EffectiveMinCapacity value
>
>
> Key: YARN-11133
> URL: https://issues.apache.org/jira/browse/YARN-11133
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 3.2.3, 3.3.2
>Reporter: Zilong Zhu
>Assignee: Zilong Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> It calls the QueueConfigurations#getEffectiveMinCapacity to get the wrong
> value when I use the YarnClient. I found some bugs with
> QueueConfigurationsPBImpl#mergeLocalToBuilder.
> {code:java}
> private void mergeLocalToBuilder() {
> if (this.effMinResource != null) {
> builder
> .setEffectiveMinCapacity(convertToProtoFormat(this.effMinResource));
> }
> if (this.effMaxResource != null) {
> builder
> .setEffectiveMaxCapacity(convertToProtoFormat(this.effMaxResource));
> }
> if (this.configuredMinResource != null) {
> builder.setEffectiveMinCapacity(
> convertToProtoFormat(this.configuredMinResource));
> }
> if (this.configuredMaxResource != null) {
> builder.setEffectiveMaxCapacity(
> convertToProtoFormat(this.configuredMaxResource));
> }
> } {code}
> configuredMinResource was incorrectly assigned to effMinResource. This causes
> the real effMinResource to be overwritten and configuredMinResource is null.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11133) YarnClient gets the wrong EffectiveMinCapacity value
[
https://issues.apache.org/jira/browse/YARN-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Boyina resolved YARN-11133.
---
Fix Version/s: 3.4.0
Resolution: Fixed
> YarnClient gets the wrong EffectiveMinCapacity value
>
>
> Key: YARN-11133
> URL: https://issues.apache.org/jira/browse/YARN-11133
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 3.2.3, 3.3.2
>Reporter: Zilong Zhu
>Assignee: Zilong Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> It calls the QueueConfigurations#getEffectiveMinCapacity to get the wrong
> value when I use the YarnClient. I found some bugs with
> QueueConfigurationsPBImpl#mergeLocalToBuilder.
> {code:java}
> private void mergeLocalToBuilder() {
> if (this.effMinResource != null) {
> builder
> .setEffectiveMinCapacity(convertToProtoFormat(this.effMinResource));
> }
> if (this.effMaxResource != null) {
> builder
> .setEffectiveMaxCapacity(convertToProtoFormat(this.effMaxResource));
> }
> if (this.configuredMinResource != null) {
> builder.setEffectiveMinCapacity(
> convertToProtoFormat(this.configuredMinResource));
> }
> if (this.configuredMaxResource != null) {
> builder.setEffectiveMaxCapacity(
> convertToProtoFormat(this.configuredMaxResource));
> }
> } {code}
> configuredMinResource was incorrectly assigned to effMinResource. This causes
> the real effMinResource to be overwritten and configuredMinResource is null.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4283) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/YARN-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17390952#comment-17390952 ] Hemanth Boyina commented on YARN-4283: -- ran the test cases locally, test cases are passing locally, failures are not related > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: YARN-4283 > URL: https://issues.apache.org/jira/browse/YARN-4283 > Project: Hadoop YARN > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic >Priority: Major > Labels: ipv6 > Attachments: YARN-4283-HADOOP-11890.1.patch, > YARN-4283-HADOOP-17800.001.patch, YARN-4283-HADOOP-17800.002.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > hadoop-yarn part of HADOOP-12122 task -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4283) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/YARN-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina updated YARN-4283: - Attachment: YARN-4283-HADOOP-17800.002.patch > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: YARN-4283 > URL: https://issues.apache.org/jira/browse/YARN-4283 > Project: Hadoop YARN > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic >Priority: Major > Labels: ipv6 > Attachments: YARN-4283-HADOOP-11890.1.patch, > YARN-4283-HADOOP-17800.001.patch, YARN-4283-HADOOP-17800.002.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > hadoop-yarn part of HADOOP-12122 task -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-4283) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/YARN-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina reopened YARN-4283: -- > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: YARN-4283 > URL: https://issues.apache.org/jira/browse/YARN-4283 > Project: Hadoop YARN > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic >Priority: Major > Labels: ipv6 > Attachments: YARN-4283-HADOOP-11890.1.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > hadoop-yarn part of HADOOP-12122 task -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4283) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/YARN-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17390001#comment-17390001 ] Hemanth Boyina commented on YARN-4283: -- reopened and uploaded patch against trunk as the current patch has conflicts , please see here https://issues.apache.org/jira/browse/HADOOP-11890?focusedCommentId=17379845=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17379845 for more details > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: YARN-4283 > URL: https://issues.apache.org/jira/browse/YARN-4283 > Project: Hadoop YARN > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic >Priority: Major > Labels: ipv6 > Attachments: YARN-4283-HADOOP-11890.1.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > hadoop-yarn part of HADOOP-12122 task -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10442) RM should make sure node label file highly available
[
https://issues.apache.org/jira/browse/YARN-10442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223811#comment-17223811
]
Hemanth Boyina commented on YARN-10442:
---
thanks [~surendrasingh] for the contribution
committed to trunk
> RM should make sure node label file highly available
>
>
> Key: YARN-10442
> URL: https://issues.apache.org/jira/browse/YARN-10442
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Fix For: 3.4.0
>
>
> In one of my cluster RM failed transition to Active because node label file
> blocks are missing. I think RM should make sure important files are highly
> available .
> {noformat}
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Could not
> obtain block: BP-2121803626-10.0.0.22-1597301807397:blk_1073832522_91774
> file=/yarn/node-labels/nodelabel.mirror
> at
> com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:238)
> at
> com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
> at
> com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
> at
> com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
> at
> org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos$AddToClusterNodeLabelsRequestProto.parseDelimitedFrom(YarnServerResourceManagerServiceProtos.java:7493)
> at
> org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.loadFromMirror(FileSystemNodeLabelsStore.java:168)
> at
> org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.recover(FileSystemNodeLabelsStore.java:205)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:254)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:268)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)(AbstractService.java:194){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10437) Destroy yarn service if any YarnException occurs during submitApp
[ https://issues.apache.org/jira/browse/YARN-10437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195998#comment-17195998 ] Hemanth Boyina commented on YARN-10437: --- thanks for the report and patch [~dmmkr] , is it possible to extend a UT for this > Destroy yarn service if any YarnException occurs during submitApp > - > > Key: YARN-10437 > URL: https://issues.apache.org/jira/browse/YARN-10437 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Minor > Attachments: YARN-10437.001.patch > > > If a user submits a yarn service with configuration such that it causes an > exception during application submission, the files related to the service are > not cleared from hdfs automatically. Ideally the files stored to hdfs cannot > be used in future to start or stop the service as the configuration itself is > invalid. So, we should destroy the service and remove the residual files in > hdfs, if any YarnException is thrown. > For example if the user submits a service with configuring with "memory" more > than the maximum resource, the service fails but the files in hdfs are not > cleared. But these files should be cleared. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10336) RM page should throw exception when command injected in RM REST API to get applications
[ https://issues.apache.org/jira/browse/YARN-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177241#comment-17177241 ] Hemanth Boyina commented on YARN-10336: --- committed to trunk and branch 3.3 , thanks [~BilwaST] for the contribution , thanks [~Rajshree] for the report and thanks [~elgoiri] for the review > RM page should throw exception when command injected in RM REST API to get > applications > --- > > Key: YARN-10336 > URL: https://issues.apache.org/jira/browse/YARN-10336 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rajshree Mishra >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: CommandInject.jpg, RM_UI.jpg, YARN-10336.001.patch, > YARN-10336.002.patch, YARN-10336.003.patch, testproof.png > > > Using a web application attacking, we see that injecting commands like > ACCEPTED, FAILED and FINISHED to RM REST API does not throw an exception. > Refer images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10336) RM page should throw exception when command injected in RM REST API to get applications
[ https://issues.apache.org/jira/browse/YARN-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177233#comment-17177233 ] Hemanth Boyina commented on YARN-10336: --- +1 , will commit shortly > RM page should throw exception when command injected in RM REST API to get > applications > --- > > Key: YARN-10336 > URL: https://issues.apache.org/jira/browse/YARN-10336 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rajshree Mishra >Assignee: Bilwa S T >Priority: Major > Attachments: CommandInject.jpg, RM_UI.jpg, YARN-10336.001.patch, > YARN-10336.002.patch, YARN-10336.003.patch, testproof.png > > > Using a web application attacking, we see that injecting commands like > ACCEPTED, FAILED and FINISHED to RM REST API does not throw an exception. > Refer images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10336) RM page should throw exception when command injected in RM REST API to get applications
[ https://issues.apache.org/jira/browse/YARN-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169498#comment-17169498 ] Hemanth Boyina commented on YARN-10336: --- thanks for the report [~Rajshree] , thanks for the patch [~BilwaST] is [ ] character was injected in the rest api request ? > RM page should throw exception when command injected in RM REST API to get > applications > --- > > Key: YARN-10336 > URL: https://issues.apache.org/jira/browse/YARN-10336 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rajshree Mishra >Assignee: Bilwa S T >Priority: Major > Attachments: CommandInject.jpg, RM_UI.jpg, YARN-10336.001.patch, > testproof.png > > > Using a web application attacking, we see that injecting commands like > ACCEPTED, FAILED and FINISHED to RM REST API does not throw an exception. > Refer images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
