[jira] [Created] (YARN-9927) RM multi-thread event processing mechanism
hcarrot created YARN-9927: - Summary: RM multi-thread event processing mechanism Key: YARN-9927 URL: https://issues.apache.org/jira/browse/YARN-9927 Project: Hadoop YARN Issue Type: Improvement Components: yarn Reporter: hcarrot Attachments: RM multi-thread event processing mechanism.pdf Recently, we have observed serious event blocking in RM event dispatcher queue. After analysis of RM event monitoring data and RM event processing logic, we found that the proportion of RMNodeStatusEvent is less than other events, but the overall processing time of it is more than other events. Meanwhile, RM event processing is in a single-thread mode, and It results in the decrease of RM's performance. So we proposed a RM multi-thread event processing mechanism to improve RM performance. Is this mechanism feasible? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9926) RM multi-thread event processing mechanism
hcarrot created YARN-9926: - Summary: RM multi-thread event processing mechanism Key: YARN-9926 URL: https://issues.apache.org/jira/browse/YARN-9926 Project: Hadoop YARN Issue Type: Improvement Components: yarn Reporter: hcarrot Attachments: RM multi-thread event processing mechanism.pdf Recently, we have observed serious event blocking in RM event dispatcher queue. After analysis of RM event monitoring data and RM event processing logic, we found that the proportion of RMNodeStatusEvent is less than other events, but the overall processing time of it is more than other events. Meanwhile, RM event processing is in a single-thread mode, and It results in the decrease of RM's performance. So we proposed a RM multi-thread event processing mechanism to improve RM performance. Is this mechanism feasible? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956730#comment-16956730 ] Kinga Marton commented on YARN-9886: [~wangda] yes. I will add whitelist, where it can be defined who can use this feature. > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map this applications to the real submitting user's > queue instead of the Hive one. > For this cases if they would pass the username in the application tag we may > read it and use that one during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9875) FSSchedulerConfigurationStore fails to update with hdfs path
[
https://issues.apache.org/jira/browse/YARN-9875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956718#comment-16956718
]
Prabhu Joseph commented on YARN-9875:
-
Thanks [~eyang].
> FSSchedulerConfigurationStore fails to update with hdfs path
>
>
> Key: YARN-9875
> URL: https://issues.apache.org/jira/browse/YARN-9875
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9875-001.patch, YARN-9875-002.patch
>
>
> FSSchedulerConfigurationStore fails to update with hdfs path -
> "java.io.IOException: Filesystem closed"
> *RM Logs:*
> {code}
> 2019-10-06 16:50:40,829 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.FSSchedulerConfigurationStore:
> write temp capacity configuration fail,
> schedulerConfigFile=hdfs:/tmp/yarn/system/capacity-scheduler.xml.1570380640828.tmp
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:475)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1232)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1214)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1196)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1134)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:530)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:527)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:541)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:468)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1136)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1116)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1005)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:993)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.FSSchedulerConfigurationStore.writeTmpConfig(FSSchedulerConfigurationStore.java:251)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.FSSchedulerConfigurationStore.logMutation(FSSchedulerConfigurationStore.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.MutableCSConfigurationProvider.logAndApplyMutation(MutableCSConfigurationProvider.java:153)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2597)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2587)
> {code}
> *Repro:*
> {code:java}
> yarn-site.xml:
>
> yarn.scheduler.configuration.fs.path
> hdfs:///tmp/yarn/system
>
>
> yarn.scheduler.configuration.store.class
> fs
>
> [yarn@yarndocker-1 yarn]$ cat /tmp/abc.xml
>
>
> root.default
>
>
> priority
> 10
>
>
>
>
> [yarn@yarndocker-1 yarn]$ curl -v -X PUT -d @/tmp/abc.xml -H "Content-type:
> application/xml"
> 'http://yarndocker-1:8088/ws/v1/cluster/scheduler-conf?user.name=yarn'
> Filesystem closed
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9915) Fix FindBug issue in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956715#comment-16956715
]
Prabhu Joseph commented on YARN-9915:
-
Thanks [~epayne].
> Fix FindBug issue in QueueMetrics
> -
>
> Key: YARN-9915
> URL: https://issues.apache.org/jira/browse/YARN-9915
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0, 3.2.2, 3.1.4
>
> Attachments: YARN-9915-01.patch
>
>
> Below FindBug issue appears in the trunk build
> {code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> invokes inefficient new Long(long) constructor; use Long.valueOf(long)
> instead
> Bug type DM_NUMBER_CTOR (click for details)
> In class org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics
> In method
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> Called method new Long(long)
> Should call Long.valueOf(long) instead
> At QueueMetrics.java:[line 468]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9887) Capacity scheduler: add support for limiting maxRunningApps per user
[
https://issues.apache.org/jira/browse/YARN-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956688#comment-16956688
]
Peter Bacsko edited comment on YARN-9887 at 10/22/19 5:21 AM:
--
[~wangda] that's correct, but are you saying that we should not spend time on
this and abandon it?
Just asking because I was under the impression that eventually we'll address
all feature gap between FS and CS. This might not be the most important one
(that is indeed YARN-9879), but nevertheless CS doesn't support it. You think
we should abandon implementing this? Or just lower the priority?
Note that it's not necessarily for Phase 2, it could be done later. What is
your position on this?
was (Author: pbacsko):
[~wangda] that's correct, but are you saying that we should not spend time on
this and abandon it?
Just asking because I was under the impression that eventually we'll address
all feature gap between FS and CS. This might not be the most important one
(that is indeed YARN-9879), but nevertheless CS doesn't support it. You think
we should abandon implementing of this? Or just lower the priority?
Note that it's not necessarily for Phase 2, it could be done later. What is
your position on this?
> Capacity scheduler: add support for limiting maxRunningApps per user
>
>
> Key: YARN-9887
> URL: https://issues.apache.org/jira/browse/YARN-9887
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Priority: Major
>
> Fair Scheduler supports limiting the number of applications that a particular
> user can submit:
> {noformat}
>
> 10
>
> {noformat}
> Capacity Scheduler does not have an exact equivalent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9887) Capacity scheduler: add support for limiting maxRunningApps per user
[
https://issues.apache.org/jira/browse/YARN-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956688#comment-16956688
]
Peter Bacsko commented on YARN-9887:
[~wangda] that's correct, but are you saying that we should not spend time on
this and abandon it?
Just asking because I was under the impression that eventually we'll address
all feature gap between FS and CS. This might not be the most important one
(that is indeed YARN-9879), but nevertheless CS doesn't support it. You think
we should abandon implementing of this? Or just lower the priority?
Note that it's not necessarily for Phase 2, it could be done later. What is
your position on this?
> Capacity scheduler: add support for limiting maxRunningApps per user
>
>
> Key: YARN-9887
> URL: https://issues.apache.org/jira/browse/YARN-9887
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Priority: Major
>
> Fair Scheduler supports limiting the number of applications that a particular
> user can submit:
> {noformat}
>
> 10
>
> {noformat}
> Capacity Scheduler does not have an exact equivalent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated YARN-9897: --- Attachment: hadoop_build.log > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > Attachments: hadoop_build.log > > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956643#comment-16956643 ] Zhenyu Zheng commented on YARN-9897: [~eyang]Hi, the log is in the attachment field. > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > Attachments: hadoop_build.log > > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9897) Add an Aarch64 CI for YARN
[
https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956640#comment-16956640
]
liusheng commented on YARN-9897:
Hi [~eyang],
Yes, we have built Hadoop with *-Pnative* flag, without the
container-executor.additional_cflags, seems everything is OK, here is my
building command:
{code:java}
mvn clean install -e -B -Pdist,native -Dtar -DskipTests -Dmaven.javadoc.skip
{code}
and attach my building log
> Add an Aarch64 CI for YARN
> --
>
> Key: YARN-9897
> URL: https://issues.apache.org/jira/browse/YARN-9897
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: build, test
>Reporter: Zhenyu Zheng
>Priority: Major
>
> As YARN is the resource manager of Hadoop and there are large number of other
> software that also uses YARN for resource management. The capability of
> running YARN on platforms with different architecture and managing hardware
> resources with different architecture could be very important and useful.
> Aarch64(ARM) architecture is currently the dominate architecture in small
> devices like phone, IOT devices, security cameras, drones etc. With the
> increasing compuiting capability and the increasing connection speed like 5G
> network, there could be greate posibility and opportunity for world chaging
> inovations and new market if we can managing and make use of those devices as
> well.
> Currently, all YARN CIs are based on x86 architecture and we have been
> performing tests on Aarch64 and proposing possible solutions for problems we
> have meet, like:
> https://issues.apache.org/jira/browse/HADOOP-16614
> we have done all YARN tests and it turns out there are only a few problems,
> and we can provide possible solutions for discussion.
> We want to propose to add an Aarch64 CI for YARN to promote the support for
> YARN on Aarch64 platforms. We are willing to provide machines to the current
> CI system and manpower to mananging the CI and fxing problems that occours.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9511) [JDK11] TestAuxServices#testRemoteAuxServiceClassPath YarnRuntimeException: The remote jarfile should not be writable by group or others. The current Permission is 436
[
https://issues.apache.org/jira/browse/YARN-9511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956634#comment-16956634
]
liusheng commented on YARN-9511:
Hi [~adam.antal],
Here are my steps:
1. Confirm current *umask* value of system, mine is *0002*
{code:java}
$ umask
0002{code}
2. Run unit tests of *TestAuxServices*, and will reproduce the problems
described in this issue.
{code:java}
$ cd hadoop-yarn-project/
$ mvn test -Dtest=TestAuxServices{code}
3. modify the permissions of hadoop source code directory to remove other and
group wirte permission and the system's umask to *022*
{code:java}
$ chmod go-w hadoop/ -R
$ umask 022
$ umask
0022{code}
4. Re-run the unitests of *TestAuxServices* again, all the unit tests can pass
{code:java}
$ cd hadoop-yarn-project/
$ mvn test -Dtest=TestAuxServices{code}
> [JDK11] TestAuxServices#testRemoteAuxServiceClassPath YarnRuntimeException:
> The remote jarfile should not be writable by group or others. The current
> Permission is 436
> ---
>
> Key: YARN-9511
> URL: https://issues.apache.org/jira/browse/YARN-9511
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Szilard Nemeth
>Priority: Major
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8.
> {code}
> [ERROR]
> testRemoteAuxServiceClassPath(org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices)
> Time elapsed: 0.551 s <<<
> ERROR!org.apache.hadoop.yarn.exceptions.YarnRuntimeException: The remote
> jarfile should not be writable by group or others. The current Permission is
> 436
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:202)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices.testRemoteAuxServiceClassPath(TestAuxServices.java:268)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9562) Add Java changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956476#comment-16956476 ] Eric Badger commented on YARN-9562: --- Patch 010 should fix the findbugs. > Add Java changes for the new RuncContainerRuntime > - > > Key: YARN-9562 > URL: https://issues.apache.org/jira/browse/YARN-9562 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9562.001.patch, YARN-9562.002.patch, > YARN-9562.003.patch, YARN-9562.004.patch, YARN-9562.005.patch, > YARN-9562.006.patch, YARN-9562.007.patch, YARN-9562.008.patch, > YARN-9562.009.patch > > > This JIRA will be used to add the Java changes for the new > RuncContainerRuntime. This will work off of YARN-9560 to use much of the > existing DockerLinuxContainerRuntime code once it is moved up into an > abstract class that can be extended. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9915) Fix FindBug issue in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956442#comment-16956442
]
Hudson commented on YARN-9915:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17558 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17558/])
YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu (ericp: rev
83d148074f9299de02d5c896a3ed4e11292cba73)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
> Fix FindBug issue in QueueMetrics
> -
>
> Key: YARN-9915
> URL: https://issues.apache.org/jira/browse/YARN-9915
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9915-01.patch
>
>
> Below FindBug issue appears in the trunk build
> {code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> invokes inefficient new Long(long) constructor; use Long.valueOf(long)
> instead
> Bug type DM_NUMBER_CTOR (click for details)
> In class org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics
> In method
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> Called method new Long(long)
> Should call Long.valueOf(long) instead
> At QueueMetrics.java:[line 468]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9922) Fix JavaDoc errors introduced by YARN-9699
[
https://issues.apache.org/jira/browse/YARN-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956426#comment-16956426
]
Hudson commented on YARN-9922:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17557 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17557/])
YARN-9922. Fixed YARN javadoc errors from YARN-9699.(eyang: rev
3f7756dc6cd541918eec2b221891864d29b174d3)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/PreconditionException.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/package-info.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/UnsupportedPropertyException.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverterParams.java
> Fix JavaDoc errors introduced by YARN-9699
> --
>
> Key: YARN-9922
> URL: https://issues.apache.org/jira/browse/YARN-9922
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9922-001.patch
>
>
> We accidentally introduced some Javadoc errors in YARN-9699:
> {code:java}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java:89:
> error: bad use of '>'
> [ERROR]* Used during FS->CS conversion. When enabled, background threads
> are
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/package-info.java:20:
> error: bad use of '>'
> [ERROR] * This package contains classes related to the Fair Scheduler ->
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java:30:
> error: bad use of '>'
> [ERROR] * Parses arguments passed to the FS->CS converter.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverterParams.java:20:
> error: bad use of '>'
> [ERROR] * POJO that holds values for the FS->CS converter.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java:38:
> error: bad use of '>'
> [ERROR] * Class that determines what should happen if the FS->CS converter
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/PreconditionException.java:23:
> error: bad use of '>'
> [ERROR] * before FS->CS conversion.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config [Phase 1]
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956427#comment-16956427 ] Hudson commented on YARN-9699: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17557 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17557/]) YARN-9922. Fixed YARN javadoc errors from YARN-9699.(eyang: rev 3f7756dc6cd541918eec2b221891864d29b174d3) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/PreconditionException.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/UnsupportedPropertyException.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverterParams.java > Migration tool that help to generate CS config based on FS config [Phase 1] > > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.3.0 > > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699-012.patch, > YARN-9699-013.patch, YARN-9699-014.patch, YARN-9699-015.patch, > YARN-9699-016.patch, YARN-9699-017.patch, YARN-9699.001.patch, > YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9915) Fix FindBug issue in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956419#comment-16956419
]
Eric Payne commented on YARN-9915:
--
Will also backport to 3.2 and 3.1.
> Fix FindBug issue in QueueMetrics
> -
>
> Key: YARN-9915
> URL: https://issues.apache.org/jira/browse/YARN-9915
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9915-01.patch
>
>
> Below FindBug issue appears in the trunk build
> {code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> invokes inefficient new Long(long) constructor; use Long.valueOf(long)
> instead
> Bug type DM_NUMBER_CTOR (click for details)
> In class org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics
> In method
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> Called method new Long(long)
> Should call Long.valueOf(long) instead
> At QueueMetrics.java:[line 468]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9915) Fix FindBug issue in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956415#comment-16956415
]
Eric Payne commented on YARN-9915:
--
Thanks for the clarificaion [~Prabhu Joseph].
+1
Will commit soon.
> Fix FindBug issue in QueueMetrics
> -
>
> Key: YARN-9915
> URL: https://issues.apache.org/jira/browse/YARN-9915
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9915-01.patch
>
>
> Below FindBug issue appears in the trunk build
> {code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> invokes inefficient new Long(long) constructor; use Long.valueOf(long)
> instead
> Bug type DM_NUMBER_CTOR (click for details)
> In class org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics
> In method
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> Called method new Long(long)
> Should call Long.valueOf(long) instead
> At QueueMetrics.java:[line 468]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9922) Fix JavaDoc errors introduced by YARN-9699
[
https://issues.apache.org/jira/browse/YARN-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956411#comment-16956411
]
Eric Yang commented on YARN-9922:
-
[~pbacsko] Thank you for the patch.
+1 for Patch 001.
> Fix JavaDoc errors introduced by YARN-9699
> --
>
> Key: YARN-9922
> URL: https://issues.apache.org/jira/browse/YARN-9922
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9922-001.patch
>
>
> We accidentally introduced some Javadoc errors in YARN-9699:
> {code:java}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java:89:
> error: bad use of '>'
> [ERROR]* Used during FS->CS conversion. When enabled, background threads
> are
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/package-info.java:20:
> error: bad use of '>'
> [ERROR] * This package contains classes related to the Fair Scheduler ->
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java:30:
> error: bad use of '>'
> [ERROR] * Parses arguments passed to the FS->CS converter.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverterParams.java:20:
> error: bad use of '>'
> [ERROR] * POJO that holds values for the FS->CS converter.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java:38:
> error: bad use of '>'
> [ERROR] * Class that determines what should happen if the FS->CS converter
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/PreconditionException.java:23:
> error: bad use of '>'
> [ERROR] * before FS->CS conversion.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/UnsupportedPropertyException.java:20:
> error: bad use of '>'
> [ERROR] * Thrown by the FS->CS converter if it encounters an
> [ERROR] ^
> {code}
>
> These should be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9915) Fix FindBug issue in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956399#comment-16956399
]
Prabhu Joseph commented on YARN-9915:
-
Thanks [~epayne] for reviewing. Yes, the above report is confusing but Findbugs
ran twice
*one on trunk before patch: trunk Compile Tests -* where warnings are shown
-1 findbugs 1m 27s
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings.
*And on trunk with patch: Patch Compile Tests -* the warnings are not shown and
+1 is given.
+1 findbugs 1m 23s
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
*From the console output: On trunk without patch:*
{code:java}
findbugs detection: trunk
cd
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
/usr/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build@2/yetus-m2/hadoop-trunk-patch-0
-Ptest-patch -DskipTests test-compile findbugs:findbugs -DskipTests=true >
/testptch/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
2>&1
Elapsed: 1m 25s
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings.
{code}
*On trunk after patch:*
{code:java}
findbugs detection: patch
cd
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
/usr/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build@2/yetus-m2/hadoop-trunk-patch-0
-Ptest-patch -DskipTests test-compile findbugs:findbugs -DskipTests=true >
/testptch/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
2>&1
Elapsed: 1m 12s
Starting with
/testptch/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.xml
Merging
/testptch/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.xml
Writing
/testptch/patchprocess/combined-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.xml
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
{code}
> Fix FindBug issue in QueueMetrics
> -
>
> Key: YARN-9915
> URL: https://issues.apache.org/jira/browse/YARN-9915
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9915-01.patch
>
>
> Below FindBug issue appears in the trunk build
> {code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> invokes inefficient new Long(long) constructor; use Long.valueOf(long)
> instead
> Bug type DM_NUMBER_CTOR (click for details)
> In class org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics
> In method
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> Called method new Long(long)
> Should call Long.valueOf(long) instead
> At QueueMetrics.java:[line 468]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9915) Fix FindBug issue in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956391#comment-16956391
]
Eric Payne commented on YARN-9915:
--
The changes look correct. The confusing thing is the the [findbugs from the
above pre-commit
build|https://builds.apache.org/job/PreCommit-YARN-Build/25017/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html]
lists the same warning in the exact same place:
{panel}
Bug type DM_NUMBER_CTOR (click for details)
In class org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics
In method
org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
Called method new Long(long)
Should call Long.valueOf(long) instead
At QueueMetrics.java:[line 468]
{panel}
> Fix FindBug issue in QueueMetrics
> -
>
> Key: YARN-9915
> URL: https://issues.apache.org/jira/browse/YARN-9915
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9915-01.patch
>
>
> Below FindBug issue appears in the trunk build
> {code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> invokes inefficient new Long(long) constructor; use Long.valueOf(long)
> instead
> Bug type DM_NUMBER_CTOR (click for details)
> In class org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics
> In method
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> Called method new Long(long)
> Should call Long.valueOf(long) instead
> At QueueMetrics.java:[line 468]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9925) CapacitySchedulerQueueManager allows unsupported Queue hierarchy
[
https://issues.apache.org/jira/browse/YARN-9925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9925:
Affects Version/s: 3.3.0
> CapacitySchedulerQueueManager allows unsupported Queue hierarchy
>
>
> Key: YARN-9925
> URL: https://issues.apache.org/jira/browse/YARN-9925
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> CapacitySchedulerQueueManager allows unsupported Queue hierarchy. When
> creating a queue with same name as an existing parent queue name - it has to
> fail with below.
> {code:java}
> Caused by: java.io.IOException: A is moved from:root.A to:root.B.A after
> refresh, which is not allowed.Caused by: java.io.IOException: A is moved
> from:root.A to:root.B.A after refresh, which is not allowed. at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.validateQueueHierarchy(CapacitySchedulerQueueManager.java:335)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:180)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:762)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:473)
> ... 70 more
> {code}
> In Some cases, the error is not thrown while creating the queue but thrown at
> submission of job "Failed to submit application_1571677375269_0002 to YARN :
> Application application_1571677375269_0002 submitted by user : systest to
> non-leaf queue : B"
> Below scenarios are allowed but it should not
> {code:java}
> It allows root.A.A1.B when root.B.B1 exists already.
>
> 1. Add root.A
> 2. Add root.A.A1
> 3. Add root.B
> 4. Add root.B.B1
> 5. Allows Add of root.A.A1.B
> It allows two root queues:
>
> 1. Add root.A
> 2. Add root.B
> 3. Add root.A.A1
> 4. Allows Add of root.A.A1.root
>
> {code}
> Below scenario is handled properly:
> {code:java}
> It does not allow root.B.A when root.A.A1 exists already.
>
> 1. Add root.A
> 2. Add root.B
> 3. Add root.A.A1
> 4. Does not Allow Add of root.B.A
> {code}
> This error handling has to be consistent in all scenarios.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9925) CapacitySchedulerQueueManager allows unsupported Queue hierarchy
[
https://issues.apache.org/jira/browse/YARN-9925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9925:
Component/s: capacityscheduler
> CapacitySchedulerQueueManager allows unsupported Queue hierarchy
>
>
> Key: YARN-9925
> URL: https://issues.apache.org/jira/browse/YARN-9925
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> CapacitySchedulerQueueManager allows unsupported Queue hierarchy. When
> creating a queue with same name as an existing parent queue name - it has to
> fail with below.
> {code:java}
> Caused by: java.io.IOException: A is moved from:root.A to:root.B.A after
> refresh, which is not allowed.Caused by: java.io.IOException: A is moved
> from:root.A to:root.B.A after refresh, which is not allowed. at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.validateQueueHierarchy(CapacitySchedulerQueueManager.java:335)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:180)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:762)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:473)
> ... 70 more
> {code}
> In Some cases, the error is not thrown while creating the queue but thrown at
> submission of job "Failed to submit application_1571677375269_0002 to YARN :
> Application application_1571677375269_0002 submitted by user : systest to
> non-leaf queue : B"
> Below scenarios are allowed but it should not
> {code:java}
> It allows root.A.A1.B when root.B.B1 exists already.
>
> 1. Add root.A
> 2. Add root.A.A1
> 3. Add root.B
> 4. Add root.B.B1
> 5. Allows Add of root.A.A1.B
> It allows two root queues:
>
> 1. Add root.A
> 2. Add root.B
> 3. Add root.A.A1
> 4. Allows Add of root.A.A1.root
>
> {code}
> Below scenario is handled properly:
> {code:java}
> It does not allow root.B.A when root.A.A1 exists already.
>
> 1. Add root.A
> 2. Add root.B
> 3. Add root.A.A1
> 4. Does not Allow Add of root.B.A
> {code}
> This error handling has to be consistent in all scenarios.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9925) CapacitySchedulerQueueManager allows unsupported Queue hierarchy
[
https://issues.apache.org/jira/browse/YARN-9925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9925:
Description:
CapacitySchedulerQueueManager allows unsupported Queue hierarchy. When creating
a queue with same name as an existing parent queue name - it has to fail with
below.
{code:java}
Caused by: java.io.IOException: A is moved from:root.A to:root.B.A after
refresh, which is not allowed.Caused by: java.io.IOException: A is moved
from:root.A to:root.B.A after refresh, which is not allowed. at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.validateQueueHierarchy(CapacitySchedulerQueueManager.java:335)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:180)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:762)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:473)
... 70 more
{code}
In Some cases, the error is not thrown while creating the queue but thrown at
submission of job "Failed to submit application_1571677375269_0002 to YARN :
Application application_1571677375269_0002 submitted by user : systest to
non-leaf queue : B"
Below scenarios are allowed but it should not
{code:java}
It allows root.A.A1.B when root.B.B1 exists already.
1. Add root.A
2. Add root.A.A1
3. Add root.B
4. Add root.B.B1
5. Allows Add of root.A.A1.B
It allows two root queues:
1. Add root.A
2. Add root.B
3. Add root.A.A1
4. Allows Add of root.A.A1.root
{code}
Below scenario is handled properly:
{code:java}
It does not allow root.B.A when root.A.A1 exists already.
1. Add root.A
2. Add root.B
3. Add root.A.A1
4. Does not Allow Add of root.B.A
{code}
This error handling has to be consistent in all scenarios.
was:
CapacitySchedulerQueueManager allows unsupported Queue hierarchy. When creating
a queue with same name as an existing parent queue name - it has to fail with
below.
{code:java}
Caused by: java.io.IOException: A is moved from:root.A to:root.B.A after
refresh, which is not allowed.Caused by: java.io.IOException: A is moved
from:root.A to:root.B.A after refresh, which is not allowed. at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.validateQueueHierarchy(CapacitySchedulerQueueManager.java:335)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:180)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:762)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:473)
... 70 more
{code}
In Some cases, the error is not thrown while creating the queue but thrown at
submission of job "Failed to submit application_1571677375269_0002 to YARN :
Application application_1571677375269_0002 submitted by user : systest to
non-leaf queue : B"
Below scenarios are allowed but it should not
{code}
1. It allows root.A.A1.B when root.B.B1 exists already.
1. Add root.A
2. Add root.A.A1
3. Add root.B
4. Add root.B.B1
5. Allows Add of root.A.A1.B
2. It allows two root queues:
1. Add root.A
2. Add root.B
3. Add root.A.A1
4. Allows Add of root.A.A1.root
{code}
Below scenario is handled properly:
{code}
1. It does not allow root.B.A when root.A.A1 exists already.
1. Add root.A
2. Add root.B
3. Add root.A.A1
4. Does not Allow Add of root.B.A
{code}
This error handling has to be consistent in all scenarios.
> CapacitySchedulerQueueManager allows unsupported Queue hierarchy
>
>
> Key: YARN-9925
> URL: https://issues.apache.org/jira/browse/YARN-9925
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> CapacitySchedulerQueueManager allows unsupported Queue hierarchy. When
> creating a queue with same name as an existing parent queue name - it has to
> fail with below.
> {code:java}
> Caused by: java.io.IOException: A is moved from:root.A to:root.B.A after
> refresh, which is not allowed.Caused by: java.io.IOException: A is moved
> from:root.A to:root.B.A after refresh, which is not allowed. at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.validateQueueHierarchy(CapacitySchedulerQueueManager.java:335)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues
[jira] [Created] (YARN-9925) CapacitySchedulerQueueManager allows unsupported Queue hierarchy
Prabhu Joseph created YARN-9925:
---
Summary: CapacitySchedulerQueueManager allows unsupported Queue
hierarchy
Key: YARN-9925
URL: https://issues.apache.org/jira/browse/YARN-9925
Project: Hadoop YARN
Issue Type: Bug
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph
CapacitySchedulerQueueManager allows unsupported Queue hierarchy. When creating
a queue with same name as an existing parent queue name - it has to fail with
below.
{code:java}
Caused by: java.io.IOException: A is moved from:root.A to:root.B.A after
refresh, which is not allowed.Caused by: java.io.IOException: A is moved
from:root.A to:root.B.A after refresh, which is not allowed. at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.validateQueueHierarchy(CapacitySchedulerQueueManager.java:335)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:180)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:762)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:473)
... 70 more
{code}
In Some cases, the error is not thrown while creating the queue but thrown at
submission of job "Failed to submit application_1571677375269_0002 to YARN :
Application application_1571677375269_0002 submitted by user : systest to
non-leaf queue : B"
Below scenarios are allowed but it should not
{code}
1. It allows root.A.A1.B when root.B.B1 exists already.
1. Add root.A
2. Add root.A.A1
3. Add root.B
4. Add root.B.B1
5. Allows Add of root.A.A1.B
2. It allows two root queues:
1. Add root.A
2. Add root.B
3. Add root.A.A1
4. Allows Add of root.A.A1.root
{code}
Below scenario is handled properly:
{code}
1. It does not allow root.B.A when root.A.A1 exists already.
1. Add root.A
2. Add root.B
3. Add root.A.A1
4. Does not Allow Add of root.B.A
{code}
This error handling has to be consistent in all scenarios.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956347#comment-16956347 ] Eric Badger commented on YARN-9884: --- Thanks for the commit, [~eyang]! Thanks for the reviews [~Jim_Brennan] and [~ccondit]! > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9884.001.patch, YARN-9884.002.patch, > YARN-9884.003.patch, YARN-9884.004.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9915) Fix FindBug issue in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956338#comment-16956338
]
Eric Payne commented on YARN-9915:
--
Will look this afternoon.
> Fix FindBug issue in QueueMetrics
> -
>
> Key: YARN-9915
> URL: https://issues.apache.org/jira/browse/YARN-9915
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9915-01.patch
>
>
> Below FindBug issue appears in the trunk build
> {code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> invokes inefficient new Long(long) constructor; use Long.valueOf(long)
> instead
> Bug type DM_NUMBER_CTOR (click for details)
> In class org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics
> In method
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.registerCustomResources()
> Called method new Long(long)
> Should call Long.valueOf(long) instead
> At QueueMetrics.java:[line 468]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9887) Capacity scheduler: add support for limiting maxRunningApps per user
[
https://issues.apache.org/jira/browse/YARN-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956297#comment-16956297
]
Wangda Tan commented on YARN-9887:
--
[~pbacsko], [~epayne], IIRC the max app per user in FS is across queues, but CS
workaround mentioned by Eric is per queue.
For the convertion tool, I think it might be good enough to document and move
on, adding another app-limit per user globally sounds like creating more issues
for troubleshooting.
> Capacity scheduler: add support for limiting maxRunningApps per user
>
>
> Key: YARN-9887
> URL: https://issues.apache.org/jira/browse/YARN-9887
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Priority: Major
>
> Fair Scheduler supports limiting the number of applications that a particular
> user can submit:
> {noformat}
>
> 10
>
> {noformat}
> Capacity Scheduler does not have an exact equivalent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956284#comment-16956284 ] Wangda Tan commented on YARN-9886: -- [~kmarton], can we also make sure apps from privileged users can do such operation? Maybe we can add a config to the mapping policy to make sure only users like "hive" can do such operation. BTW this is also a requirement from Hive when doAs set to false. cc: [~ashutoshc], [~thejas] > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map this applications to the real submitting user's > queue instead of the Hive one. > For this cases if they would pass the username in the application tag we may > read it and use that one during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956275#comment-16956275 ] Eric Yang commented on YARN-9897: - Have you tried to build with -Pnative flag? Does it require any container-executor.additional_cflags to compile native code on Aarch64? > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9865) Capacity scheduler: add support for combined %user + %secondary_group mapping
[
https://issues.apache.org/jira/browse/YARN-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956235#comment-16956235
]
Manikandan R commented on YARN-9865:
Tried to fix the checksytle issue by removing the space after {, but saving the
file brings it back to original position. Hadoop formatter is doing it. Also,
issue is not because of this patch. Can you try it out?
> Capacity scheduler: add support for combined %user + %secondary_group mapping
> -
>
> Key: YARN-9865
> URL: https://issues.apache.org/jira/browse/YARN-9865
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9865.001.patch, YARN-9865.002.patch
>
>
> Similiar to YARN-9841, but for secondary group.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9924) TimelineSchemaCreator breaks after protobuffer upgrade
[ https://issues.apache.org/jira/browse/YARN-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anil Sadineni updated YARN-9924: Description: While creating schema using HBaseTimelineSchemaCreator, observed below issue. Looks like this one broke with proto buffers upgrade (HADOOP-16557). I ran TimelineSchemaCreator against hbase version 1.4.8 which is packaged with proto buffers 2.5. 2019-10-21 12:08:25,013 ERROR storage.HBaseTimelineSchemaCreator: Error in creating hbase tables:2019-10-21 12:08:25,013 ERROR storage.HBaseTimelineSchemaCreator: Error in creating hbase tables:org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.VerifyError: class com.google.protobuf.LiteralByteString overrides final method toString.(Ljava/lang/String;)Ljava/lang/String; at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:248) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:221) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.VerifyError: class com.google.protobuf.LiteralByteString overrides final method toString.(Ljava/lang/String;)Ljava/lang/String; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.hbase.util.ByteStringer.(ByteStringer.java:44) at org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:1053) at org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:496) at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:402) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:274) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219) ... 7 more2019-10-21 12:08:25,014 WARN storage.HBaseTimelineSchemaCreator: Schema creation finished with the following exceptions2019-10-21 12:08:25,014 WARN storage.HBaseTimelineSchemaCreator: java.lang.VerifyError: class com.google.protobuf.LiteralByteString overrides final method toString.(Ljava/lang/String;)Ljava/lang/String; was: While creating schema using HBaseTimelineSchemaCreator, observed below issue. Looks like this one broke with proto buffers upgrade (HADOOP-16557). 2019-10-21 12:08:25,013 ERROR storage.HBaseTimelineSchemaCreator: Error in creating hbase tables:2019-10-21 12:08:25,013 ERROR storage.HBaseTimelineSchemaCreator: Error in creating hbase tables:org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.VerifyError: class com.google.protobuf.LiteralByteString overrides final method toString.(Ljava/lang/String;)Ljava/lang/String; at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:248) at org.apache.hadoop.hbase.client.R
[jira] [Updated] (YARN-9924) TimelineSchemaCreator breaks after protobuffer upgrade
[ https://issues.apache.org/jira/browse/YARN-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anil Sadineni updated YARN-9924: Parent: YARN-9802 Issue Type: Sub-task (was: Task) > TimelineSchemaCreator breaks after protobuffer upgrade > -- > > Key: YARN-9924 > URL: https://issues.apache.org/jira/browse/YARN-9924 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Anil Sadineni >Priority: Major > > While creating schema using HBaseTimelineSchemaCreator, observed below issue. > Looks like this one broke with proto buffers upgrade (HADOOP-16557). > 2019-10-21 12:08:25,013 ERROR storage.HBaseTimelineSchemaCreator: Error in > creating hbase tables:2019-10-21 12:08:25,013 ERROR > storage.HBaseTimelineSchemaCreator: Error in creating hbase > tables:org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.VerifyError: > class com.google.protobuf.LiteralByteString overrides final method > toString.(Ljava/lang/String;)Ljava/lang/String; at > org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:248) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:221) > at > org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388) > at > org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.VerifyError: > class com.google.protobuf.LiteralByteString overrides final method > toString.(Ljava/lang/String;)Ljava/lang/String; at > java.lang.ClassLoader.defineClass1(Native Method) at > java.lang.ClassLoader.defineClass(ClassLoader.java:763) at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at > java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at > java.net.URLClassLoader.access$100(URLClassLoader.java:74) at > java.net.URLClassLoader$1.run(URLClassLoader.java:369) at > java.net.URLClassLoader$1.run(URLClassLoader.java:363) at > java.security.AccessController.doPrivileged(Native Method) at > java.net.URLClassLoader.findClass(URLClassLoader.java:362) at > java.lang.ClassLoader.loadClass(ClassLoader.java:424) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at > java.lang.ClassLoader.loadClass(ClassLoader.java:357) at > java.lang.ClassLoader.defineClass1(Native Method) at > java.lang.ClassLoader.defineClass(ClassLoader.java:763) at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at > java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at > java.net.URLClassLoader.access$100(URLClassLoader.java:74) at > java.net.URLClassLoader$1.run(URLClassLoader.java:369) at > java.net.URLClassLoader$1.run(URLClassLoader.java:363) at > java.security.AccessController.doPrivileged(Native Method) at > java.net.URLClassLoader.findClass(URLClassLoader.java:362) at > java.lang.ClassLoader.loadClass(ClassLoader.java:424) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at > java.lang.ClassLoader.loadClass(ClassLoader.java:357) at > org.apache.hadoop.hbase.util.ByteStringer.(ByteStringer.java:44) at > org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:1053) > at > org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:496) > at > org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:402) > at > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:274) > at > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219) > ... 7 more2019-10-21 12:08:25,014 WARN storage.HBaseTimelineSchemaCreator: > Schema creation finished with the following exceptions2019-10-21 12:08:25,014 > WARN storage.HBaseTimelineSchemaCreator: java.lang.VerifyError: class > com.google.protobuf.LiteralByteString overrides final method > toString.(Ljava/lang/String;)Ljava/lang/String; -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For a
[jira] [Created] (YARN-9924) TimelineSchemaCreator breaks after protobuffer upgrade
Anil Sadineni created YARN-9924: --- Summary: TimelineSchemaCreator breaks after protobuffer upgrade Key: YARN-9924 URL: https://issues.apache.org/jira/browse/YARN-9924 Project: Hadoop YARN Issue Type: Task Components: timelineserver Reporter: Anil Sadineni While creating schema using HBaseTimelineSchemaCreator, observed below issue. Looks like this one broke with proto buffers upgrade (HADOOP-16557). 2019-10-21 12:08:25,013 ERROR storage.HBaseTimelineSchemaCreator: Error in creating hbase tables:2019-10-21 12:08:25,013 ERROR storage.HBaseTimelineSchemaCreator: Error in creating hbase tables:org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.VerifyError: class com.google.protobuf.LiteralByteString overrides final method toString.(Ljava/lang/String;)Ljava/lang/String; at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:248) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:221) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.VerifyError: class com.google.protobuf.LiteralByteString overrides final method toString.(Ljava/lang/String;)Ljava/lang/String; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.hbase.util.ByteStringer.(ByteStringer.java:44) at org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:1053) at org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:496) at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:402) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:274) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219) ... 7 more2019-10-21 12:08:25,014 WARN storage.HBaseTimelineSchemaCreator: Schema creation finished with the following exceptions2019-10-21 12:08:25,014 WARN storage.HBaseTimelineSchemaCreator: java.lang.VerifyError: class com.google.protobuf.LiteralByteString overrides final method toString.(Ljava/lang/String;)Ljava/lang/String; -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9511) [JDK11] TestAuxServices#testRemoteAuxServiceClassPath YarnRuntimeException: The remote jarfile should not be writable by group or others. The current Permission is 436
[
https://issues.apache.org/jira/browse/YARN-9511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956185#comment-16956185
]
Adam Antal commented on YARN-9511:
--
Hi [~seanlau],
Thanks for the explanation. Could you please provide details on how you run the
unit tests (proper CLI command, etc.) to determine the JDK11 part of the issue?
> [JDK11] TestAuxServices#testRemoteAuxServiceClassPath YarnRuntimeException:
> The remote jarfile should not be writable by group or others. The current
> Permission is 436
> ---
>
> Key: YARN-9511
> URL: https://issues.apache.org/jira/browse/YARN-9511
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Szilard Nemeth
>Priority: Major
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8.
> {code}
> [ERROR]
> testRemoteAuxServiceClassPath(org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices)
> Time elapsed: 0.551 s <<<
> ERROR!org.apache.hadoop.yarn.exceptions.YarnRuntimeException: The remote
> jarfile should not be writable by group or others. The current Permission is
> 436
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:202)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices.testRemoteAuxServiceClassPath(TestAuxServices.java:268)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9886) Queue mapping based on userid passed through application tag
[
https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955920#comment-16955920
]
Kinga Marton edited comment on YARN-9886 at 10/21/19 1:58 PM:
--
As a conclusion of an offline discussion with [~sunilg] we have agreed on the
following:
* We will keep the username in the tag, but I will change the pattern to
{{u=}}
* I will add a configuration to enable/disable the application placement using
the application tag. This way we can keep the backward compatibility and avoid
any undesired side effects.
was (Author: kmarton):
As a conclusion of an offline discussion with [~sunilg] we have agreed on the
following:
* We will keep the username in the tag, but I will change the pattern to
{{u=}}
* I will add a configuration to enable/disable the application placement using
the application tag. This way we can keep the backport compatibility and avoid
any undesired side effects.
> Queue mapping based on userid passed through application tag
>
>
> Key: YARN-9886
> URL: https://issues.apache.org/jira/browse/YARN-9886
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9886-WIP.patch
>
>
> There are situations when the real submitting user differs from the user what
> arrives to YARN. For example in case of a Hive application when Hive
> impersonation is turned off, the hive queries will run as Hive user and the
> mapping is done based on this username. Unfortunately in this case YARN
> doesn't have any information about the real user and there are cases when the
> customer may want to map this applications to the real submitting user's
> queue instead of the Hive one.
> For this cases if they would pass the username in the application tag we may
> read it and use that one during the queue mapping, if that user has rights to
> run on the real user's queue.
> [~sunilg] please correct me if I missed something.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9887) Capacity scheduler: add support for limiting maxRunningApps per user
[
https://issues.apache.org/jira/browse/YARN-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956102#comment-16956102
]
Peter Bacsko commented on YARN-9887:
[~epayne] sorry for the delayed answer. Thanks for the info, I'm not really an
expert at CS yet, so it's very useful.
It looks good as a workaround when we convert from an existing Fair scheduler
configuration, but I believe that later on we need something that is exactly
the same as "maxRunningApps".
cc [~sunilg], [~leftnoteasy]
> Capacity scheduler: add support for limiting maxRunningApps per user
>
>
> Key: YARN-9887
> URL: https://issues.apache.org/jira/browse/YARN-9887
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Priority: Major
>
> Fair Scheduler supports limiting the number of applications that a particular
> user can submit:
> {noformat}
>
> 10
>
> {noformat}
> Capacity Scheduler does not have an exact equivalent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9865) Capacity scheduler: add support for combined %user + %secondary_group mapping
[ https://issues.apache.org/jira/browse/YARN-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956096#comment-16956096 ] Peter Bacsko commented on YARN-9865: [~maniraj...@gmail.com] could you please fix the last checkstyle issue? You can ignore fidbugs, it's just showed up recently. > Capacity scheduler: add support for combined %user + %secondary_group mapping > - > > Key: YARN-9865 > URL: https://issues.apache.org/jira/browse/YARN-9865 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Attachments: YARN-9865.001.patch, YARN-9865.002.patch > > > Similiar to YARN-9841, but for secondary group. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9922) Fix JavaDoc errors introduced by YARN-9699
[
https://issues.apache.org/jira/browse/YARN-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956017#comment-16956017
]
Peter Bacsko commented on YARN-9922:
Findbugs is unrelated, "no tests" is obvious.
pls check [~eyang]
> Fix JavaDoc errors introduced by YARN-9699
> --
>
> Key: YARN-9922
> URL: https://issues.apache.org/jira/browse/YARN-9922
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9922-001.patch
>
>
> We accidentally introduced some Javadoc errors in YARN-9699:
> {code:java}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java:89:
> error: bad use of '>'
> [ERROR]* Used during FS->CS conversion. When enabled, background threads
> are
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/package-info.java:20:
> error: bad use of '>'
> [ERROR] * This package contains classes related to the Fair Scheduler ->
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java:30:
> error: bad use of '>'
> [ERROR] * Parses arguments passed to the FS->CS converter.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverterParams.java:20:
> error: bad use of '>'
> [ERROR] * POJO that holds values for the FS->CS converter.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java:38:
> error: bad use of '>'
> [ERROR] * Class that determines what should happen if the FS->CS converter
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/PreconditionException.java:23:
> error: bad use of '>'
> [ERROR] * before FS->CS conversion.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/UnsupportedPropertyException.java:20:
> error: bad use of '>'
> [ERROR] * Thrown by the FS->CS converter if it encounters an
> [ERROR] ^
> {code}
>
> These should be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9922) Fix JavaDoc errors introduced by YARN-9699
[
https://issues.apache.org/jira/browse/YARN-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956010#comment-16956010
]
Hadoop QA commented on YARN-9922:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m
31s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9922 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983602/YARN-9922-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e731c64832af 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 447f46d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/25022/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25022/testReport/ |
| Max. process+thread count | 828 (vs. ulimit of 550
[jira] [Created] (YARN-9923) Detect missing Docker binary or not running Docker daemon
Adam Antal created YARN-9923:
Summary: Detect missing Docker binary or not running Docker daemon
Key: YARN-9923
URL: https://issues.apache.org/jira/browse/YARN-9923
Project: Hadoop YARN
Issue Type: New Feature
Components: nodemanager, yarn
Affects Versions: 3.2.1
Reporter: Adam Antal
Assignee: Adam Antal
Currently if a NodeManager is enabled to allocate Docker containers, but the
specified binary (docker.binary in the container-executor.cfg) is missing the
container allocation fails with the following error message:
{noformat}
Container launch fails
Exit code: 29
Exception message: Launch container failed
Shell error output: sh: : No
such file or directory
Could not inspect docker network to get type /usr/bin/docker network inspect
host --format='{{.Driver}}'.
Error constructing docker command, docker error code=-1, error message='Unknown
error'
{noformat}
I suggest to add a property say "yarn.nodemanager.runtime.linux.docker.check"
to have the following options:
- STARTUP: setting this option the NodeManager would not start if Docker
binaries are missing or the Docker daemon is not running (the exception is
considered FATAL during startup)
- RUNTIME: would give a more detailed/user-friendly exception in NodeManager's
side (NM logs) if Docker binaries are missing or the daemon is not working.
This would also prevent further Docker container allocation as long as the
binaries do not exist and the docker daemon is not running.
- NONE (default): preserving the current behaviour, throwing exception during
container allocation, carrying on using the default retry procedure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9773) Add QueueMetrics for Custom Resources
[ https://issues.apache.org/jira/browse/YARN-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955962#comment-16955962 ] Prabhu Joseph commented on YARN-9773: - Have raised YARN-9915 to fix the findbug issue. Thanks. > Add QueueMetrics for Custom Resources > - > > Key: YARN-9773 > URL: https://issues.apache.org/jira/browse/YARN-9773 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.1.4 > > Attachments: YARN-9773.001.patch, YARN-9773.002.patch, > YARN-9773.003.patch > > > Although the custom resource metrics are calculated and saved as a > QueueMetricsForCustomResources object within the QueueMetrics class, the JMX > and Simon QueueMetrics do not report that information for custom resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9773) Add QueueMetrics for Custom Resources
[ https://issues.apache.org/jira/browse/YARN-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955958#comment-16955958 ] Tarun Parimi commented on YARN-9773: Got a findbugs warning from the changes done in this jira. https://builds.apache.org/job/PreCommit-YARN-Build/25021/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html > Add QueueMetrics for Custom Resources > - > > Key: YARN-9773 > URL: https://issues.apache.org/jira/browse/YARN-9773 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.1.4 > > Attachments: YARN-9773.001.patch, YARN-9773.002.patch, > YARN-9773.003.patch > > > Although the custom resource metrics are calculated and saved as a > QueueMetricsForCustomResources object within the QueueMetrics class, the JMX > and Simon QueueMetrics do not report that information for custom resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9921) Issue in PlacementConstraint when YARN Service AM retries allocation on component failure.
[
https://issues.apache.org/jira/browse/YARN-9921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955955#comment-16955955
]
Tarun Parimi commented on YARN-9921:
The Findbugs warning is due to the changes done in YARN-9773 and is not
related to the patch.
> Issue in PlacementConstraint when YARN Service AM retries allocation on
> component failure.
> --
>
> Key: YARN-9921
> URL: https://issues.apache.org/jira/browse/YARN-9921
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: YARN-9921.001.patch, differenceProtobuf.png
>
>
> When YARN Service AM tries to relaunch a container on failure, we encounter
> the below error in PlacementConstraints.
> {code:java}
> ERROR impl.AMRMClientAsyncImpl: Exception on heartbeat
> org.apache.hadoop.yarn.exceptions.YarnException:
> org.apache.hadoop.yarn.exceptions.SchedulerInvalidResoureRequestException:
> Invalid updated SchedulingRequest added to scheduler, we only allows changing
> numAllocations for the updated SchedulingRequest.
> Old=SchedulingRequestPBImpl{priority=0, allocationReqId=0,
> executionType={Execution Type: GUARANTEED, Enforce Execution Type: true},
> allocationTags=[component],
> resourceSizing=ResourceSizingPBImpl{numAllocations=0,
> resources=},
> placementConstraint=notin,node,llap:notin,node,yarn_node_partition/=[label]}
> new=SchedulingRequestPBImpl{priority=0, allocationReqId=0,
> executionType={Execution Type: GUARANTEED, Enforce Execution Type: true},
> allocationTags=[component],
> resourceSizing=ResourceSizingPBImpl{numAllocations=1,
> resources=},
> placementConstraint=notin,node,component:notin,node,yarn_node_partition/=[label]},
> if any fields need to be updated, please cancel the old request (by setting
> numAllocations to 0) and send a SchedulingRequest with different combination
> of priority/allocationId
> {code}
> But we can see from the message that the SchedulingRequest is indeed valid
> with everything same except numAllocations as expected. But still the below
> equals check in SingleConstraintAppPlacementAllocator fails.
> {code:java}
> // Compare two objects
> if (!schedulingRequest.equals(newSchedulingRequest)) {
> // Rollback #numAllocations
> sizing.setNumAllocations(newNumAllocations);
> throw new SchedulerInvalidResoureRequestException(
> "Invalid updated SchedulingRequest added to scheduler, "
> + " we only allows changing numAllocations for the updated "
> + "SchedulingRequest. Old=" + schedulingRequest.toString()
> + " new=" + newSchedulingRequest.toString()
> + ", if any fields need to be updated, please cancel the "
> + "old request (by setting numAllocations to 0) and send a "
> + "SchedulingRequest with different combination of "
> + "priority/allocationId");
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config [Phase 1]
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955926#comment-16955926 ] Peter Bacsko commented on YARN-9699: Thanks [~eyang], I created YARN-9922 to fix these Javadoc issues. > Migration tool that help to generate CS config based on FS config [Phase 1] > > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.3.0 > > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699-012.patch, > YARN-9699-013.patch, YARN-9699-014.patch, YARN-9699-015.patch, > YARN-9699-016.patch, YARN-9699-017.patch, YARN-9699.001.patch, > YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9922) Fix JavaDoc errors introduced by YARN-9699
[
https://issues.apache.org/jira/browse/YARN-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-9922:
---
Attachment: YARN-9922-001.patch
> Fix JavaDoc errors introduced by YARN-9699
> --
>
> Key: YARN-9922
> URL: https://issues.apache.org/jira/browse/YARN-9922
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9922-001.patch
>
>
> We accidentally introduced some Javadoc errors in YARN-9699:
> {code:java}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java:89:
> error: bad use of '>'
> [ERROR]* Used during FS->CS conversion. When enabled, background threads
> are
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/package-info.java:20:
> error: bad use of '>'
> [ERROR] * This package contains classes related to the Fair Scheduler ->
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java:30:
> error: bad use of '>'
> [ERROR] * Parses arguments passed to the FS->CS converter.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverterParams.java:20:
> error: bad use of '>'
> [ERROR] * POJO that holds values for the FS->CS converter.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java:38:
> error: bad use of '>'
> [ERROR] * Class that determines what should happen if the FS->CS converter
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/PreconditionException.java:23:
> error: bad use of '>'
> [ERROR] * before FS->CS conversion.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/UnsupportedPropertyException.java:20:
> error: bad use of '>'
> [ERROR] * Thrown by the FS->CS converter if it encounters an
> [ERROR] ^
> {code}
>
> These should be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9922) Fix JavaDoc errors introduced by YARN-9699
[
https://issues.apache.org/jira/browse/YARN-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-9922:
---
Description:
We accidentally introduced some Javadoc errors in YARN-9699:
{code:java}
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java:89:
error: bad use of '>'
[ERROR]* Used during FS->CS conversion. When enabled, background threads are
[ERROR] ^
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/package-info.java:20:
error: bad use of '>'
[ERROR] * This package contains classes related to the Fair Scheduler ->
[ERROR] ^
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java:30:
error: bad use of '>'
[ERROR] * Parses arguments passed to the FS->CS converter.
[ERROR] ^
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverterParams.java:20:
error: bad use of '>'
[ERROR] * POJO that holds values for the FS->CS converter.
[ERROR] ^
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java:38:
error: bad use of '>'
[ERROR] * Class that determines what should happen if the FS->CS converter
[ERROR] ^
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/PreconditionException.java:23:
error: bad use of '>'
[ERROR] * before FS->CS conversion.
[ERROR] ^
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/UnsupportedPropertyException.java:20:
error: bad use of '>'
[ERROR] * Thrown by the FS->CS converter if it encounters an
[ERROR] ^
{code}
These should be fixed.
> Fix JavaDoc errors introduced by YARN-9699
> --
>
> Key: YARN-9922
> URL: https://issues.apache.org/jira/browse/YARN-9922
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> We accidentally introduced some Javadoc errors in YARN-9699:
> {code:java}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java:89:
> error: bad use of '>'
> [ERROR]* Used during FS->CS conversion. When enabled, background threads
> are
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/package-info.java:20:
> error: bad use of '>'
> [ERROR] * This package contains classes related to the Fair Scheduler ->
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java:30:
> error: bad use of '>'
> [ERROR] * Parses arguments passed to the FS->CS converter.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverterParams.java:20:
> error: bad use of '>'
> [ERROR] * POJO that holds values for the FS->CS converter.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-
[jira] [Created] (YARN-9922) Fix JavaDoc errors introduced by YARN-9699
Peter Bacsko created YARN-9922: -- Summary: Fix JavaDoc errors introduced by YARN-9699 Key: YARN-9922 URL: https://issues.apache.org/jira/browse/YARN-9922 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Peter Bacsko Assignee: Peter Bacsko -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9886) Queue mapping based on userid passed through application tag
[
https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955920#comment-16955920
]
Kinga Marton commented on YARN-9886:
As a conclusion of an offline discussion with [~sunilg] we have agreed on the
following:
* We will keep the username in the tag, but I will change the pattern to
{{u=}}
* I will add a configuration to enable/disable the application placement using
the application tag. This way we can keep the backport compatibility and avoid
any undesired side effects.
> Queue mapping based on userid passed through application tag
>
>
> Key: YARN-9886
> URL: https://issues.apache.org/jira/browse/YARN-9886
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9886-WIP.patch
>
>
> There are situations when the real submitting user differs from the user what
> arrives to YARN. For example in case of a Hive application when Hive
> impersonation is turned off, the hive queries will run as Hive user and the
> mapping is done based on this username. Unfortunately in this case YARN
> doesn't have any information about the real user and there are cases when the
> customer may want to map this applications to the real submitting user's
> queue instead of the Hive one.
> For this cases if they would pass the username in the application tag we may
> read it and use that one during the queue mapping, if that user has rights to
> run on the real user's queue.
> [~sunilg] please correct me if I missed something.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9921) Issue in PlacementConstraint when YARN Service AM retries allocation on component failure.
[
https://issues.apache.org/jira/browse/YARN-9921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955881#comment-16955881
]
Hadoop QA commented on YARN-9921:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 19s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
47s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m
21s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9921 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983572/YARN-9921.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a4cac15d457d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 447f46d |
| maven | version: Apache Maven 3.3.
