[jira] [Deleted] (YARN-11051) .
[ https://issues.apache.org/jira/browse/YARN-11051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka deleted YARN-11051: - > . > - > > Key: YARN-11051 > URL: https://issues.apache.org/jira/browse/YARN-11051 > Project: Hadoop YARN > Issue Type: Bug >Reporter: JustinWang >Priority: Blocker > > . -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11051) FileNotFoundException: /tmp/hadoop-maprchin/nm-local-dir/filecache/970
[ https://issues.apache.org/jira/browse/YARN-11051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved YARN-11051. -- Resolution: Invalid Hadoop JIRA is not for end-user question. BTW, I suppose you should ask the question to MapR customer support because your environment is MapR. > FileNotFoundException: /tmp/hadoop-maprchin/nm-local-dir/filecache/970 > -- > > Key: YARN-11051 > URL: https://issues.apache.org/jira/browse/YARN-11051 > Project: Hadoop YARN > Issue Type: Bug >Reporter: JustinWang >Priority: Blocker > Attachments: image-2021-12-16-20-56-09-652.png > > > FileNotFoundException: > /tmp/hadoop-maprchin/nm-local-dir/filecache/970/infa_rpm.tar/services/shared/hadoop/MapR_6.1/lib/spark-xml_2.11-0.4.0.jar > (No such file or directory) > > situation is : when I run BDM workflow (contains 3 mapping), I monitor the > cache folder and found 970 created at beginning, but be deleted before the > mapping which need 970, that's so weired. any possible the cache folder will > be truncate random during these 3 mapping run? > > _2021-12-16 16:01:00.887 SEVERE: The > Integration Service failed to execute the mapping._ > _Caused by: java.io.FileNotFoundException: > /tmp/hadoop-maprchin/nm-local-dir/filecache/970/infa_rpm.tar/services/shared/hadoop/MapR_6.1/lib/spark-xml_2.11-0.4.0.jar > (No such file or directory)_ > !image-2021-12-16-20-56-09-652.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9515) Non-exclusive labels do not respect user-limit-factor/max-capacity
[
https://issues.apache.org/jira/browse/YARN-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456905#comment-17456905
]
Akira Ajisaka commented on YARN-9515:
-
According to
[https://github.com/apache/hadoop/blob/d59890404611629d364c39537e6a0a53808403e1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java#L834-L841]
{code:java}
// - When doing IGNORE_PARTITION_EXCLUSIVITY allocation, we will not
respect
// queue's max capacity, queue's max capacity on the partition will be
// considered to be 100%. Which is a queue can use all resource in the
// partition.
// Doing this because: for non-exclusive allocation, we make sure there's
// idle resource on the partition, to avoid wastage, such resource will be
// leveraged as much as we can, and preemption policy will reclaim it back
// when partitioned-resource-request comes back.
{code}
this issue looks as expected, and I think it won't be fixed.
> Non-exclusive labels do not respect user-limit-factor/max-capacity
> --
>
> Key: YARN-9515
> URL: https://issues.apache.org/jira/browse/YARN-9515
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.8.5
>Reporter: Brandon Scheller
>Priority: Major
>
> When using a cluster with non-exclusive labels. Queues will only respect
> user-limit-factor for allocations to the default partition. Non-exclusive
> label allocations will not consider these factors allowing the labeled
> partitions capacity to be completely used up.
> To reproduce this, consider this example.
> Cluster contains 1 non-exclusive nodelabel: *APPMASTER*
> "test" queue has access to all labels: *
> "test" queue has capacity/max-capacity: 20
> "test" queue has user-limit-factor: 0.1
> Job is submitted to "test" queue with label: *APPMASTER* only on its
> appMaster container request, and no-label for all other containers.
> user-limit-factor and capacity will not be respected for the job on
> allocations to the *APPMASTER* partition causing the single job to attempt to
> use the entire capacity of the *APPMASTER* partition through non-exclusive
> allocations.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9515) Non-exclusive labels do not respect user-limit-factor/max-capacity
[ https://issues.apache.org/jira/browse/YARN-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved YARN-9515. - Resolution: Won't Fix > Non-exclusive labels do not respect user-limit-factor/max-capacity > -- > > Key: YARN-9515 > URL: https://issues.apache.org/jira/browse/YARN-9515 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.8.5 >Reporter: Brandon Scheller >Priority: Major > > When using a cluster with non-exclusive labels. Queues will only respect > user-limit-factor for allocations to the default partition. Non-exclusive > label allocations will not consider these factors allowing the labeled > partitions capacity to be completely used up. > To reproduce this, consider this example. > Cluster contains 1 non-exclusive nodelabel: *APPMASTER* > "test" queue has access to all labels: * > "test" queue has capacity/max-capacity: 20 > "test" queue has user-limit-factor: 0.1 > Job is submitted to "test" queue with label: *APPMASTER* only on its > appMaster container request, and no-label for all other containers. > user-limit-factor and capacity will not be respected for the job on > allocations to the *APPMASTER* partition causing the single job to attempt to > use the entire capacity of the *APPMASTER* partition through non-exclusive > allocations. > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9063) ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing
[
https://issues.apache.org/jira/browse/YARN-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved YARN-9063.
-
Fix Version/s: 3.4.0
2.10.2
3.2.4
3.3.3
Resolution: Fixed
Committed to trunk, branch-3.3, branch-3.2, and branch-2.10. Thank you
[~tarunparimi] for your report and thanks [~groot] for your contribution.
> ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing
> -
>
> Key: YARN-9063
> URL: https://issues.apache.org/jira/browse/YARN-9063
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver, timelineservice
>Affects Versions: 2.8.0
>Reporter: Tarun Parimi
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.3
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> ATS v1.5 fails to start up if there are some missing files in
> RollingLevelDBTimelineStore. YARN-6054 fixes this issue only for the
> LevelDBTimelineStore. Since RollingLevelDBTimelineStore opens multiple level
> db and rolls them, we need a separate fix for this. The error is shown below
> {code}
> 18/11/13 07:00:56 FATAL applicationhistoryservice.ApplicationHistoryServer:
> Error starting ApplicationHistoryServer
> org.apache.hadoop.service.ServiceStateException:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1
> missing files; e.g.:
> /tmp/ats_folder/yarn/timeline/leveldb-timeline-store/owner-ldb/05.sst
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
>
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
>
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111)
>
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174)
>
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184)
>
> Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException:
> Corruption: 1 missing files; e.g.:
> /tmp/ats-folder/yarn/timeline/leveldb-timeline-store/owner-ldb/05.sst
> at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at
> org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceInit(RollingLevelDBTimelineStore.java:321)
>
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9063) ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing
[
https://issues.apache.org/jira/browse/YARN-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned YARN-9063:
---
Assignee: Ashutosh Gupta (was: Tarun Parimi)
> ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing
> -
>
> Key: YARN-9063
> URL: https://issues.apache.org/jira/browse/YARN-9063
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver, timelineservice
>Affects Versions: 2.8.0
>Reporter: Tarun Parimi
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> ATS v1.5 fails to start up if there are some missing files in
> RollingLevelDBTimelineStore. YARN-6054 fixes this issue only for the
> LevelDBTimelineStore. Since RollingLevelDBTimelineStore opens multiple level
> db and rolls them, we need a separate fix for this. The error is shown below
> {code}
> 18/11/13 07:00:56 FATAL applicationhistoryservice.ApplicationHistoryServer:
> Error starting ApplicationHistoryServer
> org.apache.hadoop.service.ServiceStateException:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1
> missing files; e.g.:
> /tmp/ats_folder/yarn/timeline/leveldb-timeline-store/owner-ldb/05.sst
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
>
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
>
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111)
>
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174)
>
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184)
>
> Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException:
> Corruption: 1 missing files; e.g.:
> /tmp/ats-folder/yarn/timeline/leveldb-timeline-store/owner-ldb/05.sst
> at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at
> org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceInit(RollingLevelDBTimelineStore.java:321)
>
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9984) FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node
[
https://issues.apache.org/jira/browse/YARN-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-9984:
Fix Version/s: 2.10.2
Cherry-picked to branch-2.10.
> FSPreemptionThread can cause NullPointerException while app is unregistered
> with containers running on a node
> -
>
> Key: YARN-9984
> URL: https://issues.apache.org/jira/browse/YARN-9984
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 3.1.4, 2.10.2
>
> Attachments: YARN-9984.001.patch
>
>
> When an application is unregistered there is a chance that there are still
> containers running on a node for that application. In all cases we handle the
> application missing from the RM gracefully (log a message and continue)
> except for the FS pre-emption thread.
> In case the application is removed but some containers are still linked to a
> node the FSPreemptionThread will crash with a NPE when it tries to retrieve
> the application id for the attempt:
> {code:java}
> FSAppAttempt app =
> scheduler.getSchedulerApp(container.getApplicationAttemptId());
> ApplicationId appId = app.getApplicationId();{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10438) Handle null containerId in ClientRMService#getContainerReport()
[
https://issues.apache.org/jira/browse/YARN-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10438:
-
Fix Version/s: 2.10.2
Cherry-picked to branch-2.10.
> Handle null containerId in ClientRMService#getContainerReport()
> ---
>
> Key: YARN-10438
> URL: https://issues.apache.org/jira/browse/YARN-10438
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.2.1
>Reporter: Raghvendra Singh
>Assignee: Shubham Gupta
>Priority: Major
> Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2
>
>
> Here is the Exception trace which we are seeing, we are suspecting because of
> this exception RM is reaching in a state where it is no more allowing any new
> job to run on the cluster.
> {noformat}
> 2020-09-15 07:08:15,496 WARN ipc.Server: IPC Server handler 18 on default
> port 8032, call Call#1463486 Retry#0
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getContainerReport
> from 10.39.91.205:49564 java.lang.NullPointerException at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getContainerReport(ClientRMService.java:520)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getContainerReport(ApplicationClientProtocolPBServiceImpl.java:466)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:639)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
> {noformat}
> We are seeing this issue with this specific node only, we do run this cluster
> at a scale of around 500 nodes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7266) Timeline Server event handler threads locked
[ https://issues.apache.org/jira/browse/YARN-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-7266: Fix Version/s: 2.10.2 Cherry-picked to branch-2.10. > Timeline Server event handler threads locked > > > Key: YARN-7266 > URL: https://issues.apache.org/jira/browse/YARN-7266 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2, timelineserver >Affects Versions: 2.7.3 >Reporter: Venkata Puneet Ravuri >Assignee: Prabhu Joseph >Priority: Major > Fix For: 2.7.8, 3.3.0, 2.8.6, 2.9.3, 2.10.2, 3.2.4 > > Attachments: YARN-7266-0005.patch, YARN-7266-001.patch, > YARN-7266-002.patch, YARN-7266-003.patch, YARN-7266-004.patch, > YARN-7266-006.patch, YARN-7266-007.patch, YARN-7266-008.patch, > YARN-7266-branch-2.7.001.patch, YARN-7266-branch-2.8.001.patch > > > Event handlers for Timeline Server seem to take a lock while parsing HTTP > headers of the request. This is causing all other threads to wait and slowing > down the overall performance of Timeline server. We have resourcemanager > metrics enabled to send to timeline server. Because of the high load on > ResourceManager, the metrics to be sent are getting backlogged and in turn > increasing heap footprint of Resource Manager (due to pending metrics). > This is the complete stack trace of a blocked thread on timeline server:- > "2079644967@qtp-1658980982-4560" #4632 daemon prio=5 os_prio=0 > tid=0x7f6ba490a000 nid=0x5eb waiting for monitor entry > [0x7f6b9142c000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:82) > - waiting to lock <0x0005c0621860> (a java.lang.Class for > com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector) > at > com.sun.xml.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:168) > at > com.sun.xml.bind.v2.runtime.reflect.Accessor$FieldReflection.optimize(Accessor.java:282) > at > com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.(SingleElementNodeProperty.java:94) > at sun.reflect.GeneratedConstructorAccessor52.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown > Source) > at java.lang.reflect.Constructor.newInstance(Unknown Source) > at > com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128) > at > com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:183) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:532) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:551) > at > com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.(ArrayElementProperty.java:112) > at > com.sun.xml.bind.v2.runtime.property.ArrayElementNodeProperty.(ArrayElementNodeProperty.java:62) > at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown > Source) > at java.lang.reflect.Constructor.newInstance(Unknown Source) > at > com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128) > at > com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:183) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:532) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.(JAXBContextImpl.java:347) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1170) > at > com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:145) > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at javax.xml.bind.ContextFinder.newInstance(Unknown Source) > at javax.xml.bind.ContextFinder.newInstance(Unknown Source) > at javax.xml.bind.ContextFinder.find(Unknown Source) > at javax.xml.bind.JAXBContext.newInstance(Unknown Source) > at javax.xml.bind.JAXBContext.newInstance(Unknown Source) > at > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.buildModelAndSchemas(WadlGeneratorJAXBGrammarGenerator.java:412) > at > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.createExternalGrammar(WadlGeneratorJAXBGrammarGenerator.java:352) > at > com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:115) >
[jira] [Updated] (YARN-7266) Timeline Server event handler threads locked
[ https://issues.apache.org/jira/browse/YARN-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-7266: Fix Version/s: 3.2.4 Cherry-picked to branch-3.2. > Timeline Server event handler threads locked > > > Key: YARN-7266 > URL: https://issues.apache.org/jira/browse/YARN-7266 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2, timelineserver >Affects Versions: 2.7.3 >Reporter: Venkata Puneet Ravuri >Assignee: Prabhu Joseph >Priority: Major > Fix For: 2.7.8, 3.3.0, 2.8.6, 2.9.3, 3.2.4 > > Attachments: YARN-7266-0005.patch, YARN-7266-001.patch, > YARN-7266-002.patch, YARN-7266-003.patch, YARN-7266-004.patch, > YARN-7266-006.patch, YARN-7266-007.patch, YARN-7266-008.patch, > YARN-7266-branch-2.7.001.patch, YARN-7266-branch-2.8.001.patch > > > Event handlers for Timeline Server seem to take a lock while parsing HTTP > headers of the request. This is causing all other threads to wait and slowing > down the overall performance of Timeline server. We have resourcemanager > metrics enabled to send to timeline server. Because of the high load on > ResourceManager, the metrics to be sent are getting backlogged and in turn > increasing heap footprint of Resource Manager (due to pending metrics). > This is the complete stack trace of a blocked thread on timeline server:- > "2079644967@qtp-1658980982-4560" #4632 daemon prio=5 os_prio=0 > tid=0x7f6ba490a000 nid=0x5eb waiting for monitor entry > [0x7f6b9142c000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:82) > - waiting to lock <0x0005c0621860> (a java.lang.Class for > com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector) > at > com.sun.xml.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:168) > at > com.sun.xml.bind.v2.runtime.reflect.Accessor$FieldReflection.optimize(Accessor.java:282) > at > com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.(SingleElementNodeProperty.java:94) > at sun.reflect.GeneratedConstructorAccessor52.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown > Source) > at java.lang.reflect.Constructor.newInstance(Unknown Source) > at > com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128) > at > com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:183) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:532) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:551) > at > com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.(ArrayElementProperty.java:112) > at > com.sun.xml.bind.v2.runtime.property.ArrayElementNodeProperty.(ArrayElementNodeProperty.java:62) > at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown > Source) > at java.lang.reflect.Constructor.newInstance(Unknown Source) > at > com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128) > at > com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:183) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:532) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.(JAXBContextImpl.java:347) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1170) > at > com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:145) > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at javax.xml.bind.ContextFinder.newInstance(Unknown Source) > at javax.xml.bind.ContextFinder.newInstance(Unknown Source) > at javax.xml.bind.ContextFinder.find(Unknown Source) > at javax.xml.bind.JAXBContext.newInstance(Unknown Source) > at javax.xml.bind.JAXBContext.newInstance(Unknown Source) > at > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.buildModelAndSchemas(WadlGeneratorJAXBGrammarGenerator.java:412) > at > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.createExternalGrammar(WadlGeneratorJAXBGrammarGenerator.java:352) > at > com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:115) > at >
[jira] [Updated] (YARN-11007) Correct words in YARN documents
[ https://issues.apache.org/jira/browse/YARN-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-11007: - Issue Type: Bug (was: Improvement) > Correct words in YARN documents > --- > > Key: YARN-11007 > URL: https://issues.apache.org/jira/browse/YARN-11007 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Assignee: guo >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.3 > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11007) Correct words in YARN documents
[ https://issues.apache.org/jira/browse/YARN-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved YARN-11007. -- Fix Version/s: 3.4.0 3.2.4 3.3.3 Resolution: Fixed Committed to trunk, branch-3.3, and branch-3.2. Thank you [~philipse] for cleaning up the documents! > Correct words in YARN documents > --- > > Key: YARN-11007 > URL: https://issues.apache.org/jira/browse/YARN-11007 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Assignee: guo >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.3 > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11007) Correct words in YARN documents
[ https://issues.apache.org/jira/browse/YARN-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned YARN-11007: Assignee: guo > Correct words in YARN documents > --- > > Key: YARN-11007 > URL: https://issues.apache.org/jira/browse/YARN-11007 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Assignee: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11007) Correct words in YARN documents
[ https://issues.apache.org/jira/browse/YARN-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-11007: - Summary: Correct words in YARN documents (was: improve doc) > Correct words in YARN documents > --- > > Key: YARN-11007 > URL: https://issues.apache.org/jira/browse/YARN-11007 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10991) Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method
[ https://issues.apache.org/jira/browse/YARN-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved YARN-10991. -- Fix Version/s: 3.4.0 3.2.4 3.3.3 (was: 3.3.1) Resolution: Fixed Committed to trunk, branch-3.3, and branch-3.2. Thanks [~groot] for your contribution. Welcome! > Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString > method > --- > > Key: YARN-10991 > URL: https://issues.apache.org/jira/browse/YARN-10991 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-shell >Affects Versions: 3.3.1 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.3 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently if the resourcesStr argument in parseResourcesString method > contains "]" at the end, its not being ignored. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9452) Fix TestDistributedShell and TestTimelineAuthFilterForV2 failures
[
https://issues.apache.org/jira/browse/YARN-9452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448361#comment-17448361
]
Akira Ajisaka commented on YARN-9452:
-
Hi [~tanakahda], would you create a backport PR?
> Fix TestDistributedShell and TestTimelineAuthFilterForV2 failures
> -
>
> Key: YARN-9452
> URL: https://issues.apache.org/jira/browse/YARN-9452
> Project: Hadoop YARN
> Issue Type: Bug
> Components: ATSv2, distributed-shell, test
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9452-001.patch, YARN-9452-002.patch,
> YARN-9452-003.patch, YARN-9452-004.patch
>
>
> *TestDistributedShell#testDSShellWithoutDomainV2CustomizedFlow*
> {code}
> [ERROR]
> testDSShellWithoutDomainV2CustomizedFlow(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 72.14 s <<< FAILURE!
> java.lang.AssertionError: Entity ID prefix should be same across each publish
> of same entity expected:<9223372036854775806> but was:<9223370482298585580>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.verifyEntityForTimelineV2(TestDistributedShell.java:695)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.checkTimelineV2(TestDistributedShell.java:588)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:459)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow(TestDistributedShell.java:330)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> *TestTimelineAuthFilterForV2#testPutTimelineEntities*
> {code}
> [ERROR]
> testPutTimelineEntities[3](org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2)
> Time elapsed: 1.047 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:712)
> at org.junit.Assert.assertNotNull(Assert.java:722)
> at
> org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.verifyEntity(TestTimelineAuthFilterForV2.java:282)
> at
> org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.testPutTimelineEntities(TestTimelineAuthFilterForV2.java:421)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
[jira] [Commented] (YARN-10438) Handle null containerId in ClientRMService#getContainerReport()
[
https://issues.apache.org/jira/browse/YARN-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445996#comment-17445996
]
Akira Ajisaka commented on YARN-10438:
--
Hi [~chaosun], I think it is good to include this in Hadoop 3.3.2 release.
Would you check this?
> Handle null containerId in ClientRMService#getContainerReport()
> ---
>
> Key: YARN-10438
> URL: https://issues.apache.org/jira/browse/YARN-10438
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.2.1
>Reporter: Raghvendra Singh
>Assignee: Shubham Gupta
>Priority: Major
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
>
> Here is the Exception trace which we are seeing, we are suspecting because of
> this exception RM is reaching in a state where it is no more allowing any new
> job to run on the cluster.
> {noformat}
> 2020-09-15 07:08:15,496 WARN ipc.Server: IPC Server handler 18 on default
> port 8032, call Call#1463486 Retry#0
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getContainerReport
> from 10.39.91.205:49564 java.lang.NullPointerException at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getContainerReport(ClientRMService.java:520)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getContainerReport(ApplicationClientProtocolPBServiceImpl.java:466)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:639)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
> {noformat}
> We are seeing this issue with this specific node only, we do run this cluster
> at a scale of around 500 nodes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10438) Handle null containerId in ClientRMService#getContainerReport()
[
https://issues.apache.org/jira/browse/YARN-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10438:
-
Fix Version/s: 3.2.3
3.3.2
> Handle null containerId in ClientRMService#getContainerReport()
> ---
>
> Key: YARN-10438
> URL: https://issues.apache.org/jira/browse/YARN-10438
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.2.1
>Reporter: Raghvendra Singh
>Assignee: Shubham Gupta
>Priority: Major
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
>
> Here is the Exception trace which we are seeing, we are suspecting because of
> this exception RM is reaching in a state where it is no more allowing any new
> job to run on the cluster.
> {noformat}
> 2020-09-15 07:08:15,496 WARN ipc.Server: IPC Server handler 18 on default
> port 8032, call Call#1463486 Retry#0
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getContainerReport
> from 10.39.91.205:49564 java.lang.NullPointerException at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getContainerReport(ClientRMService.java:520)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getContainerReport(ApplicationClientProtocolPBServiceImpl.java:466)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:639)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
> {noformat}
> We are seeing this issue with this specific node only, we do run this cluster
> at a scale of around 500 nodes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10438) Handle null containerId in ClientRMService#getContainerReport()
[
https://issues.apache.org/jira/browse/YARN-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445987#comment-17445987
]
Akira Ajisaka commented on YARN-10438:
--
Backported to branch-3.3, branch-3.2, and branch-3.2.3.
> Handle null containerId in ClientRMService#getContainerReport()
> ---
>
> Key: YARN-10438
> URL: https://issues.apache.org/jira/browse/YARN-10438
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.2.1
>Reporter: Raghvendra Singh
>Assignee: Shubham Gupta
>Priority: Major
> Fix For: 3.4.0
>
>
> Here is the Exception trace which we are seeing, we are suspecting because of
> this exception RM is reaching in a state where it is no more allowing any new
> job to run on the cluster.
> {noformat}
> 2020-09-15 07:08:15,496 WARN ipc.Server: IPC Server handler 18 on default
> port 8032, call Call#1463486 Retry#0
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getContainerReport
> from 10.39.91.205:49564 java.lang.NullPointerException at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getContainerReport(ClientRMService.java:520)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getContainerReport(ApplicationClientProtocolPBServiceImpl.java:466)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:639)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
> {noformat}
> We are seeing this issue with this specific node only, we do run this cluster
> at a scale of around 500 nodes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10991) Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method
[ https://issues.apache.org/jira/browse/YARN-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned YARN-10991: Assignee: Ashutosh Gupta > Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString > method > --- > > Key: YARN-10991 > URL: https://issues.apache.org/jira/browse/YARN-10991 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-shell >Affects Versions: 3.3.1 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently if the resourcesStr argument in parseResourcesString method > contains "]" at the end, its not being ignored. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10976) Fix resource leak due to Files.walk
[
https://issues.apache.org/jira/browse/YARN-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved YARN-10976.
--
Fix Version/s: 3.3.2
3.4.0
Resolution: Fixed
Committed to trunk and branch-3.3. Thank you [~xiaoheipangzi] for your
contribution!
> Fix resource leak due to Files.walk
> ---
>
> Key: YARN-10976
> URL: https://issues.apache.org/jira/browse/YARN-10976
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Stream creates by File.walk should be closed, like jdk said:
> * The returned stream encapsulates one or more \{@link DirectoryStream}s.
> * If timely disposal of file system resources is required, the
> * {@code try}-with-resources construct should be used to ensure that the
> * stream's \{@link Stream#close close} method is invoked after the stream
> * operations are completed. Operating on a closed stream will result in an
> * {@link java.lang.IllegalStateException}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10976) Fix resource leak due to Files.walk
[
https://issues.apache.org/jira/browse/YARN-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10976:
-
Issue Type: Bug (was: Improvement)
Priority: Minor (was: Major)
> Fix resource leak due to Files.walk
> ---
>
> Key: YARN-10976
> URL: https://issues.apache.org/jira/browse/YARN-10976
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Stream creates by File.walk should be closed, like jdk said:
> * The returned stream encapsulates one or more \{@link DirectoryStream}s.
> * If timely disposal of file system resources is required, the
> * {@code try}-with-resources construct should be used to ensure that the
> * stream's \{@link Stream#close close} method is invoked after the stream
> * operations are completed. Operating on a closed stream will result in an
> * {@link java.lang.IllegalStateException}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10976) Fix resource leak due to Files.walk
[
https://issues.apache.org/jira/browse/YARN-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10976:
-
Summary: Fix resource leak due to Files.walk (was: Resource Leak due to
Files.walk)
> Fix resource leak due to Files.walk
> ---
>
> Key: YARN-10976
> URL: https://issues.apache.org/jira/browse/YARN-10976
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: lujie
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Stream creates by File.walk should be closed, like jdk said:
> * The returned stream encapsulates one or more \{@link DirectoryStream}s.
> * If timely disposal of file system resources is required, the
> * {@code try}-with-resources construct should be used to ensure that the
> * stream's \{@link Stream#close close} method is invoked after the stream
> * operations are completed. Operating on a closed stream will result in an
> * {@link java.lang.IllegalStateException}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10976) Fix resource leak due to Files.walk
[
https://issues.apache.org/jira/browse/YARN-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned YARN-10976:
Assignee: lujie
> Fix resource leak due to Files.walk
> ---
>
> Key: YARN-10976
> URL: https://issues.apache.org/jira/browse/YARN-10976
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Stream creates by File.walk should be closed, like jdk said:
> * The returned stream encapsulates one or more \{@link DirectoryStream}s.
> * If timely disposal of file system resources is required, the
> * {@code try}-with-resources construct should be used to ensure that the
> * stream's \{@link Stream#close close} method is invoked after the stream
> * operations are completed. Operating on a closed stream will result in an
> * {@link java.lang.IllegalStateException}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10970) Standby RM should expose prom endpoint
[ https://issues.apache.org/jira/browse/YARN-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10970: - Priority: Major (was: Minor) Summary: Standby RM should expose prom endpoint (was: Prometheussink: Standby RM should expose prom endpoint) > Standby RM should expose prom endpoint > -- > > Key: YARN-10970 > URL: https://issues.apache.org/jira/browse/YARN-10970 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h > Remaining Estimate: 0h > > HADOOP-16398 exports Hadoop metrics to Prometheus. > However, standby RM redirects prom metrics to the Active. We need to > separate out prom metrics displayed so the Standby RM can also be export > metrics to prometheus. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10970) Prometheussink: Standby RM should expose prom endpoint
[ https://issues.apache.org/jira/browse/YARN-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned YARN-10970: Assignee: Max Xie > Prometheussink: Standby RM should expose prom endpoint > -- > > Key: YARN-10970 > URL: https://issues.apache.org/jira/browse/YARN-10970 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h > Remaining Estimate: 0h > > HADOOP-16398 exports Hadoop metrics to Prometheus. > However, standby RM redirects prom metrics to the Active. We need to > separate out prom metrics displayed so the Standby RM can also be export > metrics to prometheus. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10970) Prometheussink: Standby RM should expose prom endpoint
[ https://issues.apache.org/jira/browse/YARN-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved YARN-10970. -- Fix Version/s: 3.3.2 3.4.0 Resolution: Fixed Committed to trunk and branch-3.3. > Prometheussink: Standby RM should expose prom endpoint > -- > > Key: YARN-10970 > URL: https://issues.apache.org/jira/browse/YARN-10970 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Max Xie >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h > Remaining Estimate: 0h > > HADOOP-16398 exports Hadoop metrics to Prometheus. > However, standby RM redirects prom metrics to the Active. We need to > separate out prom metrics displayed so the Standby RM can also be export > metrics to prometheus. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9452) Fix TestDistributedShell and TestTimelineAuthFilterForV2 failures
[
https://issues.apache.org/jira/browse/YARN-9452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17409259#comment-17409259
]
Akira Ajisaka commented on YARN-9452:
-
I saw similar problems in branch-3.2. I'll create a backport PR.
> Fix TestDistributedShell and TestTimelineAuthFilterForV2 failures
> -
>
> Key: YARN-9452
> URL: https://issues.apache.org/jira/browse/YARN-9452
> Project: Hadoop YARN
> Issue Type: Bug
> Components: ATSv2, distributed-shell, test
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9452-001.patch, YARN-9452-002.patch,
> YARN-9452-003.patch, YARN-9452-004.patch
>
>
> *TestDistributedShell#testDSShellWithoutDomainV2CustomizedFlow*
> {code}
> [ERROR]
> testDSShellWithoutDomainV2CustomizedFlow(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 72.14 s <<< FAILURE!
> java.lang.AssertionError: Entity ID prefix should be same across each publish
> of same entity expected:<9223372036854775806> but was:<9223370482298585580>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.verifyEntityForTimelineV2(TestDistributedShell.java:695)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.checkTimelineV2(TestDistributedShell.java:588)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:459)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow(TestDistributedShell.java:330)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> *TestTimelineAuthFilterForV2#testPutTimelineEntities*
> {code}
> [ERROR]
> testPutTimelineEntities[3](org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2)
> Time elapsed: 1.047 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:712)
> at org.junit.Assert.assertNotNull(Assert.java:722)
> at
> org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.verifyEntity(TestTimelineAuthFilterForV2.java:282)
> at
> org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.testPutTimelineEntities(TestTimelineAuthFilterForV2.java:421)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
>
[jira] [Updated] (YARN-10428) Zombie applications in the YARN queue using FAIR + sizebasedweight
[ https://issues.apache.org/jira/browse/YARN-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10428: - Fix Version/s: 2.10.2 Backported to branch-2.10. > Zombie applications in the YARN queue using FAIR + sizebasedweight > -- > > Key: YARN-10428 > URL: https://issues.apache.org/jira/browse/YARN-10428 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.8.5 >Reporter: Guang Yang >Assignee: Andras Gyori >Priority: Critical > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2 > > Attachments: YARN-10428.001.patch, YARN-10428.002.patch, > YARN-10428.003.patch > > > Seeing zombie jobs in the YARN queue that uses FAIR and size based weight > ordering policy . > *Detection:* > The YARN UI shows incorrect number of "Num Schedulable Applications". > *Impact:* > The queue has an upper limit of number of running applications, with zombie > job, it hits the limit even though the number of running applications is far > less than the limit. > *Workaround:* > **Fail-over and restart Resource Manager process. > *Analysis:* > **In the heap dump, we can find the zombie jobs in the `FairOderingPolicy# > schedulableEntities` (see attachment). Take application > "application_1599157165858_29429" for example, it is still in the > `FairOderingPolicy#schedulableEntities` set, however, if we check the log of > resource manager, we can see RM already tried to remove the application: > > ./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 > 04:32:19,730 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Application removed - appId: > application_1599157165858_29429 user: svc_di_data_eng queue: core-data > #user-pending-applications: -3 #user-active-applications: 7 > #queue-pending-applications: 0 #queue-active-applications: 21 > > So it appears RM failed to removed the application from the set. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10428) Zombie applications in the YARN queue using FAIR + sizebasedweight
[ https://issues.apache.org/jira/browse/YARN-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10428: - Fix Version/s: 3.3.2 3.2.3 Backported to branch-3.3, branch-3.2, and branch-3.2.3. > Zombie applications in the YARN queue using FAIR + sizebasedweight > -- > > Key: YARN-10428 > URL: https://issues.apache.org/jira/browse/YARN-10428 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.8.5 >Reporter: Guang Yang >Assignee: Andras Gyori >Priority: Critical > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Attachments: YARN-10428.001.patch, YARN-10428.002.patch, > YARN-10428.003.patch > > > Seeing zombie jobs in the YARN queue that uses FAIR and size based weight > ordering policy . > *Detection:* > The YARN UI shows incorrect number of "Num Schedulable Applications". > *Impact:* > The queue has an upper limit of number of running applications, with zombie > job, it hits the limit even though the number of running applications is far > less than the limit. > *Workaround:* > **Fail-over and restart Resource Manager process. > *Analysis:* > **In the heap dump, we can find the zombie jobs in the `FairOderingPolicy# > schedulableEntities` (see attachment). Take application > "application_1599157165858_29429" for example, it is still in the > `FairOderingPolicy#schedulableEntities` set, however, if we check the log of > resource manager, we can see RM already tried to remove the application: > > ./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 > 04:32:19,730 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Application removed - appId: > application_1599157165858_29429 user: svc_di_data_eng queue: core-data > #user-pending-applications: -3 #user-active-applications: 7 > #queue-pending-applications: 0 #queue-active-applications: 21 > > So it appears RM failed to removed the application from the set. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10428) Zombie applications in the YARN queue using FAIR + sizebasedweight
[ https://issues.apache.org/jira/browse/YARN-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10428: - Priority: Critical (was: Major) > Zombie applications in the YARN queue using FAIR + sizebasedweight > -- > > Key: YARN-10428 > URL: https://issues.apache.org/jira/browse/YARN-10428 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.8.5 >Reporter: Guang Yang >Assignee: Andras Gyori >Priority: Critical > Fix For: 3.4.0 > > Attachments: YARN-10428.001.patch, YARN-10428.002.patch, > YARN-10428.003.patch > > > Seeing zombie jobs in the YARN queue that uses FAIR and size based weight > ordering policy . > *Detection:* > The YARN UI shows incorrect number of "Num Schedulable Applications". > *Impact:* > The queue has an upper limit of number of running applications, with zombie > job, it hits the limit even though the number of running applications is far > less than the limit. > *Workaround:* > **Fail-over and restart Resource Manager process. > *Analysis:* > **In the heap dump, we can find the zombie jobs in the `FairOderingPolicy# > schedulableEntities` (see attachment). Take application > "application_1599157165858_29429" for example, it is still in the > `FairOderingPolicy#schedulableEntities` set, however, if we check the log of > resource manager, we can see RM already tried to remove the application: > > ./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 > 04:32:19,730 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Application removed - appId: > application_1599157165858_29429 user: svc_di_data_eng queue: core-data > #user-pending-applications: -3 #user-active-applications: 7 > #queue-pending-applications: 0 #queue-active-applications: 21 > > So it appears RM failed to removed the application from the set. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10428) Zombie applications in the YARN queue using FAIR + sizebasedweight
[ https://issues.apache.org/jira/browse/YARN-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407745#comment-17407745 ] Akira Ajisaka commented on YARN-10428: -- We recently enabled fair and size based weight ordering policy, and then faced this issue. I'll backport to the release branches. > Zombie applications in the YARN queue using FAIR + sizebasedweight > -- > > Key: YARN-10428 > URL: https://issues.apache.org/jira/browse/YARN-10428 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.8.5 >Reporter: Guang Yang >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10428.001.patch, YARN-10428.002.patch, > YARN-10428.003.patch > > > Seeing zombie jobs in the YARN queue that uses FAIR and size based weight > ordering policy . > *Detection:* > The YARN UI shows incorrect number of "Num Schedulable Applications". > *Impact:* > The queue has an upper limit of number of running applications, with zombie > job, it hits the limit even though the number of running applications is far > less than the limit. > *Workaround:* > **Fail-over and restart Resource Manager process. > *Analysis:* > **In the heap dump, we can find the zombie jobs in the `FairOderingPolicy# > schedulableEntities` (see attachment). Take application > "application_1599157165858_29429" for example, it is still in the > `FairOderingPolicy#schedulableEntities` set, however, if we check the log of > resource manager, we can see RM already tried to remove the application: > > ./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 > 04:32:19,730 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Application removed - appId: > application_1599157165858_29429 user: svc_di_data_eng queue: core-data > #user-pending-applications: -3 #user-active-applications: 7 > #queue-pending-applications: 0 #queue-active-applications: 21 > > So it appears RM failed to removed the application from the set. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10882) Fix branch-3.1 build: zstd library is missing from the Dockerfile
[
https://issues.apache.org/jira/browse/YARN-10882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10882:
-
Fix Version/s: (was: 3.4.0)
> Fix branch-3.1 build: zstd library is missing from the Dockerfile
> -
>
> Key: YARN-10882
> URL: https://issues.apache.org/jira/browse/YARN-10882
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The branch-3.1 did not build on the Jenkins slave, because the zstd is
> missing from the Dockerfile.
>
> {code:java}
> [INFO] --- hadoop-maven-plugins:3.1.5-SNAPSHOT:cmake-compile (cmake-compile)
> @ hadoop-common ---
> [INFO] Running cmake
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3286/src/hadoop-common-project/hadoop-common/src
>
> -DGENERATED_JAVAH=/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3286/src/hadoop-common-project/hadoop-common/target/native/javah
> -DJVM_ARCH_DATA_MODEL=64 -DREQUIRE_BZIP2=false -DREQUIRE_ISAL=false
> -DREQUIRE_OPENSSL=true -DREQUIRE_SNAPPY=true -DREQUIRE_ZSTD=true -G Unix
> Makefiles
> [INFO] with extra environment variables {}
> [WARNING] -- The C compiler identification is GNU 7.5.0
> [WARNING] -- The CXX compiler identification is GNU 7.5.0
> [WARNING] -- Check for working C compiler: /usr/bin/cc
> [WARNING] -- Check for working C compiler: /usr/bin/cc -- works
> [WARNING] -- Detecting C compiler ABI info
> [WARNING] -- Detecting C compiler ABI info - done
> [WARNING] -- Detecting C compile features
> [WARNING] -- Detecting C compile features - done
> [WARNING] -- Check for working CXX compiler: /usr/bin/c++
> [WARNING] -- Check for working CXX compiler: /usr/bin/c++ -- works
> [WARNING] -- Detecting CXX compiler ABI info
> [WARNING] -- Detecting CXX compiler ABI info - done
> [WARNING] -- Detecting CXX compile features
> [WARNING] -- Detecting CXX compile features - done
> [WARNING] -- Looking for pthread.h
> [WARNING] -- Looking for pthread.h - found
> [WARNING] -- Looking for pthread_create
> [WARNING] -- Looking for pthread_create - not found
> [WARNING] -- Looking for pthread_create in pthreads
> [WARNING] -- Looking for pthread_create in pthreads - not found
> [WARNING] -- Looking for pthread_create in pthread
> [WARNING] -- Looking for pthread_create in pthread - found
> [WARNING] -- Found Threads: TRUE
> [WARNING] JAVA_HOME=,
> JAVA_JVM_LIBRARY=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> [WARNING] JAVA_INCLUDE_PATH=/usr/lib/jvm/java-8-openjdk-amd64/include,
> JAVA_INCLUDE_PATH2=/usr/lib/jvm/java-8-openjdk-amd64/include/linux
> [WARNING] Located all JNI components successfully.
> [WARNING] -- Found JNI:
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjawt.so
> [WARNING] -- Found ZLIB: /lib/x86_64-linux-gnu/libz.so.1 (found version
> "1.2.11")
> [WARNING] -- Found Snappy: /usr/lib/x86_64-linux-gnu/libsnappy.so.1
> [WARNING] CMake Error at CMakeLists.txt:120 (MESSAGE):
> [WARNING] Required zstandard library could not be found.
> [WARNING] ZSTD_LIBRARY=/usr/lib/x86_64-linux-gnu/libzstd.so.1,
> ZSTD_INCLUDE_DIR=,
> [WARNING] CUSTOM_ZSTD_INCLUDE_DIR=, CUSTOM_ZSTD_PREFIX=,
> CUSTOM_ZSTD_INCLUDE= {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10882) Fix branch-3.1 build: zstd library is missing from the Dockerfile
[
https://issues.apache.org/jira/browse/YARN-10882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397811#comment-17397811
]
Akira Ajisaka commented on YARN-10882:
--
Removed the fix version because branch-3.1 is EoL.
> Fix branch-3.1 build: zstd library is missing from the Dockerfile
> -
>
> Key: YARN-10882
> URL: https://issues.apache.org/jira/browse/YARN-10882
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The branch-3.1 did not build on the Jenkins slave, because the zstd is
> missing from the Dockerfile.
>
> {code:java}
> [INFO] --- hadoop-maven-plugins:3.1.5-SNAPSHOT:cmake-compile (cmake-compile)
> @ hadoop-common ---
> [INFO] Running cmake
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3286/src/hadoop-common-project/hadoop-common/src
>
> -DGENERATED_JAVAH=/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3286/src/hadoop-common-project/hadoop-common/target/native/javah
> -DJVM_ARCH_DATA_MODEL=64 -DREQUIRE_BZIP2=false -DREQUIRE_ISAL=false
> -DREQUIRE_OPENSSL=true -DREQUIRE_SNAPPY=true -DREQUIRE_ZSTD=true -G Unix
> Makefiles
> [INFO] with extra environment variables {}
> [WARNING] -- The C compiler identification is GNU 7.5.0
> [WARNING] -- The CXX compiler identification is GNU 7.5.0
> [WARNING] -- Check for working C compiler: /usr/bin/cc
> [WARNING] -- Check for working C compiler: /usr/bin/cc -- works
> [WARNING] -- Detecting C compiler ABI info
> [WARNING] -- Detecting C compiler ABI info - done
> [WARNING] -- Detecting C compile features
> [WARNING] -- Detecting C compile features - done
> [WARNING] -- Check for working CXX compiler: /usr/bin/c++
> [WARNING] -- Check for working CXX compiler: /usr/bin/c++ -- works
> [WARNING] -- Detecting CXX compiler ABI info
> [WARNING] -- Detecting CXX compiler ABI info - done
> [WARNING] -- Detecting CXX compile features
> [WARNING] -- Detecting CXX compile features - done
> [WARNING] -- Looking for pthread.h
> [WARNING] -- Looking for pthread.h - found
> [WARNING] -- Looking for pthread_create
> [WARNING] -- Looking for pthread_create - not found
> [WARNING] -- Looking for pthread_create in pthreads
> [WARNING] -- Looking for pthread_create in pthreads - not found
> [WARNING] -- Looking for pthread_create in pthread
> [WARNING] -- Looking for pthread_create in pthread - found
> [WARNING] -- Found Threads: TRUE
> [WARNING] JAVA_HOME=,
> JAVA_JVM_LIBRARY=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> [WARNING] JAVA_INCLUDE_PATH=/usr/lib/jvm/java-8-openjdk-amd64/include,
> JAVA_INCLUDE_PATH2=/usr/lib/jvm/java-8-openjdk-amd64/include/linux
> [WARNING] Located all JNI components successfully.
> [WARNING] -- Found JNI:
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjawt.so
> [WARNING] -- Found ZLIB: /lib/x86_64-linux-gnu/libz.so.1 (found version
> "1.2.11")
> [WARNING] -- Found Snappy: /usr/lib/x86_64-linux-gnu/libsnappy.so.1
> [WARNING] CMake Error at CMakeLists.txt:120 (MESSAGE):
> [WARNING] Required zstandard library could not be found.
> [WARNING] ZSTD_LIBRARY=/usr/lib/x86_64-linux-gnu/libzstd.so.1,
> ZSTD_INCLUDE_DIR=,
> [WARNING] CUSTOM_ZSTD_INCLUDE_DIR=, CUSTOM_ZSTD_PREFIX=,
> CUSTOM_ZSTD_INCLUDE= {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10459) containerLaunchedOnNode method not need to hold schedulerApptemt lock
[
https://issues.apache.org/jira/browse/YARN-10459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10459:
-
Fix Version/s: (was: 3.2.1)
> containerLaunchedOnNode method not need to hold schedulerApptemt lock
> --
>
> Key: YARN-10459
> URL: https://issues.apache.org/jira/browse/YARN-10459
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.2.0, 3.1.3
>Reporter: Ryan Wu
>Assignee: Minni Mittal
>Priority: Major
> Labels: pull-request-available
> Attachments: YARN-10459.v1.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
>
> Now, the containerLaunchedOnNode method hold the SchedulerApplicationAttempt
> writelock, but looking at the method, it does not change any field. And more
> seriously, this will affect the scheduler.
> {code:java}
> public void containerLaunchedOnNode(ContainerId containerId, NodeId nodeId)
> {
> // Inform the container
> writelock.lock
> try {
> RMContainer rmContainer = getRMContainer(containerId);
> if (rmContainer == null) {
> // Some unknown container sneaked into the system. Kill it.
> rmContext.getDispatcher().getEventHandler().handle( new
> RMNodeCleanContainerEvent(nodeId, containerId)); return;
> }
> rmContainer.handle( new RMContainerEvent(containerId,
> RMContainerEventType.LAUNCHED));
>}finally {
> writeLock.unlock();
>}
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10561) Upgrade node.js to at least 12.x in YARN application catalog webapp
[ https://issues.apache.org/jira/browse/YARN-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10561: - Summary: Upgrade node.js to at least 12.x in YARN application catalog webapp (was: Upgrade node.js to at least 10.x in YARN application catalog webapp) > Upgrade node.js to at least 12.x in YARN application catalog webapp > --- > > Key: YARN-10561 > URL: https://issues.apache.org/jira/browse/YARN-10561 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > YARN application catalog webapp is using node.js 8.11.3, and 8.x are already > EoL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10809) testWithHbaseConfAtHdfsFileSystem consistently failing
[
https://issues.apache.org/jira/browse/YARN-10809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10809:
-
Fix Version/s: 3.2.3
Target Version/s: 3.4.0, 3.3.2 (was: 3.4.0, 3.2.3, 3.3.2)
> testWithHbaseConfAtHdfsFileSystem consistently failing
> --
>
> Key: YARN-10809
> URL: https://issues.apache.org/jira/browse/YARN-10809
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> TestHBaseTimelineStorageUtils#testWithHbaseConfAtHdfsFileSystem is
> consistently failing with
> {code:java}
> java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2618)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2632)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1498)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:977)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:576)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:518)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils.testWithHbaseConfAtHdfsFileSystem(TestHBaseTimelineStorageUtils.java:86)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10809) testWithHbaseConfAtHdfsFileSystem consistently failing
[
https://issues.apache.org/jira/browse/YARN-10809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10809:
-
Target Version/s: 3.4.0, 3.2.3, 3.3.2 (was: 3.4.0, 3.3.2)
Backported to branch-3.2.
> testWithHbaseConfAtHdfsFileSystem consistently failing
> --
>
> Key: YARN-10809
> URL: https://issues.apache.org/jira/browse/YARN-10809
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> TestHBaseTimelineStorageUtils#testWithHbaseConfAtHdfsFileSystem is
> consistently failing with
> {code:java}
> java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2618)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2632)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1498)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:977)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:576)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:518)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils.testWithHbaseConfAtHdfsFileSystem(TestHBaseTimelineStorageUtils.java:86)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9990) Testcase fails with "Insufficient configured threads: required=16 < max=10"
[
https://issues.apache.org/jira/browse/YARN-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-9990:
Fix Version/s: 3.2.3
Backported to branch-3.2.
> Testcase fails with "Insufficient configured threads: required=16 < max=10"
> ---
>
> Key: YARN-9990
> URL: https://issues.apache.org/jira/browse/YARN-9990
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0, 3.2.3
>
> Attachments: YARN-9990-001.patch
>
>
> Testcase fails with "Insufficient configured threads: required=16 < max=10".
> Below testcases failing
> 1. TestWebAppProxyServlet
> 2. TestAmFilter
> 3. TestApiServiceClient
> 4. TestSecureApiServiceClient
> {code}
> [ERROR] org.apache.hadoop.yarn.server.webproxy.TestWebAppProxyServlet Time
> elapsed: 0.396 s <<< ERROR!
> java.lang.IllegalStateException: Insufficient configured threads: required=16
> < max=10 for
> QueuedThreadPool[qtp1597249648]@5f341870{STARTED,8<=8<=10,i=8,r=1,q=0}[ReservedThreadExecutor@4c762604{s=0/1,p=0}]
> at
> org.eclipse.jetty.util.thread.ThreadPoolBudget.check(ThreadPoolBudget.java:156)
> at
> org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseTo(ThreadPoolBudget.java:130)
> at
> org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseFrom(ThreadPoolBudget.java:182)
> at
> org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:255)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at
> org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:283)
> at
> org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
> at
> org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at org.eclipse.jetty.server.Server.doStart(Server.java:385)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at
> org.apache.hadoop.yarn.server.webproxy.TestWebAppProxyServlet.start(TestWebAppProxyServlet.java:102)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> [INFO] Running org.apache.hadoop.yarn.server.webproxy.amfilter.TestAmFilter
> [ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.326
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.webproxy.amfilter.TestAmFilter
> [ERROR]
> testFindRedirectUrl(org.apache.hadoop.yarn.server.webproxy.amfilter.TestAmFilter)
> Time elapsed: 0.306 s <<< ERROR!
> java.lang.IllegalStateException: Insufficient configured threads: required=16
> < max=10 for
> QueuedThreadPool[qtp485041780]@1ce92674{STARTED,8<=8<=10,i=8,r=1,q=0}[ReservedThreadExecutor@31f924f5{s=0/1,p=0}]
> at
>
[jira] [Updated] (YARN-10337) TestRMHATimelineCollectors fails on hadoop trunk
[
https://issues.apache.org/jira/browse/YARN-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10337:
-
Fix Version/s: 3.2.3
Backported to branch-3.2.
> TestRMHATimelineCollectors fails on hadoop trunk
>
>
> Key: YARN-10337
> URL: https://issues.apache.org/jira/browse/YARN-10337
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test, yarn
>Reporter: Ahmed Hussein
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
> Attachments: YARN-10337.001.patch
>
>
> {{TestRMHATimelineCollectors}} has been failing on trunk. I see it frequently
> in the qbt reports and the yetus reprts
> {code:bash}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.95
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR]
> testRebuildCollectorDataOnFailover(org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors)
> Time elapsed: 5.615 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover(TestRMHATimelineCollectors.java:105)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover:105
> NullPointer
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR] There are test failures.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10337) TestRMHATimelineCollectors fails on hadoop trunk
[
https://issues.apache.org/jira/browse/YARN-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10337:
-
Fix Version/s: 3.3.2
Already backported to branch-3.3. Updated the fix versions.
> TestRMHATimelineCollectors fails on hadoop trunk
>
>
> Key: YARN-10337
> URL: https://issues.apache.org/jira/browse/YARN-10337
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test, yarn
>Reporter: Ahmed Hussein
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0, 3.3.2
>
> Attachments: YARN-10337.001.patch
>
>
> {{TestRMHATimelineCollectors}} has been failing on trunk. I see it frequently
> in the qbt reports and the yetus reprts
> {code:bash}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.95
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR]
> testRebuildCollectorDataOnFailover(org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors)
> Time elapsed: 5.615 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover(TestRMHATimelineCollectors.java:105)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover:105
> NullPointer
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR] There are test failures.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9875) FSSchedulerConfigurationStore fails to update with hdfs path
[
https://issues.apache.org/jira/browse/YARN-9875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-9875:
Fix Version/s: 3.2.3
Backported to branch-3.2.
> FSSchedulerConfigurationStore fails to update with hdfs path
>
>
> Key: YARN-9875
> URL: https://issues.apache.org/jira/browse/YARN-9875
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.0, 3.2.3
>
> Attachments: YARN-9875-001.patch, YARN-9875-002.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> FSSchedulerConfigurationStore fails to update with hdfs path -
> "java.io.IOException: Filesystem closed"
> *RM Logs:*
> {code}
> 2019-10-06 16:50:40,829 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.FSSchedulerConfigurationStore:
> write temp capacity configuration fail,
> schedulerConfigFile=hdfs:/tmp/yarn/system/capacity-scheduler.xml.1570380640828.tmp
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:475)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1232)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1214)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1196)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1134)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:530)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:527)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:541)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:468)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1136)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1116)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1005)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:993)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.FSSchedulerConfigurationStore.writeTmpConfig(FSSchedulerConfigurationStore.java:251)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.FSSchedulerConfigurationStore.logMutation(FSSchedulerConfigurationStore.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.MutableCSConfigurationProvider.logAndApplyMutation(MutableCSConfigurationProvider.java:153)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2597)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2587)
> {code}
> *Repro:*
> {code:java}
> yarn-site.xml:
>
> yarn.scheduler.configuration.fs.path
> hdfs:///tmp/yarn/system
>
>
> yarn.scheduler.configuration.store.class
> fs
>
> [yarn@yarndocker-1 yarn]$ cat /tmp/abc.xml
>
>
> root.default
>
>
> priority
> 10
>
>
>
>
> [yarn@yarndocker-1 yarn]$ curl -v -X PUT -d @/tmp/abc.xml -H "Content-type:
> application/xml"
> 'http://yarndocker-1:8088/ws/v1/cluster/scheduler-conf?user.name=yarn'
> Filesystem closed
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8992) Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue
[
https://issues.apache.org/jira/browse/YARN-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved YARN-8992.
-
Fix Version/s: 3.2.3
Resolution: Fixed
Backported to branch-3.2.
> Fair scheduler can delete a dynamic queue while an application attempt is
> being added to the queue
> --
>
> Key: YARN-8992
> URL: https://issues.apache.org/jira/browse/YARN-8992
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.1.1
>Reporter: Haibo Chen
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Labels: pull-request-available, release-blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: YARN-8992.001.patch, YARN-8992.002.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> As discovered in YARN-8990, QueueManager can see a leaf queue being empty
> while FSLeafQueue.addApp() is called in the middle of
> {code:java}
> return queue.getNumRunnableApps() == 0 &&
> leafQueue.getNumNonRunnableApps() == 0 &&
> leafQueue.getNumAssignedApps() == 0;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8990) Fix fair scheduler race condition in app submit and queue cleanup
[
https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved YARN-8990.
-
Fix Version/s: 3.2.3
Resolution: Fixed
Backported to branch-3.2.
> Fix fair scheduler race condition in app submit and queue cleanup
> -
>
> Key: YARN-8990
> URL: https://issues.apache.org/jira/browse/YARN-8990
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.2.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Blocker
> Labels: pull-request-available, release-blocker
> Fix For: 3.2.3, 3.3.0, 3.2.0
>
> Attachments: YARN-8990.001.patch, YARN-8990.002.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> With the introduction of the dynamic queue deletion in YARN-8191 a race
> condition was introduced that can cause a queue to be removed while an
> application submit is in progress.
> The issue occurs in {{FairScheduler.addApplication()}} when an application is
> submitted to a dynamic queue which is empty or the queue does not exist yet.
> If during the processing of the application submit the
> {{AllocationFileLoaderService}} kicks of for an update the queue clean up
> will be run first. The application submit first creates the queue and get a
> reference back to the queue.
> Other checks are performed and as the last action before getting ready to
> generate an AppAttempt the queue is updated to show the submitted application
> ID..
> The time between the queue creation and the queue update to show the submit
> is long enough for the queue to be removed. The application however is lost
> and will never get any resources assigned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9338) Timeline related testcases are failing
[
https://issues.apache.org/jira/browse/YARN-9338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-9338:
Fix Version/s: 3.2.3
Backported to branch-3.2.
> Timeline related testcases are failing
> --
>
> Key: YARN-9338
> URL: https://issues.apache.org/jira/browse/YARN-9338
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Prabhu Joseph
>Assignee: Abhishek Modi
>Priority: Major
> Fix For: 3.3.0, 3.2.3
>
> Attachments: YARN-9338.001.patch, YARN-9338.002.patch,
> YARN-9338.003.patch, YARN-9338.004.patch
>
>
> Timeline related testcases are failing
> {code}
> [ERROR] Failures:
> [ERROR]
> TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV2Enabled:262->runTest:245->validateV2:382->verifyEntity:417
> Expected 2 events to be published expected:<2> but was:<1>
> [ERROR]
> TestSystemMetricsPublisherForV2.testPublishAppAttemptMetrics:259->verifyEntity:332
> Expected 2 events to be published expected:<2> but was:<1>
> [ERROR]
> TestSystemMetricsPublisherForV2.testPublishApplicationMetrics:224->verifyEntity:332
> Expected 4 events to be published expected:<4> but was:<1>
> [ERROR]
> TestSystemMetricsPublisherForV2.testPublishContainerMetrics:291->verifyEntity:332
> Expected 2 events to be published expected:<2> but was:<1>
> [ERROR] Errors:
> [ERROR]
> TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled:252->runTest:242->testSetup:123
> » YarnRuntime
> [ERROR] Failures:
> [ERROR]
> TestTimelineAuthFilterForV2.testPutTimelineEntities:343->access$000:87->publishAndVerifyEntity:307
> expected:<1> but was:<2>
> [ERROR]
> TestTimelineAuthFilterForV2.testPutTimelineEntities:352->publishWithRetries:320->publishAndVerifyEntity:307
> expected:<1> but was:<2>
> [ERROR]
> TestTimelineAuthFilterForV2.testPutTimelineEntities:352->publishWithRetries:320->publishAndVerifyEntity:307
> expected:<1> but was:<2>
> [ERROR]
> TestTimelineAuthFilterForV2.testPutTimelineEntities:343->access$000:87->publishAndVerifyEntity:307
> expected:<1> but was:<2>
> [INFO]
> [ERROR] Failures:
> [ERROR]
> TestDistributedShell.testDSShellWithoutDomainV2:313->testDSShell:317->testDSShell:458->checkTimelineV2:557->verifyEntityForTimelineV2:710
> Unexpected number of DS_APP_ATTEMPT_START event published. expected:<1> but
> was:<0>
> [ERROR]
> TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow:329->testDSShell:458->checkTimelineV2:557->verifyEntityForTimelineV2:710
> Unexpected number of DS_APP_ATTEMPT_START event published. expected:<1> but
> was:<0>
> [ERROR]
> TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow:323->testDSShell:458->checkTimelineV2:557->verifyEntityForTimelineV2:710
> Unexpected number of DS_APP_ATTEMPT_START event published. expected:<1> but
> was:<0>
> [ERROR] Failures:
> [ERROR]
> TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling:240->checkNewTimelineEvent:304->verifyEntity:462
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8992) Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue
[
https://issues.apache.org/jira/browse/YARN-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-8992:
Target Version/s: 3.2.3
Labels: release-blocker (was: )
> Fair scheduler can delete a dynamic queue while an application attempt is
> being added to the queue
> --
>
> Key: YARN-8992
> URL: https://issues.apache.org/jira/browse/YARN-8992
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.1.1
>Reporter: Haibo Chen
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Labels: release-blocker
> Fix For: 3.3.0
>
> Attachments: YARN-8992.001.patch, YARN-8992.002.patch
>
>
> As discovered in YARN-8990, QueueManager can see a leaf queue being empty
> while FSLeafQueue.addApp() is called in the middle of
> {code:java}
> return queue.getNumRunnableApps() == 0 &&
> leafQueue.getNumNonRunnableApps() == 0 &&
> leafQueue.getNumAssignedApps() == 0;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-8992) Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue
[
https://issues.apache.org/jira/browse/YARN-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reopened YARN-8992:
-
Reopened this and marked as release-blocker.
> Fair scheduler can delete a dynamic queue while an application attempt is
> being added to the queue
> --
>
> Key: YARN-8992
> URL: https://issues.apache.org/jira/browse/YARN-8992
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.1.1
>Reporter: Haibo Chen
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Labels: release-blocker
> Fix For: 3.3.0
>
> Attachments: YARN-8992.001.patch, YARN-8992.002.patch
>
>
> As discovered in YARN-8990, QueueManager can see a leaf queue being empty
> while FSLeafQueue.addApp() is called in the middle of
> {code:java}
> return queue.getNumRunnableApps() == 0 &&
> leafQueue.getNumNonRunnableApps() == 0 &&
> leafQueue.getNumAssignedApps() == 0;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-8990) Fix fair scheduler race condition in app submit and queue cleanup
[
https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reopened YARN-8990:
-
> Fix fair scheduler race condition in app submit and queue cleanup
> -
>
> Key: YARN-8990
> URL: https://issues.apache.org/jira/browse/YARN-8990
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.2.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Blocker
> Labels: pull-request-available, release-blocker
> Fix For: 3.3.0
>
> Attachments: YARN-8990.001.patch, YARN-8990.002.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With the introduction of the dynamic queue deletion in YARN-8191 a race
> condition was introduced that can cause a queue to be removed while an
> application submit is in progress.
> The issue occurs in {{FairScheduler.addApplication()}} when an application is
> submitted to a dynamic queue which is empty or the queue does not exist yet.
> If during the processing of the application submit the
> {{AllocationFileLoaderService}} kicks of for an update the queue clean up
> will be run first. The application submit first creates the queue and get a
> reference back to the queue.
> Other checks are performed and as the last action before getting ready to
> generate an AppAttempt the queue is updated to show the submitted application
> ID..
> The time between the queue creation and the queue update to show the submit
> is long enough for the queue to be removed. The application however is lost
> and will never get any resources assigned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8990) Fix fair scheduler race condition in app submit and queue cleanup
[
https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-8990:
Fix Version/s: 3.2.0
> Fix fair scheduler race condition in app submit and queue cleanup
> -
>
> Key: YARN-8990
> URL: https://issues.apache.org/jira/browse/YARN-8990
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.2.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Blocker
> Labels: pull-request-available, release-blocker
> Fix For: 3.2.0, 3.3.0
>
> Attachments: YARN-8990.001.patch, YARN-8990.002.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With the introduction of the dynamic queue deletion in YARN-8191 a race
> condition was introduced that can cause a queue to be removed while an
> application submit is in progress.
> The issue occurs in {{FairScheduler.addApplication()}} when an application is
> submitted to a dynamic queue which is empty or the queue does not exist yet.
> If during the processing of the application submit the
> {{AllocationFileLoaderService}} kicks of for an update the queue clean up
> will be run first. The application submit first creates the queue and get a
> reference back to the queue.
> Other checks are performed and as the last action before getting ready to
> generate an AppAttempt the queue is updated to show the submitted application
> ID..
> The time between the queue creation and the queue update to show the submit
> is long enough for the queue to be removed. The application however is lost
> and will never get any resources assigned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8990) Fix fair scheduler race condition in app submit and queue cleanup
[
https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-8990:
Fix Version/s: (was: 3.2.0)
Target Version/s: 3.2.3
Labels: pull-request-available release-blocker (was:
pull-request-available)
> Fix fair scheduler race condition in app submit and queue cleanup
> -
>
> Key: YARN-8990
> URL: https://issues.apache.org/jira/browse/YARN-8990
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.2.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Blocker
> Labels: pull-request-available, release-blocker
> Fix For: 3.3.0
>
> Attachments: YARN-8990.001.patch, YARN-8990.002.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With the introduction of the dynamic queue deletion in YARN-8191 a race
> condition was introduced that can cause a queue to be removed while an
> application submit is in progress.
> The issue occurs in {{FairScheduler.addApplication()}} when an application is
> submitted to a dynamic queue which is empty or the queue does not exist yet.
> If during the processing of the application submit the
> {{AllocationFileLoaderService}} kicks of for an update the queue clean up
> will be run first. The application submit first creates the queue and get a
> reference back to the queue.
> Other checks are performed and as the last action before getting ready to
> generate an AppAttempt the queue is updated to show the submitted application
> ID..
> The time between the queue creation and the queue update to show the submit
> is long enough for the queue to be removed. The application however is lost
> and will never get any resources assigned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8990) Fix fair scheduler race condition in app submit and queue cleanup
[
https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391388#comment-17391388
]
Akira Ajisaka commented on YARN-8990:
-
Backport PR opened: https://github.com/apache/hadoop/pull/3254
> Fix fair scheduler race condition in app submit and queue cleanup
> -
>
> Key: YARN-8990
> URL: https://issues.apache.org/jira/browse/YARN-8990
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.2.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.2.0, 3.3.0
>
> Attachments: YARN-8990.001.patch, YARN-8990.002.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With the introduction of the dynamic queue deletion in YARN-8191 a race
> condition was introduced that can cause a queue to be removed while an
> application submit is in progress.
> The issue occurs in {{FairScheduler.addApplication()}} when an application is
> submitted to a dynamic queue which is empty or the queue does not exist yet.
> If during the processing of the application submit the
> {{AllocationFileLoaderService}} kicks of for an update the queue clean up
> will be run first. The application submit first creates the queue and get a
> reference back to the queue.
> Other checks are performed and as the last action before getting ready to
> generate an AppAttempt the queue is updated to show the submitted application
> ID..
> The time between the queue creation and the queue update to show the submit
> is long enough for the queue to be removed. The application however is lost
> and will never get any resources assigned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8992) Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue
[
https://issues.apache.org/jira/browse/YARN-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-8992:
Fix Version/s: (was: 3.2.1)
3.3.0
> Fair scheduler can delete a dynamic queue while an application attempt is
> being added to the queue
> --
>
> Key: YARN-8992
> URL: https://issues.apache.org/jira/browse/YARN-8992
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.1.1
>Reporter: Haibo Chen
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-8992.001.patch, YARN-8992.002.patch
>
>
> As discovered in YARN-8990, QueueManager can see a leaf queue being empty
> while FSLeafQueue.addApp() is called in the middle of
> {code:java}
> return queue.getNumRunnableApps() == 0 &&
> leafQueue.getNumNonRunnableApps() == 0 &&
> leafQueue.getNumAssignedApps() == 0;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10858) [UI2] YARN-10826 breaks Queue view
[ https://issues.apache.org/jira/browse/YARN-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10858: - Fix Version/s: 3.3.2 3.2.3 Since YARN-10826 is committed to branch-3.3 and branch-3.2, backported it to the branches. > [UI2] YARN-10826 breaks Queue view > -- > > Key: YARN-10858 > URL: https://issues.apache.org/jira/browse/YARN-10858 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-ui-v2 >Reporter: Andras Gyori >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Attachments: Screenshot 2021-07-19 at 11.40.57.png > > Time Spent: 40m > Remaining Estimate: 0h > > With YARN-10826, UIv2 was upgraded to EmberJS 2.11.3. However, the Queues tab > is broken and loads an empty page. After reverting the commit, the page is > working as intended. > We need to investigate what causes this issue, and how we could mitigate it > without reverting the commit back. > cc. > [~iwasakims] [~aajisaka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10847) Build fails on MacOS 10.15
[ https://issues.apache.org/jira/browse/YARN-10847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved YARN-10847. -- Resolution: Invalid Please do not change the package name when building Apache Hadoop. > Build fails on MacOS 10.15 > -- > > Key: YARN-10847 > URL: https://issues.apache.org/jira/browse/YARN-10847 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: Abhinav Kumar >Priority: Major > > [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (depcheck) @ > hadoop-yarn-server-timelineservice-hbase-common --- > [WARNING] > Dependency convergence error for commons-lang:commons-lang:2.6 paths to > dependency are: > +-org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0 > +-org.apache.hbase:hbase-common:1.2.6 > +-commons-lang:commons-lang:2.6 > and > +-org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0 > +-org.apache.hbase:hbase-common:1.2.6 > +-org.apache.hadoop:hadoop-common:2.5.1 > +-commons-lang:commons-lang:2.6 > and > +-org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0 > +-org.apache.hbase:hbase-common:1.2.6 > +-org.apache.hadoop:hadoop-common:2.5.1 > +-commons-configuration:commons-configuration:1.6 > +-commons-lang:commons-lang:2.4 > and > +-org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0 > +-org.apache.hbase:hbase-common:1.2.6 > +-org.apache.hadoop:hadoop-mapreduce-client-core:2.5.1 > +-org.apache.hadoop:hadoop-yarn-common:2.5.1 > +-org.apache.hadoop:hadoop-yarn-api:2.5.1 > +-commons-lang:commons-lang:2.6 > and > +-org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0 > +-org.apache.hbase:hbase-common:1.2.6 > +-org.apache.hadoop:hadoop-mapreduce-client-core:2.5.1 > +-org.apache.hadoop:hadoop-yarn-common:2.5.1 > +-commons-lang:commons-lang:2.6 > [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence > failed with message: > Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with > message: > Failed while enforcing releasability. See above detailed error message. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10847) Build fails on MacOS 10.15
[
https://issues.apache.org/jira/browse/YARN-10847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376156#comment-17376156
]
Akira Ajisaka commented on YARN-10847:
--
{quote}org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0{quote}
The package name does not start with "org.apache.hadoop", so I think you are
not building Apache Hadoop.
> Build fails on MacOS 10.15
> --
>
> Key: YARN-10847
> URL: https://issues.apache.org/jira/browse/YARN-10847
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Abhinav Kumar
>Priority: Major
>
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (depcheck) @
> hadoop-yarn-server-timelineservice-hbase-common ---
> [WARNING]
> Dependency convergence error for commons-lang:commons-lang:2.6 paths to
> dependency are:
> +-org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0
> +-org.apache.hbase:hbase-common:1.2.6
> +-commons-lang:commons-lang:2.6
> and
> +-org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0
> +-org.apache.hbase:hbase-common:1.2.6
> +-org.apache.hadoop:hadoop-common:2.5.1
> +-commons-lang:commons-lang:2.6
> and
> +-org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0
> +-org.apache.hbase:hbase-common:1.2.6
> +-org.apache.hadoop:hadoop-common:2.5.1
> +-commons-configuration:commons-configuration:1.6
> +-commons-lang:commons-lang:2.4
> and
> +-org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0
> +-org.apache.hbase:hbase-common:1.2.6
> +-org.apache.hadoop:hadoop-mapreduce-client-core:2.5.1
> +-org.apache.hadoop:hadoop-yarn-common:2.5.1
> +-org.apache.hadoop:hadoop-yarn-api:2.5.1
> +-commons-lang:commons-lang:2.6
> and
> +-org.uscbrft.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:3.2.0
> +-org.apache.hbase:hbase-common:1.2.6
> +-org.apache.hadoop:hadoop-mapreduce-client-core:2.5.1
> +-org.apache.hadoop:hadoop-yarn-common:2.5.1
> +-commons-lang:commons-lang:2.6
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
> failed with message:
> Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with
> message:
> Failed while enforcing releasability. See above detailed error message.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10826) [UI2] Upgrade Node.js to at least v12.22.1
[ https://issues.apache.org/jira/browse/YARN-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368610#comment-17368610 ] Akira Ajisaka commented on YARN-10826: -- Cherry-picked the above 4 issues. > [UI2] Upgrade Node.js to at least v12.22.1 > -- > > Key: YARN-10826 > URL: https://issues.apache.org/jira/browse/YARN-10826 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Node.js 10.x is EoL. We have to upgrade to at least 12.x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10560) Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2
[ https://issues.apache.org/jira/browse/YARN-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10560: - Fix Version/s: 3.2.3 Cherry-picked to branch-3.2. > Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2 > -- > > Key: YARN-10560 > URL: https://issues.apache.org/jira/browse/YARN-10560 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp, yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Time Spent: 50m > Remaining Estimate: 0h > > Node 10.x before 10.23.1 are vulnerable: > https://groups.google.com/g/nodejs-sec/c/kyzmwvQdUfs/m/7mjPCzY2BAAJ?pli=1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10331) Upgrade node.js to 10.21.0
[ https://issues.apache.org/jira/browse/YARN-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10331: - Fix Version/s: 3.2.3 Target Version/s: 3.3.1, 3.4.0 (was: 3.4.0, 3.3.1) Cherry-picked to branch-3.2. > Upgrade node.js to 10.21.0 > -- > > Key: YARN-10331 > URL: https://issues.apache.org/jira/browse/YARN-10331 > Project: Hadoop YARN > Issue Type: Bug > Components: build, yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Fix For: 3.4.0, 3.3.1, 3.2.3 > > > YARN-10036 upgraded Node.js to 8.17.0, but Node.js 8.x is already EoL. > https://nodejs.org/en/about/releases/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10037) Upgrade build tools for YARN Web UI v2
[ https://issues.apache.org/jira/browse/YARN-10037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10037: - Fix Version/s: 3.2.3 Cherry-picked to branch-3.2. > Upgrade build tools for YARN Web UI v2 > -- > > Key: YARN-10037 > URL: https://issues.apache.org/jira/browse/YARN-10037 > Project: Hadoop YARN > Issue Type: Bug > Components: build, security, yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.3.0, 3.2.3 > > Attachments: YARN-10037.001.patch > > > The versions of the build tools are too old and have some vulnerabilities. > Update. > * node: 5.12.0 (latest: 12.13.1 LTS) > * yarn: 0.21.3 (latest: 1.21.1) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10020) Fix build instruction of hadoop-yarn-ui
[ https://issues.apache.org/jira/browse/YARN-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10020: - Fix Version/s: 3.2.3 Cherry-picked to branch-3.2. > Fix build instruction of hadoop-yarn-ui > --- > > Key: YARN-10020 > URL: https://issues.apache.org/jira/browse/YARN-10020 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 3.3.0, 3.2.3 > > > We don't need to manually install package managers such as yarn and bower as > described in README.md since frontend-maven-plugin was introduced by > YARN-6278. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10826) [UI2] Upgrade Node.js to at least v12.22.1
[ https://issues.apache.org/jira/browse/YARN-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368604#comment-17368604 ] Akira Ajisaka commented on YARN-10826: -- Thank you. I think we need to cherry-pick YARN-10020 and YARN-10037 as well. > [UI2] Upgrade Node.js to at least v12.22.1 > -- > > Key: YARN-10826 > URL: https://issues.apache.org/jira/browse/YARN-10826 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Node.js 10.x is EoL. We have to upgrade to at least 12.x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10826) [UI2] Upgrade Node.js to at least 12.x
Akira Ajisaka created YARN-10826: Summary: [UI2] Upgrade Node.js to at least 12.x Key: YARN-10826 URL: https://issues.apache.org/jira/browse/YARN-10826 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Akira Ajisaka Node.js 10.x is EoL. We have to upgrade to at least 12.x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10817) Avoid java.lang.NullPointerException when use yarn shell
[
https://issues.apache.org/jira/browse/YARN-10817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360868#comment-17360868
]
Akira Ajisaka commented on YARN-10817:
--
Thank you for your report, [~kaifeiYi].
> Avoid java.lang.NullPointerException when use yarn shell
>
>
> Key: YARN-10817
> URL: https://issues.apache.org/jira/browse/YARN-10817
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.2.1
>Reporter: yikf
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When we use yarn queue-status, we encounter a null pointer exception, As
> follow:
> Like this, and：
> yarn cluster
> yarn queue
> yarn node
> {code:java}
> Missing argument for optionsMissing argument for options
> usage: queue
> -help Displays help for all commands.
> -status List queue information about given queue.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.yarn.client.cli.YarnCLI.stop(YarnCLI.java:75)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.main(QueueCLI.java:51)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10817) Avoid java.lang.NullPointerException when use yarn shell
[
https://issues.apache.org/jira/browse/YARN-10817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360867#comment-17360867
]
Akira Ajisaka commented on YARN-10817:
--
bq. Probably we need to backport YARN-9246 to branch-3.2.
Done
> Avoid java.lang.NullPointerException when use yarn shell
>
>
> Key: YARN-10817
> URL: https://issues.apache.org/jira/browse/YARN-10817
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.2.1
>Reporter: yikf
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When we use yarn queue-status, we encounter a null pointer exception, As
> follow:
> Like this, and：
> yarn cluster
> yarn queue
> yarn node
> {code:java}
> Missing argument for optionsMissing argument for options
> usage: queue
> -help Displays help for all commands.
> -status List queue information about given queue.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.yarn.client.cli.YarnCLI.stop(YarnCLI.java:75)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.main(QueueCLI.java:51)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9246) NPE when executing a command yarn node -status or -states without additional arguments
[
https://issues.apache.org/jira/browse/YARN-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-9246:
Fix Version/s: 3.2.3
Backported to branch-3.2.
> NPE when executing a command yarn node -status or -states without additional
> arguments
> --
>
> Key: YARN-9246
> URL: https://issues.apache.org/jira/browse/YARN-9246
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Masahiro Tanaka
>Assignee: Masahiro Tanaka
>Priority: Minor
> Fix For: 3.3.0, 3.2.3
>
> Attachments: YARN-9246.001.patch, YARN-9246.002.patch
>
>
> yarn node command should not print NPE even if there are some missing
> arguments.
> {code}
> ubuntu@ip-172-31-0-78:~/hadoop-3.3.0-SNAPSHOT$ ./bin/yarn node -states
> Missing argument for options
> usage: node
> -all Works with -list to list all nodes.
> -help Displays help for all commands.
> -list List all running nodes. Supports optional use of
> -states to filter nodes based on node state, all -all
> to list all nodes, -showDetails to display more
> details about each node.
> -showDetails Works with -list to show more details about each node.
> -states Works with -list to filter nodes based on input
> comma-separated list of node states. The valid node
> state can be one of the following:
> NEW,RUNNING,UNHEALTHY,DECOMMISSIONED,LOST,REBOOTED,DEC
> OMMISSIONING,SHUTDOWN.
> -status Prints the status report of the node.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.yarn.client.cli.YarnCLI.stop(YarnCLI.java:75)
> at org.apache.hadoop.yarn.client.cli.NodeCLI.main(NodeCLI.java:63)
> ubuntu@ip-172-31-0-78:~/hadoop-3.3.0-SNAPSHOT$ ./bin/yarn node -status
> Missing argument for options
> usage: node
> -all Works with -list to list all nodes.
> -help Displays help for all commands.
> -list List all running nodes. Supports optional use of
> -states to filter nodes based on node state, all -all
> to list all nodes, -showDetails to display more
> details about each node.
> -showDetails Works with -list to show more details about each node.
> -states Works with -list to filter nodes based on input
> comma-separated list of node states. The valid node
> state can be one of the following:
> NEW,RUNNING,UNHEALTHY,DECOMMISSIONED,LOST,REBOOTED,DEC
> OMMISSIONING,SHUTDOWN.
> -status Prints the status report of the node.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.yarn.client.cli.YarnCLI.stop(YarnCLI.java:75)
> at org.apache.hadoop.yarn.client.cli.NodeCLI.main(NodeCLI.java:63)
> ubuntu@ip-172-31-0-78:~/hadoop-3.3.0-SNAPSHOT$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10817) Avoid java.lang.NullPointerException when use yarn shell
[
https://issues.apache.org/jira/browse/YARN-10817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360862#comment-17360862
]
Akira Ajisaka commented on YARN-10817:
--
Probably we need to backport YARN-9246 to branch-3.2.
> Avoid java.lang.NullPointerException when use yarn shell
>
>
> Key: YARN-10817
> URL: https://issues.apache.org/jira/browse/YARN-10817
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.2.1
>Reporter: yikf
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When we use yarn queue-status, we encounter a null pointer exception, As
> follow:
> Like this, and：
> yarn cluster
> yarn queue
> yarn node
> {code:java}
> Missing argument for optionsMissing argument for options
> usage: queue
> -help Displays help for all commands.
> -status List queue information about given queue.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.yarn.client.cli.YarnCLI.stop(YarnCLI.java:75)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.main(QueueCLI.java:51)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10817) Avoid java.lang.NullPointerException when use yarn shell
[
https://issues.apache.org/jira/browse/YARN-10817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved YARN-10817.
--
Fix Version/s: (was: 3.2.1)
Resolution: Duplicate
> Avoid java.lang.NullPointerException when use yarn shell
>
>
> Key: YARN-10817
> URL: https://issues.apache.org/jira/browse/YARN-10817
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.2.1
>Reporter: yikf
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When we use yarn queue-status, we encounter a null pointer exception, As
> follow:
> Like this, and：
> yarn cluster
> yarn queue
> yarn node
> {code:java}
> Missing argument for optionsMissing argument for options
> usage: queue
> -help Displays help for all commands.
> -status List queue information about given queue.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.yarn.client.cli.YarnCLI.stop(YarnCLI.java:75)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.main(QueueCLI.java:51)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-10817) Avoid java.lang.NullPointerException when use yarn shell
[
https://issues.apache.org/jira/browse/YARN-10817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reopened YARN-10817:
--
> Avoid java.lang.NullPointerException when use yarn shell
>
>
> Key: YARN-10817
> URL: https://issues.apache.org/jira/browse/YARN-10817
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.2.1
>Reporter: yikf
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.2.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When we use yarn queue-status, we encounter a null pointer exception, As
> follow:
> Like this, and：
> yarn cluster
> yarn queue
> yarn node
> {code:java}
> Missing argument for optionsMissing argument for options
> usage: queue
> -help Displays help for all commands.
> -status List queue information about given queue.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.yarn.client.cli.YarnCLI.stop(YarnCLI.java:75)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.main(QueueCLI.java:51)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10803) [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException
[
https://issues.apache.org/jira/browse/YARN-10803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved YARN-10803.
--
Fix Version/s: 3.3.2
3.4.0
Resolution: Fixed
Committed to trunk and branch-3.3.
> [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider
> fails by ClassCastException
>
>
> Key: YARN-10803
> URL: https://issues.apache.org/jira/browse/YARN-10803
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/178/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
> {quote}
> [ERROR]
> testFailoverChange(org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider)
> Time elapsed: 0.024 s <<< ERROR! java.lang.ClassCastException: class
> org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider$TestProxy cannot be
> cast to class org.apache.hadoop.yarn.client.RMProxy
> (org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider$TestProxy and
> org.apache.hadoop.yarn.client.RMProxy are in unnamed module of loader 'app')
> at
> org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider.testFailoverChange(TestRMFailoverProxyProvider.java:141)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10772) Stable API GetApplicationsRequest#newInstance compatibility broken by YARN-8363
[
https://issues.apache.org/jira/browse/YARN-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned YARN-10772:
Assignee: (was: Akira Ajisaka)
> Stable API GetApplicationsRequest#newInstance compatibility broken by
> YARN-8363
> ---
>
> Key: YARN-10772
> URL: https://issues.apache.org/jira/browse/YARN-10772
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 3.2.0
>Reporter: Wei-Chiu Chuang
>Priority: Major
>
> YARN-8363 migrated our usage of commons-lang to commons-lang3 in 3.2.0.
>
> Unfortunately, it changed the API signature of
> {code:java}
> /**
> *
> * The request from clients to get a report of Applications matching the
> * giving application types in the cluster from the
> * ResourceManager.
> *
> *
> * @see ApplicationClientProtocol#getApplications(GetApplicationsRequest)
> *
> * Setting any of the parameters to null, would just disable that
> * filter
> *
> * @param scope {@link ApplicationsRequestScope} to filter by
> * @param users list of users to filter by
> * @param queues list of scheduler queues to filter by
> * @param applicationTypes types of applications
> * @param applicationTags application tags to filter by
> * @param applicationStates application states to filter by
> * @param startRange range of application start times to filter by
> * @param finishRange range of application finish times to filter by
> * @param limit number of applications to limit to
> * @return {@link GetApplicationsRequest} to be used with
> * {@link ApplicationClientProtocol#getApplications(GetApplicationsRequest)}
> */
> @Public
> @Stable
> public static GetApplicationsRequest newInstance(
> ApplicationsRequestScope scope,
> Set users,
> Set queues,
> Set applicationTypes,
> Set applicationTags,
> EnumSet applicationStates,
> Range startRange,
> Range finishRange,
> Long limit) { {code}
> The startRange and finishRange changed type from LongRange to Range.
> It could cause problems when migrating applications, for example, from Hadoop
> 3.1 to 3.3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10808) TestHBaseTimelineStorageUtils fails by NoClassDefFoundError
[
https://issues.apache.org/jira/browse/YARN-10808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved YARN-10808.
--
Resolution: Duplicate
> TestHBaseTimelineStorageUtils fails by NoClassDefFoundError
> ---
>
> Key: YARN-10808
> URL: https://issues.apache.org/jira/browse/YARN-10808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: newbie
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/529/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client.txt
> {quote}
> [ERROR]
> testWithHbaseConfAtHdfsFileSystem(org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils)
> Time elapsed: 5.89 s <<< ERROR!
> java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2618)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2632)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1498)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:977)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:576)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:518)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils.testWithHbaseConfAtHdfsFileSystem(TestHBaseTimelineStorageUtils.java:86)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10808) TestHBaseTimelineStorageUtils fails by NoClassDefFoundError
[
https://issues.apache.org/jira/browse/YARN-10808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10808:
-
Labels: newbie (was: )
After HDFS-15915, MiniDFSCluster requires mockito dependency. Adding the
dependency fixes this error.
> TestHBaseTimelineStorageUtils fails by NoClassDefFoundError
> ---
>
> Key: YARN-10808
> URL: https://issues.apache.org/jira/browse/YARN-10808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: newbie
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/529/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client.txt
> {quote}
> [ERROR]
> testWithHbaseConfAtHdfsFileSystem(org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils)
> Time elapsed: 5.89 s <<< ERROR!
> java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2618)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2632)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1498)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:977)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:576)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:518)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils.testWithHbaseConfAtHdfsFileSystem(TestHBaseTimelineStorageUtils.java:86)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10808) TestHBaseTimelineStorageUtils fails by NoClassDefFoundError
Akira Ajisaka created YARN-10808:
Summary: TestHBaseTimelineStorageUtils fails by
NoClassDefFoundError
Key: YARN-10808
URL: https://issues.apache.org/jira/browse/YARN-10808
Project: Hadoop YARN
Issue Type: Bug
Components: test
Reporter: Akira Ajisaka
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/529/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client.txt
{quote}
[ERROR]
testWithHbaseConfAtHdfsFileSystem(org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils)
Time elapsed: 5.89 s <<< ERROR!
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at
org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2618)
at
org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2632)
at
org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1498)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:977)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:576)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:518)
at
org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils.testWithHbaseConfAtHdfsFileSystem(TestHBaseTimelineStorageUtils.java:86)
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10803) [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException
[
https://issues.apache.org/jira/browse/YARN-10803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned YARN-10803:
Assignee: Akira Ajisaka
> [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider
> fails by ClassCastException
>
>
> Key: YARN-10803
> URL: https://issues.apache.org/jira/browse/YARN-10803
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/178/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
> {quote}
> [ERROR]
> testFailoverChange(org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider)
> Time elapsed: 0.024 s <<< ERROR! java.lang.ClassCastException: class
> org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider$TestProxy cannot be
> cast to class org.apache.hadoop.yarn.client.RMProxy
> (org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider$TestProxy and
> org.apache.hadoop.yarn.client.RMProxy are in unnamed module of loader 'app')
> at
> org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider.testFailoverChange(TestRMFailoverProxyProvider.java:141)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10803) [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException
[
https://issues.apache.org/jira/browse/YARN-10803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10803:
-
Target Version/s: 3.4.0, 3.3.2 (was: 3.4.0)
> [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider
> fails by ClassCastException
>
>
> Key: YARN-10803
> URL: https://issues.apache.org/jira/browse/YARN-10803
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Priority: Major
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/178/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
> {quote}
> [ERROR]
> testFailoverChange(org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider)
> Time elapsed: 0.024 s <<< ERROR! java.lang.ClassCastException: class
> org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider$TestProxy cannot be
> cast to class org.apache.hadoop.yarn.client.RMProxy
> (org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider$TestProxy and
> org.apache.hadoop.yarn.client.RMProxy are in unnamed module of loader 'app')
> at
> org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider.testFailoverChange(TestRMFailoverProxyProvider.java:141)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10803) [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException
Akira Ajisaka created YARN-10803:
Summary: [JDK 11] TestRMFailoverProxyProvider and
TestNoHaRMFailoverProxyProvider fails by ClassCastException
Key: YARN-10803
URL: https://issues.apache.org/jira/browse/YARN-10803
Project: Hadoop YARN
Issue Type: Bug
Components: test
Reporter: Akira Ajisaka
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/178/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
{quote}
[ERROR]
testFailoverChange(org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider)
Time elapsed: 0.024 s <<< ERROR! java.lang.ClassCastException: class
org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider$TestProxy cannot be
cast to class org.apache.hadoop.yarn.client.RMProxy
(org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider$TestProxy and
org.apache.hadoop.yarn.client.RMProxy are in unnamed module of loader 'app') at
org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider.testFailoverChange(TestRMFailoverProxyProvider.java:141)
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10788) TestCsiClient fails
[
https://issues.apache.org/jira/browse/YARN-10788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351536#comment-17351536
]
Akira Ajisaka commented on YARN-10788:
--
I couldn't reproduce the error locally.
> TestCsiClient fails
> ---
>
> Key: YARN-10788
> URL: https://issues.apache.org/jira/browse/YARN-10788
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Priority: Major
> Attachments:
> patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt
>
>
> TestCsiClient fails to bind to unix domain socket.
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/518/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt
> {noformat}
> [INFO] Running org.apache.hadoop.yarn.csi.client.TestCsiClient
> [ERROR] Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.67
> s <<< FAILURE! - in org.apache.hadoop.yarn.csi.client.TestCsiClient
> [ERROR] testIdentityService(org.apache.hadoop.yarn.csi.client.TestCsiClient)
> Time elapsed: 0.457 s <<< ERROR!
> java.io.IOException: Failed to bind
> at io.grpc.netty.NettyServer.start(NettyServer.java:257)
> at io.grpc.internal.ServerImpl.start(ServerImpl.java:184)
> at io.grpc.internal.ServerImpl.start(ServerImpl.java:90)
> at
> org.apache.hadoop.yarn.csi.client.FakeCsiDriver.start(FakeCsiDriver.java:56)
> at
> org.apache.hadoop.yarn.csi.client.TestCsiClient.testIdentityService(TestCsiClient.java:72)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10788) TestCsiClient fails
[
https://issues.apache.org/jira/browse/YARN-10788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10788:
-
Attachment: patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt
> TestCsiClient fails
> ---
>
> Key: YARN-10788
> URL: https://issues.apache.org/jira/browse/YARN-10788
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Priority: Major
> Attachments:
> patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt
>
>
> TestCsiClient fails to bind to unix domain socket.
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/518/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt
> {noformat}
> [INFO] Running org.apache.hadoop.yarn.csi.client.TestCsiClient
> [ERROR] Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.67
> s <<< FAILURE! - in org.apache.hadoop.yarn.csi.client.TestCsiClient
> [ERROR] testIdentityService(org.apache.hadoop.yarn.csi.client.TestCsiClient)
> Time elapsed: 0.457 s <<< ERROR!
> java.io.IOException: Failed to bind
> at io.grpc.netty.NettyServer.start(NettyServer.java:257)
> at io.grpc.internal.ServerImpl.start(ServerImpl.java:184)
> at io.grpc.internal.ServerImpl.start(ServerImpl.java:90)
> at
> org.apache.hadoop.yarn.csi.client.FakeCsiDriver.start(FakeCsiDriver.java:56)
> at
> org.apache.hadoop.yarn.csi.client.TestCsiClient.testIdentityService(TestCsiClient.java:72)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10788) TestCsiClient fails
Akira Ajisaka created YARN-10788:
Summary: TestCsiClient fails
Key: YARN-10788
URL: https://issues.apache.org/jira/browse/YARN-10788
Project: Hadoop YARN
Issue Type: Bug
Components: test
Reporter: Akira Ajisaka
TestCsiClient fails to bind to unix domain socket.
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/518/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt
{noformat}
[INFO] Running org.apache.hadoop.yarn.csi.client.TestCsiClient
[ERROR] Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.67 s
<<< FAILURE! - in org.apache.hadoop.yarn.csi.client.TestCsiClient
[ERROR] testIdentityService(org.apache.hadoop.yarn.csi.client.TestCsiClient)
Time elapsed: 0.457 s <<< ERROR!
java.io.IOException: Failed to bind
at io.grpc.netty.NettyServer.start(NettyServer.java:257)
at io.grpc.internal.ServerImpl.start(ServerImpl.java:184)
at io.grpc.internal.ServerImpl.start(ServerImpl.java:90)
at
org.apache.hadoop.yarn.csi.client.FakeCsiDriver.start(FakeCsiDriver.java:56)
at
org.apache.hadoop.yarn.csi.client.TestCsiClient.testIdentityService(TestCsiClient.java:72)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10770) container-executor permission is wrong in SecureContainer.md
[
https://issues.apache.org/jira/browse/YARN-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350929#comment-17350929
]
Akira Ajisaka commented on YARN-10770:
--
+1. Thank you [~sahuja] and [~zhuqi]
> container-executor permission is wrong in SecureContainer.md
>
>
> Key: YARN-10770
> URL: https://issues.apache.org/jira/browse/YARN-10770
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Siddharth Ahuja
>Priority: Major
> Labels: newbie
> Attachments: YARN-10770.001.patch
>
>
> {noformat}
> The `container-executor` program must be owned by `root` and have the
> permission set `---sr-s---`.
> {noformat}
> It should be 6050 {noformat}---Sr-s---{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10772) Stable API GetApplicationsRequest#newInstance compatibility broken by YARN-8363
[
https://issues.apache.org/jira/browse/YARN-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346692#comment-17346692
]
Akira Ajisaka edited comment on YARN-10772 at 5/18/21, 8:51 AM:
I tried to write a patch to add the removed API, however, after adding the API,
the following code gets a compile error:
{code:title=TestClientRMService.java}
// Check tags
request = GetApplicationsRequest.newInstance(
ApplicationsRequestScope.ALL, null, null, null, null, null, null,
null, null);
{code}
{noformat}
[ERROR] both method
newInstance(org.apache.hadoop.yarn.api.protocolrecords.ApplicationsRequestScope,java.util.Set,java.util.Set,java.util.Set,java.util.Set,java.util.EnumSet,org.apache.commons.lang3.Range,org.apache.commons.lang3.Range,java.lang.Long)
in org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest and
method
newInstance(org.apache.hadoop.yarn.api.protocolrecords.ApplicationsRequestScope,java.util.Set,java.util.Set,java.util.Set,java.util.Set,java.util.EnumSet,org.apache.commons.lang.math.LongRange,org.apache.commons.lang.math.LongRange,java.lang.Long)
in org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest match
{noformat}
Perhaps it is better for users to keep the signature as-is. Hi [~weichiu], are
there any projects calling the old API?
was (Author: ajisakaa):
I tried to write a patch to add the removed API, however, after adding the API,
the following code gets a compile error:
{code:title=TestClientRMService.java}
// Check tags
request = GetApplicationsRequest.newInstance(
ApplicationsRequestScope.ALL, null, null, null, null, null, null,
null, null);
{code}
{noformat}
[ERROR] both method
newInstance(org.apache.hadoop.yarn.api.protocolrecords.ApplicationsRequestScope,java.util.Set,java.util.Set,java.util.Set,java.util.Set,java.util.EnumSet,org.apache.commons.lang3.Range,org.apache.commons.lang3.Range,java.lang.Long)
in org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest and
method
newInstance(org.apache.hadoop.yarn.api.protocolrecords.ApplicationsRequestScope,java.util.Set,java.util.Set,java.util.Set,java.util.Set,java.util.EnumSet,org.apache.commons.lang.math.LongRange,org.apache.commons.lang.math.LongRange,java.lang.Long)
in org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest match
{noformat}
Perhaps it is better for the user to keep the signature as-is. Hi [~weichiu],
are there any projects calling the old API?
> Stable API GetApplicationsRequest#newInstance compatibility broken by
> YARN-8363
> ---
>
> Key: YARN-10772
> URL: https://issues.apache.org/jira/browse/YARN-10772
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 3.2.0
>Reporter: Wei-Chiu Chuang
>Assignee: Akira Ajisaka
>Priority: Major
>
> YARN-8363 migrated our usage of commons-lang to commons-lang3 in 3.2.0.
>
> Unfortunately, it changed the API signature of
> {code:java}
> /**
> *
> * The request from clients to get a report of Applications matching the
> * giving application types in the cluster from the
> * ResourceManager.
> *
> *
> * @see ApplicationClientProtocol#getApplications(GetApplicationsRequest)
> *
> * Setting any of the parameters to null, would just disable that
> * filter
> *
> * @param scope {@link ApplicationsRequestScope} to filter by
> * @param users list of users to filter by
> * @param queues list of scheduler queues to filter by
> * @param applicationTypes types of applications
> * @param applicationTags application tags to filter by
> * @param applicationStates application states to filter by
> * @param startRange range of application start times to filter by
> * @param finishRange range of application finish times to filter by
> * @param limit number of applications to limit to
> * @return {@link GetApplicationsRequest} to be used with
> * {@link ApplicationClientProtocol#getApplications(GetApplicationsRequest)}
> */
> @Public
> @Stable
> public static GetApplicationsRequest newInstance(
> ApplicationsRequestScope scope,
> Set users,
> Set queues,
> Set applicationTypes,
> Set applicationTags,
> EnumSet applicationStates,
> Range startRange,
> Range finishRange,
> Long limit) { {code}
> The startRange and finishRange changed type from LongRange to Range.
> It could cause problems when migrating applications, for example, from Hadoop
> 3.1 to 3.3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10772) Stable API GetApplicationsRequest#newInstance compatibility broken by YARN-8363
[
https://issues.apache.org/jira/browse/YARN-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346692#comment-17346692
]
Akira Ajisaka commented on YARN-10772:
--
I tried to write a patch to add the removed API, however, after adding the API,
the following code gets a compile error:
{code:title=TestClientRMService.java}
// Check tags
request = GetApplicationsRequest.newInstance(
ApplicationsRequestScope.ALL, null, null, null, null, null, null,
null, null);
{code}
{noformat}
[ERROR] both method
newInstance(org.apache.hadoop.yarn.api.protocolrecords.ApplicationsRequestScope,java.util.Set,java.util.Set,java.util.Set,java.util.Set,java.util.EnumSet,org.apache.commons.lang3.Range,org.apache.commons.lang3.Range,java.lang.Long)
in org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest and
method
newInstance(org.apache.hadoop.yarn.api.protocolrecords.ApplicationsRequestScope,java.util.Set,java.util.Set,java.util.Set,java.util.Set,java.util.EnumSet,org.apache.commons.lang.math.LongRange,org.apache.commons.lang.math.LongRange,java.lang.Long)
in org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest match
{noformat}
Perhaps it is better for the user to keep the signature as-is. Hi [~weichiu],
are there any projects calling the old API?
> Stable API GetApplicationsRequest#newInstance compatibility broken by
> YARN-8363
> ---
>
> Key: YARN-10772
> URL: https://issues.apache.org/jira/browse/YARN-10772
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 3.2.0
>Reporter: Wei-Chiu Chuang
>Assignee: Akira Ajisaka
>Priority: Major
>
> YARN-8363 migrated our usage of commons-lang to commons-lang3 in 3.2.0.
>
> Unfortunately, it changed the API signature of
> {code:java}
> /**
> *
> * The request from clients to get a report of Applications matching the
> * giving application types in the cluster from the
> * ResourceManager.
> *
> *
> * @see ApplicationClientProtocol#getApplications(GetApplicationsRequest)
> *
> * Setting any of the parameters to null, would just disable that
> * filter
> *
> * @param scope {@link ApplicationsRequestScope} to filter by
> * @param users list of users to filter by
> * @param queues list of scheduler queues to filter by
> * @param applicationTypes types of applications
> * @param applicationTags application tags to filter by
> * @param applicationStates application states to filter by
> * @param startRange range of application start times to filter by
> * @param finishRange range of application finish times to filter by
> * @param limit number of applications to limit to
> * @return {@link GetApplicationsRequest} to be used with
> * {@link ApplicationClientProtocol#getApplications(GetApplicationsRequest)}
> */
> @Public
> @Stable
> public static GetApplicationsRequest newInstance(
> ApplicationsRequestScope scope,
> Set users,
> Set queues,
> Set applicationTypes,
> Set applicationTags,
> EnumSet applicationStates,
> Range startRange,
> Range finishRange,
> Long limit) { {code}
> The startRange and finishRange changed type from LongRange to Range.
> It could cause problems when migrating applications, for example, from Hadoop
> 3.1 to 3.3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10772) Stable API GetApplicationsRequest#newInstance compatibility broken by YARN-8363
[
https://issues.apache.org/jira/browse/YARN-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned YARN-10772:
Assignee: Akira Ajisaka
> Stable API GetApplicationsRequest#newInstance compatibility broken by
> YARN-8363
> ---
>
> Key: YARN-10772
> URL: https://issues.apache.org/jira/browse/YARN-10772
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 3.2.0
>Reporter: Wei-Chiu Chuang
>Assignee: Akira Ajisaka
>Priority: Major
>
> YARN-8363 migrated our usage of commons-lang to commons-lang3 in 3.2.0.
>
> Unfortunately, it changed the API signature of
> {code:java}
> /**
> *
> * The request from clients to get a report of Applications matching the
> * giving application types in the cluster from the
> * ResourceManager.
> *
> *
> * @see ApplicationClientProtocol#getApplications(GetApplicationsRequest)
> *
> * Setting any of the parameters to null, would just disable that
> * filter
> *
> * @param scope {@link ApplicationsRequestScope} to filter by
> * @param users list of users to filter by
> * @param queues list of scheduler queues to filter by
> * @param applicationTypes types of applications
> * @param applicationTags application tags to filter by
> * @param applicationStates application states to filter by
> * @param startRange range of application start times to filter by
> * @param finishRange range of application finish times to filter by
> * @param limit number of applications to limit to
> * @return {@link GetApplicationsRequest} to be used with
> * {@link ApplicationClientProtocol#getApplications(GetApplicationsRequest)}
> */
> @Public
> @Stable
> public static GetApplicationsRequest newInstance(
> ApplicationsRequestScope scope,
> Set users,
> Set queues,
> Set applicationTypes,
> Set applicationTags,
> EnumSet applicationStates,
> Range startRange,
> Range finishRange,
> Long limit) { {code}
> The startRange and finishRange changed type from LongRange to Range.
> It could cause problems when migrating applications, for example, from Hadoop
> 3.1 to 3.3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8363) Upgrade commons-lang version to 3.7 in hadoop-yarn-project
[ https://issues.apache.org/jira/browse/YARN-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346582#comment-17346582 ] Akira Ajisaka commented on YARN-8363: - Marking as incompatible change: YARN-10772 > Upgrade commons-lang version to 3.7 in hadoop-yarn-project > -- > > Key: YARN-8363 > URL: https://issues.apache.org/jira/browse/YARN-8363 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8363.1.patch, YARN-8363.2.patch > > > commons-lang 2.6 is widely used. Let's upgrade to 3.6. > This jira is separated from HADOOP-10783. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8363) Upgrade commons-lang version to 3.7 in hadoop-yarn-project
[ https://issues.apache.org/jira/browse/YARN-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-8363: Hadoop Flags: Incompatible change,Reviewed (was: Reviewed) > Upgrade commons-lang version to 3.7 in hadoop-yarn-project > -- > > Key: YARN-8363 > URL: https://issues.apache.org/jira/browse/YARN-8363 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8363.1.patch, YARN-8363.2.patch > > > commons-lang 2.6 is widely used. Let's upgrade to 3.6. > This jira is separated from HADOOP-10783. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Deleted] (YARN-10773) Situs Slot Online Gacor Hari Ini 2021
[ https://issues.apache.org/jira/browse/YARN-10773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka deleted YARN-10773: - > Situs Slot Online Gacor Hari Ini 2021 > - > > Key: YARN-10773 > URL: https://issues.apache.org/jira/browse/YARN-10773 > Project: Hadoop YARN > Issue Type: New Feature > Environment: Karakter Liar yang bersimbol lembah berbatu ini punya > dua fungsi. Paling awal, bila digunakan pada putaran biasa akan bisa menjadi > pengganti semua gambar yang ada ketika dibutuhkan. Satu lagi simbol [judi > slot|https://www.arnobbd.com] yang dapat menambah hasil kemenangan adalah > munculnya bonus gambar yang bergambar koin emas banteng, yang bila didapat > apabila 3 karakter saja sudah langsung langsung memberi dua puluh dua kali > dan bisa mendapatkan bayaran bertaruh dengan 100x, dan ketika free spin yang > diraih tersebut dimainkan, akan diperoleh fungsi lainnya dari dapat wild, > yaitu akan meraih penambahan 2x, tiga kali dan lima kali kepada hasil > peraihan yang sudah lebih dulu diraih. Dari gambar wild yang terdapat > perhitungan ini sangat memungkinkan untuk memperoleh lebih dari satu karakter > pada tiap permainan, dan akan bertambah terus pada perhitungannya. Benar > benar perolehan yang berganda seperti yang dijanjikan, yang bisa saja > memenangkan Sensasional. > Harga yang akan dibayarkan oleh pemain [slot online|http://5.77.39.126] > seperti enam gambar yang sama muncul berbarengan saat 1 putaran. Tiap simbol > memiliki jumlah nominal yang berbeda, apabila seperti Simbol Buffalo King > untuk memperoleh enam gambar yang sama, harga yang dapat diraih sebesar dua > puluh kali nilai uang taruhan, ini yakni nilai yang tertinggi didapat oleh > simbol Banteng, yang akan lain jumlahnya bila karakter serigala untuk datang > 6 karakter yang sama, harga yang akan diperoleh sebesar 1,5x nominal taruhan. > Bila peserta sudah mengatur uang taruhan dengan menekan tanda kurang tambah > saat mengatur taruhan dengan nilai besar, maka dapat pemain hitung berapa > harga yang akan diperoleh. kita dapat mengklik tanda -+ untuk taruhan kecil > dan menaikkan taruhan, atau krusor dapat digeser geser ke kiri untuk > mengecilkan taruhan dan geser kesamping kanan menambah taruhan. semua pemain > slot memainkan nilai kecil untuk memanaskan mesin slot. awal putaran untuk > mempelajari untuk langkah selanjutnya. > Fitur susun kebawah, orang kenal dengan mode runtuh, menghilangkan karakter > yang menghasilkan kombinasi menang dan menggantinya dengan simbol grafis yang > bergeser menggantikan dari geseran atas. Karakter yang sisa di layar tampilan > setelah dapat kemenangan jatuh turun bergeser kebawah gulungan. Dengan maksud > lain, anda akan bermain dengan grafis baru setelah setiap kemenangan > sepanjang bertaruh.Urutan gesernya simbol berakhir ketika sudah berhenti > setelah memperoleh kemenangan dikarenakan reel jatuh bergeser kebawah. >Reporter: Damien Sapien >Priority: Minor > > h1. Slot Online Gacor Hari Ini 2021 > h2. Main menang di Situs Slot Online Sering Jackpot > Kagak dikira ternyata datang idul fitri kembali,tertinggal berapa hari kita > semua untuk masuk ke waktunya saling minta maaf sesudah selagi satu bulan > menjalani amalan menahan emosi. Dengan menyesal hari fitri sekarang ini kita > masuki lewat status yang persis ibarat hari fitri warsa yang lewat. virus > berbahaya covid bersisa menemani suasana hari fitri sekarang ini. anda tengah > harus senantiasa menjamin ruang maka menyingkir menerapkan kerubungan besok > sedang tatkala berjumpa lewat keluarga. Memang mau tak mau memerlukan > kesadaran pula wawasan luas berpikir menghadapi lingkungan ini. Mempunyai > membela pula anti disekitar semua orang tiap-tiap menyimpan ragam paham diri > sendiri akan mengerjakannya. Bagaimanapun cara tingkah laku untuk mengerjakan > berhadapan hari fitri kemudian. Buktikan tata cara Kesegaran mau tak mau > selamanya diamalkan tidak teledor. Mambasahi tangan sebelum saling salam > harus dilakukan, atau simpan alat pancar cairan pembasmi dekat baju bisa > memungkinkan menjadi bersih ulang telapak kalian. Silih berganti saling tos > dapat kalian tukar lewat beradu cengkram tangan saja sudah buat silih > berganti menghormati kala ini. Kalian memahami penghalang aksi yang masih > semua orang laksanakan. Mengenakan kedok penjaga wajahpun tidak lalai. Bakal > mengurangi menyebarnya bakteri dari tumpahan air ludah bisa terbentuk tatkala > semua orang sama sama bertutur, dan mencermati langkah nyaman untuk > meningkatkan kemungkinan tidak bisa tergapai percikan ke sisi anda. Bila tiga > tata cara kesehatan ini digelar seraya teratur, bertemu lewat sanak saudara > jua diinginkan berlangsung gampang jangan ada konsekuensi di anda termuat 2 > pekan setelah hari raya, menjadi
[jira] [Commented] (YARN-10772) Stable API GetApplicationsRequest#newInstance compatibility broken by YARN-8363
[
https://issues.apache.org/jira/browse/YARN-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346543#comment-17346543
]
Akira Ajisaka commented on YARN-10772:
--
Hi [~weichiu], thank you for your report. LongRange comes from commons-lang
2.x, so we need to add back commons-lang 2.x dependency to fix the
compatibility.
BTW, in YARN-8363, we upgraded the signature to use commons-lang 3.x, which is
still not good. If we upgrade commons-lang 4.x in the future, we break
compatibility again. Instead, I want to create a public API to avoid the use of
external library.
> Stable API GetApplicationsRequest#newInstance compatibility broken by
> YARN-8363
> ---
>
> Key: YARN-10772
> URL: https://issues.apache.org/jira/browse/YARN-10772
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 3.2.0
>Reporter: Wei-Chiu Chuang
>Priority: Major
>
> YARN-8363 migrated our usage of commons-lang to commons-lang3 in 3.2.0.
>
> Unfortunately, it changed the API signature of
> {code:java}
> /**
> *
> * The request from clients to get a report of Applications matching the
> * giving application types in the cluster from the
> * ResourceManager.
> *
> *
> * @see ApplicationClientProtocol#getApplications(GetApplicationsRequest)
> *
> * Setting any of the parameters to null, would just disable that
> * filter
> *
> * @param scope {@link ApplicationsRequestScope} to filter by
> * @param users list of users to filter by
> * @param queues list of scheduler queues to filter by
> * @param applicationTypes types of applications
> * @param applicationTags application tags to filter by
> * @param applicationStates application states to filter by
> * @param startRange range of application start times to filter by
> * @param finishRange range of application finish times to filter by
> * @param limit number of applications to limit to
> * @return {@link GetApplicationsRequest} to be used with
> * {@link ApplicationClientProtocol#getApplications(GetApplicationsRequest)}
> */
> @Public
> @Stable
> public static GetApplicationsRequest newInstance(
> ApplicationsRequestScope scope,
> Set users,
> Set queues,
> Set applicationTypes,
> Set applicationTags,
> EnumSet applicationStates,
> Range startRange,
> Range finishRange,
> Long limit) { {code}
> The startRange and finishRange changed type from LongRange to Range.
> It could cause problems when migrating applications, for example, from Hadoop
> 3.1 to 3.3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10555) Missing access check before getAppAttempts
[
https://issues.apache.org/jira/browse/YARN-10555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10555:
-
Fix Version/s: 2.10.2
> Missing access check before getAppAttempts
> ---
>
> Key: YARN-10555
> URL: https://issues.apache.org/jira/browse/YARN-10555
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Labels: pull-request-available, security
> Fix For: 3.4.0, 3.3.1, 3.1.5, 2.10.2, 3.2.3
>
> Attachments: YARN-10555_1.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> It seems that we miss a security check before getAppAttempts, see
> [https://github.com/apache/hadoop/blob/513f1995adc9b73f9c7f4c7beb89725b51b313ac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L1127]
> thus we can get the some sensitive information, like logs link.
> {code:java}
> application_1609318368700_0002 belong to user2
> user1@hadoop11$ curl --negotiate -u :
> http://hadoop11:8088/ws/v1/cluster/apps/application_1609318368700_0002/appattempts/|jq
> {
> "appAttempts": {
> "appAttempt": [
> {
> "id": 1,
> "startTime": 1609318411566,
> "containerId": "container_1609318368700_0002_01_01",
> "nodeHttpAddress": "hadoop12:8044",
> "nodeId": "hadoop12:36831",
> "logsLink":
> "http://hadoop12:8044/node/containerlogs/container_1609318368700_0002_01_01/user2;,
> "blacklistedNodes": "",
> "nodesBlacklistedBySystem": ""
> }
> ]
> }
> }
> {code}
> Other apis, like getApps and getApp, has access check like "hasAccess(app,
> hsr)", they would hide the logs link if the appid do not belong to query
> user, see
> [https://github.com/apache/hadoop/blob/513f1995adc9b73f9c7f4c7beb89725b51b313ac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L1098]
> We need add hasAccess(app, hsr) for getAppAttempts.
>
> Besides, at
> [https://github.com/apache/hadoop/blob/580a6a75a3e3d3b7918edeffd6e93fc211166884/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java#L145]
> it seems that we have a access check in its caller, so now i pass "true" to
> AppAttemptInfo in the patch.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10555) Missing access check before getAppAttempts
[
https://issues.apache.org/jira/browse/YARN-10555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10555:
-
Summary: Missing access check before getAppAttempts (was: missing access
check before getAppAttempts)
> Missing access check before getAppAttempts
> ---
>
> Key: YARN-10555
> URL: https://issues.apache.org/jira/browse/YARN-10555
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Labels: pull-request-available, security
> Fix For: 3.4.0, 3.3.1, 3.1.5, 3.2.3
>
> Attachments: YARN-10555_1.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> It seems that we miss a security check before getAppAttempts, see
> [https://github.com/apache/hadoop/blob/513f1995adc9b73f9c7f4c7beb89725b51b313ac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L1127]
> thus we can get the some sensitive information, like logs link.
> {code:java}
> application_1609318368700_0002 belong to user2
> user1@hadoop11$ curl --negotiate -u :
> http://hadoop11:8088/ws/v1/cluster/apps/application_1609318368700_0002/appattempts/|jq
> {
> "appAttempts": {
> "appAttempt": [
> {
> "id": 1,
> "startTime": 1609318411566,
> "containerId": "container_1609318368700_0002_01_01",
> "nodeHttpAddress": "hadoop12:8044",
> "nodeId": "hadoop12:36831",
> "logsLink":
> "http://hadoop12:8044/node/containerlogs/container_1609318368700_0002_01_01/user2;,
> "blacklistedNodes": "",
> "nodesBlacklistedBySystem": ""
> }
> ]
> }
> }
> {code}
> Other apis, like getApps and getApp, has access check like "hasAccess(app,
> hsr)", they would hide the logs link if the appid do not belong to query
> user, see
> [https://github.com/apache/hadoop/blob/513f1995adc9b73f9c7f4c7beb89725b51b313ac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L1098]
> We need add hasAccess(app, hsr) for getAppAttempts.
>
> Besides, at
> [https://github.com/apache/hadoop/blob/580a6a75a3e3d3b7918edeffd6e93fc211166884/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java#L145]
> it seems that we have a access check in its caller, so now i pass "true" to
> AppAttemptInfo in the patch.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10555) missing access check before getAppAttempts
[
https://issues.apache.org/jira/browse/YARN-10555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10555:
-
Fix Version/s: 3.2.3
3.1.5
3.3.1
> missing access check before getAppAttempts
> ---
>
> Key: YARN-10555
> URL: https://issues.apache.org/jira/browse/YARN-10555
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Labels: pull-request-available, security
> Fix For: 3.4.0, 3.3.1, 3.1.5, 3.2.3
>
> Attachments: YARN-10555_1.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> It seems that we miss a security check before getAppAttempts, see
> [https://github.com/apache/hadoop/blob/513f1995adc9b73f9c7f4c7beb89725b51b313ac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L1127]
> thus we can get the some sensitive information, like logs link.
> {code:java}
> application_1609318368700_0002 belong to user2
> user1@hadoop11$ curl --negotiate -u :
> http://hadoop11:8088/ws/v1/cluster/apps/application_1609318368700_0002/appattempts/|jq
> {
> "appAttempts": {
> "appAttempt": [
> {
> "id": 1,
> "startTime": 1609318411566,
> "containerId": "container_1609318368700_0002_01_01",
> "nodeHttpAddress": "hadoop12:8044",
> "nodeId": "hadoop12:36831",
> "logsLink":
> "http://hadoop12:8044/node/containerlogs/container_1609318368700_0002_01_01/user2;,
> "blacklistedNodes": "",
> "nodesBlacklistedBySystem": ""
> }
> ]
> }
> }
> {code}
> Other apis, like getApps and getApp, has access check like "hasAccess(app,
> hsr)", they would hide the logs link if the appid do not belong to query
> user, see
> [https://github.com/apache/hadoop/blob/513f1995adc9b73f9c7f4c7beb89725b51b313ac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L1098]
> We need add hasAccess(app, hsr) for getAppAttempts.
>
> Besides, at
> [https://github.com/apache/hadoop/blob/580a6a75a3e3d3b7918edeffd6e93fc211166884/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java#L145]
> it seems that we have a access check in its caller, so now i pass "true" to
> AppAttemptInfo in the patch.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10555) missing access check before getAppAttempts
[
https://issues.apache.org/jira/browse/YARN-10555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17345998#comment-17345998
]
Akira Ajisaka commented on YARN-10555:
--
Thank you [~zhuqi] and [~xiaoheipangzi]. I'm backporting to the lower branches,
will update the fix versions.
> missing access check before getAppAttempts
> ---
>
> Key: YARN-10555
> URL: https://issues.apache.org/jira/browse/YARN-10555
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Labels: pull-request-available, security
> Fix For: 3.4.0
>
> Attachments: YARN-10555_1.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> It seems that we miss a security check before getAppAttempts, see
> [https://github.com/apache/hadoop/blob/513f1995adc9b73f9c7f4c7beb89725b51b313ac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L1127]
> thus we can get the some sensitive information, like logs link.
> {code:java}
> application_1609318368700_0002 belong to user2
> user1@hadoop11$ curl --negotiate -u :
> http://hadoop11:8088/ws/v1/cluster/apps/application_1609318368700_0002/appattempts/|jq
> {
> "appAttempts": {
> "appAttempt": [
> {
> "id": 1,
> "startTime": 1609318411566,
> "containerId": "container_1609318368700_0002_01_01",
> "nodeHttpAddress": "hadoop12:8044",
> "nodeId": "hadoop12:36831",
> "logsLink":
> "http://hadoop12:8044/node/containerlogs/container_1609318368700_0002_01_01/user2;,
> "blacklistedNodes": "",
> "nodesBlacklistedBySystem": ""
> }
> ]
> }
> }
> {code}
> Other apis, like getApps and getApp, has access check like "hasAccess(app,
> hsr)", they would hide the logs link if the appid do not belong to query
> user, see
> [https://github.com/apache/hadoop/blob/513f1995adc9b73f9c7f4c7beb89725b51b313ac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L1098]
> We need add hasAccess(app, hsr) for getAppAttempts.
>
> Besides, at
> [https://github.com/apache/hadoop/blob/580a6a75a3e3d3b7918edeffd6e93fc211166884/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java#L145]
> it seems that we have a access check in its caller, so now i pass "true" to
> AppAttemptInfo in the patch.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10770) container-executor permission is wrong in SecureContainer.md
Akira Ajisaka created YARN-10770:
Summary: container-executor permission is wrong in
SecureContainer.md
Key: YARN-10770
URL: https://issues.apache.org/jira/browse/YARN-10770
Project: Hadoop YARN
Issue Type: Bug
Components: documentation
Reporter: Akira Ajisaka
{noformat}
The `container-executor` program must be owned by `root` and have the
permission set `---sr-s---`.
{noformat}
It should be 6050 {noformat}---Sr-s---{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10770) container-executor permission is wrong in SecureContainer.md
[
https://issues.apache.org/jira/browse/YARN-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10770:
-
Labels: newbie (was: )
> container-executor permission is wrong in SecureContainer.md
>
>
> Key: YARN-10770
> URL: https://issues.apache.org/jira/browse/YARN-10770
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: newbie
>
> {noformat}
> The `container-executor` program must be owned by `root` and have the
> permission set `---sr-s---`.
> {noformat}
> It should be 6050 {noformat}---Sr-s---{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9333) TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently
[
https://issues.apache.org/jira/browse/YARN-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-9333:
Fix Version/s: 3.3.1
Backported to branch-3.3.
> TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes
> fails intermittently
>
>
> Key: YARN-9333
> URL: https://issues.apache.org/jira/browse/YARN-9333
> Project: Hadoop YARN
> Issue Type: Test
> Components: yarn
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-9333-001.patch, YARN-9333-002.patch,
> YARN-9333-003.patch, YARN-9333-debug1.patch
>
>
> TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes
> fails intermittent - observed in YARN-9311.
> {code}
> [ERROR]
> testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes[MinSharePreemptionWithDRF](org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption)
> Time elapsed: 11.056 s <<< FAILURE!
> java.lang.AssertionError: Incorrect # of containers on the greedy app
> expected:<6> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:296)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyRelaxLocalityPreemption(TestFairSchedulerPreemption.java:537)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes(TestFairSchedulerPreemption.java:473)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
>
[jira] [Updated] (YARN-10515) Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
[
https://issues.apache.org/jira/browse/YARN-10515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10515:
-
Fix Version/s: 3.3.1
Backported to branch-3.3.
> Fix flaky test
> TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
> --
>
> Key: YARN-10515
> URL: https://issues.apache.org/jira/browse/YARN-10515
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10515-001.patch
>
>
> The testcase
> TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags
> sometimes fails with the following error:
> {noformat}
> org.apache.hadoop.service.ServiceStateException:
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:174)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:884)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1296)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:339)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1018)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:134)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:130)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation$5.(TestCapacitySchedulerAutoQueueCreation.java:873)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags(TestCapacitySchedulerAutoQueueCreation.java:873)
> ...
> Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
> PartitionQueueMetrics,partition=,q0=root,q1=a already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getPartitionQueueMetrics(QueueMetrics.java:317)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.setAvailableResourcesToQueue(QueueMetrics.java:513)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateQueueStatistics(CSQueueUtils.java:308)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:412)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:350)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:137)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:119)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:57)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:261)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:289)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:162)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:752)
> {noformat}
> We have to reset
[jira] [Commented] (YARN-9279) Remove the old hamlet package
[ https://issues.apache.org/jira/browse/YARN-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340481#comment-17340481 ] Akira Ajisaka commented on YARN-9279: - PR opened: https://github.com/apache/hadoop/pull/2986 > Remove the old hamlet package > - > > Key: YARN-9279 > URL: https://issues.apache.org/jira/browse/YARN-9279 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Attachments: YARN-9279.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The old hamlet package was deprecated in HADOOP-11875. Let's remove this to > improve the maintenability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10756) Remove additional junit 4.11 dependency from javadoc
[ https://issues.apache.org/jira/browse/YARN-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-10756: - Summary: Remove additional junit 4.11 dependency from javadoc (was: Upgrade JUnit to 4.13.1) > Remove additional junit 4.11 dependency from javadoc > > > Key: YARN-10756 > URL: https://issues.apache.org/jira/browse/YARN-10756 > Project: Hadoop YARN > Issue Type: Bug > Components: build, test, timelineservice >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.1, 3.1.5, 3.2.3 > > Time Spent: 40m > Remaining Estimate: 0h > > Yarn Timeline Server still using 4.11 Junit version, need to upgrade it to > 4.13.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10757) jsonschema2pojo-maven-plugin version is not defined
[
https://issues.apache.org/jira/browse/YARN-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10757:
-
Target Version/s: 3.4.0
> jsonschema2pojo-maven-plugin version is not defined
> ---
>
> Key: YARN-10757
> URL: https://issues.apache.org/jira/browse/YARN-10757
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: newbie
>
> The below maven plugin version is not defined.
> {noformat}
> $ mvn install
> [INFO] Scanning for projects...
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:3.4.0-SNAPSHOT
> [WARNING] 'build.plugins.plugin.version' for
> org.jsonschema2pojo:jsonschema2pojo-maven-plugin is missing. @ line 448,
> column 15
> [WARNING]
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
> [WARNING]
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
> [WARNING]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10757) jsonschema2pojo-maven-plugin version is not defined
Akira Ajisaka created YARN-10757:
Summary: jsonschema2pojo-maven-plugin version is not defined
Key: YARN-10757
URL: https://issues.apache.org/jira/browse/YARN-10757
Project: Hadoop YARN
Issue Type: Bug
Components: build
Reporter: Akira Ajisaka
The below maven plugin version is not defined.
{noformat}
$ mvn install
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:3.4.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for
org.jsonschema2pojo:jsonschema2pojo-maven-plugin is missing. @ line 448, column
15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten
the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
[WARNING]
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10510) TestAppPage.testAppBlockRenderWithNullCurrentAppAttempt will cause NullPointerException
[
https://issues.apache.org/jira/browse/YARN-10510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17333093#comment-17333093
]
Akira Ajisaka commented on YARN-10510:
--
What is the java version configured in IntelliJ? I assume the test might fail
on JDK 11.
> TestAppPage.testAppBlockRenderWithNullCurrentAppAttempt will cause
> NullPointerException
>
>
> Key: YARN-10510
> URL: https://issues.apache.org/jira/browse/YARN-10510
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: tuyu
>Priority: Minor
> Labels: test
> Attachments: YARN-10510.001.patch
>
>
> run TestAppPage.testAppBlockRenderWithNullCurrentAppAttempt will cause blow
> exception
> {code:java}
> 2020-12-01 20:16:41,412 ERROR [main] webapp.AppBlock
> (AppBlock.java:render(124)) - Failed to read the application
> application_1234_.
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMAppBlock.getApplicationReport(RMAppBlock.java:218)
> at
> org.apache.hadoop.yarn.server.webapp.AppBlock.render(AppBlock.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMAppBlock.render(RMAppBlock.java:71)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestAppPage.testAppBlockRenderWithNullCurrentAppAttempt(TestAppPage.java:92)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
> Disconnected from the target VM, address: '127.0.0.1:60623', transport:
> 'socket'
> {code}
> because of mockClientRMService not mock getApplicationReport and
> getApplicationAttempts interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
