[jira] [Resolved] (YARN-9971) YARN Native Service HttpProbe logs THIS_HOST in error messages
[
https://issues.apache.org/jira/browse/YARN-9971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph resolved YARN-9971.
-
Fix Version/s: 3.4.0
Resolution: Fixed
Thanks [~groot] for the patch. Have committed the patch to trunk.
> YARN Native Service HttpProbe logs THIS_HOST in error messages
> --
>
> Key: YARN-9971
> URL: https://issues.apache.org/jira/browse/YARN-9971
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> YARN Native Service HttpProbe logs THIS_HOST in error messages. While
> logging, missed to use the replaced url string.
> {code:java}
> 2019-11-12 19:25:47,317 [pool-7-thread-1] INFO probe.HttpProbe - Probe
> http://${THIS_HOST}:18010/master-status failed for IP 172.27.75.198:
> java.net.ConnectException: Connection refused (Connection refused)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7463) Using getLocalPathForWrite for Container related debug information
[ https://issues.apache.org/jira/browse/YARN-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557214#comment-17557214 ] Ashutosh Gupta commented on YARN-7463: -- This issue seems to be already fixed in trunk > Using getLocalPathForWrite for Container related debug information > -- > > Key: YARN-7463 > URL: https://issues.apache.org/jira/browse/YARN-7463 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > Attachments: YARN-7463.1.patch > > > Containers debug information launch_container.sh and directory.info are > always logged into first directory of NM_LOG_DIRS instead of using the log > directory returned from getLogPathForWrite. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9373) HBaseTimelineSchemaCreator has to allow user to configure pre-splits
[ https://issues.apache.org/jira/browse/YARN-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557206#comment-17557206 ] Ashutosh Gupta commented on YARN-9373: -- The issue seems to be pending for a while. Taking it up. > HBaseTimelineSchemaCreator has to allow user to configure pre-splits > > > Key: YARN-9373 > URL: https://issues.apache.org/jira/browse/YARN-9373 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: Configurable_PreSplits.png, YARN-9373-001.patch, > YARN-9373-002.patch, YARN-9373-003.patch > > > Most of the TimelineService HBase tables is set with username splits which is > based on lowercase alphabet (a,ad,an,b,ca). This won't help if the rowkey > starts with either number or uppercase alphabet. We need to allow user to > configure based upon their data. For example, say a user has configured the > yarn.resourcemanager.cluster-id to be ATS or 123, then the splits can be > configured as A,B,C,,, or 100,200,300,,, -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9373) HBaseTimelineSchemaCreator has to allow user to configure pre-splits
[ https://issues.apache.org/jira/browse/YARN-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-9373: Assignee: Ashutosh Gupta (was: Prabhu Joseph) > HBaseTimelineSchemaCreator has to allow user to configure pre-splits > > > Key: YARN-9373 > URL: https://issues.apache.org/jira/browse/YARN-9373 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Ashutosh Gupta >Priority: Major > Attachments: Configurable_PreSplits.png, YARN-9373-001.patch, > YARN-9373-002.patch, YARN-9373-003.patch > > > Most of the TimelineService HBase tables is set with username splits which is > based on lowercase alphabet (a,ad,an,b,ca). This won't help if the rowkey > starts with either number or uppercase alphabet. We need to allow user to > configure based upon their data. For example, say a user has configured the > yarn.resourcemanager.cluster-id to be ATS or 123, then the splits can be > configured as A,B,C,,, or 100,200,300,,, -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9408) @Path("/apps/{appid}/appattempts") error message misleads
[
https://issues.apache.org/jira/browse/YARN-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557205#comment-17557205
]
Ashutosh Gupta commented on YARN-9408:
--
The issue seems to be pending for a while. Taking it up.
> @Path("/apps/{appid}/appattempts") error message misleads
> -
>
> Key: YARN-9408
> URL: https://issues.apache.org/jira/browse/YARN-9408
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9408-001.patch, YARN-9408-002.patch
>
>
> {code} @Path("/apps/{appid}/appattempts") {code} error message is misleading.
> NotFoundException "Unable to find the context flow name, and flow run id, and
> user id" is displayed while app attempts is looked.
> {code}
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0004/appattempts?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats,
> appId=application_1553258815132_0004",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats/apps/application_1553258815132_0005/appattempts?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats,
> appId=application_1553258815132_0005",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats1/apps/application_1553258815132_0001/containers/container_e14_1553258815132_0001_01_01?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats1,
> appId=application_1553258815132_0001",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats1/apps/application_1553258815132_0001/appattempts/appattempt_1553258815132_0001_01/containers?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats1,
> appId=application_1553258815132_0001",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9408) @Path("/apps/{appid}/appattempts") error message misleads
[
https://issues.apache.org/jira/browse/YARN-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-9408:
Assignee: Ashutosh Gupta (was: Prabhu Joseph)
> @Path("/apps/{appid}/appattempts") error message misleads
> -
>
> Key: YARN-9408
> URL: https://issues.apache.org/jira/browse/YARN-9408
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Minor
> Attachments: YARN-9408-001.patch, YARN-9408-002.patch
>
>
> {code} @Path("/apps/{appid}/appattempts") {code} error message is misleading.
> NotFoundException "Unable to find the context flow name, and flow run id, and
> user id" is displayed while app attempts is looked.
> {code}
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0004/appattempts?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats,
> appId=application_1553258815132_0004",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats/apps/application_1553258815132_0005/appattempts?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats,
> appId=application_1553258815132_0005",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats1/apps/application_1553258815132_0001/containers/container_e14_1553258815132_0001_01_01?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats1,
> appId=application_1553258815132_0001",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats1/apps/application_1553258815132_0001/appattempts/appattempt_1553258815132_0001_01/containers?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats1,
> appId=application_1553258815132_0001",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9822) TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
[
https://issues.apache.org/jira/browse/YARN-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557198#comment-17557198
]
Ashutosh Gupta commented on YARN-9822:
--
This is an important issue. As its pending for a while. Taking it up - making
the required changes and raising the PR
> TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
> -
>
> Key: YARN-9822
> URL: https://issues.apache.org/jira/browse/YARN-9822
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9822-001.patch, YARN-9822-002.patch
>
>
> TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
> YARN-9374 prevents the threads getting blocked when it has already identified
> that Hbase down before accessing Hbase. TimelineCollector can check if the
> Writer Backend is up or down before locking the writer.
> {code}
> synchronized (writer) {
> response = writeTimelineEntities(entities, callerUgi);
> flushBufferedTimelineEntities();
> }
> {code}
> {code}
> "qtp183259297-80" #80 daemon prio=5 os_prio=0 tid=0x7f5f567fd000
> nid=0x5fbb waiting for monitor entry [0x7f5f236d4000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector.putEntities(TimelineCollector.java:164)
> - waiting to lock <0x0006c7c05770> (a
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorWebService.putEntities(TimelineCollectorWebService.java:186)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1624)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
[jira] [Assigned] (YARN-9822) TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
[
https://issues.apache.org/jira/browse/YARN-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-9822:
Assignee: Ashutosh Gupta (was: Prabhu Joseph)
> TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
> -
>
> Key: YARN-9822
> URL: https://issues.apache.org/jira/browse/YARN-9822
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Attachments: YARN-9822-001.patch, YARN-9822-002.patch
>
>
> TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
> YARN-9374 prevents the threads getting blocked when it has already identified
> that Hbase down before accessing Hbase. TimelineCollector can check if the
> Writer Backend is up or down before locking the writer.
> {code}
> synchronized (writer) {
> response = writeTimelineEntities(entities, callerUgi);
> flushBufferedTimelineEntities();
> }
> {code}
> {code}
> "qtp183259297-80" #80 daemon prio=5 os_prio=0 tid=0x7f5f567fd000
> nid=0x5fbb waiting for monitor entry [0x7f5f236d4000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector.putEntities(TimelineCollector.java:164)
> - waiting to lock <0x0006c7c05770> (a
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorWebService.putEntities(TimelineCollectorWebService.java:186)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1624)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
>
[jira] [Updated] (YARN-9874) Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation
[
https://issues.apache.org/jira/browse/YARN-9874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-9874:
-
Labels: pull-request-available (was: )
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation
> --
>
> Key: YARN-9874
> URL: https://issues.apache.org/jira/browse/YARN-9874
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Minor
> Labels: pull-request-available
> Attachments: YARN-9874-001.patch, YARN-9874-002.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation.
> {code}
> public void confirmMutation(boolean isValid) throws IOException {
> WriteBatch updateBatch = db.createWriteBatch();
> if (isValid) {
> ...
> }
> db.write(updateBatch);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9874) Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation
[
https://issues.apache.org/jira/browse/YARN-9874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-9874:
Assignee: Ashutosh Gupta (was: Prabhu Joseph)
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation
> --
>
> Key: YARN-9874
> URL: https://issues.apache.org/jira/browse/YARN-9874
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Minor
> Attachments: YARN-9874-001.patch, YARN-9874-002.patch
>
>
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation.
> {code}
> public void confirmMutation(boolean isValid) throws IOException {
> WriteBatch updateBatch = db.createWriteBatch();
> if (isValid) {
> ...
> }
> db.write(updateBatch);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9874) Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation
[
https://issues.apache.org/jira/browse/YARN-9874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557196#comment-17557196
]
Ashutosh Gupta commented on YARN-9874:
--
The issue seems to be pending for a while. Taking it up.
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation
> --
>
> Key: YARN-9874
> URL: https://issues.apache.org/jira/browse/YARN-9874
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9874-001.patch, YARN-9874-002.patch
>
>
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation.
> {code}
> public void confirmMutation(boolean isValid) throws IOException {
> WriteBatch updateBatch = db.createWriteBatch();
> if (isValid) {
> ...
> }
> db.write(updateBatch);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10320) Replace FSDataInputStream#read with readFully in Log Aggregation
[
https://issues.apache.org/jira/browse/YARN-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-10320:
--
Labels: pull-request-available (was: )
> Replace FSDataInputStream#read with readFully in Log Aggregation
>
>
> Key: YARN-10320
> URL: https://issues.apache.org/jira/browse/YARN-10320
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Attachments: YARN-10320-001.patch, YARN-10320-002.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Have observed Log Aggregation code has used FSDataInputStream#read instead of
> readFully in multiple places like below. One of the place is fixed by
> YARN-8106.
> This Jira targets to fix at all other places.
> LogAggregationIndexedFileController#loadUUIDFromLogFile
> {code}
> byte[] b = new byte[uuid.length];
> int actual = fsDataInputStream.read(b);
> if (actual != uuid.length || Arrays.equals(b, uuid)) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10320) Replace FSDataInputStream#read with readFully in Log Aggregation
[
https://issues.apache.org/jira/browse/YARN-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557173#comment-17557173
]
Ashutosh Gupta commented on YARN-10320:
---
This issue seems to be pending for a while. Taking it up.
> Replace FSDataInputStream#read with readFully in Log Aggregation
>
>
> Key: YARN-10320
> URL: https://issues.apache.org/jira/browse/YARN-10320
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10320-001.patch, YARN-10320-002.patch
>
>
> Have observed Log Aggregation code has used FSDataInputStream#read instead of
> readFully in multiple places like below. One of the place is fixed by
> YARN-8106.
> This Jira targets to fix at all other places.
> LogAggregationIndexedFileController#loadUUIDFromLogFile
> {code}
> byte[] b = new byte[uuid.length];
> int actual = fsDataInputStream.read(b);
> if (actual != uuid.length || Arrays.equals(b, uuid)) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10320) Replace FSDataInputStream#read with readFully in Log Aggregation
[
https://issues.apache.org/jira/browse/YARN-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-10320:
-
Assignee: Ashutosh Gupta (was: Tanu Ajmera)
> Replace FSDataInputStream#read with readFully in Log Aggregation
>
>
> Key: YARN-10320
> URL: https://issues.apache.org/jira/browse/YARN-10320
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Attachments: YARN-10320-001.patch, YARN-10320-002.patch
>
>
> Have observed Log Aggregation code has used FSDataInputStream#read instead of
> readFully in multiple places like below. One of the place is fixed by
> YARN-8106.
> This Jira targets to fix at all other places.
> LogAggregationIndexedFileController#loadUUIDFromLogFile
> {code}
> byte[] b = new byte[uuid.length];
> int actual = fsDataInputStream.read(b);
> if (actual != uuid.length || Arrays.equals(b, uuid)) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11192) TestRouterWebServicesREST failing after YARN-9827
[
https://issues.apache.org/jira/browse/YARN-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-11192:
--
Labels: pull-request-available (was: )
> TestRouterWebServicesREST failing after YARN-9827
> -
>
> Key: YARN-11192
> URL: https://issues.apache.org/jira/browse/YARN-11192
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In YARN-9827, the following modifications:
> {code:java}
> GenericExceptionHandler should respond with SERVICE_UNAVAILABLE in case of
> connection and service unavailable exception instead of
> INTERNAL_SERVICE_ERROR. {code}
> This modification caused all of YARN Federation's TestRouterWebServicesREST
> unit tests to fail
> {code:java}
> [ERROR] Tests run: 201, Failures: 15, Errors: 0, Skipped: 0, Flakes: 2
> .
> [ERROR]
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST.testUpdateAppStateXML(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST)
> [ERROR] Run 1: TestRouterWebServicesREST.testUpdateAppStateXML:774
> expected:<500> but was:<503>
> [ERROR] Run 2: TestRouterWebServicesREST.testUpdateAppStateXML:774
> expected:<500> but was:<503>
> [ERROR] Run 3: TestRouterWebServicesREST.testUpdateAppStateXML:774
> expected:<500> but was:<503> {code}
> Report-URL:
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4464/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11192) TestRouterWebServicesREST failing after YARN-9827
fanshilun created YARN-11192:
Summary: TestRouterWebServicesREST failing after YARN-9827
Key: YARN-11192
URL: https://issues.apache.org/jira/browse/YARN-11192
Project: Hadoop YARN
Issue Type: Bug
Components: federation
Affects Versions: 3.4.0
Reporter: fanshilun
Assignee: fanshilun
In YARN-9827, the following modifications:
{code:java}
GenericExceptionHandler should respond with SERVICE_UNAVAILABLE in case of
connection and service unavailable exception instead of INTERNAL_SERVICE_ERROR.
{code}
This modification caused all of YARN Federation's TestRouterWebServicesREST
unit tests to fail
{code:java}
[ERROR] Tests run: 201, Failures: 15, Errors: 0, Skipped: 0, Flakes: 2
.
[ERROR]
org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST.testUpdateAppStateXML(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST)
[ERROR] Run 1: TestRouterWebServicesREST.testUpdateAppStateXML:774
expected:<500> but was:<503>
[ERROR] Run 2: TestRouterWebServicesREST.testUpdateAppStateXML:774
expected:<500> but was:<503>
[ERROR] Run 3: TestRouterWebServicesREST.testUpdateAppStateXML:774
expected:<500> but was:<503> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11192) TestRouterWebServicesREST failing after YARN-9827
[
https://issues.apache.org/jira/browse/YARN-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fanshilun updated YARN-11192:
-
Description:
In YARN-9827, the following modifications:
{code:java}
GenericExceptionHandler should respond with SERVICE_UNAVAILABLE in case of
connection and service unavailable exception instead of INTERNAL_SERVICE_ERROR.
{code}
This modification caused all of YARN Federation's TestRouterWebServicesREST
unit tests to fail
{code:java}
[ERROR] Tests run: 201, Failures: 15, Errors: 0, Skipped: 0, Flakes: 2
.
[ERROR]
org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST.testUpdateAppStateXML(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST)
[ERROR] Run 1: TestRouterWebServicesREST.testUpdateAppStateXML:774
expected:<500> but was:<503>
[ERROR] Run 2: TestRouterWebServicesREST.testUpdateAppStateXML:774
expected:<500> but was:<503>
[ERROR] Run 3: TestRouterWebServicesREST.testUpdateAppStateXML:774
expected:<500> but was:<503> {code}
Report-URL:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4464/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
was:
In YARN-9827, the following modifications:
{code:java}
GenericExceptionHandler should respond with SERVICE_UNAVAILABLE in case of
connection and service unavailable exception instead of INTERNAL_SERVICE_ERROR.
{code}
This modification caused all of YARN Federation's TestRouterWebServicesREST
unit tests to fail
{code:java}
[ERROR] Tests run: 201, Failures: 15, Errors: 0, Skipped: 0, Flakes: 2
.
[ERROR]
org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST.testUpdateAppStateXML(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST)
[ERROR] Run 1: TestRouterWebServicesREST.testUpdateAppStateXML:774
expected:<500> but was:<503>
[ERROR] Run 2: TestRouterWebServicesREST.testUpdateAppStateXML:774
expected:<500> but was:<503>
[ERROR] Run 3: TestRouterWebServicesREST.testUpdateAppStateXML:774
expected:<500> but was:<503> {code}
> TestRouterWebServicesREST failing after YARN-9827
> -
>
> Key: YARN-11192
> URL: https://issues.apache.org/jira/browse/YARN-11192
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
>
> In YARN-9827, the following modifications:
> {code:java}
> GenericExceptionHandler should respond with SERVICE_UNAVAILABLE in case of
> connection and service unavailable exception instead of
> INTERNAL_SERVICE_ERROR. {code}
> This modification caused all of YARN Federation's TestRouterWebServicesREST
> unit tests to fail
> {code:java}
> [ERROR] Tests run: 201, Failures: 15, Errors: 0, Skipped: 0, Flakes: 2
> .
> [ERROR]
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST.testUpdateAppStateXML(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST)
> [ERROR] Run 1: TestRouterWebServicesREST.testUpdateAppStateXML:774
> expected:<500> but was:<503>
> [ERROR] Run 2: TestRouterWebServicesREST.testUpdateAppStateXML:774
> expected:<500> but was:<503>
> [ERROR] Run 3: TestRouterWebServicesREST.testUpdateAppStateXML:774
> expected:<500> but was:<503> {code}
> Report-URL:
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4464/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] (YARN-5764) NUMA awareness support for launching containers
[ https://issues.apache.org/jira/browse/YARN-5764 ] Samrat Deb deleted comment on YARN-5764: -- was (Author: samrat007): hi [~devaraj] , [~cheersyang] , The patch enabled only for LinuxContainerExecutor (which is for secure) DefaultContainerExecutor is for unsecure which does not have any code change. Does that mean that numa enable and actually this patch will work on the secure cluster ? Even if we configure numa , it wont use the numa awareness if the cluster is not secure ? > NUMA awareness support for launching containers > --- > > Key: YARN-5764 > URL: https://issues.apache.org/jira/browse/YARN-5764 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, yarn >Reporter: Olasoji >Assignee: Devaraj Kavali >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: NUMA Awareness for YARN Containers.pdf, NUMA Performance > Results.pdf, YARN-5764-v0.patch, YARN-5764-v1.patch, YARN-5764-v10.patch, > YARN-5764-v11.patch, YARN-5764-v2.patch, YARN-5764-v3.patch, > YARN-5764-v4.patch, YARN-5764-v5.patch, YARN-5764-v6.patch, > YARN-5764-v7.patch, YARN-5764-v8.patch, YARN-5764-v9.patch > > > The purpose of this feature is to improve Hadoop performance by minimizing > costly remote memory accesses on non SMP systems. Yarn containers, on launch, > will be pinned to a specific NUMA node and all subsequent memory allocations > will be served by the same node, reducing remote memory accesses. The current > default behavior is to spread memory across all NUMA nodes. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5764) NUMA awareness support for launching containers
[ https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556772#comment-17556772 ] Samrat Deb commented on YARN-5764: -- hi [~devaraj] , [~cheersyang] , The patch enabled only for LinuxContainerExecutor (which is for secure) DefaultContainerExecutor is for unsecure which does not have any code change. Does that mean that numa enable and actually this patch will work on the secure cluster ? Even if we configure numa , it wont use the numa awareness if the cluster is not secure ? > NUMA awareness support for launching containers > --- > > Key: YARN-5764 > URL: https://issues.apache.org/jira/browse/YARN-5764 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, yarn >Reporter: Olasoji >Assignee: Devaraj Kavali >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: NUMA Awareness for YARN Containers.pdf, NUMA Performance > Results.pdf, YARN-5764-v0.patch, YARN-5764-v1.patch, YARN-5764-v10.patch, > YARN-5764-v11.patch, YARN-5764-v2.patch, YARN-5764-v3.patch, > YARN-5764-v4.patch, YARN-5764-v5.patch, YARN-5764-v6.patch, > YARN-5764-v7.patch, YARN-5764-v8.patch, YARN-5764-v9.patch > > > The purpose of this feature is to improve Hadoop performance by minimizing > costly remote memory accesses on non SMP systems. Yarn containers, on launch, > will be pinned to a specific NUMA node and all subsequent memory allocations > will be served by the same node, reducing remote memory accesses. The current > default behavior is to spread memory across all NUMA nodes. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
