Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2

2019-08-21 Thread Jim Brennan
ritize this. Hoping to get to this next >> week. >> >> Jonathan >> >> -- >> *From:* Jim Brennan >> *Sent:* Thursday, April 18, 2019 7:28 AM >> *To:* Jonathan Hung >> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.

Re: [VOTE] Merge YARN-8200 to branch-2 and branch-3.0

2019-08-26 Thread Jim Brennan
+1 (non-binding). I have built branch-2 with the latest YARN-8200 patch (YARN-8200-branch-2.003.patch). I ran all of the NM/RM tests and ran a few test jobs on a one-node cluster with default settings. On Mon, Aug 26, 2019 at 3:51 PM Oliver Hu wrote: > +1 (non-binding) > > We have used this pa

Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)

2019-10-25 Thread Jim Brennan
+1 (non-binding) on RC1 I built from source on Mac and RHEL7, ran hdfs, nodemanager, and resourcemanager unit tests, and set up a one-node cluster and ran some test jobs (pi and sleep). - Jim On Tue, Oct 22, 2019 at 4:55 PM Jonathan Hung wrote: > Hi folks, > > This is the second release candida

Re: [DISCUSS] Making 2.10 the last minor 2.x release

2019-12-31 Thread Jim Brennan
It looks like QBT tests are still being run on branch-2 ( https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/), and they are not very helpful at this point. Can we change the QBT tests to run against branch-2.10 instead? Jim On Mon, Dec 23, 2019 at 7:44 PM Akira

Re: [E] Re: [DISCUSS] Hadoop 2.10.1 release

2020-09-02 Thread Jim Brennan
Thanks Masatake Iwasaki! I am willing to help out with Hadoop 2.10.1 release. Jim Brennan On Tue, Sep 1, 2020 at 2:13 AM Masatake Iwasaki wrote: > Thanks, Mingliang Liu. > > I volunteer to take the RM role then. > I will appreciate advice from who have the experience. > >

Re: [E] [VOTE] Release Apache Hadoop 2.10.1 (RC0)

2020-09-16 Thread Jim Brennan
Thanks for your work on this Masatake! I am +1 (non-binding) on this 2.10.1 release. I built from source and ran hdfs, resourcemanager, and nodemanager unit tests. I set up a one-node-cluster and ran some example jobs (pi, sleep). I tested NM/RM recovery by killing NM/RM during jobs and verifying

Re: [E] Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4

2020-12-21 Thread Jim Brennan
I put up a patch for https://issues.apache.org/jira/browse/YARN-10540. Thanks for bringing it to my attention. Jim On Mon, Dec 21, 2020 at 10:36 AM Sunil Govindan wrote: > I had some offline talks with a few folks. > This issue is happening only in Mac, hence ideally it does not cause much > of

Re: [E] Re: Java 8 Lambdas

2021-04-29 Thread Jim Brennan
I just think that we should be cognizant of changes (particularly bug fixes), that will need to be ported to branch-2.10. Since it is still on Java7, anytime you use a lambda in code on trunk, we need to change it for branch-2.10. While not difficult, this is extra work and it increases the diff

Re: [E] [VOTE] Hadoop 3.1.x EOL

2021-06-03 Thread Jim Brennan
+1 On Thu, Jun 3, 2021 at 1:14 AM Akira Ajisaka wrote: > Dear Hadoop developers, > > Given the feedback from the discussion thread [1], I'd like to start > an official vote > thread for the community to vote and start the 3.1 EOL process. > > What this entails: > > (1) an official announcement

Re: [ANNOUNCE] Eric Badger is now a committer!

2019-03-05 Thread Jim Brennan
Congratulations Eric! On Tue, Mar 5, 2019 at 11:20 AM Eric Payne wrote: > It is my pleasure to announce that Eric Badger has accepted an invitation > to become a Hadoop Core committer. > > Congratulations, Eric! This is well-deserved! > > -Eric Payne >

Re: [DISCUSS] Docker build process

2019-03-19 Thread Jim Brennan
I agree with Steve and Marton. I am ok with having the docker build as an option, but I don't want it to be the default. Jim On Tue, Mar 19, 2019 at 12:19 PM Eric Yang wrote: > Hi Marton, > > Thank you for your input. I agree with most of what you said with a few > exceptions. Security fix

Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2

2019-04-02 Thread Jim Brennan
Thanks for working on this! One concern for us is support for a rolling upgrade. If we are running a cluster based on branch-2.8, will we be able to do a rolling upgrade (no cluster down-time) to a branch containing these changes? Have you tested rolling upgrades? Thanks. Jim On Fri, Mar 29, 20

Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2

2019-04-18 Thread Jim Brennan
ing an > issue, but we’ll try it out and report back. > > Jonathan > > ------ > *From:* Jim Brennan > *Sent:* Tuesday, April 2, 2019 9:17 AM > *To:* Jonathan Hung > *Cc:* yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org > *Subject:* Re: [

[jira] [Created] (YARN-9844) TestCapacitySchedulerPerf test errors in branch-2

2019-09-19 Thread Jim Brennan (Jira)
Jim Brennan created YARN-9844: - Summary: TestCapacitySchedulerPerf test errors in branch-2 Key: YARN-9844 URL: https://issues.apache.org/jira/browse/YARN-9844 Project: Hadoop YARN Issue Type

[jira] [Resolved] (YARN-9906) When setting multi volumes throurh the "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS" setting is not valid

2019-10-16 Thread Jim Brennan (Jira)
[ https://issues.apache.org/jira/browse/YARN-9906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan resolved YARN-9906. --- Resolution: Invalid > When setting multi volumes throurh the "YARN_CONTAINER_RUNTIME_DOCKE

[jira] [Created] (YARN-9914) Use separate configs for free disk space checking for full and not-full disks

2019-10-18 Thread Jim Brennan (Jira)
Jim Brennan created YARN-9914: - Summary: Use separate configs for free disk space checking for full and not-full disks Key: YARN-9914 URL: https://issues.apache.org/jira/browse/YARN-9914 Project: Hadoop

[jira] [Created] (YARN-10072) TestCSAllocateCustomResource failures

2020-01-07 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10072: -- Summary: TestCSAllocateCustomResource failures Key: YARN-10072 URL: https://issues.apache.org/jira/browse/YARN-10072 Project: Hadoop YARN Issue Type: Test

[jira] [Created] (YARN-10161) TestRouterWebServicesREST is corrupting STDOUT

2020-02-24 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10161: -- Summary: TestRouterWebServicesREST is corrupting STDOUT Key: YARN-10161 URL: https://issues.apache.org/jira/browse/YARN-10161 Project: Hadoop YARN Issue Type

[jira] [Created] (YARN-10227) Pull YARN-8242 back to branch-2.10

2020-04-08 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10227: -- Summary: Pull YARN-8242 back to branch-2.10 Key: YARN-10227 URL: https://issues.apache.org/jira/browse/YARN-10227 Project: Hadoop YARN Issue Type: Bug

[jira] [Created] (YARN-10312) Add support for yarn logs -logFile to retain backward compatibility

2020-06-11 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10312: -- Summary: Add support for yarn logs -logFile to retain backward compatibility Key: YARN-10312 URL: https://issues.apache.org/jira/browse/YARN-10312 Project: Hadoop YARN

[jira] [Created] (YARN-10348) Allow RM to always cancel tokens after app completes

2020-07-08 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10348: -- Summary: Allow RM to always cancel tokens after app completes Key: YARN-10348 URL: https://issues.apache.org/jira/browse/YARN-10348 Project: Hadoop YARN Issue

[jira] [Created] (YARN-10353) Log vcores used and cumulative cpu in containers monitor

2020-07-16 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10353: -- Summary: Log vcores used and cumulative cpu in containers monitor Key: YARN-10353 URL: https://issues.apache.org/jira/browse/YARN-10353 Project: Hadoop YARN

[jira] [Created] (YARN-10363) TestRMAdminCLI.testHelp is failing in branch-2.10

2020-07-22 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10363: -- Summary: TestRMAdminCLI.testHelp is failing in branch-2.10 Key: YARN-10363 URL: https://issues.apache.org/jira/browse/YARN-10363 Project: Hadoop YARN Issue Type

[jira] [Created] (YARN-10369) Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG

2020-07-27 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10369: -- Summary: Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG Key: YARN-10369 URL: https://issues.apache.org/jira/browse/YARN-10369 Project: Hadoop YARN

[jira] [Created] (YARN-10450) Add cpu and memory utilization per node and cluster-wide metrics

2020-09-29 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10450: -- Summary: Add cpu and memory utilization per node and cluster-wide metrics Key: YARN-10450 URL: https://issues.apache.org/jira/browse/YARN-10450 Project: Hadoop YARN

[jira] [Created] (YARN-10475) Scale RM-NM heartbeat interval based on node utilization

2020-10-27 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10475: -- Summary: Scale RM-NM heartbeat interval based on node utilization Key: YARN-10475 URL: https://issues.apache.org/jira/browse/YARN-10475 Project: Hadoop YARN

[jira] [Created] (YARN-10477) runc launch failure should not cause nodemanager to go unhealthy

2020-10-28 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10477: -- Summary: runc launch failure should not cause nodemanager to go unhealthy Key: YARN-10477 URL: https://issues.apache.org/jira/browse/YARN-10477 Project: Hadoop YARN

[jira] [Resolved] (YARN-10477) runc launch failure should not cause nodemanager to go unhealthy

2020-10-28 Thread Jim Brennan (Jira)
[ https://issues.apache.org/jira/browse/YARN-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan resolved YARN-10477. Resolution: Invalid Closing this as invalid. The problem was only there in our internal version

[jira] [Created] (YARN-10478) Make RM-NM heartbeat scaling calculator pluggable

2020-11-02 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10478: -- Summary: Make RM-NM heartbeat scaling calculator pluggable Key: YARN-10478 URL: https://issues.apache.org/jira/browse/YARN-10478 Project: Hadoop YARN Issue Type

[jira] [Created] (YARN-10479) RMProxy should retry on SocketTimeout Exceptions

2020-11-02 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10479: -- Summary: RMProxy should retry on SocketTimeout Exceptions Key: YARN-10479 URL: https://issues.apache.org/jira/browse/YARN-10479 Project: Hadoop YARN Issue Type

[jira] [Resolved] (YARN-10485) TimelineConnector swallows InterruptedException

2020-11-13 Thread Jim Brennan (Jira)
[ https://issues.apache.org/jira/browse/YARN-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan resolved YARN-10485. Resolution: Fixed > TimelineConnector swallows InterruptedExcept

[jira] [Resolved] (YARN-10485) TimelineConnector swallows InterruptedException

2020-11-16 Thread Jim Brennan (Jira)
[ https://issues.apache.org/jira/browse/YARN-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan resolved YARN-10485. Fix Version/s: 3.2.3 3.4.1 3.1.5 3.3.1

[jira] [Created] (YARN-10542) Node Utilization on UI is misleading if nodes don't report utilization

2020-12-21 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10542: -- Summary: Node Utilization on UI is misleading if nodes don't report utilization Key: YARN-10542 URL: https://issues.apache.org/jira/browse/YARN-10542 Project: H

[jira] [Created] (YARN-10562) Alternate fix for DirectoryCollection.checkDirs() race

2021-01-06 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10562: -- Summary: Alternate fix for DirectoryCollection.checkDirs() race Key: YARN-10562 URL: https://issues.apache.org/jira/browse/YARN-10562 Project: Hadoop YARN Issue

[jira] [Resolved] (YARN-5853) TestDelegationTokenRenewer#testRMRestartWithExpiredToken fails intermittently on Power

2021-02-11 Thread Jim Brennan (Jira)
[ https://issues.apache.org/jira/browse/YARN-5853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan resolved YARN-5853. --- Resolution: Duplicate This is fixed by YARN-10500 > TestDelegationTokenRene

[jira] [Created] (YARN-10664) Allow parameter expansion in NM_ADMIN_USER_ENV

2021-03-02 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10664: -- Summary: Allow parameter expansion in NM_ADMIN_USER_ENV Key: YARN-10664 URL: https://issues.apache.org/jira/browse/YARN-10664 Project: Hadoop YARN Issue Type

[jira] [Created] (YARN-10665) TestContainerManagerRecover sometimes fails

2021-03-02 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10665: -- Summary: TestContainerManagerRecover sometimes fails Key: YARN-10665 URL: https://issues.apache.org/jira/browse/YARN-10665 Project: Hadoop YARN Issue Type

[jira] [Created] (YARN-10702) Add cluster metric for amount of CPU used by RM Event Processor

2021-03-17 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10702: -- Summary: Add cluster metric for amount of CPU used by RM Event Processor Key: YARN-10702 URL: https://issues.apache.org/jira/browse/YARN-10702 Project: Hadoop YARN

[jira] [Resolved] (YARN-10733) TimelineService Hbase tests are failing with timeout error on branch-2.10

2021-04-14 Thread Jim Brennan (Jira)
[ https://issues.apache.org/jira/browse/YARN-10733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan resolved YARN-10733. Fix Version/s: 2.10.2 Resolution: Fixed Thanks [~ahussein], I have committed this to branch

[jira] [Created] (YARN-10855) yarn logs cli fails to retrieve logs if any TFile is corrupt or empty

2021-07-15 Thread Jim Brennan (Jira)
Jim Brennan created YARN-10855: -- Summary: yarn logs cli fails to retrieve logs if any TFile is corrupt or empty Key: YARN-10855 URL: https://issues.apache.org/jira/browse/YARN-10855 Project: Hadoop YARN

[jira] [Created] (YARN-7678) Logging of container memory stats is missing in 2.8

2017-12-21 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-7678: - Summary: Logging of container memory stats is missing in 2.8 Key: YARN-7678 URL: https://issues.apache.org/jira/browse/YARN-7678 Project: Hadoop YARN Issue Type

[jira] [Created] (YARN-7857) -fstack-check compilation flag causes binary incompatibility for container-executor between RHEL 6 and RHEL 7

2018-01-30 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-7857: - Summary: -fstack-check compilation flag causes binary incompatibility for container-executor between RHEL 6 and RHEL 7 Key: YARN-7857 URL: https://issues.apache.org/jira/browse/YARN

[jira] [Created] (YARN-8027) Setting hostname of docker container breaks for --net=host in docker 1.13

2018-03-12 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-8027: - Summary: Setting hostname of docker container breaks for --net=host in docker 1.13 Key: YARN-8027 URL: https://issues.apache.org/jira/browse/YARN-8027 Project: Hadoop YARN

[jira] [Created] (YARN-8029) YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS should not use commas as separators

2018-03-14 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-8029: - Summary: YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS should not use commas as separators Key: YARN-8029 URL: https://issues.apache.org/jira/browse/YARN-8029 Project: Hadoop YARN

[jira] [Created] (YARN-8071) Provide Spark-like API for setting Environment Variables to enable vars with commas

2018-03-23 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-8071: - Summary: Provide Spark-like API for setting Environment Variables to enable vars with commas Key: YARN-8071 URL: https://issues.apache.org/jira/browse/YARN-8071 Project

[jira] [Created] (YARN-8444) NodeResourceMonitor crashes on bad swapFree value

2018-06-20 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-8444: - Summary: NodeResourceMonitor crashes on bad swapFree value Key: YARN-8444 URL: https://issues.apache.org/jira/browse/YARN-8444 Project: Hadoop YARN Issue Type

[jira] [Created] (YARN-8515) container-executor can crash with SIGPIPE after nodemanager restart

2018-07-10 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-8515: - Summary: container-executor can crash with SIGPIPE after nodemanager restart Key: YARN-8515 URL: https://issues.apache.org/jira/browse/YARN-8515 Project: Hadoop YARN

[jira] [Created] (YARN-8518) test-container-executor test_is_empty() is broken

2018-07-11 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-8518: - Summary: test-container-executor test_is_empty() is broken Key: YARN-8518 URL: https://issues.apache.org/jira/browse/YARN-8518 Project: Hadoop YARN Issue Type

[jira] [Created] (YARN-8640) Restore previous state in container-executor if write_exit_code_file_as_nm fails

2018-08-09 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-8640: - Summary: Restore previous state in container-executor if write_exit_code_file_as_nm fails Key: YARN-8640 URL: https://issues.apache.org/jira/browse/YARN-8640 Project

[jira] [Created] (YARN-8648) Container cgroups are leaked when using docker

2018-08-10 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-8648: - Summary: Container cgroups are leaked when using docker Key: YARN-8648 URL: https://issues.apache.org/jira/browse/YARN-8648 Project: Hadoop YARN Issue Type: Bug

[jira] [Created] (YARN-8656) container-executor should not write cgroup tasks files for docker containers

2018-08-13 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-8656: - Summary: container-executor should not write cgroup tasks files for docker containers Key: YARN-8656 URL: https://issues.apache.org/jira/browse/YARN-8656 Project: Hadoop

[jira] [Created] (YARN-9442) container working directory has group read permissions

2019-04-04 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-9442: - Summary: container working directory has group read permissions Key: YARN-9442 URL: https://issues.apache.org/jira/browse/YARN-9442 Project: Hadoop YARN Issue

[jira] [Created] (YARN-9527) Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file

2019-05-02 Thread Jim Brennan (JIRA)
Jim Brennan created YARN-9527: - Summary: Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file Key: YARN-9527 URL: https://issues.apache.org/jira/browse/YARN-9527 Project: Hadoop YARN