Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/ [Aug 3, 2022, 1:44:39 AM] (noreply) HADOOP-18301. Upgrade commons-io to 2.11.0 (#4455) [Aug 3, 2022, 7:31:41 AM] (noreply) HDFS-16712. Fix incorrect placeholder in DataNode.java (#4672). Contributed by ZanderXu. [Aug 3, 2022, 4:14:11 PM] (noreply) HDFS-16687. RouterFsckServlet replicates code from DfsServlet base class (#4681) [Aug 3, 2022, 4:35:40 PM] (noreply) YARN-6972. Adding RM ClusterId in AppInfo. (#4673) [Aug 3, 2022, 4:38:00 PM] (noreply) YARN-11029. Refactor AMRMProxy Service code and Added Some Metrics. (#4650) [Aug 3, 2022, 6:21:48 PM] (noreply) YARN-11230. [Federation] Add getContainer, signalToContainer REST APIs for Router. (#4689) [Aug 3, 2022, 10:38:04 PM] (noreply) HADOOP-18355. Update previous index properly while validating overlapping ranges. (#4647) -1 overall The following subsystems voted -1: blanks pathlen xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/artifact/out/results-compile-javac-root.txt [536K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/artifact/out/blanks-eol.txt [14M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/942/artifact/out/results-javadoc-javadoc-root.txt [400K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11242) New Map Reduce Example - Simple Sentiment Analysis
Meetu Patel created YARN-11242: -- Summary: New Map Reduce Example - Simple Sentiment Analysis Key: YARN-11242 URL: https://issues.apache.org/jira/browse/YARN-11242 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 3.4.0 Reporter: Meetu Patel Fix For: 3.4.0 Attachments: sample_data.txt, sample_words.txt I am looking to add a new map reduce example, i.e, sentiment analysis. Sentiment analysis map reduce job helps in determining the sentiment score for a user. It takes each tweet made by an user and assigns a sentiment score for that tweet/sentence for a particular user and then aggregates the sentiment scores for all tweets made by all users. This example takes the twitter dataset which contains users and the tweets made by users and gives the output as . For each user, the sentiment score is calculated for all the tweets made by that particular user. This mapreduce examples takes in two input files - input twitter dataset and a file containing list of words. The word list file contains positive, negative and negation words which are used to give a sentiment score to the words in tweets. You can use command: bin/hadoop jar /HADOOP_PATH/share/hadoop/mapreduce/mapreduce-examples.jar sentimentanalysis For example, you can use the sample files and run the above command as: bin/hadoop jar /HADOOP_PATH/share/hadoop/mapreduce/mapreduce-examples.jar sentimentanalysis sample_data.txt sample_words.txt -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.4
+1 (Sorry, took me a while) Ran: ./dev-support/hadoop-vote.sh --source https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/ * Signature: ok * Checksum : failed * Rat check (17.0.1): ok - mvn clean apache-rat:check * Built from source (17.0.1): ok - mvn clean install -DskipTests * Built tar from source (17.0.1): ok - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true Took a look at website. Home page says stuff like, “ARM Support: This is the first release to support ARM architectures.“, which I don’t think is true of 3.3.4 but otherwise, looks fine. Only played with HDFS. UIs looked right. Deployed to ten node arm64 cluster. Ran the hbase verification job on top of it and all passed. Did some kills, stuff came back. I didn't spend time on unit tests but one set passed on a local rig here: [image: image.png] Stack On Fri, Jul 29, 2022 at 11:48 AM Steve Loughran wrote: > I have put together a release candidate (RC1) for Hadoop 3.3.4 > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/ > > The git tag is release-3.3.4-RC1, commit a585a73c3e0 > > The maven artifacts are staged at > https://repository.apache.org/content/repositories/orgapachehadoop-1358/ > > You can find my public key at: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > Change log > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md > > Release notes > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md > > There's a very small number of changes, primarily critical code/packaging > issues and security fixes. > > See the release notes for details. > > Please try the release and vote. The vote will run for 5 days. > > steve >
Re: [VOTE] Release Apache Hadoop 3.3.4
I'm still trying to get a successful run from Spark side here: https://github.com/apache/spark/pull/37281. The last CI run failed, but most likely due to some issue not related to Hadoop. On Thu, Aug 4, 2022 at 10:27 AM Steve Loughran wrote: > > The vote passed with the following result. > > Binding PMC members: > > +1 Chris Nauroth > +1 Steve Loughran > +1 Masatake Iwasaki > > Non binding votes: > > +1 Ashutosh Gupta > > Cheng Pan was worried about the new transient kotlin dependency. They are > working on a PR there which we can target for the successor to this release > https://issues.apache.org/jira/browse/HDFS-16714 > > I'm going to publish the artifacts, site docs, maven artifacts, then > announce. > > Thank you to all who helped to push this release out. > > > On Thu, 4 Aug 2022 at 11:48, Ashutosh Gupta > wrote: > > > +1 (non-binding) > > > > * Builds from source looks good. > > * Checksums and signatures are correct. > > * Running basic HDFS commands and running simple MapReduce jobs looks good. > > * Skimmed through the contents of site documentation and it looks good. > > > > Thanks Steve for driving this release. > > > > Ashutosh > > > > > > On Wed, Aug 3, 2022 at 9:39 PM Chris Nauroth wrote: > > > > > +1 (binding) > > > > > > * Verified all checksums. > > > * Verified all signatures. > > > * Built from source, including native code on Linux. > > > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy > > > -Drequire.zstd -DskipTests > > > * Tests passed. > > > * mvn --fail-never clean test -Pnative -Dparallel-tests > > > -Drequire.snappy -Drequire.zstd -Drequire.openssl > > > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > > > * Checked dependency tree to make sure we have all of the expected > > library > > > updates that are mentioned in the release notes. > > > * mvn -o dependency:tree > > > > > > I saw a LibHDFS test failure, but I know it's something flaky that's > > > already tracked in a JIRA issue. The release looks good. Steve, thank you > > > for driving this. > > > > > > Chris Nauroth > > > > > > > > > On Wed, Aug 3, 2022 at 11:27 AM Steve Loughran > > > > > > > > wrote: > > > > > > > my vote for this is +1, binding. > > > > > > > > obviously I`m biased, but i do not want to have to issue any more > > interim > > > > releases before the feature release off branch-3.3, so I am trying to > > be > > > > ruthless. > > > > > > > > my client vaidator ant project has a more targets to help with > > releasing, > > > > and now builds a lot mor of my local projects > > > > https://github.com/steveloughran/validate-hadoop-client-artifacts > > > > all good as far as my test coverage goes, with these projects > > validating > > > > the staged dependencies. > > > > > > > > now, who else can review > > > > > > > > On Fri, 29 Jul 2022 at 19:47, Steve Loughran > > > wrote: > > > > > > > > > > > > > > > > > > > I have put together a release candidate (RC1) for Hadoop 3.3.4 > > > > > > > > > > The RC is available at: > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/ > > > > > > > > > > The git tag is release-3.3.4-RC1, commit a585a73c3e0 > > > > > > > > > > The maven artifacts are staged at > > > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1358/ > > > > > > > > > > You can find my public key at: > > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > > > Change log > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md > > > > > > > > > > Release notes > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md > > > > > > > > > > There's a very small number of changes, primarily critical > > > code/packaging > > > > > issues and security fixes. > > > > > > > > > > See the release notes for details. > > > > > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > > > > > steve > > > > > > > > > > > > > > - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.4
The vote passed with the following result. Binding PMC members: +1 Chris Nauroth +1 Steve Loughran +1 Masatake Iwasaki Non binding votes: +1 Ashutosh Gupta Cheng Pan was worried about the new transient kotlin dependency. They are working on a PR there which we can target for the successor to this release https://issues.apache.org/jira/browse/HDFS-16714 I'm going to publish the artifacts, site docs, maven artifacts, then announce. Thank you to all who helped to push this release out. On Thu, 4 Aug 2022 at 11:48, Ashutosh Gupta wrote: > +1 (non-binding) > > * Builds from source looks good. > * Checksums and signatures are correct. > * Running basic HDFS commands and running simple MapReduce jobs looks good. > * Skimmed through the contents of site documentation and it looks good. > > Thanks Steve for driving this release. > > Ashutosh > > > On Wed, Aug 3, 2022 at 9:39 PM Chris Nauroth wrote: > > > +1 (binding) > > > > * Verified all checksums. > > * Verified all signatures. > > * Built from source, including native code on Linux. > > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy > > -Drequire.zstd -DskipTests > > * Tests passed. > > * mvn --fail-never clean test -Pnative -Dparallel-tests > > -Drequire.snappy -Drequire.zstd -Drequire.openssl > > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > > * Checked dependency tree to make sure we have all of the expected > library > > updates that are mentioned in the release notes. > > * mvn -o dependency:tree > > > > I saw a LibHDFS test failure, but I know it's something flaky that's > > already tracked in a JIRA issue. The release looks good. Steve, thank you > > for driving this. > > > > Chris Nauroth > > > > > > On Wed, Aug 3, 2022 at 11:27 AM Steve Loughran > > > > > wrote: > > > > > my vote for this is +1, binding. > > > > > > obviously I`m biased, but i do not want to have to issue any more > interim > > > releases before the feature release off branch-3.3, so I am trying to > be > > > ruthless. > > > > > > my client vaidator ant project has a more targets to help with > releasing, > > > and now builds a lot mor of my local projects > > > https://github.com/steveloughran/validate-hadoop-client-artifacts > > > all good as far as my test coverage goes, with these projects > validating > > > the staged dependencies. > > > > > > now, who else can review > > > > > > On Fri, 29 Jul 2022 at 19:47, Steve Loughran > > wrote: > > > > > > > > > > > > > > > I have put together a release candidate (RC1) for Hadoop 3.3.4 > > > > > > > > The RC is available at: > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/ > > > > > > > > The git tag is release-3.3.4-RC1, commit a585a73c3e0 > > > > > > > > The maven artifacts are staged at > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1358/ > > > > > > > > You can find my public key at: > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > Change log > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md > > > > > > > > Release notes > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md > > > > > > > > There's a very small number of changes, primarily critical > > code/packaging > > > > issues and security fixes. > > > > > > > > See the release notes for details. > > > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > > > steve > > > > > > > > > >
[jira] [Created] (YARN-11241) Add uncleaning option for local app log file with log-aggregation enabled
groot created YARN-11241: Summary: Add uncleaning option for local app log file with log-aggregation enabled Key: YARN-11241 URL: https://issues.apache.org/jira/browse/YARN-11241 Project: Hadoop YARN Issue Type: New Feature Components: log-aggregation Reporter: groot Assignee: groot Add uncleaning option for local app log file with log-aggregation enabled -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11235) Refactor Policy Code and Define getReservationHomeSubcluster
[ https://issues.apache.org/jira/browse/YARN-11235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved YARN-11235. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Refactor Policy Code and Define getReservationHomeSubcluster > > > Key: YARN-11235 > URL: https://issues.apache.org/jira/browse/YARN-11235 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: [RESERVATION] Add support for reservation-based > routing.pdf > > Time Spent: 8h 20m > Remaining Estimate: 0h > > Refer to 2.1 Router Policy, which describes the changes to be made. The > documentation will continue to improve, the current version is V1. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.4
+1 (non-binding) * Builds from source looks good. * Checksums and signatures are correct. * Running basic HDFS commands and running simple MapReduce jobs looks good. * Skimmed through the contents of site documentation and it looks good. Thanks Steve for driving this release. Ashutosh On Wed, Aug 3, 2022 at 9:39 PM Chris Nauroth wrote: > +1 (binding) > > * Verified all checksums. > * Verified all signatures. > * Built from source, including native code on Linux. > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy > -Drequire.zstd -DskipTests > * Tests passed. > * mvn --fail-never clean test -Pnative -Dparallel-tests > -Drequire.snappy -Drequire.zstd -Drequire.openssl > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > * Checked dependency tree to make sure we have all of the expected library > updates that are mentioned in the release notes. > * mvn -o dependency:tree > > I saw a LibHDFS test failure, but I know it's something flaky that's > already tracked in a JIRA issue. The release looks good. Steve, thank you > for driving this. > > Chris Nauroth > > > On Wed, Aug 3, 2022 at 11:27 AM Steve Loughran > > wrote: > > > my vote for this is +1, binding. > > > > obviously I`m biased, but i do not want to have to issue any more interim > > releases before the feature release off branch-3.3, so I am trying to be > > ruthless. > > > > my client vaidator ant project has a more targets to help with releasing, > > and now builds a lot mor of my local projects > > https://github.com/steveloughran/validate-hadoop-client-artifacts > > all good as far as my test coverage goes, with these projects validating > > the staged dependencies. > > > > now, who else can review > > > > On Fri, 29 Jul 2022 at 19:47, Steve Loughran > wrote: > > > > > > > > > > > I have put together a release candidate (RC1) for Hadoop 3.3.4 > > > > > > The RC is available at: > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/ > > > > > > The git tag is release-3.3.4-RC1, commit a585a73c3e0 > > > > > > The maven artifacts are staged at > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1358/ > > > > > > You can find my public key at: > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > Change log > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md > > > > > > Release notes > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md > > > > > > There's a very small number of changes, primarily critical > code/packaging > > > issues and security fixes. > > > > > > See the release notes for details. > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > steve > > > > > >
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.TestRollingUpgrade hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.TestEncryptionZonesWithKMS hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.yarn.sls.appmaster.TestAMSimulator hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/diff-compile-javac-root.txt [508K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/patch-mvnsite-root.txt [692K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [244K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [436K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [76K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/743/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [128K]
[jira] [Resolved] (YARN-4741) RM is flooded with RMNodeFinishedContainersPulledByAMEvents in the async dispatcher event queue
[
https://issues.apache.org/jira/browse/YARN-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fanshilun resolved YARN-4741.
-
Resolution: Duplicate
> RM is flooded with RMNodeFinishedContainersPulledByAMEvents in the async
> dispatcher event queue
> ---
>
> Key: YARN-4741
> URL: https://issues.apache.org/jira/browse/YARN-4741
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Priority: Critical
> Attachments: nm.log
>
>
> We had a pretty major incident with the RM where it was continually flooded
> with RMNodeFinishedContainersPulledByAMEvents in the async dispatcher event
> queue.
> In our setup, we had the RM HA or stateful restart *disabled*, but NM
> work-preserving restart *enabled*. Due to other issues, we did a cluster-wide
> NM restart.
> Some time during the restart (which took multiple hours), we started seeing
> the async dispatcher event queue building. Normally it would log 1,000. In
> this case, it climbed all the way up to tens of millions of events.
> When we looked at the RM log, it was full of the following messages:
> {noformat}
> 2016-02-18 01:47:29,530 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Invalid
> event FINISHED_CONTAINERS_PULLED_BY_AM on Node worker-node-foo.bar.net:8041
> 2016-02-18 01:47:29,535 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Can't handle
> this event at current state
> 2016-02-18 01:47:29,535 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Invalid
> event FINISHED_CONTAINERS_PULLED_BY_AM on Node worker-node-foo.bar.net:8041
> 2016-02-18 01:47:29,538 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Can't handle
> this event at current state
> 2016-02-18 01:47:29,538 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Invalid
> event FINISHED_CONTAINERS_PULLED_BY_AM on Node worker-node-foo.bar.net:8041
> {noformat}
> And that node in question was restarted a few minutes earlier.
> When we inspected the RM heap, it was full of
> RMNodeFinishedContainersPulledByAMEvents.
> Suspecting the NM work-preserving restart, we disabled it and did another
> cluster-wide rolling restart. Initially that seemed to have helped reduce the
> queue size, but the queue built back up to several millions and continued for
> an extended period. We had to restart the RM to resolve the problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
