Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/320/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.util.TestDiskCheckerWithDiskIo hadoop.fs.TestFileUtil hadoop.crypto.key.kms.server.TestKMS hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.tracing.TestTraceAdmin hadoop.hdfs.TestGetBlocks hadoop.hdfs.TestDataTransferKeepalive hadoop.fs.viewfs.TestViewFsWithXAttrs hadoop.hdfs.TestSetrepIncreasing hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.fs.contract.hdfs.TestHDFSContractDelete hadoop.cli.TestAclCLI hadoop.fs.permission.TestStickyBit hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting hadoop.hdfs.TestSnapshotCommands hadoop.hdfs.TestDFSShellGenericOptions hadoop.fs.contract.hdfs.TestHDFSContractSetTimes hadoop.hdfs.server.datanode.TestDataNodeReconfiguration hadoop.hdfs.TestRollingUpgradeDowngrade hadoop.hdfs.server.datanode.TestFsDatasetCache hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling hadoop.hdfs.TestLocalDFS hadoop.hdfs.TestHFlush hadoop.hdfs.TestFileAppend4 hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation hadoop.hdfs.TestFSOutputSummer hadoop.hdfs.TestModTime hadoop.hdfs.TestSafeMode hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer hadoop.hdfs.server.balancer.TestBalancer hadoop.hdfs.TestFileCorruption hadoop.hdfs.TestParallelShortCircuitReadNoChecksum hadoop.hdfs.TestDataStream hadoop.fs.viewfs.TestViewFileSystemWithTruncate hadoop.hdfs.TestAclsEndToEnd hadoop.fs.TestSymlinkHdfsFileContext hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.TestFileConcurrentReader hadoop.hdfs.TestDataTransferProtocol hadoop.hdfs.client.impl.TestClientBlockVerification hadoop.hdfs.TestReplication hadoop.hdfs.TestMaintenanceState hadoop.hdfs.TestFetchImage hadoop.hdfs.tools.TestGetGroups hadoop.fs.viewfs.TestViewFsAtHdfsRoot hadoop.fs.TestSymlinkHdfsFileSystem hadoop.hdfs.TestDFSOutputStream hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.TestRestartDFS hadoop.hdfs.TestEncryptionZonesWithHA hadoop.cli.TestCryptoAdminCLI hadoop.fs.contract.hdfs.TestHDFSContractRename hadoop.hdfs.tools.TestDFSAdminWithHA hadoop.fs.TestWebHdfsFileContextMainOperations hadoop.hdfs.TestSetTimes hadoop.hdfs.TestWriteRead hadoop.fs.TestFcHdfsSetUMask hadoop.fs.viewfs.TestViewFileSystemWithAcls hadoop.hdfs.crypto.TestHdfsCryptoStreams hadoop.hdfs.TestParallelRead hadoop.hdfs.TestRollingUpgrade hadoop.hdfs.TestRenameWhileOpen hadoop.hdfs.TestHDFSTrash hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead hadoop.hdfs.TestMissingBlocksAlert hadoop.hdfs.tools.TestDFSZKFailoverController hadoop.fs.viewfs.TestViewFileSystemWithXAttrs hadoop.hdfs.TestBlockTokenWrappingQOP hadoop.hdfs.TestDatanodeLayoutUpgrade hadoop.hdfs.TestExtendedAcls hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup hadoop.hdfs.TestDFSClientRetries hadoop.hdfs.TestBlockStoragePolicy hadoop.hdfs.server.datanode.TestBlockRecovery hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl hadoop.hdfs.TestReplaceDatanodeFailureReplication hadoop.hdfs.TestMultipleNNPortQOP hadoop.fs.TestEnhancedByteBufferAccess hadoop.fs.loadGenerator.TestLoadGenerator hadoop.hdfs.TestHdfsAdmin hadoop.cli.TestCacheAdminCLI hadoop.hdfs.TestByteBufferPread hadoop.hdfs.TestReservedRawPaths hadoop.hdfs.qjournal.TestSecureNNWithQJM hadoop.hdfs.security.token.block.TestBlockToken hadoop.hdfs.TestSetrepDecreasing hadoop.hdfs.tools.TestDebugAdmin hadoop.hdfs.server.datanode.TestDataNodeVolumeMetrics hadoop.hdfs.TestConnCache hadoop.fs.contract.hdfs.TestHDFSContractMkdir hadoop.hdfs.TestEncryptionZonesWithKMS
[jira] [Resolved] (YARN-10566) Elapsed time should be measured monotonicNow
[
https://issues.apache.org/jira/browse/YARN-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein resolved YARN-10566.
--
Release Note: see discussions in HADOOP-15901
Resolution: Won't Fix
> Elapsed time should be measured monotonicNow
>
>
> Key: YARN-10566
> URL: https://issues.apache.org/jira/browse/YARN-10566
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> I noticed that there is a widespread incorrect usage of
> {{System.currentTimeMillis()}} throughout the yarn code.
> For example:
> {code:java}
> // Some comments here
> long start = System.currentTimeMillis();
> while (System.currentTimeMillis() - start < timeout) {
> // Do something
> }
> {code}
> Elapsed time should be measured using `monotonicNow()`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/ [Jun 3, 2021 8:56:15 AM] (noreply) HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const (#2795) [Jun 3, 2021 9:56:00 AM] (noreply) HADOOP-17152. Provide Hadoop's own Lists utility to reduce dependency on Guava (#3061) [Jun 3, 2021 11:27:36 AM] (Szilard Nemeth) YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if its capacity is 0%. Contributed by Peter Bacsko -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils hadoop.yarn.csi.client.TestCsiClient hadoop.tools.dynamometer.workloadgenerator.TestWorkloadGenerator hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.blockgenerator.TestBlockGen hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.blockgenerator.TestBlockGen hadoop.tools.dynamometer.workloadgenerator.TestWorkloadGenerator cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/results-compile-javac-root.txt [380K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/results-checkstyle-root.txt [16M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/results-javadoc-javadoc-root.txt [408K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-workload.txt [32K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-blockgen.txt [28K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/528/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer.txt [68K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail:
[jira] [Created] (YARN-10807) Parent node labels are incorrectly added to child queues in weight mode
Benjamin Teke created YARN-10807:
Summary: Parent node labels are incorrectly added to child queues
in weight mode
Key: YARN-10807
URL: https://issues.apache.org/jira/browse/YARN-10807
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Benjamin Teke
Assignee: Benjamin Teke
In ParentQueue.updateClusterResource when calculating the normalized weights CS
will iterate through the parent's nodelabels. If the parent has a node label
that a specific child doesn't it will incorrectly added to the child's node
label list through the queueCapacities.setNormalizedWeights(label, weight) call:
{code:java}
// Normalize weight of children
if (getCapacityConfigurationTypeForQueues(childQueues)
== QueueCapacityType.WEIGHT) {
for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
float sumOfWeight = 0;
for (CSQueue queue : childQueues) {
float weight = Math.max(0,
queue.getQueueCapacities().getWeight(nodeLabel));
sumOfWeight += weight;
}
// When sum of weight == 0, skip setting normalized_weight (so
// normalized weight will be 0).
if (Math.abs(sumOfWeight) > 1e-6) {
for (CSQueue queue : childQueues) {
queue.getQueueCapacities().setNormalizedWeight(nodeLabel,
queue.getQueueCapacities().getWeight(nodeLabel) /
sumOfWeight);
}
}
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.1 RC3
Dropped them. Please check again. On Fri, Jun 4, 2021 at 1:19 PM Chao Sun wrote: > Hi Wei-Chiu, > > It seems the Maven staging repository is still pointing at RC0: > https://repository.apache.org/content/repositories/staging/org/apache/hadoop/hadoop-client/3.3.1/, > and it's probably because you need to drop the old RCs in Apache Nexus > server. Could you do that? Thanks. > > Chao > > On Thu, Jun 3, 2021 at 12:52 PM Steve Loughran > wrote: > >> Extend for a bit from the last RC, as it takes time to qualify. >> >> I'm busy testing, doing >> >> - packaging >> - CLI working with abfs and s3, both fs and cloudstore library calls >> - building downstream projects (so validating maven artifacts). cloudstore >> and spark there >> - building downstream of downstream projects, i.e. my spark cloud >> IO/committer test module. Moving to spark 3 cost me the afternoon, not >> through any incompatible changes there but because the upgraded scalatest >> "moved" their foundational FunTest class to a different package and name. >> Not happy with Team Scalatest there. >> - reviewing the docs in the -aws and azure modules to see they link >> together OK. >> >> So far so good. >> >> One troublespot (which isn't any reason to hold up the release), is that >> the table in the directory_markers markdown file doesn't render right. >> Created https://issues.apache.org/jira/browse/HADOOP-17746. >> >> This is *not a blocker* >> >> I can prepare a fix and we can have it in so that if any other changes >> come >> in the page will look OK. >> >> >> >> >> On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang wrote: >> >> > Hello, >> > do we want to extend the release vote? I understand a big release like >> this >> > takes time to validate. >> > >> > I am aware a number of people are testing it: Attila tested Ozone on >> Hadoop >> > 3.3.1 RC3, Stack is testing HBase, Chao tested Spark. >> > I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by >> 20% >> > over Hadoop 3.2 library. >> > >> >> ooh. That'll be from Mukund's listing improvements translating into query >> planning speedups, >> >> Nice >> >> If someone benchmarking this stuff were to enable directory marker >> retention >> fs.s3a.directory.marker.retention=keep , I'd be interested to know how >> much >> speedup that >> delivers on versioned and unversioned buckets. >> >> Unversioned: reduces risk of IO throttling on writes >> Versioned: that and should stop subsequent LIST operations from getting >> slowed down from all the tombstones >> >> >> > >> > Looks like we may need some more time to test. How about extending it >> by a >> > week? >> > >> >> >> That would be good. This week included some holidays for people in the >> US/UK which is why I'm a bit behind on my testing. >> >> >> > >> > >> >
[jira] [Created] (YARN-10806) Dispatcher thread crash caused by NoSuchElementException
chaosju created YARN-10806: -- Summary: Dispatcher thread crash caused by NoSuchElementException Key: YARN-10806 URL: https://issues.apache.org/jira/browse/YARN-10806 Project: Hadoop YARN Issue Type: Bug Reporter: chaosju 2021-06-03 08:18:18,586 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread java.util.NoSuchElementException at java.base/java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:721) at java.base/java.util.LinkedHashMap$LinkedValueIterator.next(LinkedHashMap.java:746) at java.base/java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1047) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:1091) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:1055) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133) at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
