Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/279/ [Mar 27, 2022 1:23:48 PM] (noreply) HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) [Mar 28, 2022 8:02:10 AM] (Masatake Iwasaki) Make upstream aware of 3.2.3 release. [Mar 28, 2022 8:07:50 AM] (Masatake Iwasaki) Make upstream aware of 3.2.3 release. [Mar 28, 2022 7:52:46 PM] (Owen O'Malley) HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. Fixed in HADOOP-16161, but [Mar 28, 2022 8:15:28 PM] (Owen O'Malley) HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache -1 overall The following subsystems voted -1: blanks mvnsite pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-hdfs-project/hadoop-hdfs Redundant nullcheck of oldLock, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory) Redundant null check at DataStorage.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory) Redundant null check at DataStorage.java:[line 695] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:[line 138] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:[line 75] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:[line 85] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$PmemMappedRegion, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$PmemMappedRegion, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:[line 130] org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts doesn't override java.util.ArrayList.equals(Object) At RollingWindowManager.java:At RollingWindowManager.java:[line 1] spotbugs : module:hadoop-yarn-project/hadoop-yarn Redundant nullcheck of it, which is known to be non-null in
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/823/ [Mar 28, 2022 8:02:10 AM] (Masatake Iwasaki) Make upstream aware of 3.2.3 release. [Mar 28, 2022 8:07:50 AM] (Masatake Iwasaki) Make upstream aware of 3.2.3 release. [Mar 28, 2022 7:52:46 PM] (Owen O'Malley) HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. Fixed in HADOOP-16161, but [Mar 28, 2022 8:15:28 PM] (Owen O'Malley) HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache -1 overall The following subsystems voted -1: blanks pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-tools/hadoop-sls org.apache.hadoop.yarn.sls.AMRunner.setInputTraces(String[]) may expose internal representation by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:[line 267] Write to static field org.apache.hadoop.yarn.sls.AMRunner.REMAINING_APPS from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:[line 116] spotbugs : module:hadoop-tools org.apache.hadoop.yarn.sls.AMRunner.setInputTraces(String[]) may expose internal representation by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:[line 267] Write to static field org.apache.hadoop.yarn.sls.AMRunner.REMAINING_APPS from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:[line 116] spotbugs : module:root org.apache.hadoop.yarn.sls.AMRunner.setInputTraces(String[]) may expose internal representation by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:[line 267] Write to static field org.apache.hadoop.yarn.sls.AMRunner.REMAINING_APPS from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:[line 116] Failed junit tests : hadoop.yarn.conf.TestYarnConfigurationFields hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor hadoop.yarn.sls.TestSLSRunner hadoop.yarn.sls.TestSLSGenericSynth hadoop.yarn.sls.TestSLSStreamAMSynth cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/823/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/823/artifact/out/results-compile-javac-root.txt [340K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/823/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/823/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/823/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/823/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/823/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/823/artifact/out/results-shellcheck.txt [28K] xml:
[jira] [Created] (HADOOP-18190) s3a prefetching streams to collect iostats on prefetching operations
Steve Loughran created HADOOP-18190: --- Summary: s3a prefetching streams to collect iostats on prefetching operations Key: HADOOP-18190 URL: https://issues.apache.org/jira/browse/HADOOP-18190 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran There is a lot more happening in reads, so lot of more to collect and publish in IO stats for us to view in a summary at the end of processes as well as get from the stream while it is active Some useful ones would seem to be counters * is in memory. using 0 or 1 here lets aggregation reports count total #of memory cached files. * prefetching operations executed * errors during prefetching gauges * number of blocks in cache * total size of blocks * active prefetches + active memory used duration tracking count/min/max/ave * time to fetch a block * time queued before the actual fetch begins * time a reader is blocked waiting for a block fetch to complete and some info on cache use itself * number of blocks discarded unread * number of prefetched blocks later used * number of backward seeks to a prefetched block * number of forward seeks to a prefetched block the key ones I care about are # memory consumption # can we determine if cache is working (reads with cache hit) and when it is not (misses, wasted prefetches) # time blocked on executors The stats need to be accessible on a stream even when closed, and aggregated into the FS. once we get per-thread stats contexts we can publish there too and collect in worker threads for reporting in task commits -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18189) S3PrefetchingInputStream to support status probes when closed
Steve Loughran created HADOOP-18189: --- Summary: S3PrefetchingInputStream to support status probes when closed Key: HADOOP-18189 URL: https://issues.apache.org/jira/browse/HADOOP-18189 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran S3PrefetchingInputStream is a bit over aggressive on raising exceptions/downgrading responses after a stream is closed * MUST: getPos() to return last read location, or a least 0 (maybe we should add this to filesystem spec) * MUST: getIOStatistics(). critical for collecting stats in processes * MUST: seekToNewSource() (it's a no op anyway) * MAY S3AInputStreamStatistics and getStreamStatistics() though that is only in used in testing...ioStatistics have given us a stable stats api. we may want to tag the method @VisibleForTesting to discourage use. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18188) Support touch command for directory
Akira Ajisaka created HADOOP-18188: -- Summary: Support touch command for directory Key: HADOOP-18188 URL: https://issues.apache.org/jira/browse/HADOOP-18188 Project: Hadoop Common Issue Type: Improvement Reporter: Akira Ajisaka Currently hadoop fs -touch command cannot update the mtime and the atime of directory. The feature would be useful when we check whether the filesystem is ready to write or not without creating any file. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18187) Use JavaDoc for many field and enum comments
Daniel Carl Jones created HADOOP-18187: -- Summary: Use JavaDoc for many field and enum comments Key: HADOOP-18187 URL: https://issues.apache.org/jira/browse/HADOOP-18187 Project: Hadoop Common Issue Type: Sub-task Reporter: Daniel Carl Jones There's lots of good comments for fields and enum values in the current code. Let's expose these to your IDE with JavaDoc instead. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18186) s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
Steve Loughran created HADOOP-18186: --- Summary: s3a prefetching to use SemaphoredDelegatingExecutor for submitting work Key: HADOOP-18186 URL: https://issues.apache.org/jira/browse/HADOOP-18186 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran Use SemaphoredDelegatingExecutor for each to stream to submit work, if possible, for better fairness in processes with many streams. this also takes a DurationTrackerFactory to count how long was spent in the queue, something we would want to know -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18185) tune logging of prefetch problems
Steve Loughran created HADOOP-18185: --- Summary: tune logging of prefetch problems Key: HADOOP-18185 URL: https://issues.apache.org/jira/browse/HADOOP-18185 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran we need to think what we want to do here. * we would want errors to be counted and included in stream iostats * but we've seen abfs apps flooded with stack traces when a transient network error breaks every prefetch. * logging through LogExactlyOnce would highlight a problem, but on a long lived process, not show enough, unless the cache manager had a per instance value. # PrefetchTask needs to be non static and update the CachingBlockManager when there is a problem; it can LogExactlyOnce # stream statistics need to include cache values (hits, misses, prefech failures, how long futures were queued before execution...) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/ [Mar 28, 2022 8:31:50 PM] (Owen O'Malley) HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.io.compress.snappy.TestSnappyCompressorDecompressor hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/diff-compile-javac-root.txt [476K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-mvnsite-root.txt [560K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [224K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [428K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [128K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/615/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K]
