Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/ [Nov 14, 2021 2:29:15 PM] (noreply) HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. [Nov 14, 2021 2:40:24 PM] (noreply) HDFS-16319. Add metrics doc for ReadLockLongHoldCount and WriteLockLongHoldCount (#3653). Contributed by tomscut. -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.TestViewDistributedFileSystemContract hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure hadoop.yarn.csi.client.TestCsiClient hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.TestDynamometerInfra cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/results-compile-javac-root.txt [356K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/results-javadoc-javadoc-root.txt [408K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [544K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt [152K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/690/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer.txt [24K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14480) Shut down DataNode gracefully when responding to stop-dfs.sh/stop-dfs.cmd
[ https://issues.apache.org/jira/browse/HDFS-14480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feilong He resolved HDFS-14480. --- Resolution: Won't Fix > Shut down DataNode gracefully when responding to stop-dfs.sh/stop-dfs.cmd > - > > Key: HDFS-14480 > URL: https://issues.apache.org/jira/browse/HDFS-14480 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > > Currently, DataNode has #shutdown method to tackle something before shutdown. > But its shutdown hook does't call this method. In HDFS-14401 for HDFS > persistent memory cache optimization, we added clean cache logic in DN's > #shutdown method. And we expect DN will clean up cache during shut down by > stop-dfs.sh/stop-dfs.cmd, which depends on this Jira's patch. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16315) Add metrics related to Transfer and NativeCopy for DataNode
[ https://issues.apache.org/jira/browse/HDFS-16315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma resolved HDFS-16315. - Fix Version/s: 3.4.0 Resolution: Fixed > Add metrics related to Transfer and NativeCopy for DataNode > --- > > Key: HDFS-16315 > URL: https://issues.apache.org/jira/browse/HDFS-16315 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2021-11-11-08-26-33-074.png > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Datanodes already have Read, Write, Sync and Flush metrics. We should add > NativeCopy and Transfer as well. > Here is a partial look after the change: > !image-2021-11-11-08-26-33-074.png|width=205,height=235! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16327) Change dfs.namenode.max.slowpeer.collect.nodes to a proportional value
tomscut created HDFS-16327: -- Summary: Change dfs.namenode.max.slowpeer.collect.nodes to a proportional value Key: HDFS-16327 URL: https://issues.apache.org/jira/browse/HDFS-16327 Project: Hadoop HDFS Issue Type: Wish Reporter: tomscut Assignee: tomscut Currently, dfs.namenode.max.slowpeer.collect.nodes is a fixed value, but often needs to be changed as the cluster size changes. We can change it to a scale value and make it reconfigurable. See [HDFS-15879|https://issues.apache.org/jira/browse/HDFS-15879]. And dfs.datanode.max.disks.to.report can be changed similarly. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16326) Simplify the code for DiskBalancer
tomscut created HDFS-16326: -- Summary: Simplify the code for DiskBalancer Key: HDFS-16326 URL: https://issues.apache.org/jira/browse/HDFS-16326 Project: Hadoop HDFS Issue Type: Wish Reporter: tomscut Assignee: tomscut Simplify the code for DiskBalancer. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16325) some time delay (0.3s) for swebhdfs + kerberos + observer setting.
Chuan-Heng Hsiao created HDFS-16325: --- Summary: some time delay (0.3s) for swebhdfs + kerberos + observer setting. Key: HDFS-16325 URL: https://issues.apache.org/jira/browse/HDFS-16325 Project: Hadoop HDFS Issue Type: Bug Components: ha, webhdfs Affects Versions: 3.3.1 Environment: ubuntu 20.04 hadoop 3.3.1 openjdk-8 Reporter: Chuan-Heng Hsiao Settings: 1 master namenode (A), 1 standby namenode (B), 1 observer namenode (C). following [https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html] except that dfs.client.failover.observer.auto-msync-period. is set to -1 (not auto -msync) uable to do curl - - negotiate -u ':' 'https://:/webhdfs/v1/...' because it seems like due to the following issue: https://issues.apache.org/jira/browse/HDFS-14443 using curl --negotiate -u ':' 'https://:/webhdfs/v1/...' can successfully get 307 redirect with the corresponding Location. but got token (token for xxx HDFS_DELEGATION_TOKEN owner=xxx renewer=xxx masterKeyID=ooo) can't be found in cache" if redirect the url within 300ms. Not issue if waiting for more than 300ms and then do the redirect. No issue if changing (C) to Standby (no observers) (and redirect within 10 ms) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16324) fix error log in BlockManagerSafeMode
guo created HDFS-16324: -- Summary: fix error log in BlockManagerSafeMode Key: HDFS-16324 URL: https://issues.apache.org/jira/browse/HDFS-16324 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.3.1 Reporter: guo if `recheckInterval` was set as invalid value, there will be warning log output, but the message seems not that proper ,we can improve it. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.client.api.impl.TestAMRMClient hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.tools.TestDistCpSystem hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/diff-compile-javac-root.txt [496K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-mvnsite-root.txt [584K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/diff-patch-pylint.txt [48K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/diff-patch-shelldocs.txt [48K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-javadoc-root.txt [32K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [232K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [424K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [40K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [128K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/482/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-map
