Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/435/ [Jan 27, 2023, 11:50:17 AM] (github) HADOOP-17717. Update wildfly openssl to 1.1.3.Final. (#5310) [Jan 27, 2023, 3:32:50 PM] (Szilard Nemeth) HADOOP-18602. Remove netty3 dependency [Jan 28, 2023, 8:49:19 AM] (github) HADOOP-18604. Add compile platform in the hadoop version output. (#5327). Contributed by Ayush Saxena. -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-hdfs-project/hadoop-hdfs Redundant nullcheck of oldLock, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)) Redundant null check at DataStorage.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)) Redundant null check at DataStorage.java:[line 695] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:[line 138] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:[line 75] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:[line 85] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:[line 130] org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts doesn't override java.util.ArrayList.equals(Object) At RollingWindowManager.java:At RollingWindowManager.java:[line 1] spotbugs : module:hadoop-yarn-project/hadoop-yarn Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState)) Redundant null check at
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/922/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.hdfs.shortcircuit.TestShortCircuitCache hadoop.hdfs.TestQuota hadoop.hdfs.TestLargeBlock hadoop.security.TestRefreshUserMappings hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot hadoop.hdfs.server.namenode.TestNameNodeStatusMXBean hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics hadoop.hdfs.TestWriteConfigurationToDFS hadoop.hdfs.TestDFSShell hadoop.hdfs.TestDisableConnCache hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate hadoop.hdfs.server.namenode.TestEditLog hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA hadoop.hdfs.server.namenode.ha.TestStandbyInProgressTail hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters hadoop.hdfs.server.namenode.TestListOpenFiles hadoop.hdfs.server.namenode.TestNamenodeCapacityReport hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot hadoop.hdfs.TestDecommission hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.namenode.TestMetadataVersionOutput hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.server.namenode.ha.TestDNFencing hadoop.hdfs.server.namenode.TestStorageRestore hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy hadoop.hdfs.server.namenode.ha.TestStandbyIsHot hadoop.hdfs.TestRollingUpgradeRollback hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement hadoop.hdfs.server.namenode.TestGetContentSummaryWithPermission hadoop.hdfs.TestByteBufferPread hadoop.hdfs.server.namenode.snapshot.TestSnapshot hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication hadoop.tracing.TestTraceAdmin hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover hadoop.tracing.TestTracing hadoop.hdfs.TestDataStream hadoop.hdfs.TestDeadNodeDetection hadoop.hdfs.server.namenode.TestMetaSave hadoop.hdfs.server.datanode.TestBatchIbr hadoop.hdfs.server.datanode.TestBlockCountersInPendingIBR hadoop.hdfs.server.namenode.TestLargeDirectoryDelete hadoop.hdfs.server.namenode.TestMalformedURLs hadoop.hdfs.TestTrashWithEncryptionZones hadoop.hdfs.server.namenode.ha.TestUpdateBlockTailing hadoop.hdfs.server.namenode.ha.TestObserverNode hadoop.hdfs.server.namenode.ha.TestNNHealthCheck hadoop.TestRefreshCallQueue hadoop.hdfs.server.namenode.TestFileContextAcl hadoop.hdfs.server.namenode.ha.TestBootstrapStandby hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean hadoop.hdfs.server.namenode.ha.TestHAMetrics hadoop.hdfs.TestFileAppendRestart hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing hadoop.hdfs.TestParallelShortCircuitReadNoChecksum hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead hadoop.hdfs.server.namenode.ha.TestMultiObserverNode hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages hadoop.hdfs.TestListFilesInDFS hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion hadoop.hdfs.server.namenode.TestDecommissioningStatus hadoop.cli.TestCacheAdminCLI hadoop.hdfs.TestEncryptedTransfer hadoop.hdfs.TestHAAuxiliaryPort hadoop.cli.TestAclCLI hadoop.hdfs.server.namenode.TestEditLogRace hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade hadoop.hdfs.server.namenode.TestHDFSConcat hadoop.TestGenericRefresh hadoop.hdfs.server.namenode.TestAuditLogs hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots hadoop.hdfs.server.namenode.TestParallelImageWrite
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1120/ [Jan 28, 2023, 8:49:19 AM] (github) HADOOP-18604. Add compile platform in the hadoop version output. (#5327). Contributed by Ayush Saxena. -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-mapreduce-project/hadoop-mapreduce-client Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:hadoop-mapreduce-project Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:root Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] Failed junit tests : hadoop.hdfs.TestLeaseRecovery2 hadoop.mapreduce.v2.hs.TestJobHistoryParsing hadoop.mapreduce.v2.hs.TestJobHistoryEvents hadoop.mapreduce.v2.hs.TestJobHistoryServer hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1120/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1120/artifact/out/results-compile-javac-root.txt [528K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1120/artifact/out/blanks-eol.txt [14M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1120/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1120/artifact/out/results-checkstyle-root.txt [13M] hadolint:
[jira] [Created] (HDFS-16898) Make write lock fine-grain in processCommandFromActor method
ZhangHB created HDFS-16898:
--
Summary: Make write lock fine-grain in processCommandFromActor
method
Key: HDFS-16898
URL: https://issues.apache.org/jira/browse/HDFS-16898
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 3.3.4
Reporter: ZhangHB
Now in method processCommandFromActor, we have code like below:
{code:java}
writeLock();
try {
if (actor == bpServiceToActive) {
return processCommandFromActive(cmd, actor);
} else {
return processCommandFromStandby(cmd, actor);
}
} finally {
writeUnlock();
} {code}
if method processCommandFromActive costs much time, the write lock would not
release.
It maybe block the updateActorStatesFromHeartbeat method in
offerService，furthermore, it can cause the lastcontact of datanode very high,
even dead when lastcontact beyond 600s.
{code:java}
bpos.updateActorStatesFromHeartbeat(
this, resp.getNameNodeHaState());{code}
here we can make write lock fine-grain in processCommandFromActor method to
address this problem
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16897) fix abundant Broken pipe exception in BlockSender
fanluo created HDFS-16897: - Summary: fix abundant Broken pipe exception in BlockSender Key: HDFS-16897 URL: https://issues.apache.org/jira/browse/HDFS-16897 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 3.3.4 Reporter: fanluo in our production cluster env , we found some exception in datanode logs，its frequently print below error in HDFS-2054 we only ignored message starting with `Broken pipe` which may not enough for the following case: !https://user-images.githubusercontent.com/20748856/215264829-5f16dbc3-fea2-4883-a3d6-ded367564b8c.png! this situation look like related to short-circuit read. in HDFS-4354 the error has been wrapped, so that our previous judgment conditions are invalid. !https://user-images.githubusercontent.com/20748856/215314257-2064637b-ea46-42f5-b53f-a29e68bb50ea.png! maybe we can improve it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
