[jira] [Resolved] (HDFS-16502) Reconfigure Block Invalidate limit
[
https://issues.apache.org/jira/browse/HDFS-16502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HDFS-16502.
Fix Version/s: 3.4.0
3.3.3
Resolution: Fixed
> Reconfigure Block Invalidate limit
> --
>
> Key: HDFS-16502
> URL: https://issues.apache.org/jira/browse/HDFS-16502
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.3
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Based on the cluster load, it would be helpful to consider tuning block
> invalidate limit (dfs.block.invalidate.limit). The only way we can do this
> without restarting Namenode as of today is by reconfiguring heartbeat
> interval
> {code:java}
> Math.max(heartbeatInt*20, blockInvalidateLimit){code}
> , this logic is not straightforward and operators are usually not aware of it
> (lack of documentation), also updating heartbeat interval is not desired in
> all the cases.
> We should provide the ability to alter block invalidation limit without
> affecting heartbeat interval on the live cluster to adjust some load at
> Datanode level.
> We should also take this opportunity to keep (heartbeatInterval * 20)
> computation logic in a common method.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16507) Purged edit logs which is in process
tomscut created HDFS-16507:
--
Summary: Purged edit logs which is in process
Key: HDFS-16507
URL: https://issues.apache.org/jira/browse/HDFS-16507
Project: Hadoop HDFS
Issue Type: Bug
Environment: {code:java}
// code placeholder
{code}
Reporter: tomscut
We introduced Standby read functionality in branch-3.1.0, but found a FATAL
exception. It looks like it's purging edit logs which is in process.
According to the analysis, I suspect that the Editlog to be purged does not
finalize normally .
I post some key logs for your reference:
1. ANN. Create editlog,
{color:#FF}edits_InProgresS_00024207987{color}.
{code:java}
2022-03-15 17:24:52,558 INFO namenode.FSEditLog
(FSEditLog.java:startLogSegment(1394)) - Starting log segment at 24207987
2022-03-15 17:24:52,609 INFO namenode.FSEditLog
(FSEditLog.java:startLogSegment(1423)) - Ending log segment at 24207987
2022-03-15 17:24:52,610 INFO namenode.FSEditLog
(FSEditLog.java:startLogSegmentAndWriteHeaderTxn(1432)) - logEdit at 24207987
2022-03-15 17:24:52,624 INFO namenode.FSEditLog
(FSEditLog.java:startLogSegmentAndWriteHeaderTxn(1434)) - logSync at 24207987
{code}
2. SNN. Checkpoint.
{color:#FF}25892513 + 1 - 100 = 24892514{color}
{color:#FF}dfs.namenode.num.extra.edits.retained=100{color}
{code:java}
2022-03-15 17:28:02,640 INFO ha.StandbyCheckpointer
(StandbyCheckpointer.java:doWork(443)) - Triggering checkpoint because there
have been 1189661 txns since the last checkpoint, which exceeds the configured
threshold 2
2022-03-15 17:28:02,648 INFO namenode.FSImage
(FSEditLogLoader.java:loadFSEdits(188)) - Edits file
ByteStringEditLog[27082175, 27082606], ByteStringEditLog[27082175, 27082606],
ByteStringEditLog[27082175, 27082606] of size 60008 edits # 432 loaded in 0
seconds
2022-03-15 17:28:02,649 INFO namenode.FSImage
(FSImage.java:saveNamespace(1121)) - Save namespace ...
2022-03-15 17:28:02,650 INFO namenode.FSImageFormatProtobuf
(FSImageFormatProtobuf.java:save(718)) - Saving image file
/data/hadoop/hdfs/namenode/current/fsimage.ckpt_00027082606 using no
compression
2022-03-15 17:28:03,180 INFO namenode.FSImageFormatProtobuf
(FSImageFormatProtobuf.java:save(722)) - Image file
/data/hadoop/hdfs/namenode/current/fsimage.ckpt_00027082606 of size
17885002 bytes saved in 0 seconds .
2022-03-15 17:28:03,183 INFO namenode.NNStorageRetentionManager
(NNStorageRetentionManager.java:getImageTxIdToRetain(211)) - Going to retain 2
images with txid >= 25892513
2022-03-15 17:28:03,183 INFO namenode.NNStorageRetentionManager
(NNStorageRetentionManager.java:purgeImage(233)) - Purging old image
FSImageFile(file=/data/hadoop/hdfs/namenode/current/fsimage_00024794305,
cpktTxId=00024794305)
2022-03-15 17:28:03,188 INFO namenode.NNStorageRetentionManager
(NNStorageRetentionManager.java:purgeOldStorage(169)) - purgeLogsFrom: 24892514
2022-03-15 17:28:03,282 INFO namenode.TransferFsImage
(TransferFsImage.java:copyFileToStream(396)) - Sending fileName:
/data/hadoop/hdfs/namenode/current/fsimage_00027082606, fileSize:
17885002. Sent total: 17885002 bytes. Size of last segment intended to send: -1
bytes.
2022-03-15 17:28:03,536 INFO namenode.TransferFsImage
(TransferFsImage.java:uploadImageFromStorage(240)) - Uploaded image with txid
27082606 to namenode at http://sg-test-ambari-nn1.bigdata.bigo.inner:50070 in
0.343 seconds
2022-03-15 17:28:03,640 INFO namenode.TransferFsImage
(TransferFsImage.java:copyFileToStream(396)) - Sending fileName:
/data/hadoop/hdfs/namenode/current/fsimage_00027082606, fileSize:
17885002. Sent total: 17885002 bytes. Size of last segment intended to send: -1
bytes.
2022-03-15 17:28:03,684 INFO namenode.TransferFsImage
(TransferFsImage.java:uploadImageFromStorage(240)) - Uploaded image with txid
27082606 to namenode at http://sg-test-ambari-dn1.bigdata.bigo.inner:50070 in
0.148 seconds
2022-03-15 17:28:03,748 INFO namenode.TransferFsImage
(TransferFsImage.java:copyFileToStream(396)) - Sending fileName:
/data/hadoop/hdfs/namenode/current/fsimage_00027082606, fileSize:
17885002. Sent total: 17885002 bytes. Size of last segment intended to send: -1
bytes.
2022-03-15 17:28:03,798 INFO namenode.TransferFsImage
(TransferFsImage.java:uploadImageFromStorage(240)) - Uploaded image with txid
27082606 to namenode at http://sg-test-ambari-dn2.bigdata.bigo.inner:50070 in
0.113 seconds
2022-03-15 17:28:03,798 INFO ha.StandbyCheckpointer
(StandbyCheckpointer.java:doWork(482)) - Checkpoint finished successfully.
{code}
3. ANN. Purge edit logs.
{color:#FF}25892513 + 1 - 100 = 24892514{color}
{color:#FF}dfs.namenode.num.extra.edits.retained=100{color}
{code:java}
2022-03-15 17:28:03,515 INFO namenode.NNStorageRetentionManager
Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/272/ [Mar 14, 2022 5:21:35 PM] (Owen O'Malley) HDFS-16495: RBF should prepend the client ip rather than append it. [Mar 14, 2022 6:29:48 PM] (Owen O'Malley) HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. [Error replacing 'FILE' - Workspace is not accessible] - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/ [Mar 14, 2022 5:21:35 PM] (Owen O'Malley) HDFS-16495: RBF should prepend the client ip rather than append it. [Mar 14, 2022 6:29:48 PM] (Owen O'Malley) HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. -1 overall The following subsystems voted -1: blanks pathlen xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/artifact/out/results-compile-javac-root.txt [340K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/809/artifact/out/results-javadoc-javadoc-root.txt [404K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.io.compress.snappy.TestSnappyCompressorDecompressor hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/diff-compile-javac-root.txt [476K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-mvnsite-root.txt [556K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [224K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [424K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [124K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/601/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K] asflicense:
