[jira] [Commented] (HDFS-16726) There is a memory-related problem about HDFS namenode
[ https://issues.apache.org/jira/browse/HDFS-16726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797090#comment-17797090 ] Yanlei Yu commented on HDFS-16726: -- [~zinx] hi,You mean upgrade jdk to 12 and remove G1RSetRegionEntries param. After that, you can solve the problem? > There is a memory-related problem about HDFS namenode > - > > Key: HDFS-16726 > URL: https://issues.apache.org/jira/browse/HDFS-16726 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs, namenode >Affects Versions: 2.7.2 > Environment: -Xms280G -Xmx280G -XX:MaxDirectMemorySize=10G > -XX:MetaspaceSize=128M -server \ > -XX:+UseG1GC -XX:+UseStringDeduplication > -XX:MaxGCPauseMillis=250 -XX:+UnlockExperimentalVMOptions > -XX:+PrintGCApplicationStoppedTime -XX:+PrintSafepointStatistics > -XX:PrintSafepointStatisticsCount=1 \ > -XX:G1OldCSetRegionThresholdPercent=1 > -XX:G1MixedGCCountTarget=9 -XX:+SafepointTimeout > -XX:SafepointTimeoutDelay=4000 \ > -XX:ParallelGCThreads=24 -XX:ConcGCThreads=6 > -XX:G1RSetRegionEntries=4096 -XX:+AggressiveOpts -XX:+DisableExplicitGC \ > -XX:G1HeapWastePercent=9 > -XX:G1MixedGCLiveThresholdPercent=85 -XX:InitiatingHeapOccupancyPercent=75 \ > -XX:+ParallelRefProcEnabled -XX:-ResizePLAB > -XX:+PrintAdaptiveSizePolicy \ > -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps \ > -Xloggc:$HADOOP_LOG_DIR/namenode.gc.log \ > -XX:+HeapDumpOnOutOfMemoryError > -XX:ErrorFile=$HADOOP_LOG_DIR/hs_err_pid%p.log > -XX:HeapDumpPath=$HADOOP_LOG_DIR \ > -Dcom.sun.management.jmxremote \ > -Dcom.sun.management.jmxremote.port=9009 \ > -Dcom.sun.management.jmxremote.ssl=false \ > -Dcom.sun.management.jmxremote.authenticate=false \ > $HADOOP_NAMENODE_OPTS >Reporter: Yanlei Yu >Priority: Critical > > In the cluster, the memory usage of Namenode exceeds the XMX setting (XMX > =280GB). The actual memory usage of Namenode is 479GB > Output via pamp: > Address Perm Offset Device Inode Size Rss Pss > Referenced Anonymous Swap Locked Mapping > 2b42f000 rw-p 00:00 0 294174720 293756960 293756960 > 293756960 293756960 0 0 > 01e21000 rw-p 00:00 0 195245456 195240848 195240848 > 195240848 195240848 0 0 [heap] > 2b897c00 rw-p 00:00 0 9246724 9246724 9246724 > 9246724 9246724 0 0 > 2b8bb0905000 rw-p 00:00 0 1781124 1754572 1754572 > 1754572 1754572 0 0 > 2b893600 rw-p 00:00 0 1146880 1002084 1002084 > 1002084 1002084 0 0 > 2b42db652000 rwxp 00:00 0 57792 55252 55252 > 55252 55252 0 0 > 2b42ec12a000 rw-p 00:00 0 25696 24700 24700 > 24700 24700 0 0 > 2b42ef25b000 rw-p 00:00 0 9988 8972 8972 > 8972 8972 0 0 > 2b8c1d467000 rw-p 00:00 0 9216 8204 8204 > 8204 8204 0 0 > 2b8d6f8db000 rw-p 00:00 0 7160 6228 6228 > 6228 6228 0 0 > The first line should configure the memory footprint for XMX, and [heap] is > unusually large, so a memory leak is suspected! > > * [heap] is associated with malloc > After configuring JCMD in the test environment, we found that the malloc part > of Internal in JCMD increased significantly when the client was writing to a > gz file (XMX =40g in the test environment, and the Internal area was 900MB > before the client wrote) : > Total: reserved=47276MB, committed=47070MB > - Java Heap (reserved=40960MB, committed=40960MB) > (mmap: reserved=40960MB, committed=40960MB) > > - Class (reserved=53MB, committed=52MB) > (classes #7423) > (malloc=1MB #17053) > (mmap: reserved=52MB, committed=52MB) > > - Thread (reserved=2145MB, committed=2145MB) > (thread #2129) > (stack: reserved=2136MB, committed=2136MB) > (malloc=7MB #10673) > (arena=2MB #4256) > > - Code (reserved=251MB, committed=45MB) > (malloc=7MB #10661) > (mmap: reserved=244MB, committed=38MB) > > -
[jira] [Created] (HDFS-17291) DataNode metric bytesWritten is not totally accurate in some situations.
farmmamba created HDFS-17291: Summary: DataNode metric bytesWritten is not totally accurate in some situations. Key: HDFS-17291 URL: https://issues.apache.org/jira/browse/HDFS-17291 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.3.6 Reporter: farmmamba As the title described, dataNode metric bytesWritten is not totally accurate in some situations, such as failure recovery, re-send data. We should fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17291) DataNode metric bytesWritten is not totally accurate in some situations.
[ https://issues.apache.org/jira/browse/HDFS-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17291: -- Labels: pull-request-available (was: ) > DataNode metric bytesWritten is not totally accurate in some situations. > > > Key: HDFS-17291 > URL: https://issues.apache.org/jira/browse/HDFS-17291 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > As the title described, dataNode metric bytesWritten is not totally accurate > in some situations, such as failure recovery, re-send data. We should fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17291) DataNode metric bytesWritten is not totally accurate in some situations.
[ https://issues.apache.org/jira/browse/HDFS-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797105#comment-17797105 ] ASF GitHub Bot commented on HDFS-17291: --- hfutatzhanghb commented on PR #6360: URL: https://github.com/apache/hadoop/pull/6360#issuecomment-1857568147 @Hexiaoqiao @ayushtkn @tomscut @zhangshuyan0 Hi, sir. Could you please help me review this simple modification when you have free time? Thanks ahead. > DataNode metric bytesWritten is not totally accurate in some situations. > > > Key: HDFS-17291 > URL: https://issues.apache.org/jira/browse/HDFS-17291 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > As the title described, dataNode metric bytesWritten is not totally accurate > in some situations, such as failure recovery, re-send data. We should fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17289) Should not consider the size of non-lastBlocks equals to complete block size.
[ https://issues.apache.org/jira/browse/HDFS-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797167#comment-17797167 ] ASF GitHub Bot commented on HDFS-17289: --- zhangshuyan0 commented on code in PR #6357: URL: https://github.com/apache/hadoop/pull/6357#discussion_r1427941430 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java: ## @@ -368,14 +368,14 @@ private DFSOutputStream(DFSClient dfsClient, String src, } } - private void adjustPacketChunkSize(HdfsFileStatus stat) throws IOException{ + private void adjustPacketChunkSize(LocatedBlock lastBlock) throws IOException{ -long usedInLastBlock = stat.getLen() % blockSize; +long usedInLastBlock = lastBlock.getBlockSize() % blockSize; Review Comment: Why don't we use `long usedInLastBlock = lastBlock.getBlockSize()` directly? > Should not consider the size of non-lastBlocks equals to complete block size. > - > > Key: HDFS-17289 > URL: https://issues.apache.org/jira/browse/HDFS-17289 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17289) Should not consider the size of non-lastBlocks equals to complete block size.
[ https://issues.apache.org/jira/browse/HDFS-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797170#comment-17797170 ] ASF GitHub Bot commented on HDFS-17289: --- hfutatzhanghb commented on code in PR #6357: URL: https://github.com/apache/hadoop/pull/6357#discussion_r1427952107 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java: ## @@ -368,14 +368,14 @@ private DFSOutputStream(DFSClient dfsClient, String src, } } - private void adjustPacketChunkSize(HdfsFileStatus stat) throws IOException{ + private void adjustPacketChunkSize(LocatedBlock lastBlock) throws IOException{ -long usedInLastBlock = stat.getLen() % blockSize; +long usedInLastBlock = lastBlock.getBlockSize() % blockSize; Review Comment: @zhangshuyan0 Sir, Thanks for your reviewing, very nice suggestion. I will fix it. > Should not consider the size of non-lastBlocks equals to complete block size. > - > > Key: HDFS-17289 > URL: https://issues.apache.org/jira/browse/HDFS-17289 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17289) Should not consider the size of non-lastBlocks equals to complete block size.
[ https://issues.apache.org/jira/browse/HDFS-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797171#comment-17797171 ] ASF GitHub Bot commented on HDFS-17289: --- hfutatzhanghb commented on code in PR #6357: URL: https://github.com/apache/hadoop/pull/6357#discussion_r1427954912 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java: ## @@ -368,14 +368,14 @@ private DFSOutputStream(DFSClient dfsClient, String src, } } - private void adjustPacketChunkSize(HdfsFileStatus stat) throws IOException{ + private void adjustPacketChunkSize(LocatedBlock lastBlock) throws IOException{ -long usedInLastBlock = stat.getLen() % blockSize; +long usedInLastBlock = lastBlock.getBlockSize() % blockSize; Review Comment: Have fixed. > Should not consider the size of non-lastBlocks equals to complete block size. > - > > Key: HDFS-17289 > URL: https://issues.apache.org/jira/browse/HDFS-17289 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17291) DataNode metric bytesWritten is not totally accurate in some situations.
[ https://issues.apache.org/jira/browse/HDFS-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797198#comment-17797198 ] ASF GitHub Bot commented on HDFS-17291: --- hadoop-yetus commented on PR #6360: URL: https://github.com/apache/hadoop/pull/6360#issuecomment-1857940102 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 21s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 35s | | trunk passed | | +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | +1 :green_heart: | checkstyle | 0m 37s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 43s | | trunk passed | | +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | +1 :green_heart: | spotbugs | 1m 40s | | trunk passed | | +1 :green_heart: | shadedclient | 20m 14s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 35s | | the patch passed | | +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javac | 0m 37s | | the patch passed | | +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | +1 :green_heart: | javac | 0m 31s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 29s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 37s | | the patch passed | | +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | +1 :green_heart: | spotbugs | 1m 44s | | the patch passed | | +1 :green_heart: | shadedclient | 19m 57s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | -1 :x: | unit | 182m 37s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6360/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 24s | | The patch does not generate ASF License warnings. | | | | 267m 42s | | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6360/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/6360 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | Linux 3728c0e20f4b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / c789a691a780bbbf1452736961725a2be413ca39 | | Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6360/1/testReport/ | | Max. process+thread count | 4783 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-p
[jira] [Commented] (HDFS-17289) Should not consider the size of non-lastBlocks equals to complete block size.
[ https://issues.apache.org/jira/browse/HDFS-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797207#comment-17797207 ] ASF GitHub Bot commented on HDFS-17289: --- hadoop-yetus commented on PR #6357: URL: https://github.com/apache/hadoop/pull/6357#issuecomment-1857973804 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 21s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 59s | | trunk passed | | +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | +1 :green_heart: | checkstyle | 0m 20s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 34s | | trunk passed | | +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | +1 :green_heart: | spotbugs | 1m 24s | | trunk passed | | +1 :green_heart: | shadedclient | 20m 5s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 24s | | the patch passed | | +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javac | 0m 29s | | the patch passed | | +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | +1 :green_heart: | javac | 0m 24s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 10s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 25s | | the patch passed | | +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | +1 :green_heart: | spotbugs | 1m 26s | | the patch passed | | +1 :green_heart: | shadedclient | 20m 8s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 1m 49s | | hadoop-hdfs-client in the patch passed. | | +1 :green_heart: | asflicense | 0m 25s | | The patch does not generate ASF License warnings. | | | | 84m 9s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6357/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/6357 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | Linux 031aba3ffdb8 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / b127b3a68aaec22bf29bc9683defd10ec0f52486 | | Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6357/2/testReport/ | | Max. process+thread count | 554 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6357/2/console | | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |