[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646808 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 06/Sep/21 09:08 Start Date: 06/Sep/21 09:08 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-913481069 @tomscut @virajjasani, Thanks. I think It's a good way to add description here. BTW, change the title of JIRA and PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646808) Time Spent: 1h 50m (was: 1h 40m) > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646729=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646729 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 06/Sep/21 01:45 Start Date: 06/Sep/21 01:45 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-913278183 @ayushtkn @ferhui @virajjasani Thank you very much for your comments and suggestions. I think what you are saying is reasonable, we should not change the default value of this parameter. But we can add a caption, as @virajjasani said. I have changed the parameter description information, please have a look. Thanks again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646729) Time Spent: 1h 40m (was: 1.5h) > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646691 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 05/Sep/21 11:23 Start Date: 05/Sep/21 11:23 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-913134902 Thanks for the reference and find @tomscut ! I agree to the general opinion of not making incompatible change until absolutely required, and on the other hand, this change looks quite obvious given that write lock overhead is redundant for non-Cache use-cases. If we want to use the feature, enable the config, that sounds right. However, I believe, as of now, we should provide one fat warning log at appropriate place stating that "please disable this config unless you are using Cache feature and we are going to disable this config by default in 4.0.0 and above releases". And we might also want to reference this Jira for perf degradation case. Thoughts? Overall, perhaps we might want to wait at least one more major release before disabling this by default rather than making incompatible change on 3.x releases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646691) Time Spent: 1.5h (was: 1h 20m) > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646255 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 03/Sep/21 11:08 Start Date: 03/Sep/21 11:08 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-912456375 > As @ayushtkn said, facing the same problem, [HDFS-13820](https://issues.apache.org/jira/browse/HDFS-13820) add ability to disable the feature, you can also set it false. > If you change the default value, it's an incompatible change, especially for upgrading(using this feature). Seem that it's not so good. Thanks @ferhui for your comments. Maybe we can add a release note for this change. For new users who may not know this feature(Centralized Cache Management) exists, but it already runs quietly in the background. I think it's not a very elegant way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646255) Time Spent: 1h 20m (was: 1h 10m) > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646251 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 03/Sep/21 10:34 Start Date: 03/Sep/21 10:34 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-912438441 As @ayushtkn said, facing the same problem, HDFS-13820 add ability to disable the feature, you can also set it false. If you change the default value, it's an incompatible change, especially for upgrading(using this feature). Seem that it's not so good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646251) Time Spent: 1h 10m (was: 1h) > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[
https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646247
]
ASF GitHub Bot logged work on HDFS-16209:
-
Author: ASF GitHub Bot
Created on: 03/Sep/21 10:16
Start Date: 03/Sep/21 10:16
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3378:
URL: https://github.com/apache/hadoop/pull/3378#issuecomment-912427687
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 59s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 249m 5s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3378/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 345m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.fs.TestEnhancedByteBufferAccess |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestPmemCacheRecovery |
| |
hadoop.hdfs.server.datanode.fsdataset.impl.TestCacheByPmemMappableBlockLoader |
| | hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetCache |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3378/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3378 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux 8f8fcb9e3d81 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646215=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646215 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 03/Sep/21 08:12 Start Date: 03/Sep/21 08:12 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-912349488 > [HDFS-13820](https://issues.apache.org/jira/browse/HDFS-13820), added this configuration to disable the feature, But still it was made to true by default, guess due to compatibility reasons. > Folks using the Cache feature would get impacted with this change, right? they have to now enable this explicitly. There was a proposal on on [HDFS-13820](https://issues.apache.org/jira/browse/HDFS-13820) > > ``` > Please implement a way to disable the CacheReplicationMonitor class if there are no paths specified. Adding the first cached path to the NameNode should kick off the CacheReplicationMonitor and when the last one is deleted, the CacheReplicationMonitor should be disabled again. > ``` > > Is something like this possible? Thanks @ayushtkn for your comments. I have also seen [HDFS-13820](https://issues.apache.org/jira/browse/HDFS-13820). But that feature(auto enable or auto disable) is not currently implemented. For new users who may not know this feature(Centralized Cache Management) exists, but it already runs quietly in the background, which incurs performance overhead. IMO, if we need to use this feature, it makes sense to turn it on and specify the path. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646215) Time Spent: 50m (was: 40m) > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646204 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 03/Sep/21 07:56 Start Date: 03/Sep/21 07:56 Worklog Time Spent: 10m Work Description: ayushtkn commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-912338368 HDFS-13820, added this configuration to disable the feature, But still it was made to true by default, guess due to compatibility reasons. Folks using the Cache feature would get impacted with this change, right? they have to now enable this explicitly. There was a proposal on on HDFS-13820 ``` Please implement a way to disable the CacheReplicationMonitor class if there are no paths specified. Adding the first cached path to the NameNode should kick off the CacheReplicationMonitor and when the last one is deleted, the CacheReplicationMonitor should be disabled again. ``` Is something like this possible? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646204) Time Spent: 40m (was: 0.5h) > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646194 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 03/Sep/21 07:06 Start Date: 03/Sep/21 07:06 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-912309108 Hi @ayushtkn , could you please also take a look. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646194) Time Spent: 0.5h (was: 20m) > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646187 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 03/Sep/21 06:10 Start Date: 03/Sep/21 06:10 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-912284182 @tasanuma @jojochuang @Hexiaoqiao @ferhui Please help review the change. Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646187) Time Spent: 20m (was: 10m) > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646173=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646173 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 03/Sep/21 04:29 Start Date: 03/Sep/21 04:29 Worklog Time Spent: 10m Work Description: tomscut opened a new pull request #3378: URL: https://github.com/apache/hadoop/pull/3378 **Namenode config:** dfs.namenode.write-lock-reporting-threshold-ms=50ms dfs.namenode.caching.enabled=true (default) In fact, the caching feature is not used in our cluster, but this switch is turned on by default(dfs.namenode.caching.enabled=true), incurring some additional write lock overhead. We count the number of write lock warnings in a log file, and find that the number of rescan cache warnings reaches about 32%, which greatly affects the performance of Namenode.  We should set 'dfs.namenode.caching.enabled' to false by default and turn it on when we wants to use it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646173) Remaining Estimate: 0h Time Spent: 10m > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg|width=713,height=82! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
