[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754809#comment-17754809
]
ASF GitHub Bot commented on HDFS-16864:
---
hadoop-yetus commented on PR #5204:
URL: https://github.com/apache/hadoop/pull/5204#issuecomment-1679740737
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 16s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 1s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 269m 40s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 426m 14s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.balancer.TestBalancerService |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5204 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 18119cbe6f31 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737354#comment-17737354
]
ASF GitHub Bot commented on HDFS-16864:
---
hadoop-yetus commented on PR #5204:
URL: https://github.com/apache/hadoop/pull/5204#issuecomment-1608309531
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 244m 19s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 362m 18s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5204 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6b7ecf878ee6 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17696338#comment-17696338
]
ASF GitHub Bot commented on HDFS-16864:
---
hadoop-yetus commented on PR #5204:
URL: https://github.com/apache/hadoop/pull/5204#issuecomment-1454152028
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 57s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 3m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 3m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 259m 54s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 392m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.balancer.TestBalancerService |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5204 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8bd6412786bc 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 U
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690559#comment-17690559
]
ASF GitHub Bot commented on HDFS-16864:
---
hadoop-yetus commented on PR #5204:
URL: https://github.com/apache/hadoop/pull/5204#issuecomment-1435166654
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 263m 12s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 391m 3s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.TestFSNamesystemLockReport |
| | hadoop.hdfs.server.balancer.TestBalancerService |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.server.namenode.TestAuditLogger |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5204 |
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690391#comment-17690391
]
ASF GitHub Bot commented on HDFS-16864:
---
dlmarion commented on PR #5216:
URL: https://github.com/apache/hadoop/pull/5216#issuecomment-1434641482
Looks like this won't make 3.3.5. Closing, #5204 is still open for 3.4.0.
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690392#comment-17690392
]
ASF GitHub Bot commented on HDFS-16864:
---
dlmarion closed pull request #5216: HDFS-16864. Drop cache behind entire block
on close
URL: https://github.com/apache/hadoop/pull/5216
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680890#comment-17680890
]
ASF GitHub Bot commented on HDFS-16864:
---
hadoop-yetus commented on PR #5204:
URL: https://github.com/apache/hadoop/pull/5204#issuecomment-1404561494
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 37s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 29m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 416m 55s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 547m 30s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.balancer.TestBalancerService |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5204 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e2a6fe387b8e 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680862#comment-17680862
]
ASF GitHub Bot commented on HDFS-16864:
---
hadoop-yetus commented on PR #5216:
URL: https://github.com/apache/hadoop/pull/5216#issuecomment-1404497942
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3.5 Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 33s | | branch-3.3.5 passed |
| +1 :green_heart: | compile | 1m 12s | | branch-3.3.5 passed |
| +1 :green_heart: | checkstyle | 0m 53s | | branch-3.3.5 passed |
| +1 :green_heart: | mvnsite | 1m 22s | | branch-3.3.5 passed |
| +1 :green_heart: | javadoc | 1m 33s | | branch-3.3.5 passed |
| +1 :green_heart: | spotbugs | 3m 29s | | branch-3.3.5 passed |
| +1 :green_heart: | shadedclient | 30m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 295m 53s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 417m 46s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.balancer.TestBalancerService |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.TestFileLengthOnClusterRestart |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5216 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 96f8292b3ab2 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3.5 / f03c37c59f8529211a81030b6e57bd
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677870#comment-17677870
]
ASF GitHub Bot commented on HDFS-16864:
---
dlmarion commented on PR #5204:
URL: https://github.com/apache/hadoop/pull/5204#issuecomment-1385796995
Not sure what to make of these test failures. I ran 'mvn clean package
-Dtest=TestBalancer*` locally and they all passed.
```
[INFO] -
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17656969#comment-17656969
]
ASF GitHub Bot commented on HDFS-16864:
---
hadoop-yetus commented on PR #5204:
URL: https://github.com/apache/hadoop/pull/5204#issuecomment-1377854988
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 23s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 341m 34s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 464m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
|
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.server.balancer.TestBalancerService |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks |
| | hadoop.hdfs.TestSetrepIncreasing |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5204 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Li
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17656767#comment-17656767
]
ASF GitHub Bot commented on HDFS-16864:
---
hadoop-yetus commented on PR #5216:
URL: https://github.com/apache/hadoop/pull/5216#issuecomment-1377780674
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 9m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3.5 Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 58s | | branch-3.3.5 passed |
| +1 :green_heart: | compile | 1m 18s | | branch-3.3.5 passed |
| +1 :green_heart: | checkstyle | 0m 53s | | branch-3.3.5 passed |
| +1 :green_heart: | mvnsite | 1m 24s | | branch-3.3.5 passed |
| +1 :green_heart: | javadoc | 1m 33s | | branch-3.3.5 passed |
| +1 :green_heart: | spotbugs | 3m 42s | | branch-3.3.5 passed |
| +1 :green_heart: | shadedclient | 28m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 42s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed |
| +1 :green_heart: | spotbugs | 3m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 306m 23s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 428m 58s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.balancer.TestBalancerService |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.TestFileLengthOnClusterRestart |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.TestDFSInotifyEventInputStream |
| | hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5216 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux cedc282c2794 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3.5 / 759e32019413fa3ddf933cd8339efb142865ebc1 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~18.04-b08
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646837#comment-17646837
]
ASF GitHub Bot commented on HDFS-16864:
---
hadoop-yetus commented on PR #5216:
URL: https://github.com/apache/hadoop/pull/5216#issuecomment-1349790736
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 5m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3.5 Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 48s | | branch-3.3.5 passed |
| +1 :green_heart: | compile | 1m 14s | | branch-3.3.5 passed |
| +1 :green_heart: | checkstyle | 0m 53s | | branch-3.3.5 passed |
| +1 :green_heart: | mvnsite | 1m 25s | | branch-3.3.5 passed |
| +1 :green_heart: | javadoc | 1m 35s | | branch-3.3.5 passed |
| +1 :green_heart: | spotbugs | 3m 34s | | branch-3.3.5 passed |
| +1 :green_heart: | shadedclient | 27m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 42s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 320m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 437m 46s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.balancer.TestBalancerService |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.TestBlockStoragePolicy |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5216 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ae5b53b557f0 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3.5 / fea23420b1ea1d4a66e325d67aab937d36baf9ca |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~18.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/1/test
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646660#comment-17646660
]
ASF GitHub Bot commented on HDFS-16864:
---
dlmarion commented on PR #5216:
URL: https://github.com/apache/hadoop/pull/5216#issuecomment-1348682631
This PR is the same as #5204 , but targeted on the 3.3.5 branch. I'm curious
if the build passes.
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646659#comment-17646659
]
ASF GitHub Bot commented on HDFS-16864:
---
dlmarion opened a new pull request, #5216:
URL: https://github.com/apache/hadoop/pull/5216
As blocks are written posix_fadvise is called when dropCacheBehindWrites is
true, but the range supplied does not include the most recent 8MB written. This
change modifies BlockReceiver.close such that when dropCacheBehindWrites is
true, then posix_fadvise is called on the entire block.
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646155#comment-17646155
]
Dave Marion commented on HDFS-16864:
FWIW, this patch applies cleanly to 3.3.5. Also, I reran my tests and with
SYNC_BLOCK and setDropBehind, about 99% of the pages for the block are *not*
present in the page cache on file close.
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646092#comment-17646092
]
Steve Loughran commented on HDFS-16864:
---
although i want to do rc0 out this week, as xmas is coming it will take a while
for testing to finish and we are all having our own vacations. if others in
the hdfs team are happy it could go in now for that testing coverage
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645630#comment-17645630
]
Dave Marion commented on HDFS-16864:
I don't think there is much risk here, the code being used here is the same
that is used in another method in the BlockReceiver class. It would be good if
maybe [~cmccabe] or [~tlipcon] took a quick look to assess (not sure if they
are available for this or not, just looking at who was involved with the prior
ticket). As for criticality, it's not critical, it's a nice to have. I would
certainly be willing to wait if there is going to be another release in a few
months.
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645620#comment-17645620
]
Steve Loughran commented on HDFS-16864:
---
wanted to kick out an rc 0 next week (been testing the staging process while
getting an abfs blocker in)
* how critical is this
* how risky is it in terms of "if it plays up, what will go wrong"
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645510#comment-17645510
]
ASF GitHub Bot commented on HDFS-16864:
---
hadoop-yetus commented on PR #5204:
URL: https://github.com/apache/hadoop/pull/5204#issuecomment-1344866721
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 14m 49s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 350m 43s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 436m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerService |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5204/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5204 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsec
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645420#comment-17645420
]
Dave Marion commented on HDFS-16864:
[~ste...@apache.org] - not sure how far along you are with the 3.3.5 release
testing. Too late to get this in?
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645338#comment-17645338
]
ASF GitHub Bot commented on HDFS-16864:
---
dlmarion opened a new pull request, #5204:
URL: https://github.com/apache/hadoop/pull/5204
As blocks are written posix_fadvise is called when dropCacheBehindWrites is
true, but the range supplied does not include the most recent 8MB written. This
change modifies BlockReceiver.close such that when dropCacheBehindWrites is
true, then posix_fadvise is called on the entire block.
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
