[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680862#comment-17680862
]
ASF GitHub Bot commented on HDFS-16864:
---------------------------------------
hadoop-yetus commented on PR #5216:
URL: https://github.com/apache/hadoop/pull/5216#issuecomment-1404497942
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ branch-3.3.5 Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 33s | | branch-3.3.5 passed |
| +1 :green_heart: | compile | 1m 12s | | branch-3.3.5 passed |
| +1 :green_heart: | checkstyle | 0m 53s | | branch-3.3.5 passed |
| +1 :green_heart: | mvnsite | 1m 22s | | branch-3.3.5 passed |
| +1 :green_heart: | javadoc | 1m 33s | | branch-3.3.5 passed |
| +1 :green_heart: | spotbugs | 3m 29s | | branch-3.3.5 passed |
| +1 :green_heart: | shadedclient | 30m 46s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 14s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 295m 53s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 417m 46s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.balancer.TestBalancerService |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.TestFileLengthOnClusterRestart |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5216 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 96f8292b3ab2 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3.5 / f03c37c59f8529211a81030b6e57bdb621c35156 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~18.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/3/testReport/ |
| Max. process+thread count | 2221 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5216/3/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> HDFS advisory caching should drop cache behind block when block closed
> ----------------------------------------------------------------------
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Affects Versions: 3.3.4
> Reporter: Dave Marion
> Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
