[
https://issues.apache.org/jira/browse/HDFS-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155114#comment-15155114
]
Colin Patrick McCabe commented on HDFS-9549:
--------------------------------------------
Thanks, [~xiaochen].
{code}
if (dn.getCacheRemainingPercent() > watermark
|| dn.getPendingCached().isEmpty()) {
continue;
}
{code}
I can see why you want to do this, but it isn't quite correct. We could have
10% of our DN's cache remaining, but a really big uncacheable block in its
pending_cached list. I would say just start with the current cacheUsed that
the DN has reported in its latest heartbeat and keep adding to it as you add
pendingCached blocks. Drop any pendingCached blocks which would cause it to
exceed cacheCapacity. Perhaps we can optimize this more later (like by keeping
a running total of pending cached), but for now that should make it correct.
> TestCacheDirectives#testExceedsCapacity is flaky
> ------------------------------------------------
>
> Key: HDFS-9549
> URL: https://issues.apache.org/jira/browse/HDFS-9549
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.0.0
> Environment: Jenkins
> Reporter: Wei-Chiu Chuang
> Assignee: Xiao Chen
> Labels: unittest
> Attachments: HDFS-9549.01.patch, HDFS-9549.02.patch,
> HDFS-9549.03.patch, TestCacheDirectives.rtf
>
>
> I have observed that this test (TestCacheDirectives.testExceedsCapacity)
> fails quite frequently in Jenkins (trunk, trunk-Java8)
> Error Message
> Pending cached list of 127.0.0.1:54134 is not empty, [{blockId=1073741841,
> replication=1, mark=true}]
> Stacktrace
> java.lang.AssertionError: Pending cached list of 127.0.0.1:54134 is not
> empty, [{blockId=1073741841, replication=1, mark=true}]
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.checkPendingCachedEmpty(TestCacheDirectives.java:1479)
> at
> org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity(TestCacheDirectives.java:1502)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
