[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982311#comment-14982311
]
Anoop Sam John commented on HBASE-14189:
So at CF level it can be only more permissive. We can document that at least
clearly. In javadoc.
Mind changing the title and /or desc of jira as per that?
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch, HBASE-14189_v3.patch
>
>
> The original description is ambiguous. I think i will rewrite it.
> Let's see {{BlockCache}} constructor firstly
> {code}
> public CacheConfig(Configuration conf, HColumnDescriptor family) {
> this(CacheConfig.instantiateBlockCache(conf),
> family.isBlockCacheEnabled(),
> family.isInMemory(),
> // For the following flags we enable them regardless of per-schema
> settings
> // if they are enabled in the global configuration.
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite(),
> conf.getBoolean(CACHE_INDEX_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_INDEXES_ON_WRITE) || family.isCacheIndexesOnWrite(),
> conf.getBoolean(CACHE_BLOOM_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_BLOOMS_ON_WRITE) || family.isCacheBloomsOnWrite(),
> conf.getBoolean(EVICT_BLOCKS_ON_CLOSE_KEY,
> DEFAULT_EVICT_ON_CLOSE) || family.isEvictBlocksOnClose(),
> conf.getBoolean(CACHE_DATA_BLOCKS_COMPRESSED_KEY,
> DEFAULT_CACHE_DATA_COMPRESSED),
> conf.getBoolean(PREFETCH_BLOCKS_ON_OPEN_KEY,
> DEFAULT_PREFETCH_ON_OPEN) || family.isPrefetchBlocksOnOpen(),
> conf.getBoolean(HColumnDescriptor.CACHE_DATA_IN_L1,
> HColumnDescriptor.DEFAULT_CACHE_DATA_IN_L1) ||
> family.isCacheDataInL1(),
>
> conf.getBoolean(DROP_BEHIND_CACHE_COMPACTION_KEY,DROP_BEHIND_CACHE_COMPACTION_DEFAULT)
> );
> }
> {code}
> If we dig in it, we will see {{CacheConfig.cacheDataOnRead}} is used to
> accept {{family.isBlockCacheEnabled()}}.
> I think it is confused as comments about {{cacheDataOnRead}}
> {code}
> /**
>* Whether blocks should be cached on read (default is on if there is a
>* cache but this can be turned off on a per-family or per-request basis).
>* If off we will STILL cache meta blocks; i.e. INDEX and BLOOM types.
>* This cannot be disabled.
>*/
> private boolean cacheDataOnRead;
> {code}
> So i think we should use another variable to represent for
> {{family.isBlockCacheEnabled()}}.
> The secondary point is we use 'or' to decide {{cacheDataOnWrite}} is on/off
> when both CF and global has this setting.
> {code}
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite()
> {code}
> Should we use CF Level setting to override global setting?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978536#comment-14978536
]
Anoop Sam John commented on HBASE-14189:
bq.The secondary point is we use 'or' to decide cacheDataOnWrite is on/off when
both CF and global has this setting.
Seems this is not done in this patch.
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch, HBASE-14189_v3.patch
>
>
> The original description is ambiguous. I think i will rewrite it.
> Let's see {{BlockCache}} constructor firstly
> {code}
> public CacheConfig(Configuration conf, HColumnDescriptor family) {
> this(CacheConfig.instantiateBlockCache(conf),
> family.isBlockCacheEnabled(),
> family.isInMemory(),
> // For the following flags we enable them regardless of per-schema
> settings
> // if they are enabled in the global configuration.
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite(),
> conf.getBoolean(CACHE_INDEX_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_INDEXES_ON_WRITE) || family.isCacheIndexesOnWrite(),
> conf.getBoolean(CACHE_BLOOM_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_BLOOMS_ON_WRITE) || family.isCacheBloomsOnWrite(),
> conf.getBoolean(EVICT_BLOCKS_ON_CLOSE_KEY,
> DEFAULT_EVICT_ON_CLOSE) || family.isEvictBlocksOnClose(),
> conf.getBoolean(CACHE_DATA_BLOCKS_COMPRESSED_KEY,
> DEFAULT_CACHE_DATA_COMPRESSED),
> conf.getBoolean(PREFETCH_BLOCKS_ON_OPEN_KEY,
> DEFAULT_PREFETCH_ON_OPEN) || family.isPrefetchBlocksOnOpen(),
> conf.getBoolean(HColumnDescriptor.CACHE_DATA_IN_L1,
> HColumnDescriptor.DEFAULT_CACHE_DATA_IN_L1) ||
> family.isCacheDataInL1(),
>
> conf.getBoolean(DROP_BEHIND_CACHE_COMPACTION_KEY,DROP_BEHIND_CACHE_COMPACTION_DEFAULT)
> );
> }
> {code}
> If we dig in it, we will see {{CacheConfig.cacheDataOnRead}} is used to
> accept {{family.isBlockCacheEnabled()}}.
> I think it is confused as comments about {{cacheDataOnRead}}
> {code}
> /**
>* Whether blocks should be cached on read (default is on if there is a
>* cache but this can be turned off on a per-family or per-request basis).
>* If off we will STILL cache meta blocks; i.e. INDEX and BLOOM types.
>* This cannot be disabled.
>*/
> private boolean cacheDataOnRead;
> {code}
> So i think we should use another variable to represent for
> {{family.isBlockCacheEnabled()}}.
> The secondary point is we use 'or' to decide {{cacheDataOnWrite}} is on/off
> when both CF and global has this setting.
> {code}
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite()
> {code}
> Should we use CF Level setting to override global setting?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978539#comment-14978539
]
Heng Chen commented on HBASE-14189:
---
{quote}
Seems this is not done in this patch.
{quote}
yeah.. I am not sure whether we should do it...
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch, HBASE-14189_v3.patch
>
>
> The original description is ambiguous. I think i will rewrite it.
> Let's see {{BlockCache}} constructor firstly
> {code}
> public CacheConfig(Configuration conf, HColumnDescriptor family) {
> this(CacheConfig.instantiateBlockCache(conf),
> family.isBlockCacheEnabled(),
> family.isInMemory(),
> // For the following flags we enable them regardless of per-schema
> settings
> // if they are enabled in the global configuration.
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite(),
> conf.getBoolean(CACHE_INDEX_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_INDEXES_ON_WRITE) || family.isCacheIndexesOnWrite(),
> conf.getBoolean(CACHE_BLOOM_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_BLOOMS_ON_WRITE) || family.isCacheBloomsOnWrite(),
> conf.getBoolean(EVICT_BLOCKS_ON_CLOSE_KEY,
> DEFAULT_EVICT_ON_CLOSE) || family.isEvictBlocksOnClose(),
> conf.getBoolean(CACHE_DATA_BLOCKS_COMPRESSED_KEY,
> DEFAULT_CACHE_DATA_COMPRESSED),
> conf.getBoolean(PREFETCH_BLOCKS_ON_OPEN_KEY,
> DEFAULT_PREFETCH_ON_OPEN) || family.isPrefetchBlocksOnOpen(),
> conf.getBoolean(HColumnDescriptor.CACHE_DATA_IN_L1,
> HColumnDescriptor.DEFAULT_CACHE_DATA_IN_L1) ||
> family.isCacheDataInL1(),
>
> conf.getBoolean(DROP_BEHIND_CACHE_COMPACTION_KEY,DROP_BEHIND_CACHE_COMPACTION_DEFAULT)
> );
> }
> {code}
> If we dig in it, we will see {{CacheConfig.cacheDataOnRead}} is used to
> accept {{family.isBlockCacheEnabled()}}.
> I think it is confused as comments about {{cacheDataOnRead}}
> {code}
> /**
>* Whether blocks should be cached on read (default is on if there is a
>* cache but this can be turned off on a per-family or per-request basis).
>* If off we will STILL cache meta blocks; i.e. INDEX and BLOOM types.
>* This cannot be disabled.
>*/
> private boolean cacheDataOnRead;
> {code}
> So i think we should use another variable to represent for
> {{family.isBlockCacheEnabled()}}.
> The secondary point is we use 'or' to decide {{cacheDataOnWrite}} is on/off
> when both CF and global has this setting.
> {code}
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite()
> {code}
> Should we use CF Level setting to override global setting?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976208#comment-14976208
]
Heng Chen commented on HBASE-14189:
---
I rewrite the description of this issue. Should we pick it up again? Any
suggestions?
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch, HBASE-14189_v3.patch
>
>
> The original description is ambiguous. I think i will rewrite it.
> Let see {{BlockCache}} constructor
> {code}
> public CacheConfig(Configuration conf, HColumnDescriptor family) {
> this(CacheConfig.instantiateBlockCache(conf),
> family.isBlockCacheEnabled(),
> family.isInMemory(),
> // For the following flags we enable them regardless of per-schema
> settings
> // if they are enabled in the global configuration.
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite(),
> conf.getBoolean(CACHE_INDEX_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_INDEXES_ON_WRITE) || family.isCacheIndexesOnWrite(),
> conf.getBoolean(CACHE_BLOOM_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_BLOOMS_ON_WRITE) || family.isCacheBloomsOnWrite(),
> conf.getBoolean(EVICT_BLOCKS_ON_CLOSE_KEY,
> DEFAULT_EVICT_ON_CLOSE) || family.isEvictBlocksOnClose(),
> conf.getBoolean(CACHE_DATA_BLOCKS_COMPRESSED_KEY,
> DEFAULT_CACHE_DATA_COMPRESSED),
> conf.getBoolean(PREFETCH_BLOCKS_ON_OPEN_KEY,
> DEFAULT_PREFETCH_ON_OPEN) || family.isPrefetchBlocksOnOpen(),
> conf.getBoolean(HColumnDescriptor.CACHE_DATA_IN_L1,
> HColumnDescriptor.DEFAULT_CACHE_DATA_IN_L1) ||
> family.isCacheDataInL1(),
>
> conf.getBoolean(DROP_BEHIND_CACHE_COMPACTION_KEY,DROP_BEHIND_CACHE_COMPACTION_DEFAULT)
> );
> }
> {code}
> If we dig in it, we will see {{CacheConfig.cacheDataOnRead}} is used to
> accept {{family.isBlockCacheEnabled()}}.
> I think it is confused as comments about {{cacheDataOnRead}}
> {code}
> /**
>* Whether blocks should be cached on read (default is on if there is a
>* cache but this can be turned off on a per-family or per-request basis).
>* If off we will STILL cache meta blocks; i.e. INDEX and BLOOM types.
>* This cannot be disabled.
>*/
> private boolean cacheDataOnRead;
> {code}
> So i think we should use another variable to represent for
> {{family.isBlockCacheEnabled()}}.
> The secondary point is we use 'or' to decide {{cacheDataOnWrite}} is on/off
> when both CF and global has this setting.
> {code}
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite()
> {code}
> Should we use CF Level setting to override global setting?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723207#comment-14723207
]
Heng Chen commented on HBASE-14189:
---
Any suggestions? [~anoop.hbase]
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch, HBASE-14189_v3.patch
>
>
> While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
> represents for whether we should cache block after read(write) block from(to)
> hdfs. We should honour BC setting and CF Level cache setting while using
> BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723213#comment-14723213
]
Anoop Sam John commented on HBASE-14189:
Sorry for being late.. I was out of work for some days. Will see it day after
tomorrow.
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch, HBASE-14189_v3.patch
>
>
> While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
> represents for whether we should cache block after read(write) block from(to)
> hdfs. We should honour BC setting and CF Level cache setting while using
> BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711294#comment-14711294
]
Heng Chen commented on HBASE-14189:
---
Any concerns? [~anoop.hbase] [~Apache9] [~ndimiduk]
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
HBASE-14189_v2.patch, HBASE-14189_v3.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710796#comment-14710796
]
Hadoop QA commented on HBASE-14189:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12752140/HBASE-14189_v2.patch
against master branch at commit d0873f5a8cc060adbc5a1ae0ed52b84a8942a868.
ATTACHMENT ID: 12752140
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15239//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15239//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15239//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15239//console
This message is automatically generated.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
HBASE-14189_v2.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711279#comment-14711279
]
Hadoop QA commented on HBASE-14189:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12752194/HBASE-14189_v3.patch
against master branch at commit d0873f5a8cc060adbc5a1ae0ed52b84a8942a868.
ATTACHMENT ID: 12752194
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15243//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15243//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15243//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15243//console
This message is automatically generated.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
HBASE-14189_v2.patch, HBASE-14189_v3.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702642#comment-14702642
]
Anoop Sam John commented on HBASE-14189:
bq.private boolean cfBlockCacheEnabled;
So this indicates whether the cache on read in ON at CF level.
I see getters in HCD for the cache on read and cache compressed data. WHat abt
setters?
When cache on read is OFF at CF level but cache on write is ON, what we will
do? Ideally at that time we should not cache at read and write time. Also make
sure we have enough test coverage for all combinations. Per read level also,
user can specify whether to cache on read. If it is OFF at read level, we
should not cache the read blocks. But we may expect the block to be available
in cache and look for it in cache. (I think this is already done)
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch, HBASE-14189_v1.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702710#comment-14702710
]
Heng Chen commented on HBASE-14189:
---
Thanks for your reply!
1.
{quote}
So this indicates whether the cache on read in ON at CF level.
{quote}
No no..
{{cfBlockCacheEnabled}} only means whether BC is enabled or not at CF Level, it
refer to the HCD schema option {{BLOCKCACHE}}.
If it is OFF, all BC operations will be disabled no matter write or read.
So there need some changes in patch, and {{shouldCacheBlockOnRead}} should be
something like below:
{code}
public boolean shouldCacheBlockOnRead(BlockCategory category) {
if (!isBlockCacheEnabled()) {
return false;
}
// skip CF Level BC check
if (category == BlockCategory.INDEX ||
category == BlockCategory.BLOOM ||
(prefetchOnOpen
(category != BlockCategory.META
category != BlockCategory.UNKNOWN))) {
return true;
}
return shouldCacheDataOnRead();
}
{code}
2.
{quote}
When cache on read is OFF at CF level but cache on write is ON, what we will do?
{quote}
{{cacheDataOnRead}} means we will cache block after read it. and
{{cacheDataOnWrite}} means we will cache block after writer it.
So if {{cacheDataOnRead}} is OFF and {{cacheDataOnWrite}} is ON, we will not
cache block after read, but cache it after write, and when we read the block,
we should read it from cache.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch, HBASE-14189_v1.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699329#comment-14699329
]
Hadoop QA commented on HBASE-14189:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12750758/HBASE-14189_v1.patch
against master branch at commit 737f264509284420e6fa8c14d92fe9fbdb49f67f.
ATTACHMENT ID: 12750758
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15125//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15125//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15125//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15125//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15125//console
This message is automatically generated.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch, HBASE-14189_v1.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699525#comment-14699525
]
Hadoop QA commented on HBASE-14189:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12750789/HBASE-14189_v1.patch
against master branch at commit 737f264509284420e6fa8c14d92fe9fbdb49f67f.
ATTACHMENT ID: 12750789
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning(TestHRegion.java:3798)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15127//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15127//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15127//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15127//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15127//console
This message is automatically generated.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch, HBASE-14189_v1.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695105#comment-14695105
]
Hadoop QA commented on HBASE-14189:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12750245/HBASE-14189_v1.patch
against master branch at commit 9c69bf766fcad024bef5760f242cae2bc609b374.
ATTACHMENT ID: 12750245
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testWALfencing(TestMasterFailoverWithProcedures.java:216)
at
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testWALfencingWithWALRolling(TestMasterFailoverWithProcedures.java:183)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15083//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15083//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15083//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15083//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15083//console
This message is automatically generated.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch, HBASE-14189_v1.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694221#comment-14694221
]
Heng Chen commented on HBASE-14189:
---
[~ndimiduk] any suggestions?
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694683#comment-14694683
]
Anoop Sam John commented on HBASE-14189:
I think the changes we are trying here in this patch is too much.. Not really
convinced abt the removal of cache on write stuff. And what is the new config
added used for?
I agree that we can have CF level setting for cache data on read and cache
compressed data while read. This change has to any way be backward compatible
also. Means we should continue to support global level setting (for cache data
in compressed form)
When we have cache on write and read settings we have to make sure which gets
precedence. That is what was missing/confusing in CacheConfig. We can correct
them. IMO the cache on read related values should override the on write values.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679606#comment-14679606
]
Hadoop QA commented on HBASE-14189:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12749511/HBASE-14189.patch
against master branch at commit 5bdb0eb91290e306213bca62cea82c5d1b24d317.
ATTACHMENT ID: 12749511
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 44 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1863 checkstyle errors (more than the master's current 1862 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 8 zombie test(s):
at
org.apache.hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient.testCloneSnapshot(TestMobRestoreFlushSnapshotFromClient.java:188)
at
org.apache.hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient.testCloneSnapshot(TestMobRestoreFlushSnapshotFromClient.java:179)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:288)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:262)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:195)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWriteInternals(TestCacheOnWrite.java:265)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWrite(TestCacheOnWrite.java:492)
at
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.testBulkOutputWithTsvImporterTextMapper(TestImportTSVWithVisibilityLabels.java:256)
at
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot.testExportFileSystemState(TestMobExportSnapshot.java:285)
at
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot.testExportFileSystemState(TestMobExportSnapshot.java:259)
at
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot.testSnapshotWithRefsExportFileSystemState(TestMobExportSnapshot.java:253)
at
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot.testSnapshotWithRefsExportFileSystemState(TestMobExportSnapshot.java:233)
at
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot.testExportFileSystemState(TestMobExportSnapshot.java:285)
at
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot.testExportFileSystemState(TestMobExportSnapshot.java:259)
at
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot.testExportFileSystemStateWithSkipTmp(TestMobExportSnapshot.java:198)
at
org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders.testSeekingOnSample(TestDataBlockEncoders.java:217)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15024//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15024//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15024//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15024//console
This message is automatically generated.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch
While using
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662728#comment-14662728
]
Hadoop QA commented on HBASE-14189:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12749342/HBASE-14189.patch
against master branch at commit f1c1692d511f72ba0bef42e22d27098c25bd10d9.
ATTACHMENT ID: 12749342
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 44 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1863 checkstyle errors (more than the master's current 1862 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestRegionServerHostname
{color:red}-1 core zombie tests{color}. There are 5 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoadWithSplit(TestHFileOutputFormat.java:378)
at
org.apache.hadoop.hbase.mapreduce.TestCellCounter.testCellCounterForCompleteTable(TestCellCounter.java:301)
at
org.apache.hadoop.hbase.mapreduce.TestImportTsv.testBulkOutputWithTsvImporterTextMapper(TestImportTsv.java:247)
at
org.apache.hadoop.hbase.mapreduce.TestRowCounter.testRowCounterExclusiveColumn(TestRowCounter.java:118)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15003//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15003//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15003//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15003//console
This message is automatically generated.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661765#comment-14661765
]
Hadoop QA commented on HBASE-14189:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12749228/HBASE-14189.patch
against master branch at commit 741783585306e03eec8074841b342ab742cf37e7.
ATTACHMENT ID: 12749228
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 44 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1862 checkstyle errors (more than the master's current 1861 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestRegionServerHostname
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14995//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14995//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14995//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14995//console
This message is automatically generated.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
Attachments: HBASE-14189.patch
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14659804#comment-14659804
]
Heng Chen commented on HBASE-14189:
---
After review the code in BlockConfig, I has a plan.
I will submit a patch as soon as possible.
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Reporter: Heng Chen
Assignee: Heng Chen
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661240#comment-14661240
]
Heng Chen commented on HBASE-14189:
---
Now the options about cache configuration are below:
{code}
/**
* Whether blocks should be cached on read (default is on if there is a
* cache but this can be turned off on a per-family or per-request basis).
* If off we will STILL cache meta blocks; i.e. INDEX and BLOOM types.
* This cannot be disabled.
*/
private boolean cacheDataOnRead;
/** Whether blocks should be flagged as in-memory when being cached */
private final boolean inMemory;
/** Whether data blocks should be cached when new files are written */
private boolean cacheDataOnWrite;
/** Whether index blocks should be cached when new files are written */
private final boolean cacheIndexesOnWrite;
/** Whether compound bloom filter blocks should be cached on write */
private final boolean cacheBloomsOnWrite;
/** Whether blocks of a file should be evicted when the file is closed */
private boolean evictOnClose;
/** Whether data blocks should be stored in compressed and/or encrypted form
in the cache */
private final boolean cacheDataCompressed;
/** Whether data blocks should be prefetched into the cache */
private final boolean prefetchOnOpen;
/**
* If true and if more than one tier in this cache deploy -- e.g.
CombinedBlockCache has an L1
* and an L2 tier -- then cache data blocks up in the L1 tier (The meta
blocks are likely being
* cached up in L1 already. At least this is the case if CombinedBlockCache).
*/
private boolean cacheDataInL1;
{code}
I think all the options are CF Level Cache Setting, and they are useless for
the whole RS cache setting.
The whole RS cache only cares about whether the BC is enabled or not, and it
has no business with how to use BC
So i think the options in config as like blow are useless, and i will remove
them. At the same time, i will add one option
to decide whether open BC or not, for example, {{hbase.rs.blockcacheopen}}
{code}
public static final String CACHE_BLOCKS_ON_READ_KEY =
hbase.rs.cacheblockonread;
/**
* Configuration key to cache data blocks on write. There are separate
* switches for bloom blocks and non-root index blocks.
*/
public static final String CACHE_BLOCKS_ON_WRITE_KEY =
hbase.rs.cacheblocksonwrite;
/**
* Configuration key to cache leaf and intermediate-level index blocks on
* write.
*/
public static final String CACHE_INDEX_BLOCKS_ON_WRITE_KEY =
hfile.block.index.cacheonwrite;
/**
* Configuration key to cache compound bloom filter blocks on write.
*/
public static final String CACHE_BLOOM_BLOCKS_ON_WRITE_KEY =
hfile.block.bloom.cacheonwrite;
/**
* Configuration key to cache data blocks in compressed and/or encrypted
format.
*/
public static final String CACHE_DATA_BLOCKS_COMPRESSED_KEY =
hbase.block.data.cachecompressed;
/**
* Configuration key to evict all blocks of a given file from the block cache
* when the file is closed.
*/
public static final String EVICT_BLOCKS_ON_CLOSE_KEY =
hbase.rs.evictblocksonclose;
{code}
Am i right?
BlockCache options should consider CF Level BlockCacheEnabled setting
-
Key: HBASE-14189
URL: https://issues.apache.org/jira/browse/HBASE-14189
Project: HBase
Issue Type: Improvement
Components: BlockCache
Reporter: Heng Chen
Assignee: Heng Chen
While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
represents for whether we should cache block after read(write) block from(to)
hdfs. We should honour BC setting and CF Level cache setting while using
BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
