[jira] [Commented] (HBASE-14768) bin/graceful_stop.sh logs nothing as a balancer state to be stored
[
https://issues.apache.org/jira/browse/HBASE-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991300#comment-14991300
]
Hudson commented on HBASE-14768:
SUCCESS: Integrated in HBase-1.3 #349 (See
[https://builds.apache.org/job/HBase-1.3/349/])
HBASE-14768 bin/graceful_stop.sh logs nothing as a balancer state to be (stack:
rev 32642ac97116ad14b143ab20fb908020dac3f88d)
* bin/graceful_stop.sh
> bin/graceful_stop.sh logs nothing as a balancer state to be stored
> --
>
> Key: HBASE-14768
> URL: https://issues.apache.org/jira/browse/HBASE-14768
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14768.patch
>
>
> In bin/graceful_stop.sh
> {code}
> log() {
> echo `date +%Y-%m-%dT%H:%M:%S` $1
> }
> ...skip...
> if [ $HBASE_BALANCER_STATE != "false" ]; then
> log "Restoring balancer state to " $HBASE_BALANCER_STATE
> {code}
> The position of the above last double quotation mark is wrong, and the
> balancer state will be not shown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14767) Remove deprecated functions from HBaseAdmin
[
https://issues.apache.org/jira/browse/HBASE-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991307#comment-14991307
]
Hadoop QA commented on HBASE-14767:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770745/HBASE-14767-master-v2.patch
against master branch at commit 21b607e322df0404c9941bc3dd12faef6b7957fe.
ATTACHMENT ID: 12770745
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 112
new or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1727 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16402//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16402//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16402//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16402//console
This message is automatically generated.
> Remove deprecated functions from HBaseAdmin
> ---
>
> Key: HBASE-14767
> URL: https://issues.apache.org/jira/browse/HBASE-14767
> Project: HBase
> Issue Type: Bug
>Reporter: Apekshit Sharma
>Assignee: Apekshit Sharma
> Attachments: HBASE-14767-master-v2.patch, HBASE-14767-master.patch
>
>
> Many functions in HBaseAdmin are marked deprecated. Removing them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14659) Fix flakey TestHFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-14659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen resolved HBASE-14659. --- Resolution: Won't Fix > Fix flakey TestHFileOutputFormat > > > Key: HBASE-14659 > URL: https://issues.apache.org/jira/browse/HBASE-14659 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > > As i said in HBASE-14654 > This testcase was hanged twice recently. > the two QA information: > https://builds.apache.org/job/PreCommit-HBASE-Build/16118/console > https://builds.apache.org/job/PreCommit-HBASE-Build/16117/console > I notice that the two QA running simultaneously on same machine H0. > I doubt If there are some common resources the two QA conflicts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14189) CF Level BC setting should override global one.
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14189:
--
Status: Patch Available (was: Open)
> CF Level BC setting should override global one.
> ---
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch, HBASE-14189_v3.patch, HBASE-14189_v4.patch
>
>
> The original description is ambiguous. I think i will rewrite it.
> Let's see {{BlockCache}} constructor firstly
> {code}
> public CacheConfig(Configuration conf, HColumnDescriptor family) {
> this(CacheConfig.instantiateBlockCache(conf),
> family.isBlockCacheEnabled(),
> family.isInMemory(),
> // For the following flags we enable them regardless of per-schema
> settings
> // if they are enabled in the global configuration.
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite(),
> conf.getBoolean(CACHE_INDEX_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_INDEXES_ON_WRITE) || family.isCacheIndexesOnWrite(),
> conf.getBoolean(CACHE_BLOOM_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_BLOOMS_ON_WRITE) || family.isCacheBloomsOnWrite(),
> conf.getBoolean(EVICT_BLOCKS_ON_CLOSE_KEY,
> DEFAULT_EVICT_ON_CLOSE) || family.isEvictBlocksOnClose(),
> conf.getBoolean(CACHE_DATA_BLOCKS_COMPRESSED_KEY,
> DEFAULT_CACHE_DATA_COMPRESSED),
> conf.getBoolean(PREFETCH_BLOCKS_ON_OPEN_KEY,
> DEFAULT_PREFETCH_ON_OPEN) || family.isPrefetchBlocksOnOpen(),
> conf.getBoolean(HColumnDescriptor.CACHE_DATA_IN_L1,
> HColumnDescriptor.DEFAULT_CACHE_DATA_IN_L1) ||
> family.isCacheDataInL1(),
>
> conf.getBoolean(DROP_BEHIND_CACHE_COMPACTION_KEY,DROP_BEHIND_CACHE_COMPACTION_DEFAULT)
> );
> }
> {code}
> If we dig in it, we will see {{CacheConfig.cacheDataOnRead}} is used to
> accept {{family.isBlockCacheEnabled()}}.
> I think it is confused as comments about {{cacheDataOnRead}}
> {code}
> /**
>* Whether blocks should be cached on read (default is on if there is a
>* cache but this can be turned off on a per-family or per-request basis).
>* If off we will STILL cache meta blocks; i.e. INDEX and BLOOM types.
>* This cannot be disabled.
>*/
> private boolean cacheDataOnRead;
> {code}
> So i think we should use another variable to represent for
> {{family.isBlockCacheEnabled()}}.
> The secondary point is we use 'or' to decide {{cacheDataOnWrite}} is on/off
> when both CF and global has this setting.
> {code}
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite()
> {code}
> IMO we should use CF Level setting to override global setting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14768) bin/graceful_stop.sh logs nothing as a balancer state to be stored
[
https://issues.apache.org/jira/browse/HBASE-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991303#comment-14991303
]
Hudson commented on HBASE-14768:
FAILURE: Integrated in HBase-1.3-IT #295 (See
[https://builds.apache.org/job/HBase-1.3-IT/295/])
HBASE-14768 bin/graceful_stop.sh logs nothing as a balancer state to be (stack:
rev 32642ac97116ad14b143ab20fb908020dac3f88d)
* bin/graceful_stop.sh
> bin/graceful_stop.sh logs nothing as a balancer state to be stored
> --
>
> Key: HBASE-14768
> URL: https://issues.apache.org/jira/browse/HBASE-14768
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14768.patch
>
>
> In bin/graceful_stop.sh
> {code}
> log() {
> echo `date +%Y-%m-%dT%H:%M:%S` $1
> }
> ...skip...
> if [ $HBASE_BALANCER_STATE != "false" ]; then
> log "Restoring balancer state to " $HBASE_BALANCER_STATE
> {code}
> The position of the above last double quotation mark is wrong, and the
> balancer state will be not shown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14768) bin/graceful_stop.sh logs nothing as a balancer state to be stored
[
https://issues.apache.org/jira/browse/HBASE-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991373#comment-14991373
]
Hudson commented on HBASE-14768:
FAILURE: Integrated in HBase-1.2 #349 (See
[https://builds.apache.org/job/HBase-1.2/349/])
HBASE-14768 bin/graceful_stop.sh logs nothing as a balancer state to be (stack:
rev 97f7857b7be4eae565b6b0bb0cc0c25aa25ccc9e)
* bin/graceful_stop.sh
> bin/graceful_stop.sh logs nothing as a balancer state to be stored
> --
>
> Key: HBASE-14768
> URL: https://issues.apache.org/jira/browse/HBASE-14768
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14768.patch
>
>
> In bin/graceful_stop.sh
> {code}
> log() {
> echo `date +%Y-%m-%dT%H:%M:%S` $1
> }
> ...skip...
> if [ $HBASE_BALANCER_STATE != "false" ]; then
> log "Restoring balancer state to " $HBASE_BALANCER_STATE
> {code}
> The position of the above last double quotation mark is wrong, and the
> balancer state will be not shown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14755) Fix some broken links and HTML problems
[
https://issues.apache.org/jira/browse/HBASE-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991299#comment-14991299
]
Hadoop QA commented on HBASE-14755:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770742/HBASE-14755-v1.patch
against master branch at commit 21b607e322df0404c9941bc3dd12faef6b7957fe.
ATTACHMENT ID: 12770742
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ The full HBase API test code, including private
and unstable APIs
+
org.apache.hadoop.hbase.backup*:org.apache.hadoop.hbase.catalog:org.apache.hadoop.hbase.client.coprocessor:org.apache.hadoop.hbase.client.metrics:org.apache.hadoop.hbase.codec*:org.apache.hadoop.hbase.constraint:org.apache.hadoop.hbase.coprocessor.*:org.apache.hadoop.hbase.executor:org.apache.hadoop.hbase.fs:*.generated.*:org.apache.hadoop.hbase.io.hfile.*:org.apache.hadoop.hbase.mapreduce.hadoopbackport:org.apache.hadoop.hbase.mapreduce.replication:org.apache.hadoop.hbase.master.*:org.apache.hadoop.hbase.metrics*:org.apache.hadoop.hbase.migration:org.apache.hadoop.hbase.monitoring:org.apache.hadoop.hbase.p*:org.apache.hadoop.hbase.regionserver.compactions:org.apache.hadoop.hbase.regionserver.handler:org.apache.hadoop.hbase.regionserver.snapshot:org.apache.hadoop.hbase.replication.*:org.apache.hadoop.hbase.rest.filter:org.apache.hadoop.hbase.rest.model:org.apache.hadoop.hbase.rest.p*:org.apache.hadoop.hbase.security.*:org.apache.hadoop.hbase.thrift*:org.apache.hadoop.hbase.tmpl.*:org.apache.hadoop.hbase.tool:org.apache.hadoop.hbase.trace:org.apache.hadoop.hbase.util.byterange*:org.apache.hadoop.hbase.util.test:org.apache.hadoop.hbase.util.vint:org.apache.hadoop.hbase.zookeeper.lock:org.apache.hadoop.metrics2*:org.apache.hadoop.hbase.io.compress*
+
org.apache.hadoop.hbase.backup*:org.apache.hadoop.hbase.catalog:org.apache.hadoop.hbase.client.coprocessor:org.apache.hadoop.hbase.client.metrics:org.apache.hadoop.hbase.codec*:org.apache.hadoop.hbase.constraint:org.apache.hadoop.hbase.coprocessor.*:org.apache.hadoop.hbase.executor:org.apache.hadoop.hbase.fs:*.generated.*:org.apache.hadoop.hbase.io.hfile.*:org.apache.hadoop.hbase.mapreduce.hadoopbackport:org.apache.hadoop.hbase.mapreduce.replication:org.apache.hadoop.hbase.master.*:org.apache.hadoop.hbase.metrics*:org.apache.hadoop.hbase.migration:org.apache.hadoop.hbase.monitoring:org.apache.hadoop.hbase.p*:org.apache.hadoop.hbase.regionserver.compactions:org.apache.hadoop.hbase.regionserver.handler:org.apache.hadoop.hbase.regionserver.snapshot:org.apache.hadoop.hbase.replication.*:org.apache.hadoop.hbase.rest.filter:org.apache.hadoop.hbase.rest.model:org.apache.hadoop.hbase.rest.p*:org.apache.hadoop.hbase.security.*:org.apache.hadoop.hbase.thrift*:org.apache.hadoop.hbase.tmpl.*:org.apache.hadoop.hbase.tool:org.apache.hadoop.hbase.trace:org.apache.hadoop.hbase.util.byterange*:org.apache.hadoop.hbase.util.test:org.apache.hadoop.hbase.util.vint:org.apache.hadoop.hbase.zookeeper.lock:org.apache.hadoop.metrics2*:org.apache.hadoop.hbase.io.compress*
+. 8 bytes: Block type, a sequence of bytes equivalent to version 1's "magic
records". Supported block types are:
+ As opposed to the above, this is not an HFile v2 block but a fixed>size (for
each HFile version) data structure
+We are supporting "meta" blocks in version 2 the same way they were supported
in version 1, even though we do not store Bloom filter data in these blocks
anymore.
+There are three types of block indexes in HFile version 2, stored in two
different formats (root and non-root):
+.. Optionally, version 2 intermediate levels, stored in the non%root format in
the data index section of the file. Intermediate levels can only be present
if leaf level blocks are present
+A version 2
[jira] [Updated] (HBASE-14189) CF Level BC setting should override global one.
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14189:
--
Attachment: HBASE-14189_v4.patch
update patch:
* use cf level setting override global setting.
> CF Level BC setting should override global one.
> ---
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch, HBASE-14189_v3.patch, HBASE-14189_v4.patch
>
>
> The original description is ambiguous. I think i will rewrite it.
> Let's see {{BlockCache}} constructor firstly
> {code}
> public CacheConfig(Configuration conf, HColumnDescriptor family) {
> this(CacheConfig.instantiateBlockCache(conf),
> family.isBlockCacheEnabled(),
> family.isInMemory(),
> // For the following flags we enable them regardless of per-schema
> settings
> // if they are enabled in the global configuration.
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite(),
> conf.getBoolean(CACHE_INDEX_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_INDEXES_ON_WRITE) || family.isCacheIndexesOnWrite(),
> conf.getBoolean(CACHE_BLOOM_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_BLOOMS_ON_WRITE) || family.isCacheBloomsOnWrite(),
> conf.getBoolean(EVICT_BLOCKS_ON_CLOSE_KEY,
> DEFAULT_EVICT_ON_CLOSE) || family.isEvictBlocksOnClose(),
> conf.getBoolean(CACHE_DATA_BLOCKS_COMPRESSED_KEY,
> DEFAULT_CACHE_DATA_COMPRESSED),
> conf.getBoolean(PREFETCH_BLOCKS_ON_OPEN_KEY,
> DEFAULT_PREFETCH_ON_OPEN) || family.isPrefetchBlocksOnOpen(),
> conf.getBoolean(HColumnDescriptor.CACHE_DATA_IN_L1,
> HColumnDescriptor.DEFAULT_CACHE_DATA_IN_L1) ||
> family.isCacheDataInL1(),
>
> conf.getBoolean(DROP_BEHIND_CACHE_COMPACTION_KEY,DROP_BEHIND_CACHE_COMPACTION_DEFAULT)
> );
> }
> {code}
> If we dig in it, we will see {{CacheConfig.cacheDataOnRead}} is used to
> accept {{family.isBlockCacheEnabled()}}.
> I think it is confused as comments about {{cacheDataOnRead}}
> {code}
> /**
>* Whether blocks should be cached on read (default is on if there is a
>* cache but this can be turned off on a per-family or per-request basis).
>* If off we will STILL cache meta blocks; i.e. INDEX and BLOOM types.
>* This cannot be disabled.
>*/
> private boolean cacheDataOnRead;
> {code}
> So i think we should use another variable to represent for
> {{family.isBlockCacheEnabled()}}.
> The secondary point is we use 'or' to decide {{cacheDataOnWrite}} is on/off
> when both CF and global has this setting.
> {code}
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite()
> {code}
> IMO we should use CF Level setting to override global setting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14768) bin/graceful_stop.sh logs nothing as a balancer state to be stored
[
https://issues.apache.org/jira/browse/HBASE-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991312#comment-14991312
]
Hudson commented on HBASE-14768:
FAILURE: Integrated in HBase-Trunk_matrix #436 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/436/])
HBASE-14768 bin/graceful_stop.sh logs nothing as a balancer state to be (stack:
rev 050ebe850b32057860fb94b46f955352db139db1)
* bin/graceful_stop.sh
> bin/graceful_stop.sh logs nothing as a balancer state to be stored
> --
>
> Key: HBASE-14768
> URL: https://issues.apache.org/jira/browse/HBASE-14768
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14768.patch
>
>
> In bin/graceful_stop.sh
> {code}
> log() {
> echo `date +%Y-%m-%dT%H:%M:%S` $1
> }
> ...skip...
> if [ $HBASE_BALANCER_STATE != "false" ]; then
> log "Restoring balancer state to " $HBASE_BALANCER_STATE
> {code}
> The position of the above last double quotation mark is wrong, and the
> balancer state will be not shown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14693) Add client-side metrics for received pushback signals
[ https://issues.apache.org/jira/browse/HBASE-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-14693: -- Attachment: HBASE-14693_v4.patch Fix typo and checkstyle errors. > Add client-side metrics for received pushback signals > - > > Key: HBASE-14693 > URL: https://issues.apache.org/jira/browse/HBASE-14693 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Heng Chen > Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16 > > Attachments: HBASE-14693.patch, HBASE-14693_v1.patch, > HBASE-14693_v2.patch, HBASE-14693_v3.patch, HBASE-14693_v4.patch > > > HBASE-12911 added client side metrics. HBASE-5162 added a mechanism for > sending advisory backpressure signals to clients when the server is heavily > loaded, and HBASE-12702 and subtasks backported this to all active branches. > Add client-side metrics for received pushback signal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14712) MasterProcWALs never clean up
[
https://issues.apache.org/jira/browse/HBASE-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991328#comment-14991328
]
Hadoop QA commented on HBASE-14712:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770746/HBASE-14712-v1.patch
against master branch at commit 21b607e322df0404c9941bc3dd12faef6b7957fe.
ATTACHMENT ID: 12770746
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16401//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16401//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16401//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16401//console
This message is automatically generated.
> MasterProcWALs never clean up
> -
>
> Key: HBASE-14712
> URL: https://issues.apache.org/jira/browse/HBASE-14712
> Project: HBase
> Issue Type: Bug
>Reporter: Elliott Clark
>Assignee: Matteo Bertozzi
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14712-v0.patch, HBASE-14712-v1.patch, state.tar.gz
>
>
> MasterProcWALs directory grows pretty much un-bounded. Because of that when
> master failover happens the NN is flooded with connections and everything
> grinds to a halt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14768) bin/graceful_stop.sh logs nothing as a balancer state to be stored
[
https://issues.apache.org/jira/browse/HBASE-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991347#comment-14991347
]
Hudson commented on HBASE-14768:
FAILURE: Integrated in HBase-1.2-IT #265 (See
[https://builds.apache.org/job/HBase-1.2-IT/265/])
HBASE-14768 bin/graceful_stop.sh logs nothing as a balancer state to be (stack:
rev 97f7857b7be4eae565b6b0bb0cc0c25aa25ccc9e)
* bin/graceful_stop.sh
> bin/graceful_stop.sh logs nothing as a balancer state to be stored
> --
>
> Key: HBASE-14768
> URL: https://issues.apache.org/jira/browse/HBASE-14768
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14768.patch
>
>
> In bin/graceful_stop.sh
> {code}
> log() {
> echo `date +%Y-%m-%dT%H:%M:%S` $1
> }
> ...skip...
> if [ $HBASE_BALANCER_STATE != "false" ]; then
> log "Restoring balancer state to " $HBASE_BALANCER_STATE
> {code}
> The position of the above last double quotation mark is wrong, and the
> balancer state will be not shown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Status: Patch Available (was: Open)
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-async.patch, HBASE-14703-start.patch,
> HBASE-14703.patch, HBASE-14703_v1.patch, HBASE-14703_v2.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14659) Fix flakey TestHFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-14659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991324#comment-14991324 ] Heng Chen commented on HBASE-14659: --- This class is removed. So resolve this issue. > Fix flakey TestHFileOutputFormat > > > Key: HBASE-14659 > URL: https://issues.apache.org/jira/browse/HBASE-14659 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > > As i said in HBASE-14654 > This testcase was hanged twice recently. > the two QA information: > https://builds.apache.org/job/PreCommit-HBASE-Build/16118/console > https://builds.apache.org/job/PreCommit-HBASE-Build/16117/console > I notice that the two QA running simultaneously on same machine H0. > I doubt If there are some common resources the two QA conflicts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14498) Master stuck in infinite loop when all Zookeeper servers are unreachable.
[
https://issues.apache.org/jira/browse/HBASE-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991727#comment-14991727
]
Ted Yu commented on HBASE-14498:
For the abort message, suggest adding the actual wait duration.
Is it possible to add a test ?
Thanks
> Master stuck in infinite loop when all Zookeeper servers are unreachable.
> -
>
> Key: HBASE-14498
> URL: https://issues.apache.org/jira/browse/HBASE-14498
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Pankaj Kumar
>Priority: Blocker
> Attachments: HBASE-14498.patch
>
>
> We met a weird scenario in our production environment.
> In a HA cluster,
> > Active Master (HM1) is not able to connect to any Zookeeper server (due to
> > N/w breakdown on master machine network with Zookeeper servers).
> {code}
> 2015-09-26 15:24:47,508 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 33463ms for sessionid 0x104576b8dda0002, closing socket connection and
> attempting reconnect
> 2015-09-26 15:24:47,877 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:48,236 INFO [main-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:49,879 WARN
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:49,879 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-IP1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:24:50,238 WARN [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:50,238 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-Host1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:25:17,470 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 30023ms for sessionid 0x2045762cc710006, closing socket connection and
> attempting reconnect
> 2015-09-26 15:25:17,571 WARN [master/HM1-Host/HM1-IP:16000]
> zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
> quorum=ZK-Host:2181,ZK-Host1:2181,ZK-Host2:2181,
> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2015-09-26 15:25:17,872 INFO [main-SendThread(ZK-Host:2181)]
> client.FourLetterWordMain: connecting to ZK-Host 2181
> 2015-09-26 15:25:19,874 WARN [main-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host
> 2015-09-26 15:25:19,874 INFO [main-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server ZK-Host/ZK-IP:2181.
> Will not attempt to authenticate using SASL (unknown error)
> {code}
> > Since HM1 was not able to connect to any ZK, so session timeout didnt
> > happen at Zookeeper server side and HM1 didnt abort.
> > On Zookeeper session timeout standby master (HM2) registered himself as an
> > active master.
> > HM2 is keep on waiting for region server to report him as part of active
> > master intialization.
> {noformat}
> 2015-09-26 15:24:44,928 | INFO | HM2-Host:21300.activeMasterManager | Waiting
> for region servers count to settle; currently checked in 0, slept for 0 ms,
> expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval
> of 1500 ms. |
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> ---
> ---
> 2015-09-26 15:32:50,841 | INFO | HM2-Host:21300.activeMasterManager | Waiting
> for region servers count to settle; currently checked in 0, slept for 483913
> ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms,
> interval of 1500 ms. |
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> {noformat}
> > At other end, region servers are reporting to HM1 on 3 sec interval. Here
> > region server retrieve master location from zookeeper only when they
> > couldn't connect to Master (ServiceException).
> Region Server will not report HM2 as per current design until unless HM1
> abort,so HM2 will exit(InitializationMonitor) and again wait for region
> servers in loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14771) RpcServer.getRemoteAddress always returns null.
[
https://issues.apache.org/jira/browse/HBASE-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14771:
---
Status: Patch Available (was: Open)
> RpcServer.getRemoteAddress always returns null.
> ---
>
> Key: HBASE-14771
> URL: https://issues.apache.org/jira/browse/HBASE-14771
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 1.2.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14771.patch
>
>
> RpcServer.getRemoteAddress always returns null, because Call object is
> getting initialized with null.This seems to be happening because of using
> RpcServer.getRemoteIp() in Call object constructor before RpcServer thread
> local 'CurCall' being set in CallRunner.run method:
> {noformat}
> // --- RpcServer.java ---
> protected void processRequest(byte[] buf) throws IOException,
> InterruptedException {
> .
> // Call object getting initialized here with address
> // obtained from RpcServer.getRemoteIp()
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, RpcServer.getRemoteIp());
> scheduler.dispatch(new CallRunner(RpcServer.this, call));
> }
> // getRemoteIp method gets address from threadlocal 'CurCall' which
> // gets set in CallRunner.run and calling it before this as in above case,
> will return null
> // --- CallRunner.java ---
> public void run() {
> .
> Pair resultPair = null;
> RpcServer.CurCall.set(call);
> ..
> }
> // Using 'this.addr' in place of getRemoteIp method in RpcServer.java seems
> to be fixing this issue
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, this.addr);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14498) Master stuck in infinite loop when all Zookeeper servers are unreachable.
[
https://issues.apache.org/jira/browse/HBASE-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991732#comment-14991732
]
Hadoop QA commented on HBASE-14498:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770788/HBASE-14498.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770788
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16408//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16408//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16408//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16408//console
This message is automatically generated.
> Master stuck in infinite loop when all Zookeeper servers are unreachable.
> -
>
> Key: HBASE-14498
> URL: https://issues.apache.org/jira/browse/HBASE-14498
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Pankaj Kumar
>Priority: Blocker
> Attachments: HBASE-14498.patch
>
>
> We met a weird scenario in our production environment.
> In a HA cluster,
> > Active Master (HM1) is not able to connect to any Zookeeper server (due to
> > N/w breakdown on master machine network with Zookeeper servers).
> {code}
> 2015-09-26 15:24:47,508 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 33463ms for sessionid 0x104576b8dda0002, closing socket connection and
> attempting reconnect
> 2015-09-26 15:24:47,877 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:48,236 INFO [main-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:49,879 WARN
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:49,879 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-IP1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:24:50,238 WARN [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:50,238 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-Host1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:25:17,470 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 30023ms for sessionid 0x2045762cc710006, closing socket connection and
> attempting reconnect
> 2015-09-26 15:25:17,571 WARN [master/HM1-Host/HM1-IP:16000]
> zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
> quorum=ZK-Host:2181,ZK-Host1:2181,ZK-Host2:2181,
>
[jira] [Commented] (HBASE-14737) Clear cachedMaxVersions when setValue(VERSIONS, value) called
[
https://issues.apache.org/jira/browse/HBASE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991733#comment-14991733
]
Hadoop QA commented on HBASE-14737:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770787/HBASE-14737.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770787
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16407//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16407//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16407//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16407//console
This message is automatically generated.
> Clear cachedMaxVersions when setValue(VERSIONS, value) called
> -
>
> Key: HBASE-14737
> URL: https://issues.apache.org/jira/browse/HBASE-14737
> Project: HBase
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Pankaj Kumar
> Attachments: HBASE-14737.patch
>
>
> HColumnDescriptor caches the value of VERSIONS in a cachedMaxVersions member
> variable. This member variable should be reset or cleared when
> setValue(HConstants.VERSIONS, value) is called, like this:
> {code}
> static final bytes[] VERSIONS_BYTES = Bytes.toBytes(HConstants.VERSIONS);
> public HColumnDescriptor setValue(byte[] key, byte[] value) {
> if (Bytes.compare(HConstants.VERSIONS_BYTES, key) == 0) {
> cachedMaxVersions = UNINITIALIZED;
> }
> values.put(new ImmutableBytesWritable(key),
> new ImmutableBytesWritable(value));
> return this;
> }
> {code}
> Otherwise, you continue getting back cachedMaxVersions instead of the updated
> value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14664) Master failover issue: Backup master is unable to start if active master is killed and started in short time interval
[
https://issues.apache.org/jira/browse/HBASE-14664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991456#comment-14991456
]
Samir Ahmic commented on HBASE-14664:
-
Thanks for review [~stack],
regarding unit test what do you have on mind?
ActiveMasterManager#handleMasterNodeChange() is covered in
TestActiveMasterManager and is also tested in TestMasterFailover as part of
failover process.
Should we create test covering this scenario killing and restarting master to
reproduce issue or focus on aftermath of removing meta-region-znode ?
> Master failover issue: Backup master is unable to start if active master is
> killed and started in short time interval
> -
>
> Key: HBASE-14664
> URL: https://issues.apache.org/jira/browse/HBASE-14664
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14664.patch, HBASE-14664.patch
>
>
> I notice this issue while running IntegrationTestDDLMasterFailover, it can be
> simply reproduced by executing this on active master (tested on two masters +
> 3rs cluster setup)
> {code}
> $ kill -9 master_pid; hbase-daemon.sh start master
> {code}
> Logs show that new active master is trying to locate hbase:meta table on
> restarted active master
> {code}
> 2015-10-21 19:28:20,804 INFO [hnode2:16000.activeMasterManager]
> zookeeper.MetaTableLocator: Failed verification of hbase:meta,,1 at
> address=hnode1,16000,1445447051681,
> exception=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is
> not running yet
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1092)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1330)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.getRegionInfo(MasterRpcServices.java:1525)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22233)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2136)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> 2015-10-21 19:28:20,805 INFO [hnode2:16000.activeMasterManager]
> master.HMaster: Meta was in transition on hnode1,16000,1445447051681
> 2015-10-21 19:28:20,805 INFO [hnode2:16000.activeMasterManager]
> master.AssignmentManager: Processing {1588230740 state=OPEN,
> ts=1445448500598, server=hnode1,16000,1445447051681
> {code}
> and because of above master is unable to read hbase:meta table:
> {code}
> 2015-10-21 19:28:49,429 INFO [hconnection-0x6e9cebcc-shared--pool6-t1]
> client.AsyncProcess: #2, table=hbase:meta, attempt=10/351 failed=1ops, last
> exception: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException:
> org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not
> running yet
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1092)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2083)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32462)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2136)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> which cause master is unable to complete start.
> I have also notices that in this case value of /hbase/meta-region-server
> znode is always pointing on restarted active master (hnode1 in my cluster ).
> I was able to workaround this issue by repeating same scenario with following:
> {code}
> $ kill -9 master_pid; hbase zkcli rmr /hbase/meta-region-server;
> hbase-daemon.sh start master
> {code}
> So issue is probably caused by staled value in /hbase/meta-region-server
> znode. I will try to create patch based on above.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991516#comment-14991516
]
Hadoop QA commented on HBASE-14703:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770758/HBASE-14703_v2.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770758
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1729 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+tableName, actions, nonceGroup, getPool(pool), needResults,
results, callback, callable, curTimeout);
+// If statisitcs is off, the action number will not equals
resultOrExceptoins number.
+//return rpcCallerFactory.
newCaller().callWithRetries(callable, this.operationTimeout);
+
RetryingCallable callable, int t) {
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16404//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16404//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16404//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16404//console
This message is automatically generated.
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-async.patch, HBASE-14703-start.patch,
> HBASE-14703.patch, HBASE-14703_v1.patch, HBASE-14703_v2.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14498) Master stuck in infinite loop when all Zookeeper servers are unreachable.
[
https://issues.apache.org/jira/browse/HBASE-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-14498:
-
Attachment: HBASE-14498.patch
> Master stuck in infinite loop when all Zookeeper servers are unreachable.
> -
>
> Key: HBASE-14498
> URL: https://issues.apache.org/jira/browse/HBASE-14498
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.0.1
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Pankaj Kumar
>Priority: Blocker
> Attachments: HBASE-14498.patch
>
>
> We met a weird scenario in our production environment.
> In a HA cluster,
> > Active Master (HM1) is not able to connect to any Zookeeper server (due to
> > N/w breakdown on master machine network with Zookeeper servers).
> {code}
> 2015-09-26 15:24:47,508 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 33463ms for sessionid 0x104576b8dda0002, closing socket connection and
> attempting reconnect
> 2015-09-26 15:24:47,877 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:48,236 INFO [main-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:49,879 WARN
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:49,879 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-IP1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:24:50,238 WARN [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:50,238 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-Host1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:25:17,470 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 30023ms for sessionid 0x2045762cc710006, closing socket connection and
> attempting reconnect
> 2015-09-26 15:25:17,571 WARN [master/HM1-Host/HM1-IP:16000]
> zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
> quorum=ZK-Host:2181,ZK-Host1:2181,ZK-Host2:2181,
> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2015-09-26 15:25:17,872 INFO [main-SendThread(ZK-Host:2181)]
> client.FourLetterWordMain: connecting to ZK-Host 2181
> 2015-09-26 15:25:19,874 WARN [main-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host
> 2015-09-26 15:25:19,874 INFO [main-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server ZK-Host/ZK-IP:2181.
> Will not attempt to authenticate using SASL (unknown error)
> {code}
> > Since HM1 was not able to connect to any ZK, so session timeout didnt
> > happen at Zookeeper server side and HM1 didnt abort.
> > On Zookeeper session timeout standby master (HM2) registered himself as an
> > active master.
> > HM2 is keep on waiting for region server to report him as part of active
> > master intialization.
> {noformat}
> 2015-09-26 15:24:44,928 | INFO | HM2-Host:21300.activeMasterManager | Waiting
> for region servers count to settle; currently checked in 0, slept for 0 ms,
> expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval
> of 1500 ms. |
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> ---
> ---
> 2015-09-26 15:32:50,841 | INFO | HM2-Host:21300.activeMasterManager | Waiting
> for region servers count to settle; currently checked in 0, slept for 483913
> ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms,
> interval of 1500 ms. |
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> {noformat}
> > At other end, region servers are reporting to HM1 on 3 sec interval. Here
> > region server retrieve master location from zookeeper only when they
> > couldn't connect to Master (ServiceException).
> Region Server will not report HM2 as per current design until unless HM1
> abort,so HM2 will exit(InitializationMonitor) and again wait for region
> servers in loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14737) Clear cachedMaxVersions when setValue(VERSIONS, value) called
[
https://issues.apache.org/jira/browse/HBASE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-14737:
-
Attachment: HBASE-14737.patch
> Clear cachedMaxVersions when setValue(VERSIONS, value) called
> -
>
> Key: HBASE-14737
> URL: https://issues.apache.org/jira/browse/HBASE-14737
> Project: HBase
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Pankaj Kumar
> Attachments: HBASE-14737.patch
>
>
> HColumnDescriptor caches the value of VERSIONS in a cachedMaxVersions member
> variable. This member variable should be reset or cleared when
> setValue(HConstants.VERSIONS, value) is called, like this:
> {code}
> static final bytes[] VERSIONS_BYTES = Bytes.toBytes(HConstants.VERSIONS);
> public HColumnDescriptor setValue(byte[] key, byte[] value) {
> if (Bytes.compare(HConstants.VERSIONS_BYTES, key) == 0) {
> cachedMaxVersions = UNINITIALIZED;
> }
> values.put(new ImmutableBytesWritable(key),
> new ImmutableBytesWritable(value));
> return this;
> }
> {code}
> Otherwise, you continue getting back cachedMaxVersions instead of the updated
> value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14490) [RpcServer] reuse request read buffer
[
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991531#comment-14991531
]
Hiroshi Ikeda commented on HBASE-14490:
---
I just noticed that your patch is not equivalent to the original code when
using SASL. [~gzh1992n] have you confirmed that is OK?
> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
> Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: ByteBufferPool.java, HBASE-14490-v1.patch,
> HBASE-14490-v10.patch, HBASE-14490-v11.patch, HBASE-14490-v12.patch,
> HBASE-14490-v2.patch, HBASE-14490-v3.patch, HBASE-14490-v4.patch,
> HBASE-14490-v5.patch, HBASE-14490-v6.patch, HBASE-14490-v7.patch,
> HBASE-14490-v8.patch, HBASE-14490-v9.patch, test-v12-patch
>
>
> Reusing buffer to read request.It's not necessary to every request free
> buffer.The idea of optimization is to reduce the times that allocate
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}},
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a buffer pool in each {{Connection}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991542#comment-14991542
]
ramkrishna.s.vasudevan commented on HBASE-13082:
I just saw this
{code}
if (FSUtils.WINDOWS) {
// compaction cannot move files while they are open in secondary on
windows. Skip remaining.
return;
}
{code}
In one of the test case - so may this is what I was observing. Anyway will
verify once in linux box also.
> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt,
> 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf,
> HBASE-13082_1_WIP.patch, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch,
> HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, gc.png,
> gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and
> required in the documentation (not picking on Phoenix, this one is my fault
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load.
> RegionScanner operations would keep getting the locks and the
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14737) Clear cachedMaxVersions when setValue(VERSIONS, value) called
[
https://issues.apache.org/jira/browse/HBASE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-14737:
-
Status: Patch Available (was: Open)
Attached patch for review.
> Clear cachedMaxVersions when setValue(VERSIONS, value) called
> -
>
> Key: HBASE-14737
> URL: https://issues.apache.org/jira/browse/HBASE-14737
> Project: HBase
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Pankaj Kumar
> Attachments: HBASE-14737.patch
>
>
> HColumnDescriptor caches the value of VERSIONS in a cachedMaxVersions member
> variable. This member variable should be reset or cleared when
> setValue(HConstants.VERSIONS, value) is called, like this:
> {code}
> static final bytes[] VERSIONS_BYTES = Bytes.toBytes(HConstants.VERSIONS);
> public HColumnDescriptor setValue(byte[] key, byte[] value) {
> if (Bytes.compare(HConstants.VERSIONS_BYTES, key) == 0) {
> cachedMaxVersions = UNINITIALIZED;
> }
> values.put(new ImmutableBytesWritable(key),
> new ImmutableBytesWritable(value));
> return this;
> }
> {code}
> Otherwise, you continue getting back cachedMaxVersions instead of the updated
> value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14189) CF Level BC setting should override global one.
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991549#comment-14991549
]
Hadoop QA commented on HBASE-14189:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770772/HBASE-14189_v4.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770772
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1727 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestInterfaceAudienceAnnotations
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16406//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16406//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16406//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16406//console
This message is automatically generated.
> CF Level BC setting should override global one.
> ---
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch, HBASE-14189_v3.patch, HBASE-14189_v4.patch
>
>
> The original description is ambiguous. I think i will rewrite it.
> Let's see {{BlockCache}} constructor firstly
> {code}
> public CacheConfig(Configuration conf, HColumnDescriptor family) {
> this(CacheConfig.instantiateBlockCache(conf),
> family.isBlockCacheEnabled(),
> family.isInMemory(),
> // For the following flags we enable them regardless of per-schema
> settings
> // if they are enabled in the global configuration.
> conf.getBoolean(CACHE_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_DATA_ON_WRITE) || family.isCacheDataOnWrite(),
> conf.getBoolean(CACHE_INDEX_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_INDEXES_ON_WRITE) || family.isCacheIndexesOnWrite(),
> conf.getBoolean(CACHE_BLOOM_BLOCKS_ON_WRITE_KEY,
> DEFAULT_CACHE_BLOOMS_ON_WRITE) || family.isCacheBloomsOnWrite(),
> conf.getBoolean(EVICT_BLOCKS_ON_CLOSE_KEY,
> DEFAULT_EVICT_ON_CLOSE) || family.isEvictBlocksOnClose(),
> conf.getBoolean(CACHE_DATA_BLOCKS_COMPRESSED_KEY,
> DEFAULT_CACHE_DATA_COMPRESSED),
> conf.getBoolean(PREFETCH_BLOCKS_ON_OPEN_KEY,
> DEFAULT_PREFETCH_ON_OPEN) || family.isPrefetchBlocksOnOpen(),
> conf.getBoolean(HColumnDescriptor.CACHE_DATA_IN_L1,
> HColumnDescriptor.DEFAULT_CACHE_DATA_IN_L1) ||
> family.isCacheDataInL1(),
>
> conf.getBoolean(DROP_BEHIND_CACHE_COMPACTION_KEY,DROP_BEHIND_CACHE_COMPACTION_DEFAULT)
> );
> }
> {code}
> If we dig in it, we will see {{CacheConfig.cacheDataOnRead}} is used to
> accept {{family.isBlockCacheEnabled()}}.
> I think it is confused as comments about {{cacheDataOnRead}}
> {code}
> /**
>* Whether blocks should be cached on read (default is on if there is a
>* cache but this can be turned off on a per-family or per-request basis).
>* If off we will STILL cache meta blocks; i.e. INDEX and BLOOM types.
>* This cannot be disabled.
>*/
> private boolean cacheDataOnRead;
> {code}
> So i think we should use
[jira] [Updated] (HBASE-14498) Master stuck in infinite loop when all Zookeeper servers are unreachable.
[
https://issues.apache.org/jira/browse/HBASE-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-14498:
-
Affects Version/s: (was: 1.0.1)
Status: Patch Available (was: Open)
Thanks [~bkjain].
Attached patch for review.
> Master stuck in infinite loop when all Zookeeper servers are unreachable.
> -
>
> Key: HBASE-14498
> URL: https://issues.apache.org/jira/browse/HBASE-14498
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Pankaj Kumar
>Priority: Blocker
> Attachments: HBASE-14498.patch
>
>
> We met a weird scenario in our production environment.
> In a HA cluster,
> > Active Master (HM1) is not able to connect to any Zookeeper server (due to
> > N/w breakdown on master machine network with Zookeeper servers).
> {code}
> 2015-09-26 15:24:47,508 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 33463ms for sessionid 0x104576b8dda0002, closing socket connection and
> attempting reconnect
> 2015-09-26 15:24:47,877 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:48,236 INFO [main-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:49,879 WARN
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:49,879 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-IP1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:24:50,238 WARN [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:50,238 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-Host1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:25:17,470 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 30023ms for sessionid 0x2045762cc710006, closing socket connection and
> attempting reconnect
> 2015-09-26 15:25:17,571 WARN [master/HM1-Host/HM1-IP:16000]
> zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
> quorum=ZK-Host:2181,ZK-Host1:2181,ZK-Host2:2181,
> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2015-09-26 15:25:17,872 INFO [main-SendThread(ZK-Host:2181)]
> client.FourLetterWordMain: connecting to ZK-Host 2181
> 2015-09-26 15:25:19,874 WARN [main-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host
> 2015-09-26 15:25:19,874 INFO [main-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server ZK-Host/ZK-IP:2181.
> Will not attempt to authenticate using SASL (unknown error)
> {code}
> > Since HM1 was not able to connect to any ZK, so session timeout didnt
> > happen at Zookeeper server side and HM1 didnt abort.
> > On Zookeeper session timeout standby master (HM2) registered himself as an
> > active master.
> > HM2 is keep on waiting for region server to report him as part of active
> > master intialization.
> {noformat}
> 2015-09-26 15:24:44,928 | INFO | HM2-Host:21300.activeMasterManager | Waiting
> for region servers count to settle; currently checked in 0, slept for 0 ms,
> expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval
> of 1500 ms. |
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> ---
> ---
> 2015-09-26 15:32:50,841 | INFO | HM2-Host:21300.activeMasterManager | Waiting
> for region servers count to settle; currently checked in 0, slept for 483913
> ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms,
> interval of 1500 ms. |
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> {noformat}
> > At other end, region servers are reporting to HM1 on 3 sec interval. Here
> > region server retrieve master location from zookeeper only when they
> > couldn't connect to Master (ServiceException).
> Region Server will not report HM2 as per current design until unless HM1
> abort,so HM2 will exit(InitializationMonitor) and again wait for region
> servers in loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14693) Add client-side metrics for received pushback signals
[
https://issues.apache.org/jira/browse/HBASE-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991548#comment-14991548
]
Hadoop QA commented on HBASE-14693:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770761/HBASE-14693_v4.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770761
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16405//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16405//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16405//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16405//console
This message is automatically generated.
> Add client-side metrics for received pushback signals
> -
>
> Key: HBASE-14693
> URL: https://issues.apache.org/jira/browse/HBASE-14693
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-14693.patch, HBASE-14693_v1.patch,
> HBASE-14693_v2.patch, HBASE-14693_v3.patch, HBASE-14693_v4.patch
>
>
> HBASE-12911 added client side metrics. HBASE-5162 added a mechanism for
> sending advisory backpressure signals to clients when the server is heavily
> loaded, and HBASE-12702 and subtasks backported this to all active branches.
> Add client-side metrics for received pushback signal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991977#comment-14991977 ] Apekshit Sharma commented on HBASE-14769: - HBASE-9182 added getTableNames(). But the reasons I think it's okay to remove all these functions are, these are all unused String/byte[] counterparts of TableName and do not need to be marked deprecated because the class was marked Audience private in 1.0.0. I am not removing any functions which are still required to support Admin class' deprecated functions. Those can be removed in 3.0.0. Thanks [~stack] for reviewing. > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma > Attachments: HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14768) bin/graceful_stop.sh logs nothing as a balancer state to be stored
[
https://issues.apache.org/jira/browse/HBASE-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991996#comment-14991996
]
Hudson commented on HBASE-14768:
FAILURE: Integrated in HBase-0.98-matrix #257 (See
[https://builds.apache.org/job/HBase-0.98-matrix/257/])
HBASE-14768 bin/graceful_stop.sh logs nothing as a balancer state to be (stack:
rev 317bfa8a24ed2afce61cb857febb7a37050a2b64)
* bin/graceful_stop.sh
> bin/graceful_stop.sh logs nothing as a balancer state to be stored
> --
>
> Key: HBASE-14768
> URL: https://issues.apache.org/jira/browse/HBASE-14768
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14768.patch
>
>
> In bin/graceful_stop.sh
> {code}
> log() {
> echo `date +%Y-%m-%dT%H:%M:%S` $1
> }
> ...skip...
> if [ $HBASE_BALANCER_STATE != "false" ]; then
> log "Restoring balancer state to " $HBASE_BALANCER_STATE
> {code}
> The position of the above last double quotation mark is wrong, and the
> balancer state will be not shown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14758) Add UT case for unchecked error/exception thrown in AsyncProcess#sendMultiAction
[ https://issues.apache.org/jira/browse/HBASE-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991995#comment-14991995 ] Hudson commented on HBASE-14758: FAILURE: Integrated in HBase-0.98-matrix #257 (See [https://builds.apache.org/job/HBase-0.98-matrix/257/]) HBASE-14758 Add UT case for unchecked error/exception thrown in (tedyu: rev 415cf6c4e7f62f09d1acb6233fe06271dbda6e15) * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java > Add UT case for unchecked error/exception thrown in > AsyncProcess#sendMultiAction > > > Key: HBASE-14758 > URL: https://issues.apache.org/jira/browse/HBASE-14758 > Project: HBase > Issue Type: Test >Affects Versions: 1.1.2, 0.98.15 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.17, 1.0.4 > > Attachments: HBASE-14758.branch-0.98.patch, HBASE-14758.patch > > > This is an addendum for HBASE-14359, open a new JIRA to trigger HadoopQA run -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14771) RpcServer.getRemoteAddress always returns null.
[
https://issues.apache.org/jira/browse/HBASE-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992032#comment-14992032
]
Hadoop QA commented on HBASE-14771:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770801/HBASE-14771.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770801
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16409//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16409//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16409//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16409//console
This message is automatically generated.
> RpcServer.getRemoteAddress always returns null.
> ---
>
> Key: HBASE-14771
> URL: https://issues.apache.org/jira/browse/HBASE-14771
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 1.2.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14771.patch
>
>
> RpcServer.getRemoteAddress always returns null, because Call object is
> getting initialized with null.This seems to be happening because of using
> RpcServer.getRemoteIp() in Call object constructor before RpcServer thread
> local 'CurCall' being set in CallRunner.run method:
> {noformat}
> // --- RpcServer.java ---
> protected void processRequest(byte[] buf) throws IOException,
> InterruptedException {
> .
> // Call object getting initialized here with address
> // obtained from RpcServer.getRemoteIp()
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, RpcServer.getRemoteIp());
> scheduler.dispatch(new CallRunner(RpcServer.this, call));
> }
> // getRemoteIp method gets address from threadlocal 'CurCall' which
> // gets set in CallRunner.run and calling it before this as in above case,
> will return null
> // --- CallRunner.java ---
> public void run() {
> .
> Pair resultPair = null;
> RpcServer.CurCall.set(call);
> ..
> }
> // Using 'this.addr' in place of getRemoteIp method in RpcServer.java seems
> to be fixing this issue
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, this.addr);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14755) Fix some broken links and HTML problems
[ https://issues.apache.org/jira/browse/HBASE-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992111#comment-14992111 ] stack commented on HBASE-14755: --- These 'zombies' look like they are just a concurrent build that is happening on this machine. At least we ain't killing them now as we used to. Let me retry just in case (though this patch does not change code). > Fix some broken links and HTML problems > --- > > Key: HBASE-14755 > URL: https://issues.apache.org/jira/browse/HBASE-14755 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14755-v1.patch, HBASE-14755.patch > > > Problems seen in > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Website%20Link%20Ckecker/3/artifact/link_report/index.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992100#comment-14992100 ] stack commented on HBASE-12790: --- Phoenix shouldn't have to add its own scheduler. Phoenix changing basic functionality of hbase is not sustainable; for phoenix nor hbase; neither project has resources to ensure custom scheduler and executor setups perform and are bug free at scale. Phoenix shouldn't have to do hbase customization given that what Phoenix wants of the scheduler is super basic (staccato scan rather than sustained). I've asked if this groupid specialization is a deal breaker or will pulling on exisiting attributes whether connection identifier or scan attributes will do. I may have been answered above or over in rb but have not seen it if I have. > Support fairness across parallelized scans > -- > > Key: HBASE-12790 > URL: https://issues.apache.org/jira/browse/HBASE-12790 > Project: HBase > Issue Type: New Feature >Reporter: James Taylor >Assignee: ramkrishna.s.vasudevan > Labels: Phoenix > Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, > HBASE-12790_1.patch, HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, > HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip > > > Some HBase clients parallelize the execution of a scan to reduce latency in > getting back results. This can lead to starvation with a loaded cluster and > interleaved scans, since the RPC queue will be ordered and processed on a > FIFO basis. For example, if there are two clients, A & B that submit largish > scans at the same time. Say each scan is broken down into 100 scans by the > client (broken down into equal depth chunks along the row key), and the 100 > scans of client A are queued first, followed immediately by the 100 scans of > client B. In this case, client B will be starved out of getting any results > back until the scans for client A complete. > One solution to this is to use the attached AbstractRoundRobinQueue instead > of the standard FIFO queue. The queue to be used could be (maybe it already > is) configurable based on a new config parameter. Using this queue would > require the client to have the same identifier for all of the 100 parallel > scans that represent a single logical scan from the clients point of view. > With this information, the round robin queue would pick off a task from the > queue in a round robin fashion (instead of a strictly FIFO manner) to prevent > starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991881#comment-14991881 ] Solomon Duskis commented on HBASE-14769: FWIW, it's probably worth while checking the jruby shell code's uses of HBaseAdmin's deprecated methods. The shell uses the String version of overloaded methods rather than TableName and byte[] counterparts that exist in the Admin interface. I don't think we have tests for all of the shell's functionality, so it's worth doing a manual review. > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma > Attachments: HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14768) bin/graceful_stop.sh logs nothing as a balancer state to be stored
[
https://issues.apache.org/jira/browse/HBASE-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992039#comment-14992039
]
Hudson commented on HBASE-14768:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1130 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1130/])
HBASE-14768 bin/graceful_stop.sh logs nothing as a balancer state to be (stack:
rev 317bfa8a24ed2afce61cb857febb7a37050a2b64)
* bin/graceful_stop.sh
> bin/graceful_stop.sh logs nothing as a balancer state to be stored
> --
>
> Key: HBASE-14768
> URL: https://issues.apache.org/jira/browse/HBASE-14768
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14768.patch
>
>
> In bin/graceful_stop.sh
> {code}
> log() {
> echo `date +%Y-%m-%dT%H:%M:%S` $1
> }
> ...skip...
> if [ $HBASE_BALANCER_STATE != "false" ]; then
> log "Restoring balancer state to " $HBASE_BALANCER_STATE
> {code}
> The position of the above last double quotation mark is wrong, and the
> balancer state will be not shown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14279:
--
Attachment: HBASE-14279_v3.patch
update patch as [~ikeda] code.
Sorry again for being late.
Maybe we need a testcase for it.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14768) bin/graceful_stop.sh logs nothing as a balancer state to be stored
[
https://issues.apache.org/jira/browse/HBASE-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991983#comment-14991983
]
Hudson commented on HBASE-14768:
FAILURE: Integrated in HBase-1.0 #1108 (See
[https://builds.apache.org/job/HBase-1.0/1108/])
HBASE-14768 bin/graceful_stop.sh logs nothing as a balancer state to be (stack:
rev d9d3fe71fd705bd3d995773242556a893d633dee)
* bin/graceful_stop.sh
> bin/graceful_stop.sh logs nothing as a balancer state to be stored
> --
>
> Key: HBASE-14768
> URL: https://issues.apache.org/jira/browse/HBASE-14768
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14768.patch
>
>
> In bin/graceful_stop.sh
> {code}
> log() {
> echo `date +%Y-%m-%dT%H:%M:%S` $1
> }
> ...skip...
> if [ $HBASE_BALANCER_STATE != "false" ]; then
> log "Restoring balancer state to " $HBASE_BALANCER_STATE
> {code}
> The position of the above last double quotation mark is wrong, and the
> balancer state will be not shown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991986#comment-14991986 ] Apekshit Sharma commented on HBASE-14769: - Will look into ruby code and fix hudson errors. Thanks [~sduskis]. > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma > Attachments: HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14758) Add UT case for unchecked error/exception thrown in AsyncProcess#sendMultiAction
[ https://issues.apache.org/jira/browse/HBASE-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992038#comment-14992038 ] Hudson commented on HBASE-14758: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1130 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1130/]) HBASE-14758 Add UT case for unchecked error/exception thrown in (tedyu: rev 415cf6c4e7f62f09d1acb6233fe06271dbda6e15) * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java > Add UT case for unchecked error/exception thrown in > AsyncProcess#sendMultiAction > > > Key: HBASE-14758 > URL: https://issues.apache.org/jira/browse/HBASE-14758 > Project: HBase > Issue Type: Test >Affects Versions: 1.1.2, 0.98.15 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.17, 1.0.4 > > Attachments: HBASE-14758.branch-0.98.patch, HBASE-14758.patch > > > This is an addendum for HBASE-14359, open a new JIRA to trigger HadoopQA run -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992024#comment-14992024 ] stack commented on HBASE-14769: --- Sounds good to me then [~appy] Yeah on what [~sduskis] says... be careful though... TestShell is temporarily disabled IIRC (as part of the build stabilization project). > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma > Attachments: HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992097#comment-14992097
]
Hadoop QA commented on HBASE-14279:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770824/HBASE-14279_v3.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770824
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.bucket.TestBucketCache
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16410//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16410//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16410//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16410//console
This message is automatically generated.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14772) Improve zombie detector; be more discerning
stack created HBASE-14772:
-
Summary: Improve zombie detector; be more discerning
Key: HBASE-14772
URL: https://issues.apache.org/jira/browse/HBASE-14772
Project: HBase
Issue Type: Sub-task
Reporter: stack
Currently, any surefire process with the hbase flag is a potential zombie. Our
zombie check currently takes a reading and if it finds candidate zombies, it
waits 30 seconds and then does another reading. If a concurrent build going on,
in both cases the zombie detector will come up positive though the adjacent
test run may be making progress; i.e. the cast of surefire processes may have
changed between readings but our detector just sees presence of hbase surefire
processes.
Here is example:
{code}
Suspicious java process found - waiting 30s to see if there are just slow to
stop
There appear to be 5 zombie tests, they should have been killed by surefire but
survived
12823 surefirebooter852180186418035480.jar -enableassertions -Dhbase.test
-Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
7653 surefirebooter8579074445899448699.jar -enableassertions -Dhbase.test
-Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
12614 surefirebooter136529596936417090.jar -enableassertions -Dhbase.test
-Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
7836 surefirebooter3217047564606450448.jar -enableassertions -Dhbase.test
-Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
13566 surefirebooter2084039411151963494.jar -enableassertions -Dhbase.test
-Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
BEGIN zombies jstack extract
END zombies jstack extract
{code}
5 is the number of forked processes we allow when doing medium and large
tests so an adjacent build will always show as '5 zombies'.
Need to add discerning if list of processes changes between readings.
Can I also add a tag per build run that all forked processes pick up so I can
look at the current builds progeny only?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14763) Remove usages of deprecated HConnection
[ https://issues.apache.org/jira/browse/HBASE-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14763: --- Attachment: hbase-14763.v3.patch v3. cleaned up some checkstyle. Looks like the javac warns are actually in the spark code, which I didn't touch. > Remove usages of deprecated HConnection > > > Key: HBASE-14763 > URL: https://issues.apache.org/jira/browse/HBASE-14763 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14763.patch, hbase-14763.v2.patch, > hbase-14763.v3.patch > > > HConnection was deprecated in 1.0.0. There are two interfaces that are > supposed to be used instead -- Connection for client programs and > ClusterConnection for internal hbaes and special tools (LoadIncremental, > HBCK, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992149#comment-14992149 ] huaxiang sun commented on HBASE-14766: -- s/patch/path in the previous comment. > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache
[ https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992171#comment-14992171 ] Elliott Clark commented on HBASE-14708: --- bq.It is quite confusing that some of methods provide copies, and some of methods expose the internal data that the client should not modify. That's only for tree map. The array based version has non-modifiable versions of everything. The tree map is less risky as it relies on a know and tested version of map. However it's pretty specialized and should only really ever be used in meta cache. The array based map is faster and more complete. However it is also more risky. My thought was that in branch-1.2 we would use the tree map and in branch-1 + we would use the array map. > Use copy on write Map for region location cache > --- > > Key: HBASE-14708 > URL: https://issues.apache.org/jira/browse/HBASE-14708 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, > HBASE-14708-v12.patch, HBASE-14708-v2.patch, HBASE-14708-v3.patch, > HBASE-14708-v4.patch, HBASE-14708-v5.patch, HBASE-14708-v6.patch, > HBASE-14708-v7.patch, HBASE-14708-v8.patch, HBASE-14708-v9.patch, > HBASE-14708.patch, anotherbench.zip, location_cache_times.pdf, result.csv > > > Internally a co-worker profiled their application that was talking to HBase. > > 60% of the time was spent in locating a region. This was while the cluster > was stable and no regions were moving. > To figure out if there was a faster way to cache region location I wrote up a > benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache > This tries to simulate a heavy load on the location cache. > * 24 different threads. > * 2 Deleting location data > * 2 Adding location data > * Using floor to get the result. > To repeat my work just run ./run.sh and it should produce a result.csv > Results: > ConcurrentSkiplistMap is a good middle ground. It's got equal speed for > reading and writing. > However most operations will not need to remove or add a region location. > There will be potentially several orders of magnitude more reads for cached > locations than there will be on clearing the cache. > So I propose a copy on write tree map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14772) Improve zombie detector; be more discerning
[
https://issues.apache.org/jira/browse/HBASE-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992223#comment-14992223
]
Sean Busbey edited comment on HBASE-14772 at 11/5/15 6:46 PM:
--
yeah, a UUID or some such specific to the current build would be perfect. on
Jenkins at least, there's $\{BUILD_ID} that we can use.
was (Author: busbey):
yeah, a UUID or some such specific to the current build would be perfect. on
Jenkins at least, there's ${BUILD_ID} that we can use.
> Improve zombie detector; be more discerning
> ---
>
> Key: HBASE-14772
> URL: https://issues.apache.org/jira/browse/HBASE-14772
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>
> Currently, any surefire process with the hbase flag is a potential zombie.
> Our zombie check currently takes a reading and if it finds candidate zombies,
> it waits 30 seconds and then does another reading. If a concurrent build
> going on, in both cases the zombie detector will come up positive though the
> adjacent test run may be making progress; i.e. the cast of surefire processes
> may have changed between readings but our detector just sees presence of
> hbase surefire processes.
> Here is example:
> {code}
> Suspicious java process found - waiting 30s to see if there are just slow to
> stop
> There appear to be 5 zombie tests, they should have been killed by surefire
> but survived
> 12823 surefirebooter852180186418035480.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7653 surefirebooter8579074445899448699.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 12614 surefirebooter136529596936417090.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7836 surefirebooter3217047564606450448.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 13566 surefirebooter2084039411151963494.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> BEGIN zombies jstack extract
> END zombies jstack extract
> {code}
> 5 is the number of forked processes we allow when doing medium and large
> tests so an adjacent build will always show as '5 zombies'.
> Need to add discerning if list of processes changes between readings.
> Can I also add a tag per build run that all forked processes pick up so I can
> look at the current builds progeny only?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14765: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.2.0 Status: Resolved (was: Patch Available) > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992246#comment-14992246 ] Elliott Clark commented on HBASE-14765: --- bq.Thats some old stuff you are removing there Mr Clark Yeah I barely remember ever using it. > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12072) Standardize retry handling for master operations
[
https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992306#comment-14992306
]
Enis Soztutar commented on HBASE-12072:
---
[~appy] can you please not add the fixVersions=1.0.0 to jiras with fix versions
= 0.99.x.
The fixVersions tracks the first release that the issue appeared in. Since we
have done 0.99.x releases before 1.0.0, some of the jiras that you recently
modified appeared on earlier 0.99.x releases.
Let me know whether this helps:
http://markmail.org/message/u43qluenc7soxloe
> Standardize retry handling for master operations
>
>
> Key: HBASE-12072
> URL: https://issues.apache.org/jira/browse/HBASE-12072
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 1.0.0, 2.0.0, 0.99.2
>
> Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch,
> hbase-12072_v2.patch, hbase-12072_v2.patch, hbase-12072_v3.patch
>
>
> For master requests, there are two retry mechanisms in effect. The first one
> is from HBaseAdmin.executeCallable()
> {code}
> private V executeCallable(MasterCallable callable) throws
> IOException {
> RpcRetryingCaller caller = rpcCallerFactory.newCaller();
> try {
> return caller.callWithRetries(callable);
> } finally {
> callable.close();
> }
> }
> {code}
> And inside, the other one is from StubMaker.makeStub():
> {code}
> /**
>* Create a stub against the master. Retry if necessary.
>* @return A stub to do intf against the master
>* @throws MasterNotRunningException
>*/
> @edu.umd.cs.findbugs.annotations.SuppressWarnings
> (value="SWL_SLEEP_WITH_LOCK_HELD")
> Object makeStub() throws MasterNotRunningException {
> {code}
> The tests will just hang for 10 min * 35 ~= 6hours.
> {code}
> 2014-09-23 16:19:05,151 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 1 of 35
> failed; retrying after sleep of 100, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,253 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 2 of 35
> failed; retrying after sleep of 200, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,456 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 3 of 35
> failed; retrying after sleep of 300, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,759 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 4 of 35
> failed; retrying after sleep of 500, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:06,262 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 5 of 35
> failed; retrying after sleep of 1008, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:07,273 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 6 of 35
> failed; retrying after sleep of 2011, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:09,286 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 7 of 35
> failed; retrying after sleep of 4012, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:13,303 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 8 of 35
> failed; retrying after sleep of 10033, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:23,343 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 9 of 35
> failed; retrying after sleep of 10089, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:33,439 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 10 of
> 35 failed; retrying after sleep of 10027, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:43,473 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 11 of
> 35 failed; retrying after sleep of 10004, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:53,485 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 12 of
> 35
[jira] [Commented] (HBASE-14427) Fix 'should' assertions in TestFastFail
[ https://issues.apache.org/jira/browse/HBASE-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992237#comment-14992237 ] Abhishek Singh Chouhan commented on HBASE-14427: [~stack] in case you missed this. This test is not same as TestFailFastWithoutTestUtil which was removed. This one is still there on master but is currently ignored. > Fix 'should' assertions in TestFastFail > --- > > Key: HBASE-14427 > URL: https://issues.apache.org/jira/browse/HBASE-14427 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: Abhishek Singh Chouhan >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-14427.addendum.patch, HBASE-14427.patch > > > Over in HBASE-14421, TestFastFail has been failing assertions that talk of > events that 'should' be happening. Fix. For now HBASE-14421 has disabled the > 'should' assertions. They seem fine on apache jenkins build but fail fairly > reliably for me on alternate HW. > To address, get familiar with the test. Change the commented out asserts to > be yes/no instead of a 'likely' (On a cursory scan, it is possible that a > test run may not involve preemption and it is these runs that are throwing > asserts). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12072) Standardize retry handling for master operations
[
https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992342#comment-14992342
]
Apekshit Sharma commented on HBASE-12072:
-
You're right, i should have seen the pattern and corrected myself on seeing the
same thing in 3 jiras.
To confirm, we add unreleased version corresponding to every brach the change
was pushed, right?
If yes, seems like 0.99.2 was made from 1.0 branch.
Also let me know if I should revert the changes i made.
> Standardize retry handling for master operations
>
>
> Key: HBASE-12072
> URL: https://issues.apache.org/jira/browse/HBASE-12072
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 1.0.0, 2.0.0, 0.99.2
>
> Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch,
> hbase-12072_v2.patch, hbase-12072_v2.patch, hbase-12072_v3.patch
>
>
> For master requests, there are two retry mechanisms in effect. The first one
> is from HBaseAdmin.executeCallable()
> {code}
> private V executeCallable(MasterCallable callable) throws
> IOException {
> RpcRetryingCaller caller = rpcCallerFactory.newCaller();
> try {
> return caller.callWithRetries(callable);
> } finally {
> callable.close();
> }
> }
> {code}
> And inside, the other one is from StubMaker.makeStub():
> {code}
> /**
>* Create a stub against the master. Retry if necessary.
>* @return A stub to do intf against the master
>* @throws MasterNotRunningException
>*/
> @edu.umd.cs.findbugs.annotations.SuppressWarnings
> (value="SWL_SLEEP_WITH_LOCK_HELD")
> Object makeStub() throws MasterNotRunningException {
> {code}
> The tests will just hang for 10 min * 35 ~= 6hours.
> {code}
> 2014-09-23 16:19:05,151 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 1 of 35
> failed; retrying after sleep of 100, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,253 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 2 of 35
> failed; retrying after sleep of 200, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,456 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 3 of 35
> failed; retrying after sleep of 300, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,759 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 4 of 35
> failed; retrying after sleep of 500, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:06,262 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 5 of 35
> failed; retrying after sleep of 1008, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:07,273 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 6 of 35
> failed; retrying after sleep of 2011, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:09,286 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 7 of 35
> failed; retrying after sleep of 4012, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:13,303 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 8 of 35
> failed; retrying after sleep of 10033, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:23,343 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 9 of 35
> failed; retrying after sleep of 10089, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:33,439 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 10 of
> 35 failed; retrying after sleep of 10027, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:43,473 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 11 of
> 35 failed; retrying after sleep of 10004, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:53,485 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 12 of
> 35 failed; retrying after sleep of 20160,
[jira] [Updated] (HBASE-14708) Use copy on write Map for region location cache
[ https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14708: -- Attachment: HBASE-14708-v13.patch > Use copy on write Map for region location cache > --- > > Key: HBASE-14708 > URL: https://issues.apache.org/jira/browse/HBASE-14708 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, > HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v2.patch, > HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, > HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, > HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, > location_cache_times.pdf, result.csv > > > Internally a co-worker profiled their application that was talking to HBase. > > 60% of the time was spent in locating a region. This was while the cluster > was stable and no regions were moving. > To figure out if there was a faster way to cache region location I wrote up a > benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache > This tries to simulate a heavy load on the location cache. > * 24 different threads. > * 2 Deleting location data > * 2 Adding location data > * Using floor to get the result. > To repeat my work just run ./run.sh and it should produce a result.csv > Results: > ConcurrentSkiplistMap is a good middle ground. It's got equal speed for > reading and writing. > However most operations will not need to remove or add a region location. > There will be potentially several orders of magnitude more reads for cached > locations than there will be on clearing the cache. > So I propose a copy on write tree map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-14766: - Attachment: HBASE-14766-v002.patch The change in ScopeWALEntryFilter.java has been done in master branch. InTableCfWALEntryFilter's filter(), no clone is needed. > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992125#comment-14992125 ] huaxiang sun commented on HBASE-14766: -- [~ted_yu] Thanks! Updated a new patch. The change in ScopeWALEntryFilter.filter() has been done. The new patch avoids a clone of cell in the replication patch. > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14755) Fix some broken links and HTML problems
[ https://issues.apache.org/jira/browse/HBASE-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14755: -- Attachment: HBASE-14755-v1.patch Retry > Fix some broken links and HTML problems > --- > > Key: HBASE-14755 > URL: https://issues.apache.org/jira/browse/HBASE-14755 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14755-v1.patch, HBASE-14755-v1.patch, > HBASE-14755.patch > > > Problems seen in > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Website%20Link%20Ckecker/3/artifact/link_report/index.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14771) RpcServer.getRemoteAddress always returns null.
[
https://issues.apache.org/jira/browse/HBASE-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992161#comment-14992161
]
stack commented on HBASE-14771:
---
Good one. Hmmm. This used to work. Stuff got moved around? Do you know [~appy]
? Can you see? Is this the only place a Call is made? There are a few others
if only on error. We should set address there too? Nice one.
> RpcServer.getRemoteAddress always returns null.
> ---
>
> Key: HBASE-14771
> URL: https://issues.apache.org/jira/browse/HBASE-14771
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 1.2.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14771.patch
>
>
> RpcServer.getRemoteAddress always returns null, because Call object is
> getting initialized with null.This seems to be happening because of using
> RpcServer.getRemoteIp() in Call object constructor before RpcServer thread
> local 'CurCall' being set in CallRunner.run method:
> {noformat}
> // --- RpcServer.java ---
> protected void processRequest(byte[] buf) throws IOException,
> InterruptedException {
> .
> // Call object getting initialized here with address
> // obtained from RpcServer.getRemoteIp()
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, RpcServer.getRemoteIp());
> scheduler.dispatch(new CallRunner(RpcServer.this, call));
> }
> // getRemoteIp method gets address from threadlocal 'CurCall' which
> // gets set in CallRunner.run and calling it before this as in above case,
> will return null
> // --- CallRunner.java ---
> public void run() {
> .
> Pair resultPair = null;
> RpcServer.CurCall.set(call);
> ..
> }
> // Using 'this.addr' in place of getRemoteIp method in RpcServer.java seems
> to be fixing this issue
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, this.addr);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14772) Improve zombie detector; be more discerning
[
https://issues.apache.org/jira/browse/HBASE-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992223#comment-14992223
]
Sean Busbey commented on HBASE-14772:
-
yeah, a UUID or some such specific to the current build would be perfect. on
Jenkins at least, there's ${BUILD_ID} that we can use.
> Improve zombie detector; be more discerning
> ---
>
> Key: HBASE-14772
> URL: https://issues.apache.org/jira/browse/HBASE-14772
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>
> Currently, any surefire process with the hbase flag is a potential zombie.
> Our zombie check currently takes a reading and if it finds candidate zombies,
> it waits 30 seconds and then does another reading. If a concurrent build
> going on, in both cases the zombie detector will come up positive though the
> adjacent test run may be making progress; i.e. the cast of surefire processes
> may have changed between readings but our detector just sees presence of
> hbase surefire processes.
> Here is example:
> {code}
> Suspicious java process found - waiting 30s to see if there are just slow to
> stop
> There appear to be 5 zombie tests, they should have been killed by surefire
> but survived
> 12823 surefirebooter852180186418035480.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7653 surefirebooter8579074445899448699.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 12614 surefirebooter136529596936417090.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7836 surefirebooter3217047564606450448.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 13566 surefirebooter2084039411151963494.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> BEGIN zombies jstack extract
> END zombies jstack extract
> {code}
> 5 is the number of forked processes we allow when doing medium and large
> tests so an adjacent build will always show as '5 zombies'.
> Need to add discerning if list of processes changes between readings.
> Can I also add a tag per build run that all forked processes pick up so I can
> look at the current builds progeny only?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992285#comment-14992285 ] Enis Soztutar commented on HBASE-13082: --- bq. In one of the test case - so may this is what I was observing. Anyway will verify once in linux box also. Yeah, when FS is local filesystem on windows, the rename while having readers does not work. If fs is hdfs on top of windows, it works. This particular unit test is using the local file system, hence the workaround needed, but in production, we are not using ntfs as a local fs, we are using hdfs on top of ntfs. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: ramkrishna.s.vasudevan > Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, > 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, > HBASE-13082_1_WIP.patch, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, > HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, gc.png, > gc.png, gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992301#comment-14992301
]
Enis Soztutar commented on HBASE-14223:
---
The logic is something like this in server failover:
- check whether the failed region server was carrying meta, if so split the
meta wals and assign meta. Split the other wals after that.
- else, split only the files that matches the regular log file pattern and
assign the regions. If there is a .meta WAL, the file is left there in the
splitting directory.
So, if a region server dies while hosting meta, the regular recovery process
recovers the meta wal files and cleans them. Only in cases where a meta region
was opened in some region server, then the meta was moved to someplace else
when the region server failed, we are left with a meta WAL file in the WALs
directory.
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
>
[jira] [Updated] (HBASE-13982) Add info for visibility labels/cell TTLs to ImportTsv
[ https://issues.apache.org/jira/browse/HBASE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] NIDHI GAMBHIR updated HBASE-13982: -- Attachment: HBASE_13982_1.patch Fixed the line length issue. > Add info for visibility labels/cell TTLs to ImportTsv > - > > Key: HBASE-13982 > URL: https://issues.apache.org/jira/browse/HBASE-13982 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 1.1.0.1 >Reporter: Lars George >Assignee: NIDHI GAMBHIR > Labels: beginner > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13982.patch, HBASE_13982_1.patch > > > HBASE-9832 added support for two more optional, special TSV columns, but no > usage info was added. Add. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992670#comment-14992670 ] Hudson commented on HBASE-14765: FAILURE: Integrated in HBase-1.2 #350 (See [https://builds.apache.org/job/HBase-1.2/350/]) HBASE-14765 Remove snappy profile (eclark: rev b378b3459da34a64633fb41aeac157c006756267) * hbase-server/pom.xml * src/main/asciidoc/_chapters/developer.adoc * pom.xml * hbase-shell/pom.xml * hbase-common/pom.xml > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992712#comment-14992712 ] Hudson commented on HBASE-14765: SUCCESS: Integrated in HBase-1.3 #350 (See [https://builds.apache.org/job/HBase-1.3/350/]) HBASE-14765 Remove snappy profile (eclark: rev 2b84c40f014bc57808f726412b976bb67e337f0b) * hbase-server/pom.xml * src/main/asciidoc/_chapters/developer.adoc * pom.xml * hbase-shell/pom.xml * hbase-common/pom.xml > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14763) Remove usages of deprecated HConnection
[ https://issues.apache.org/jira/browse/HBASE-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992721#comment-14992721 ] stack commented on HBASE-14763: --- Failed here: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (secondPartTestsExecution) on project hbase-server: ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.maven.surefire.report.ReporterException: When writing xml report stdout/stderr: /tmp/stderr2779164428906562928deferred (No such file or directory) -> [Help 1] I've not seen this one before. Retrying. > Remove usages of deprecated HConnection > > > Key: HBASE-14763 > URL: https://issues.apache.org/jira/browse/HBASE-14763 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14763.patch, hbase-14763.v2.patch, > hbase-14763.v3.patch > > > HConnection was deprecated in 1.0.0. There are two interfaces that are > supposed to be used instead -- Connection for client programs and > ClusterConnection for internal hbaes and special tools (LoadIncremental, > HBCK, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992783#comment-14992783 ] Hudson commented on HBASE-14765: FAILURE: Integrated in HBase-Trunk_matrix #437 (See [https://builds.apache.org/job/HBase-Trunk_matrix/437/]) HBASE-14765 Remove snappy profile (eclark: rev 86da57f498c68703979dcd7ecc4b0ef2ce03723d) * src/main/asciidoc/_chapters/developer.adoc * pom.xml * hbase-shell/pom.xml * hbase-server/pom.xml * hbase-common/pom.xml > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14773: -- Attachment: HBASE-14773.patch > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992788#comment-14992788 ] Elliott Clark commented on HBASE-14773: --- [~appy] http://knowyourmeme.com/memes/copypasta :-) > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy paste error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HBASE-14773. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.3.0 1.2.0 2.0.0 > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy paste error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992723#comment-14992723 ] stack commented on HBASE-14773: --- +1 > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14763) Remove usages of deprecated HConnection
[ https://issues.apache.org/jira/browse/HBASE-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14763: -- Attachment: hbase-14763.v3.patch > Remove usages of deprecated HConnection > > > Key: HBASE-14763 > URL: https://issues.apache.org/jira/browse/HBASE-14763 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14763.patch, hbase-14763.v2.patch, > hbase-14763.v3.patch, hbase-14763.v3.patch > > > HConnection was deprecated in 1.0.0. There are two interfaces that are > supposed to be used instead -- Connection for client programs and > ClusterConnection for internal hbaes and special tools (LoadIncremental, > HBCK, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14427) Fix 'should' assertions in TestFastFail
[ https://issues.apache.org/jira/browse/HBASE-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992733#comment-14992733 ] stack commented on HBASE-14427: --- [~abhishek.chouhan] Want me to apply both patches then sir? The first for fix and the second to reenable the test? Let me put them together and get some test runs in here. > Fix 'should' assertions in TestFastFail > --- > > Key: HBASE-14427 > URL: https://issues.apache.org/jira/browse/HBASE-14427 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: Abhishek Singh Chouhan >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-14427.addendum.patch, HBASE-14427.patch > > > Over in HBASE-14421, TestFastFail has been failing assertions that talk of > events that 'should' be happening. Fix. For now HBASE-14421 has disabled the > 'should' assertions. They seem fine on apache jenkins build but fail fairly > reliably for me on alternate HW. > To address, get familiar with the test. Change the commented out asserts to > be yes/no instead of a 'likely' (On a cursory scan, it is possible that a > test run may not involve preemption and it is these runs that are throwing > asserts). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14775) Replication can't authenticate with peer Zookeeper with different server principal
Gary Helmling created HBASE-14775:
-
Summary: Replication can't authenticate with peer Zookeeper with
different server principal
Key: HBASE-14775
URL: https://issues.apache.org/jira/browse/HBASE-14775
Project: HBase
Issue Type: Bug
Reporter: Gary Helmling
Assignee: Gary Helmling
When replication is setup with security, where the local ZK cluster and peer ZK
cluster use different server principals, the source HBase cluster is unable to
authenticate with the peer ZK cluster.
When ZK is configured for SASL authentication and a server principal other than
the default ("zookeeper") is used, the correct server principal must be
specified on the client as a system property -- the confusingly named
{{zookeeper.sasl.client.username}}. However, since this is given as a system
property, authentication with the peer cluster breaks when it uses a different
ZK server principal than the local cluster.
We need a way of tying this setting to the replication peer config and then
setting the property when the peer's ZooKeeperWatcher is created.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14775) Replication can't authenticate with peer Zookeeper with different server principal
[
https://issues.apache.org/jira/browse/HBASE-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-14775:
--
Component/s: security
Replication
> Replication can't authenticate with peer Zookeeper with different server
> principal
> --
>
> Key: HBASE-14775
> URL: https://issues.apache.org/jira/browse/HBASE-14775
> Project: HBase
> Issue Type: Bug
> Components: Replication, security
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>
> When replication is setup with security, where the local ZK cluster and peer
> ZK cluster use different server principals, the source HBase cluster is
> unable to authenticate with the peer ZK cluster.
> When ZK is configured for SASL authentication and a server principal other
> than the default ("zookeeper") is used, the correct server principal must be
> specified on the client as a system property -- the confusingly named
> {{zookeeper.sasl.client.username}}. However, since this is given as a system
> property, authentication with the peer cluster breaks when it uses a
> different ZK server principal than the local cluster.
> We need a way of tying this setting to the replication peer config and then
> setting the property when the peer's ZooKeeperWatcher is created.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14776) Rewrite smart-patch.sh to use 'git am' or 'git apply' rather than 'patch'
Misty Stanley-Jones created HBASE-14776: --- Summary: Rewrite smart-patch.sh to use 'git am' or 'git apply' rather than 'patch' Key: HBASE-14776 URL: https://issues.apache.org/jira/browse/HBASE-14776 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 2.0.0 We require patches to be created using 'git format-patch' or 'git diff', so patches should be tested using 'git am' or 'git apply', not 'patch -pX'. This causes false errors in the Jenkins patch tester. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14773: - Description: Looks like a copy paste error. (was: Looks like a copy pasta error.) > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14773.patch > > > Looks like a copy paste error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache
[
https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992489#comment-14992489
]
Hadoop QA commented on HBASE-14708:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770848/HBASE-14708-v13.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770848
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1727 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16413//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16413//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16413//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16413//console
This message is automatically generated.
> Use copy on write Map for region location cache
> ---
>
> Key: HBASE-14708
> URL: https://issues.apache.org/jira/browse/HBASE-14708
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch,
> HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v2.patch,
> HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch,
> HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch,
> HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip,
> location_cache_times.pdf, result.csv
>
>
> Internally a co-worker profiled their application that was talking to HBase.
> > 60% of the time was spent in locating a region. This was while the cluster
> was stable and no regions were moving.
> To figure out if there was a faster way to cache region location I wrote up a
> benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache
> This tries to simulate a heavy load on the location cache.
> * 24 different threads.
> * 2 Deleting location data
> * 2 Adding location data
> * Using floor to get the result.
> To repeat my work just run ./run.sh and it should produce a result.csv
> Results:
> ConcurrentSkiplistMap is a good middle ground. It's got equal speed for
> reading and writing.
> However most operations will not need to remove or add a region location.
> There will be potentially several orders of magnitude more reads for cached
> locations than there will be on clearing the cache.
> So I propose a copy on write tree map.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
Elliott Clark created HBASE-14773: - Summary: Fix HBase shell tests are skipped when skipping server tests. Key: HBASE-14773 URL: https://issues.apache.org/jira/browse/HBASE-14773 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14755) Fix some broken links and HTML problems
[ https://issues.apache.org/jira/browse/HBASE-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992728#comment-14992728 ] stack commented on HBASE-14755: --- +1 > Fix some broken links and HTML problems > --- > > Key: HBASE-14755 > URL: https://issues.apache.org/jira/browse/HBASE-14755 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14755-v1.patch, HBASE-14755-v1.patch, > HBASE-14755.patch > > > Problems seen in > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Website%20Link%20Ckecker/3/artifact/link_report/index.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14774) Raise the font size on high-DPI small-screen devices like iphone 6+
[ https://issues.apache.org/jira/browse/HBASE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14774: Status: Patch Available (was: Open) > Raise the font size on high-DPI small-screen devices like iphone 6+ > --- > > Key: HBASE-14774 > URL: https://issues.apache.org/jira/browse/HBASE-14774 > Project: HBase > Issue Type: Bug > Components: website >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14774.patch > > > On iPads and things like that, the website looks fine. But the fonts are too > small on high-DPI small screens. It's tiny on my iPhone 6+. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[
https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992774#comment-14992774
]
Hadoop QA commented on HBASE-14766:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770874/HBASE-14766-v003.patch
against master branch at commit 86da57f498c68703979dcd7ecc4b0ef2ce03723d.
ATTACHMENT ID: 12770874
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are possible 1 zombie
test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16416//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16416//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16416//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16416//console
This message is automatically generated.
> WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with
> the new low-cost implementation.
> --
>
> Key: HBASE-14766
> URL: https://issues.apache.org/jira/browse/HBASE-14766
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch,
> HBASE-14766-v003.patch
>
>
> Cell's getFamily() gets an array copy of the cell's family, while in the
> filter function, it just needs to peek into the family and do a compare.
> Replace
> Bytes.toString(cell.getFamily())
> with
> Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength())
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14763) Remove usages of deprecated HConnection
[
https://issues.apache.org/jira/browse/HBASE-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992654#comment-14992654
]
Hadoop QA commented on HBASE-14763:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770855/hbase-14763.v3.patch
against master branch at commit 86da57f498c68703979dcd7ecc4b0ef2ce03723d.
ATTACHMENT ID: 12770855
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 75 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:red}-1 javac{color}. The applied patch generated 36 javac compiler
warnings (more than the master's current 35 warnings).
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16415//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16415//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16415//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16415//console
This message is automatically generated.
> Remove usages of deprecated HConnection
>
>
> Key: HBASE-14763
> URL: https://issues.apache.org/jira/browse/HBASE-14763
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-14763.patch, hbase-14763.v2.patch,
> hbase-14763.v3.patch
>
>
> HConnection was deprecated in 1.0.0. There are two interfaces that are
> supposed to be used instead -- Connection for client programs and
> ClusterConnection for internal hbaes and special tools (LoadIncremental,
> HBCK, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14771) RpcServer.getRemoteAddress always returns null.
[
https://issues.apache.org/jira/browse/HBASE-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992714#comment-14992714
]
Appy commented on HBASE-14771:
--
getRemoteAddress() always returns null because CurCall is thread local variable
and is only set in threads responsible for executing CallRunners (executor in
FifoRpcScheduler and those being created in RpcExecutor.startHandlers).
RpcServer.getRemoteIp also uses CurCall, so it'll always return null here.
Using {code}this.addr{code} makes sense.
It'll definitely fix 2 of the 3 uses of getRemoteAddress(). I am not sure about
the use in AccessController since I don't know it's internals.
[~a72877] what do you mean by "... fixing this issue"? Where are you seeing the
null values which change to legitimate address after this change?
> RpcServer.getRemoteAddress always returns null.
> ---
>
> Key: HBASE-14771
> URL: https://issues.apache.org/jira/browse/HBASE-14771
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 1.2.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14771.patch
>
>
> RpcServer.getRemoteAddress always returns null, because Call object is
> getting initialized with null.This seems to be happening because of using
> RpcServer.getRemoteIp() in Call object constructor before RpcServer thread
> local 'CurCall' being set in CallRunner.run method:
> {noformat}
> // --- RpcServer.java ---
> protected void processRequest(byte[] buf) throws IOException,
> InterruptedException {
> .
> // Call object getting initialized here with address
> // obtained from RpcServer.getRemoteIp()
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, RpcServer.getRemoteIp());
> scheduler.dispatch(new CallRunner(RpcServer.this, call));
> }
> // getRemoteIp method gets address from threadlocal 'CurCall' which
> // gets set in CallRunner.run and calling it before this as in above case,
> will return null
> // --- CallRunner.java ---
> public void run() {
> .
> Pair resultPair = null;
> RpcServer.CurCall.set(call);
> ..
> }
> // Using 'this.addr' in place of getRemoteIp method in RpcServer.java seems
> to be fixing this issue
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, this.addr);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992736#comment-14992736 ] Hudson commented on HBASE-14765: SUCCESS: Integrated in HBase-1.3-IT #296 (See [https://builds.apache.org/job/HBase-1.3-IT/296/]) HBASE-14765 Remove snappy profile (eclark: rev 2b84c40f014bc57808f726412b976bb67e337f0b) * hbase-common/pom.xml * hbase-shell/pom.xml * pom.xml * hbase-server/pom.xml * src/main/asciidoc/_chapters/developer.adoc > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14774) Raise the font size on high-DPI small-screen devices like iphone 6+
[ https://issues.apache.org/jira/browse/HBASE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14774: Attachment: HBASE-14774.patch This patch raises the font size a bit if the screen size is less than 768px and the density is more than 1.5. > Raise the font size on high-DPI small-screen devices like iphone 6+ > --- > > Key: HBASE-14774 > URL: https://issues.apache.org/jira/browse/HBASE-14774 > Project: HBase > Issue Type: Bug > Components: website >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14774.patch > > > On iPads and things like that, the website looks fine. But the fonts are too > small on high-DPI small screens. It's tiny on my iPhone 6+. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14774) Raise the font size on high-DPI small-screen devices like iphone 6+
Misty Stanley-Jones created HBASE-14774: --- Summary: Raise the font size on high-DPI small-screen devices like iphone 6+ Key: HBASE-14774 URL: https://issues.apache.org/jira/browse/HBASE-14774 Project: HBase Issue Type: Bug Components: website Affects Versions: 2.0.0 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 2.0.0 On iPads and things like that, the website looks fine. But the fonts are too small on high-DPI small screens. It's tiny on my iPhone 6+. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache
[
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992448#comment-14992448
]
Anoop Sam John commented on HBASE-14463:
Sorry for the delay.
I did the tests again by fixing the RKs what we fetch (via multi get) every
time. ( Not exactly this way as in patchpe_use_same_keys.patch. instead of
doing a random RK generation, I can use the incoming int 'i' to testRow(final
int i) to make RKs )
Both with and with out patch gives almost similar avg total run time for
threads. So in my test, I have ensured that every thread doing distinct RK
fetch only. And there also we don't degrade perf.
So here is my +1
Thanks for the perseverance [~carp84]
> Severe performance downgrade when parallel reading a single key from
> BucketCache
>
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch,
> HBASE-14463_v11.patch, HBASE-14463_v12.patch, HBASE-14463_v2.patch,
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch,
> TestBucketCache-new_with_IdLock.png,
> TestBucketCache-new_with_IdReadWriteLock.png,
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png,
> TestBucketCache_with_IdReadWriteLock-latest.png,
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
> TestBucketCache_with_IdReadWriteLock.png, pe_use_same_keys.patch,
> test-results.tar.gz
>
>
> We store feature data of online items in HBase, do machine learning on these
> features, and supply the outputs to our online search engine. In such
> scenario we will launch hundreds of yarn workers and each worker will read
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc
> issue, and just as titled we have observed severe performance downgrade.
> After some analytics we found the root cause is the lock in
> BucketCache#getBlock, as shown below
> {code}
> try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
> // ...
> int len = bucketEntry.getLength();
> Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
> bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the
> operation in LruCache. And since we're using synchronized in
> IdLock#getLockEntry, parallel read dropping on the same bucket would be
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14755) Fix some broken links and HTML problems
[
https://issues.apache.org/jira/browse/HBASE-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992458#comment-14992458
]
Hadoop QA commented on HBASE-14755:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770846/HBASE-14755-v1.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770846
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ The full HBase API test code, including private
and unstable APIs
+
org.apache.hadoop.hbase.backup*:org.apache.hadoop.hbase.catalog:org.apache.hadoop.hbase.client.coprocessor:org.apache.hadoop.hbase.client.metrics:org.apache.hadoop.hbase.codec*:org.apache.hadoop.hbase.constraint:org.apache.hadoop.hbase.coprocessor.*:org.apache.hadoop.hbase.executor:org.apache.hadoop.hbase.fs:*.generated.*:org.apache.hadoop.hbase.io.hfile.*:org.apache.hadoop.hbase.mapreduce.hadoopbackport:org.apache.hadoop.hbase.mapreduce.replication:org.apache.hadoop.hbase.master.*:org.apache.hadoop.hbase.metrics*:org.apache.hadoop.hbase.migration:org.apache.hadoop.hbase.monitoring:org.apache.hadoop.hbase.p*:org.apache.hadoop.hbase.regionserver.compactions:org.apache.hadoop.hbase.regionserver.handler:org.apache.hadoop.hbase.regionserver.snapshot:org.apache.hadoop.hbase.replication.*:org.apache.hadoop.hbase.rest.filter:org.apache.hadoop.hbase.rest.model:org.apache.hadoop.hbase.rest.p*:org.apache.hadoop.hbase.security.*:org.apache.hadoop.hbase.thrift*:org.apache.hadoop.hbase.tmpl.*:org.apache.hadoop.hbase.tool:org.apache.hadoop.hbase.trace:org.apache.hadoop.hbase.util.byterange*:org.apache.hadoop.hbase.util.test:org.apache.hadoop.hbase.util.vint:org.apache.hadoop.hbase.zookeeper.lock:org.apache.hadoop.metrics2*:org.apache.hadoop.hbase.io.compress*
+
org.apache.hadoop.hbase.backup*:org.apache.hadoop.hbase.catalog:org.apache.hadoop.hbase.client.coprocessor:org.apache.hadoop.hbase.client.metrics:org.apache.hadoop.hbase.codec*:org.apache.hadoop.hbase.constraint:org.apache.hadoop.hbase.coprocessor.*:org.apache.hadoop.hbase.executor:org.apache.hadoop.hbase.fs:*.generated.*:org.apache.hadoop.hbase.io.hfile.*:org.apache.hadoop.hbase.mapreduce.hadoopbackport:org.apache.hadoop.hbase.mapreduce.replication:org.apache.hadoop.hbase.master.*:org.apache.hadoop.hbase.metrics*:org.apache.hadoop.hbase.migration:org.apache.hadoop.hbase.monitoring:org.apache.hadoop.hbase.p*:org.apache.hadoop.hbase.regionserver.compactions:org.apache.hadoop.hbase.regionserver.handler:org.apache.hadoop.hbase.regionserver.snapshot:org.apache.hadoop.hbase.replication.*:org.apache.hadoop.hbase.rest.filter:org.apache.hadoop.hbase.rest.model:org.apache.hadoop.hbase.rest.p*:org.apache.hadoop.hbase.security.*:org.apache.hadoop.hbase.thrift*:org.apache.hadoop.hbase.tmpl.*:org.apache.hadoop.hbase.tool:org.apache.hadoop.hbase.trace:org.apache.hadoop.hbase.util.byterange*:org.apache.hadoop.hbase.util.test:org.apache.hadoop.hbase.util.vint:org.apache.hadoop.hbase.zookeeper.lock:org.apache.hadoop.metrics2*:org.apache.hadoop.hbase.io.compress*
+. 8 bytes: Block type, a sequence of bytes equivalent to version 1's "magic
records". Supported block types are:
+ As opposed to the above, this is not an HFile v2 block but a fixed>size (for
each HFile version) data structure
+We are supporting "meta" blocks in version 2 the same way they were supported
in version 1, even though we do not store Bloom filter data in these blocks
anymore.
+There are three types of block indexes in HFile version 2, stored in two
different formats (root and non-root):
+.. Optionally, version 2 intermediate levels, stored in the non%root format in
the data index section of the file. Intermediate levels can only be present
if leaf level blocks are present
+A version 2
[jira] [Updated] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-14766: - Attachment: HBASE-14766-v003.patch Address checkstyle errors. > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch, > HBASE-14766-v003.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13982) Add info for visibility labels/cell TTLs to ImportTsv
[
https://issues.apache.org/jira/browse/HBASE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992423#comment-14992423
]
Hadoop QA commented on HBASE-13982:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770860/HBASE_13982_1.patch
against master branch at commit 86da57f498c68703979dcd7ecc4b0ef2ce03723d.
ATTACHMENT ID: 12770860
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16414//console
This message is automatically generated.
> Add info for visibility labels/cell TTLs to ImportTsv
> -
>
> Key: HBASE-13982
> URL: https://issues.apache.org/jira/browse/HBASE-13982
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.1.0.1
>Reporter: Lars George
>Assignee: NIDHI GAMBHIR
> Labels: beginner
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13982.patch, HBASE_13982_1.patch
>
>
> HBASE-9832 added support for two more optional, special TSV columns, but no
> usage info was added. Add.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[
https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992424#comment-14992424
]
Hadoop QA commented on HBASE-14766:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770845/HBASE-14766-v002.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770845
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1728 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16412//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16412//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16412//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16412//console
This message is automatically generated.
> WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with
> the new low-cost implementation.
> --
>
> Key: HBASE-14766
> URL: https://issues.apache.org/jira/browse/HBASE-14766
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch
>
>
> Cell's getFamily() gets an array copy of the cell's family, while in the
> filter function, it just needs to peek into the family and do a compare.
> Replace
> Bytes.toString(cell.getFamily())
> with
> Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength())
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992481#comment-14992481 ] Hudson commented on HBASE-14765: SUCCESS: Integrated in HBase-1.2-IT #266 (See [https://builds.apache.org/job/HBase-1.2-IT/266/]) HBASE-14765 Remove snappy profile (eclark: rev b378b3459da34a64633fb41aeac157c006756267) * hbase-shell/pom.xml * hbase-server/pom.xml * src/main/asciidoc/_chapters/developer.adoc * pom.xml * hbase-common/pom.xml > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache
[ https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992554#comment-14992554 ] Elliott Clark commented on HBASE-14708: --- [~enis] Thoughts on this? You seem to be the last person to touch MetaCache. [~busbey] Thoughts on the plan for branch-1.2 ? > Use copy on write Map for region location cache > --- > > Key: HBASE-14708 > URL: https://issues.apache.org/jira/browse/HBASE-14708 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, > HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v2.patch, > HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, > HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, > HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, > location_cache_times.pdf, result.csv > > > Internally a co-worker profiled their application that was talking to HBase. > > 60% of the time was spent in locating a region. This was while the cluster > was stable and no regions were moving. > To figure out if there was a faster way to cache region location I wrote up a > benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache > This tries to simulate a heavy load on the location cache. > * 24 different threads. > * 2 Deleting location data > * 2 Adding location data > * Using floor to get the result. > To repeat my work just run ./run.sh and it should produce a result.csv > Results: > ConcurrentSkiplistMap is a good middle ground. It's got equal speed for > reading and writing. > However most operations will not need to remove or add a region location. > There will be potentially several orders of magnitude more reads for cached > locations than there will be on clearing the cache. > So I propose a copy on write tree map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache
[ https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992568#comment-14992568 ] Sean Busbey commented on HBASE-14708: - the plan sounds reasonable for 1.2. > Use copy on write Map for region location cache > --- > > Key: HBASE-14708 > URL: https://issues.apache.org/jira/browse/HBASE-14708 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, > HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v2.patch, > HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, > HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, > HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, > location_cache_times.pdf, result.csv > > > Internally a co-worker profiled their application that was talking to HBase. > > 60% of the time was spent in locating a region. This was while the cluster > was stable and no regions were moving. > To figure out if there was a faster way to cache region location I wrote up a > benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache > This tries to simulate a heavy load on the location cache. > * 24 different threads. > * 2 Deleting location data > * 2 Adding location data > * Using floor to get the result. > To repeat my work just run ./run.sh and it should produce a result.csv > Results: > ConcurrentSkiplistMap is a good middle ground. It's got equal speed for > reading and writing. > However most operations will not need to remove or add a region location. > There will be potentially several orders of magnitude more reads for cached > locations than there will be on clearing the cache. > So I propose a copy on write tree map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache
[
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-14463:
--
Attachment: HBASE-14463_v12.patch
Thanks [~anoop.hbase] for the double check and confirmation!
Checked and confirmed v12 patch could apply in the latest code base, re-attach
to see what HadoopQA will say (with recent commits/code changes)
> Severe performance downgrade when parallel reading a single key from
> BucketCache
>
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch,
> HBASE-14463_v11.patch, HBASE-14463_v12.patch, HBASE-14463_v12.patch,
> HBASE-14463_v2.patch, HBASE-14463_v3.patch, HBASE-14463_v4.patch,
> HBASE-14463_v5.patch, TestBucketCache-new_with_IdLock.png,
> TestBucketCache-new_with_IdReadWriteLock.png,
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png,
> TestBucketCache_with_IdReadWriteLock-latest.png,
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
> TestBucketCache_with_IdReadWriteLock.png, pe_use_same_keys.patch,
> test-results.tar.gz
>
>
> We store feature data of online items in HBase, do machine learning on these
> features, and supply the outputs to our online search engine. In such
> scenario we will launch hundreds of yarn workers and each worker will read
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc
> issue, and just as titled we have observed severe performance downgrade.
> After some analytics we found the root cause is the lock in
> BucketCache#getBlock, as shown below
> {code}
> try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
> // ...
> int len = bucketEntry.getLength();
> Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
> bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the
> operation in LruCache. And since we're using synchronized in
> IdLock#getLockEntry, parallel read dropping on the same bucket would be
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993108#comment-14993108 ] Hudson commented on HBASE-14773: SUCCESS: Integrated in HBase-1.2-IT #267 (See [https://builds.apache.org/job/HBase-1.2-IT/267/]) HBASE-14773 Fix HBase shell tests are skipped when skipping server (eclark: rev d1bdd68a813ea5c56eeb2576839177eb7b83726d) * hbase-shell/pom.xml > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14771) RpcServer.getRemoteAddress always returns null.
[
https://issues.apache.org/jira/browse/HBASE-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993165#comment-14993165
]
Abhishek Kumar commented on HBASE-14771:
it was mainly for logging purpose such as the use of
HMaster.getClientIdAuditPrefix() in create/modify etc operations and we have
some custom co-processor also making use of this getRemoteAddress method which
was causing some issues for us.
> RpcServer.getRemoteAddress always returns null.
> ---
>
> Key: HBASE-14771
> URL: https://issues.apache.org/jira/browse/HBASE-14771
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 1.2.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14771.patch
>
>
> RpcServer.getRemoteAddress always returns null, because Call object is
> getting initialized with null.This seems to be happening because of using
> RpcServer.getRemoteIp() in Call object constructor before RpcServer thread
> local 'CurCall' being set in CallRunner.run method:
> {noformat}
> // --- RpcServer.java ---
> protected void processRequest(byte[] buf) throws IOException,
> InterruptedException {
> .
> // Call object getting initialized here with address
> // obtained from RpcServer.getRemoteIp()
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, RpcServer.getRemoteIp());
> scheduler.dispatch(new CallRunner(RpcServer.this, call));
> }
> // getRemoteIp method gets address from threadlocal 'CurCall' which
> // gets set in CallRunner.run and calling it before this as in above case,
> will return null
> // --- CallRunner.java ---
> public void run() {
> .
> Pair resultPair = null;
> RpcServer.CurCall.set(call);
> ..
> }
> // Using 'this.addr' in place of getRemoteIp method in RpcServer.java seems
> to be fixing this issue
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, this.addr);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14767) Remove deprecated functions from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993216#comment-14993216 ] Appy commented on HBASE-14767: -- working on changes in shell code > Remove deprecated functions from HBaseAdmin > --- > > Key: HBASE-14767 > URL: https://issues.apache.org/jira/browse/HBASE-14767 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-14767-master-v2.patch, > HBASE-14767-master-v3.patch, HBASE-14767-master.patch > > > Many functions in HBaseAdmin are marked deprecated. Removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992820#comment-14992820 ] Appy commented on HBASE-14773: -- Mmmm, delicious bug. > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy paste error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14759) Avoid using Math.abs when selecting SyncRunner in FSHLog
[
https://issues.apache.org/jira/browse/HBASE-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-14759:
--
Affects Version/s: 1.3.0
Fix Version/s: 1.3.0
> Avoid using Math.abs when selecting SyncRunner in FSHLog
>
>
> Key: HBASE-14759
> URL: https://issues.apache.org/jira/browse/HBASE-14759
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.0.2, 1.2.0, 1.1.2, 1.3.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3
>
> Attachments: HBASE-14759.patch
>
>
> {code:title=FSHLog.java}
> int index = Math.abs(this.syncRunnerIndex++) % this.syncRunners.length;
> try {
> this.syncRunners[index].offer(sequence, this.syncFutures,
> this.syncFuturesCount);
> } catch (Exception e) {
> // Should NEVER get here.
> requestLogRoll();
> this.exception = new DamagedWALException("Failed offering sync",
> e);
> }
> {code}
> Math.abs will return Integer.MIN_VALUE if you pass Integer.MIN_VALUE in since
> the actual absolute value of Integer.MIN_VALUE is out of range.
> I think {{this.syncRunnerIndex++}} will overflow eventually if we keep the
> regionserver running for enough time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993026#comment-14993026 ] Hudson commented on HBASE-14773: FAILURE: Integrated in HBase-1.3-IT #297 (See [https://builds.apache.org/job/HBase-1.3-IT/297/]) HBASE-14773 Fix HBase shell tests are skipped when skipping server (eclark: rev 7c9e1738306264def7f19ed1607974acb98cecfb) * hbase-shell/pom.xml > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
