[jira] [Commented] (HBASE-27146) Avoid CellUtil.cloneRow in MetaCellComparator
[
https://issues.apache.org/jira/browse/HBASE-27146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558728#comment-17558728
]
Hudson commented on HBASE-27146:
Results for branch branch-2
[build #576 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576//console].
> Avoid CellUtil.cloneRow in MetaCellComparator
> -
>
> Key: HBASE-27146
> URL: https://issues.apache.org/jira/browse/HBASE-27146
> Project: HBase
> Issue Type: Improvement
> Components: meta, Offheaping, Performance
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-4, 2.4.14
>
>
> In HBASE-26981, a flame graph shows that we spend a lot of CPUs in
> CellUtils.cloneXXX.
> Since we need to split the row to different parts when comparing, we can not
> use ByteBuffer compare directly, but as least, we should try to avoid copy
> the content when the Cell is stored in a ByteBuffer.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26790) getAllRegionLocations can cache locations with null hostname
[
https://issues.apache.org/jira/browse/HBASE-26790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558727#comment-17558727
]
Hudson commented on HBASE-26790:
Results for branch branch-2
[build #576 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576//console].
> getAllRegionLocations can cache locations with null hostname
>
>
> Key: HBASE-26790
> URL: https://issues.apache.org/jira/browse/HBASE-26790
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Fix For: 2.5.1, 3.0.0-alpha-4, 2.4.14
>
>
> RegionLocator methods typically delegate to
> ConnectionImplementation.locateRegion, which throws a
> NoServerForRegionException if the located region's serverName is null.
> RegionLocator.getAllRegionLocations does not go through that path, instead
> caching all returned region locations without any validation. This can result
> in a "dirty" meta cache, since clients do not expect to have null serverNames
> in the meta cache. We should add the same throwing of
> NoServerForRegionException to this method as used in the others. Or at least
> we should not cache the result if the serverName is null.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26945) Quotas causes too much load on meta for large clusters
[
https://issues.apache.org/jira/browse/HBASE-26945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558725#comment-17558725
]
Hudson commented on HBASE-26945:
Results for branch branch-2
[build #576 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576//console].
> Quotas causes too much load on meta for large clusters
> --
>
> Key: HBASE-26945
> URL: https://issues.apache.org/jira/browse/HBASE-26945
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Fix For: 2.5.1, 3.0.0-alpha-4, 2.4.14
>
>
> We've been upgrading larger clusters to hbase 2, and ran into this issue
> where two equivalent clusters (one on 1.2 and the other on 2.4.x) running
> almost identical workloads, but the hbase2 cluster was doing way more meta
> requests.
> One of the reasons seems to be from
> https://issues.apache.org/jira/browse/HBASE-21820, which added an
> updateQuotaFactors method to the QuotaCache QuotaRefreshChore. This new
> method calls MetaTableAcessor.scanMeta for every table in the cluster. This
> is happening on every regionserver, so if you have many tables and/or many
> regionservers you can easily start sending lots of traffic to meta. One way
> to offset this is to reduce frequency of hbase.quota.refresh.period, but this
> means you are less able to quickly respond to changes in load.
> We should figure out a caching or notification mechanism that can reduce the
> need scan meta so much. It seems like we're mainly scanning meta to get a
> count of total regions per table, which should not be changing so often.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-27060) Allow sharing connections between AggregationClient instances
[
https://issues.apache.org/jira/browse/HBASE-27060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558729#comment-17558729
]
Hudson commented on HBASE-27060:
Results for branch branch-2
[build #576 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576//console].
> Allow sharing connections between AggregationClient instances
> -
>
> Key: HBASE-27060
> URL: https://issues.apache.org/jira/browse/HBASE-27060
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
> Fix For: 2.5.1, 3.0.0-alpha-4, 2.4.14
>
>
> AggregationClient only has a single constructor which takes a Configuration.
> The constructor uses the Configuration to create a Connection.
> However, some of the AggregationClient methods take a Table argument. In
> those cases it doesn't use the created Connection at all.
> We should add another constructor which does not create a Connection so that
> people can use AggregationClient with externally managed Connection.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-27111) Make Netty channel bytebuf allocator configurable
[
https://issues.apache.org/jira/browse/HBASE-27111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558726#comment-17558726
]
Hudson commented on HBASE-27111:
Results for branch branch-2
[build #576 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/576//console].
> Make Netty channel bytebuf allocator configurable
> -
>
> Key: HBASE-27111
> URL: https://issues.apache.org/jira/browse/HBASE-27111
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-4
>
>
> Netty supports different strategies for allocating byte buffers for IO and
> channel modules. We do not allow the site operator to fine tune this but
> could. It would be particularly useful to allow preference of heap buffers
> where direct memory may be limited or utilized for other purposes.
> Support site configuration of the bytebuf allocator that Netty will use for
> NettyRpcServer channels. Property name is 'hbase.netty.rpcserver.allocator'.
> Default is no value, which is equivalent to "pooled". Valid values are:
> - "pooled": use PooledByteBufAllocator
> - "unpooled": use UnpooledByteBufAllocator
> - "heap": use HeapByteBufAllocator, which is a PooledByteBufAllocator that
> preferentially allocates buffers on heap wherever possible
> - : If the value is none of the recognized labels, treat it as a class
> name implementing
> org.apache.hbase.thirdparty.io.netty.buffer.ByteBufAllocator. This allows the
> user to add a custom implementation, perhaps for debugging.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-27146) Avoid CellUtil.cloneRow in MetaCellComparator
[
https://issues.apache.org/jira/browse/HBASE-27146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558718#comment-17558718
]
Hudson commented on HBASE-27146:
Results for branch branch-2.4
[build #377 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377//console].
> Avoid CellUtil.cloneRow in MetaCellComparator
> -
>
> Key: HBASE-27146
> URL: https://issues.apache.org/jira/browse/HBASE-27146
> Project: HBase
> Issue Type: Improvement
> Components: meta, Offheaping, Performance
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-4, 2.4.14
>
>
> In HBASE-26981, a flame graph shows that we spend a lot of CPUs in
> CellUtils.cloneXXX.
> Since we need to split the row to different parts when comparing, we can not
> use ByteBuffer compare directly, but as least, we should try to avoid copy
> the content when the Cell is stored in a ByteBuffer.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26790) getAllRegionLocations can cache locations with null hostname
[
https://issues.apache.org/jira/browse/HBASE-26790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558717#comment-17558717
]
Hudson commented on HBASE-26790:
Results for branch branch-2.4
[build #377 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377//console].
> getAllRegionLocations can cache locations with null hostname
>
>
> Key: HBASE-26790
> URL: https://issues.apache.org/jira/browse/HBASE-26790
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Fix For: 2.5.1, 3.0.0-alpha-4, 2.4.14
>
>
> RegionLocator methods typically delegate to
> ConnectionImplementation.locateRegion, which throws a
> NoServerForRegionException if the located region's serverName is null.
> RegionLocator.getAllRegionLocations does not go through that path, instead
> caching all returned region locations without any validation. This can result
> in a "dirty" meta cache, since clients do not expect to have null serverNames
> in the meta cache. We should add the same throwing of
> NoServerForRegionException to this method as used in the others. Or at least
> we should not cache the result if the serverName is null.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26945) Quotas causes too much load on meta for large clusters
[
https://issues.apache.org/jira/browse/HBASE-26945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558716#comment-17558716
]
Hudson commented on HBASE-26945:
Results for branch branch-2.4
[build #377 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377//console].
> Quotas causes too much load on meta for large clusters
> --
>
> Key: HBASE-26945
> URL: https://issues.apache.org/jira/browse/HBASE-26945
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Fix For: 2.5.1, 3.0.0-alpha-4, 2.4.14
>
>
> We've been upgrading larger clusters to hbase 2, and ran into this issue
> where two equivalent clusters (one on 1.2 and the other on 2.4.x) running
> almost identical workloads, but the hbase2 cluster was doing way more meta
> requests.
> One of the reasons seems to be from
> https://issues.apache.org/jira/browse/HBASE-21820, which added an
> updateQuotaFactors method to the QuotaCache QuotaRefreshChore. This new
> method calls MetaTableAcessor.scanMeta for every table in the cluster. This
> is happening on every regionserver, so if you have many tables and/or many
> regionservers you can easily start sending lots of traffic to meta. One way
> to offset this is to reduce frequency of hbase.quota.refresh.period, but this
> means you are less able to quickly respond to changes in load.
> We should figure out a caching or notification mechanism that can reduce the
> need scan meta so much. It seems like we're mainly scanning meta to get a
> count of total regions per table, which should not be changing so often.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-27060) Allow sharing connections between AggregationClient instances
[
https://issues.apache.org/jira/browse/HBASE-27060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558719#comment-17558719
]
Hudson commented on HBASE-27060:
Results for branch branch-2.4
[build #377 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/377//console].
> Allow sharing connections between AggregationClient instances
> -
>
> Key: HBASE-27060
> URL: https://issues.apache.org/jira/browse/HBASE-27060
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
> Fix For: 2.5.1, 3.0.0-alpha-4, 2.4.14
>
>
> AggregationClient only has a single constructor which takes a Configuration.
> The constructor uses the Configuration to create a Connection.
> However, some of the AggregationClient methods take a Table argument. In
> those cases it doesn't use the created Connection at all.
> We should add another constructor which does not create a Connection so that
> people can use AggregationClient with externally managed Connection.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4562: HBASE-27048 Server side scanner time limit should account for time in queue
Apache-HBase commented on PR #4562: URL: https://github.com/apache/hbase/pull/4562#issuecomment-1166186371 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 26s | master passed | | +1 :green_heart: | compile | 0m 34s | master passed | | +1 :green_heart: | shadedjars | 3m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 3s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | shadedjars | 3m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 203m 27s | hbase-server in the patch passed. | | | | 219m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4562/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4562 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 064c36c3903b 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d447fa01ba | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4562/7/testReport/ | | Max. process+thread count | 2244 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4562/7/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4562: HBASE-27048 Server side scanner time limit should account for time in queue
Apache-HBase commented on PR #4562: URL: https://github.com/apache/hbase/pull/4562#issuecomment-1166186336 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 20s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 33s | master passed | | +1 :green_heart: | compile | 0m 46s | master passed | | +1 :green_heart: | shadedjars | 3m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 34s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 3m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 202m 35s | hbase-server in the patch passed. | | | | 219m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4562/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4562 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux aa7e8584ee17 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d447fa01ba | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4562/7/testReport/ | | Max. process+thread count | 2555 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4562/7/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26856) BufferedDataBlockEncoder.OnheapDecodedCell value can get corrupted
[
https://issues.apache.org/jira/browse/HBASE-26856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558705#comment-17558705
]
Duo Zhang commented on HBASE-26856:
---
Reviewing the related code again, I wonder why the value can be changed? Could
you please provide a UT to reproduce the problem? For me, seems the block data
will be read either from the HFileBlock directly, or from block cache? None of
them can be changed, unless you do not reference the block any more.
So I prefer we revert the changes here first, and do a more deep analysis on
what is the root cause of the problem, and then find a suitable way to fix.
Thoughts? Thanks.
> BufferedDataBlockEncoder.OnheapDecodedCell value can get corrupted
> --
>
> Key: HBASE-26856
> URL: https://issues.apache.org/jira/browse/HBASE-26856
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> In our production cluster we observed the cell value is modified after
> successful scanner read. After analyzing we observed OnheapDecodedCell is not
> created properly.
> We create OnheapDecodedCell with complete valAndTagsBuffer underlying array.
> {code:java}
> return new OnheapDecodedCell(Bytes.copy(keyBuffer, 0, this.keyLength),
> currentKey.getRowLength(), currentKey.getFamilyOffset(),
> currentKey.getFamilyLength(),
> currentKey.getQualifierOffset(), currentKey.getQualifierLength(),
> currentKey.getTimestamp(), currentKey.getTypeByte(),
> valAndTagsBuffer.array(),
> valAndTagsBuffer.arrayOffset() + vOffset, this.valueLength,
> memstoreTS, tagsArray,
> tOffset, this.tagsLength);
> {code}
> Here we are passing valAndTagsBuffer.array() for value extraction.
> The underlying array will be modified if it is altered anywhere.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HBASE-26101) Release 3.0.0
[ https://issues.apache.org/jira/browse/HBASE-26101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26101: -- Component/s: community > Release 3.0.0 > - > > Key: HBASE-26101 > URL: https://issues.apache.org/jira/browse/HBASE-26101 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-13126) Provide alternate mini cluster classes other than HBTU for downstream users to write unit tests
[ https://issues.apache.org/jira/browse/HBASE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-13126: -- Release Note: Introduce a TestingHBaseCluster for users to implement integration test with mini hbase cluster. See this section https://hbase.apache.org/book.html#_integration_testing_with_an_hbase_mini_cluster in the ref guide for more details on how to use it. TestingHBaseCluster also allowes you to start a mini hbase cluster based on external HDFS cluster and zookeeper cluster, please see the release note of HBASE-26167 for more details. HBaseTestingUtility is marked as deprecated and will be 'removed' in the future. > Provide alternate mini cluster classes other than HBTU for downstream users > to write unit tests > --- > > Key: HBASE-13126 > URL: https://issues.apache.org/jira/browse/HBASE-13126 > Project: HBase > Issue Type: Umbrella > Components: API, test >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0-alpha-4 > > > Over in the review for HBASE-12972, [~enis] mentioned that one of the HBTU > methods wasn't intended for public consumption. > Can we build a list of such methods across the API, appropriately annotate > them for 2.0, and deprecate them in earlier versions with a warning that > they're going to be restricted? -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-13126) Provide alternate mini cluster classes other than HBTU for downstream users to write unit tests
[ https://issues.apache.org/jira/browse/HBASE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-13126: -- Fix Version/s: 2.5.0 > Provide alternate mini cluster classes other than HBTU for downstream users > to write unit tests > --- > > Key: HBASE-13126 > URL: https://issues.apache.org/jira/browse/HBASE-13126 > Project: HBase > Issue Type: Umbrella > Components: API, test >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-4 > > > Over in the review for HBASE-12972, [~enis] mentioned that one of the HBTU > methods wasn't intended for public consumption. > Can we build a list of such methods across the API, appropriately annotate > them for 2.0, and deprecate them in earlier versions with a warning that > they're going to be restricted? -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-27159) Emit source metrics for BlockCacheExpressHitPercent, blockCache counts of hits and misses for cacheable requests
[
https://issues.apache.org/jira/browse/HBASE-27159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Manning updated HBASE-27159:
--
Description:
[https://github.com/apache/hbase/blob/d447fa01ba36a11d57927b78cce1bbca361b1d52/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java#L346-L400]
{code:java}
public double getHitCachingRatio() {
double requestCachingCount = getRequestCachingCount();
if (requestCachingCount == 0) {
return 0;
}
return getHitCachingCount() / requestCachingCount;
} {code}
This code is responsible for the metric {{{}BlockCacheExpressHitPercent{}}}.
The metric represents the percentage of requests which were cacheable, but not
found in the cache. Unfortunately, since the counters are process-level
counters, the ratio is for the lifetime of the process. This makes it less
useful for looking at cache behavior during a smaller time period.
The underlying counters are {{hitCachingCount}} and {{{}missCachingCount{}}}.
Having access to the underlying counters allows for offline computation of the
same metric for any given time period. But these counters are not emitted today
from {{{}MetricsRegionServerWrapperImpl.java{}}}.
Compare this to {{hitCount}} and {{missCount}} which are emitted as metrics
{{blockCacheHitCount}} and {{{}blockCacheMissCount{}}}. But these are raw
counts for the cache, which include requests that are not cacheable. The
cacheable metrics are more interesting, since it can be common to miss on a
request which is not cacheable.
Interestingly, these metrics are emitted regularly as part of a log line in
{{{}StatisticsThread.logStats{}}}.
We should emit blockCache{{{}HitCachingCount{}}} and
{{blockCacheMissCachingCount}} along with the current metrics.
was:
[https://github.com/apache/hbase/blob/d447fa01ba36a11d57927b78cce1bbca361b1d52/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java#L346-L400]
{code:java}
public double getHitCachingRatio() {
double requestCachingCount = getRequestCachingCount();
if (requestCachingCount == 0) {
return 0;
}
return getHitCachingCount() / requestCachingCount;
} {code}
This code is responsible for the metric {{{}BlockCacheExpressHitPercent{}}}.
The metric represents the percentage of requests which were cacheable, but not
found in the cache. Unfortunately, since the counters are process-level
counters, the ratio is for the lifetime of the process. This makes it less
useful for looking at cache behavior during a smaller time period.
The underlying counters are {{hitCachingCount}} and {{{}missCachingCount{}}}.
Having access to the underlying counters allows for offline computation of the
same metric for any given time period. But these counters are not emitted today
from {{{}MetricsRegionServerWrapperImpl.java{}}}.
Compare this to {{hitCount}} and {{missCount}} which are emitted as metrics
{{blockCacheHitCount}} and {{{}blockCacheMissCount{}}}. But these are raw
counts for the cache, which include requests that are not cacheable. The
cacheable metrics are more interesting, since it can be common to miss on a
request which is not cacheable.
We should emit blockCache{{{}HitCachingCount{}}} and
{{blockCacheMissCachingCount}} along with the current metrics.
> Emit source metrics for BlockCacheExpressHitPercent, blockCache counts of
> hits and misses for cacheable requests
>
>
> Key: HBASE-27159
> URL: https://issues.apache.org/jira/browse/HBASE-27159
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, metrics
>Affects Versions: 3.0.0-alpha-1, 2.0.0
>Reporter: David Manning
>Priority: Minor
>
> [https://github.com/apache/hbase/blob/d447fa01ba36a11d57927b78cce1bbca361b1d52/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java#L346-L400]
> {code:java}
> public double getHitCachingRatio() {
> double requestCachingCount = getRequestCachingCount();
> if (requestCachingCount == 0) {
> return 0;
> }
> return getHitCachingCount() / requestCachingCount;
> } {code}
> This code is responsible for the metric {{{}BlockCacheExpressHitPercent{}}}.
> The metric represents the percentage of requests which were cacheable, but
> not found in the cache. Unfortunately, since the counters are process-level
> counters, the ratio is for the lifetime of the process. This makes it less
> useful for looking at cache behavior during a smaller time period.
> The underlying counters are {{hitCachingCount}} and {{{}missCachingCount{}}}.
> Having access to the underlying counters allows for offline computation of
> the same metric for any given time period. But these counters are not emitted
> today from {{{}MetricsRegionServerWrapperImpl.java{}}}.
[GitHub] [hbase] Apache-HBase commented on pull request #4562: HBASE-27048 Server side scanner time limit should account for time in queue
Apache-HBase commented on PR #4562: URL: https://github.com/apache/hbase/pull/4562#issuecomment-1166158663 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 26s | master passed | | +1 :green_heart: | compile | 2m 13s | master passed | | +1 :green_heart: | checkstyle | 0m 28s | master passed | | +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 32s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 6s | the patch passed | | +1 :green_heart: | compile | 2m 12s | the patch passed | | +1 :green_heart: | javac | 2m 12s | the patch passed | | +1 :green_heart: | checkstyle | 0m 29s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 54s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotless | 0m 39s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 33m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4562/7/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4562 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 00b660b9f898 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d447fa01ba | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4562/7/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-27159) Emit source metrics for BlockCacheExpressHitPercent, blockCache counts of hits and misses for cacheable requests
[
https://issues.apache.org/jira/browse/HBASE-27159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Manning updated HBASE-27159:
--
Summary: Emit source metrics for BlockCacheExpressHitPercent, blockCache
counts of hits and misses for cacheable requests (was: Emit source metrics for
BlockCacheExpressHitPercent, getHitCachingRatio, getHitCachingCount,
getMissCachingCount)
> Emit source metrics for BlockCacheExpressHitPercent, blockCache counts of
> hits and misses for cacheable requests
>
>
> Key: HBASE-27159
> URL: https://issues.apache.org/jira/browse/HBASE-27159
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, metrics
>Affects Versions: 3.0.0-alpha-1, 2.0.0
>Reporter: David Manning
>Priority: Minor
>
> [https://github.com/apache/hbase/blob/d447fa01ba36a11d57927b78cce1bbca361b1d52/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java#L346-L400]
> {code:java}
> public double getHitCachingRatio() {
> double requestCachingCount = getRequestCachingCount();
> if (requestCachingCount == 0) {
> return 0;
> }
> return getHitCachingCount() / requestCachingCount;
> } {code}
> This code is responsible for the metric {{{}BlockCacheExpressHitPercent{}}}.
> The metric represents the percentage of requests which were cacheable, but
> not found in the cache. Unfortunately, since the counters are process-level
> counters, the ratio is for the lifetime of the process. This makes it less
> useful for looking at cache behavior during a smaller time period.
> The underlying counters are {{hitCachingCount}} and {{{}missCachingCount{}}}.
> Having access to the underlying counters allows for offline computation of
> the same metric for any given time period. But these counters are not emitted
> today from {{{}MetricsRegionServerWrapperImpl.java{}}}.
> Compare this to {{hitCount}} and {{missCount}} which are emitted as metrics
> {{blockCacheHitCount}} and {{{}blockCacheMissCount{}}}. But these are raw
> counts for the cache, which include requests that are not cacheable. The
> cacheable metrics are more interesting, since it can be common to miss on a
> request which is not cacheable.
> We should emit blockCache{{{}HitCachingCount{}}} and
> {{blockCacheMissCachingCount}} along with the current metrics.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (HBASE-27159) Emit source metrics for BlockCacheExpressHitPercent, getHitCachingRatio, getHitCachingCount, getMissCachingCount
David Manning created HBASE-27159:
-
Summary: Emit source metrics for BlockCacheExpressHitPercent,
getHitCachingRatio, getHitCachingCount, getMissCachingCount
Key: HBASE-27159
URL: https://issues.apache.org/jira/browse/HBASE-27159
Project: HBase
Issue Type: Improvement
Components: BlockCache, metrics
Affects Versions: 2.0.0, 3.0.0-alpha-1
Reporter: David Manning
[https://github.com/apache/hbase/blob/d447fa01ba36a11d57927b78cce1bbca361b1d52/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java#L346-L400]
{code:java}
public double getHitCachingRatio() {
double requestCachingCount = getRequestCachingCount();
if (requestCachingCount == 0) {
return 0;
}
return getHitCachingCount() / requestCachingCount;
} {code}
This code is responsible for the metric {{{}BlockCacheExpressHitPercent{}}}.
The metric represents the percentage of requests which were cacheable, but not
found in the cache. Unfortunately, since the counters are process-level
counters, the ratio is for the lifetime of the process. This makes it less
useful for looking at cache behavior during a smaller time period.
The underlying counters are {{hitCachingCount}} and {{{}missCachingCount{}}}.
Having access to the underlying counters allows for offline computation of the
same metric for any given time period. But these counters are not emitted today
from {{{}MetricsRegionServerWrapperImpl.java{}}}.
Compare this to {{hitCount}} and {{missCount}} which are emitted as metrics
{{blockCacheHitCount}} and {{{}blockCacheMissCount{}}}. But these are raw
counts for the cache, which include requests that are not cacheable. The
cacheable metrics are more interesting, since it can be common to miss on a
request which is not cacheable.
We should emit blockCache{{{}HitCachingCount{}}} and
{{blockCacheMissCachingCount}} along with the current metrics.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HBASE-27078) Allow configuring a separate timeout for meta scans
[
https://issues.apache.org/jira/browse/HBASE-27078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-27078:
Fix Version/s: 2.5.0
3.0.0-alpha-4
> Allow configuring a separate timeout for meta scans
> ---
>
> Key: HBASE-27078
> URL: https://issues.apache.org/jira/browse/HBASE-27078
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
> Fix For: 2.5.0, 3.0.0-alpha-4
>
>
> There is a {{hbase.client.meta.operation.timeout}} but it does not apply to
> meta scans, which are the primary use-case for clients (i.e. through
> RegionLocator).
> Many user-facing clients may want to have low rpc and scan timeouts. However,
> in periods of meta hotspotting, those timeouts can be way too low for the
> meta scans. The problem with low timeouts for meta scans is that without a
> populated MetaCache, user requests cannot succeed. In fact, user requests
> will continually try to re-scan meta until the MetaCache is populated. So
> having a lower rpc timeout will cause a situation where meta scans cannot
> succeed, and thus user requests cannot succeed. In this case I think it'd be
> preferable to relax the rpc timeout for meta requests so that a few long
> requests can unblock many faster requests.
> My suggestion would be to add an {{hbase.client.meta.rpc.timeout}} and ensure
> that it applies to meta scans. I also think it would be less confusing to
> have {{hbase.client.meta.operation.timeout}} apply as the scanner timeout
> period for meta scans.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26945) Quotas causes too much load on meta for large clusters
[
https://issues.apache.org/jira/browse/HBASE-26945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558672#comment-17558672
]
Hudson commented on HBASE-26945:
Results for branch branch-2.5
[build #150 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Quotas causes too much load on meta for large clusters
> --
>
> Key: HBASE-26945
> URL: https://issues.apache.org/jira/browse/HBASE-26945
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Fix For: 2.5.1, 3.0.0-alpha-4, 2.4.14
>
>
> We've been upgrading larger clusters to hbase 2, and ran into this issue
> where two equivalent clusters (one on 1.2 and the other on 2.4.x) running
> almost identical workloads, but the hbase2 cluster was doing way more meta
> requests.
> One of the reasons seems to be from
> https://issues.apache.org/jira/browse/HBASE-21820, which added an
> updateQuotaFactors method to the QuotaCache QuotaRefreshChore. This new
> method calls MetaTableAcessor.scanMeta for every table in the cluster. This
> is happening on every regionserver, so if you have many tables and/or many
> regionservers you can easily start sending lots of traffic to meta. One way
> to offset this is to reduce frequency of hbase.quota.refresh.period, but this
> means you are less able to quickly respond to changes in load.
> We should figure out a caching or notification mechanism that can reduce the
> need scan meta so much. It seems like we're mainly scanning meta to get a
> count of total regions per table, which should not be changing so often.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-27111) Make Netty channel bytebuf allocator configurable
[
https://issues.apache.org/jira/browse/HBASE-27111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558673#comment-17558673
]
Hudson commented on HBASE-27111:
Results for branch branch-2.5
[build #150 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Make Netty channel bytebuf allocator configurable
> -
>
> Key: HBASE-27111
> URL: https://issues.apache.org/jira/browse/HBASE-27111
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-4
>
>
> Netty supports different strategies for allocating byte buffers for IO and
> channel modules. We do not allow the site operator to fine tune this but
> could. It would be particularly useful to allow preference of heap buffers
> where direct memory may be limited or utilized for other purposes.
> Support site configuration of the bytebuf allocator that Netty will use for
> NettyRpcServer channels. Property name is 'hbase.netty.rpcserver.allocator'.
> Default is no value, which is equivalent to "pooled". Valid values are:
> - "pooled": use PooledByteBufAllocator
> - "unpooled": use UnpooledByteBufAllocator
> - "heap": use HeapByteBufAllocator, which is a PooledByteBufAllocator that
> preferentially allocates buffers on heap wherever possible
> - : If the value is none of the recognized labels, treat it as a class
> name implementing
> org.apache.hbase.thirdparty.io.netty.buffer.ByteBufAllocator. This allows the
> user to add a custom implementation, perhaps for debugging.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-27105) HBaseInterClusterReplicationEndpoint should honor replication adaptive timeout
[
https://issues.apache.org/jira/browse/HBASE-27105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558670#comment-17558670
]
Hudson commented on HBASE-27105:
Results for branch branch-2.5
[build #150 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBaseInterClusterReplicationEndpoint should honor replication adaptive timeout
> --
>
> Key: HBASE-27105
> URL: https://issues.apache.org/jira/browse/HBASE-27105
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-4
>
>
> HBASE-23293 introduced replication.source.shipedits.timeout which is
> adaptive, ReplicationSourceShipper#shipEdits() set the adaptive timeout based
> on retries. But on CallTimeoutException in
> HBaseInterClusterReplicationEndpoint#replicate(), it keep retrying the
> replication after sleep with same timeout value.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-27001) The deleted variable cannot be printed out
[
https://issues.apache.org/jira/browse/HBASE-27001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558671#comment-17558671
]
Hudson commented on HBASE-27001:
Results for branch branch-2.5
[build #150 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/150/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The deleted variable cannot be printed out
> --
>
> Key: HBASE-27001
> URL: https://issues.apache.org/jira/browse/HBASE-27001
> Project: HBase
> Issue Type: Bug
>Reporter: selina.yan
>Assignee: selina.yan
>Priority: Minor
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> In the deleteAction method of CleanerChore class, the delete variable cannot
> be printed out
> {code:java}
> LOG.trace("Finish deleting {} under {}, deleted=", type, dir, deleted); {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Closed] (HBASE-18175) Add hbase-spark integration test into hbase-spark-it
[ https://issues.apache.org/jira/browse/HBASE-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-18175. --- > Add hbase-spark integration test into hbase-spark-it > > > Key: HBASE-18175 > URL: https://issues.apache.org/jira/browse/HBASE-18175 > Project: HBase > Issue Type: Test > Components: hbase-connectors, spark >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Critical > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: addendum-master.patch, hbase-18175-master-v2.patch, > hbase-18175-master-v3.patch, hbase-18175-master-v4.patch, > hbase-18175-master-v5.patch, hbase-18175-master-v6.patch, > hbase-18175-master-v7.patch, hbase-18175-master-v8.patch, > hbase-18175-master-v9.patch, hbase-18175-v1.patch > > > After HBASE-17574, all test under hbase-spark are regarded as unit test, and > this jira will add integration test of hbase-spark into hbase-it. This patch > run same tests as mapreduce.IntegrationTestBulkLoad, just change mapreduce to > spark. > test in Maven: > mvn verify -Dit.test=IntegrationTestSparkBulkLoad > test on cluster: > spark-submit --class > org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad > HBASE_HOME/lib/hbase-it-2.0.0-SNAPSHOT-tests.jar > -Dhbase.spark.bulkload.chainlength=50 -m slowDeterministic -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14406) The dataframe datasource filter is wrong, and will result in data loss or unexpected behavior
[ https://issues.apache.org/jira/browse/HBASE-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14406. --- > The dataframe datasource filter is wrong, and will result in data loss or > unexpected behavior > - > > Key: HBASE-14406 > URL: https://issues.apache.org/jira/browse/HBASE-14406 > Project: HBase > Issue Type: Bug > Components: hbase-connectors, spark >Affects Versions: 2.0.0 >Reporter: Zhan Zhang >Assignee: Theodore michael Malaska >Priority: Blocker > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-14406.1.patch, HBASE-14406.10.patch, > HBASE-14406.11.patch, HBASE-14406.2.patch, HBASE-14406.3.patch, > HBASE-14406.4.patch, HBASE-14406.5.patch, HBASE-14406.6.patch, > HBASE-14406.7.patch, HBASE-14406.9.patch > > > Following condition will result in the same filter. It will have data loss > with the current filter construction. > col1 > 4 && col2 < 3 > col1 > 4 || col2 < 3 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-14216) Consolidate MR and Spark BulkLoad shared functions and string consts
[ https://issues.apache.org/jira/browse/HBASE-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-14216. - Assignee: (was: Theodore michael Malaska) Resolution: Incomplete > Consolidate MR and Spark BulkLoad shared functions and string consts > > > Key: HBASE-14216 > URL: https://issues.apache.org/jira/browse/HBASE-14216 > Project: HBase > Issue Type: Improvement >Reporter: Theodore michael Malaska >Priority: Minor > > This is a follow up jira is HBASE-14150. Andrew P had noticed code that > could be consolidate between MR and Spark bulk load code. > Before I get started I need advice to know where the consolidated code should > live. Once I have the location I can start coding. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13978) Variable never assigned in SimpleTotalOrderPartitioner.getPartition()
[
https://issues.apache.org/jira/browse/HBASE-13978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell closed HBASE-13978.
---
> Variable never assigned in SimpleTotalOrderPartitioner.getPartition()
> --
>
> Key: HBASE-13978
> URL: https://issues.apache.org/jira/browse/HBASE-13978
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.1.0.1
>Reporter: Lars George
>Assignee: Bhupendra Kumar Jain
>Priority: Major
> Labels: beginner
> Fix For: 0.98.14, 1.2.0, 2.0.0
>
> Attachments:
> 0001-HBASE-13978-Variable-never-assigned-in-SimpleTotalOr.patch
>
>
> See
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SimpleTotalOrderPartitioner.java#L104,
> which has an {{if}} statement that tries to limit the code to run only once,
> but since it does not assign {{this.lastReduces}} it will always trigger and
> recompute the splits (and log them).
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Closed] (HBASE-13985) Add configuration to skip validating HFile format when bulk loading
[ https://issues.apache.org/jira/browse/HBASE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13985. --- > Add configuration to skip validating HFile format when bulk loading > --- > > Key: HBASE-13985 > URL: https://issues.apache.org/jira/browse/HBASE-13985 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.13 >Reporter: Victor Xu >Assignee: Victor Xu >Priority: Minor > Labels: regionserver > Fix For: 0.98.14, 1.2.0, 1.3.0, 2.0.0 > > Attachments: HBASE-13985-v2.patch, HBASE-13985-v3.patch, > HBASE-13985.patch > > > When bulk loading millions of HFile into one HTable, checking HFile format is > the most time-consuming phase. Maybe we could use a parallel mechanism to > increase the speed, but when it comes to millions of HFiles, it may still > cost dozens of minutes. So I think it's necessary to add an option for > advanced user to bulkload without checking HFile format at all. > Of course, the default value of this option should be true. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13976) release manager list in ref guide is missing 0.94 line
[ https://issues.apache.org/jira/browse/HBASE-13976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13976. --- > release manager list in ref guide is missing 0.94 line > -- > > Key: HBASE-13976 > URL: https://issues.apache.org/jira/browse/HBASE-13976 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Sean Busbey >Assignee: M Linville >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-13976.patch > > > 0.94 has slowed substantially, but we haven't EOLed it yet so we should make > sure folks looking to get a fix included know where to look. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14181) Add Spark DataFrame DataSource to HBase-Spark Module
[ https://issues.apache.org/jira/browse/HBASE-14181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14181. --- > Add Spark DataFrame DataSource to HBase-Spark Module > > > Key: HBASE-14181 > URL: https://issues.apache.org/jira/browse/HBASE-14181 > Project: HBase > Issue Type: New Feature > Components: hbase-connectors, spark >Reporter: Theodore michael Malaska >Assignee: Theodore michael Malaska >Priority: Minor > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-14181.1.patch, HBASE-14181.10.patch, > HBASE-14181.11.patch, HBASE-14181.2.patch, HBASE-14181.3.patch, > HBASE-14181.4.patch, HBASE-14181.5.patch, HBASE-14181.6.patch, > HBASE-14181.7.patch, HBASE-14181.8.patch, HBASE-14181.8.patch, > HBASE-14181.9.patch > > > Build a RelationProvider for HBase-Spark Module. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-17574) Clean up how to run tests under hbase-spark module
[
https://issues.apache.org/jira/browse/HBASE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell closed HBASE-17574.
---
> Clean up how to run tests under hbase-spark module
> ---
>
> Key: HBASE-17574
> URL: https://issues.apache.org/jira/browse/HBASE-17574
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors, spark
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Fix For: 3.0.0-alpha-1, connector-1.0.0
>
> Attachments: HBase-17574-V1.patch, HBase-17574-V2.patch
>
>
> In master brunch, the test of hbase-spark module needs clean-up.
> I think we need to let hbase-spark follow the rules that exist in the whole
> hbase project
> 1. In hbase-spark, all the scala test cases are regarded as integration test,
> i.e. we need to go to hbase-spark folder to use mvn verify to run the test
> case. I think these tests had better to be regard as unit test for the
> following reasons:
> (1) All the scala test are very small, most of them can be finished within
> 20s.
> (2) Integration test usually put into hbase-it module, not in its own module.
> (3) Hadoop QA could not run those scala test in hbase-spark, I guess Hadoop
> QA will only run mvn test under root dir, however hbase-spark need mvn verify.
> (4) From its pom.xml below, you can see that, both
> integration-test and test point to same
> test. From MVN reference,
> http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Built-in_Lifecycle_Bindings,
> we know that if a goal is bound to one or more build phases, that goal will
> be called in all those phases. it means that mvn test and mvn
> integration-test will do same thing, however true in
> test phase just disable the mvn test command. It is uncommon to have define
> like that.
> {code}
>
>
> test
> test
>
> test
>
>
> true
>
>
>
> integration-test
> integration-test
>
> test
>
>
> Integration-Test
>
> -Xmx1536m -XX:MaxPermSize=512m
> -XX:ReservedCodeCacheSize=512m
>
> false
>
>
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Closed] (HBASE-15644) Add maven-scala-plugin for scaladoc
[ https://issues.apache.org/jira/browse/HBASE-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15644. --- > Add maven-scala-plugin for scaladoc > --- > > Key: HBASE-15644 > URL: https://issues.apache.org/jira/browse/HBASE-15644 > Project: HBase > Issue Type: Task > Components: build >Reporter: Sean Busbey >Assignee: Apekshit Sharma >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-15644.1.patch, bogus-scala-change.patch, > scala-tools.patch > > > Added scala-tools.org to repository (as a side effect, all common artifacts > get downloaded twice now, once from apache repo and once from scala-tools) > This fixes scala:doc precommit. > The patch 'bogus-scala-change' adds a blank line to a scala file to trigger > scala:doc precommit. As expected, the target failed for master and passed for > the patch. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-15597) Clean up configuration keys used in hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15597. --- > Clean up configuration keys used in hbase-spark module > -- > > Key: HBASE-15597 > URL: https://issues.apache.org/jira/browse/HBASE-15597 > Project: HBase > Issue Type: Bug > Components: hbase-connectors, spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Yi Liang >Priority: Critical > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-15597-V2.patch, HBASE-15597-V3.patch, > HBASE-15597-v1.patch, HBase-15597-V4.patch > > > This should be considered a blocker for backport to branch-1 since it will > impact our compatibility. > The constants we expose in configuration should all start with "hbase". Since > our configurations keys for the spark integration all relate to that system, > the prefix for all configuration keys (excluding those cases where we need to > do something special due to restrictions in how properties are handled by > e.g. spark) should be "hbase.spark". > Before publishing a public api labeled version of our spark integration we > should review all of our configuration keys to make sure they either conform > to the "hbase.spark" prefix or they have a comment documenting why they need > to be otherwise. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-15201) Add hbase-spark to hbase assembly
[ https://issues.apache.org/jira/browse/HBASE-15201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15201. --- > Add hbase-spark to hbase assembly > - > > Key: HBASE-15201 > URL: https://issues.apache.org/jira/browse/HBASE-15201 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15201.patch > > > hbase-spark currently is missing from hbase assembly. > We should add it. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-17192) remove use of scala-tools.org from pom
[
https://issues.apache.org/jira/browse/HBASE-17192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell closed HBASE-17192.
---
> remove use of scala-tools.org from pom
> --
>
> Key: HBASE-17192
> URL: https://issues.apache.org/jira/browse/HBASE-17192
> Project: HBase
> Issue Type: Bug
> Components: spark, website
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-17192.1.patch, HBASE-17192.1.test.patch
>
>
> our pom makes use of scala-tools.org for a repository. That domain currently
> issues redirects for all URLs; for maven coordinates those redirects lead to
> 'not found' and the 'permantenly moved' HTML gets saved. this corrupts the
> local maven repository in a way that cause the mvn:site goal to give an
> opaque error:
> {code}
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 01:46 min
> [INFO] Finished at: 2016-11-28T14:17:10+00:00
> [INFO] Final Memory: 292M/6583M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.4:site (default-site) on project
> hbase: Execution default-site of goal
> org.apache.maven.plugins:maven-site-plugin:3.4:site failed: For artifact
> {null:null:null:jar}: The groupId cannot be empty. -> [Help 1]
> [ERROR]
> {code}
> Rerunning in debug mode with {{mvn -X}} gives no additional useful
> information.
> All artifacts from scala-tools.org are now found in maven central.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Closed] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-16845. --- Assignee: (was: Ted Yu) > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Priority: Major > Attachments: 16845.v1.txt, 16845.v2.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-15184) SparkSQL Scan operation doesn't work on kerberos cluster
[ https://issues.apache.org/jira/browse/HBASE-15184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15184. --- > SparkSQL Scan operation doesn't work on kerberos cluster > > > Key: HBASE-15184 > URL: https://issues.apache.org/jira/browse/HBASE-15184 > Project: HBase > Issue Type: Bug > Components: hbase-connectors, spark >Reporter: Theodore michael Malaska >Assignee: Theodore michael Malaska >Priority: Critical > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-15184.1.patch, HBaseSparkModule.zip > > > I was using the HBase Spark Module at a client with Kerberos and I ran into > an issue with the Scan. > I made a fix for the client but we need to put it back into HBase. I will > attach my solution, but it has a major problem. I had to over ride a > protected class in spark. I will need help to decover a better approach -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-16246) Ensure avro version in hbase-spark matches that shipped in our default hadoop profile
[ https://issues.apache.org/jira/browse/HBASE-16246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-16246. --- > Ensure avro version in hbase-spark matches that shipped in our default hadoop > profile > - > > Key: HBASE-16246 > URL: https://issues.apache.org/jira/browse/HBASE-16246 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Blocker > > currently, the hbase-spark module pulls in avro 1.7.6. We should keep this > matching the version in our default hadoop to keep things consistent (as of > Hadoop 2.7.1 that's avro 1.7.4). -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14376) move hbase spark integration examples into their own module
[ https://issues.apache.org/jira/browse/HBASE-14376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14376. --- > move hbase spark integration examples into their own module > --- > > Key: HBASE-14376 > URL: https://issues.apache.org/jira/browse/HBASE-14376 > Project: HBase > Issue Type: Task > Components: pom, spark >Reporter: Sean Busbey >Priority: Major > Labels: beginner > Attachments: HBASE-14376.1.patch, warnings.diff > > > take the examples that are currently in the hbase-spark module and move them > into a hbase-spark-examples module. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-16242) Upgrade Avro to 1.7.7
[ https://issues.apache.org/jira/browse/HBASE-16242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-16242. --- > Upgrade Avro to 1.7.7 > - > > Key: HBASE-16242 > URL: https://issues.apache.org/jira/browse/HBASE-16242 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: Ben McCann >Assignee: Sean Busbey >Priority: Major > Fix For: 1.4.0, 2.0.0-alpha-2, 2.0.0 > > Attachments: HBASE-16242.0.patch > > > I'd like to see Avro upgraded to 1.8.1 or at the very least 1.7.7 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14375) Define public API for spark integration module
[ https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14375. --- > Define public API for spark integration module > -- > > Key: HBASE-14375 > URL: https://issues.apache.org/jira/browse/HBASE-14375 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Jerry He >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-14375-v1.patch, HBASE-14375-v2.patch > > > before we can put the spark integration module into a release, we need to > annotate its public api surface. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests
[
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell closed HBASE-14167.
---
> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
> Issue Type: Improvement
> Components: pom, spark
>Affects Versions: 2.0.0
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Attachments: HBASE-14167-master-v2.patch, HBASE-14167.11.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark
> module's integration tests do not respect the flag and run anyway. Fix.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Closed] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job
[ https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14161. --- > Add hbase-spark integration tests to IT jenkins job > --- > > Key: HBASE-14161 > URL: https://issues.apache.org/jira/browse/HBASE-14161 > Project: HBase > Issue Type: Task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0-alpha-1 > > > expand the set of ITs we run to include the new hbase-spark tests. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-15271) Spark Bulk Load: Need to write HFiles to tmp location then rename to protect from Spark Executor Failures
[ https://issues.apache.org/jira/browse/HBASE-15271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15271. --- > Spark Bulk Load: Need to write HFiles to tmp location then rename to protect > from Spark Executor Failures > - > > Key: HBASE-15271 > URL: https://issues.apache.org/jira/browse/HBASE-15271 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Theodore michael Malaska >Assignee: Theodore michael Malaska >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-15271.1.patch, HBASE-15271.2.patch, > HBASE-15271.3.patch, HBASE-15271.4.patch > > > With the current code if an executor failure before the HFile is close it > will cause problems. This jira will have the files first write out to a file > that starts with an underscore. Then when the HFile is complete it will be > renamed and the underscore will be removed. > The underscore is important because the load bulk functionality will skip > files with an underscore. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14159) Resolve warning introduced by HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14159. --- > Resolve warning introduced by HBase-Spark module > > > Key: HBASE-14159 > URL: https://issues.apache.org/jira/browse/HBASE-14159 > Project: HBase > Issue Type: Improvement > Components: build, hbase-connectors, spark >Reporter: Theodore michael Malaska >Assignee: Apekshit Sharma >Priority: Minor > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-14159-master-v1.patch > > > Fix the following warning that is a result of something in the modules pom > file > [WARNING] warning: Class org.apache.hadoop.mapred.MiniMRCluster not found - > continuing with a stub. > [WARNING] one warning found -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-15350) Enable individual unit test in hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15350. --- Assignee: (was: Zhan Zhang) > Enable individual unit test in hbase-spark module > - > > Key: HBASE-15350 > URL: https://issues.apache.org/jira/browse/HBASE-15350 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Priority: Major > Attachments: HBASE-15350-1.patch, HBASE-15350-2.patch > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-15333) [hbase-spark] Enhance dataframe filters to handle naively encoded short, integer, long, float and double
[ https://issues.apache.org/jira/browse/HBASE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15333. --- > [hbase-spark] Enhance dataframe filters to handle naively encoded short, > integer, long, float and double > > > Key: HBASE-15333 > URL: https://issues.apache.org/jira/browse/HBASE-15333 > Project: HBase > Issue Type: Sub-task > Components: hbase-connectors, spark >Reporter: Zhan Zhang >Assignee: Zhan Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-15333-1.patch, HBASE-15333-10.patch, > HBASE-15333-2.patch, HBASE-15333-3.patch, HBASE-15333-4.patch, > HBASE-15333-5.patch, HBASE-15333-6.patch, HBASE-15333-7.patch, > HBASE-15333-8.patch, HBASE-15333-9.patch > > > Currently, the range filter is based on the order of bytes. But for java > primitive type, such as short, int, long, double, float, etc, their order is > not consistent with their byte order, extra manipulation has to be in place > to take care of them correctly. > For example, for the integer range (-100, 100), the filter <= 1, the current > filter will return 0 and 1, and the right return value should be (-100, 1] -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14789) Enhance the current spark-hbase connector
[ https://issues.apache.org/jira/browse/HBASE-14789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14789. --- > Enhance the current spark-hbase connector > - > > Key: HBASE-14789 > URL: https://issues.apache.org/jira/browse/HBASE-14789 > Project: HBase > Issue Type: Improvement > Components: hbase-connectors, spark >Reporter: Zhan Zhang >Assignee: Zhan Zhang >Priority: Major > Labels: spark > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: shc.pdf > > > This JIRA is to optimize the RDD construction in the current connector > implementation. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-15334) Add avro support for spark hbase connector
[ https://issues.apache.org/jira/browse/HBASE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15334. --- > Add avro support for spark hbase connector > -- > > Key: HBASE-15334 > URL: https://issues.apache.org/jira/browse/HBASE-15334 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-15334-1.patch, HBASE-15334-2.patch, > HBASE-15334-3.patch, HBASE-15334-4.patch, HBASE-15334-5.patch > > > Avro is a popular format for hbase storage. User may want the support > natively in the connector. > With the support, user can save serialized avro into hbase table, and then > query on top of it using spark sql. The conversion between avro and catalyst > datatype will be handled automatically. This is one way of support complex > data types. Otherwise, user has to define their own customized Serdes to > support complex data types. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-15473) Documentation for the usage of hbase dataframe user api (JSON, Avro, etc)
[ https://issues.apache.org/jira/browse/HBASE-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15473. --- > Documentation for the usage of hbase dataframe user api (JSON, Avro, etc) > - > > Key: HBASE-15473 > URL: https://issues.apache.org/jira/browse/HBASE-15473 > Project: HBase > Issue Type: Sub-task > Components: documentation, hbase-connectors, spark >Reporter: Zhan Zhang >Assignee: Weiqing Yang >Priority: Blocker > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-15473_v1.patch > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-15336) Support Dataframe writer to the spark connector
[ https://issues.apache.org/jira/browse/HBASE-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15336. --- > Support Dataframe writer to the spark connector > --- > > Key: HBASE-15336 > URL: https://issues.apache.org/jira/browse/HBASE-15336 > Project: HBase > Issue Type: Sub-task > Components: hbase-connectors, spark >Reporter: Zhan Zhang >Assignee: Zhan Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-15336-1.patch, HBASE-15336-2.patch, > HBASE-15336-3.patch > > > Currently, the connector only support read path. A complete solution should > support both read and writer. This subtask add write support. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-15825) Fix the null pointer in DynamicLogicExpressionSuite
[ https://issues.apache.org/jira/browse/HBASE-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-15825. --- > Fix the null pointer in DynamicLogicExpressionSuite > --- > > Key: HBASE-15825 > URL: https://issues.apache.org/jira/browse/HBASE-15825 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-15825-1.patch > > > It only happens in test cases. Not sure why it is not caught. Will submit > patch soon -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13927) Allow hbase-daemon.sh to conditionally redirect the log or not
[ https://issues.apache.org/jira/browse/HBASE-13927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13927. --- > Allow hbase-daemon.sh to conditionally redirect the log or not > -- > > Key: HBASE-13927 > URL: https://issues.apache.org/jira/browse/HBASE-13927 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 1.2.0, 2.0.0 >Reporter: Elliott Neil Clark >Assignee: Elliott Neil Clark >Priority: Major > Labels: shell > Fix For: 1.2.0, 2.0.0 > > Attachments: HBASE-13927.patch, HBASE-13927.patch > > > Kind of like HBASE_NOEXEC allow hbase-daemon to skip redirecting to a log. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14801) Enhance the Spark-HBase connector catalog with json format
[ https://issues.apache.org/jira/browse/HBASE-14801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14801. --- > Enhance the Spark-HBase connector catalog with json format > -- > > Key: HBASE-14801 > URL: https://issues.apache.org/jira/browse/HBASE-14801 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang >Priority: Major > Attachments: HBASE-14801-1.patch, HBASE-14801-2.patch, > HBASE-14801-3.patch, HBASE-14801-4.patch, HBASE-14801-5.patch, > HBASE-14801-6.patch, HBASE-14801-7.patch, HBASE-14801-8.patch > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14160) backport hbase-spark module to branch-1 and branch-2
[ https://issues.apache.org/jira/browse/HBASE-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14160. --- > backport hbase-spark module to branch-1 and branch-2 > > > Key: HBASE-14160 > URL: https://issues.apache.org/jira/browse/HBASE-14160 > Project: HBase > Issue Type: New Feature > Components: spark >Affects Versions: 1.3.0 >Reporter: Sean Busbey >Priority: Major > > Once the hbase-spark module gets polished a bit, we should backport to > branch-1 and branch-2 so we can publish it sooner. > needs (per previous discussion): > * examples refactored into different module > * user facing documentation > * defined public API -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-17933) [hbase-spark] Support Java api for bulkload
[ https://issues.apache.org/jira/browse/HBASE-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-17933. --- > [hbase-spark] Support Java api for bulkload > > > Key: HBASE-17933 > URL: https://issues.apache.org/jira/browse/HBASE-17933 > Project: HBase > Issue Type: New Feature > Components: hbase-connectors, spark >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Major > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBase-17933-V1.patch, HBase-17933-V2.patch, > HBase-17933-V3.patch, HBase-17933-V4.patch > > > In JavaHBaseContext, there are java api for bulkPut, bulkDelete , but no > Java api for bulkload. And this jira will add bulkload java api to hbase-spark -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14217) Add Java access to Spark bulk load functionality
[ https://issues.apache.org/jira/browse/HBASE-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14217. --- > Add Java access to Spark bulk load functionality > > > Key: HBASE-14217 > URL: https://issues.apache.org/jira/browse/HBASE-14217 > Project: HBase > Issue Type: Improvement >Reporter: Theodore michael Malaska >Priority: Minor > > HBASE-14150 added bulk load functionality for Scala users. This jira will > add the Java layer that will make this functionality accessable to Java > developers. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Reopened] (HBASE-14217) Add Java access to Spark bulk load functionality
[ https://issues.apache.org/jira/browse/HBASE-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reopened HBASE-14217: - > Add Java access to Spark bulk load functionality > > > Key: HBASE-14217 > URL: https://issues.apache.org/jira/browse/HBASE-14217 > Project: HBase > Issue Type: Improvement >Reporter: Theodore michael Malaska >Priority: Minor > > HBASE-14150 added bulk load functionality for Scala users. This jira will > add the Java layer that will make this functionality accessable to Java > developers. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14158) Add documentation for Initial Release for HBase-Spark Module integration
[ https://issues.apache.org/jira/browse/HBASE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14158. --- > Add documentation for Initial Release for HBase-Spark Module integration > - > > Key: HBASE-14158 > URL: https://issues.apache.org/jira/browse/HBASE-14158 > Project: HBase > Issue Type: Improvement > Components: documentation, hbase-connectors, spark >Reporter: Theodore michael Malaska >Assignee: Theodore michael Malaska >Priority: Major > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-14158-v8.patch, HBASE-14158.1.patch, > HBASE-14158.2.patch, HBASE-14158.5.patch, HBASE-14158.5.patch, > HBASE-14158.6.patch, HBASE-14158.7.patch > > > Add documentation for Initial Release for HBase-Spark Module integration -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14150) Add BulkLoad functionality to HBase-Spark Module
[ https://issues.apache.org/jira/browse/HBASE-14150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14150. --- > Add BulkLoad functionality to HBase-Spark Module > > > Key: HBASE-14150 > URL: https://issues.apache.org/jira/browse/HBASE-14150 > Project: HBase > Issue Type: New Feature > Components: hbase-connectors, spark >Reporter: Theodore michael Malaska >Assignee: Theodore michael Malaska >Priority: Major > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-14150.1.patch, HBASE-14150.2.patch, > HBASE-14150.3.patch, HBASE-14150.4.patch, HBASE-14150.5.patch > > > Add on to the work done in HBASE-13992 to add functionality to do a bulk load > from a given RDD. > This will do the following: > 1. figure out the number of regions and sort and partition the data correctly > to be written out to HFiles > 2. Also unlike the MR bulkload I would like that the columns to be sorted in > the shuffle stage and not in the memory of the reducer. This will allow this > design to support super wide records with out going out of memory. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14340) Add second bulk load option to Spark Bulk Load to send puts as the value
[ https://issues.apache.org/jira/browse/HBASE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14340. --- > Add second bulk load option to Spark Bulk Load to send puts as the value > > > Key: HBASE-14340 > URL: https://issues.apache.org/jira/browse/HBASE-14340 > Project: HBase > Issue Type: New Feature > Components: hbase-connectors, spark >Reporter: Theodore michael Malaska >Assignee: Theodore michael Malaska >Priority: Minor > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-14340.1.patch, HBASE-14340.2.patch > > > The initial bulk load option for Spark bulk load sends values over one by one > through the shuffle. This is the similar to how the original MR bulk load > worked. > How ever the MR bulk loader have more then one bulk load option. There is a > second option that allows for all the Column Families, Qualifiers, and Values > or a row to be combined in the map side. > This only works if the row is not super wide. > But if the row is not super wide this method of sending values through the > shuffle will reduce the data and work the shuffle has to deal with. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14149) Add Data Frame support for HBase-Spark Module
[ https://issues.apache.org/jira/browse/HBASE-14149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14149. --- Assignee: (was: Theodore michael Malaska) > Add Data Frame support for HBase-Spark Module > - > > Key: HBASE-14149 > URL: https://issues.apache.org/jira/browse/HBASE-14149 > Project: HBase > Issue Type: New Feature > Components: spark >Reporter: Theodore michael Malaska >Priority: Major > > Add on to the work done in HBASE-13992 and add support for dataframes for > bulk puts, bulk gets, and scans -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-14217) Add Java access to Spark bulk load functionality
[ https://issues.apache.org/jira/browse/HBASE-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-14217. - Resolution: Implemented > Add Java access to Spark bulk load functionality > > > Key: HBASE-14217 > URL: https://issues.apache.org/jira/browse/HBASE-14217 > Project: HBase > Issue Type: Improvement >Reporter: Theodore michael Malaska >Priority: Minor > > HBASE-14150 added bulk load functionality for Scala users. This jira will > add the Java layer that will make this functionality accessable to Java > developers. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-20570) CLONE - Integrate SparkOnHBase into HBase
[ https://issues.apache.org/jira/browse/HBASE-20570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-20570. --- Assignee: (was: Theodore michael Malaska) > CLONE - Integrate SparkOnHBase into HBase > - > > Key: HBASE-20570 > URL: https://issues.apache.org/jira/browse/HBASE-20570 > Project: HBase > Issue Type: New Feature > Components: spark >Reporter: ujjawal kumar >Priority: Major > > This Jira is to ask if SparkOnHBase can find a home in side HBase core. > Here is the github: > https://github.com/cloudera-labs/SparkOnHBase > I am the core author of this project and the license is Apache 2.0 > A blog explaining this project is here > http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/ > A spark Streaming example is here > http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ > A real customer using this in produce is blogged here > http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/ > Please debate and let me know what I can do to make this happen. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14216) Consolidate MR and Spark BulkLoad shared functions and string consts
[ https://issues.apache.org/jira/browse/HBASE-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14216. --- > Consolidate MR and Spark BulkLoad shared functions and string consts > > > Key: HBASE-14216 > URL: https://issues.apache.org/jira/browse/HBASE-14216 > Project: HBase > Issue Type: Improvement >Reporter: Theodore michael Malaska >Priority: Minor > > This is a follow up jira is HBASE-14150. Andrew P had noticed code that > could be consolidate between MR and Spark bulk load code. > Before I get started I need advice to know where the consolidated code should > live. Once I have the location I can start coding. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13992) Integrate SparkOnHBase into HBase
[ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13992. --- > Integrate SparkOnHBase into HBase > - > > Key: HBASE-13992 > URL: https://issues.apache.org/jira/browse/HBASE-13992 > Project: HBase > Issue Type: New Feature > Components: hbase-connectors, spark >Reporter: Theodore michael Malaska >Assignee: Theodore michael Malaska >Priority: Major > Fix For: 3.0.0-alpha-1, connector-1.0.0 > > Attachments: HBASE-13992.10.patch, HBASE-13992.11.patch, > HBASE-13992.12.patch, HBASE-13992.5.patch, HBASE-13992.6.patch, > HBASE-13992.7.patch, HBASE-13992.8.patch, HBASE-13992.9.patch, > HBASE-13992.patch, HBASE-13992.patch.3, HBASE-13992.patch.4, > HBASE-13992.patch.5 > > > This Jira is to ask if SparkOnHBase can find a home in side HBase core. > Here is the github: > https://github.com/cloudera-labs/SparkOnHBase > I am the core author of this project and the license is Apache 2.0 > A blog explaining this project is here > http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/ > A spark Streaming example is here > http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ > A real customer using this in produce is blogged here > http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/ > Please debate and let me know what I can do to make this happen. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14000) Region server failed to report to Master and was stuck in reportForDuty retry loop
[
https://issues.apache.org/jira/browse/HBASE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell closed HBASE-14000.
---
> Region server failed to report to Master and was stuck in reportForDuty retry
> loop
> --
>
> Key: HBASE-14000
> URL: https://issues.apache.org/jira/browse/HBASE-14000
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.2.0, 1.1.2, 1.3.0, 2.0.0
>
> Attachments: HBASE-14000.patch, HM_RS-Log_snippet.txt
>
>
> In a HA cluster, region server got stuck in reportForDuty retry loop if the
> active master is restarting and later on master switch happens before it
> reports successfully.
> Root cause is same as HBASE-13317, but the region server tried to connect
> master when it was starting, so rssStub reset didnt happen as
> {code}
> if (ioe instanceof ServerNotRunningYetException) {
> LOG.debug("Master is not running yet");
> }
> {code}
> When master starts, master switch happened. So RS always tried to connect to
> standby master.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Closed] (HBASE-13997) ScannerCallableWithReplicas cause Infinitely blocking
[
https://issues.apache.org/jira/browse/HBASE-13997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell closed HBASE-13997.
---
> ScannerCallableWithReplicas cause Infinitely blocking
> -
>
> Key: HBASE-13997
> URL: https://issues.apache.org/jira/browse/HBASE-13997
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.0.1.1
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Minor
> Fix For: 1.2.0, 1.1.2, 1.3.0, 2.0.0
>
> Attachments: HBASE-13997.patch, hbase-13997_v2.patch
>
>
> Bug in ScannerCallableWithReplicas.addCallsForOtherReplicas method
> {code:title=code in ScannerCallableWithReplicas.addCallsForOtherReplicas
> |borderStyle=solid}
> private int addCallsForOtherReplicas(
> BoundedCompletionService> cs,
> RegionLocations rl, int min,
> int max) {
> if (scan.getConsistency() == Consistency.STRONG) {
> return 0; // not scheduling on other replicas for strong consistency
> }
> for (int id = min; id <= max; id++) {
> if (currentScannerCallable.getHRegionInfo().getReplicaId() == id) {
> continue; //this was already scheduled earlier
> }
> ScannerCallable s =
> currentScannerCallable.getScannerCallableForReplica(id);
> if (this.lastResult != null) {
> s.getScan().setStartRow(this.lastResult.getRow());
> }
> outstandingCallables.add(s);
> RetryingRPC retryingOnReplica = new RetryingRPC(s);
> cs.submit(retryingOnReplica);
> }
> return max - min + 1; //bug? should be "max - min",because "continue"
> //always happen once
> }
> {code}
> It can cause completed < submitted always so that the following code will be
> infinitely blocked.
> {code:title=code in ScannerCallableWithReplicas.call|borderStyle=solid}
> // submitted larger than the actual one
> submitted += addCallsForOtherReplicas(cs, rl, 0, rl.size() - 1);
> try {
> //here will be affected
> while (completed < submitted) {
> try {
> Future> f = cs.take();
> Pair r = f.get();
> if (r != null && r.getSecond() != null) {
> updateCurrentlyServingReplica(r.getSecond(), r.getFirst(), done,
> pool);
> }
> return r == null ? null : r.getFirst(); // great we got an answer
> } catch (ExecutionException e) {
> // if not cancel or interrupt, wait until all RPC's are done
> // one of the tasks failed. Save the exception for later.
> if (exceptions == null) exceptions = new
> ArrayList(rl.size());
> exceptions.add(e);
> completed++;
> }
> }
> } catch (CancellationException e) {
> throw new InterruptedIOException(e.getMessage());
> } catch (InterruptedException e) {
> throw new InterruptedIOException(e.getMessage());
> } finally {
> // We get there because we were interrupted or because one or more of
> the
> // calls succeeded or failed. In all case, we stop all our tasks.
> cs.cancelAll(true);
> }
> {code}
> If all replica-RS occur ExecutionException ,it will be infinitely blocked in
> cs.take()
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Closed] (HBASE-13996) Add write sniffing in canary
[ https://issues.apache.org/jira/browse/HBASE-13996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13996. --- > Add write sniffing in canary > > > Key: HBASE-13996 > URL: https://issues.apache.org/jira/browse/HBASE-13996 > Project: HBase > Issue Type: Improvement > Components: canary >Affects Versions: 0.98.13, 1.1.0.1 >Reporter: Shaohui Liu >Assignee: Shaohui Liu >Priority: Major > Fix For: 0.98.14, 1.2.0, 1.3.0, 2.0.0 > > Attachments: HBASE-13996-0.98.patch, HBASE-13996-branch-1.patch, > HBASE-13996-v001.diff, HBASE-13996-v002.diff, HBASE-13996-v003.diff, > HBASE-13996-v004.diff > > > Currently the canary tool only sniff the read operations, it's hard to > finding the problem in write path. > To support the write sniffing, we create a system table named '_canary_' in > the canary tool. And the tool will make sure that the region number is large > than the number of the regionserver and the regions will be distributed onto > all regionservers. > Periodically, the tool will put data to these regions to calculate the write > availability of HBase and send alerts if needed. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14001) Optimize write(OutputStream out, boolean withTags) for SizeCachedNoTagsKeyValue
[ https://issues.apache.org/jira/browse/HBASE-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14001. --- > Optimize write(OutputStream out, boolean withTags) for > SizeCachedNoTagsKeyValue > --- > > Key: HBASE-14001 > URL: https://issues.apache.org/jira/browse/HBASE-14001 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14001.patch > > > We can override this method in SizeCachedNoTagsKeyValue. KeyValue impl do > value length parsing and do some maths to find length with out tags length. > In SizeCachedNoTagsKeyValue we know there are no tags. Same optimize we did > in NoTagsKeyValue. But I missed to do it in SizeCachedNoTagsKeyValue. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13866) Add endpoint coprocessor to the section hbase.coprocessor.region.classes in HBase book
[
https://issues.apache.org/jira/browse/HBASE-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell closed HBASE-13866.
---
> Add endpoint coprocessor to the section hbase.coprocessor.region.classes in
> HBase book
> --
>
> Key: HBASE-13866
> URL: https://issues.apache.org/jira/browse/HBASE-13866
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: M Linville
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-13866.patch
>
>
> {quote}
> hbase.coprocessor.region.classes
> Description
> A comma-separated list of Coprocessors that are loaded by default on all
> tables. For any override coprocessor method, these classes will be called in
> order. After implementing your own Coprocessor, just put it in HBase’s
> classpath and add the fully qualified class name here. A coprocessor can also
> be loaded on demand by setting HTableDescriptor.
> {quote}
> This must be more specific: not Coprocessors, but Region observers and
> *endpoint coprocessors*.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Closed] (HBASE-13965) Stochastic Load Balancer JMX Metrics
[ https://issues.apache.org/jira/browse/HBASE-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13965. --- > Stochastic Load Balancer JMX Metrics > > > Key: HBASE-13965 > URL: https://issues.apache.org/jira/browse/HBASE-13965 > Project: HBase > Issue Type: Improvement > Components: Balancer, metrics >Reporter: Lei Chen >Assignee: Lei Chen >Priority: Major > Fix For: 1.3.0, 2.0.0 > > Attachments: 13965-addendum.txt, HBASE-13965-branch-1-v2.patch, > HBASE-13965-branch-1.patch, HBASE-13965-v10.patch, HBASE-13965-v11.patch, > HBASE-13965-v3.patch, HBASE-13965-v4.patch, HBASE-13965-v5.patch, > HBASE-13965-v6.patch, HBASE-13965-v7.patch, HBASE-13965-v8.patch, > HBASE-13965-v9.patch, HBASE-13965_v2.patch, HBase-13965-JConsole.png, > HBase-13965-v1.patch, stochasticloadbalancerclasses_v2.png > > > Today’s default HBase load balancer (the Stochastic load balancer) is cost > function based. The cost function weights are tunable but no visibility into > those cost function results is directly provided. > A driving example is a cluster we have been tuning which has skewed rack size > (one rack has half the nodes of the other few racks). We are tuning the > cluster for uniform response time from all region servers with the ability to > tolerate a rack failure. Balancing LocalityCost, RegionReplicaRack Cost and > RegionCountSkew Cost is difficult without a way to attribute each cost > function’s contribution to overall cost. > What this jira proposes is to provide visibility via JMX into each cost > function of the stochastic load balancer, as well as the overall cost of the > balancing plan. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13949) Expand hadoop2 versions built on the pre-commit
[ https://issues.apache.org/jira/browse/HBASE-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13949. --- Assignee: (was: Nick Dimiduk) > Expand hadoop2 versions built on the pre-commit > --- > > Key: HBASE-13949 > URL: https://issues.apache.org/jira/browse/HBASE-13949 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Priority: Major > Fix For: 2.0.0 > > > For the HBase 1.1 line I've been validating builds against the following > hadoop versions: 2.2.0 2.3.0 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0. Let's > do the same in pre-commit. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13999) Improve storefile split performance by creating daughter reference files concurrently
[ https://issues.apache.org/jira/browse/HBASE-13999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13999. --- > Improve storefile split performance by creating daughter reference files > concurrently > - > > Key: HBASE-13999 > URL: https://issues.apache.org/jira/browse/HBASE-13999 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.98.12 >Reporter: Hari Krishna Dara >Priority: Minor > Labels: performance, split > > The fix for HBASE-13959 improves the concurrency of region split operation, > and also identifies another optimization by extending the concurrency to the > creation of individual reference files. This jira aims to do just that! Since > creation of reference files take time in the order of 300 to 400ms, this has > the potential of reducing the split time by as much in most common scenarios. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-13999) Improve storefile split performance by creating daughter reference files concurrently
[ https://issues.apache.org/jira/browse/HBASE-13999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-13999. - Assignee: (was: Hari Krishna Dara) Resolution: Incomplete > Improve storefile split performance by creating daughter reference files > concurrently > - > > Key: HBASE-13999 > URL: https://issues.apache.org/jira/browse/HBASE-13999 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.98.12 >Reporter: Hari Krishna Dara >Priority: Minor > Labels: performance, split > > The fix for HBASE-13959 improves the concurrency of region split operation, > and also identifies another optimization by extending the concurrency to the > creation of individual reference files. This jira aims to do just that! Since > creation of reference files take time in the order of 300 to 400ms, this has > the potential of reducing the split time by as much in most common scenarios. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-14002) Add --noReplicationSetup option to IntegrationTestReplication
[ https://issues.apache.org/jira/browse/HBASE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-14002. --- > Add --noReplicationSetup option to IntegrationTestReplication > - > > Key: HBASE-14002 > URL: https://issues.apache.org/jira/browse/HBASE-14002 > Project: HBase > Issue Type: Improvement > Components: integration tests >Reporter: Dima Spivak >Assignee: Dima Spivak >Priority: Major > Fix For: 0.98.14, 1.2.0, 1.1.2, 1.3.0, 2.0.0 > > Attachments: HBASE-14002_master.patch > > > IntegrationTestReplication has been flaky for me on pre-1.1 versions of HBase > because of not-actually-synchronous operations in HBaseAdmin/Admin, which > hamper its setupTablesAndReplication method. To get around this, I'd like to > add a \-nrs/--noReplicationSetup option to the test to allow it to be run on > clusters in which the necessary tables and replication have already been > setup. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13966) Limit column width in table.jsp
[ https://issues.apache.org/jira/browse/HBASE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13966. --- > Limit column width in table.jsp > --- > > Key: HBASE-13966 > URL: https://issues.apache.org/jira/browse/HBASE-13966 > Project: HBase > Issue Type: Bug > Components: Operability, UI >Affects Versions: 1.1.0.1 >Reporter: Jean-Marc Spaggiari >Assignee: Matt Warhaftig >Priority: Minor > Labels: beginner > Fix For: 1.2.0, 1.1.2, 1.3.0, 2.0.0 > > Attachments: HBASE-13966_post.tiff, HBASE-13966_pre.tiff, > hbase-13966-v1.patch > > > In table.jsp, for tables with very wide keys like URLs, the page can be very > wide. On my own cluster, this page is 8 screens wide, almost un-usable. > Might be good to have a way to resize the columns, or wrap the long keys or > truncate them, or anything else. When we want to look at the last colums, > this is a big difficult. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13962) Invalid HFile block magic
[ https://issues.apache.org/jira/browse/HBASE-13962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13962. --- > Invalid HFile block magic > - > > Key: HBASE-13962 > URL: https://issues.apache.org/jira/browse/HBASE-13962 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.12.1 > Environment: hadoop 1.2.1 > hbase 0.98.12.1 > jdk 1.7.0.79 > os : ubuntu 12.04.1 amd64 >Reporter: reaz hedayati >Priority: Major > > hi every body > my table has some cell that load with bulk load scenario and some cells for > increment. > we use 2 job to load data into table, first job use increment in reduce site > and second job use bulk load. > first we run increment job, next run bulk job and run completebulkload job, > after that we got this exception: > 2015-06-24 17:40:01,557 INFO > [regionserver60020-smallCompactions-1434448531302] regionserver.HRegion: > Starting compaction on c2 in region table1,\x04C#P1"\x07\x94 > ,1435065082383.0fe38a6c782600e4d46f1f148144b489. > 2015-06-24 17:40:01,558 INFO > [regionserver60020-smallCompactions-1434448531302] regionserver.HStore: > Starting compaction of 3 file(s) in c2 of table1,\x04C#P1"\x07\x94 > ,1435065082383.0fe38a6c782600e4d46f1f148144b489. into > tmpdir=hdfs://m2/hbase2/data/default/table1/0fe38a6c782600e4d46f1f148144b489/.tmp, > totalSize=43.1m > 2015-06-24 17:40:01,558 DEBUG > [regionserver60020-smallCompactions-1434448531302] > regionserver.StoreFileInfo: reference > 'hdfs://m2/hbase2/data/default/table1/0fe38a6c782600e4d46f1f148144b489/c2/6b1249a3b474474db5cf6c664f2d98dc.d21f8ee8b3c915fd9e1c143a0f1892e5' > to region=d21f8ee8b3c915fd9e1c143a0f1892e5 > hfile=6b1249a3b474474db5cf6c664f2d98dc > 2015-06-24 17:40:01,558 DEBUG > [regionserver60020-smallCompactions-1434448531302] compactions.Compactor: > Compacting > hdfs://m2/hbase2/data/default/table1/0fe38a6c782600e4d46f1f148144b489/c2/6b1249a3b474474db5cf6c664f2d98dc.d21f8ee8b3c915fd9e1c143a0f1892e5-hdfs://m2/hbase2/data/default/table1/d21f8ee8b3c915fd9e1c143a0f1892e5/c2/6b1249a3b474474db5cf6c664f2d98dc-top, > keycount=575485, bloomtype=ROW, size=20.8m, encoding=NONE, seqNum=9, > earliestPutTs=1434875448405 > 2015-06-24 17:40:01,558 DEBUG > [regionserver60020-smallCompactions-1434448531302] compactions.Compactor: > Compacting > hdfs://m2/hbase2/data/default/table1/0fe38a6c782600e4d46f1f148144b489/c2/41e13b20ee79435ebc260d11d3bf9920_SeqId_11_, > keycount=562988, bloomtype=ROW, size=10.1m, encoding=NONE, seqNum=11, > earliestPutTs=1435076732205 > 2015-06-24 17:40:01,558 DEBUG > [regionserver60020-smallCompactions-1434448531302] compactions.Compactor: > Compacting > hdfs://m2/hbase2/data/default/table1/0fe38a6c782600e4d46f1f148144b489/c2/565c45ff05b14a419978834c86defa1a_SeqId_12_, > keycount=554577, bloomtype=ROW, size=12.2m, encoding=NONE, seqNum=12, > earliestPutTs=1435136926850 > 2015-06-24 17:40:01,560 ERROR > [regionserver60020-smallCompactions-1434448531302] > regionserver.CompactSplitThread: Compaction failed Request = > regionName=table1,\x04C#P1"\x07\x94 > ,1435065082383.0fe38a6c782600e4d46f1f148144b489., storeName=c2, fileCount=3, > fileSize=43.1m (20.8m, 10.1m, 12.2m), priority=1, time=6077271921381072 > java.io.IOException: Could not seek > StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@1d1eb574, > cur=null] to key /c2:/LATEST_TIMESTAMP/DeleteFamily/vlen=0/mvcc=0 > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:164) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:252) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:214) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:299) > at > org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:87) > at > org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:112) > at > org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1113) > at > org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1519) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:498) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: Failed to read compressed block at 10930320, > onDiskSizeWithoutHeader=22342,
[jira] [Closed] (HBASE-13951) hbase regionserver shunt down
[ https://issues.apache.org/jira/browse/HBASE-13951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13951. --- > hbase regionserver shunt down > -- > > Key: HBASE-13951 > URL: https://issues.apache.org/jira/browse/HBASE-13951 > Project: HBase > Issue Type: Bug >Reporter: longzai >Priority: Major > > [main] util.DirectMemoryUtils: Failed to retrieve nio.BufferPool direct > MemoryUsed attribute. > javax.management.InstanceNotFoundException: > java.nio:type=BufferPool,name=direct > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) > at > org.apache.hadoop.hbase.util.DirectMemoryUtils.(DirectMemoryUtils.java:72) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:369) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.(CacheConfig.java:166) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:576) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2323) > at > org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61) > at > org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2340) > 2015-06-18 20:42:40,748 INFO [main] hfile.CacheConfig: Allocating > LruBlockCache with maximum size 398.4 M -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13948) Expand hadoop2 versions built on the pre-commit
[ https://issues.apache.org/jira/browse/HBASE-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13948. --- > Expand hadoop2 versions built on the pre-commit > --- > > Key: HBASE-13948 > URL: https://issues.apache.org/jira/browse/HBASE-13948 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 2.0.0 > > Attachments: 13948.patch, HBASE-13948-addendum.patch, > HBASE-13948.01.patch > > > For the HBase 1.1 line I've been validating builds against the following > hadoop versions: 2.2.0 2.3.0 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0. Let's > do the same in pre-commit. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13960) HConnection stuck with UnknownHostException
[
https://issues.apache.org/jira/browse/HBASE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell closed HBASE-13960.
---
Assignee: (was: Yu Li)
> HConnection stuck with UnknownHostException
>
>
> Key: HBASE-13960
> URL: https://issues.apache.org/jira/browse/HBASE-13960
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.8
>Reporter: Kurt Young
>Priority: Major
> Attachments: HBASE-13960-0.98-v1.patch, HBASE-13960-update.patch,
> HBASE-13960-update.v2.patch, HBASE-13960-v2.patch
>
>
> when put/get from hbase, if we meet a temporary dns failure causes resolve
> RS's host, the error will never recovered. put/get will failed with
> UnknownHostException forever.
> I checked the code, and the reason maybe:
> 1. when RegionServerCallable or MultiServerCallable prepare(), it gets a
> ClientService.BlockingInterface stub from Hconnection
> 2. In HConnectionImplementation::getClient, it caches the stub with a
> BlockingRpcChannelImplementation
> 3. In BlockingRpcChannelImplementation(),
> this.isa = new InetSocketAddress(sn.getHostname(), sn.getPort()); If we
> meet a temporary dns failure then the "address" in isa will be null.
> 4. then we launch the real rpc call, the following stack is:
> Caused by: java.net.UnknownHostException: unknown host: xxx.host2
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.(RpcClient.java:385)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:351)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1523)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1435)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
> Besides, i noticed there is a protection in RpcClient:
> if (remoteId.getAddress().isUnresolved()) {
> throw new UnknownHostException("unknown host: " +
> remoteId.getAddress().getHostName());
> }
> shouldn't we do something when this situation occurred?
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-13951) hbase regionserver shunt down
[ https://issues.apache.org/jira/browse/HBASE-13951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-13951. - Resolution: Invalid > hbase regionserver shunt down > -- > > Key: HBASE-13951 > URL: https://issues.apache.org/jira/browse/HBASE-13951 > Project: HBase > Issue Type: Bug >Reporter: longzai >Priority: Major > > [main] util.DirectMemoryUtils: Failed to retrieve nio.BufferPool direct > MemoryUsed attribute. > javax.management.InstanceNotFoundException: > java.nio:type=BufferPool,name=direct > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) > at > org.apache.hadoop.hbase.util.DirectMemoryUtils.(DirectMemoryUtils.java:72) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:369) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.(CacheConfig.java:166) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:576) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2323) > at > org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61) > at > org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2340) > 2015-06-18 20:42:40,748 INFO [main] hfile.CacheConfig: Allocating > LruBlockCache with maximum size 398.4 M -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13956) Add myself as 1.1 release manager
[ https://issues.apache.org/jira/browse/HBASE-13956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13956. --- > Add myself as 1.1 release manager > - > > Key: HBASE-13956 > URL: https://issues.apache.org/jira/browse/HBASE-13956 > Project: HBase > Issue Type: Task > Components: website >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Trivial > Fix For: 2.0.0 > > Attachments: 13956.patch > > > Just saw we have an RM section. Add myself. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13955) Review 'hbasecon2015 Lars Hofhansl Perf talk' and ensure our defaults align
[ https://issues.apache.org/jira/browse/HBASE-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13955. --- > Review 'hbasecon2015 Lars Hofhansl Perf talk' and ensure our defaults align > --- > > Key: HBASE-13955 > URL: https://issues.apache.org/jira/browse/HBASE-13955 > Project: HBase > Issue Type: Task > Components: documentation, Operability >Reporter: Michael Stack >Priority: Major > > I was looking at Lars slides this morning. A bunch of his recommendations are > defaulted to align with his recommendations, especially in later versions, > but we should review and ensure we do as he suggests (or better). Need to > also call in out in refguide such as his critical HDFS configurations (or > better, do as he suggests, and not start if HDFS does not have synconclose > set -- can't we make these defaults in our hdfs-default.xml altogether?) > Here are his slides: > http://www.slideshare.net/lhofhansl/h-base-tuninghbasecon2015ok -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13896) Multi-actions in hbase-client could fall in dead loop when region moves.
[ https://issues.apache.org/jira/browse/HBASE-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13896. --- > Multi-actions in hbase-client could fall in dead loop when region moves. > > > Key: HBASE-13896 > URL: https://issues.apache.org/jira/browse/HBASE-13896 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.13 >Reporter: Victor Xu >Priority: Minor > Attachments: HBASE-13896-0.98-v1.patch > > > The code in AsyncProcess.receiveGlobalFailure() use only one row to update > region cache in hbase-client. When we use HTable.put(List) api to write > some data which are from different regions and some of them are > moved/balanced while writing, the client may fall into a dead loop: > multi-actions fails because some regions moved => update only one region > cache(not the wrong ones) => resubmit => failed again. > It only happens in 0.98 branch, and the master branch is ok. > The patch followed should update all cached region locations when > multi-actions fails. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13894) Avoid visitor alloc each call of ByteBufferArray get/putMultiple()
[ https://issues.apache.org/jira/browse/HBASE-13894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13894. --- > Avoid visitor alloc each call of ByteBufferArray get/putMultiple() > -- > > Key: HBASE-13894 > URL: https://issues.apache.org/jira/browse/HBASE-13894 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.0.1, 1.1.0, 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 1.2.0, 1.1.1, 2.0.0 > > Attachments: HBASE-13894-v0.patch > > > ByteBufferArray getMultiple()/putMultiple() creates a new Visitor each call. > we can move that visitor out. > also, ByteBufferArray.multiple() seems to be an hotspot, I tried to do some > experiments using slice and read/write lock instead of a mutex, doesn't seems > to change much. any thoughts? -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13981) Fix ImportTsv spelling and usage issues
[
https://issues.apache.org/jira/browse/HBASE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell closed HBASE-13981.
---
> Fix ImportTsv spelling and usage issues
> ---
>
> Key: HBASE-13981
> URL: https://issues.apache.org/jira/browse/HBASE-13981
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.1.0.1
>Reporter: Lars George
>Priority: Major
> Labels: beginner
> Attachments: HBASE-13981.1.patch, HBASE-13981.2.patch,
> HBASE-13981.3.patch, HBASE-13981.4.patch
>
>
> The {{ImportTsv}} tool has various spelling and formatting issues. Fix those.
> In code:
> {noformat}
> public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
> "attributes.seperator";
> {noformat}
> It is "separator".
> In usage text:
> {noformat}
> "input data. Another special columnHBASE_TS_KEY designates that this column
> should be"
> {noformat}
> Space missing.
> {noformat}
> "Record with invalid timestamps (blank, non-numeric) will be treated as bad
> record."
> {noformat}
> "Records ... as bad records" - plural missing twice.
> {noformat}
> "HBASE_ATTRIBUTES_KEY can be used to specify Operation Attributes per record.
> Should be specified as key=>value where -1 is used
> as the seperator. Note that more than one OperationAttributes can be
> specified."
> {noformat}
> - Remove line wraps and indentation.
> - Fix "separator".
> - Fix wrong separator being output, it is not "-1" (wrong constant use in
> code)
> - General wording/style could be better (eg. last sentence now uses
> OperationAttributes without a space).
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-13981) Fix ImportTsv spelling and usage issues
[
https://issues.apache.org/jira/browse/HBASE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-13981.
-
Assignee: (was: Gabor Liptak)
Resolution: Abandoned
> Fix ImportTsv spelling and usage issues
> ---
>
> Key: HBASE-13981
> URL: https://issues.apache.org/jira/browse/HBASE-13981
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.1.0.1
>Reporter: Lars George
>Priority: Major
> Labels: beginner
> Attachments: HBASE-13981.1.patch, HBASE-13981.2.patch,
> HBASE-13981.3.patch, HBASE-13981.4.patch
>
>
> The {{ImportTsv}} tool has various spelling and formatting issues. Fix those.
> In code:
> {noformat}
> public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
> "attributes.seperator";
> {noformat}
> It is "separator".
> In usage text:
> {noformat}
> "input data. Another special columnHBASE_TS_KEY designates that this column
> should be"
> {noformat}
> Space missing.
> {noformat}
> "Record with invalid timestamps (blank, non-numeric) will be treated as bad
> record."
> {noformat}
> "Records ... as bad records" - plural missing twice.
> {noformat}
> "HBASE_ATTRIBUTES_KEY can be used to specify Operation Attributes per record.
> Should be specified as key=>value where -1 is used
> as the seperator. Note that more than one OperationAttributes can be
> specified."
> {noformat}
> - Remove line wraps and indentation.
> - Fix "separator".
> - Fix wrong separator being output, it is not "-1" (wrong constant use in
> code)
> - General wording/style could be better (eg. last sentence now uses
> OperationAttributes without a space).
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Closed] (HBASE-13979) Move out the lowReplication roll form FSHLog to be reusable
[ https://issues.apache.org/jira/browse/HBASE-13979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13979. --- > Move out the lowReplication roll form FSHLog to be reusable > --- > > Key: HBASE-13979 > URL: https://issues.apache.org/jira/browse/HBASE-13979 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Priority: Trivial > Attachments: HBASE-13979-v0.patch > > > I'd like to reuse the low-replication detection and log roll logic used in > the FSHLog. most of that logic doesn't know about Region or WALEdits, so we > can move it out. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-13979) Move out the lowReplication roll form FSHLog to be reusable
[ https://issues.apache.org/jira/browse/HBASE-13979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-13979. - Resolution: Abandoned > Move out the lowReplication roll form FSHLog to be reusable > --- > > Key: HBASE-13979 > URL: https://issues.apache.org/jira/browse/HBASE-13979 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Priority: Trivial > Attachments: HBASE-13979-v0.patch > > > I'd like to reuse the low-replication detection and log roll logic used in > the FSHLog. most of that logic doesn't know about Region or WALEdits, so we > can move it out. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13987) Modify the result of shell cmd list_quotas when not enable quota
[ https://issues.apache.org/jira/browse/HBASE-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13987. --- > Modify the result of shell cmd list_quotas when not enable quota > > > Key: HBASE-13987 > URL: https://issues.apache.org/jira/browse/HBASE-13987 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Priority: Minor > Attachments: HBASE-13987.patch > > > When not enable quota, use shell cmd list_quotas will get result as belows: > hbase(main):008:0> list_quotas > OWNERQUOTAS > > > ERROR: Unknown table hbase:quota! > It is confuse if user doesn't know quotas are stored in hbase:quota. I add > check isQuotaEnabled before scan the table hbase:quota. So it will return > result "ERROR: quota support disabled", which is same with set_quota. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13986) HMaster instance always returns false for isAborted() check.
[ https://issues.apache.org/jira/browse/HBASE-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13986. --- > HMaster instance always returns false for isAborted() check. > > > Key: HBASE-13986 > URL: https://issues.apache.org/jira/browse/HBASE-13986 > Project: HBase > Issue Type: Bug >Reporter: Abhishek Kumar >Priority: Minor > Attachments: HBASE-13986.patch > > > It seems that HMaster never set abortRequested flag to true as done by > HRegionServer in its abort() method.We can see isAborted method being used in > few places for HMaster instance (like in HMasterCommandLine.startMaster) > where code flow being determined based on the result of isAborted() call. > We can set this abortRequested flag in Hmaster's abort() method as well like > in HRegionServer's abort method, let me know if it seems ok. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-13830) Hbase REVERSED may throw Exception sometimes
[
https://issues.apache.org/jira/browse/HBASE-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-13830.
-
Resolution: Invalid
> Hbase REVERSED may throw Exception sometimes
>
>
> Key: HBASE-13830
> URL: https://issues.apache.org/jira/browse/HBASE-13830
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.1
>Reporter: ryan.jin
>Priority: Major
>
> run a scan at hbase shell command.
> {code}
> scan
> 'analytics_access',{ENDROW=>'9223370603647713262-flume01.hadoop-10.32.117.111-373563509',LIMIT=>10,REVERSED=>true}
> {code}
> will throw exception
> {code}
> java.io.IOException: java.io.IOException: Could not seekToPreviousRow
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://nameservice1/hbase/data/default/analytics_access/a54c47c568c00dd07f9d92cfab1accc7/cf/2e3a107e9fec4930859e992b61fb22f6,
> compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false],
> firstKey=9223370603542781142-flume01.hadoop-10.32.117.111-378180911/cf:key/1433311994702/Put,
>
> lastKey=9223370603715515112-flume01.hadoop-10.32.117.111-370923552/cf:timestamp/1433139261951/Put,
> avgKeyLen=80, avgValueLen=115, entries=43544340, length=1409247455,
> cur=9223370603647710245-flume01.hadoop-10.32.117.111-373563545/cf:payload/1433207065597/Put/vlen=644/mvcc=0]
> to key
> 9223370603647710245-flume01.hadoop-10.32.117.111-373563545/cf:payload/1433207065597/Put/vlen=644/mvcc=0
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekToPreviousRow(StoreFileScanner.java:448)
> at
> org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.seekToPreviousRow(ReversedKeyValueHeap.java:88)
> at
> org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.seekToPreviousRow(ReversedStoreScanner.java:128)
> at
> org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.seekToNextRow(ReversedStoreScanner.java:88)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:503)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3866)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3946)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3814)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3805)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3136)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29497)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: On-disk size without header provided is
> 47701, but block header contains 10134. Block offset: -1, data starts with:
> DATABLK*\x00\x00'\x96\x00\x01\x00\x04\x00\x00\x00\x005\x96^\xD2\x01\x00\x00@\x00\x00\x00'
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.validateOnDiskSizeWithoutHeader(HFileBlock.java:451)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.access$400(HFileBlock.java:87)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1466)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1314)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:355)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekBefore(HFileReaderV2.java:569)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekToPreviousRow(StoreFileScanner.java:413)
> ... 17 more
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[na:1.6.0_65]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> ~[na:1.6.0_65]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> ~[na:1.6.0_65]
> at
[jira] [Resolved] (HBASE-13987) Modify the result of shell cmd list_quotas when not enable quota
[ https://issues.apache.org/jira/browse/HBASE-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-13987. - Assignee: (was: Guanghao Zhang) Resolution: Abandoned > Modify the result of shell cmd list_quotas when not enable quota > > > Key: HBASE-13987 > URL: https://issues.apache.org/jira/browse/HBASE-13987 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Priority: Minor > Attachments: HBASE-13987.patch > > > When not enable quota, use shell cmd list_quotas will get result as belows: > hbase(main):008:0> list_quotas > OWNERQUOTAS > > > ERROR: Unknown table hbase:quota! > It is confuse if user doesn't know quotas are stored in hbase:quota. I add > check isQuotaEnabled before scan the table hbase:quota. So it will return > result "ERROR: quota support disabled", which is same with set_quota. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13907) Document how to deploy a coprocessor
[ https://issues.apache.org/jira/browse/HBASE-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13907. --- > Document how to deploy a coprocessor > > > Key: HBASE-13907 > URL: https://issues.apache.org/jira/browse/HBASE-13907 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: M Linville >Assignee: M Linville >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-13907-1.patch, HBASE-13907-2.patch, > HBASE-13907-v3.patch, HBASE-13907-v4.patch, HBASE-13907-v5.patch, > HBASE-13907-v6.patch, HBASE-13907.patch > > > Capture this information: > > Where are the dependencies located for these classes? Is there a path on > > HDFS or local disk that dependencies need to be placed so that each > > RegionServer has access to them? > It is suggested to bundle them as a single jar so that RS can load the whole > jar and resolve dependencies. If you are not able to do that, you need place > the dependencies in regionservers class path so that they are loaded during > RS startup. Do either of these options work for you? Btw, you can load the > coprocessors/filters into path specified by hbase.dynamic.jars.dir [1], so > that they are loaded dynamically by regionservers when the class is accessed > (or you can place them in the RS class path too, so that they are loaded > during RS JVM startup). > > How would one deploy these using an automated system? > > (puppet/chef/ansible/etc) > You can probably use these tools to automate shipping the jars to above > locations? > > Tests our developers have done suggest that simply disabling a coprocessor, > > replacing the jar with a different version, and enabling the coprocessor > > again does not load the newest version. With that in mind how does one know > > which version is currently deployed and enabled without resorting to > > parsing `hbase shell` output or restarting hbase? > Actually this is a design issue with current classloader. You can't reload a > class in a JVM unless you delete all the current references to it. Since the > current JVM (classloader) has reference to it, you can't overwrite it unless > you kill the JVM, which is equivalent to restarting it. So you still have the > older class loaded in place. For this to work, classloader design should be > changed. If it works for you, you can rename the coprocessor class name and > the new version of jar and RS loads it properly. > > Where does logging go, and how does one access it? Does logging need to be > > configured in a certain way? > Can you please specify which logging you are referring to? > > Where is a good location to place configuration files? > Same as above, are these hbase configs or something else? If hbase configs, > are these gateway configs/server side? -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13909) create 1.2 branch
[ https://issues.apache.org/jira/browse/HBASE-13909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13909. --- > create 1.2 branch > - > > Key: HBASE-13909 > URL: https://issues.apache.org/jira/browse/HBASE-13909 > Project: HBase > Issue Type: Sub-task > Components: build >Affects Versions: 1.2.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.3.0 > > Attachments: HBASE-13909.1.patch > > > create branch for 1.2 and update poms for branch-1 to 1.3.0-SNAPSHOT -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-13986) HMaster instance always returns false for isAborted() check.
[ https://issues.apache.org/jira/browse/HBASE-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-13986. - Assignee: (was: Abhishek Kumar) Resolution: Abandoned > HMaster instance always returns false for isAborted() check. > > > Key: HBASE-13986 > URL: https://issues.apache.org/jira/browse/HBASE-13986 > Project: HBase > Issue Type: Bug >Reporter: Abhishek Kumar >Priority: Minor > Attachments: HBASE-13986.patch > > > It seems that HMaster never set abortRequested flag to true as done by > HRegionServer in its abort() method.We can see isAborted method being used in > few places for HMaster instance (like in HMasterCommandLine.startMaster) > where code flow being determined based on the result of isAborted() call. > We can set this abortRequested flag in Hmaster's abort() method as well like > in HRegionServer's abort method, let me know if it seems ok. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13910) add branch-1.2 to precommit branches
[ https://issues.apache.org/jira/browse/HBASE-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13910. --- > add branch-1.2 to precommit branches > > > Key: HBASE-13910 > URL: https://issues.apache.org/jira/browse/HBASE-13910 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-13910.1.patch > > > update the precommit test properties so that patches targeting branch-1.2 work -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Closed] (HBASE-13908) 1.2 release umbrella
[ https://issues.apache.org/jira/browse/HBASE-13908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell closed HBASE-13908. --- > 1.2 release umbrella > > > Key: HBASE-13908 > URL: https://issues.apache.org/jira/browse/HBASE-13908 > Project: HBase > Issue Type: Umbrella > Components: build >Affects Versions: 1.2.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.2.0 > > > Umbrella for tasks related to the 1.2 release line starting. -- This message was sent by Atlassian Jira (v8.20.7#820007)
