[jira] [Updated] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[ https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13954: -- Attachment: HBASE-13954-v6.patch Fixed one of the line length warning rest are from the generated code. Remove HTableInterface#getRowOrBefore related server side code -- Key: HBASE-13954 URL: https://issues.apache.org/jira/browse/HBASE-13954 Project: HBase Issue Type: Sub-task Components: API Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch, HBASE-13954-v2.patch, HBASE-13954-v3.patch, HBASE-13954-v4.patch, HBASE-13954-v5.patch, HBASE-13954-v6.patch, HBASE-13954.patch As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the review board to remove all the server side related code for getRowOrBefore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13981) Fix ImportTsv spelling and usage issues
[
https://issues.apache.org/jira/browse/HBASE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632772#comment-14632772
]
Ashish Singhi commented on HBASE-13981:
---
Looks like it was added as part of HBASE-9767, but not not referred any where.
[~ramkrishna.s.vasude...@gmail.com] does it hold any meaning in our code ?
Fix ImportTsv spelling and usage issues
---
Key: HBASE-13981
URL: https://issues.apache.org/jira/browse/HBASE-13981
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Gabor Liptak
Labels: beginner
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-13981.1.patch, HBASE-13981.2.patch,
HBASE-13981.3.patch, HBASE-13981.4.patch
The {{ImportTsv}} tool has various spelling and formatting issues. Fix those.
In code:
{noformat}
public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.seperator;
{noformat}
It is separator.
In usage text:
{noformat}
input data. Another special columnHBASE_TS_KEY designates that this column
should be
{noformat}
Space missing.
{noformat}
Record with invalid timestamps (blank, non-numeric) will be treated as bad
record.
{noformat}
Records ... as bad records - plural missing twice.
{noformat}
HBASE_ATTRIBUTES_KEY can be used to specify Operation Attributes per record.
Should be specified as key=value where -1 is used
as the seperator. Note that more than one OperationAttributes can be
specified.
{noformat}
- Remove line wraps and indentation.
- Fix separator.
- Fix wrong separator being output, it is not -1 (wrong constant use in
code)
- General wording/style could be better (eg. last sentence now uses
OperationAttributes without a space).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[
https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632801#comment-14632801
]
Hadoop QA commented on HBASE-13954:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745998/HBASE-13954-v6.patch
against master branch at commit 338e39970ba8e4835733669b9252d073b2157b8a.
ATTACHMENT ID: 12745998
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 34 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ ualifier\030\002
\003(\014\\336\002\n\003Get\022\013\n\003row\030\001 \002(\014\022 \n\006c +
+ \016\n\006exists\030\003 \001(\010\022\024\n\005stale\030\004
\001(\010:\005false\022\026\n +
+ d\030\r \001(\010\022\r\n\005small\030\016
\001(\010\022\027\n\010reversed\030\017 \001(\010 +
+ new java.lang.String[] { Row, Column, Attribute, Filter,
TimeRange, MaxVersions, CacheBlocks, StoreLimit, StoreOffset,
ExistenceOnly, Consistency, });
+private static class getRegionInfo_argsStandardScheme extends
StandardSchemegetRegionInfo_args {
+private static class getRegionInfo_resultStandardScheme extends
StandardSchemegetRegionInfo_result {
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14831//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14831//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14831//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14831//console
This message is automatically generated.
Remove HTableInterface#getRowOrBefore related server side code
--
Key: HBASE-13954
URL: https://issues.apache.org/jira/browse/HBASE-13954
Project: HBase
Issue Type: Sub-task
Components: API
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0
Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch,
HBASE-13954-v2.patch, HBASE-13954-v3.patch, HBASE-13954-v4.patch,
HBASE-13954-v5.patch, HBASE-13954-v6.patch, HBASE-13954.patch
As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the
review board to remove all the server side related code for getRowOrBefore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632806#comment-14632806 ] Eshcar Hillel commented on HBASE-13408: --- Snapshot active set and the pipeline components are all memstore segments, it's an abstraction that allows to treat all these parts equally. The memstore compaction should work also with flush-by-column-family. However, even when flushing by column the WAL sequence id is defined per region (right?) so WAL truncation is not trivial. forceflushsize is not a new config, instead we take the average of flush size and the blocking flush size: flush-size forceflushsize blockingflushsize. When considering a flush-by-column-family mode, if the active segment is greater than flush size then flush is invoked and the active segment is pushed to the pipeline. If the active +pipeline segments are greater the forceflushsize then the flush is forced and snapshot is flushed to disk. All entries (active, pipeline, snapshot) are stored in a skip-list. The performance gain comes from accessing only memory and not the disk. The skip lists are not too large as multiple versions of the same key are removed within the compacted pipeline, but are not too small either, e.g., active is pushed to pipeline only when it gets to 128MB. When there is no duplication, i.e., a large set of active keys and no multiple versions per active key compaction is of no help, data is flushed to disk anyway but the compaction pipeline consumes memory and cpu. We don't see slow down in our experiments but in a setting where the memory/cpu resources are limited and contended for might show slow down. HBase In-Memory Memstore Compaction --- Key: HBASE-13408 URL: https://issues.apache.org/jira/browse/HBASE-13408 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Attachments: HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, InMemoryMemstoreCompactionEvaluationResults.pdf A store unit holds a column family in a region, where the memstore is its in-memory component. The memstore absorbs all updates to the store; from time to time these updates are flushed to a file on disk, where they are compacted. Unlike disk components, the memstore is not compacted until it is written to the filesystem and optionally to block-cache. This may result in underutilization of the memory due to duplicate entries per row, for example, when hot data is continuously updated. Generally, the faster the data is accumulated in memory, more flushes are triggered, the data sinks to disk more frequently, slowing down retrieval of data, even if very recent. In high-churn workloads, compacting the memstore can help maintain the data in memory, and thereby speed up data retrieval. We suggest a new compacted memstore with the following principles: 1.The data is kept in memory for as long as possible 2.Memstore data is either compacted or in process of being compacted 3.Allow a panic mode, which may interrupt an in-progress compaction and force a flush of part of the memstore. We suggest applying this optimization only to in-memory column families. A design document is attached. This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13867) Add endpoint coprocessor guide to HBase book
[ https://issues.apache.org/jira/browse/HBASE-13867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurav Bhardwaj updated HBASE-13867: Status: In Progress (was: Patch Available) Add endpoint coprocessor guide to HBase book Key: HBASE-13867 URL: https://issues.apache.org/jira/browse/HBASE-13867 Project: HBase Issue Type: Task Components: Coprocessors, documentation Reporter: Vladimir Rodionov Assignee: Gaurav Bhardwaj Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.2 Attachments: HBASE-13867.1.patch Endpoint coprocessors are very poorly documented. Coprocessor section of HBase book must be updated either with its own endpoint coprocessors HOW-TO guide or, at least, with the link(s) to some other guides. There is good description here: http://www.3pillarglobal.com/insights/hbase-coprocessors -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14122) Client API for determining if server side supports cell level security
Andrew Purtell created HBASE-14122:
--
Summary: Client API for determining if server side supports cell
level security
Key: HBASE-14122
URL: https://issues.apache.org/jira/browse/HBASE-14122
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0
Add a client API for determining if the server side supports cell level
security.
Ask the master, assuming as we do in many other instances that the master and
regionservers all have a consistent view of site configuration.
Return {{true}} if all features required for cell level security are present,
{{false}} otherwise, or throw {{UnsupportedOperationException}} if the master
does not have support for the RPC call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[ https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632872#comment-14632872 ] Ashish Singhi commented on HBASE-13954: --- line length warnings are from auto generate proto and thrift classes. Remove HTableInterface#getRowOrBefore related server side code -- Key: HBASE-13954 URL: https://issues.apache.org/jira/browse/HBASE-13954 Project: HBase Issue Type: Sub-task Components: API Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch, HBASE-13954-v2.patch, HBASE-13954-v3.patch, HBASE-13954-v4.patch, HBASE-13954-v5.patch, HBASE-13954-v6.patch, HBASE-13954.patch As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the review board to remove all the server side related code for getRowOrBefore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu reassigned HBASE-6721: -- Assignee: Francis Liu (was: Vandana Ayyalasomayajula) RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Francis Liu Attachments: 6721-master-webUI.patch, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721_10.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, HBASE-6721_94_7.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, HBASE-6721_trunk2.patch In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13992) Integrate SparkOnHBase into HBase
[ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Malaska updated HBASE-13992: Attachment: HBASE-13992.patch.5 This patch gets us much closer to the finish line. There are a bunch of review fixes, javadoc, Pom fixes, and unit tests. Please review. Integrate SparkOnHBase into HBase - Key: HBASE-13992 URL: https://issues.apache.org/jira/browse/HBASE-13992 Project: HBase Issue Type: New Feature Components: spark Reporter: Ted Malaska Assignee: Ted Malaska Fix For: 2.0.0 Attachments: HBASE-13992.patch, HBASE-13992.patch.3, HBASE-13992.patch.4, HBASE-13992.patch.5 This Jira is to ask if SparkOnHBase can find a home in side HBase core. Here is the github: https://github.com/cloudera-labs/SparkOnHBase I am the core author of this project and the license is Apache 2.0 A blog explaining this project is here http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/ A spark Streaming example is here http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ A real customer using this in produce is blogged here http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/ Please debate and let me know what I can do to make this happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14059) We should add a RS to the dead servers list if admin calls fail more than a threshold
[
https://issues.apache.org/jira/browse/HBASE-14059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633048#comment-14633048
]
Duo Zhang commented on HBASE-14059:
---
{quote}
In the issue I ran into it was a bad region causing the RS to be blocked for a
long time.
{quote}
More details? What does 'bad' mean? And you said the region is back to normal
when you kill the RS, so I think there maybe another bug?
In general, I agree with you, we should offline a RS if admin calls always
fail. But it should be used to fix a 'bad' RS, not a 'bad' region. If there is
a 'bad' region that can not be fixed by reassign, then as [~chenheng] said, the
'bad' region will kill all regionservers in your cluster...
Thanks.
We should add a RS to the dead servers list if admin calls fail more than a
threshold
-
Key: HBASE-14059
URL: https://issues.apache.org/jira/browse/HBASE-14059
Project: HBase
Issue Type: Bug
Components: master, regionserver, rpc
Affects Versions: 0.98.13
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Priority: Critical
I ran into this problem twice this week: calls from the HBase master to a RS
can timeout since the RS call queue size has been maxed out, however since
the RS is not dead (ephemeral znode still present) the master keeps
attempting to perform admin tasks like trying to open or close a region but
those operations eventually fail after we run out of retries or the
assignment manager attempts to re-assign to other RSs. From the side effects
of this I've noticed master operations to be fully blocked or RITs since we
cannot close the region and open the region in a new location since RS is not
dead.
A potential solution for this is to add the RS to the list of dead RSs after
certain number of calls from the master to the RS fail.
I've noticed only the problem in 0.98.x but it should be present in all
versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
