[jira] [Updated] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked
[ https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiang Tian updated HBASE-11096: --- Status: Open (was: Patch Available) > stop method of Master and RegionServer coprocessor is not invoked > -- > > Key: HBASE-11096 > URL: https://issues.apache.org/jira/browse/HBASE-11096 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.19, 0.98.1, 0.96.2 >Reporter: Qiang Tian >Assignee: Qiang Tian > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v0.patch, > HBASE-11096-trunk-v1.patch, HBASE-11096-trunk-v2.patch > > > the stop method of coprocessor specified by > "hbase.coprocessor.master.classes" and > "hbase.coprocessor.regionserver.classes" is not invoked. > If coprocessor allocates OS resources, it could cause master/regionserver > resource leak or hang during exit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9740) A corrupt HFile could cause endless attempts to assign the region without a chance of success
[ https://issues.apache.org/jira/browse/HBASE-9740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9740: - Resolution: Won't Fix Fix Version/s: (was: 0.94.20) Status: Resolved (was: Patch Available) After pushing this around for a few releases... Do we need this in 0.94? It's not clear how anybody would find out about these regions. Currently it is clear that something is wrong. We can resurrect if it is important to have this in 0.94. > A corrupt HFile could cause endless attempts to assign the region without a > chance of success > - > > Key: HBASE-9740 > URL: https://issues.apache.org/jira/browse/HBASE-9740 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.16 >Reporter: Aditya Kishore >Assignee: Ping > Attachments: HBase-9740_0.94_v4.patch, HBase-9749_0.94_v2.patch, > HBase-9749_0.94_v3.patch, patch-9740_0.94.txt > > > As described in HBASE-9737, a corrupt HFile in a region could lead to an > assignment storm in the cluster since the Master will keep trying to assign > the region to each region server one after another and obviously none will > succeed. > The region server, upon detecting such a scenario should mark the region as > "RS_ZK_REGION_FAILED_ERROR" (or something to the effect) in the Zookeeper > which should indicate the Master to stop assigning the region until the error > has been resolved (via an HBase shell command, probably "assign"?) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10936) Add zeroByte encoding test
[ https://issues.apache.org/jira/browse/HBASE-10936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-10936: -- Summary: Add zeroByte encoding test (was: Port zeroByte encoding test from parent.) > Add zeroByte encoding test > -- > > Key: HBASE-10936 > URL: https://issues.apache.org/jira/browse/HBASE-10936 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Lars Hofhansl >Priority: Minor > Fix For: 0.96.3, 0.94.20 > > Attachments: 10936-0.94.txt, 10936-0.96.txt, 10936-0.98.txt > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11143) ageOfLastShippedOp metric is confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-11143: -- Fix Version/s: (was: 0.96.3) > ageOfLastShippedOp metric is confusing > -- > > Key: HBASE-11143 > URL: https://issues.apache.org/jira/browse/HBASE-11143 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.99.0, 0.94.20, 0.98.3 > > Attachments: 11143-0.94-v2.txt, 11143-0.94.txt, 11143-trunk.txt > > > We are trying to report on replication lag and find that there is no good > single metric to do that. > ageOfLastShippedOp is close, but unfortunately it is increased even when > there is nothing to ship on a particular RegionServer. > I would like discuss a few options here: > Add a new metric: replicationQueueTime (or something) with the above meaning. > I.e. if we have something to ship we set the age of that last shipped edit, > if we fail we increment that last time (just like we do now). But if there is > nothing to replicate we set it to current time (and hence that metric is > reported to close to 0). > Alternatively we could change the meaning of ageOfLastShippedOp to mean to do > that. That might lead to surprises, but the current behavior is clearly weird > when there is nothing to replicate. > Comments? [~jdcryans], [~stack]. > If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10915) Decouple region closing (HM and HRS) from ZK
[
https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993467#comment-13993467
]
Hadoop QA commented on HBASE-10915:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644074/HBASE-10915.patch
against trunk revision .
ATTACHMENT ID: 12644074
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9490//console
This message is automatically generated.
> Decouple region closing (HM and HRS) from ZK
>
>
> Key: HBASE-10915
> URL: https://issues.apache.org/jira/browse/HBASE-10915
> Project: HBase
> Issue Type: Sub-task
> Components: Consensus, Zookeeper
>Affects Versions: 0.99.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
> Attachments: HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch,
> HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch,
> HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch,
> HBASE-10915.patch, HBASE-10915.patch
>
>
> Decouple region closing from ZK.
> Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the
> code using (HRegionServer, RSRpcServices etc).
> May need small changes in AssignmentManager.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996086#comment-13996086 ] stack commented on HBASE-7912: -- Very nice. This doc. with perhaps a little more commentary like it could go into the hbase refguide when this feature is committed? https://issues.apache.org/jira/secure/attachment/12638283/HBase_BackupRestore-Jira-7912-CLI-v1.pdf We should add a 'backups' chapter to hbase refguide when this comes in? >From design doc: bq. We execute roll log across region servers to track the WALs that need to be in the backup. Late question. How we ensure the WAL is not moved aside or even deleted between time of snapshot and copy aside? bq. We’ll convert/replay the backed-up Hlogs into HFiles for fast incremental restore. This is interesting. It is done against a cluster or it is just a MR job/tool? Looks very nice. Have you lads had much chance testing it out? What needs to go in first? What should we review first? Thanks lads. > HBase Backup/Restore Based on HBase Snapshot > > > Key: HBASE-7912 > URL: https://issues.apache.org/jira/browse/HBASE-7912 > Project: HBase > Issue Type: Sub-task >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf, > HBase_BackupRestore-Jira-7912-CLI-v1.pdf > > > Finally, we completed the implementation of our backup/restore solution, and > would like to share with community through this jira. > We are leveraging existing hbase snapshot feature, and provide a general > solution to common users. Our full backup is using snapshot to capture > metadata locally and using exportsnapshot to move data to another cluster; > the incremental backup is using offline-WALplayer to backup HLogs; we also > leverage global distribution rolllog and flush to improve performance; other > added-on values such as convert, merge, progress report, and CLI commands. So > that a common user can backup hbase data without in-depth knowledge of hbase. > Our solution also contains some usability features for enterprise users. > The detail design document and CLI command will be attached in this jira. We > plan to use 10~12 subtasks to share each of the following features, and > document the detail implement in the subtasks: > * *Full Backup* : provide local and remote back/restore for a list of tables > * *offline-WALPlayer* to convert HLog to HFiles offline (for incremental > backup) > * *distributed* Logroll and distributed flush > * Backup *Manifest* and history > * *Incremental* backup: to build on top of full backup as daily/weekly backup > * *Convert* incremental backup WAL files into hfiles > * *Merge* several backup images into one(like merge weekly into monthly) > * *add and remove* table to and from Backup image > * *Cancel* a backup process > * backup progress *status* > * full backup based on *existing snapshot* > *-* > *Below is the original description, to keep here as the history for the > design and discussion back in 2013* > There have been attempts in the past to come up with a viable HBase > backup/restore solution (e.g., HBASE-4618). Recently, there are many > advancements and new features in HBase, for example, FileLink, Snapshot, and > Distributed Barrier Procedure. This is a proposal for a backup/restore > solution that utilizes these new features to achieve better performance and > consistency. > > A common practice of backup and restore in database is to first take full > baseline backup, and then periodically take incremental backup that capture > the changes since the full baseline backup. HBase cluster can store massive > amount data. Combination of full backups with incremental backups has > tremendous benefit for HBase as well. The following is a typical scenario > for full and incremental backup. > # The user takes a full backup of a table or a set of tables in HBase. > # The user schedules periodical incremental backups to capture the changes > from the full backup, or from last incremental backup. > # The user needs to restore table data to a past point of time. > # The full backup is restored to the table(s) or to different table name(s). > Then the incremental backups that are up to the desired point in time are > applied on top of the full backup. > We would support the following key features and capabilities. > * Full backup uses HBase snapshot to capture HFiles. > * Use HBase WALs to capture incremental changes, but we use bulk load of > HFiles for fast incremental restore. > * Support single table or a set of tables, and column family level backup and > restore. > * Restore to different table
[jira] [Commented] (HBASE-11140) LocalHBaseCluster should create ConsensusProvider per each server
[ https://issues.apache.org/jira/browse/HBASE-11140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995748#comment-13995748 ] Mikhail Antonov commented on HBASE-11140: - Thanks [~stack] > LocalHBaseCluster should create ConsensusProvider per each server > - > > Key: HBASE-11140 > URL: https://issues.apache.org/jira/browse/HBASE-11140 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11140.patch > > > Right now there's a bug there when single ConsensusProvider instance is > shared across all threads running region servers and masters within > processes, which breaks certain tests in patches which used to pass > successfully before. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11069) Decouple region merging from ZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996045#comment-13996045
]
Hadoop QA commented on HBASE-11069:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644525/HBASE_11069-v2.patch
against trunk revision .
ATTACHMENT ID: 12644525
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ rmd = server != null && server.getConsensusProvider() != null ?
((BaseConsensusProvider) server
+ public void prepareForMerge(Server server, RegionServerServices services,
HRegionInfo mergedRegionInfo,
+ public void prepareForMerge(Server server, RegionServerServices services,
HRegionInfo mergedRegionInfo,
+ *
org.apache.hadoop.hbase.regionserver.consensus.RegionMergeConsensus#finishRegionMergeTransaction
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9510//console
This message is automatically generated.
> Decouple region merging from ZooKeeper
> --
>
> Key: HBASE-11069
> URL: https://issues.apache.org/jira/browse/HBASE-11069
> Project: HBase
> Issue Type: Sub-task
> Components: Consensus, Zookeeper
>Reporter: Sergey Soldatov
> Attachments: HBASE-11069.patch, HBASE_11069-v2.patch
>
>
> Region Merge should be decoupled from ZK.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11151) move tracing modules from hbase-server to hbase-common
[ https://issues.apache.org/jira/browse/HBASE-11151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996034#comment-13996034 ] Nick Dimiduk commented on HBASE-11151: -- Yep, sounds about right. Thanks. > move tracing modules from hbase-server to hbase-common > -- > > Key: HBASE-11151 > URL: https://issues.apache.org/jira/browse/HBASE-11151 > Project: HBase > Issue Type: Improvement > Components: documentation, util >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-11151-0.patch > > > Not only servers but also clients using tracing needs SpanReceiverHost. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-1990) Add methods accepting strings for family/qualifier in client
[
https://issues.apache.org/jira/browse/HBASE-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-1990.
---
Resolution: Won't Fix
Not wanted badly enough I'd say
> Add methods accepting strings for family/qualifier in client
> -
>
> Key: HBASE-1990
> URL: https://issues.apache.org/jira/browse/HBASE-1990
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 0.20.0
>Reporter: Doug Meil
>Priority: Minor
> Attachments: TestHTableGenerics.java
>
>
> Consider the following client code...
> byte b[] = result.getValue( Bytes.toBytes("family"),
> Bytes.toBytes("qualifier") );
> put.add( Bytes.toBytes("family"), Bytes.toBytes("qualifer"),
> Bytes.toBytes( "value") );
> ... the requirement to supply family and qualifiers as bytes causes code to
> get cluttered and verbose. At worst, it scares peoples un-necessarily about
> HBase development, and at best, developers inevitably will get tired of doing
> all this casting and then add their own wrapper classes around the HBase
> client to make their code more readable.
> I would like to see something like this in the API...
> byte b[] = result.getValue( "family"), "qualifier" );
> put.add( "family", "qualifer", Bytes.toBytes( "value") );
> ... where the Hbase client can perform the required Bytes.toBytes()
> conversion behind the scenes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10504) Define Replication Interface
[ https://issues.apache.org/jira/browse/HBASE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995685#comment-13995685 ] Jean-Daniel Cryans commented on HBASE-10504: [~enis] So IIUC you are going a completely different way than what was discussed before, right? I don't mind it but I want to make sure that we're on the same track. So instead of having people define their sinks, they instead put that code in the ReplicationConsumer. Seems like a good idea, you don't have to mess around getting a fake cluster up and running, among other issues. Might still be nice to offer some tools like removeNonReplicableEdits(), the basic metrics the handling of a disabled peer, etc, for the other consumers. > Define Replication Interface > > > Key: HBASE-10504 > URL: https://issues.apache.org/jira/browse/HBASE-10504 > Project: HBase > Issue Type: Task >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.99.0 > > Attachments: hbase-10504_wip1.patch > > > HBase has replication. Fellas have been hijacking the replication apis to do > all kinds of perverse stuff like indexing hbase content (hbase-indexer > https://github.com/NGDATA/hbase-indexer) and our [~toffer] just showed up w/ > overrides that replicate via an alternate channel (over a secure thrift > channel between dcs over on HBASE-9360). This issue is about surfacing these > APIs as public with guarantees to downstreamers similar to those we have on > our public client-facing APIs (and so we don't break them for downstreamers). > Any input [~phunt] or [~gabriel.reid] or [~toffer]? > Thanks. > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11135) Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file
[
https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996046#comment-13996046
]
Jeffrey Zhong commented on HBASE-11135:
---
{quote}
I am not going to migrate your patch on top of mine. I don't know the
unification project well enough
{quote}
No. Sorry for the confusion. I mean I'll migrate my patch on top of yours and
run tests.
{quote}
Should I commit it?
{quote}
Sure. +1. Thanks.
> Change region sequenceid generation so happens earlier in the append cycle
> rather than just before added to file
>
>
> Key: HBASE-11135
> URL: https://issues.apache.org/jira/browse/HBASE-11135
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: stack
>Assignee: stack
> Attachments: 11135.wip.txt, 11135v2.txt, 11135v5.txt, 11135v5.txt,
> 11135v5.txt, 11135v6.txt, 11135v7.txt
>
>
> Currently we assign the region edit/sequence id just before we put it in the
> WAL. We do it in the single thread that feeds from the ring buffer. Doing
> it at this point, we can ensure order, that the edits will be in the file in
> accordance w/ the ordering of the region sequence id.
> But the point at which region sequence id is assigned an edit is deep down in
> the WAL system and there is a lag between our putting an edit into the WAL
> system and the edit actually getting its edit/sequence id.
> This lag -- "late-binding" -- complicates the unification of mvcc and region
> sequence id, especially around async WAL writes (and, related, for no-WAL
> writes) -- the parent for this issue (For async, how you get the edit id in
> our system when the threads have all gone home -- unless you make them wait?)
> Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of
> getting the region sequence id near-immediately. We'll run two ringbuffers.
> The first will mesh all handler threads and the consumer will generate ids
> (we will have order on other side of this first ring buffer), and then if
> async or no sync, we will just let the threads return ... updating mvcc just
> before we let them go. All other calls will go up on to the second ring
> buffer to be serviced as now (batching, distribution out among the sync'ing
> threads). The first rb will have no friction and should turn at fast rates
> compared to the second. There should not be noticeable slowdown nor do I
> foresee this refactor intefering w/ our multi-WAL plans.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11123) Upgrade instructions from 0.94 to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11123: Status: Patch Available (was: Open) > Upgrade instructions from 0.94 to 0.98 > -- > > Key: HBASE-11123 > URL: https://issues.apache.org/jira/browse/HBASE-11123 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Minor > Attachments: HBASE-11123.patch > > > I cloned this from the 0.96 upgrade docs task. It was suggested that we need > upgrade instructions from 0.94 to 0.98. I will need source material to even > prioritize this. Assuming this is Minor. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11123) Upgrade instructions from 0.94 to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11123: Attachment: HBASE-11123.patch Added upgrade info from 0.94 to 0.98 and fixed the "TODO" item in the 0.98 section. > Upgrade instructions from 0.94 to 0.98 > -- > > Key: HBASE-11123 > URL: https://issues.apache.org/jira/browse/HBASE-11123 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Priority: Minor > Attachments: HBASE-11123.patch > > > I cloned this from the 0.96 upgrade docs task. It was suggested that we need > upgrade instructions from 0.94 to 0.98. I will need source material to even > prioritize this. Assuming this is Minor. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11119) Update ExportSnapShot to optionally not use a tmp file on external file system
[ https://issues.apache.org/jira/browse/HBASE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995884#comment-13995884 ] Hudson commented on HBASE-9: SUCCESS: Integrated in hbase-0.96-hadoop2 #276 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/276/]) HBASE-9 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593339) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java > Update ExportSnapShot to optionally not use a tmp file on external file system > -- > > Key: HBASE-9 > URL: https://issues.apache.org/jira/browse/HBASE-9 > Project: HBase > Issue Type: Improvement > Components: snapshots >Reporter: Ted Malaska >Assignee: Ted Malaska >Priority: Minor > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: HBASE-9.patch > > > There are FileSystem like S3 where renaming is extremely expensive. This > patch will add a parameter that says something like > use.tmp.folder > It will be defaulted to true. So default behavior is the same. If false is > set them the files will land in the final destination with no need for a > rename. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11128) Add -target option to ExportSnapshot to export with a different name
[ https://issues.apache.org/jira/browse/HBASE-11128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995979#comment-13995979 ] Hudson commented on HBASE-11128: SUCCESS: Integrated in HBase-0.94-JDK7 #134 (See [https://builds.apache.org/job/HBase-0.94-JDK7/134/]) HBASE-11128 Add -target option to ExportSnapshot to export with a different name (mbertozzi: rev 1593778) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java > Add -target option to ExportSnapshot to export with a different name > > > Key: HBASE-11128 > URL: https://issues.apache.org/jira/browse/HBASE-11128 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 0.99.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: HBASE-11128-v0.patch > > > Add a "-target" option to export the snapshot using a different name -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11135) Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file
[ https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11135: -- Attachment: 11135v7.txt Fix findbugs and javadoc. The failure is odd (says only one replica). Can't repro down here. Retry. > Change region sequenceid generation so happens earlier in the append cycle > rather than just before added to file > > > Key: HBASE-11135 > URL: https://issues.apache.org/jira/browse/HBASE-11135 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Assignee: stack > Attachments: 11135.wip.txt, 11135v2.txt, 11135v5.txt, 11135v5.txt, > 11135v5.txt, 11135v6.txt, 11135v7.txt > > > Currently we assign the region edit/sequence id just before we put it in the > WAL. We do it in the single thread that feeds from the ring buffer. Doing > it at this point, we can ensure order, that the edits will be in the file in > accordance w/ the ordering of the region sequence id. > But the point at which region sequence id is assigned an edit is deep down in > the WAL system and there is a lag between our putting an edit into the WAL > system and the edit actually getting its edit/sequence id. > This lag -- "late-binding" -- complicates the unification of mvcc and region > sequence id, especially around async WAL writes (and, related, for no-WAL > writes) -- the parent for this issue (For async, how you get the edit id in > our system when the threads have all gone home -- unless you make them wait?) > Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of > getting the region sequence id near-immediately. We'll run two ringbuffers. > The first will mesh all handler threads and the consumer will generate ids > (we will have order on other side of this first ring buffer), and then if > async or no sync, we will just let the threads return ... updating mvcc just > before we let them go. All other calls will go up on to the second ring > buffer to be serviced as now (batching, distribution out among the sync'ing > threads). The first rb will have no friction and should turn at fast rates > compared to the second. There should not be noticeable slowdown nor do I > foresee this refactor intefering w/ our multi-WAL plans. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11069) Decouple region merging from ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995872#comment-13995872 ] Sergey Soldatov commented on HBASE-11069: - Could someone add me to the project contributors, so I will be assign it to me? > Decouple region merging from ZooKeeper > -- > > Key: HBASE-11069 > URL: https://issues.apache.org/jira/browse/HBASE-11069 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Reporter: Sergey Soldatov > Attachments: HBASE-11069.patch, HBASE_11069-v2.patch > > > Region Merge should be decoupled from ZK. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10985) Decouple Split Transaction from Zookeeper
[ https://issues.apache.org/jira/browse/HBASE-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov updated HBASE-10985: Status: Open (was: Patch Available) > Decouple Split Transaction from Zookeeper > - > > Key: HBASE-10985 > URL: https://issues.apache.org/jira/browse/HBASE-10985 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Reporter: Sergey Soldatov > Attachments: HBASE-10985.patch, HBASE-10985.patch, HBASE-10985.patch, > HBASE_10985-v2.patch, HBASE_10985-v3.patch, HBASE_10985-v4.patch > > > As part of HBASE-10296 SplitTransaction should be decoupled from Zookeeper. > This is an initial patch for review. At the moment the consensus provider > placed directly to SplitTransaction to minimize affected code. In the ideal > world it should be done in HServer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11154) Document how to use Reverse Scan API
[ https://issues.apache.org/jira/browse/HBASE-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995922#comment-13995922 ] Misty Stanley-Jones commented on HBASE-11154: - I think the primary value-proposition for reverse scan might belong here, as "One time when you don't need to use secondary indexes anymore": http://hbase.apache.org/book.html#secondary.indexes 6.11.1.3 here is also a good place to have a note about the new functionality: http://hbase.apache.org/book.html#schema.casestudies Maybe something here too: http://hbase.apache.org/book.html#reverse.timestamp Or maybe I need a whole more abstract session about scanners. > Document how to use Reverse Scan API > > > Key: HBASE-11154 > URL: https://issues.apache.org/jira/browse/HBASE-11154 > Project: HBase > Issue Type: Task > Components: documentation, Scanners >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10942) support parallel request cancellation for multi-get
[ https://issues.apache.org/jira/browse/HBASE-10942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995631#comment-13995631 ] Sergey Shelukhin commented on HBASE-10942: -- About config - I think it's important to have way to turn off any new feature in case there's a bug in it... otherwise making new build and redeploying a cluster can be very costly for whoever is maintaining it. Wrt confusion, I'll rename variables. What do you mean by having CompletionService i.e. what would we use it for > support parallel request cancellation for multi-get > --- > > Key: HBASE-10942 > URL: https://issues.apache.org/jira/browse/HBASE-10942 > Project: HBase > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-10070 > > Attachments: HBASE-10942.01.patch, HBASE-10942.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10909) Abstract out ZooKeeper usage in HBase
[ https://issues.apache.org/jira/browse/HBASE-10909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-10909: Attachment: HBaseConsensus.pdf > Abstract out ZooKeeper usage in HBase > - > > Key: HBASE-10909 > URL: https://issues.apache.org/jira/browse/HBASE-10909 > Project: HBase > Issue Type: Umbrella > Components: Consensus, Zookeeper >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Attachments: HBaseConsensus.pdf, HBaseConsensus.pdf, > HBaseConsensus.pdf > > > As some sort of follow-up or initial step towards HBASE-10296. > Whatever consensus algorithm/library may be the chosen, perhaps one of first > practical steps towards this goal would be to better abstract ZK-related API > and details, which are now throughout the codebase (mostly leaked throuth > ZkUtil, ZooKeeperWatcher and listeners). > This jira is umbrella for relevant subtasks. > As the design doc is in process of peer-review now, please use Google Doc > (linked below) instead of pdf. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11154) Document how to use Reverse Scan API
[ https://issues.apache.org/jira/browse/HBASE-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11154: Fix Version/s: 0.98.3 Status: Patch Available (was: Open) > Document how to use Reverse Scan API > > > Key: HBASE-11154 > URL: https://issues.apache.org/jira/browse/HBASE-11154 > Project: HBase > Issue Type: Task > Components: documentation, Scanners >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 0.98.3 > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11148) Provide a distributed procedure to globally roll logs
[ https://issues.apache.org/jira/browse/HBASE-11148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995877#comment-13995877 ] Jerry He commented on HBASE-11148: -- [~stack] I will provide a patch shortly. > Provide a distributed procedure to globally roll logs > - > > Key: HBASE-11148 > URL: https://issues.apache.org/jira/browse/HBASE-11148 > Project: HBase > Issue Type: New Feature >Reporter: Jerry He > Fix For: 0.99.0 > > > Propose a distributed procedure here to globally roll logs. > Currently HBaseAdmin and HBase shell provides a way to roll the WAL on a > single RS. > Some use cases may require that all the RS roll the logs at the same time and > in a coordinated way. Also there may be requirement that some tasks be done > together with the roll log on each region server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jiajia updated HBASE-11144: -- Attachment: MultiRowRangeFilter2.patch MultiRowRangeFilter2.patch is generated for trunk. > Filter to support scan multiple row key ranges > -- > > Key: HBASE-11144 > URL: https://issues.apache.org/jira/browse/HBASE-11144 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Li Jiajia > Attachments: MultiRowRangeFilter.patch, MultiRowRangeFilter2.patch > > > Provide a filter feature to support scan multiple row key ranges. It can > construct the row key ranges from the passed list which can be accessed by > each region server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jiajia updated HBASE-11144: -- Status: Open (was: Patch Available) > Filter to support scan multiple row key ranges > -- > > Key: HBASE-11144 > URL: https://issues.apache.org/jira/browse/HBASE-11144 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Li Jiajia > Attachments: MultiRowRangeFilter.patch > > > Provide a filter feature to support scan multiple row key ranges. It can > construct the row key ranges from the passed list which can be accessed by > each region server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11069) Decouple region merging from ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov updated HBASE-11069: Status: Patch Available (was: Open) > Decouple region merging from ZooKeeper > -- > > Key: HBASE-11069 > URL: https://issues.apache.org/jira/browse/HBASE-11069 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Reporter: Sergey Soldatov > Attachments: HBASE-11069.patch, HBASE_11069-v2.patch > > > Region Merge should be decoupled from ZK. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-6626) Add a chapter on HDFS in the troubleshooting section of the HBase reference guide.
[ https://issues.apache.org/jira/browse/HBASE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995674#comment-13995674 ] Misty Stanley-Jones commented on HBASE-6626: Stack, yes please. Unless you agree with Nicholas. > Add a chapter on HDFS in the troubleshooting section of the HBase reference > guide. > -- > > Key: HBASE-6626 > URL: https://issues.apache.org/jira/browse/HBASE-6626 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.95.2 >Reporter: Nicolas Liochon >Assignee: Doug Meil >Priority: Blocker > Attachments: troubleshooting.txt > > > I looked mainly at the major failure case, but here is what I have: > New sub chapter in the existing chapter "Troubleshooting and Debugging > HBase": "HDFS & HBASE" > 1) HDFS & HBase > 2) Connection related settings > 2.1) Number of retries > 2.2) Timeouts > 3) Log samples > 1) HDFS & HBase > HBase uses HDFS to store its HFile, i.e. the core HBase files and the > Write-Ahead-Logs, i.e. the files that will be used to restore the data after > a crash. > In both cases, the reliability of HBase comes from the fact that HDFS writes > the data to multiple locations. To be efficient, HBase needs the data to be > available locally, hence it's highly recommended to have the HDFS datanode on > the same machines as the HBase Region Servers. > Detailed information on how HDFS works can be found at [1]. > Important features are: > - HBase is a client application of HDFS, i.e. uses the HDFS DFSClient class. > This class can appears in HBase logs with other HDFS client related logs. > - Some HDFS settings are HDFS-server-side, i.e. must be set on the HDFS > side, while some other are HDFS-client-side, i.e. must be set in HBase, while > some other must be set in both places. > - the HDFS writes are pipelined from one datanode to another. When writing, > there are communications between: > - HBase and HDFS namenode, through the HDFS client classes. > - HBase and HDFS datanodes, through the HDFS client classes. > - HDFS datanode between themselves: issues on these communications are in > HDFS logs, not HBase. HDFS writes are always local when possible. As a > consequence, there should not be much write error in HBase Region Servers: > they write to the local datanode. If this datanode can't replicate the > blocks, it will appear in its logs, not in the region servers logs. > - datanodes can be contacted through the ipc.Client interface (once again > this class can shows up in HBase logs) and the data transfer interface > (usually shows up as the DataNode class in the HBase logs). There are on > different ports (defaults being: 50010 and 50020). > - To understand exactly what's going on, you must look that the HDFS log > files as well: HBase logs represent the client side. > - With the default setting, HDFS needs 630s to mark a datanode as dead. For > this reason, this node will still be tried by HBase or by other datanodes > when writing and reading until HDFS definitively decides it's dead. This will > add some extras lines in the logs. This monitoring is performed by the > NameNode. > - The HDFS clients (i.e. HBase using HDFS client code) don't fully rely on > the NameNode, but can mark temporally a node as dead if they had an error > when they tried to use it. > 2) Settings for retries and timeouts > 2.1) Retries > ipc.client.connect.max.retries > Default 10 > Indicates the number of retries a client will make to establish a server > connection. Not taken into account if the error is a SocketTimeout. In this > case the number of retries is 45 (fixed on branch, HADOOP-7932 or in > HADOOP-7397). For SASL, the number of retries is hard-coded to 15. Can be > increased, especially if the socket timeouts have been lowered. > ipc.client.connect.max.retries.on.timeouts > Default 45 > If you have HADOOP-7932, max number of retries on timeout. Counter is > different than ipc.client.connect.max.retries so if you mix the socket errors > you will get 55 retries with the default values. Could be lowered, once it is > available. With HADOOP-7397 ipc.client.connect.max.retries is reused so there > would be 10 tries. > dfs.client.block.write.retries > Default 3 > Number of tries for the client when writing a block. After a failure, will > connect to the namenode a get a new location, sending the list of the > datanodes already tried without success. Could be increased, especially if > the socket timeouts have been lowered. See HBASE-6490. > dfs.client.block.write.locateFollowingBlock.retries > Default 5 > Number of retries to the namenode when the client got > NotReplicatedYetException, i.e. the existing nodes of the
[jira] [Updated] (HBASE-11154) Document how to use Reverse Scan API
[ https://issues.apache.org/jira/browse/HBASE-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11154: Attachment: HBASE-11154.patch Fixed how I generated the patch > Document how to use Reverse Scan API > > > Key: HBASE-11154 > URL: https://issues.apache.org/jira/browse/HBASE-11154 > Project: HBase > Issue Type: Task > Components: documentation, Scanners >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11154.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[
https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995983#comment-13995983
]
Hadoop QA commented on HBASE-11137:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644518/HBASE-11137.01.patch
against trunk revision .
ATTACHMENT ID: 12644518
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.snapshot.TestExportSnapshot
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestTableMapReduceBase.testMultiRegionTable(TestTableMapReduceBase.java:96)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9508//console
This message is automatically generated.
> Add mapred.TableSnapshotInputFormat
> ---
>
> Key: HBASE-11137
> URL: https://issues.apache.org/jira/browse/HBASE-11137
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce, Performance
>Affects Versions: 0.98.0, 0.96.2
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: HBASE-11137.00.patch, HBASE-11137.01.patch
>
>
> We should have feature parity between mapreduce and mapred implementations.
> This is important for Hive.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11136) Add permission check to roll WAL writer
[
https://issues.apache.org/jira/browse/HBASE-11136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995333#comment-13995333
]
Jerry He commented on HBASE-11136:
--
There is no use for postRollWALWriter() now.
Is it customary to have both pre and post in there?
{code}
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] HBase . SUCCESS [ 1.712 s]
[INFO] HBase - Common SUCCESS [ 25.713 s]
[INFO] HBase - Protocol .. SUCCESS [ 0.265 s]
[INFO] HBase - Client SUCCESS [ 36.078 s]
[INFO] HBase - Hadoop Compatibility .. SUCCESS [ 6.542 s]
[INFO] HBase - Hadoop Two Compatibility .. SUCCESS [ 1.359 s]
[INFO] HBase - Prefix Tree ... SUCCESS [ 3.470 s]
[INFO] HBase - Server SUCCESS [43:07 min]
[INFO] HBase - Testing Util .. SUCCESS [ 1.276 s]
[INFO] HBase - Thrift SUCCESS [01:45 min]
[INFO] HBase - Shell . SUCCESS [ 1.054 s]
[INFO] HBase - Integration Tests . SUCCESS [ 1.267 s]
[INFO] HBase - Examples .. SUCCESS [ 1.064 s]
[INFO] HBase - Assembly .. SUCCESS [ 0.916 s]
[INFO]
[INFO] BUILD SUCCESS
{code}
> Add permission check to roll WAL writer
>
>
> Key: HBASE-11136
> URL: https://issues.apache.org/jira/browse/HBASE-11136
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, security
>Affects Versions: 0.96.2, 0.98.2
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 0.99.0
>
> Attachments: HBASE-11136-trunk-v1.patch
>
>
> Currently HBase provides HBaseAdmin.rollHLogWriter() and shell command to
> roll WAL on a region server. But no permission check is done on this
> operation in a secure cluster.
> We need to add permission check to prevent un-authorized user from running
> this operation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11154) Document how to use Reverse Scan API
[ https://issues.apache.org/jira/browse/HBASE-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995937#comment-13995937 ] Misty Stanley-Jones commented on HBASE-11154: - I'm not sure if I actually need more than this. Ideally, the Scan API would have a working example. I can't figure out where in the Ref Guide would be the appropriate place for a working example of reverse scanning, since there doesn't even seem to be a working example of scanning. > Document how to use Reverse Scan API > > > Key: HBASE-11154 > URL: https://issues.apache.org/jira/browse/HBASE-11154 > Project: HBase > Issue Type: Task > Components: documentation, Scanners >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11153) http webUI's should redirect to https when enabled
[ https://issues.apache.org/jira/browse/HBASE-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11153: - Labels: noob (was: ) > http webUI's should redirect to https when enabled > -- > > Key: HBASE-11153 > URL: https://issues.apache.org/jira/browse/HBASE-11153 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Affects Versions: 0.98.0 >Reporter: Nick Dimiduk >Priority: Minor > Labels: noob > > When configured to listen on https, we should redirect non-secure requests to > the appropriate port/protocol. Currently we respond with a 200 and no data, > which is perplexing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11128) Add -target option to ExportSnapshot to export with a different name
[ https://issues.apache.org/jira/browse/HBASE-11128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995640#comment-13995640 ] Hudson commented on HBASE-11128: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #289 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/289/]) HBASE-11128 Add -target option to ExportSnapshot to export with a different name (mbertozzi: rev 1593776) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java > Add -target option to ExportSnapshot to export with a different name > > > Key: HBASE-11128 > URL: https://issues.apache.org/jira/browse/HBASE-11128 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 0.99.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: HBASE-11128-v0.patch > > > Add a "-target" option to export the snapshot using a different name -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11123) Upgrade instructions from 0.94 to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995506#comment-13995506 ] Jean-Marc Spaggiari commented on HBASE-11123: - I confirm this statement. I test 0.94 => 0.98 migration as part of my release testing and it works well. > Upgrade instructions from 0.94 to 0.98 > -- > > Key: HBASE-11123 > URL: https://issues.apache.org/jira/browse/HBASE-11123 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Priority: Minor > > I cloned this from the 0.96 upgrade docs task. It was suggested that we need > upgrade instructions from 0.94 to 0.98. I will need source material to even > prioritize this. Assuming this is Minor. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995883#comment-13995883 ] Enis Soztutar commented on HBASE-11137: --- bq. HBASE-7987 looks like the major culprit Not sure backporting HBASE-7987 to 0.98 is a good idea. It is changing the snapshot file formats and layouts. > Add mapred.TableSnapshotInputFormat > --- > > Key: HBASE-11137 > URL: https://issues.apache.org/jira/browse/HBASE-11137 > Project: HBase > Issue Type: Improvement > Components: mapreduce, Performance >Affects Versions: 0.98.0, 0.96.2 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: HBASE-11137.00.patch, HBASE-11137.01.patch > > > We should have feature parity between mapreduce and mapred implementations. > This is important for Hive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11151) move tracing modules from hbase-server to hbase-common
[ https://issues.apache.org/jira/browse/HBASE-11151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995958#comment-13995958 ] Masatake Iwasaki commented on HBASE-11151: -- Thanks for your comment [~ndimiduk]. PerfEval seems worked with this patch dut to intra-project dependencies. I think TestFromClientSide should be left in hbase-server because it refers to modules in hbase-server and hbase-client. hbase-server depends on hbase-common (and hbase-client) but not vice versa. > move tracing modules from hbase-server to hbase-common > -- > > Key: HBASE-11151 > URL: https://issues.apache.org/jira/browse/HBASE-11151 > Project: HBase > Issue Type: Improvement > Components: documentation, util >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-11151-0.patch > > > Not only servers but also clients using tracing needs SpanReceiverHost. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11136) Add permission check to roll WAL writer
[ https://issues.apache.org/jira/browse/HBASE-11136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-11136: - Fix Version/s: 0.99.0 Status: Patch Available (was: Open) > Add permission check to roll WAL writer > > > Key: HBASE-11136 > URL: https://issues.apache.org/jira/browse/HBASE-11136 > Project: HBase > Issue Type: Improvement > Components: regionserver, security >Affects Versions: 0.98.2, 0.96.2 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-11136-trunk-v1.patch > > > Currently HBase provides HBaseAdmin.rollHLogWriter() and shell command to > roll WAL on a region server. But no permission check is done on this > operation in a secure cluster. > We need to add permission check to prevent un-authorized user from running > this operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10909) Abstract out ZooKeeper usage in HBase
[ https://issues.apache.org/jira/browse/HBASE-10909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-10909: Attachment: (was: HBaseConsensus.pdf) > Abstract out ZooKeeper usage in HBase > - > > Key: HBASE-10909 > URL: https://issues.apache.org/jira/browse/HBASE-10909 > Project: HBase > Issue Type: Umbrella > Components: Consensus, Zookeeper >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Attachments: HBase Consensus.pdf, HBaseConsensus.pdf, > HBaseConsensus.pdf > > > As some sort of follow-up or initial step towards HBASE-10296. > Whatever consensus algorithm/library may be the chosen, perhaps one of first > practical steps towards this goal would be to better abstract ZK-related API > and details, which are now throughout the codebase (mostly leaked throuth > ZkUtil, ZooKeeperWatcher and listeners). > This jira is umbrella for relevant subtasks. > As the design doc is in process of peer-review now, please use Google Doc > (linked below) instead of pdf. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10909) Abstract out ZooKeeper usage in HBase
[ https://issues.apache.org/jira/browse/HBASE-10909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-10909: Attachment: HBase Consensus.pdf Updated PDF version. design doc after numerous questions have been asked and discussed in the google doc. leaving ref to google doc in place.. > Abstract out ZooKeeper usage in HBase > - > > Key: HBASE-10909 > URL: https://issues.apache.org/jira/browse/HBASE-10909 > Project: HBase > Issue Type: Umbrella > Components: Consensus, Zookeeper >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Attachments: HBase Consensus.pdf, HBaseConsensus.pdf, > HBaseConsensus.pdf > > > As some sort of follow-up or initial step towards HBASE-10296. > Whatever consensus algorithm/library may be the chosen, perhaps one of first > practical steps towards this goal would be to better abstract ZK-related API > and details, which are now throughout the codebase (mostly leaked throuth > ZkUtil, ZooKeeperWatcher and listeners). > This jira is umbrella for relevant subtasks. > As the design doc is in process of peer-review now, please use Google Doc > (linked below) instead of pdf. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11069) Decouple region merging from ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov updated HBASE-11069: Status: Open (was: Patch Available) > Decouple region merging from ZooKeeper > -- > > Key: HBASE-11069 > URL: https://issues.apache.org/jira/browse/HBASE-11069 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Reporter: Sergey Soldatov > Attachments: HBASE-11069.patch > > > Region Merge should be decoupled from ZK. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HBASE-11135) Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file
[
https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994282#comment-13994282
]
stack edited comment on HBASE-11135 at 5/10/14 7:03 PM:
[~jeffreyz] has been reviewing offline. Comments that if CP is slow doing
pre/post append, then as is, we will slow throughput. But as he notes, this is
as it is in 0.98 and not really a way around it since cp, and wal listeners are
inline w/ the actual append.
In this patch over most of the appendNoSyncs to use new method (so
edit/sequence id is availble on return out of append). Changed the append to
use a latch. This gained me back a few percent... maybe 5%... but this patch
makes us slower appending. When 200 threads contending, about half slower
again (180seconds vs 280 seconds to complete test):
{code}
$ for i in 1 3 5 10 50 200; do for j in 1 2 3; do perf stat
./hbase-0.99.0-SNAPSHOT/bin/hbase --config /home/stack/conf_hbase
org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation -threads $i
-iterations 10 -keySize 50 -valueSize 100 &>
"/tmp/wpatch_latch_barrier.${i}.${j}.txt"; done; done
{code}
I tried then also using a cyclicbarrier in the SyncFuture for the handler
thread and the the sync'ing thread to coordinate around but no noticeable
difference (perhaps a little slower).
I'd be good committing this for now since it simplifies the unification of mvcc
and sequence id. Even when we are half slower again we are still about 0.98
throughput in trunk since the claim in HBASE-10156 is that it doubled our
throughput at 200 threads.
After the unification happens, we can come back here again to do see if we can
get perf back.
was (Author: stack):
[~jeffreyz] has been reviewing offline. Comments that if CP is slow doing
pre/post append, then as is, we will slow throughput. But as he notes, this is
as it is in 0.98 and not really a way around it since cp, and wal listeners are
inline w/ the actual append.
In this patch over most of the appendNoSyncs to use new method (so
edit/sequence id is availble on return out of append). Changed the append to
use a latch. This gained me back a few percent... maybe 5%... but this patch
makes us slower appending. When 200 threads contending, about 1/3rd slower
again (180seconds vs 280 seconds to complete test):
{code}
$ for i in 1 3 5 10 50 200; do for j in 1 2 3; do perf stat
./hbase-0.99.0-SNAPSHOT/bin/hbase --config /home/stack/conf_hbase
org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation -threads $i
-iterations 10 -keySize 50 -valueSize 100 &>
"/tmp/wpatch_latch_barrier.${i}.${j}.txt"; done; done
{code}
I tried then also using a cyclicbarrier in the SyncFuture for the handler
thread and the the sync'ing thread to coordinate around but no noticeable
difference (perhaps a little slower).
I'd be good committing this for now since it simplifies the unification of mvcc
and sequence id. Even when we are 1/3rd slower we are still ahead of 0.98 in
trunk since the claim in HBASE-10156 is that it doubled our throughput at 200
threads.
After the unification happens, we can come back here again to do see if we can
get perf back.
> Change region sequenceid generation so happens earlier in the append cycle
> rather than just before added to file
>
>
> Key: HBASE-11135
> URL: https://issues.apache.org/jira/browse/HBASE-11135
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: stack
>Assignee: stack
> Attachments: 11135.wip.txt, 11135v2.txt, 11135v5.txt
>
>
> Currently we assign the region edit/sequence id just before we put it in the
> WAL. We do it in the single thread that feeds from the ring buffer. Doing
> it at this point, we can ensure order, that the edits will be in the file in
> accordance w/ the ordering of the region sequence id.
> But the point at which region sequence id is assigned an edit is deep down in
> the WAL system and there is a lag between our putting an edit into the WAL
> system and the edit actually getting its edit/sequence id.
> This lag -- "late-binding" -- complicates the unification of mvcc and region
> sequence id, especially around async WAL writes (and, related, for no-WAL
> writes) -- the parent for this issue (For async, how you get the edit id in
> our system when the threads have all gone home -- unless you make them wait?)
> Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of
> getting the region sequence id near-immediately. We'll run two ringbuffers.
> The first will mesh all handler threads and the consumer will generate ids
> (we will have order on other side of this first ring buffer), and then i
[jira] [Updated] (HBASE-11154) Document how to use Reverse Scan API
[ https://issues.apache.org/jira/browse/HBASE-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11154: Status: Patch Available (was: Open) Patch attached, which adds some notes to the chapter about schema design pointing out the new reverse scan API > Document how to use Reverse Scan API > > > Key: HBASE-11154 > URL: https://issues.apache.org/jira/browse/HBASE-11154 > Project: HBase > Issue Type: Task > Components: documentation, Scanners >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11154.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11154) Document how to use Reverse Scan API
[ https://issues.apache.org/jira/browse/HBASE-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11154: Fix Version/s: (was: 0.98.3) Status: Open (was: Patch Available) > Document how to use Reverse Scan API > > > Key: HBASE-11154 > URL: https://issues.apache.org/jira/browse/HBASE-11154 > Project: HBase > Issue Type: Task > Components: documentation, Scanners >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11154.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11154) Document how to use Reverse Scan API
[ https://issues.apache.org/jira/browse/HBASE-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11154: Attachment: HBASE-11154.patch > Document how to use Reverse Scan API > > > Key: HBASE-11154 > URL: https://issues.apache.org/jira/browse/HBASE-11154 > Project: HBase > Issue Type: Task > Components: documentation, Scanners >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 0.98.3 > > Attachments: HBASE-11154.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11154) Document how to use Reverse Scan API
[ https://issues.apache.org/jira/browse/HBASE-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995929#comment-13995929 ] Misty Stanley-Jones commented on HBASE-11154: - Added the following note in the first two places above: Reverse Scan API https://issues.apache.org/jira/browse/HBASE-4811";>HBASE-4811 implements an API to scan a table or a range within a table in reverse, reducing the need to optimize your schema for forward or reverse scanning. This feature is available in HBase 0.98 and later. See https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setReversed%28boolean"; /> for more information. > Document how to use Reverse Scan API > > > Key: HBASE-11154 > URL: https://issues.apache.org/jira/browse/HBASE-11154 > Project: HBase > Issue Type: Task > Components: documentation, Scanners >Affects Versions: 0.98.2 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10909) Abstract out ZooKeeper usage in HBase
[ https://issues.apache.org/jira/browse/HBASE-10909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-10909: Affects Version/s: 0.99.0 > Abstract out ZooKeeper usage in HBase > - > > Key: HBASE-10909 > URL: https://issues.apache.org/jira/browse/HBASE-10909 > Project: HBase > Issue Type: Umbrella > Components: Consensus, Zookeeper >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Attachments: HBaseConsensus.pdf, HBaseConsensus.pdf, > HBaseConsensus.pdf > > > As some sort of follow-up or initial step towards HBASE-10296. > Whatever consensus algorithm/library may be the chosen, perhaps one of first > practical steps towards this goal would be to better abstract ZK-related API > and details, which are now throughout the codebase (mostly leaked throuth > ZkUtil, ZooKeeperWatcher and listeners). > This jira is umbrella for relevant subtasks. Design doc is attached, for > comments/questions there's a google doc linked. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11136) Add permission check to roll WAL writer
[ https://issues.apache.org/jira/browse/HBASE-11136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995578#comment-13995578 ] Jerry He commented on HBASE-11136: -- Hi, Ted Good question. I am not very clear regarding Permission.Action.CREATE with a global permission scope Some examples of only Permission.Action.ADMIN is balance, snapshot, stop rs. > Add permission check to roll WAL writer > > > Key: HBASE-11136 > URL: https://issues.apache.org/jira/browse/HBASE-11136 > Project: HBase > Issue Type: Improvement > Components: regionserver, security >Affects Versions: 0.96.2, 0.98.2 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-11136-trunk-v1.patch > > > Currently HBase provides HBaseAdmin.rollHLogWriter() and shell command to > roll WAL on a region server. But no permission check is done on this > operation in a secure cluster. > We need to add permission check to prevent un-authorized user from running > this operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11128) Add -target option to ExportSnapshot to export with a different name
[ https://issues.apache.org/jira/browse/HBASE-11128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995734#comment-13995734 ] Hudson commented on HBASE-11128: SUCCESS: Integrated in HBase-0.98 #304 (See [https://builds.apache.org/job/HBase-0.98/304/]) HBASE-11128 Add -target option to ExportSnapshot to export with a different name (mbertozzi: rev 1593776) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java > Add -target option to ExportSnapshot to export with a different name > > > Key: HBASE-11128 > URL: https://issues.apache.org/jira/browse/HBASE-11128 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 0.99.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: HBASE-11128-v0.patch > > > Add a "-target" option to export the snapshot using a different name -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11154) Document how to use Reverse Scan API
Misty Stanley-Jones created HBASE-11154: --- Summary: Document how to use Reverse Scan API Key: HBASE-11154 URL: https://issues.apache.org/jira/browse/HBASE-11154 Project: HBase Issue Type: Task Components: documentation, Scanners Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11119) Update ExportSnapShot to optionally not use a tmp file on external file system
[ https://issues.apache.org/jira/browse/HBASE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993393#comment-13993393 ] Hudson commented on HBASE-9: SUCCESS: Integrated in HBase-0.94-JDK7 #133 (See [https://builds.apache.org/job/HBase-0.94-JDK7/133/]) HBASE-9 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593340) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java > Update ExportSnapShot to optionally not use a tmp file on external file system > -- > > Key: HBASE-9 > URL: https://issues.apache.org/jira/browse/HBASE-9 > Project: HBase > Issue Type: Improvement > Components: snapshots >Reporter: Ted Malaska >Assignee: Ted Malaska >Priority: Minor > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: HBASE-9.patch > > > There are FileSystem like S3 where renaming is extremely expensive. This > patch will add a parameter that says something like > use.tmp.folder > It will be defaulted to true. So default behavior is the same. If false is > set them the files will land in the final destination with no need for a > rename. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10357) Failover RPC's for scans
[ https://issues.apache.org/jira/browse/HBASE-10357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-10357: Fix Version/s: hbase-10070 > Failover RPC's for scans > > > Key: HBASE-10357 > URL: https://issues.apache.org/jira/browse/HBASE-10357 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Enis Soztutar > Fix For: 0.99.0, hbase-10070 > > Attachments: 10357-1.txt, 10357-2.txt, 10357-3.2.txt, 10357-3.txt, > 10357-4.2.txt, 10357-4.3.txt, 10357-4.txt > > > This is extension of HBASE-10355 to add failover support for scans. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10357) Failover RPC's for scans
[ https://issues.apache.org/jira/browse/HBASE-10357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-10357: Attachment: 10357-4.3.1.txt This is without the pom.xml change. > Failover RPC's for scans > > > Key: HBASE-10357 > URL: https://issues.apache.org/jira/browse/HBASE-10357 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Enis Soztutar > Fix For: 0.99.0, hbase-10070 > > Attachments: 10357-1.txt, 10357-2.txt, 10357-3.2.txt, 10357-3.txt, > 10357-4.2.txt, 10357-4.3.1.txt, 10357-4.3.txt, 10357-4.txt > > > This is extension of HBASE-10355 to add failover support for scans. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11098) Improve documentation around our blockcache options
[ https://issues.apache.org/jira/browse/HBASE-11098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995830#comment-13995830 ] Nick Dimiduk commented on HBASE-11098: -- Docstrings looks good to me. Maybe we need an example configuration to help get folks started, but I guess that's for the book, not docstrings. What's up with concurrent map changes? These were intentional? > Improve documentation around our blockcache options > --- > > Key: HBASE-11098 > URL: https://issues.apache.org/jira/browse/HBASE-11098 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack > Attachments: 11098.txt, 11098v2.txt > > > Adding as a subtask on [~ndimiduk] necessary cleanup. Trying to make sense > of this stuff I started adding doc (package-info files, javadoc). I see this > as complementary to the parent task work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11143) ageOfLastShippedOp metric is confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995895#comment-13995895 ] Lars Hofhansl commented on HBASE-11143: --- [~apurtell], you OK with this in 0.98? It's just a new metric. > ageOfLastShippedOp metric is confusing > -- > > Key: HBASE-11143 > URL: https://issues.apache.org/jira/browse/HBASE-11143 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.99.0, 0.94.20, 0.98.3 > > Attachments: 11143-0.94-v2.txt, 11143-0.94.txt, 11143-trunk.txt > > > We are trying to report on replication lag and find that there is no good > single metric to do that. > ageOfLastShippedOp is close, but unfortunately it is increased even when > there is nothing to ship on a particular RegionServer. > I would like discuss a few options here: > Add a new metric: replicationQueueTime (or something) with the above meaning. > I.e. if we have something to ship we set the age of that last shipped edit, > if we fail we increment that last time (just like we do now). But if there is > nothing to replicate we set it to current time (and hence that metric is > reported to close to 0). > Alternatively we could change the meaning of ageOfLastShippedOp to mean to do > that. That might lead to surprises, but the current behavior is clearly weird > when there is nothing to replicate. > Comments? [~jdcryans], [~stack]. > If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11136) Add permission check to roll WAL writer
[ https://issues.apache.org/jira/browse/HBASE-11136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995314#comment-13995314 ] Ted Yu commented on HBASE-11136: What's the purpose of adding postRollWALWriter() hook ? Can you post the result for unit test suite ? QA bot is down. > Add permission check to roll WAL writer > > > Key: HBASE-11136 > URL: https://issues.apache.org/jira/browse/HBASE-11136 > Project: HBase > Issue Type: Improvement > Components: regionserver, security >Affects Versions: 0.96.2, 0.98.2 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-11136-trunk-v1.patch > > > Currently HBase provides HBaseAdmin.rollHLogWriter() and shell command to > roll WAL on a region server. But no permission check is done on this > operation in a secure cluster. > We need to add permission check to prevent un-authorized user from running > this operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11137: - Attachment: HBASE-11137.01.patch Updating patch: fix line lengths and add support for both namespaces to IntegrationTestTableSnapshotInputFormat. Looks like trunk has enjoyed some progress around snapshots, so rebasing to 0.98 and 0.96 looks messy. HBASE-7987 looks like the major culprit. I'd prefer to see this backported before rebasing the patch (otherwise, I basically repeat the refactor manually for each branch). > Add mapred.TableSnapshotInputFormat > --- > > Key: HBASE-11137 > URL: https://issues.apache.org/jira/browse/HBASE-11137 > Project: HBase > Issue Type: Improvement > Components: mapreduce, Performance >Affects Versions: 0.98.0, 0.96.2 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: HBASE-11137.00.patch, HBASE-11137.01.patch > > > We should have feature parity between mapreduce and mapred implementations. > This is important for Hive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10985) Decouple Split Transaction from Zookeeper
[ https://issues.apache.org/jira/browse/HBASE-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov updated HBASE-10985: Status: Patch Available (was: Open) > Decouple Split Transaction from Zookeeper > - > > Key: HBASE-10985 > URL: https://issues.apache.org/jira/browse/HBASE-10985 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Reporter: Sergey Soldatov > Attachments: HBASE-10985.patch, HBASE-10985.patch, HBASE-10985.patch, > HBASE_10985-v2.patch, HBASE_10985-v3.patch, HBASE_10985-v4.patch > > > As part of HBASE-10296 SplitTransaction should be decoupled from Zookeeper. > This is an initial patch for review. At the moment the consensus provider > placed directly to SplitTransaction to minimize affected code. In the ideal > world it should be done in HServer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11069) Decouple region merging from ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov updated HBASE-11069: Attachment: HBASE_11069-v2.patch Rebased to the latest trunk and did better decoupling. > Decouple region merging from ZooKeeper > -- > > Key: HBASE-11069 > URL: https://issues.apache.org/jira/browse/HBASE-11069 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Reporter: Sergey Soldatov > Attachments: HBASE-11069.patch, HBASE_11069-v2.patch > > > Region Merge should be decoupled from ZK. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11118) non environment variable solution for "IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995264#comment-13995264 ] Nick Dimiduk commented on HBASE-8: -- I took the shade question to [user@maven|http://www.mail-archive.com/users@maven.apache.org/msg133791.html]. Maybe some helpful insight will surface. > non environment variable solution for "IllegalAccessError: class > com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass > com.google.protobuf.LiteralByteString" > -- > > Key: HBASE-8 > URL: https://issues.apache.org/jira/browse/HBASE-8 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.2 >Reporter: André Kelpe >Priority: Blocker > Fix For: 0.99.0, 0.98.3 > > Attachments: shade_attempt.patch > > > I am running into the problem described in > https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a > newer version within cascading.hbase > (https://github.com/cascading/cascading.hbase). > One of the features of cascading.hbase is that you can use it from lingual > (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. > lingual has a notion of providers, which are fat jars that we pull down > dynamically at runtime. Those jars give users the ability to talk to any > system or format from SQL. They are added to the classpath programmatically > before we submit jobs to a hadoop cluster. > Since lingual does not know upfront , which providers are going to be used in > a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really > clunky and breaks the ease of use we had before. No other provider requires > this right now. > It would be great to have a programmatical way to fix this, when using fat > jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11133) Add an option to skip snapshot verification after Export
[ https://issues.apache.org/jira/browse/HBASE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995641#comment-13995641 ] Hudson commented on HBASE-11133: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #289 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/289/]) HBASE-11133 Add an option to skip snapshot verification after ExportSnapshot (mbertozzi: rev 1593770) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java > Add an option to skip snapshot verification after Export > > > Key: HBASE-11133 > URL: https://issues.apache.org/jira/browse/HBASE-11133 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 0.99.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 0.99.0, 0.96.3, 0.98.3 > > Attachments: HBASE-11133-v0.patch, HBASE-11133-v1.patch > > > Add a "-skip-dst-verify" option to skip snapshot verification after Export -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10357) Failover RPC's for scans
[ https://issues.apache.org/jira/browse/HBASE-10357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-10357: Attachment: 10357-4.3.txt Addresses the comments last raised. The main change is that the "close" of the unneeded scanners are queued up in the executor service to make sure all scanners (primary or otherwise) are closed when not needed. Also a semantic change - when a scanner is obtained, the underlying table shouldn't have been closed. Updated some unit tests to reflect this. So for example, one pattern is - one would create a table, do a bunch of puts, and then close the table. Subsequently, a scanner would be obtained from that table. In the approach on this jira, since the scanner uses the pool always, if the table is closed, the underlying pool would be shut down as well. This would lead to issues. In earlier versions of the patch, I would create a new pool if the passed pool was shut down but it makes handling the closing of the pool, etc. an issue. I think for 1.0 we should make the change that a table shouldn't be closed if we want to invoke methods on that table. > Failover RPC's for scans > > > Key: HBASE-10357 > URL: https://issues.apache.org/jira/browse/HBASE-10357 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Enis Soztutar > Fix For: 0.99.0 > > Attachments: 10357-1.txt, 10357-2.txt, 10357-3.2.txt, 10357-3.txt, > 10357-4.2.txt, 10357-4.3.txt, 10357-4.txt > > > This is extension of HBASE-10355 to add failover support for scans. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7987) Snapshot Manifest file instead of multiple empty files
[
https://issues.apache.org/jira/browse/HBASE-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995793#comment-13995793
]
Nick Dimiduk commented on HBASE-7987:
-
Hi [~mbertozzi]. Any plan for a backport to 0.98? I'm seeing some conflict in
TableSnapshotInputFormat while backporting my patch on HBASE-11137. If you
intend a backport, I'd rather not further complicate.
> Snapshot Manifest file instead of multiple empty files
> --
>
> Key: HBASE-7987
> URL: https://issues.apache.org/jira/browse/HBASE-7987
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 0.99.0
>
> Attachments: HBASE-7987-v0.patch, HBASE-7987-v1.patch,
> HBASE-7987-v2.patch, HBASE-7987-v2.sketch, HBASE-7987-v3.patch,
> HBASE-7987-v4.patch, HBASE-7987-v5.patch, HBASE-7987-v6.patch,
> HBASE-7987.sketch
>
>
> Currently taking a snapshot means creating one empty file for each file in
> the source table directory, plus copying the .regioninfo file for each
> region, the table descriptor file and a snapshotInfo file.
> during the restore or snapshot verification we traverse the filesystem
> (fs.listStatus()) to find the snapshot files, and we open the .regioninfo
> files to get the information.
> to avoid hammering the NameNode and having lots of empty files, we can use a
> manifest file that contains the list of files and information that we need.
> To keep the RS parallelism that we have, each RS can write its own manifest.
> {code}
> message SnapshotDescriptor {
> required string name;
> optional string table;
> optional int64 creationTime;
> optional Type type;
> optional int32 version;
> }
> message SnapshotRegionManifest {
> optional int32 version;
> required RegionInfo regionInfo;
> repeated FamilyFiles familyFiles;
> message StoreFile {
> required string name;
> optional Reference reference;
> }
> message FamilyFiles {
> required bytes familyName;
> repeated StoreFile storeFiles;
> }
> }
> {code}
> {code}
> /hbase/.snapshot/
> /hbase/.snapshot//snapshotInfo
> /hbase/.snapshot//
> /hbase/.snapshot///tableInfo
> /hbase/.snapshot///regionManifest(.n)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11108) Split ZKTable into interface and implementation
[
https://issues.apache.org/jira/browse/HBASE-11108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995792#comment-13995792
]
Hadoop QA commented on HBASE-11108:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644478/HBASE-11108.patch
against trunk revision .
ATTACHMENT ID: 12644478
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 37 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9506//console
This message is automatically generated.
> Split ZKTable into interface and implementation
> ---
>
> Key: HBASE-11108
> URL: https://issues.apache.org/jira/browse/HBASE-11108
> Project: HBase
> Issue Type: Sub-task
> Components: Consensus, Zookeeper
>Affects Versions: 0.99.0
>Reporter: Konstantin Boudnik
>Assignee: Mikhail Antonov
> Attachments: HBASE-11108.patch, HBASE-11108.patch, HBASE-11108.patch
>
>
> In HBASE-11071 we are trying to split admin handlers away from ZK. However, a
> ZKTable instance is being used in multiple places, hence it would be
> beneficial to hide its implementation behind a well defined interface.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[
https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995790#comment-13995790
]
Hadoop QA commented on HBASE-11137:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644212/HBASE-11137.00.patch
against trunk revision .
ATTACHMENT ID: 12644212
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+public TableSnapshotRegionSplit(HTableDescriptor htd, HRegionInfo
regionInfo, List locations) {
+ public static void setInput(JobConf job, String snapshotName, Path
restoreDir) throws IOException {
+public TableSnapshotRegionSplit(HTableDescriptor htd, HRegionInfo
regionInfo, List locations) {
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9504//console
This message is automatically generated.
> Add mapred.TableSnapshotInputFormat
> ---
>
> Key: HBASE-11137
> URL: https://issues.apache.org/jira/browse/HBASE-11137
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce, Performance
>Affects Versions: 0.98.0, 0.96.2
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: HBASE-11137.00.patch
>
>
> We should have feature parity between mapreduce and mapred implementations.
> This is important for Hive.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-10235) Backport HBASE-10226 (Namespace grants not always checked) to 0.96
[ https://issues.apache.org/jira/browse/HBASE-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-10235. Resolution: Won't Fix Assignee: (was: Andrew Purtell) > Backport HBASE-10226 (Namespace grants not always checked) to 0.96 > -- > > Key: HBASE-10235 > URL: https://issues.apache.org/jira/browse/HBASE-10235 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.2 >Reporter: Andrew Purtell > > Namespace level grants are not checked by the AccessController in 0.96 also. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11135) Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file
[
https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995591#comment-13995591
]
Hadoop QA commented on HBASE-11135:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644431/11135v5.txt
against trunk revision .
ATTACHMENT ID: 12644431
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 5
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+i.visitLogEntryBeforeWrite(entry.getHTableDescriptor(),
entry.getKey(), entry.getEdit());
+ * @deprecated Use {@link #appendNoSync(HRegionInfo, HLogKey, WALEdit,
HTableDescriptor, AtomicLong, boolean)}
+fsUtils.recoverFileLease(((HFileSystem)fs).getBackingFs(), p,
TEST_UTIL.getConfiguration(), null);
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9502//console
This message is automatically generated.
> Change region sequenceid generation so happens earlier in the append cycle
> rather than just before added to file
>
>
> Key: HBASE-11135
> URL: https://issues.apache.org/jira/browse/HBASE-11135
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: stack
>Assignee: stack
> Attachments: 11135.wip.txt, 11135v2.txt, 11135v5.txt, 11135v5.txt,
> 11135v5.txt
>
>
> Currently we assign the region edit/sequence id just before we put it in the
> WAL. We do it in the single thread that feeds from the ring buffer. Doing
> it at this point, we can ensure order, that the edits will be in the file in
> accordance w/ the ordering of the region sequence id.
> But the point at which region sequence id is assigned an edit is deep down in
> the WAL system and there is a lag between our putting an edit into the WAL
> system and the edit actually getting its edit/sequence id.
> This lag -- "late-binding" -- complicates the unification of mvcc and region
> sequence id, especially around async WAL writes (and, related, for no-WAL
> writes) -- the parent for this issue (For async, how you get the edit id in
> our system when the threads have all gone home -- unless you make them wait?)
> Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of
> getting the region sequence id near-immediately. We'll run two ringbuffers.
> The first will mesh all han
[jira] [Commented] (HBASE-11143) ageOfLastShippedOp metric is confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995661#comment-13995661 ] Lars Hofhansl commented on HBASE-11143: --- [~stack], the new metric? Sure. I suppose our message at this point is to upgrade from 0.94 to 0.98, right? OK... Lemme me put it in 0.94, 0.98, and trunk. > ageOfLastShippedOp metric is confusing > -- > > Key: HBASE-11143 > URL: https://issues.apache.org/jira/browse/HBASE-11143 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: 11143-0.94-v2.txt, 11143-0.94.txt, 11143-trunk.txt > > > We are trying to report on replication lag and find that there is no good > single metric to do that. > ageOfLastShippedOp is close, but unfortunately it is increased even when > there is nothing to ship on a particular RegionServer. > I would like discuss a few options here: > Add a new metric: replicationQueueTime (or something) with the above meaning. > I.e. if we have something to ship we set the age of that last shipped edit, > if we fail we increment that last time (just like we do now). But if there is > nothing to replicate we set it to current time (and hence that metric is > reported to close to 0). > Alternatively we could change the meaning of ageOfLastShippedOp to mean to do > that. That might lead to surprises, but the current behavior is clearly weird > when there is nothing to replicate. > Comments? [~jdcryans], [~stack]. > If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11140) LocalHBaseCluster should create ConsensusProvider per each server
[ https://issues.apache.org/jira/browse/HBASE-11140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11140: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~mantonov] > LocalHBaseCluster should create ConsensusProvider per each server > - > > Key: HBASE-11140 > URL: https://issues.apache.org/jira/browse/HBASE-11140 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11140.patch > > > Right now there's a bug there when single ConsensusProvider instance is > shared across all threads running region servers and masters within > processes, which breaks certain tests in patches which used to pass > successfully before. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10985) Decouple Split Transaction from Zookeeper
[
https://issues.apache.org/jira/browse/HBASE-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995720#comment-13995720
]
Mikhail Antonov commented on HBASE-10985:
-
Some small comments:
- unused import of ZkSplitTransactionConsensus in AssignmentManager,
znodeVersion in SplitTransaction isn't used (and shouldn't be needed?)
- Why do we need to pass in Server instance to methods of
SplitTransactionConsensus? We can retrieve it from the consensusProvider
reference kept in ZkSplitTransactionConsensus instance (also the split
consensus could better be field in SplitTransaction class I guess, shouldn't
need to retrieve it from ConsensusProvider each time?)
- fields consensus and watcher in ZkSplitTransactionConsensus are unused
- can we have the methods on split consensus to not use zk-related terms like
"transition node", or it's hard because of the zk is too tightly integrated
here? Ideally guys glancing thru code other than consensus impl shall not need
to know that. E.g. we could have it renamed like: (transitionSplittingNode ->
preCommitByRegionServer; transitionZkNode -> commitByRegionServer,
createNodeSplitting -> startTxnOnRegionServer, getNode ->
heartbeatRegionSplitting - do you think this naming accurately reflects the
actual meaning?), also - could we not return values like -1 coming from ZK
world from methods of consensus?
- to eliminate KeeperException from SplitTransaction, could we move some
portion of post splitting steps in openDaughers() like this:
{code}
ry {
// add 2nd daughter first (see HBASE-4335)
services.postOpenDeployTasks(b, server.getCatalogTracker());
// Should add it to OnlineRegions
services.addToOnlineRegions(b);
services.postOpenDeployTasks(a, server.getCatalogTracker());
services.addToOnlineRegions(a);
{code}
to separate consensus method? This way we can remove KeeperException from
imports.
> Decouple Split Transaction from Zookeeper
> -
>
> Key: HBASE-10985
> URL: https://issues.apache.org/jira/browse/HBASE-10985
> Project: HBase
> Issue Type: Sub-task
> Components: Consensus, Zookeeper
>Reporter: Sergey Soldatov
> Attachments: HBASE-10985.patch, HBASE-10985.patch, HBASE-10985.patch,
> HBASE_10985-v2.patch, HBASE_10985-v3.patch, HBASE_10985-v4.patch
>
>
> As part of HBASE-10296 SplitTransaction should be decoupled from Zookeeper.
> This is an initial patch for review. At the moment the consensus provider
> placed directly to SplitTransaction to minimize affected code. In the ideal
> world it should be done in HServer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11133) Add an option to skip snapshot verification after Export
[ https://issues.apache.org/jira/browse/HBASE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995735#comment-13995735 ] Hudson commented on HBASE-11133: SUCCESS: Integrated in HBase-0.98 #304 (See [https://builds.apache.org/job/HBase-0.98/304/]) HBASE-11133 Add an option to skip snapshot verification after ExportSnapshot (mbertozzi: rev 1593770) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java > Add an option to skip snapshot verification after Export > > > Key: HBASE-11133 > URL: https://issues.apache.org/jira/browse/HBASE-11133 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 0.99.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 0.99.0, 0.96.3, 0.98.3 > > Attachments: HBASE-11133-v0.patch, HBASE-11133-v1.patch > > > Add a "-skip-dst-verify" option to skip snapshot verification after Export -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11143) ageOfLastShippedOp metric is confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995638#comment-13995638 ] stack commented on HBASE-11143: --- Suggest do not put in 0.96 unless someone asks explicitly for it. > ageOfLastShippedOp metric is confusing > -- > > Key: HBASE-11143 > URL: https://issues.apache.org/jira/browse/HBASE-11143 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: 11143-0.94-v2.txt, 11143-0.94.txt, 11143-trunk.txt > > > We are trying to report on replication lag and find that there is no good > single metric to do that. > ageOfLastShippedOp is close, but unfortunately it is increased even when > there is nothing to ship on a particular RegionServer. > I would like discuss a few options here: > Add a new metric: replicationQueueTime (or something) with the above meaning. > I.e. if we have something to ship we set the age of that last shipped edit, > if we fail we increment that last time (just like we do now). But if there is > nothing to replicate we set it to current time (and hence that metric is > reported to close to 0). > Alternatively we could change the meaning of ageOfLastShippedOp to mean to do > that. That might lead to surprises, but the current behavior is clearly weird > when there is nothing to replicate. > Comments? [~jdcryans], [~stack]. > If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11151) move tracing modules from hbase-server to hbase-common
[
https://issues.apache.org/jira/browse/HBASE-11151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995671#comment-13995671
]
Hadoop QA commented on HBASE-11151:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644320/HBASE-11151-0.patch
against trunk revision .
ATTACHMENT ID: 12644320
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9503//console
This message is automatically generated.
> move tracing modules from hbase-server to hbase-common
> --
>
> Key: HBASE-11151
> URL: https://issues.apache.org/jira/browse/HBASE-11151
> Project: HBase
> Issue Type: Improvement
> Components: documentation, util
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Attachments: HBASE-11151-0.patch
>
>
> Not only servers but also clients using tracing needs SpanReceiverHost.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11135) Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file
[ https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11135: -- Attachment: 11135v6.txt The TestLogRolling was a real issue (I love unit tests). TestMultiParallel is not me. This has just started failing generally. Fix line lengths. > Change region sequenceid generation so happens earlier in the append cycle > rather than just before added to file > > > Key: HBASE-11135 > URL: https://issues.apache.org/jira/browse/HBASE-11135 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Assignee: stack > Attachments: 11135.wip.txt, 11135v2.txt, 11135v5.txt, 11135v5.txt, > 11135v5.txt, 11135v6.txt > > > Currently we assign the region edit/sequence id just before we put it in the > WAL. We do it in the single thread that feeds from the ring buffer. Doing > it at this point, we can ensure order, that the edits will be in the file in > accordance w/ the ordering of the region sequence id. > But the point at which region sequence id is assigned an edit is deep down in > the WAL system and there is a lag between our putting an edit into the WAL > system and the edit actually getting its edit/sequence id. > This lag -- "late-binding" -- complicates the unification of mvcc and region > sequence id, especially around async WAL writes (and, related, for no-WAL > writes) -- the parent for this issue (For async, how you get the edit id in > our system when the threads have all gone home -- unless you make them wait?) > Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of > getting the region sequence id near-immediately. We'll run two ringbuffers. > The first will mesh all handler threads and the consumer will generate ids > (we will have order on other side of this first ring buffer), and then if > async or no sync, we will just let the threads return ... updating mvcc just > before we let them go. All other calls will go up on to the second ring > buffer to be serviced as now (batching, distribution out among the sync'ing > threads). The first rb will have no friction and should turn at fast rates > compared to the second. There should not be noticeable slowdown nor do I > foresee this refactor intefering w/ our multi-WAL plans. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10513) Provide user documentation for region replicas
[ https://issues.apache.org/jira/browse/HBASE-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-10513: -- Attachment: timeline_consitency.png hbase-10513_v1.patch Updated the doc for addressing the comments, and adding some more clarity. Attaching the docbook version. I'll commit this version. > Provide user documentation for region replicas > -- > > Key: HBASE-10513 > URL: https://issues.apache.org/jira/browse/HBASE-10513 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.99.0 > > Attachments: UserdocumentationforHBASE-10070.pdf, > hbase-10513_v1.patch, timeline_consitency.png > > > We need some documentation for the feature introduced in HBASE-10070. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11108) Split ZKTable into interface and implementation
[ https://issues.apache.org/jira/browse/HBASE-11108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11108: Status: Open (was: Patch Available) > Split ZKTable into interface and implementation > --- > > Key: HBASE-11108 > URL: https://issues.apache.org/jira/browse/HBASE-11108 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Affects Versions: 0.99.0 >Reporter: Konstantin Boudnik >Assignee: Mikhail Antonov > Attachments: HBASE-11108.patch, HBASE-11108.patch, HBASE-11108.patch > > > In HBASE-11071 we are trying to split admin handlers away from ZK. However, a > ZKTable instance is being used in multiple places, hence it would be > beneficial to hide its implementation behind a well defined interface. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10942) support parallel request cancellation for multi-get
[ https://issues.apache.org/jira/browse/HBASE-10942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995613#comment-13995613 ] Sergey Shelukhin commented on HBASE-10942: -- isReplica there is only used for logging; fixed. Will fix rest and update the patch > support parallel request cancellation for multi-get > --- > > Key: HBASE-10942 > URL: https://issues.apache.org/jira/browse/HBASE-10942 > Project: HBase > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-10070 > > Attachments: HBASE-10942.01.patch, HBASE-10942.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11108) Split ZKTable into interface and implementation
[ https://issues.apache.org/jira/browse/HBASE-11108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11108: Attachment: HBASE-11108.patch fixed broken test, patch tests on local machine. > Split ZKTable into interface and implementation > --- > > Key: HBASE-11108 > URL: https://issues.apache.org/jira/browse/HBASE-11108 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Affects Versions: 0.99.0 >Reporter: Konstantin Boudnik >Assignee: Mikhail Antonov > Attachments: HBASE-11108.patch, HBASE-11108.patch, HBASE-11108.patch > > > In HBASE-11071 we are trying to split admin handlers away from ZK. However, a > ZKTable instance is being used in multiple places, hence it would be > beneficial to hide its implementation behind a well defined interface. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11135) Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file
[ https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11135: -- Attachment: 11135.wip.txt wip Includes doc cleanup from HBASE-11109. Does not do double ring buffer. Instead, when we appendNoSync, we just wait for the append to complete before returning (early-binding instead of late-binding). Also changed signature for HLog#appendNoSync so it takes a HLogKey. We stick the region edit/sequence id into the HLogKey so it is available on return from appendNoSync. Region sequence id is now always updated by the WAL system (Need to finish getting an id -- plumb it through). Should fix HBASE-11109. This patch includes the HBASE-11109 tests. Need to see how much perf we are losing doing this wait on append and then a wait on sync. Need to do cleanup too. > Change region sequenceid generation so happens earlier in the append cycle > rather than just before added to file > > > Key: HBASE-11135 > URL: https://issues.apache.org/jira/browse/HBASE-11135 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Assignee: stack > Attachments: 11135.wip.txt, 11135v2.txt > > > Currently we assign the region edit/sequence id just before we put it in the > WAL. We do it in the single thread that feeds from the ring buffer. Doing > it at this point, we can ensure order, that the edits will be in the file in > accordance w/ the ordering of the region sequence id. > But the point at which region sequence id is assigned an edit is deep down in > the WAL system and there is a lag between our putting an edit into the WAL > system and the edit actually getting its edit/sequence id. > This lag -- "late-binding" -- complicates the unification of mvcc and region > sequence id, especially around async WAL writes (and, related, for no-WAL > writes) -- the parent for this issue (For async, how you get the edit id in > our system when the threads have all gone home -- unless you make them wait?) > Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of > getting the region sequence id near-immediately. We'll run two ringbuffers. > The first will mesh all handler threads and the consumer will generate ids > (we will have order on other side of this first ring buffer), and then if > async or no sync, we will just let the threads return ... updating mvcc just > before we let them go. All other calls will go up on to the second ring > buffer to be serviced as now (batching, distribution out among the sync'ing > threads). The first rb will have no friction and should turn at fast rates > compared to the second. There should not be noticeable slowdown nor do I > foresee this refactor intefering w/ our multi-WAL plans. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-1811) Snapshot HFile and region statistics at compaction time and make info available to clients
[ https://issues.apache.org/jira/browse/HBASE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995307#comment-13995307 ] Jesse Yates commented on HBASE-1811: Yeah, 7958 looks like a dup (timeline wise). Looks like good ideas always come back around :) We could make 7958 be the scan-time infrastructure and this one to be adding comprehensive stats? > Snapshot HFile and region statistics at compaction time and make info > available to clients > -- > > Key: HBASE-1811 > URL: https://issues.apache.org/jira/browse/HBASE-1811 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Priority: Minor > > Consider snapshotting HFile and region statistics at major and minor > compaction time and making the info available to clients: > * Key statistics > ** cardinality > ** length avg/min/max/stdev > ** information content measure (entropy, etc.) > ** histogram > etc. > * Value statistics > ** length avg/min/max/stdev > ** information content measure (entropy, etc.) > ** histogram > etc. > * Region statistics > ** density estimation > ** KV count > ** total storage size (on disk) > ** total storage size (uncompressed) > etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11112) PerformanceEvaluation should document --multiGet option on its printUsage.
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-2: Status: Open (was: Patch Available) > PerformanceEvaluation should document --multiGet option on its printUsage. > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Bug > Components: documentation, Performance >Affects Versions: 0.98.2 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-2-v0-trunk.patch, HBASE-2-v1-trunk.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11098) Improve documentation around our blockcache options
[ https://issues.apache.org/jira/browse/HBASE-11098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995343#comment-13995343 ] stack commented on HBASE-11098: --- [~ndimiduk] Mind if I commit this as is Nick (I'll not do the file removes, just the doc fixup?) I'd like to get this in and backport. It is just javadoc to help folks trying to navigate in here. > Improve documentation around our blockcache options > --- > > Key: HBASE-11098 > URL: https://issues.apache.org/jira/browse/HBASE-11098 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack > Attachments: 11098.txt, 11098v2.txt > > > Adding as a subtask on [~ndimiduk] necessary cleanup. Trying to make sense > of this stuff I started adding doc (package-info files, javadoc). I see this > as complementary to the parent task work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked
[
https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995235#comment-13995235
]
Ted Yu commented on HBASE-11096:
{code}
+LOG.error("on RegionCoprocessorEnvironment!!", new IOException(
+"in RegionCoprocessorEnvironment"));
{code}
Is your intention of creating IOException just for record keeping ?
Looks like IOException doesn't have to be created.
> stop method of Master and RegionServer coprocessor is not invoked
> --
>
> Key: HBASE-11096
> URL: https://issues.apache.org/jira/browse/HBASE-11096
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.2, 0.98.1, 0.94.19
>Reporter: Qiang Tian
>Assignee: Qiang Tian
> Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3
>
> Attachments: HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v0.patch,
> HBASE-11096-trunk-v1.patch, HBASE-11096-trunk-v2.patch
>
>
> the stop method of coprocessor specified by
> "hbase.coprocessor.master.classes" and
> "hbase.coprocessor.regionserver.classes" is not invoked.
> If coprocessor allocates OS resources, it could cause master/regionserver
> resource leak or hang during exit.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-2834) Deferred deletes
[
https://issues.apache.org/jira/browse/HBASE-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-2834.
---
Resolution: Duplicate
> Deferred deletes
>
>
> Key: HBASE-2834
> URL: https://issues.apache.org/jira/browse/HBASE-2834
> Project: HBase
> Issue Type: New Feature
>Reporter: Andrew Purtell
>
> Tangentally mentioned in a blog post, James Hamilton talks about deferred
> deletes:
> {quote}
> If you have an application error, administrative error, or database
> implementation bug that losses data, then it is simply gone unless you have
> an offline copy. This, by the way, is why I'm a big fan of deferred delete.
> This is a technique where deleted items are marked as deleted but not garbage
> collected until some days or preferably weeks later. Deferred delete is not
> full protection but it has saved my butt more than once and I'm a believer.
> See On Designing and Deploying Internet-Scale Services
> (http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf) for more detail.
> {quote}
> (See
> http://perspectives.mvdirona.com/2010/04/07/StonebrakerOnCAPTheoremAndDatabases.aspx)
> Because deletes -- at least, after the initial write has been flushed from
> memstore -- are tombstones, deferred delete in HBase could be supported if
> somehow tombstones could be invalidated, an undelete operation in effect.
> This could be accomplished by adding support for tombstones for deletes.
> Would complicate major compaction but otherwise not touch much. A typical use
> case might be "resurrect any data deleted from _ts1_ to _ts2_ ", a period of
> 4 hours when an application error was operative. In this case a new write
> would be issued to the table that is a tombstone covering any deletes over
> that period of time. Users would defer major compactions until safe
> checkpoint periods.
> Such guarantees could optionally be extended to the memstoe by using
> tombstones there as well. But it would probably be sufficient to provide
> guidance that forcing a flush is necessary to insure edits are persisted in
> a way that allows for undeletion.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11143) ageOfLastShippedOp metric is confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995525#comment-13995525 ] Jean-Daniel Cryans commented on HBASE-11143: I'm +1 for the trunk patch too. > ageOfLastShippedOp metric is confusing > -- > > Key: HBASE-11143 > URL: https://issues.apache.org/jira/browse/HBASE-11143 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: 11143-0.94-v2.txt, 11143-0.94.txt, 11143-trunk.txt > > > We are trying to report on replication lag and find that there is no good > single metric to do that. > ageOfLastShippedOp is close, but unfortunately it is increased even when > there is nothing to ship on a particular RegionServer. > I would like discuss a few options here: > Add a new metric: replicationQueueTime (or something) with the above meaning. > I.e. if we have something to ship we set the age of that last shipped edit, > if we fail we increment that last time (just like we do now). But if there is > nothing to replicate we set it to current time (and hence that metric is > reported to close to 0). > Alternatively we could change the meaning of ageOfLastShippedOp to mean to do > that. That might lead to surprises, but the current behavior is clearly weird > when there is nothing to replicate. > Comments? [~jdcryans], [~stack]. > If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11136) Add permission check to roll WAL writer
[ https://issues.apache.org/jira/browse/HBASE-11136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995280#comment-13995280 ] Jerry He commented on HBASE-11136: -- Attached a patch. The patch also includes a missing documentation on 'hbase.coprocessor.regionserver.classes'. > Add permission check to roll WAL writer > > > Key: HBASE-11136 > URL: https://issues.apache.org/jira/browse/HBASE-11136 > Project: HBase > Issue Type: Improvement > Components: regionserver, security >Affects Versions: 0.96.2, 0.98.2 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-11136-trunk-v1.patch > > > Currently HBase provides HBaseAdmin.rollHLogWriter() and shell command to > roll WAL on a region server. But no permission check is done on this > operation in a secure cluster. > We need to add permission check to prevent un-authorized user from running > this operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11136) Add permission check to roll WAL writer
[
https://issues.apache.org/jira/browse/HBASE-11136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995486#comment-13995486
]
Ted Yu commented on HBASE-11136:
{code}
+requirePermission("preRollLogWriter", Permission.Action.ADMIN);
{code}
Should we consider Permission.Action.CREATE as well ?
> Add permission check to roll WAL writer
>
>
> Key: HBASE-11136
> URL: https://issues.apache.org/jira/browse/HBASE-11136
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, security
>Affects Versions: 0.96.2, 0.98.2
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 0.99.0
>
> Attachments: HBASE-11136-trunk-v1.patch
>
>
> Currently HBase provides HBaseAdmin.rollHLogWriter() and shell command to
> roll WAL on a region server. But no permission check is done on this
> operation in a secure cluster.
> We need to add permission check to prevent un-authorized user from running
> this operation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HBASE-11143) ageOfLastShippedOp metric is confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995242#comment-13995242 ] Lars Hofhansl edited comment on HBASE-11143 at 5/12/14 5:22 PM: \+1, and can we get the new metric in 0.96+? was (Author: jdcryans): +1, and can we get the new metric in 0.96+? > ageOfLastShippedOp metric is confusing > -- > > Key: HBASE-11143 > URL: https://issues.apache.org/jira/browse/HBASE-11143 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.20 > > Attachments: 11143-0.94-v2.txt, 11143-0.94.txt > > > We are trying to report on replication lag and find that there is no good > single metric to do that. > ageOfLastShippedOp is close, but unfortunately it is increased even when > there is nothing to ship on a particular RegionServer. > I would like discuss a few options here: > Add a new metric: replicationQueueTime (or something) with the above meaning. > I.e. if we have something to ship we set the age of that last shipped edit, > if we fail we increment that last time (just like we do now). But if there is > nothing to replicate we set it to current time (and hence that metric is > reported to close to 0). > Alternatively we could change the meaning of ageOfLastShippedOp to mean to do > that. That might lead to surprises, but the current behavior is clearly weird > when there is nothing to replicate. > Comments? [~jdcryans], [~stack]. > If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11135) Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file
[ https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995355#comment-13995355 ] Jeffrey Zhong commented on HBASE-11135: --- Great! Should I wait for you to migrate my patch on top of your latest change? > Change region sequenceid generation so happens earlier in the append cycle > rather than just before added to file > > > Key: HBASE-11135 > URL: https://issues.apache.org/jira/browse/HBASE-11135 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Assignee: stack > Attachments: 11135.wip.txt, 11135v2.txt, 11135v5.txt, 11135v5.txt, > 11135v5.txt > > > Currently we assign the region edit/sequence id just before we put it in the > WAL. We do it in the single thread that feeds from the ring buffer. Doing > it at this point, we can ensure order, that the edits will be in the file in > accordance w/ the ordering of the region sequence id. > But the point at which region sequence id is assigned an edit is deep down in > the WAL system and there is a lag between our putting an edit into the WAL > system and the edit actually getting its edit/sequence id. > This lag -- "late-binding" -- complicates the unification of mvcc and region > sequence id, especially around async WAL writes (and, related, for no-WAL > writes) -- the parent for this issue (For async, how you get the edit id in > our system when the threads have all gone home -- unless you make them wait?) > Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of > getting the region sequence id near-immediately. We'll run two ringbuffers. > The first will mesh all handler threads and the consumer will generate ids > (we will have order on other side of this first ring buffer), and then if > async or no sync, we will just let the threads return ... updating mvcc just > before we let them go. All other calls will go up on to the second ring > buffer to be serviced as now (batching, distribution out among the sync'ing > threads). The first rb will have no friction and should turn at fast rates > compared to the second. There should not be noticeable slowdown nor do I > foresee this refactor intefering w/ our multi-WAL plans. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11140) LocalHBaseCluster should create ConsensusProvider per each server
[ https://issues.apache.org/jira/browse/HBASE-11140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995270#comment-13995270 ] Ted Yu commented on HBASE-11140: lgtm > LocalHBaseCluster should create ConsensusProvider per each server > - > > Key: HBASE-11140 > URL: https://issues.apache.org/jira/browse/HBASE-11140 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11140.patch > > > Right now there's a bug there when single ConsensusProvider instance is > shared across all threads running region servers and masters within > processes, which breaks certain tests in patches which used to pass > successfully before. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995431#comment-13995431 ] Demai Ni commented on HBASE-7912: - the ping on May 9 was lost during apache email outage. so reposted here -- hi, folks, We have patches for both full backup (v4) and incremental backup (v1) uploaded today. With that, it is easy to apply this [HBASE-11085-trunk-v1-contains-HBASE-10900-trunk-v4.patch|https://issues.apache.org/jira/secure/attachment/12644215/HBASE-11085-trunk-v1-contains-HBASE-10900-trunk-v4.patch] directly on trunk, and give it a try. Please see the example in both incremental backup jira : [HBASE-11085|https://issues.apache.org/jira/browse/HBASE-11085] and fullbackup jira : [HBASE-10900|https://issues.apache.org/jira/browse/HBASE-10900]. We will open more jiras and patches for other features (just as, merge, convert, delete, history, progress) in the coming weeks. Also, thanks for the review comments from [~tedyu], [~mbertozzi], [~stack] and others. We will have a few follow-up improvements about zookeeper, protobuff, and leveraging the new snapshot manifest Demai > HBase Backup/Restore Based on HBase Snapshot > > > Key: HBASE-7912 > URL: https://issues.apache.org/jira/browse/HBASE-7912 > Project: HBase > Issue Type: Sub-task >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf, > HBase_BackupRestore-Jira-7912-CLI-v1.pdf > > > Finally, we completed the implementation of our backup/restore solution, and > would like to share with community through this jira. > We are leveraging existing hbase snapshot feature, and provide a general > solution to common users. Our full backup is using snapshot to capture > metadata locally and using exportsnapshot to move data to another cluster; > the incremental backup is using offline-WALplayer to backup HLogs; we also > leverage global distribution rolllog and flush to improve performance; other > added-on values such as convert, merge, progress report, and CLI commands. So > that a common user can backup hbase data without in-depth knowledge of hbase. > Our solution also contains some usability features for enterprise users. > The detail design document and CLI command will be attached in this jira. We > plan to use 10~12 subtasks to share each of the following features, and > document the detail implement in the subtasks: > * *Full Backup* : provide local and remote back/restore for a list of tables > * *offline-WALPlayer* to convert HLog to HFiles offline (for incremental > backup) > * *distributed* Logroll and distributed flush > * Backup *Manifest* and history > * *Incremental* backup: to build on top of full backup as daily/weekly backup > * *Convert* incremental backup WAL files into hfiles > * *Merge* several backup images into one(like merge weekly into monthly) > * *add and remove* table to and from Backup image > * *Cancel* a backup process > * backup progress *status* > * full backup based on *existing snapshot* > *-* > *Below is the original description, to keep here as the history for the > design and discussion back in 2013* > There have been attempts in the past to come up with a viable HBase > backup/restore solution (e.g., HBASE-4618). Recently, there are many > advancements and new features in HBase, for example, FileLink, Snapshot, and > Distributed Barrier Procedure. This is a proposal for a backup/restore > solution that utilizes these new features to achieve better performance and > consistency. > > A common practice of backup and restore in database is to first take full > baseline backup, and then periodically take incremental backup that capture > the changes since the full baseline backup. HBase cluster can store massive > amount data. Combination of full backups with incremental backups has > tremendous benefit for HBase as well. The following is a typical scenario > for full and incremental backup. > # The user takes a full backup of a table or a set of tables in HBase. > # The user schedules periodical incremental backups to capture the changes > from the full backup, or from last incremental backup. > # The user needs to restore table data to a past point of time. > # The full backup is restored to the table(s) or to different table name(s). > Then the incremental backups that are up to the desired point in time are > applied on top of the full backup. > We would support the following key features and capabilities. > * Full backup uses HBase snapshot to capture HFiles. > * Use HBase WALs to capture incremental changes, but we use bulk load of > HFiles for
[jira] [Updated] (HBASE-11143) ageOfLastShippedOp metric is confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-11143: -- Attachment: 11143-trunk.txt Trunk patch. > ageOfLastShippedOp metric is confusing > -- > > Key: HBASE-11143 > URL: https://issues.apache.org/jira/browse/HBASE-11143 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: 11143-0.94-v2.txt, 11143-0.94.txt, 11143-trunk.txt > > > We are trying to report on replication lag and find that there is no good > single metric to do that. > ageOfLastShippedOp is close, but unfortunately it is increased even when > there is nothing to ship on a particular RegionServer. > I would like discuss a few options here: > Add a new metric: replicationQueueTime (or something) with the above meaning. > I.e. if we have something to ship we set the age of that last shipped edit, > if we fail we increment that last time (just like we do now). But if there is > nothing to replicate we set it to current time (and hence that metric is > reported to close to 0). > Alternatively we could change the meaning of ageOfLastShippedOp to mean to do > that. That might lead to surprises, but the current behavior is clearly weird > when there is nothing to replicate. > Comments? [~jdcryans], [~stack]. > If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11135) Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file
[
https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995267#comment-13995267
]
Jeffrey Zhong commented on HBASE-11135:
---
{quote}
One thought I had was could you update the mvcc in the ring buffer consumer
thread just after we up the region edit/sequence id? If so, I could undo the
latching.
{quote}
Yes, I can do that. Today I'll migrate my patch and run tests to see if there
is any issue.
> Change region sequenceid generation so happens earlier in the append cycle
> rather than just before added to file
>
>
> Key: HBASE-11135
> URL: https://issues.apache.org/jira/browse/HBASE-11135
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: stack
>Assignee: stack
> Attachments: 11135.wip.txt, 11135v2.txt, 11135v5.txt, 11135v5.txt,
> 11135v5.txt
>
>
> Currently we assign the region edit/sequence id just before we put it in the
> WAL. We do it in the single thread that feeds from the ring buffer. Doing
> it at this point, we can ensure order, that the edits will be in the file in
> accordance w/ the ordering of the region sequence id.
> But the point at which region sequence id is assigned an edit is deep down in
> the WAL system and there is a lag between our putting an edit into the WAL
> system and the edit actually getting its edit/sequence id.
> This lag -- "late-binding" -- complicates the unification of mvcc and region
> sequence id, especially around async WAL writes (and, related, for no-WAL
> writes) -- the parent for this issue (For async, how you get the edit id in
> our system when the threads have all gone home -- unless you make them wait?)
> Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of
> getting the region sequence id near-immediately. We'll run two ringbuffers.
> The first will mesh all handler threads and the consumer will generate ids
> (we will have order on other side of this first ring buffer), and then if
> async or no sync, we will just let the threads return ... updating mvcc just
> before we let them go. All other calls will go up on to the second ring
> buffer to be serviced as now (batching, distribution out among the sync'ing
> threads). The first rb will have no friction and should turn at fast rates
> compared to the second. There should not be noticeable slowdown nor do I
> foresee this refactor intefering w/ our multi-WAL plans.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-7958) Statistics per-column family per-region
[ https://issues.apache.org/jira/browse/HBASE-7958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995304#comment-13995304 ] Jesse Yates commented on HBASE-7958: [~apurtell] originally this was going to be done to support phoenix. so they could get a better idea of distribution of keys to do better region chunking. In phoenix we decided even sized chunks would work well enough, so the impetus to get this done fizzled a bit. Other reasons for fizzle: * do we just expose the metric for things like otsdb? or just store it in HBase in our own format? both? * do we need to a UI components? * everyone wants all the metrics * should this even be part of core and instead just done via a scan-time coprocessor? Those considerations aside, I think it still would still be great to do for core :) We can always add more stats once we have a way to handle all the rest. How to expose them is an open question (especially considering phoenix will want to read these stats too) - maybe a pluggable sink/use a custom tag for metrics2 so people can use their own sinks? > Statistics per-column family per-region > --- > > Key: HBASE-7958 > URL: https://issues.apache.org/jira/browse/HBASE-7958 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.95.2 >Reporter: Jesse Yates >Assignee: Jesse Yates > Attachments: hbase-7958-v0-parent.patch, hbase-7958-v0.patch, > hbase-7958_rough-cut-v0.patch > > > Originating from this discussion on the dev list: > http://search-hadoop.com/m/coDKU1urovS/Simple+stastics+per+region/v=plain > Essentially, we should have built-in statistics gathering for HBase tables. > This allows clients to have a better understanding of the distribution of > keys within a table and a given region. We could also surface this > information via the UI. > There are a couple different proposals from the email, the overview is this: > We add in something on compactions that gathers stats about the keys that are > written and then we surface them to a table. > The possible proposals include: > *How to implement it?* > # Coprocessors - > ** advantage - it easily plugs in and people could pretty easily add their > own statistics. > ** disadvantage - UI elements would also require this, we get into dependent > loading, which leads down the OSGi path. Also, these CPs need to be installed > _after_ all the other CPs on compaction to ensure they see exactly what gets > written (doable, but a pain) > # Built into HBase as a custom scanner > ** advantage - always goes in the right place and no need to muck about with > loading CPs etc. > ** disadvantage - less pluggable, at least for the initial cut > *Where do we store data?* > # .META. > ** advantage - its an existing table, so we can jam it into another CF there > ** disadvantage - this would make META much larger, possibly leading to > splits AND will make it much harder for other processes to read the info > # A new stats table > ** advantage - cleanly separates out the information from META > ** disadvantage - should use a 'system table' idea to prevent accidental > deletion, manipulation by arbitrary clients, but still allow clients to read > it. > Once we have this framework, we can then move to an actual implementation of > various statistics. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11136) Add permission check to roll WAL writer
[ https://issues.apache.org/jira/browse/HBASE-11136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-11136: - Attachment: HBASE-11136-trunk-v1.patch > Add permission check to roll WAL writer > > > Key: HBASE-11136 > URL: https://issues.apache.org/jira/browse/HBASE-11136 > Project: HBase > Issue Type: Improvement > Components: regionserver, security >Affects Versions: 0.96.2, 0.98.2 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-11136-trunk-v1.patch > > > Currently HBase provides HBaseAdmin.rollHLogWriter() and shell command to > roll WAL on a region server. But no permission check is done on this > operation in a secure cluster. > We need to add permission check to prevent un-authorized user from running > this operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11145) Issue with HLog sync
[
https://issues.apache.org/jira/browse/HBASE-11145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-11145:
--
Fix Version/s: 0.99.0
> Issue with HLog sync
>
>
> Key: HBASE-11145
> URL: https://issues.apache.org/jira/browse/HBASE-11145
> Project: HBase
> Issue Type: Bug
>Reporter: Anoop Sam John
>Assignee: stack
> Fix For: 0.99.0
>
>
> Got the below Exceptions Log in case of a write heavy test
> {code}
> 2014-05-07 11:29:56,417 ERROR [main.append-pool1-t1]
> wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
> java.lang.IllegalStateException: Queue full
> at java.util.AbstractQueue.add(Unknown Source)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.offer(FSHLog.java:1227)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1878)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
> at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 2014-05-07 11:29:56,418 ERROR [main.append-pool1-t1]
> wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
> java.lang.ArrayIndexOutOfBoundsException: 5
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1838)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
> at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 2014-05-07 11:29:56,419 ERROR [main.append-pool1-t1]
> wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
> java.lang.ArrayIndexOutOfBoundsException: 6
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1838)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
> at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 2014-05-07 11:29:56,419 ERROR [main.append-pool1-t1]
> wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
> java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1838)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
> at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> {code}
> In FSHLog$SyncRunner.offer we do BlockingQueue.add() which throws Exception
> as it is full. The problem is the below shown catch() we do not do any
> cleanup.
> {code}
> this.syncRunners[index].offer(sequence, this.syncFutures,
> this.syncFuturesCount);
> attainSafePoint(sequence);
> this.syncFuturesCount = 0;
> } catch (Throwable t) {
> LOG.error("UNEXPECTED!!!", t);
> }
> {code}
> syncFuturesCount is not getting reset to 0 and so the subsequent onEvent()
> handling throws ArrayIndexOutOfBoundsException.
> I think we should do the below
> 1. Handle the Exception and call cleanupOutstandingSyncsOnException() as in
> other cases of Exception handling
> 2. Instead of BlockingQueue.add() use offer() (?)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11143) ageOfLastShippedOp metric is confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-11143: -- Attachment: 11143-trunk.txt Right patch this time. > ageOfLastShippedOp metric is confusing > -- > > Key: HBASE-11143 > URL: https://issues.apache.org/jira/browse/HBASE-11143 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > Attachments: 11143-0.94-v2.txt, 11143-0.94.txt, 11143-trunk.txt > > > We are trying to report on replication lag and find that there is no good > single metric to do that. > ageOfLastShippedOp is close, but unfortunately it is increased even when > there is nothing to ship on a particular RegionServer. > I would like discuss a few options here: > Add a new metric: replicationQueueTime (or something) with the above meaning. > I.e. if we have something to ship we set the age of that last shipped edit, > if we fail we increment that last time (just like we do now). But if there is > nothing to replicate we set it to current time (and hence that metric is > reported to close to 0). > Alternatively we could change the meaning of ageOfLastShippedOp to mean to do > that. That might lead to surprises, but the current behavior is clearly weird > when there is nothing to replicate. > Comments? [~jdcryans], [~stack]. > If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11129) Expose Scan conversion methods in TableMapReduceUtil as public methods
[
https://issues.apache.org/jira/browse/HBASE-11129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992872#comment-13992872
]
Nick Dimiduk commented on HBASE-11129:
--
Looking for some context here, I cannot find the commit that makes this
interface private. Do you know off the top of your head
([~yuzhih...@gmail.com], [~jdcryans], [~stack])?
> Expose Scan conversion methods in TableMapReduceUtil as public methods
> --
>
> Key: HBASE-11129
> URL: https://issues.apache.org/jira/browse/HBASE-11129
> Project: HBase
> Issue Type: Task
> Components: mapreduce
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
>Priority: Minor
> Attachments: HBASE-11129.patch
>
>
> Scan#readFields() from 0.92 has been removed.
> TableMapReduceUtil has the following package private methods:
> {code}
> static String convertScanToString(Scan scan) throws IOException {
> {code}
> {code}
> static Scan convertStringToScan(String base64) throws IOException {
> {code}
> We should consider exposing them as public methods so that user can interpret
> Scan objects easily in mapreduce jobs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-7030) Configurable whitelist for coprocessor class loader
[ https://issues.apache.org/jira/browse/HBASE-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-7030. --- Resolution: Not a Problem > Configurable whitelist for coprocessor class loader > --- > > Key: HBASE-7030 > URL: https://issues.apache.org/jira/browse/HBASE-7030 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.95.2 >Reporter: Andrew Purtell > > The coprocessor class loader should allow the user to supply a list of > packages or classes to resolve directly with the parent (RegionServer) > classloader. > See HBASE-6843 for some background. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11143) ageOfLastShippedOp metric is confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995242#comment-13995242 ] Jean-Daniel Cryans commented on HBASE-11143: +1, and can we get the new metric in 0.96+? > ageOfLastShippedOp metric is confusing > -- > > Key: HBASE-11143 > URL: https://issues.apache.org/jira/browse/HBASE-11143 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.20 > > Attachments: 11143-0.94-v2.txt, 11143-0.94.txt > > > We are trying to report on replication lag and find that there is no good > single metric to do that. > ageOfLastShippedOp is close, but unfortunately it is increased even when > there is nothing to ship on a particular RegionServer. > I would like discuss a few options here: > Add a new metric: replicationQueueTime (or something) with the above meaning. > I.e. if we have something to ship we set the age of that last shipped edit, > if we fail we increment that last time (just like we do now). But if there is > nothing to replicate we set it to current time (and hence that metric is > reported to close to 0). > Alternatively we could change the meaning of ageOfLastShippedOp to mean to do > that. That might lead to surprises, but the current behavior is clearly weird > when there is nothing to replicate. > Comments? [~jdcryans], [~stack]. > If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
