Re: Themis : implements cross-row/corss-table transaction on HBase.
Thanks for updating the list with the nice Themis updates Jianwei. I added Themis to the powered by list (and the other missing transactions managers, Tephra and Haeinsa). St.Ack On Mon, Nov 10, 2014 at 12:50 AM, 崔建伟 wrote: > Hi everyone: > In last few months, we have updated Themis to achieve better performance > and include more features: > > 1. Improve the single-row write performance from 23%(relative drop > compared with HBase's put) to 60%(for most test cases). For single-row > write transaction, we only write lock to MemStore in prewrite-phase, then, > we erase corresponding lock, write data and commit information to HLog in > commit-phase. This won't break the correctness of percolator algorithm and > will help improve the performance a lot for single-row write. > > 2. Support HBase 0.98. We create a branch: > https://github.com/XiaoMi/themis/tree/for_hbase_0.98 to make themis > support HBase 0.98(Currently, support HBase 0.98.5). All the functions of > master branch will also be implemented in this branch. > > 3. Transaction TTL support and Old Data Clean. Users could set TTL for > read/write transaction respectively. Then, old data which could not be read > will be cleaned periodically. > > 4. MapReduce Support. We implement a group of classes to support read data > by themis transaction in Mapper job and write data by themis transaction in > Reduce job. > > For more details, please see the github: > https://github.com/XiaoMi/themis(or > https://github.com/XiaoMi/themis/tree/for_hbase_0.98) or jira: > https://issues.apache.org/jira/browse/HBASE-10999 . If you find Themis > interesting, please leave us comment in the mail, jira or github. > > Best > cuijianwei > > > From: saint@gmail.com on behalf of Stack < > st...@duboce.net> > Sent: Sunday, July 13, 2014 1:12 PM > To: HBase Dev List > Cc: u...@hbase.apache.org > Subject: Re: Themis : implements cross-row/corss-table transaction on > HBase. > > On Tue, Jul 8, 2014 at 12:34 AM, 崔建伟 wrote: > > > Hi everyone, I want to introduce our open-source project Themis which > > implements cross-row/corss-table transaction on HBase. > > > > Themis follows google's percolator algorithm( > > http://research.google.com/pubs/pub36726.html), which provides > > ACID-compliant transaction and snapshot isolation. The cross-row > > transaction is based on HBase's single-row atomic semantics and doesn't > use > > a central transaction server, so that supports linear-scalability. > > > > Themis depends on a timestamp server to provides global strictly > > incremental timestamp to define the order of transactions, which will be > > used to resolve the write-write and read-write conflicts. The timestamp > > server is lightweight and could achieve hight throughput(500, 000 + qps), > > and Themis will batch timestamp requests across transactions in one Rpc, > so > > that it won't become the bottleneck of the system even when processing > > billions of transactions every day. > > > > Although Themis could be implemented totally in client-side, we adopt > > coprocessor framework of HBase to achieve higher performance. Themis > > includes a client-side library to provides transaction APIs, such as > > themisPut/themisGet/themisScan/themisDelete, and a coprocessor library > > loaded on regionserver. Therefore, Themis could be used without changing > > the code and logic of HBase. > > > > We have been validating the correctness of Themis for a few months by a > > AccountTransfer simulation program, which concurrently does cross-row > > transactions by transferring money among different accounts(each account > is > > a row in HBase) and verifies total money of all accounts doesn't change > in > > the simulation. We have also run Themis on our production environment. > > > > We test the performance of Themis and get comparable result as > percolator. > > The single-column transaction represents the worst performance case for > > Themis compared with HBase, the result is: > > 1) For read, the performance of percolator is 90% of HBase; > > 2) For write, the performance of percolator is 23% of HBase. > > The write performance drops a lot because Themis uses two-phase commit > > protocol to achieve ACID of transaction. For multi-row write, we improve > > the performance by paralleling all writes of pre-write phase. For > > single-row write, we are optimizing two-phase commit protocol to achieve > > better performance and will update the result when it is ready. The > details > > of performance result could be found in github. > > > > The repository and introduction of Themis include: > > 1. Themis github: https://github.com/XiaoMi/themis/. The source code, > > performance test result and user guide could be found here. > > 2. Themis jira : https://issues.apache.org/jira/browse/HBASE-10999 > > 3. Chronos github: https://github.com/XiaoMi/chronos. Chronos is our > > open-source high-availability, high-performance timestamp server to > p
[jira] [Resolved] (HBASE-12453) Make region available once it's open
[ https://issues.apache.org/jira/browse/HBASE-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang resolved HBASE-12453. - Resolution: Invalid > Make region available once it's open > > > Key: HBASE-12453 > URL: https://issues.apache.org/jira/browse/HBASE-12453 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > Currently (in trunk, with zk-less assignment), a region is available to > serving requests only after RS notifies the master the region is open, and > the meta is updated with the new location. We may be able to do better than > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12458) Improve CellCounter command line parsing
Esteban Gutierrez created HBASE-12458: - Summary: Improve CellCounter command line parsing Key: HBASE-12458 URL: https://issues.apache.org/jira/browse/HBASE-12458 Project: HBase Issue Type: Improvement Reporter: Esteban Gutierrez Priority: Minor Command line options parsing in CellCounter are different form other tools like CopyTable, RowCounter or VerifyReplication. It should be consistent with the other tools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12459) Use a non-managed Table in mapred.TableOutputFormat
Nick Dimiduk created HBASE-12459: Summary: Use a non-managed Table in mapred.TableOutputFormat Key: HBASE-12459 URL: https://issues.apache.org/jira/browse/HBASE-12459 Project: HBase Issue Type: Bug Components: mapreduce Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 2.0.0, 0.99.2 Analog to [~sduskis]'s HBASE-12423 but for mapred implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12460) Moving Chore to hbase-common module.
Srikanth Srungarapu created HBASE-12460: --- Summary: Moving Chore to hbase-common module. Key: HBASE-12460 URL: https://issues.apache.org/jira/browse/HBASE-12460 Project: HBase Issue Type: Bug Reporter: Srikanth Srungarapu Priority: Minor Right now, Chore which is being used in hbase-server and hbase-client modules is in hbase-common module. It would be better to move it to hbase-common module. This doesn't break any compatibility, as hbase-client is dependent on hbase-common already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12461) FSVisitor logging is excessive
Nick Dimiduk created HBASE-12461: Summary: FSVisitor logging is excessive Key: HBASE-12461 URL: https://issues.apache.org/jira/browse/HBASE-12461 Project: HBase Issue Type: Bug Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Fix For: 2.0.0, 0.98.9, 0.99.2 I noticed this in master log while debugging other issues. FSVisitor leaving walls of logs at INFO level making debugging difficult and creating huge log file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[CANCELLED] [VOTE] The 1st HBase 0.98.8 release candidate (RC0) is available, vote closing 11/14/2014
-1 Critical issue HBASE-12319 came in after the RC. Also, we are investigating long stalls during splitting+closing and would like to fix it if possible. See HBASE-12457. I'm planning on cutting another RC in a day or two. On Fri, Nov 7, 2014 at 11:25 PM, Andrew Purtell wrote: > The 1st HBase 0.98.8 release candidate (RC0) is available for download > at http://people.apache.org/~apurtell/0.98.8RC0/ and Maven artifacts are > also available in the temporary repository > https://repository.apache.org/content/repositories/orgapachehbase-1044/ > > Signed with my code signing key D5365CCD. > > The issues resolved in this release can be found here: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12327955 > > > Please try out the candidate and vote +1/-1 by midnight Pacific Time > (00:00 -0800 GMT) on November 14 on whether or not we should release > this as 0.98.8. Three +1 votes from PMC will be required to release. > > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Resolved] (HBASE-12319) Inconsistencies during region recovery due to close/open of a region during recovery
[
https://issues.apache.org/jira/browse/HBASE-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong resolved HBASE-12319.
---
Resolution: Fixed
Fix Version/s: (was: 2.0.0)
I've integrated the fix into 0.98 & branch-1. Thanks.
> Inconsistencies during region recovery due to close/open of a region during
> recovery
>
>
> Key: HBASE-12319
> URL: https://issues.apache.org/jira/browse/HBASE-12319
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.7, 0.99.1
>Reporter: Devaraj Das
>Assignee: Jeffrey Zhong
>Priority: Critical
> Fix For: 0.99.2, 0.98.8
>
> Attachments: HBASE-12319-v2.patch, HBASE-12319.patch
>
>
> In one of my test runs, I saw the following:
> {noformat}
> 2014-10-14 13:45:30,782 DEBUG
> [StoreOpener-51af4bd23dc32a940ad2dd5435f00e1d-1] regionserver.HStore: loaded
> hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d/test_cf/d6df5cfe15ca41d68c619489fbde4d04,
> isReference=false, isBulkLoadResult=false, seqid=141197, majorCompaction=true
> 2014-10-14 13:45:30,788 DEBUG [RS_OPEN_REGION-hor9n01:60020-1]
> regionserver.HRegion: Found 3 recovered edits file(s) under
> hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d
> .
> .
> 2014-10-14 13:45:31,916 WARN [RS_OPEN_REGION-hor9n01:60020-1]
> regionserver.HRegion: Null or non-existent edits file:
> hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d/recovered.edits/0198080
> {noformat}
> The above logs is from a regionserver, say RS2. From the initial analysis it
> seemed like the master asked a certain regionserver to open the region (let's
> say RS1) and for some reason asked it to close soon after. The open was still
> proceeding on RS1 but the master reassigned the region to RS2. This also
> started the recovery but it ended up seeing an inconsistent view of the
> recovered-edits files (it reports missing files as per the logs above) since
> the first regionserver (RS1) deleted some files after it completed the
> recovery. When RS2 really opens the region, it might not see the recent data
> that was written by flushes on hor9n10 during the recovery process. Reads of
> that data would have inconsistencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
RE: Themis : implements cross-row/corss-table transaction on HBase.
Hi Stack: Thanks for your concern and work:). I will continue to improve Themis and report the new progress. Best jianwei From: saint@gmail.com on behalf of Stack Sent: Wednesday, November 12, 2014 12:15 AM To: HBase Dev List Cc: u...@hbase.apache.org Subject: Re: Themis : implements cross-row/corss-table transaction on HBase. Thanks for updating the list with the nice Themis updates Jianwei. I added Themis to the powered by list (and the other missing transactions managers, Tephra and Haeinsa). St.Ack On Mon, Nov 10, 2014 at 12:50 AM, 崔建伟 wrote: > Hi everyone: > In last few months, we have updated Themis to achieve better performance > and include more features: > > 1. Improve the single-row write performance from 23%(relative drop > compared with HBase's put) to 60%(for most test cases). For single-row > write transaction, we only write lock to MemStore in prewrite-phase, then, > we erase corresponding lock, write data and commit information to HLog in > commit-phase. This won't break the correctness of percolator algorithm and > will help improve the performance a lot for single-row write. > > 2. Support HBase 0.98. We create a branch: > https://github.com/XiaoMi/themis/tree/for_hbase_0.98 to make themis > support HBase 0.98(Currently, support HBase 0.98.5). All the functions of > master branch will also be implemented in this branch. > > 3. Transaction TTL support and Old Data Clean. Users could set TTL for > read/write transaction respectively. Then, old data which could not be read > will be cleaned periodically. > > 4. MapReduce Support. We implement a group of classes to support read data > by themis transaction in Mapper job and write data by themis transaction in > Reduce job. > > For more details, please see the github: > https://github.com/XiaoMi/themis(or > https://github.com/XiaoMi/themis/tree/for_hbase_0.98) or jira: > https://issues.apache.org/jira/browse/HBASE-10999 . If you find Themis > interesting, please leave us comment in the mail, jira or github. > > Best > cuijianwei > > > From: saint@gmail.com on behalf of Stack < > st...@duboce.net> > Sent: Sunday, July 13, 2014 1:12 PM > To: HBase Dev List > Cc: u...@hbase.apache.org > Subject: Re: Themis : implements cross-row/corss-table transaction on > HBase. > > On Tue, Jul 8, 2014 at 12:34 AM, 崔建伟 wrote: > > > Hi everyone, I want to introduce our open-source project Themis which > > implements cross-row/corss-table transaction on HBase. > > > > Themis follows google's percolator algorithm( > > http://research.google.com/pubs/pub36726.html), which provides > > ACID-compliant transaction and snapshot isolation. The cross-row > > transaction is based on HBase's single-row atomic semantics and doesn't > use > > a central transaction server, so that supports linear-scalability. > > > > Themis depends on a timestamp server to provides global strictly > > incremental timestamp to define the order of transactions, which will be > > used to resolve the write-write and read-write conflicts. The timestamp > > server is lightweight and could achieve hight throughput(500, 000 + qps), > > and Themis will batch timestamp requests across transactions in one Rpc, > so > > that it won't become the bottleneck of the system even when processing > > billions of transactions every day. > > > > Although Themis could be implemented totally in client-side, we adopt > > coprocessor framework of HBase to achieve higher performance. Themis > > includes a client-side library to provides transaction APIs, such as > > themisPut/themisGet/themisScan/themisDelete, and a coprocessor library > > loaded on regionserver. Therefore, Themis could be used without changing > > the code and logic of HBase. > > > > We have been validating the correctness of Themis for a few months by a > > AccountTransfer simulation program, which concurrently does cross-row > > transactions by transferring money among different accounts(each account > is > > a row in HBase) and verifies total money of all accounts doesn't change > in > > the simulation. We have also run Themis on our production environment. > > > > We test the performance of Themis and get comparable result as > percolator. > > The single-column transaction represents the worst performance case for > > Themis compared with HBase, the result is: > > 1) For read, the performance of percolator is 90% of HBase; > > 2) For write, the performance of percolator is 23% of HBase. > > The write performance drops a lot because Themis uses two-phase commit > > protocol to achieve ACID of transaction. For multi-row write, we improve > > the performance by paralleling all writes of pre-write phase. For > > single-row write, we are optimizing two-phase commit protocol to achieve > > better performance and will update the result when it is ready. The > details > > of performance result could be found in github. > > > > The repository and introdu
