[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539865#comment-13539865
]
shen guanpu commented on HBASE-7263:
hi Gregory Chanan
Does your test just operate one rowkey?
How much will it be slower when you put different rowkey? as you mentioned ,you
have to wait for MVCC on other rows.
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539873#comment-13539873
]
shen guanpu commented on HBASE-7263:
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned, is
that you have to wait for updates on other rows, since MVCC is per-row.
Do you mean it is not per-row for option 2( Have an MVCC per-row (table
configuration): this avoids the unnecessary contention of 1))
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540080#comment-13540080
]
Gregory Chanan commented on HBASE-7263:
---
bq. Does your test just operate one rowkey?
The full test description is given above. The relevant part to your question:
- Each increment uniformly distributed over 500,000 rows
bq. Do you mean it is not per-row for option 2( Have an MVCC per-row (table
configuration): this avoids the unnecessary contention of 1))
Option 2 is per-row but that is not what is implemented today and this JIRA
takes a different approach.
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526619#comment-13526619
]
stack commented on HBASE-7263:
--
bq. Is it possible to introduce a system wide switch with default value
allowing user row locks to be taken ?
It maybe be possible but -1 on our ever doing it. Why expend effort to add a
config so exotic, something no one would ever use?
[~gchanan] How much work would it take to get to your running the TestAcid unit
test?
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526675#comment-13526675
]
Gregory Chanan commented on HBASE-7263:
---
bq. It maybe be possible but -1 on our ever doing it. Why expend effort to add
a config so exotic, something no one would ever use?
Agreed. If users want row locks, they can implement it at the app level.
bq. Gregory Chanan How much work would it take to get to your running the
TestAcid unit test?
Good question :). Ideally we would spend some effort refactoring the
write-path code, i.e. merge the code that does almost the same thing
(doMiniBatchMutation and mutateWithRowLocks is one example, possibly
append/increment/mutateWithPut as well). I don't think I have time for that
right now.
Instead, I'll do the following:
- Remove user-visible row locks (I'll file a linked JIRA). Doing this first
will make the next step easier, and I don't think it will be too difficult.
- Remove RowLocks altogether and replace with read/write locks as described. I
already have code for doMiniBatchMutation and increment that seems to work,
except it doesn't do memory management. I think adding to the other write-path
functions will be straightforward; memory management may take some thinking.
If you are asking for a timeframe, I don't think it will gate 0.96 :).
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13525981#comment-13525981
]
Gregory Chanan commented on HBASE-7263:
---
Arg, looks like JIRA ate my previous comment.
[~lhofhansl] I wanted to run one more thing by you before I write more code for
this.
Why do we need the row locks (at least in the case the user never explictly
grabs a rowlock). On the read side,
the scanner just ignores KVs with a memstoreTS higher than its readPoint, so
doesn't seem to be a problem there.
On the write side, we update the memstore and prepare the WALEdit under the row
lock, both of which seem like
they can be done concurrently.
Now, given that we allow the user to grab row locks, we need some concurrency
control. Could we use the read-locks
I describe above for the common case (no lock passed in) and grant the user a
write lock if they explicitly ask for it
via HTable.lockRow? I guess frequent updates could stall the call to lockRow,
but I think those calls are rare.
Am I missing something?
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13525987#comment-13525987
]
stack commented on HBASE-7263:
--
We've talked often enough in the past on dropping user ordained row locking and
have been suggesting they not be used w/ a while now. We could just drop them
from 0.96 though they have not been deprecated?
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526009#comment-13526009
]
Lars Hofhansl commented on HBASE-7263:
--
You're talking heresy now :)
I have never questioned the need to rowlock, but since you ask directly... If
we have MVCC and your read-write locks, it seems we won't need the rowlocks any
longer. They were only really needed for the atomic operation such as
increment/append and checkAndXXX.
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526100#comment-13526100
]
Andrew Purtell commented on HBASE-7263:
---
I like this thinking.
Now is the time (0.96 singularity) to make changes like dropping user row
locks.
We need to manage against having too many major changes destabilizing code at
once. That said, I would vote for this one because removing or mitigating
contention points will be increasingly important as some storage shifts away
from spinning media.
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526127#comment-13526127
]
Ted Yu commented on HBASE-7263:
---
Is it possible to introduce a system wide switch with default value allowing
user row locks to be taken ?
This way, users consciously manage concurrency if they utilize user row locks.
Otherwise we manage locking and achieve better (intrinsic) performance.
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509963#comment-13509963
]
Gregory Chanan commented on HBASE-7263:
---
I should also mention in the above numbers, I'm not doing real memory
management. I create the ReadWriteRowLock when needed and never delete. Need
to do something smarter there which may have performance impact.
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509422#comment-13509422
]
Lars Hofhansl commented on HBASE-7263:
--
Interesting!
This looks right to me:
* multiple Puts can be in the section guarded by the row read lock
* the readlock covers the MVCC section
* an increment will wait out all concurrent Puts for the same row by attempting
to take the row write lock
Do these new locks time out like original RowLock?
This has the potential that Puts starve out Increments, right? (not a big deal,
though)
Also, I'd be a bit skeptical about another mechanism for change visibility.
Do the result even get better if you spread the increments out over fewer rows?
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509440#comment-13509440
]
Gregory Chanan commented on HBASE-7263:
---
bq. Do these new locks time out like original RowLock?
Not yet, the write-lock should though.
bq. This has the potential that Puts starve out Increments, right? (not a big
deal, though)
I don't think so. Consider some set of puts that are holding the readlock, but
not the rowlock. The next operation to go will be whoever grabs the rowlock
next (since it is grabbed before the read or write lock), which is exactly the
same as today.
bq. Also, I'd be a bit skeptical about another mechanism for change visibility.
Well, I also investigated making MVCC more aware (either making it totally per
row or having it track the rows being modified and providing a
waitForRowComplete method, but those were way too complicated :). That's not
an argument for adding another mechanism, though. I agree we should only do
this if the performance is worth it.
bq. Do the result even get better if you spread the increments out over fewer
rows?
I'm not sure. My thinking was to spread the increments over a large number of
rows so that the contention is on MVCC, not on the row lock. I thought the
Today number for the increment test would be way worse, actually, because
you'd essentially get only one increment at a time actually doing work. I need
to investigate further. It's possible that multiple increments are doing work
at the same time, though; any number can technically finish
completeMemstoreInsert(beginMemstoreInsert) before another starts a new
transaction with beginMemstoreInsert.
Investigate more fine grained locking for checkAndPut/append/increment
--
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned,
is that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it
until the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts
with HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I
need to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more
