Mike Matrigali wrote:
What is your definition of a "valid" RowLocation? Mine is:
1) A non-deleted RowLocation is guaranteed to point at the intended
record until a delete is executed, by a client of the access method.
This part requires no locking, and is the current protocol.
2) A row lock is required to prevent a delete of the row by another
transaction. There is no way to prevent a delete of the RowLocation
by the same transaction. This is the current protocol.
3) I think SUR requires that upon committed delete access to the
RowLocation always return some sort of failure, and never access
to a different row. Truncate currently breaks this, for
conglomerates that guarantee non-reusable RowLocations. In current
access methods this could be enforced this while holding a
easily if the table level intent
lock requirement is added.
I would be comfortable adding this to store contract. It
seems reasonable and allows coordination through locking.
Yes, I agree with you definition, and I would appreciate if you could
add the table intent lock requirement to the contract.
Note this does not adress other current client usages of the access
methods. But I believe that all those clients could easily agree
to the table intent lock protocol. This would mean that any client
that wanted to break this protol must get an exclusive table lock
first. I believe all those clients already do, but for different
reasons.
Does this solve your non-holdable case? The holdable case is a
different beast, and should be a different thread.
Yes, as far as I can see, this solves the non-holdable case.
And to be entirely safe, would it be possible to add a requirement that
truncate/defragment and other compress operations always should run in
autocommit mode ?
I am thinking of the cases were a user could run these in the same
transaction as they have an updatable cursor ? If they are in autocommit
mode, I think the cursor (non-holdable) will be closed.
Andreas
Andreas Korneliussen wrote:
Mike Matrigali wrote:
It looks like I may have been thinking about future directions vs.
current reality. The question still is what should the contract
be, rather than what a specific implementation currently provides.
I agree. I think the contract could be one of these:
1. A RowLocation is valid as long as the transaction has a table
intent lock
2. A RowLocation is valid as long as the transaction has a row lock
for the row.
Both contracts have different tradeoffs, I guess. for store, alt. 2
gives more flexibility in future online compress operations, however I
think it would require SURs to set and hold locks for all isolation
levels.
JavaDoc for RowLocation do say:
See the conglomerate implementation specification for
information about the conditions under which a row location
remains valid.
So currently, it is implementation defined.
Andreas
Andreas Korneliussen wrote:
Mike Matrigali wrote:
I posted some questions about how the delete/update is done, those
answers would help me understand better what is needed for a solution.
I am going to start a separate thread concentrating on RowLocation
guarantees from store.
That is great.
Some other answers below.
Andreas Korneliussen wrote:
Mike Matrigali wrote:
There are very few cross thread dependencies not managed by locks
currently. These things add extra complications to current and
future code. Also I want to understand clearly the new restrictions
be imposted on the access methods (both current and possible
future). In the future we would like to do more automatic space
reclamation as part of the zero-admin goal, the ability to do this
in the future internal to access methods is probably affected by
the proposals here.
I think the complexities are there already. It seems very hard to
know
under exactly which conditions, a RowLocation remains valid. We have
assumed that the RowLocation will remain valid as long as we hold a
table intent lock (valid means that it either points to the same
row or
to a deleted row).
That is what we concluded from the previous discusssion about
validity
of RowLocation. If you in the future need to make code, or there
already
is code, which breaks this assumption, we would need to know which
other
mechanisms we should use to either verify that the RowLocation is
valid,
or to block the system to make it invalid.
There is already code that breaks this assumption, the in place
compress
table. It currently is only executed as a call to a system procedure:
http://db.apache.org/derby/docs/dev/ref/rrefproceduresinplacecompress.html
But the hope was in the future to do the same kind of work that this
procedure does, internally in background.
As you have already determined that off-line compress is even more
of a
problem, but it does run under an exclusive table lock. After it runs
the container you were connected to does not even exist any more.
The online compress seems to require table-exclusive locks when
passing the defragment or truncate_end argument.
There are two testcases in jdbcapi/ConcurrencyTest which test this
(testDefragmentDuringScan and testTruncateDuringScan).
The tests first deletes all rows except the first and the last. Then
it commits. Then it opens a SUR in read-uncommitted mode. Read all
records into the resultset, and position the cursor to afterlast.
Then it runs the defragment or truncate. The test will hang waiting
for a lock. Finally, it updates the rows correctly.
Output from the tests (modified the test slightly to get more output):
T1: delete records
T1: commit
T2: Read next Tuple:(0,0,17)
T2: Read next Tuple:(9,9,35)
T3: call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE
T3: DEFRAGMENT
ERROR 40XL1: A lock could not be obtained within the time requested
....
org.apache.derby.impl.store.access.RAMTransaction.openScan(RAMTransaction.java:1582)
at
org.apache.derby.iapi.db.OnlineCompress.setup_indexes(OnlineCompress.java:605)
at
org.apache.derby.iapi.db.OnlineCompress.defragmentRows(OnlineCompress.java:359)
at
org.apache.derby.iapi.db.OnlineCompress.compressTable(OnlineCompress.java:227)
at
org.apache.derby.catalog.SystemProcedures.SYSCS_INPLACE_COMPRESS_TABLE(SystemProcedures.java:858)
...
at
org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testCompressDuringScan(ConcurrencyTest.java:777)
at
org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testDefragmentDuringScan(ConcurrencyTest.java:712)
...
T3: got expected exception
T1: Read first Tuple:(0,0,17)
T1: updateInt(2, 3);
T1: updateRow()
T1: Read last Tuple:(9,9,35)
T1: updateInt(2, 3);
T1: updateRow()
T1: commit
T4: select * from table
T4: Read next Tuple:(0,3,17)
T4: Read next Tuple:(9,3,35)
So to me it seems that our assumptions are correct.. Only a purge is
allowed with row-level locking, online compress and online
defragment seems to be blocked by an open cursor on the table.
Maybe that is not how online compress was intended to be ?
Andreas
Locks can be used to manage cross thread dependencies, however
they are
bound to the transaction, and therefore does not help very much for
cursors held across commits. So if the only mechanism we can have to
ensure that the RowLocations are valid, is by the means of locks, I
doubt we can support the feature of scrollable insensitive *holdable*
updatable resultset.
I agree, I don't believe derby is currently architected to correctly
implement "holdable" SUR. I don't think the outside event driven
approach is the right way to go.
It is true that the current access methods don't reuse row locations
until a table level lock is granted. But your project would be the
first dependency on this outside of the access method
implementations
themselves. It is very clear the contract that the access methods
have with their clients while locks are held on the data they are
looking at, what you are proposing is a contract on unlocked data.
I guess we are the first to use RowLocation without holding a lock on
the row. This is necessary, unless we simply make SUR cursors set
locks
for all rows in the cursor independent from isolation level.
Note that the current "in-place" compress will MOVE rows from one
row location to another if one does not have a row lock on the row.
This is done in the 2nd phase and only holds an intent lock, and
exclusive row locks on the rows being moved.
The off-line compress only does work under an X table lock.
So the row that you are updating actually will exist in the table,
but currently you will request the old location and will get back
a delete row indicator. I think because of this option 1 does not
work.
Are you saing that RowLocations can be invalidated by "in-place"
compress even if we hold a Table intent lock ?
The "in-place" compress will do the following take a row a row
location
N and do the
following:
insert row at new row location M
delete row at old row location N
possibly purge old row location
I don't understand how this will affect your code, it depends how you
search for the row, and what errors are ok to throw.
