It looks like I may have been thinking about future directions vs.
current reality. The question still is what should the contract
be, rather than what a specific implementation currently provides.
Andreas Korneliussen wrote:
Mike Matrigali wrote:
I posted some questions about how the delete/update is done, those
answers would help me understand better what is needed for a solution.
I am going to start a separate thread concentrating on RowLocation
guarantees from store.
That is great.
Some other answers below.
Andreas Korneliussen wrote:
Mike Matrigali wrote:
There are very few cross thread dependencies not managed by locks
currently. These things add extra complications to current and
future code. Also I want to understand clearly the new restrictions
be imposted on the access methods (both current and possible
future). In the future we would like to do more automatic space
reclamation as part of the zero-admin goal, the ability to do this
in the future internal to access methods is probably affected by
the proposals here.
I think the complexities are there already. It seems very hard to know
under exactly which conditions, a RowLocation remains valid. We have
assumed that the RowLocation will remain valid as long as we hold a
table intent lock (valid means that it either points to the same row or
to a deleted row).
That is what we concluded from the previous discusssion about validity
of RowLocation. If you in the future need to make code, or there already
is code, which breaks this assumption, we would need to know which other
mechanisms we should use to either verify that the RowLocation is valid,
or to block the system to make it invalid.
There is already code that breaks this assumption, the in place compress
table. It currently is only executed as a call to a system procedure:
http://db.apache.org/derby/docs/dev/ref/rrefproceduresinplacecompress.html
But the hope was in the future to do the same kind of work that this
procedure does, internally in background.
As you have already determined that off-line compress is even more of a
problem, but it does run under an exclusive table lock. After it runs
the container you were connected to does not even exist any more.
The online compress seems to require table-exclusive locks when passing
the defragment or truncate_end argument.
There are two testcases in jdbcapi/ConcurrencyTest which test this
(testDefragmentDuringScan and testTruncateDuringScan).
The tests first deletes all rows except the first and the last. Then it
commits. Then it opens a SUR in read-uncommitted mode. Read all records
into the resultset, and position the cursor to afterlast.
Then it runs the defragment or truncate. The test will hang waiting for
a lock. Finally, it updates the rows correctly.
Output from the tests (modified the test slightly to get more output):
T1: delete records
T1: commit
T2: Read next Tuple:(0,0,17)
T2: Read next Tuple:(9,9,35)
T3: call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE
T3: DEFRAGMENT
ERROR 40XL1: A lock could not be obtained within the time requested
....
org.apache.derby.impl.store.access.RAMTransaction.openScan(RAMTransaction.java:1582)
at
org.apache.derby.iapi.db.OnlineCompress.setup_indexes(OnlineCompress.java:605)
at
org.apache.derby.iapi.db.OnlineCompress.defragmentRows(OnlineCompress.java:359)
at
org.apache.derby.iapi.db.OnlineCompress.compressTable(OnlineCompress.java:227)
at
org.apache.derby.catalog.SystemProcedures.SYSCS_INPLACE_COMPRESS_TABLE(SystemProcedures.java:858)
...
at
org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testCompressDuringScan(ConcurrencyTest.java:777)
at
org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testDefragmentDuringScan(ConcurrencyTest.java:712)
...
T3: got expected exception
T1: Read first Tuple:(0,0,17)
T1: updateInt(2, 3);
T1: updateRow()
T1: Read last Tuple:(9,9,35)
T1: updateInt(2, 3);
T1: updateRow()
T1: commit
T4: select * from table
T4: Read next Tuple:(0,3,17)
T4: Read next Tuple:(9,3,35)
So to me it seems that our assumptions are correct.. Only a purge is
allowed with row-level locking, online compress and online defragment
seems to be blocked by an open cursor on the table.
Maybe that is not how online compress was intended to be ?
Andreas
Locks can be used to manage cross thread dependencies, however they are
bound to the transaction, and therefore does not help very much for
cursors held across commits. So if the only mechanism we can have to
ensure that the RowLocations are valid, is by the means of locks, I
doubt we can support the feature of scrollable insensitive *holdable*
updatable resultset.
I agree, I don't believe derby is currently architected to correctly
implement "holdable" SUR. I don't think the outside event driven
approach is the right way to go.
It is true that the current access methods don't reuse row locations
until a table level lock is granted. But your project would be the
first dependency on this outside of the access method implementations
themselves. It is very clear the contract that the access methods
have with their clients while locks are held on the data they are
looking at, what you are proposing is a contract on unlocked data.
I guess we are the first to use RowLocation without holding a lock on
the row. This is necessary, unless we simply make SUR cursors set locks
for all rows in the cursor independent from isolation level.
Note that the current "in-place" compress will MOVE rows from one
row location to another if one does not have a row lock on the row.
This is done in the 2nd phase and only holds an intent lock, and
exclusive row locks on the rows being moved.
The off-line compress only does work under an X table lock.
So the row that you are updating actually will exist in the table,
but currently you will request the old location and will get back
a delete row indicator. I think because of this option 1 does not
work.
Are you saing that RowLocations can be invalidated by "in-place"
compress even if we hold a Table intent lock ?
The "in-place" compress will do the following take a row a row location
N and do the
following:
insert row at new row location M
delete row at old row location N
possibly purge old row location
I don't understand how this will affect your code, it depends how you
search for the row, and what errors are ok to throw.
