from:"Andreas Korneliussen"


Daniel John Debrunner wrote:

DerbyJUnitTest has many cases where the SQLException thrown by Derby
(through JDBC) is ignored. To me, this seems very bad practice for test
code.


Isn't that bad practice for any code ?

Anyway, I think there are cases when you are interested in not throwing 
the SQLException to the JUnit framework.


If your test has already failed with a meaningful assertion which 
describes the error, the junit- test runner will call the tearDown() 
method on your TestCase object, which may throw an exception.
If the tearDown() throws i.e "ResultSet is closed", that error will 
overwrite the original test failure in the JUnit test report.


An alternative to swallowing the exception, is then of course to log it, 
or to gather them somehow, and report them at the end of running the 
testsuite.



For example:

//
// Swallow uninteresting exceptions when disposing of jdbc objects.
//
protected   static  voidclose( ResultSet rs )
{
try {
if ( rs != null ) { rs.close(); }
}
catch (SQLException e) {}
}   

If ResultSet.close is throwing an exception when being closed then to my
mind that's a bug, but we would never see it through a Junit test.

Unless the resultset is already closed, or you have closed the 
connection for the resultset.


Andreas

Re: Ignored exceptions in DerbyJUnitTest




If ResultSet.close is throwing an exception when being closed then to my
mind that's a bug, but we would never see it through a Junit test.

Unless the resultset is already closed, or you have closed the 
connection for the resultset.


Correction: I guess ResultSet.close() should not throw an exception if 
it is already closed.


My point was anyway that we should be careful about throwing exceptions 
to the junit-framework in the tearDown() since it will overwrite the 
"real" error.


Andreas

running JUnit tests

It seems to me that for including a new JUnit test into i.e derby-all we 
need to make a new java class with a main() method, which parses a 
command line and set up the testsuite and run it, just like any java 
program. Basically we are running the junit tests as test type "java".


Instead of having to do this for every junit test going into a derby 
test suite, I would propose a different strategy.


I propose to introduce a new test type called "junit" (current test 
types are: sql,sql2,unit,java,multi,demo - unit is not junit)


Then you can use:

java org.apache.derbyTesting.functionTests.harness.RunTest 
.junit


to run a Junit test - instead of:

java org.apache.derbyTesting.functionTests.harness.RunTest 
.java


When starting a test of type junit, the RunTest class may simply use the
junit.textui.TestRunner class, which has a main method which takes a 
TestCase class name as parameter.  The junit.textui.TestRunner  runs the 
tests defined by the suite() method of the TestCase class.


I think this strategy will make it easier to integrate new JUnit tests 
into the current test suites, since it save you the trouble of creating 
a java class with a main method for every test.



-- Andreas

Re: running JUnit tests





Is there any write up of the correct way to write new Junit tests?


I am trying to follow the junit cookbook when writing new junit tests.
http://junit.sourceforge.net/doc/cookbook/cookbook.htm

I do have some thoughts on how to write new JUnit Tests.

When writing tests for Derby (using JDBC), I think most tests need to 
have access to some configuration data, like jdbc url, jdbc classname 
etc.  This could be put/gathered  into a immutable singleton object

(i.e DerbyTestConfig) which can be accessed by the tests.
Alternatively it could be gathered in a common testcase class, like 
DerbyJUnitTestCase, however then you may need to duplicate this code if 
you make i.e TestDecorators which are not subclasses of DerbyJUnitTestCase.


Since many tests are using JDBC connections, it could be useful to have 
a common TestCase base class which sets up a connection in the setUp() 
and closes it in the tearDown().  I.e:


class DerbyConnectionTestCase extends TestCase {

  protected Connection con;

  final void setUp() throws Exception
  {
 con =  DriverManager.connect(config.getJdbcUrl(), ..);
 doSetup();
  }

  final void tearDown() throws Exception
  {
 doTearDown();
 con.close(); <- of course with some better exception handling..
  }
..
}


I think a big mistake we made with the old harness was no formal way to
add Java tests, which meant multiple ways to start the engine and obtain
the connection and multiple utility methods (almost) performing the same
action.



When it comes to setting up additional fixture before running a set of 
testcases, I think a powerful and reusable mechanism is the 
junit.extensions.TestSetup. I.e one could make a DerbyNetworkServerSetup 
class. In its setUp() method it starts the network server, and in the 
tearDown() in stops it.  So if you have a suite of tests which requires 
a running network server, they can be wrapped like this:


TestSuite suite = new TestSuite(MyDerbyTestCase.class);
Test t = new DerbyNetworkServerSetup(suite);

When running Test t, it will first call setUp() in 
DerbyNetworkServerSetup, then it will run all testcases in 
MyDerbyTestCase, and finally call tearDown() in DerbyNetworkServerSetup.


Of course, one should consider if DerbyNetworkServerSetup should simply 
do nothing if the test is only testing in embedded mode (i.e make it 
dependent on the test configuration).




Would be nice if there were guidelines provided by whoever set up the
initial Junit framework.



I think Rick H /David set it up, so maybe they have some more guidelines 
or ideas.


--Andreas

Re: [VOTE] Rick Hillegas as a committer

2006-01-11 Thread Andreas Korneliussen


+1

Andreas

Re: running JUnit tests

2006-01-11 Thread Andreas Korneliussen


John Embretsen wrote:

Andreas Korneliussen wrote:

I propose to introduce a new test type called "junit" (current test 
types are: sql,sql2,unit,java,multi,demo - unit is not junit)


Then you can use:

java org.apache.derbyTesting.functionTests.harness.RunTest 
.junit


to run a Junit test - instead of:

java org.apache.derbyTesting.functionTests.harness.RunTest 
.java



If I understand this proposal correctly, there will be a one-to-one 
mapping between so-called ".junit" tests and ".java" files, right? And 
there will be no actual files ending with ".junit", but the harness will 
 map the test names ending with ".junit" to actual java/class files 
containing the actual JUnit test code?


Yes, the class name.

The harness uses the extension of the file submitted to figure out the 
test type.


I.e if it is .sql it starts ij, if it is .java it starts java with the 
class as argument, if it is unit it starts 
org.apache.derbyTesting.unitTests.harness.UnitTestMain, if it is multi 
it starts org.apache.derbyTesting.functionTests.harness.MultiTest


Which java program it starts is defined in the
buildTestCommand(..) method in 
org.apache.derbyTesting.functionTests.harness.RunTest


The proposal is that if it is junit, it starts "java 
junit.textui.TestRunner" application with the class as argument.


Andreas

Re: Features of the JUnit test execution harness

2006-02-02 Thread Andreas Korneliussen


Myrna van Lunteren wrote:
On 1/31/06, *Kristian Waagan* <[EMAIL PROTECTED] 
> wrote:


Differences in output should be irrelevant. Although not what you
mentioned above, the issue of (execution) control is very relevant. The
logic for running the tests multiple times, each time with a different
setup/environment must be located somewhere. I think Andreas' proposal
of introducing a separate JUnit test type (see
http://www.nabble.com/running-JUnit-tests-t887682.html#a2300670) makes
sense, as it gives us more freedom w.r.t. handling of JUnit tests.

 
Yes, that proposal made sense to me. I personally like the approach of 
having a class for various/different configurations. Although that could 
get out of hand.
 
Does this 'throw away' the work that Rick is doing on DERBY-874?
 


I think the work currently done on DERBY-874 was mainly to improve the 
DerbyJUnitTest's JavaDoc, and to log exceptions. So I would not throw 
that away.


However I do propose to change DerbyJUnitTest to move out everything 
about configuration into a separate class.


Following Andreas' approach we'd still be able to run the individual 
tests separately, yes?
 


Yes - definetly.


Andreas

Re: Features of the JUnit test execution harness

2006-02-03 Thread Andreas Korneliussen


Myrna van Lunteren wrote:
On 2/2/06, *Andreas Korneliussen* <[EMAIL PROTECTED] 
<mailto:[EMAIL PROTECTED]>> wrote:


I think the work currently done on DERBY-874 was mainly to improve the
DerbyJUnitTest's JavaDoc, and to log exceptions. So I would not throw
that away.

However I do propose to change DerbyJUnitTest to move out everything
about configuration into a separate class.

 
cool. thx for the reply.
 
I now noticed that the wiki says all suggestions are to be put on the 
list, so here I go rather than plopping them directly on the wiki:
 

Great feature list.

What I think is important is that we provide a common mechanism to get 
access to the configuration of the test.


If someone then need to do some framework/version specific logic, they 
have a mechanism to get i.e the framework enum, and can write logic 
based on that.


I think the following could qualify as 'more details' to the jvm, 
framework, version specific logic:
 
1. jvm-specific:

1.1.
not all parameters are consistent for all jvms. Think here of jit 
settings / configurations, memory settings. For j2ME testing, that jvm 
doesn't come with a DriverManager implementation, so already from the 
start you know you have to go with DataSources.


So I guess what you are saying is that if the test framework provides a 
common mechanism to give a Connection to a derby database, it should go 
through a DataSource, instead of using DriverManager ?


1.2. Different versions of a vendor's jvm also may have slightly 
different implementations resulting in slightly different behavior - 
e.g. of the order of rows, for instance, or rounding of certain numeric 
values.

1.3. Some behavior will be only supported by later versions...
 
2. version specific.
This really falls back to the discussion on ...(can't find right now, 
raman's working on it, I think)... re mixed version testing. I think the 
conclusion was that the harness needs a way to cope with results from 
newer servers and clients - if they differ from results with same 
versions as the harness.
 
3. framework specific

The tests needs to be able to cope with the following
3.1. different client drivers (e.g. DerbyClient, IBM Universal JDBC Driver)
3.2. server may need to be started by harness, or not
3.3. server may be on the localhost, or be running on a remote machine.
 certain individual tests may not be able to run in with this 
mechanism...

3.4 should be able to have the harness start the server in a differrent jvm.
 
4. one thing the current harness has no way of doing is to cope with 
different OSs. For instance, sometimes there are slight differences in 
behaviour of the same jvm version on different OSs. Like slightly 
different error messages (although this may be irrelevant if we're not 
gathering & comparing output).
 
I think the following details would be useful (in addition to the above 
and item 1 on the wiki):
- there must be a way to skip individual tests without causing an error 
but with an informational message for certain configurations. eg. 
absence of optional jars (specifically thinking of db2jcc.jar), 
unsupported functionality with older jvms..., or when there is a problem 
that's being worked on, or that's been referred to some other 
organization ( e.g. in the case of jvm bugs, OS bugs...).
 
- some way to compare runtimestatistics.
   Currently this is done by comparing the output, I have a hard time 
thinking of another mechanism.




I am not sure which runtimestatistics you think of. Which output ? 
Output from a SQL statement ?



Thanks

--Andreas

testScript.xml and DerbyJUnitTest refactoring


I have considered refactoring DerbyJUnitTest because of the following:

The way DerbyJUnitTest supports configuring the testcase may be error 
phrone, since it requires the calling of a number of static methods in a 
 specific order for the static members to initialize.  I.e it is not 
possible to run any of the exisiting JUnit tests using a standard Junit 
 testrunner without getting a NullPointerException. The static members 
are also non-final, and this allows testcases to call modifiers, 
potentially causing side-effects in other testcase objects.


Example:DerbyJUnitTest has a public static method called 
setDatabaseName(String dbName). If I do not call it, the _databaseName 
variable is null.  If one of my testcases call the setDatabaseName(..) 
method, it has side-effects which will affect all other testcases (they 
may start using another database).


The way DerbyJUnitTest supports configuring the correct JDBC client, is 
by having a String[] of client settings for a framework. This can be 
initialized by calling setClient(..) or findClientFromProperties()


private static String[] _defaultClientSettings;

	public	static	void		setClient( String[] client ) { 
_defaultClientSettings = client; }


	public	boolean	usingDB2Client() { return ( _defaultClientSettings == 
DB2JCC_CLIENT ); }


public  static  final   String[][]  LEGAL_CLIENTS =
{
DB2JCC_CLIENT,
DERBY_CLIENT,
EMBEDDED_CLIENT
};

The problem with this approach is that:
1. I may call setClient(..) with any array of strings not being part of 
LEGAL_CLIENTS. This will cause the methods usingDB2Client(), 
using...Client() to all return false, even if my String[] array contains 
a valid set of strings for JDBC client settings.


2. Calling setClient(..) has side-effects on other testcases, and it 
allows "non-standard" initialization of the client settings


3. If I do not call setClient(..) or findClientFromProperties().. 
methods like getConnection() fails with NullPointerException


4. The problem of configuring the client settings has been solved 
before, i.e in TestUtil, where there is a integer enum representing the 
framework.


One thing preventing me from refactoring DerbyJUnitTest is that I 
noticed the testScript.xml in CompatibilitySuite.  This seems to be a 
well-documented test harness for running the CompatibilitySuite against 
different versions of jdbc clients and derby servers. It consists of 
multiple "for loops" implemented in ant, calling 
CompatibiltySuite.main() with different arguments.


CompatibilitySuite.main takes advantage of the public setClient(..) 
method in DerbyJUnitTest, and the client settings are initialized using 
another mechanism than the one in i.e findClientFromProperties(), or any 
other mechanism found in the Derby test harness.


The main() method in the CompatibilitySuite is called from this script 
with the drivername, and by using stringmatching against drivernames in 
the LEGAL_CLIENTS[i][DRIVER_NAME], the correct client is loaded, and set 
using setClient(..).


My goal is to be able to run Derby junit tests from the junit 
testrunner, and to allow easy integration of JUnit tests into the 
current harness in Derby. Maintaining another test harness written as an 
ant.xml script is not really my itch. My question is therefore: could I 
leave it to someone else to maintain testScript.xml, or could it 
alternatively be removed ?


(Script 
org/apache/derbyTesting/functionTests/tests/junitTests/compatibility/testScript.xml)


--Sincerely

Andreas

Re: [jira] Commented: (DERBY-934) create a set of JUnit tests for Scrollable Updatable Resultsets


Daniel John Debrunner (JIRA) wrote:
[ http://issues.apache.org/jira/browse/DERBY-934?page=comments#action_12365574 ] 


Daniel John Debrunner commented on DERBY-934:
-

Is this the plan for junit tests, to have a test directory per function, namely 
the 'sur' directory here?



I did not find that the SUR tests naturally belongs in any of the 
existing directories in junitTests, so I created a new package which is 
descriptive for the type of tests. If someone finds a better package 
name, I will gladly update it.


 junitTests/sur/SURTest.junit 


If these tests were added under the old harness the correct location would be



Note: I consider junit tests as a new test type which can be run under 
the (old) harness, and which in the future can be run in another harness.



tests/jdbcapi



Or should it be tests/lang, since updatableResultSet.java (tests 
non-scrollable updatable results in the old harness) is located in 
tests/lang ?



The existing junitTests sub-directories are higher level than a set of 
functionality and two of the three match the exisitng harness layout, lang and 
bderyNet



True.

Andreas

Re: [jira] Commented: (DERBY-934) create a set of JUnit tests for Scrollable Updatable Resultsets

Hi all,
Thanks for reviewing the patch.

The junitTest directory was not introduced as part of this patch. it had
been introduced from before, and all (3-4) junit tests are in that
subdirectory.

However I agree that it is not necessary to have a separate tree for
JUnit type tests - so what should I do ?

a, put the new tests into the tests/jdbcapi directory
b, put them into a new directory : tests/junitTest/jdbcapi

If I go for a, the existing junit tests should then also be moved.

Either way is fine with me.

--Andreas

David W. Van Couvering wrote:

I am also a bit uncomfortable with a separate subdirectory for junit
tests. Is there a specific reason for this segregation? If we have a
new test type, .junit, why can't that be sufficient to distinguish
between a JUnit test and an "old-style" test? I'd like to reuse the
existing subdirectories like lang and jdbcapi rather than create a
mirror of subdirectories, or even a completely orthogonal set of
subdirectories, under junitTests.

David

Daniel John Debrunner (JIRA) wrote:

[
http://issues.apache.org/jira/browse/DERBY-934?page=comments#action_12365574
]

Daniel John Debrunner commented on DERBY-934:
-

Is this the plan for junit tests, to have a test directory per
function, namely the 'sur' directory here?

junitTests/sur/SURTest.junit
If these tests were added under the old harness the correct location
would be

tests/jdbcapi

The existing junitTests sub-directories are higher level than a set
of functionality and two of the three match the exisitng harness
layout, lang and bderyNet

create a set of JUnit tests for Scrollable Updatable Resultsets
---

Key: DERBY-934
URL: http://issues.apache.org/jira/browse/DERBY-934
Project: Derby
Type: Sub-task
Components: Test
Reporter: Andreas Korneliussen
Assignee: Andreas Korneliussen
Attachments: DERBY-934.diff, DERBY-934.stat

Add a set of JUnit tests which tests the implementation for
Scrollable Updatable ResultSets.

The following is a description of how the tests will be implemented:
Data model in test:
We use one table containing three int fields and one varchar(5000)
field. Then we run the tests on a number of variants of this model:
1. None of the fields are indexed (no primary key, no secondary key)
2. One of the fields is indexed as primary key
3. One of the fields is indexed as primary key, another field is
indexed as secondary key
4. One field is indexed as secondary key
(primary key is unique, secondary key is not unique)
By having these variations in the data model, we cover a number of
variations where the ScrollInsensitiveResultSet implementation uses
different classes of source ResultSets, and the CurrentOfResultSet
uses different classes of target and source ResultSet.
The table can be created with the following fields:
(id int, a int, b int, c varchar(5000))
-
Queries for testing SUR:
Select conditions:
* Full table scan
SQL: SELECT * FROM T1
* Full table scan with criteria on non-indexed field
SQL: .. WHERE c like ?
SQL: .. WHERE b > ?
* Full table scan with criteria on indexed field
SQL: .. WHERE id>a
* SELECT on primary key conditionals:
- Upper and lower bond criteria:
SQL: .. WHERE ID>? and IDSQL: .. WHERE ID in (1,2,3,4) SQL: .. WHERE a in (1,2,3,4) (Other
nested queries containing a table seems to not permit updates)

* SELECT on secondary key conditionals:
SQL: .. WHERE a>? and aTest that you get an exception when specifying update clause "FOR
UPDATE"

along with a query which is not updatable.
Cases:
* Query containing order by
* Query containing a join
Test that you get an exception when attempting to update a ResultSet
which has been downgraded to a read only ResultSet due to the query
Cases:

* Query contained a join
* Query contained a read only update clause
* Query contained a order by
Test that you get an exception when attempting to update a ResultSet
which has concurrency mode CONCUR_READ_ONLY

Concurrency tests:
(ConcurrencyTest)
Cases: * Test that update locks are downgraded to shared locks after
repositioning. (fails with derby)
* Test that we can aquire a update lock even if the row is locked with
a shared lock.
* Test that we can aquire a shared lock even if the row is locked with
an update lock.
* Test that we do not get a concurrency problem when opening two
cursors as readonly.
* Test what happens if you update a deleted and purged record
* Test what happens if you update a deleted and purged record using
positioned update
* Test what happens if you update a tuple which is deleted and then
reinserted.
* Test what happens if you update a tuple which is deleted and then
reinserted with the exact same values
* Test what happens if you update a tuple which has been modified by
another transaction.
* Test that you cannot comp

Re: [VOTE] Knut Anders Hatlen as committer

+1

Re: testScript.xml and DerbyJUnitTest refactoring


Rick Hillegas wrote:


..


When you have a harness that runs JUnit tests, then I can write a new 
script for running the compatibility combinations. But please don't 
throw this suite away. Instead, before you change DerbyJUnitTest, 
maybe you can clone it to some other class just for use by the 
compatiblity suite. Once you have revamped DerbyJUnitTest and built 
out the harness, I can convert the compatibility test and suite.



Great, thanks for the feedback - I think your suggestion will work fine

-- Andreas

Re: [jira] Commented: (DERBY-934) create a set of JUnit tests for Scrollable Updatable Resultsets

2006-02-09 Thread Andreas Korneliussen


David W. Van Couvering wrote:
I would say you could put the new tests into the tests/jdbcapi directory 
and log a JIRA to move over the other tests, rather than try and swallow 
that into your patch, that's too much for one patch.


David


Yes, I agree, it should go in different patches.

Andreas

Re: [patch] cleanup of org.apache.derby.iapi.services.cache.ClassSize

2006-02-09 Thread Andreas Korneliussen


Daniel John Debrunner wrote:

Kev Jackson wrote:


Following on from yesterdays post, here's a patch that tries to get this
class to conform to the coding standards chosen by the Derby developers



Except that we have not chosen a coding standard. :-)


I just read the following on http://db.apache.org/source.html :

-
All Java Language source code in the repository must be written in 
conformance to the " Code Conventions for the Java Programming Language 
as published by Sun, or in conformance with another well-defined 
convention specified by the subproject. See the FAQ page  for links to 
subproject conventions.

-

The link to the FAQ page was dead.

I guess that if the subproject has not defined a coding standard, it 
should use the standard of the "super" project ?


Andreas



http://wiki.apache.org/db-derby/DerbyContributorChecklist

and in another e-mail:


I'm willing to contribute as I'd like to learn a bit about how databases work 
under the hood,
but I can't stand the current code style and I'd like to know if this is an 
agreed upon convention,
or a legacy of the various owners of the codebase up to now.



The history of the code (from its inception over 8 years ago) was that
the only coding standard in force was to be clear, and don't make
changes in the code just for formatting sake. It was just not worth the
time originally to try and get everyone to agree on a single standard,
some people like braces one way, some the other, some like spaces in
different positions in loops or if statements etc, sometimes a single
rule is not applicable for all situations etc. etc.

The issue is that you say you can't stand the current code style, but
maybe someone else can't stand a code style you like, hard to please
everyone. :-)

Please do get involved in Derby, look at the jira issues marked with a
component of Newcomer, or fix a bug thet scratches your itch with Derby.
Many on the list will be willing to guide you, just ask.



- remove unused import
- change static final variables from camelCase to CONST_NAMES (except
where they are public and could be used elsewhere BWC concern)


[just curious, what's BWC?]



- add {} for conditionals
- strip lots of extra white space



As Myrna said, these type of changes can cause problems for merges of
fixes across branches and also to others with existing edits against the
same files.



As for the CLA/ICLA, I've already signed one (for ant), so do I need to
sign another for Derby?



Nope, I belive a single ICLA at the ASF is sufficient.

Thanks,
Dan.

Re: [jira] Commented: (DERBY-934) create a set of JUnit tests for Scrollable Updatable Resultsets

2006-02-09 Thread Andreas Korneliussen


Myrna van Lunteren wrote:
When - I think Rick - made the first junit test, I suggested a different 
directory for it.
 
I was thinking of a future time, when the current test harness would be 
completely replaced by a junit-based framework. In that rosy future, all 
'converted', and any 'new' tests would be junit based, until finally, 
nothing worthwhile remained under the current 
/java/testing/org/apache/derbyTesting/functionTests dir.
In that scenario, I was thinking the junit tests, and presumably some 
sort of framework/harness for gathering totals etc. to show up under 
/java/testing/org/apache/derbyTesting/junitTests/...

With comparable directory structure as is currently under functionTests.
 
This is not where the tests junit tests ended up. As it is now, we might 
as well put them in their logical place under functionTests/tests.




I think the general feedback I get is that it is OK to put the junit 
SUR-tests into: ../functionTests/tests/jdbcapi, so I have therefore 
updated the patch.


As far as I know, svn is capable of handling moving of repository files 
(svn move), so if someone want to refactor the test hiearchy for junit 
tests in the future, they could do so while keeping the history of the 
junit tests. Of course, getting it perfect the first time would have 
been preferrable.


-- Andreas

Re: [jira] Updated: (DERBY-918) introduce a new test type to run junit tests from the current harness

David W. Van Couvering wrote:
I would have added this as a patch but as I understand it the JIRA 
comment mailer is down...

I applied this patch and everything builds fine, but I did notice that I 
got failures with the existing Junit tests when I ran them as .junit 
instead of .java.

Do the existing JUnit tests need to be modified somehow?  Has this 
thread already been run and I just missed it?

Yes, they need to be modified to successfully run, since they depend on 
running some static methods to initialize some variables - otherwise 
they will give nullpointerexception.

I think I commented on that while uploading the patch.

I'm trying to find out how to validate if this patch actually works, any 
suggestions, since it doesn't appear to come with any junit tests that 
actually work with it.  Any suggestions?

You could download the the tests in 934 and run them with this patch.
--Andreas

Thanks,

David

+ java org.apache.derbyTesting.functionTests.harness.RunTest 
junitTests/lang/Boo

leanTest.junit
*** Start: BooleanTest jdk1.4.2_07 2006-02-09 10:54:27 ***
0 add
 > .E
 > There was 1 error:
 > 1) 
testBoolean(org.apache.derbyTesting.functionTests.tests.junitTests.lang.Boo

leanTest)java.lang.NullPointerException

*** Start: CompatibilityTest jdk1.4.2_07 2006-02-09 10:55:43 ***
0 add
 > .F
 > There was 1 failure:
 > 1) 
warning(junit.framework.TestSuite$1)junit.framework.AssertionFailedError: N
o tests found in 
org.apache.derbyTesting.functionTests.tests.junitTests.derbyNet

.CompatibilityTest
 > FAILURES!!!
 > Tests run: 1,  Failures: 1,  Errors: 0
Test Failed.

$ runtest.sh junitTests/lang/BooleanTest.java
+ java org.apache.derbyTesting.functionTests.harness.RunTest 
junitTests/lang/Boo

leanTest.java
*** Start: BooleanTest jdk1.4.2_07 2006-02-09 15:49:15 ***
*** End:   BooleanTest jdk1.4.2_07 2006-02-09 15:49:35 ***

$ runtest.sh junitTests/derbyNet/CompatibilityTest.java
+ java org.apache.derbyTesting.functionTests.harness.RunTest 
junitTests/derbyNet

/CompatibilityTest.java
*** Start: CompatibilityTest jdk1.4.2_07 2006-02-09 15:48:37 ***
0 add
 > java.sql.SQLException: No suitable driver
 > Exception in thread "main"
Test Failed.
*** End:   CompatibilityTest jdk1.4.2_07 2006-02-09 15:48:53 ***

Andreas Korneliussen (JIRA) wrote:

 [ http://issues.apache.org/jira/browse/DERBY-918?page=all ]

Andreas Korneliussen updated DERBY-918:
---

Attachment: (was: DERBY-918.diff)

introduce a new test type to run junit tests from the current harness
-

Key: DERBY-918
URL: http://issues.apache.org/jira/browse/DERBY-918
Project: Derby
   Type: Improvement
 Components: Test
Environment: All
   Reporter: Andreas Korneliussen
   Assignee: Andreas Korneliussen

It seems to me that for including a new JUnit test into i.e derby-all 
we need to make a new java class with a main() method, which parses a 
command line and set up the testsuite and run it, just like any java 
program. Basically we are running the junit tests as test type "java".
Instead of having to do this for every junit test going into a derby 
test suite, I would propose a different strategy.
I propose to introduce a new test type called "junit" (current test 
types are: sql,sql2,unit,java,multi,demo - unit is not junit)

Then you can use:
java org.apache.derbyTesting.functionTests.harness.RunTest 
.junit

to run a Junit test - instead of:
java org.apache.derbyTesting.functionTests.harness.RunTest 
.java

When starting a test of type junit, the RunTest class may simply use the
junit.textui.TestRunner class, which has a main method which takes a 
TestCase class name as parameter.  The junit.textui.TestRunner  runs 
the tests defined by the suite() method of the TestCase class.
I think this strategy will make it easier to integrate new JUnit 
tests into the current test suites, since it save you the trouble of 
creating a java class with a main method for every test.

Re: [jira] Commented: (DERBY-934) create a set of JUnit tests for Scrollable Updatable Resultsets

David Van Couvering (JIRA) wrote:
[ http://issues.apache.org/jira/browse/DERBY-934?page=comments#action_12365844 ]

David Van Couvering commented on DERBY-934:
---

One final comment: that was a *lot* of code to review in one go. In the future it would
be great if you could submit this in smaller, more manageable chunks. It will ensure a
better, closer review and won't cause "review fatigue" on us poor committers.

Hi David,
Thank you very much for the hard work of reviewing all my patches. I am
going through your comments and will update the patch today.

I did think this patch would be fairly managable, since it consist of
new files only, however I could of course have splitted it into three
patches, one for concurrency tests, one for sur query mix and one for
the surtest. Wouldn't that just be more work for both the committer
and developer ?

--Andreas

David

create a set of JUnit tests for Scrollable Updatable Resultsets
---

Add a set of JUnit tests which tests the implementation for Scrollable
Updatable ResultSets.
The following is a description of how the tests will be implemented:
Data model in test:
We use one table containing three int fields and one varchar(5000)
field.
Then we run the tests on a number of variants of this model:

1. None of the fields are indexed (no primary key, no secondary key)
2. One of the fields is indexed as primary key
3. One of the fields is indexed as primary key, another field is
indexed as secondary key
4. One field is indexed as secondary key
(primary key is unique, secondary key is not unique)
By having these variations in the data model, we cover a number of
variations where the ScrollInsensitiveResultSet implementation uses
different classes of source ResultSets, and the CurrentOfResultSet
uses different classes of target and source ResultSet.
The table can be created with the following fields:
(id int, a int, b int, c varchar(5000))
-
Queries for testing SUR:
Select conditions:
* Full table scan
SQL: SELECT * FROM T1
* Full table scan with criteria on non-indexed field
SQL: .. WHERE c like ?
SQL: .. WHERE b > ?
* Full table scan with criteria on indexed field
SQL: .. WHERE id>a
* SELECT on primary key conditionals:
- Upper and lower bond criteria:
SQL: .. WHERE ID>? and IDSQL: .. WHERE ID in (1,2,3,4)
SQL: .. WHERE a in (1,2,3,4)
(Other nested queries containing a table seems to not permit updates)

* SELECT on secondary key conditionals:
SQL: .. WHERE a>? and aTest that you get an exception when attempting to update a ResultSet
which has been downgraded to a read only ResultSet due to the query
Cases:

Concurrency tests:
(ConcurrencyTest)
Cases:
* Test that update locks are downgraded to shared locks after

repositioning. (fails with derby)
* Test that we can aquire a update lock even if the row is locked with
a shared lock.
* Test that we can aquire a shared lock even if the row is locked with
an update lock.
* Test that we do not get a concurrency problem when opening two
cursors as readonly.
* Test what happens if you update a deleted and purged record
* Test what happens if you update a deleted and purged record using
positioned update
* Test what happens if you update a tuple which is deleted and then
reinserted.
* Test what happens if you update a tuple which is deleted and then
reinserted with the exact same values
* Test what happens if you update a tuple which has been modified by
another transaction.
* Test that you cannot compress the table while the ResultSet is open,
and the transaction is open (for validity of RowLocation)
* Test that you cannot purge a row if it is locked
* Test that Derby set updatelock on current row when using
read-uncommitted

Re: [jira] Updated: (DERBY-795) After calling ResultSet.relative(0) the cursor loses its position


David W. Van Couvering wrote:
Hi, Andreas.  Upon further thought, once we work through the comments on 
DERBY-934 (still to come) I am going to go ahead and apply all these 
patches at once, no need for you to do extra work.  But a request for 
next time to please try and keep your patches independent instead of 
interdependent.



That is great.
I think the patches are independent, however they are slightly related.

DERBY-918: a patch for improving the test harness

DERBY-934: a set of tests which can be run independentely using 
junit.textui.TestRunner or by using the test harness with patch 918


DERBY-795: a patch for a specific bug in derby. I did not submit extra 
tests for this, since it is covered in 934, however there is a simple 
java program there, which can be run to verify the fix and the bug


There are no compile dependencies between these patches. I think it was 
much better to submit these as independent patches, instead of in one 
huge patch.


Andreas



Thanks!

David

David W. Van Couvering wrote:

Hi, Andreas, your set of patches have a set of dependencies which are 
a little confusing at first, and ultimately somewhat intractable:


DERBY-795 is tested by DERBY-934
DERBY-934 can't be run without, and therefore depends upon, DERBY-918

I really can't just commit one of these patches at a time, it has to 
be all or none.


I really would like each of these patches stand on their own, or at a 
minimum don't submit a dependent patch until the patch it depends upon 
has been committed.


Here's what I would like to see:

DERBY-918 comes with its own sample unit test that verifies that the 
.junit test type works.  Something very simple and easy.


DERBY-795 has its own test that comes with it, rather than being 
tested by DERBY-934


I have some comments on DERBY-934 too, I'll send these in a separate 
email.


Thanks,

David

David W. Van Couvering wrote:

Crud, I missed this comment somehow, I'll look at DERBY-934 again, I 
bet *both* my questions will be answered :)


I'll get back to you if I need anything else, Andreas.

David

Andreas Korneliussen (JIRA) wrote:


 [ http://issues.apache.org/jira/browse/DERBY-795?page=all ]

Andreas Korneliussen updated DERBY-795:
---

Attachment: DERBY-795.diff
DERBY-795.diff

Attached is a fix for this issue.
The problem is detected by the jdbcapi/SURQueryMix.junit test 
provided in DERBY-934, when running in embedded mode.




After calling ResultSet.relative(0) the cursor loses its position
-

Key: DERBY-795
URL: http://issues.apache.org/jira/browse/DERBY-795
Project: Derby
   Type: Bug
 Components: JDBC
   Versions: 10.1.2.1
Environment: Any
   Reporter: Andreas Korneliussen
   Assignee: Andreas Korneliussen
   Priority: Minor
Attachments: DERBY-795.diff, DERBY-795.diff

After calling rs.relative(0), on a scrollable ResultSet, the cursor 
looses its position, and a rs.getXXX(..) fails with:

SQL Exception: Invalid cursor state - no current row.
Probably caused by the following logic in 
ScrollInsensitiveResultSet.getRelativeRow(int row):

// Return the current row for 0
if (row == 0)
{
   if ((beforeFirst || afterLast) ||
   (!beforeFirst && !afterLast)) {
   return null;
   } else {
return getRowFromHashTable(currentPosition);
   }
}
The if () will always evaluate to true, regardless of the values of 
beforeFirst and afterLast

Test code:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class RelativeZeroIssue {
  public static void main(String[] args) throws Exception {
  Class.forName("org.apache.derby.jdbc.EmbeddedDriver");
   Connection con = 
DriverManager.getConnection("jdbc:derby:testdb2;create=true");

   con.setAutoCommit(false);
   try {   Statement statement = 
con.createStatement();

  /** Create the table */
   statement.execute("create table t1(id int)");
   statement.execute("insert into t1 values 1,2,3,4,5,6,7,8");
  Statement s = 
con.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,

   ResultSet.CONCUR_READ_ONLY);
   ResultSet rs = s.executeQuery("select * from t1");
   rs.next();
   System.out.println(rs.getInt(1));
   System.out.println(rs.relative(0));
   System.out.println(rs.getInt(1));
   }  finally {
  con.rollback();
   con.close();
   }
   }
 
}

Output from test:
1
false
Exception in thread "main" SQL Exception: Invalid cursor state - no

Re: [jira] Updated: (DERBY-100) Add support for insert functionality using JDBC 2.0 updatable resultset apis


Knut Anders Hatlen wrote:

For rowInserted(), my intention was to make the implementation of
this method as similar as possible to that of the rowDeteled() and
rowUpdated() methods. Since these two methods do not have entry
trancing, I removed it from the rowInserted() method.



Why not add tracing to rowDeleted() and rowUpdated()? :)



I guess because this is for a patch for inserting rows, and someone 
would probably question why she changed unrelated code, arguing that 
such changes should go in a different patch ;-)



Andreas

Re: [jira] Updated: (DERBY-795) After calling ResultSet.relative(0) the cursor loses its position

2006-02-13 Thread Andreas Korneliussen


David W. Van Couvering wrote:
I agree they should be submitted as independent patches.  But they 
should also be committable independently.




They are committable independently, however it is a good idea to commit 
the patches incrementally, as they were submitted:


1. Review and commit DERBY-918. Test with existing junit tests

2. Review and commit DERBY-934. Test using junit.textui.TestRunner or by 
using the feature provided in DERBY-918.


3. Review and commit 795. Test using existing junit tests (934)


I can't verify DERBY-918 without a test that works with the .junit test 
type.  That's provided in DERBY-934.  Why not provide a simple sanity 
Junit test with this patch that I can use to verify?




I agree that it would be great to have a simple sanity junit test.
The feature DERBY-918 is the ability to launch a junit test from the old 
harness, without having to make a new main method for every junit test.


It can be tested with the old junit tests. The fact that the old tests 
fails, (due to the fact that they were never designed to run in 
junit.textui.TestRunner), do not make them valueless in terms of 
verifying the feature.


Then, I can't verify DERBY-795 is fixed without applying and running the 
tests in DERBY-934.  It would be great if DERBY-795 included a test that 
verifies it is fixed as part of the patch.




DERBY-795 comes with a test in the description.  It is not part of the 
patch, since I know this will be tested as part of another patch.


DERBY-934 can be run directly from junit.textui.TestRunner's main, and 
can therefore be verified without DERBY-918.


That would make them *truly* independent, not just in terms of what they 
are about, but in terms of the ability of the committer to review and 
test them independently.




I do not see how it affects the committers ability to review the patches 
independently.  Huge patches with a lot of soon-to-be obsolete code, is 
harder to review than small patches, which can be reviewed and committed 
incrementally.



Andreas


David

Andreas Korneliussen wrote:


David W. Van Couvering wrote:

Hi, Andreas.  Upon further thought, once we work through the comments 
on DERBY-934 (still to come) I am going to go ahead and apply all 
these patches at once, no need for you to do extra work.  But a 
request for next time to please try and keep your patches independent 
instead of interdependent.



That is great.
I think the patches are independent, however they are slightly related.

DERBY-918: a patch for improving the test harness

DERBY-934: a set of tests which can be run independentely using 
junit.textui.TestRunner or by using the test harness with patch 918


DERBY-795: a patch for a specific bug in derby. I did not submit extra 
tests for this, since it is covered in 934, however there is a simple 
java program there, which can be run to verify the fix and the bug


There are no compile dependencies between these patches. I think it 
was much better to submit these as independent patches, instead of in 
one huge patch.


Andreas



Thanks!

David

David W. Van Couvering wrote:

Hi, Andreas, your set of patches have a set of dependencies which 
are a little confusing at first, and ultimately somewhat intractable:


DERBY-795 is tested by DERBY-934
DERBY-934 can't be run without, and therefore depends upon, DERBY-918

I really can't just commit one of these patches at a time, it has to 
be all or none.


I really would like each of these patches stand on their own, or at 
a minimum don't submit a dependent patch until the patch it depends 
upon has been committed.


Here's what I would like to see:

DERBY-918 comes with its own sample unit test that verifies that the 
.junit test type works.  Something very simple and easy.


DERBY-795 has its own test that comes with it, rather than being 
tested by DERBY-934


I have some comments on DERBY-934 too, I'll send these in a separate 
email.


Thanks,

David

David W. Van Couvering wrote:

Crud, I missed this comment somehow, I'll look at DERBY-934 again, 
I bet *both* my questions will be answered :)


I'll get back to you if I need anything else, Andreas.

David

Andreas Korneliussen (JIRA) wrote:


 [ http://issues.apache.org/jira/browse/DERBY-795?page=all ]

Andreas Korneliussen updated DERBY-795:
---

Attachment: DERBY-795.diff
DERBY-795.diff

Attached is a fix for this issue.
The problem is detected by the jdbcapi/SURQueryMix.junit test 
provided in DERBY-934, when running in embedded mode.




After calling ResultSet.relative(0) the cursor loses its position
-

Key: DERBY-795
URL: http://issues.apache.org/jira/browse/DERBY-795
Project: Derby
   Type: Bug
 Components: JDBC
   Versions: 10.1.2.1
Environment: Any
   Reporter: Andreas Korneliussen
   Assignee: Andreas Korneliusse

conflict detection strategies

2006-02-14 Thread Andreas Korneliussen

Some context: In scrollable updatable resultsets, we populate an 
internal table with the following data:


[]+

Example layeout:

  1 <1,10> falsefalse1,"a",3
  2 <1,11> falsefalse2,"b",2
  3 <1,12> falsefalse3,"c",9


When doing updateRow(), or deleteRow(), we use the RowLocation to 
navigate to the row being updated.


Problem:
For holdable cursors, we will release the table intent lock when doing 
commit on the transaction for the cursor.


The table intent lock, prevents the system from doing a compress of the 
table, causing all RowLocations to be invalid. In addition, it prevents 
reuse of RowLocation for deleted + purged rows.


In order to support holdable scrollable updatable cursors, we consider 
having a service which allows the system to notify subscribers (i.e 
cursors) that it has executed i.e a compress.


If the user then calls updateRow() or deleteRow(), we can then give an 
exception like:


"The row could not be updated, because its location has been updated by 
the system"


In addition, we consider having a reclaim of locks, so that immediatly 
after a commit, the new transaction with the holdable cursor, may 
reclaim the table intent lock.  This will reduce the time period which 
the system may compress the table, however not completely remove the 
possibility of a compress.


Any comments on implementing such strategy ?

An alternative to this strategy, could be to go the other way: cursors 
notify the system that it should not do compress.


I would appreciate feedback on this topic, especially if you find any 
pitfalls with the proposed strategies, or have better alternatives.


Andreas

Re: conflict detection strategies

2006-02-14 Thread Andreas Korneliussen


Hi,

The implementation of SUR  just builds on the existing scrollable 
resultsets, which collects all rows into a table. We have extended this 
table to also contain RowLocation and some metadata.
This means we do not need to change the store module to navigate 
backward etc - no changes in the store module.


Updatable cursors in derby  uses RowLocation, however the row is 
guaranteed to be locked (current row has update lock, I think, 
regardless of isolation level).
As for holdable cursors, forward only updatable cursors require the user 
to navigate to the next row after a commit, thereby getting a new 
rowlocation, on a row which is locked.


I will propose a more detailed solution tomorrow, so it becomes more 
clear, and less mysterious, what I really propose :-)

Any other suggestions are of course welcome.

--Andreas

-
From: Mike Matrigali <[EMAIL PROTECTED] 
<http://gmane.org/get-address.php?address=mikem%5fapp%2drphTv4pjVZMJGwgDXS7ZQA%40public.gmane.org>>
Subject: Re: conflict detection strategies 
<http://news.gmane.org/find-root.php?message_id=%3c43F22581.7090600%40sbcglobal.net%3e>
Newsgroups: gmane.comp.apache.db.derby.devel 
<http://news.gmane.org/gmane.comp.apache.db.derby.devel>

Date: 2006-02-14 18:46:25 GMT (59 minutes ago)

I have not been following the scrollable updatable result set work,
I had assumed that the work would be similar to the other resultset
work in Derby with no special requirements from the store.  Is there
a proposed design for this project that I should go look at?  I looked
at the doc associated with DERBY-690, but there are a lot of suggested
approaches - but not clear which choices have been made.

There is a lot of discussion about using the current support for
update where current of.  In the current system how does the system
translate a user request for an update where current of, to the
actual update of the row.  Does it currently use RowLocation?

If possible I would like to see a solution that does not require special
messages sent back and forth between modules about state.



Andreas Korneliussen wrote:

Some context: In scrollable updatable resultsets, we populate an 
internal table with the following data:


[]+

Example layeout:

  1 <1,10> falsefalse1,"a",3
  2 <1,11> falsefalse2,"b",2
  3 <1,12> falsefalse3,"c",9


When doing updateRow(), or deleteRow(), we use the RowLocation to 
navigate to the row being updated.


Problem:
For holdable cursors, we will release the table intent lock when doing 
commit on the transaction for the cursor.


The table intent lock, prevents the system from doing a compress of 
the table, causing all RowLocations to be invalid. In addition, it 
prevents reuse of RowLocation for deleted + purged rows.


In order to support holdable scrollable updatable cursors, we consider 
having a service which allows the system to notify subscribers (i.e 
cursors) that it has executed i.e a compress.


If the user then calls updateRow() or deleteRow(), we can then give an 
exception like:


"The row could not be updated, because its location has been updated 
by the system"


In addition, we consider having a reclaim of locks, so that immediatly 
after a commit, the new transaction with the holdable cursor, may 
reclaim the table intent lock.  This will reduce the time period which 
the system may compress the table, however not completely remove the 
possibility of a compress.


Any comments on implementing such strategy ?

An alternative to this strategy, could be to go the other way: cursors 
notify the system that it should not do compress.


I would appreciate feedback on this topic, especially if you find any 
pitfalls with the proposed strategies, or have better alternatives.


Andreas

Re: Adding SUR tests to derbyall (was Re: [jira] Updated: (DERBY-934) create a set of JUnit tests for Scrollable Updatable Resultsets)


This is great, getting these tests into the suites.

I just ran the SURTest on my sandbox,to compare the output.  The output 
you attached is the same as mine when running with DerbyNetClient.


In network mode, it seems that the JDBC network client is not consistent 
with the embedded client in terms of giving warnings and propagating the 
correct sql state and error message.  I think some of these issues calls 
for new JIRAs.


I have also checked looked more closely at the output from the SURTest 
running in embedded mode.

I think SURTest in some cases reveals some bugs in exisiting Derby code.

Especially, the output from running

testScrollablePositionedUpdateWithForUpdate1() seems to reveal a serious 
bug:


It seems that the current scrollable read-only resultsets allows 
positioned updates if you use "for update" in the query. However, the 
tuple actually being updated, is not necessarily the same as the current 
tuple in the resultset/cursor.  The test therefore fails with an 
assertion when it tries to verify the data in the result set.


I reproduced this error in ij:

ij> get scroll insensitive cursor c1 as 'select * from t for update';
ij> next c1;
ID |C
--
1  |hei
ij> next c1;
ID |C
--
2  |hei igjen
ij> next c1;
ID |C
--
3  |hei hei hei
ij> previous c1;
ID |C
--
2  |hei igjen
ij> previous c1;
ID |C
--
1  |hei
ij> update t set C='hade' where current of c1;
1 row inserted/updated/deleted
ij> select * from t;
ID |C
--
1  |hei
2  |hei igjen
3  |hade

3 rows selected
ij>

So the positioned update, updated tuple 3, instead of tuple 1...!
Once we provide the SUR implementation, this will no longer be an issue.

Andreas


David W. Van Couvering wrote:
So, I pinged Andreas on the backchannel, and he responded that the 
failures with SURTest are expected (as we are still lacking some of the 
Scrollable Updatable ResultSet (SUR) functionality), that the 
SURQueryMixTest should pass when I appy the patch for DERBY-795, and 
that if I remove some printStackTrace debug calls from ConcurrencyTest 
that should run clean.


I made his recommended changes, and confirmed.

So, I would like to see these tests in derbyall.  I re-ran the tests 
with framework=DerbyNetClient and SURQueryMixTest and ConcurrencyTest 
both again pass.  They also pass running with the 1.3 JDK.  I have no 
idea if they will pass or succeed with the IBM JDK as I don't have 
access to that.


Under DerbyNetClient, SURTest has more/different failures and errors. 
Some of these appear to be related to null SQL States.  Andreas, perhaps 
you can look at these failures and tell me if they look OK.  For now I 
won't be adding SURTest to the derbynetclientmats suite.


I also ran with framework=DerbyNet and got hard stop failures saying a 
null username is not supported.


ConcurrencyTest currently has a long timeout so I don't think we should 
have it in derbyall unless we reduce the timeout


So, that all said, I am going to add the following:

- Add master for SURTest for embedded client that has all the expected 
failures.  Andreas, please make sure to update the master as more 
functionality comes in and the results change.


- Add SURTest and SURQueryMixTest to jdbcapi.runall.  NOTE that there 
may be failures with the IBM JDK tests related to this.


- Add SURQueryMixTest to derbynetclientmats.runall.  I'm holding off on 
SURTest.


- Holding off adding ConcurrencyTest to jdbcapi.runall and 
derbynetclientmats.runall until the timeout duration is reduced to a 
reasonable length


Please let me know if you have any comments.

Thanks,

David

David W. Van Couvering wrote:

Hi, Andreas. Are you expecting these tests to pass?  In my environment 
SURTest fails with 8 failures and 52 errors, SURQueryMixTest has 64 
failures and 0 errors, and ConcurrencyTest fails on a lock timeout.


David

Andreas Korneliussen wrote:


David W. Van Couvering wrote:

My JIRA rendering is completely broken, so I am unable to add a 
comment.


I took a look at your patch file, but it appears to be incomplete?  
It ends half way through a method in ConcurrencyTest.




I just downloaded my diff from JIRA, and it was all ok. Maybe it was 
JIRA problem when you downloaded the patch ?


Attached is the diff

Andreas






.F.FF.F.F.F.F...E.E.E.E.E.E.E.E..java.sql.SQLException: invalid 
operation: connection closed
Caused by: org.apache.derby.clie

Re: [jira] Commented: (DERBY-796) jdbc 4.0 specific Blob and Clob method support


  if ((int) pos <= 0) {
  throw new SqlException(agent_.logWriter_,
  new MessageId(SQLState.BLOB_BAD_POSITION), new Long(pos));
  }


Is the casting of pos from long to int safe ? Consider the case if pos 
is > Integer.MAXINT. Is it intentional that pos > Integer.MAXINT gives 
this exception ?


How about:

>   if (pos <= 0L) {
>   throw new SqlException(agent_.logWriter_,
>   new MessageId(SQLState.BLOB_BAD_POSITION), new Long(pos));
>   }

or if it is intentional to give an exception if pos is bigger than 
Integer.MAXINT, one could write it more explicitly:


>   if (pos <= 0L || pos >= MAXPOS) {
>   throw new SqlException(agent_.logWriter_,
>   new MessageId(SQLState.BLOB_BAD_POSITION), new Long(pos));
>   }


Andreas

Re: conflict detection strategies


Daniel John Debrunner wrote:

Andreas Korneliussen wrote:



Problem:
For holdable cursors, we will release the table intent lock when doing
commit on the transaction for the cursor.

The table intent lock, prevents the system from doing a compress of the
table, causing all RowLocations to be invalid. In addition, it prevents
reuse of RowLocation for deleted + purged rows.



I think this last paragraph is an incorrect assuption. The table intent
lock prevents other transactions from doing a compress, but not the
transaction holding the lock.



That is a good point.

The main problem would be the system doing a compress, however we should 
take into account the fact that the user can run compress from the same 
transaction, and then maybe invalidate the resultset, or prevent the 
compress from running.



I think there are other situations where the RowLocation will become
invalid, such as the transaction deleteing the row.



Yes, however as far as I understood, the RowLocation would not be reused 
as long as at least some sort of table level intent lock is held, and 
the store will simply return false if one tries to do update / delete / 
fetch on a RowLocation which is deleted, or deleted+purged.


Andreas

Re: conflict detection strategies


Mike Matrigali wrote:
..

If possible I would like to see a solution that does not require special
messages sent back and forth between modules about state.



I am not entirely sure what restrictions you want to put on the design, 
it is a bit unclear to me.


I have considered some other solutions:

1. Change the locking behaviour, so that a table intent lock which is 
set by an updatable cursor, is kept as long as the cursor is open - this 
will ensure that the RowLocations are valid.


2. After a commit, we could clear all data in the internal table in the 
SUR. The problem with this approach is that the resultset would not 
necessarily be repopulated with the same data - it would be sensitive 
for changes across its own transactions commits, it would be highly 
ineffecient.


3. Let the cursors notify the OnlineCompress module that it should fail 
any attempt to compress/defragment or purge the table.


More details on what I suggested yesterday:

The OnlineCompress class could provide an event mechanism, where 
subscribers (OnlineCompressListener) register themselves to listen to 
OnlineCompressEvents. The ScrollInsensitiveResultSet class could then 
implement the OnlineCompressListener interface, and register itself once 
it starts populating the table with RowLocations. The OnlineCompress 
class then simply notifies all listeners once it is doing defragment / 
compress.
The listeners should unregister themselves (i.e 
ScrollInsensitiveResultSet class could do it once it closes). The 
OnlineCompress class could use a WeakHashMap to put the listeners into, 
in case they are not well-behaved. I have not checked if derby already 
has event manager type of modules, if it does, I would attempt to reuse 
them.


Please also let me know if any of the other alternatives seems better.


Andreas



Andreas Korneliussen wrote:



Some context: In scrollable updatable resultsets, we populate an
internal table with the following data:

[]+

Example layeout:

 1 <1,10> falsefalse1,"a",3
 2 <1,11> falsefalse2,"b",2
 3 <1,12> falsefalse3,"c",9


When doing updateRow(), or deleteRow(), we use the RowLocation to
navigate to the row being updated.

Problem:
For holdable cursors, we will release the table intent lock when doing
commit on the transaction for the cursor.

The table intent lock, prevents the system from doing a compress of the
table, causing all RowLocations to be invalid. In addition, it prevents
reuse of RowLocation for deleted + purged rows.

In order to support holdable scrollable updatable cursors, we consider
having a service which allows the system to notify subscribers (i.e
cursors) that it has executed i.e a compress.

If the user then calls updateRow() or deleteRow(), we can then give an
exception like:

"The row could not be updated, because its location has been updated by
the system"

In addition, we consider having a reclaim of locks, so that immediatly
after a commit, the new transaction with the holdable cursor, may
reclaim the table intent lock.  This will reduce the time period which
the system may compress the table, however not completely remove the
possibility of a compress.

Any comments on implementing such strategy ?

An alternative to this strategy, could be to go the other way: cursors
notify the system that it should not do compress.

I would appreciate feedback on this topic, especially if you find any
pitfalls with the proposed strategies, or have better alternatives.

Andreas

Re: conflict detection strategies




I think there are other situations where the RowLocation will become
invalid, such as the transaction deleteing the row.



Yes, however as far as I understood, the RowLocation would not be reused 
as long as at least some sort of table level intent lock is held, and 
the store will simply return false if one tries to do update / delete / 
fetch on a RowLocation which is deleted, or deleted+purged.




To be clear: we do handle the situation were RowLocation points to a 
deleted row by giving a WARNING if the user tries to do updateRow() or 
deleteRow(). For positioned updates, we will give an update count of 0. 
Therefore we do not really think of those RowLocations as invalid.



Andreas



Andreas

Re: [jira] Updated: (DERBY-934) create a set of JUnit tests for Scrollable Updatable Resultsets

2006-02-16 Thread Andreas Korneliussen


Myrna van Lunteren wrote:
oy, I get better results after svn update than I got with the patches. 
I'll check with various jvms now.
 


Hi,
I do not expect the tests to pass in j2me, since they use DriverManager. 
I did experiement a bit with using ij.startJBMS() to get the Connection, 
however it just returned null, so I guess it failed somewhere and 
swallowed the excpetion. Maybe the TestUtil class could be used instead 
(I have already used it for getting jdbc url etc)


Andreas

Re: conflict detection strategies

2006-02-16 Thread Andreas Korneliussen


Mike Matrigali wrote:
There are very few cross thread dependencies not managed by locks 
currently.  These things add extra complications to current and

future code.  Also I want to understand clearly the new restrictions
be imposted on the access methods (both current and possible
future).  In the future we would like to do more automatic space
reclamation as part of the zero-admin goal, the ability to do this
in the future internal to access methods is probably affected by
the proposals here.



I think the complexities are there already. It seems very hard to know 
under exactly which conditions, a RowLocation remains valid.  We have 
assumed that the RowLocation will remain valid as long as we hold a 
table intent lock (valid means that it either points to the same row or 
to a deleted row).


That is what we concluded from the previous discusssion about validity 
of RowLocation. If you in the future need to make code, or there already 
is code, which breaks this assumption, we would need to know which other 
mechanisms we should use to either verify that the RowLocation is valid, 
or to block the system to make it invalid.


Locks can be used to manage cross thread dependencies, however they are 
bound to the transaction, and therefore does not help very much for 
cursors held across commits. So if the only mechanism we can have to 
ensure that the RowLocations are valid, is by the means of locks, I 
doubt we can support the feature of scrollable insensitive *holdable* 
updatable resultset.



It is true that the current access methods don't reuse row locations
until a table level lock is granted.  But your project would be the
first dependency on this outside of the access method implementations
themselves.  It is very clear the contract that the access methods
have with their clients while locks are held on the data they are
looking at, what you are proposing is a contract on unlocked data.



I guess we are the first to use RowLocation without holding a lock on 
the row. This is necessary, unless we simply make SUR cursors set locks 
for all rows in the cursor independent from isolation level.



Note that the current "in-place" compress will MOVE rows from one
row location to another if one does not have a row lock on the row.
This is done in the 2nd phase and only holds an intent lock, and
exclusive row locks on the rows being moved.
The off-line compress only does work under an X table lock.
So the row that you are updating actually will exist in the table,
but currently you will request the old location and will get back
a delete row indicator.  I think because of this option 1 does not
work.

Are you saing that RowLocations can be invalidated by "in-place" 
compress even if we hold a Table intent lock ?


How do you call "in-place" compress today ? Does the system use it 
automatically, or do the user have to call it manually ?



The state of held cursors across commits is very murky in the standards.
We looked very carefully at forward only held cursors, and the standards
there are carefully worded to basically not promise anything about the 
rows that were viewed that preceded the commit (clearly since the 
standard says the only thing you can do after the commit is a next to 
get a new row or close - never can access rows looked at before the

commit).  What options are legal
implementations of updatable scrollable result sets for held cursors 
across commits?  Do the standards guarantee anything about data in the

cursor looked at before the commit?



I looked at the SQL standard, and for cursors held over commit, it says:

"If the cursor is insensitive, then significant changes are not visible"

Andreas




Andreas Korneliussen wrote:


Mike Matrigali wrote:
..


If possible I would like to see a solution that does not require special
messages sent back and forth between modules about state.



I am not entirely sure what restrictions you want to put on the 
design, it is a bit unclear to me.


I have considered some other solutions:

1. Change the locking behaviour, so that a table intent lock which is 
set by an updatable cursor, is kept as long as the cursor is open - 
this will ensure that the RowLocations are valid.


2. After a commit, we could clear all data in the internal table in 
the SUR. The problem with this approach is that the resultset would 
not necessarily be repopulated with the same data - it would be 
sensitive for changes across its own transactions commits, it would be 
highly ineffecient.


3. Let the cursors notify the OnlineCompress module that it should 
fail any attempt to compress/defragment or purge the table.


More details on what I suggested yesterday:

The OnlineCompress class could provide an event mechanism, where 
subscribers (OnlineCompressListener) register themselves to listen to 
OnlineCompressEvents. The ScrollInsensitiveResultSet class could then 
implement the OnlineCompressListener interface, a

Re: conflict detection strategies

Mike Matrigali wrote:

I posted some questions about how the delete/update is done, those
answers would help me understand better what is needed for a solution.

I am going to start a separate thread concentrating on RowLocation
guarantees from store.

That is great.

Some other answers below.

Andreas Korneliussen wrote:

Mike Matrigali wrote:

There are very few cross thread dependencies not managed by locks
currently. These things add extra complications to current and
future code. Also I want to understand clearly the new restrictions
be imposted on the access methods (both current and possible
future). In the future we would like to do more automatic space
reclamation as part of the zero-admin goal, the ability to do this
in the future internal to access methods is probably affected by
the proposals here.

I think the complexities are there already. It seems very hard to know
under exactly which conditions, a RowLocation remains valid. We have
assumed that the RowLocation will remain valid as long as we hold a
table intent lock (valid means that it either points to the same row or
to a deleted row).

That is what we concluded from the previous discusssion about validity
of RowLocation. If you in the future need to make code, or there already
is code, which breaks this assumption, we would need to know which other
mechanisms we should use to either verify that the RowLocation is valid,
or to block the system to make it invalid.

There is already code that breaks this assumption, the in place compress
table. It currently is only executed as a call to a system procedure:
http://db.apache.org/derby/docs/dev/ref/rrefproceduresinplacecompress.html

But the hope was in the future to do the same kind of work that this
procedure does, internally in background.

As you have already determined that off-line compress is even more of a
problem, but it does run under an exclusive table lock. After it runs
the container you were connected to does not even exist any more.

The online compress seems to require table-exclusive locks when passing
the defragment or truncate_end argument.
There are two testcases in jdbcapi/ConcurrencyTest which test this
(testDefragmentDuringScan and testTruncateDuringScan).

The tests first deletes all rows except the first and the last. Then it
commits. Then it opens a SUR in read-uncommitted mode. Read all records
into the resultset, and position the cursor to afterlast.
Then it runs the defragment or truncate. The test will hang waiting for
a lock. Finally, it updates the rows correctly.

Output from the tests (modified the test slightly to get more output):
T1: delete records
T1: commit
T2: Read next Tuple:(0,0,17)
T2: Read next Tuple:(9,9,35)
T3: call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE
T3: DEFRAGMENT
ERROR 40XL1: A lock could not be obtained within the time requested

org.apache.derby.impl.store.access.RAMTransaction.openScan(RAMTransaction.java:1582)
at
org.apache.derby.iapi.db.OnlineCompress.setup_indexes(OnlineCompress.java:605)
at
org.apache.derby.iapi.db.OnlineCompress.defragmentRows(OnlineCompress.java:359)
at
org.apache.derby.iapi.db.OnlineCompress.compressTable(OnlineCompress.java:227)
at
org.apache.derby.catalog.SystemProcedures.SYSCS_INPLACE_COMPRESS_TABLE(SystemProcedures.java:858)

...
at
org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testCompressDuringScan(ConcurrencyTest.java:777)
at
org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testDefragmentDuringScan(ConcurrencyTest.java:712)

...

T3: got expected exception
T1: Read first Tuple:(0,0,17)
T1: updateInt(2, 3);
T1: updateRow()
T1: Read last Tuple:(9,9,35)
T1: updateInt(2, 3);
T1: updateRow()
T1: commit
T4: select * from table
T4: Read next Tuple:(0,3,17)
T4: Read next Tuple:(9,3,35)

So to me it seems that our assumptions are correct.. Only a purge is
allowed with row-level locking, online compress and online defragment
seems to be blocked by an open cursor on the table.

Maybe that is not how online compress was intended to be ?

Andreas

Locks can be used to manage cross thread dependencies, however they are
bound to the transaction, and therefore does not help very much for
cursors held across commits. So if the only mechanism we can have to
ensure that the RowLocations are valid, is by the means of locks, I
doubt we can support the feature of scrollable insensitive *holdable*
updatable resultset.

I agree, I don't believe derby is currently architected to correctly
implement "holdable" SUR. I don't think the outside event driven
approach is the right way to go.

It is true that the current access methods don't reuse row locations
until a table level lock is granted. But your project would be the
first dependency on this outside of the access method implementations
themselves. It is very clear the contract that the access methods
have with their cli

Re: [jira] Updated: (DERBY-934) create a set of JUnit tests for Scrollable Updatable Resultsets


Kathey Marsden wrote:

Myrna van Lunteren wrote:



I'm not certain it's so bad - the test expects failures.
But I logged DERBY-999. Unfortunately, I can't upload the .tmp file right
now, Jira gives me an error...I'll try that again later.




I am sorry. I did not realize that the test masters  had so many
failures checked in and read the diff backwards.
Kathey

Actually, ideally it should not be necessary to have master files to 
junit tests. The reason these specific tests now have master files, is 
because they are expecting to fail since the feature is not submitted yet.


Andreas

Re: conflict detection strategies

Mike Matrigali wrote:

It looks like I may have been thinking about future directions vs.
current reality. The question still is what should the contract
be, rather than what a specific implementation currently provides.

I agree. I think the contract could be one of these:

1. A RowLocation is valid as long as the transaction has a table intent lock
2. A RowLocation is valid as long as the transaction has a row lock for
the row.

Both contracts have different tradeoffs, I guess. for store, alt. 2
gives more flexibility in future online compress operations, however I
think it would require SURs to set and hold locks for all isolation levels.

JavaDoc for RowLocation do say:

See the conglomerate implementation specification for
information about the conditions under which a row location
remains valid.

So currently, it is implementation defined.

Andreas

Andreas Korneliussen wrote:

Mike Matrigali wrote:

I posted some questions about how the delete/update is done, those
answers would help me understand better what is needed for a solution.

I am going to start a separate thread concentrating on RowLocation
guarantees from store.

That is great.

Some other answers below.

Andreas Korneliussen wrote:

Mike Matrigali wrote:

That is what we concluded from the previous discusssion about validity
of RowLocation. If you in the future need to make code, or there
already
is code, which breaks this assumption, we would need to know which
other
mechanisms we should use to either verify that the RowLocation is
valid,

or to block the system to make it invalid.

But the hope was in the future to do the same kind of work that this
procedure does, internally in background.

The online compress seems to require table-exclusive locks when
passing the defragment or truncate_end argument.
There are two testcases in jdbcapi/ConcurrencyTest which test this
(testDefragmentDuringScan and testTruncateDuringScan).

The tests first deletes all rows except the first and the last. Then
it commits. Then it opens a SUR in read-uncommitted mode. Read all
records into the resultset, and position the cursor to afterlast.
Then it runs the defragment or truncate. The test will hang waiting
for a lock. Finally, it updates the rows correctly.

org.apache.derby.impl.store.access.RAMTransaction.openScan(RAMTransaction.java:1582)

at
org.apache.derby.iapi.db.OnlineCompress.setup_indexes(OnlineCompress.java:605)

at
org.apache.derby.iapi.db.OnlineCompress.defragmentRows(OnlineCompress.java:359)

at
org.apache.derby.iapi.db.OnlineCompress.compressTable(OnlineCompress.java:227)

at
org.apache.derby.catalog.SystemProcedures.SYSCS_INPLACE_COMPRESS_TABLE(SystemProcedures.java:858)

...
at
org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testCompressDuringScan(ConcurrencyTest.java:777)

at
org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testDefragmentDuringScan(ConcurrencyTest.java:712)

...

So to me it seems that our assumptions are correct.. Only a purge is
allowed with row-level locking, online compress and online defragment
seems to be blocked by an open cursor on the

Re: conflict detection strategies