Re: location of derbyTesting.jar (again)
Rick Hillegas wrote: Svein Erik Reknes Løvland wrote: Hi, I recently downloaded the bin release of 10.4.1.3 and ran some tests on it. I found that derbyTesting.jar was located in $DERBY_HOME/test and not in $DERBY_HOME/lib which makes me wounder if the patch I submitted to DERBY-3638 was wrong? Why the inconsistency with the location of derbyTesting.jar ? Which location is correct, or have I misunderstood something? http://issues.apache.org/jira/browse/DERBY-3638 Hi Svein Erik, The location of the jar files in the 10.4.1.3 distribution looks right to me. The lib directory is supposed to hold the product jars, that is, the jars which we expect people will typically bundle with their own applications. We don't expect that people will bundle our tests with their applications, so when we generate a release we put the tests in a separate location. I think perhaps Svein Erik is thinking about the fact that when building your own sane or insane jars, derbyTesting.jar will typically end up in jars/sane/ or jars/insane/ together with the product jars, but when downloading the release binaries it is not located with the product jars. I don't think there is a conflict here, Svein Erik. As Rick explains, the binary distribution is created that way on purpose. The location of derbyTesting.jar when building it yourself has a different reasoning I guess, as when building the code yourself it will normally be for testing purposes and not deployment. Vemund
Re: [VOTE] Jørgen Løland as a committ er
Rick Hillegas wrote: Please vote on whether we should make Jørgen Løland a Derby committer. The polls close at 5:00 pm San Francisco time on Thursday April 10. Jørgen has contributed significantly to adding replication functionality to Derby. Although he readily seeks and cheerfully incorporates community feedback, his work does not need supervision by other committers. In addition, Jørgen fields questions on the user lists and his interactions with the broader community are respectful, thoughtful, and thorough. +1 Vemund
Re: [VOTE] V. Narayanan as a committer
Rick Hillegas wrote: Please vote on whether we should make V. Narayanan a Derby committer. The polls close at 5:00 pm San Francisco time on Thursday April 10. For several years, Narayanan has contributed valuable features and fixes to Derby, starting with JDBC4 and continuing through recent work on Derby replication. Narayanan eagerly seeks the community's advice and thoroughly responds to feedback. He also fields issues on the user list--there his responses are detailed and respectful. With commit privilege he will be even more effective. +1 Vemund
Re: HTML reports now available for any ant junit test run
Daniel John Debrunner wrote: Daniel John Debrunner wrote: I made a simple change to build.xml to separate out the junit HTML report generation from junitreport which always ran junit-all. The new target junit-html can be included as the last target in any junit ant run to produce an HTML report of all the tests run in that ant invocation. E.g. ant junit-clean junit-system-mini-codeline-jars junit-html This needs a little more work, if a target fails before junit-html then junit-html is never executed and hence no report. E.g. due to these: fail if=tests.failedTests Failed!/fail Really need fail if tests.failed AND junit-html is not present (another possibile check is and not the last target). No idea if this can be done in ant though. junit-html itself contains the fail test which would then cause the build to fail correctly after generating the report. I had the same problem with the emma-targets, as mentioned in DERBY-3445. If there is no elegant way to solve this in ant, I guess it is possible to just restructure the targets so that only the top level targets, intended to be executed directly by a user, use fail if=tests.failedTests Failed!/fail These targets would mostly have no logic, just use internal targets and add the fail command. All the logic that is in the current top level junit- related targets could be moved to various internal targets that would not fail the build even if a test failed. These would be used by the top level targets and not intended for direct use. Vemund
Re: Question about collatedSuite()
Kathey Marsden wrote: I noticed in CollationTest.collatedSuite() we run DatabaseMetaDataTest, BatchUpdateTest, GroupByExpressionTest, and UpdateableResultSetTest for each of four locales. I wonder if that really adds a lot of coverage or if it would be sufficient to run them with just one of the locales. Thoughts? It adds a fair amount of time to lang._Suite. I agree that it seems a bit excessive. I think it would be enough to run all the tests you list with the default (en_US ?) and one other locale. The default, as it is the most common case and one other as it represents the equivalence class of locale != default. I think testing with more locales should be targeted at specific known problems or challenges, like the testlocaleCollation() methods do. Vemund
Re: Junit hang
Rick Hillegas wrote: I would appreciate people's advice about how to track down a hang in the Junit tests. I have added a new test case to TableFunctionTest. When I execute the test standalone, it runs fine. When I execute the surrounding suite, everything succeeds also. However, when I run the full set of junit tests, the run hangs in a later suite. I tried increasing the memory given to the testrun, but this does not fix the problem. I would appreciate: 1) advice about how to instrument the test run so that I know which test case I'm hanging in 2) random theories which may occur to you This is how I run the full suite: java -Xmx600m -DderbyTesting.oldReleasePath=/Users/rh161140/derby/upgradeReleases junit.textui.TestRunner org.apache.derbyTesting.functionTests.suites.All Hi Rick, have you tried running with -Dderby.tests.trace=true added to your commandline? Vemund
Problem with addSuiteByReflection() on phoneME advanced platform
The past few days it has not been possible to run suites.All on phoneME
advanced, junit will just exit with an InvocationTargetException when
trying to invoke the suites.All.suite() method. Unwrapping the exception
shows that the underlying reason is a NoClassDefFoundError from the
invoke() call in AllPackages.addSuiteByReflection() when trying to load
the new JMX suite.
Now, this suite is compiled into 1.5 classfiles, so the Class.forName()
call before the invoke() should have failed with
UnsupportedClassVersionError. It does fail as expected when running with
jdk 1.4, but on phoneME advanced it does not, I think it has to be a bug
in phoneME advanced.
Now, I was pondering if it would make sense to also catch
InvocationTargetException from the try block below, unwrap it and see if
it is an instance of LinkageError and if so skip the testsuite. This
would make it possible to run the tests on phoneME advanced, despite the
bug. Any reason why this would be a bad idea?
private static Test addSuiteByReflection(String className) throws Exception
{
try {
Class clz = Class.forName(className);
Method sm = clz.getMethod(suite, null);
return (Test) sm.invoke(null, null);
} catch (LinkageError e) {
return new TestSuite(SKIPPED: + className + - +
e.getMessage());
}
}
Re: NetworkServerTestSetup issues with starting server as a separate process
Daniel John Debrunner wrote:
- The server is booted without setting derby.system.home. For the vm
running the junit tests derby.system.home=${user.dir}/system. Since
d.s.h is not set the server is running with a system home of user.dir.
I think this explains the wombat database appearing in the current
directory when running junit tests. I'm not sure if this intended when
this decorator was added, the concept of the system folder was to have
all Derby databases beneath that folder. Of course if d.s.h is set to
${user.dir}/system for the spawned server then we have two jvms
accessing the same system and poentially the same databases. There are
three possible solutions to this:
a) Set d.s.h to ${user.dir}/system but shutdown Derby in the test
vm before starting the separate server.
b) Set d.s.h to ${user.dir}/ns (a different sub-folder) and ensure
tests wrapped in this decorator only use databases under d.s.h.
c) Leave it as-is, d.s.h defaults to ${user.dir} which leaves a
wombat in the current folder.
Wiki page on junit folders ...
http://wiki.apache.org/db-derby/DerbyJunitTestConfiguration?#head-af5a240fccda5c96f104776dfbd20a8cabde55e0
I think my vote goes to alternative b.
It would be good to not have any db's directly at ${user.dir}. Then you
would know it is a bug somewhere if you find one there and not just
assume it was created by a networkserver test. From the wiki page on
junit folders I also see an argument that putting db's in subfolders
allows more clear policyfiles.
I think having two vm's sharing the same subfolder has the potential of
causing problems that are hard to analyze if for some reason the Derby
in the test vm did not get shut down when a networkserver test runs, or
vice versa.
Vemund
Re: {VOTE] Kim Haase as a committer
Rick Hillegas wrote: Please vote on whether we should make Kim Haase a committer. The vote will close at 5:00 pm San Francisco time on Monday March 10. Kim has made an outstanding contribution to Derby's documentation effort. With commit privileges, she will be even more effective. +1 Vemund
Re: [jira] Commented: (DERBY-2109) System privileges
Myrna van Lunteren wrote: On 3/3/08, Daniel John Debrunner [EMAIL PROTECTED] wrote: Daniel John Debrunner wrote: Around line 401 of jvm.java there is code that sets the other properties that are used by the harness, I meant used by the policy file Dan. Yes, thx, I was thinking the same thing. I got better results when I added -Demma.active= to my -Djvmflags. My -Djvmflags now show: -Djvmflags=-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource^-Demma.active= We'll see how that goes... The idea with the emma.active property was that setting it (to ) would enable extra permissions in the policy files that were needed when measuring code coverage. If the emma.active property is not set, these permissions should be ignored by the security manager, to avoid them affecting the regular testing by accident. These permissions should only be needed if you are running with classes or jar files that have been instrumented with EMMA. If you are just running regular testing without EMMA-instrumented code, they should not be needed. When running the junit tests the emma.active property is automatically set by the SercurityManagerSetup if it finds a specific EMMA-class on the classpath, so that the junit tests can be run with classes and jars that are instrumented with EMMA. Something similar (as Dan suggests) could be done with the old harness to make it easier to measure code coverage for those tests. If the changes for DERBY-3445 affects the old harness tests, then it was unintentional and probably a bug. Vemund
Re: [jira] Commented: (DERBY-2109) System privileges
Vemund Ostgaard wrote: If the changes for DERBY-3445 affects the old harness tests, then it was unintentional and probably a bug. Ah, I hadn't read the discussion around DERBY-3483, which explained why this had an effect on weme6.1-testing. Vemund
Re: [jira] Commented: (DERBY-3445) Make it easier to use the EMMA tool to measure the code coverage of the Derby testing
Bryan Pendleton wrote: As a follow on patch it would be great to have comments explaining the new emma targets in the top-level build.xml file including a quick summary of how to use them. I've added some comments explaining how to use the new targets in a patch I uploaded today. E.g. what does one need to download (versions etc.), where does it get put. Alternatively, or perhaps in addition, this seems worthy of a page in our wiki, similar to the very useful page on how to run the junit targets. I've added a wiki page as well, here: http://wiki.apache.org/db-derby/CodeCoverageWithEMMA I hope the explanations make sense. Vemund
Optional permissions in policy files
Based on Dans suggestions to fix an issue with my patch for DERBY-3445,
I read up a bit more on policy files and discovered a feature that can
be used to make parts of the policy file optional. Since I didn't know
about this feature myself, I thought I'd share it with the community, as
I think it could be useful if you are trying to figure out how to get a
test to run with the security manager.
I found the following details on property expansion in policy files
here:
http://java.sun.com/j2se/1.4.2/docs/guide/security/PolicyFiles.html#PropertyExp
---
Also note: If a property can't be expanded in a grant entry, permission
entry, or keystore entry, that entry is ignored. For example, if the
system property foo is not defined and you have:
grant codeBase ${foo} {
permission ...;
permission ...;
};
then all the permissions in this grant entry are ignored. If you have
grant {
permission Foo ${foo};
permission Bar;
};
then only the permission Foo... entry is ignored. And finally, if you
have
keystore ${foo};
then the keystore entry is ignored.
---
So, if you have some permission you want to be used only in special
circumstances, it can be controlled for instance by using a property
that is either not set (disabled) or set to an empty string (enabled).
Adding this property to some part of the permissions you want to be
optional means that these permissions are enabled when the property is
set and ignored if the property is not set. Since the property is just
set to an empty string, the permission itself is not affected in any way
by the property.
Vemund
Daniel John Debrunner (JIRA) wrote:
[ https://issues.apache.org/jira/browse/DERBY-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12572140#action_12572140 ]
Daniel John Debrunner commented on DERBY-3445:
--
It's a concern that permissions are always granted to derby.jar, even though
they are only needed if performing code coverage with EMMA.
This creates an opportunity where Derby code could unknowingly start to depend
on those permissions.
These permissions are as follows, are they needed for all code in the stack, or
just a subset?
+// These permissions are needed when testing code instrumented with EMMA.
+permission java.util.PropertyPermission user.dir, read;
+permission java.io.FilePermission ${user.dir}${/}coverage.ec, read;
+permission java.lang.RuntimePermission writeFileDescriptor;
A work-around may be to add separate sections in the policy file, e.g. if these
permissions are only needed for a subset of jars then:
grant codebase ${emmaActive}/derby.jar {
permission java.util.PropertyPermission user.dir, read;
permission java.io.FilePermission ${user.dir}${/}coverage.ec, read;
permission java.lang.RuntimePermission writeFileDescriptor;
};
then emmaActive is only set when running code coverage with EMMA. It possible a
similar trick could be done if they are needed for all code,
something that resolves to:
grant {
permission java.util.PropertyPermission user.dir, read;
permission java.io.FilePermission ${user.dir}${/}coverage.ec, read;
permission java.lang.RuntimePermission writeFileDescriptor;
};
if and only if some emma property is set
Re: Using EMMA for codecoverage analysis
Myrna van Lunteren wrote: On 2/21/08, Vemund Ostgaard [EMAIL PROTECTED] wrote: I'm now working my way through the extra permissions needed for junit.jar and ant-junit.jar when running with instrumented jarfiles. I also think some permissions are needed when running with ant 1.7 that wasn't needed when running with ant 1.6.5. Vemund Are you referring to what's mentioned in DERBY-3153? I recently tried out the patch there and it didn't work well for me, but I didn't have time to focus on it... I hadn't seen that Jira, thank you! It looks like the same problem. My solution was a bit different, though. By setting the tempdir attribute of the junit task, I got the files written to the same directory the test is using for the rest of its files, so I only had to give permissions for writing to that directory. Not that it matters much I guess. Vemund
Re: Using EMMA for codecoverage analysis
Daniel John Debrunner wrote: Vemund Ostgaard wrote: I got suites.All to run with EMMA, but I had to add 7-8 lines to derby_tests.policy and modify a couple of tests before it ran cleanly. I think it would be good if this worked without editing the source, so I'll make a Jira and upload a patch with these changes. It might be good to separate out the test changes, I'd be interested to understand why they needed to be changed to run under code coverage. I will do that. I've only fixed suites.All, but I saw similar cases in the derbyall test as well. I think the emma instrumentation affects the tests in two ways: 1. Running instrumented code requires emma.jar in the classpath. If a test forks a new process that runs instrumented code (for instance ij) the new process also needs emma.jar in the classpath. The same is true if a test creates a new classloader that loads an instrumented class, emma.jar is needed in that classloaders search URL as well. 2. Code instrumented by EMMA will by default write some output, which some tests don't like. Using -Demma.verbosity.level=silent when starting junit solves this for most tests, but if a test forks a new process it may have to set the emma.verbosity.level for that process as well if it is touchy about the extra output. It was only two tests in suites.All that needed modifications, but I guess we might see these two issues also in the future when new tests are written. I'm thinking about adding some ant tasks for instrumenting classes/jars and running the junit-tests with EMMA, making it easier to do this. +1 I hadn't run the junit tests with ant before and had some problems running them with ant 1.6.5, so I switched to ant 1.7 and it is working better. I'm now working my way through the extra permissions needed for junit.jar and ant-junit.jar when running with instrumented jarfiles. I also think some permissions are needed when running with ant 1.7 that wasn't needed when running with ant 1.6.5. Vemund
Re: Using EMMA for codecoverage analysis
I got suites.All to run with EMMA, but I had to add 7-8 lines to derby_tests.policy and modify a couple of tests before it ran cleanly. I think it would be good if this worked without editing the source, so I'll make a Jira and upload a patch with these changes. I'm thinking about adding some ant tasks for instrumenting classes/jars and running the junit-tests with EMMA, making it easier to do this. Vemund Vemund Ostgaard wrote: Hi, I've been trying to run the Derby testsuites with EMMA to measure codecoverage, and after troubling a little I have a few questions and comments. I hit some problems both with the old test harness and the new junit testsuite in relation to the securitymanager. I haven't analyzed it fully, but both seemed to be caused by the emma codebase wanting to access files on the disk and/or system properties. I also stumbled on a test for derbyrun.jar in the old harness. It forks a java process running derbyrun.jar with the -jar option, so no matter what you set your CLASSPATH env to you still only have derbyrun.jar in the classpath. This test then failed when running with instrumented jarfiles as it couldn't find the classes it needed from emma.jar. There might be other tests with similar problems, this was just the first one I stumbled on when running the derbytools subsuite. Ideally I think the tests should be run in their normal configuration (with security manager if that is the default), also when running codecoverage measurements. It would also be good if it was possible to run all the tests. My immediate thought was that this would require modifications to policy files and possibly individual tests like the test for derbyrun.jar, but there may be better ways to get around these issues that I haven't thought about. I know that EMMA is being used by others in the community, how did you get around these problems? Is there in principle anything wrong with making changes to policy files and/or tests to make it easier to run codecoverage measurements with the EMMA tool? Vemund
Re: Junit test problem
Jazarine Jamal wrote: I made a code change and then when i run the junit test: org.apache.derbyTesting.functionTests.suites.All some 5 tests get passed and then some E occurs and then the test never gets further. No error message comes out too... This is the same state for 12 hours.. What could be the reason.? I wouldn't be surprised if it is related to OutOfMemory problems. Try increasing the heap size for the jvm. With Suns jvms its these options: -Xmssizeset initial Java heap size -Xmxsizeset maximum Java heap size Vemund
Using EMMA for codecoverage analysis
Hi, I've been trying to run the Derby testsuites with EMMA to measure codecoverage, and after troubling a little I have a few questions and comments. I hit some problems both with the old test harness and the new junit testsuite in relation to the securitymanager. I haven't analyzed it fully, but both seemed to be caused by the emma codebase wanting to access files on the disk and/or system properties. I also stumbled on a test for derbyrun.jar in the old harness. It forks a java process running derbyrun.jar with the -jar option, so no matter what you set your CLASSPATH env to you still only have derbyrun.jar in the classpath. This test then failed when running with instrumented jarfiles as it couldn't find the classes it needed from emma.jar. There might be other tests with similar problems, this was just the first one I stumbled on when running the derbytools subsuite. Ideally I think the tests should be run in their normal configuration (with security manager if that is the default), also when running codecoverage measurements. It would also be good if it was possible to run all the tests. My immediate thought was that this would require modifications to policy files and possibly individual tests like the test for derbyrun.jar, but there may be better ways to get around these issues that I haven't thought about. I know that EMMA is being used by others in the community, how did you get around these problems? Is there in principle anything wrong with making changes to policy files and/or tests to make it easier to run codecoverage measurements with the EMMA tool? Vemund
Re: [Running] Top-level junit tests - WAS Re: [jira] Commented: (DERBY-3316) Leak in client if ResultSet not closed
Daniel John Debrunner wrote: Bryan Pendleton wrote: misconception. Apparently ant junit-all runs some tests that are not run by junit suites.All, and I also think junit suites.All runs some tests that are not run by ant junit-all. I, too, thought these ran the same tests. Do you have a list of the test differences between the two runs? Also, I have been routinely using ant junitreport; which set of tests is that running? Does this help? http://wiki.apache.org/db-derby/DerbyTopLevelJunitTests That page claims junit-all runs all the tests that are included in suites.All (and more), so I think org.apache.derbyTesting.functionTests.tests.demo._Suite.suite() and org.apache.derbyTesting.unitTests.junit._Suite.suite() should be added to junit-all to keep it in synch with suites.All. Would it be a good idea for junit-all to just run suites.All directly instead of the different subsuites? This way they would always be in synch. Is the definition of top-level suite or test on this page that there is an ant target to run it in build.xml? If so is the case, maybe it makes sense to add a column for included in suites.All as well to that page. This way it might be easier to see how junit-all and suites-all fit together. Vemund
Re: [Running] Top-level junit tests - WAS Re: [jira] Commented: (DERBY-3316) Leak in client if ResultSet not closed
Daniel John Debrunner wrote: Vemund Ostgaard wrote: So, I think it would be good to work towards tests being executed in environments that are as similar as possible, and in the same order, etc. when using different mechanisms that are intended to be equivalent. On the other hand having the tests run differently has exposed bugs that might not have been seen if the different mechanisms were made as identical as possible. True. Still it would be good if we were clear on whether differences exist by design to achieve a specific purpose like finding more bugs, or if its unintentional, to avoid misunderstandings. Ideally I think the mechanisms themselves (like ant or other tools) should affect the tests as little as possible, but if we want variation to expose more bugs for instance it could achieved with different targets or suites that are clearly intended for that purpose. I guess it is mostly about documenting what is intended to be equivalent and what is different by design. Vemund
Re: [Running] Top-level junit tests - WAS Re: [jira] Commented: (DERBY-3316) Leak in client if ResultSet not closed
Daniel John Debrunner wrote: Myrna van Lunteren wrote: I want to point out that to run platform tests it's not always possible to use ant - not all OSs or machines I try to run on during a platform test have ant available. So if another solution can be found than relying on junit-all I'd appreciate it. It's not really relying on ant, the junit-all target just provides a mechanism to run a set of tests, just as suites.All provides a mechanism to run almost the same set of tests. I think a problem here is that the naming implies that both junit-all and suites.All will run all the tests, at least that has been my own misconception. Apparently ant junit-all runs some tests that are not run by junit suites.All, and I also think junit suites.All runs some tests that are not run by ant junit-all. Because of this there is no easy way to run all the tests, and it is not trivial to figure out which tests gets run when I use this target but not when I use that. Ideally it should be easy for someone that just joined the community to figure out how to run all the tests, to qualify Derby on a different OS or jvm for instance. There's no requirement for anyone to use ant junit-all or suites.All. In addition there's nothing stopping anyone from contributing another mechanism to run tests, e.g. a script file, a different grouping of tests etc. The advantage of using ant or a Junit suite is that they are standard mechanisms, and thus the tests can be run by other tools easily, such as Elipse, junit test runners or cruise control. I agree, this is a good thing. Still I think it would be valuable to make clear if two mechanisms for running the tests are intended to be equivalent, and if so try to synchronize their behavior as much as possible. For instance, I believe the ant junit-all mechanism requires significantly less memory for the jvm than junit suites.All does, as it executes the different subsuites in suites.All in different jvms. I could add a new test and run it successfully with ant junit-all, but it might break for anyone using junit suites.All because of different memory constraints. If I contribute a patch I would not want the mechanism I use to run the tests possibly hide problems that will hit others that run the same tests but with a different mechanism, to the extent possible. So, I think it would be good to work towards tests being executed in environments that are as similar as possible, and in the same order, etc. when using different mechanisms that are intended to be equivalent. Vemund
Re: Java ME and ToursDB
Vemund Ostgaard wrote: Myrna van Lunteren wrote: Why there is a difference in the number of tests that are run I can't say off the top...I can look into that next week. Is it possible for you to run with -Dderby.tests.trace=true junit.textui.TestRunner or some other mechanism that gets the list of tests, and attach them (or send them to me)? I'll do that. Myrna discovered that the upgrade tests were not run in the weme6.1-nightlies, so that explains the difference in the number of tests run. Case closed. Vemund
Re: Java ME and ToursDB
Daniel John Debrunner wrote:
I just added a test org.apache.derbyTesting.junit.EnvTest that shows
the environment that the Junit setup sees. It does this by changing
the name of the fixture, e.g.
public void testJSR169() {
setName(JDBC.vmSupportsJSR169() + _vmSupportsJSR169());
}
When I run this in the swing ui test runner I see the updated names in
the test hierarchy window.
I wonder if it would show different settings for weme6.1 and phoneME,
leading to a different number of tests being run?
No swing in CDC/FP, but my modified textbased testrunner showed:
Test ended:
true_hasTools(org.apache.derbyTesting.junit.EnvTest)duration: 0.014 (+0.014)
Test ended:
true_hasEmbedded(org.apache.derbyTesting.junit.EnvTest)duration: 0.018
(+0.001)
Test ended:
false_hasClient(org.apache.derbyTesting.junit.EnvTest)duration: 0.018 (+0)
Test ended:
true_hasServer(org.apache.derbyTesting.junit.EnvTest)duration: 0.021
(+0.002)
Test ended:
false_vmSupportsJDBC4()(org.apache.derbyTesting.junit.EnvTest)duration:
0.022 (+0)
Test ended:
false_vmSupportsJDBC3()(org.apache.derbyTesting.junit.EnvTest)duration:
0.022 (+0)
Test ended:
true_vmSupportsJSR169()(org.apache.derbyTesting.junit.EnvTest)duration:
0.023 (+0)
Vemund
Java ME and ToursDB
The new test for ToursDB fails in my phoneME advanced environment, and as far as I can understand the problem is that the demo uses PreparedStatement.setBigDecimal() and this method does not exist in jsr169. Should the demo and test be changed to work in Java ME, or should the test not be run in that environment? I looked at the latest nightly results with weme6.1 on windows: http://people.apache.org/~fuzzylogic/derby_test_results/main/derbymain/testSummary-611996.html but I am surprised to see that the test hasn't failed there. However, only 2765 tests were run, when I run suites.All on phoneME advanced I think I get around 3400 tests executed. Maybe the ToursDB test wasn't executed on weme6.1? Anyone know why there is a difference in the number of tests that are run? Vemund
Re: Java ME and ToursDB
Myrna van Lunteren wrote: Apologies, this test cannot (and thus should not be) run with JSR169. I missed the 'runwithfoundation=false' in the _app.properties file for the old test harness. I'll fix this up today. Those latest nightly results for j9 you quote (svn updated to revision 611996) don't include the new test yet, the check-in of the test conversion was done with revision 612218. No apology needed, thank you for the quick reply and explanation. Why there is a difference in the number of tests that are run I can't say off the top...I can look into that next week. Is it possible for you to run with -Dderby.tests.trace=true junit.textui.TestRunner or some other mechanism that gets the list of tests, and attach them (or send them to me)? I'll do that. Vemund
Re: Setting collation=TERRITORY_BASED when territory is set to an unsupported locale
Thank you for sharing your opinions, Knut and Mamta. I''ve reported this in Jira as DERBY-3320. Vemund Mamta Satoor wrote: I think that we should fail at both database creation time and boot time if the requested locale can not be found when collation is territory based. Mamta On 1/14/08, Knut Anders Hatlen [EMAIL PROTECTED] wrote: Vemund Ostgaard [EMAIL PROTECTED] writes: So, should database creation/boot fail if the collation is territory based and the locale is not supported? That sounds reasonable to me. If the Java runtime silently uses en_US collators when the database's territory is something different, I think we can see a variety of problems if the database is moved between systems that support collators for the database's territory and systems that don't. For instance, the index entries may appear to be unordered and queries that use indices may give strange results. I think at least database boot should fail when collation is territory based and locale isn't supported. For database creation we could alternatively detect that the situation has occurred, change the collation to UCS_BASIC before the database is created, and add a warning explaining what has happened. But I think failing is the best option for creation too. -- Knut Anders
Setting collation=TERRITORY_BASED when territory is set to an unsupported locale
A problem I've discovered when testing with the phoneME advanced platform is that the collationtests expect other locales than Locale.US to be available on the platform that is used for the test, and for phoneME advanced (when compiled as foundation profile) only Locale.US is available. From the jdk1.6 javadoc of Collator.getAvailableLocales() I see that only Locale.US is strictly required: public static Locale http://java.sun.com/javase/6/docs/api/java/util/Locale.html[] *getAvailableLocales*() Returns an array of all locales for which the |getInstance| methods of this class can return localized instances. The returned array represents the union of locales supported by the Java runtime and by installed |CollatorProvider| http://java.sun.com/javase/6/docs/api/java/text/spi/CollatorProvider.html implementations. It must contain at least a Locale instance equal to |Locale.US| http://java.sun.com/javase/6/docs/api/java/util/Locale.html#US. *Returns:* An array of locales for which localized |Collator| instances are available. My thought on this is to change the collationtests to check if for instance a Collator for polish locale is available before running the test for polish collation. This however led me to thinking about how Derby should behave if created/booted with collation=TERRITORY_BASED and territory=some unsupported locale. I'm not sure what the consequences could be if the database is first created on a platform that supports whatever locale is set and later booted with one that doesn't, or created on a platform missing support and later booted with one that has. In any case I think it may confuse a user needlessly to see the database boot successfully with his collation setting and later behave in a way he does not expect. So, should database creation/boot fail if the collation is territory based and the locale is not supported? Vemund
Re: DERBY-2636/DERBY-3068
Myrna van Lunteren wrote: Hi, I see fairly regular failures caused by a failure in cleaning up extinout/t1.dat in the nightly tests run at IBM, see for example: http://people.apache.org/~fuzzylogic/derby_test_results/main/derbymain/testSummary-609084.html I am wondering if anyone has any suggestions on how to debug or even solve these...? Since it seems to be failing only on Windows, it is likely caused by Derby having an open handle to that file, which would prevent it from being deleted om Windows. The problematic subtest is run twice (embedded and networkserver). The test fails as in DERBY-3068 if an open handle is kept after the first run of the subtest and the second run stumbles into this when it first tries to export to that same file. The test fails as in DERBY-2636 if an open handle is kept after either of the subtests when the DropDatabaseSetup tries to clean up. Since it happens when the second run of the subtest first tries to access the file again, I am guessing the problem is triggered by something in the subtest that happens after the last export statement to that file, or that statement would have failed for the same reason. After the last export using that file there is a number of import statements using that file that are expected to fail, I am guessing one or more of these can trigger the problem or it is the last export. One way to debug (as the failure seems easily reproducible) would be to just remove the import statements at the end of the subtest (testImportExportProcedureNegative) and see if that removes the problem, and then try to narrow down exactly what statement triggers the handle leak. That might get you close enough to start looking at why that leaves an open file handle when running on some jvms. Vemund
test* methods may get executed in a different order
I'm trying to get all the tests in suites.All to run successfully with the phoneME advanced platform. A problem I have discovered is that when JUnit creates a suite of tests from the test* methods in a class, I get them executed in a different order on this platform (seems to be the exact opposite order). I guess reflection behaves a little different, I don't think it's a bug. This causes some tests to fail because they are not really independent of the order that the test* methods in the class get executed, but the methods happen to be executed in an order that works when running on other JVMs. I don't think it's a huge issue, I don't know yet how many tests are failing because of this as there is still too much noise in my testresults, but I've discovered a few so far. I just wanted to mention it so that people are aware that the order the methods get executed in may be different in other environments, even though they normally get executed in the order they have in the file, so all the test* methods do have to work independent of the order. Vemund
Re: Running suites.All on the phoneME platform
Daniel John Debrunner wrote: Vemund Ostgaard wrote: Thanks for the feedback Myrna and Dan, I will definitely use the oldReleasePath property. Still, the OLD_JAR_URL is static and causes my suites.All to fail when junit is constructing the test classes even if I use that property, so I'll make a Jira to change it to http unless someone has objections. Can you provide the failure information (stack trace etc.) when the property is set and https is used. Even though OLD_JAR_URL is a static it's just a String, so it can't be causing the failure. Good point, I jumped to a conclusion. From looking at the code the URL using OLD_JAR_URL should only be created if the property is not set. So it would be a bug if the property is set but OLD_JAR_URL continues to be used to attempt to connect to the svn repo. I didn't get a stack trace, junit.textui.Testrunner only said: Failed to invoke suite():junit.framework.AssertionFailedError: java.net.MalformedURLException: unknown protocol: https Setting derbyTesting.oldReleasePath to a dummy value does get around the problem, so no bug there. Vemund
Re: Running suites.All on the phoneME platform
Daniel John Debrunner wrote: static final String OLD_JAR_URL = https://svn.apache.org/repos/asf/db/derby/jars;; I don't think this needs to be https. I modified the _Suite.java with revision 597467. If I was wrong, it can be changed back. +1 It's also much faster! To do this just check out the jars svn co http://svn.apache.org/repos/asf/db/derby/jars and then point derbyTesting.oldReleasePath to the resulting jars directory. Thanks for the feedback Myrna and Dan, I will definitely use the oldReleasePath property. Still, the OLD_JAR_URL is static and causes my suites.All to fail when junit is constructing the test classes even if I use that property, so I'll make a Jira to change it to http unless someone has objections. Vemund
Running suites.All on the phoneME platform
Hi, I'm trying to run suites.All on the phoneME advanced platform. I've compiled my own cvm for foundation profile 1.1, with the optional jsr169 jdbc support. The first hurdle I've struck is that I get a MalformedURLException saying unknown protocol: https. It seems that some SSL support is missing, and I wasn't immediately able to find out how to add it. The line in the test code that causes problems for me is: static final String OLD_JAR_URL = https://svn.apache.org/repos/asf/db/derby/jars;; in org.apache.derbyTesting.functionTests.tests.upgradeTests._Suite. When I change this URL to http I am able to start the test. Is there a reason this has to be https? I seem to be able to access that URL fine in my browser when i change it to http. Vemund
Re: Missing Jakarta jar, was: Automatic setting of compiler classpath properties at build time
Rick Hillegas wrote: One thing that's puzzling to me is why the build is trying to compile Sed.java (used by the old test harness) in the target that compiles support for our junit harness. According to the build output, the error is raised in the second javac block of junitcomponents. That block doesn't directly try to compile Sed and the classpath doesn't include the jakarta jar. Perplexed, -Rick Turns out I had a custom junit-runner lying around in the java/testing/org/apache/derbyTesting/junit/ directory that I had forgot about. This runner had a reference to org.apache.derbyTesting.functionTests.suites.All, which caused the chain of events leading to my problem. So, case closed. Thanks for putting me on the right track. Vemund
lang/infostreams.sql
While searching the old testsuite for more tests related to derby.log that I could port to junit, I came across lang/infostreams.sql. This test seems to do basically nothing, except that the database gets booted by the test framework with a number of special properties set in lang/infostreams_derby.properties that seem related to the derby error stream. The problem is that most or all of these properties don't seem to be supported, at least I have trouble finding any code in Derby that handles them. I found a list of properties in the tuning guide, but couldn't find these properties there either, can I trust that list to be complete? Does anyone know anything about this test, or the properties it is designed to test? If these properties are in fact supported I would like to expand the new junit test, ErrorStream.java, to test them. If not, then I think lang/infostreams.sql should just be removed from the old testsuite. Vemund
Re: Automatic setting of compiler classpath properties at build time
Rick Hillegas wrote: I have checked in some build logic which automatically sets the compiler classpath properties. This is part of the work on DERBY-3117 and the goal, ultimately, is to have a Derby build script which works out of the box without any customizing of ant.properties. In order to take advantage of this automatic property setting today, you need to remove some existing properties from your ant.properties (j14lib, j15lib, java14compile.classpath, java15compile.classpath) and, instead, set the following in ant.properties: autosetProps=on That is, right now you have to request the automatic setting of properties. A month from now, I would like to make automatic property setting the default. Note that even after automatic property setting becomes the default, you will still be able to override the defaults by setting the properties as you do today. Over the next month, I would appreciate it if people could try out the automatic property setting in their own environments and let me know if something breaks. I would like this to be as solid as possible before switching the default behavior. I tried it out in my sandbox Rick, and it worked fine except for one thing that probably isn't related to your changes. The first time I do ant all after having done ant clean or ant clobber, the build fails when compiling junitcomponents: [javac] Compiling 1 source file to /usr/local/cluwin/vo136787/derby/source/trunk/classes [javac] Compiling 11 source files to /usr/local/cluwin/vo136787/derby/source/trunk/classes [javac] /usr/local/cluwin/vo136787/derby/source/trunk/java/testing/org/apache/derbyTesting/functionTests/harness/Sed.java:35: package org.apache.oro.text.regex does not exist [javac] import org.apache.oro.text.regex.*; As far as I can understand, for some reason it cannot find the jakarta-oro-2.0.8.jar, but it is in my tools/java/ directory where it is supposed to be. When I do ant all again, it works, but it fails the same way every time I have done clean or clobber before. Any idea? Vemund
Re: Automatic setting of compiler classpath properties at build time
I tried it out in my sandbox Rick, and it worked fine except for one thing that probably isn't related to your changes. The first time I do ant all after having done ant clean or ant clobber, the build fails when compiling junitcomponents: [javac] Compiling 1 source file to /usr/local/cluwin/vo136787/derby/source/trunk/classes [javac] Compiling 11 source files to /usr/local/cluwin/vo136787/derby/source/trunk/classes [javac] /usr/local/cluwin/vo136787/derby/source/trunk/java/testing/org/apache/derbyTesting/functionTests/harness/Sed.java:35: package org.apache.oro.text.regex does not exist [javac] import org.apache.oro.text.regex.*; As far as I can understand, for some reason it cannot find the jakarta-oro-2.0.8.jar, but it is in my tools/java/ directory where it is supposed to be. When I do ant all again, it works, but it fails the same way every time I have done clean or clobber before. Any idea? Vemund Hm. Could you try backing up your subversion client to just before my submission (596304) and re-run the experiment? I get the same behavior with pre-596304 revisions. Also, could you attach your ant.properties? Right now it looks like this: # autosetProps=on jdk16=/usr/local/java/jdk1.6 j14lib=/usr/local/java/jdk1.4/jre/lib j13lib=/usr/local/java/jdk1.3/jre/lib proceed=false sane=false debug=false Vemund
Re: Testing of derby.log
Knut Anders Hatlen wrote: I think derby.log is overwritten several times during suites.All anyway, so that shouldn't be too much of a problem. The engine is also rebooted a number of times, see for instance DatabasePropertyTestSetup.setUp(). Even if we were able to keep a single derby.log with all the history from suites.All, it would probably be very hard to find the relevant part of it because of all the negative tests. Thank you Knut and Dyre, I'll just add it to suites.All without worrying about this then. Vemund
Re: Converting tests to junit
Knut Anders Hatlen wrote: Vemund Ostgaard [EMAIL PROTECTED] writes: I was a bit surprised to find these two tests that test functionality of the derby error stream in the derbylang suite, which I expected to be only sql-related tests. Maybe they didn't fit well in any other suites either. Is it a goal for the conversion to keep basically the same set of subsuites as in the old harness testsuite? Hi Vemund, If there are tests that you think are misplaced, they should be moved into the appropriate suite when they are converted to JUnit, IMO. If none of the existing ones seems to fit, I think you should just go ahead and create a new suite. The suites are fairly lightweight in JUnit, so if it makes the structure clearer, I think it's worth the extra effort. I was thinking of the directory/package structure under java/testing/org/apache/derbyTesting/functionTests/tests/ As far as I could understand the subdirectories there are the basis both for the old harness suites and for the subsuites under suites.All? The tests I am looking at are for the derby.log file and derby.stream.error.* properties. Looking at the Derby modules this functionality seemed closest to the engine module. Maybe a new engine suite could be the right place? Vemund
Testing of derby.log
I'm looking at converting some old tests for derby.log to JUnit. The nature of these tests mean that derby.log will be deleted by the tests, and I will have to shut down the engine to be able to test the effect of changes to properties. Does this mean that these tests should not be a part of one of the regular suites in suites.All, but rather be run in it's own jvm to avoid deleting history from other tests and rebooting the Derby engine? Any advice on how to best handle this? Vemund
Converting tests to junit
Having a wish to contribute to the test conversion effort I've been looking at the wiki to figure out where to start. I found this wiki page, http://wiki.apache.org/db-derby/KillDerbyTestHarness, that has a list of all the subsuites from the old harness testsuite and a status for the number of tests converted. The last update on this page seems to have been in May, so I'm not confident it is totally up to date with the current status. Anyhow, from that page I gathered there was a focus on the derbylang suite, which seemed a good place to start, so I looked at http://wiki.apache.org/db-derby/LangJunitTestConversion for the details. Wanting to start with something not too big to get to know the junit framework first, I found the tests logStream.java and errorStream.java, that are related to the same area and aparently noone is working on. So, I hope I can just create a jira for the conversion of these two tests, and add the jira to the notes column on http://wiki.apache.org/db-derby/LangJunitTestConversion, and that will be sufficient to avoid anyone else starting to work on the same tests? I also hope that it is sufficient to look at that page to make sure noone else is allready working on it, or should I search jira as well? Anyhow, I'm asking here to be sure. I was a bit surprised to find these two tests that test functionality of the derby error stream in the derbylang suite, which I expected to be only sql-related tests. Maybe they didn't fit well in any other suites either. Is it a goal for the conversion to keep basically the same set of subsuites as in the old harness testsuite? Vemund
Re: JUnit memory consumption
Kathey Marsden wrote: Upon more research it seems that JUnit creates one instance of each test class per fixture and keeps that reference until the end of the suites.All run. That means any state in the test hangs around as well, which slowly adds up as the test runs. This is particularly problematic for tests such as ClosedObjectTest which ends up with 3150 instances + state for each: Thanks for bringing this up Kathey, I guess we will keep sliding into this problem again and again without a more long term solution. Options to reduce footprint include: - Make sure tests have tearDown() methods that null out fields. - Rework tests such as ClosedObjectTest to not have so many fixtures. - Make a request to JUnit that the Suite dereferences the test after it runs. - Make the tearDown method in BaseJDBCTest null out fields. (I think this means that fields for tests would have to be public.) - Something else? Thoughts on this are greatly appreciated. I guess it might be a possibility to run the smaller subsuites one at a time in different junit-executions, getting a fresh jvm each time, instead of the monolithic suites.All. As far as I can understand, this is how it is currently done if you run the junit tests using ant. Actually, looking at the build.xml I see that the autoloading tests have to be executed in their own jvm for other reasons. Having some way to execute the tests in different jvms also when running from the jar-files would make it possible to run these tests as well. I would suspect that other tests in the future would also need their own fresh jvm. Finally, when we at some point are able to configure portnumber for the junit runs, it would be possible to run several of the subsuites in parallell if we ran them in different jvms, and in this way probably cut the time required to run suites.All a lot. So, maybe we just need a utility or testrunner of some sort that could execute the subtsuites in suites.All in different jvms. Vemund
Re: Running the junit testsuite with a remote server
Oops, hit the send button by accident in the middle of writing, so resending this mail. Myrna van Lunteren wrote: Starting to reason from the tests, I think part of the issue is that there are tests that test the correct starting and shutting down of the network server; or that need the server to be started with specific properties as opposed to the 'normal' mode. And as far as I know we cannot start and shutdown a server on a remote machine, and thus these tests cannot run with a remote server. So there is a group of tests that can only be run with the server on the localhost and/or only make sense to run with the server on the localhost. Probably there is also a group of tests that only make sense to run with the server on a remote machine, for instance verifying that you are not allowed to shut it down? Finally there is the group of tests that make sense to run both with a remote server and with a local server, and possibly also in embedded mode. Theoretically, all tests that do not need to test starting/shutting down, or that do not test the server with specific properties set, should, be 'runnable' with remote server that is already running. However, is that really necessary? What things could go wrong when connecting to a remote server versus a server on localhost? I would think mostly things to do with connections, permissions, import/export. So, maybe it makes sense after all to have a dedicated set of tests for remote server testing. From reading about the junit tests on the wiki and taking a look at the code, I understand that the logic for controlling which test fixtures are run with an embedded database and which fixtures are run with a network server, is in the suite() method of the various test classes. I guess this is the place to put logic that will give you a suite with the right fixtures in it and with the right decorators, depending on whether you want to run tests with a local server, a remote server, or embedded. It should be possible to run only the fixtures for the modes that you want to test, but ideally it should also be possible to run all the fixtures in all the modes that they support. So I guess what is needed in terms of functionality for the test framework is to be able to say * I do/don't want to run the embedded tests * I do/don't want to run the local server tests * I do/don't want to run the remote server tests The default behavior could be as it is today, to run the tests for all modes, provided that the framework has been instructed about the whereabouts of a remote server. If not, skip the remote server tests. Note that with remote server testing, trouble brews in the area of cleanup; I believe the current junit framework attempts to delete files that get created by the server; and I think deleting of files on a remote machine should be impossible. I wasted considerable time with the old test harness trying to identify tests that would run cleanly in sequence; only to find that a previously working set was made to fail because a uncooperative test/testcase was added. Is it necessary to delete files to be able to clean properly between test fixtures? It wouldn't be sufficient in most cases to just drop everything with SQL? It may be better to create a suite of tests, normally part of suites.All, which can/should be run against a remote server as well. I am concerned about a top level suite knowing about which tests are supposed to run in this or that mode. I think the way it is now, that this knowledge is contained in each testcase class, makes sense. For the desire to be able to run all client tests; that is more an efficiency issue useful for someone working on client only changes; when one is certain there is no need to run the tests with embedded. I think maybe a class that travels through the test suites and identifies the ones that run with DerbyNetClient would cover that requirement. The converse could also be true (run only embedded tests), but I believe this can be achieved by excluding derbyclient.jar from the classpath. Another reason to be able to run only the embedded tests or client tests would be to limit which jars/classes that are in the classpath to the ones that are strictly required in that mode and test that nothing breaks because of that. Vemund
Running the junit testsuite with a remote server
I'm interested in running the junit tests with a remote server, but as far as I can see there is no easy way to do so yet. I found two relevant jiras: https://issues.apache.org/jira/browse/DERBY-1973 (Support running JUnit tests directly with a remote server) http://issues.apache.org/jira/browse/DERBY-2638 (Create an option for junit tests to run only client tests) As these are currently unassigned I was thinking of taking a closer look myself. A couple of questions: What interface should be used to instruct the test framework that you want to use a network server (probably started manually) running on a given host portnumber, instead of the default localhost location? And along the same lines, what is the best way to instruct the test framework that you want to run only the client tests (or only the embedded tests)? For choosing to run only a subset of tests, like the client tests, I see a few possibilities: * Create some new top level suite, for instance Suites.client that runs only the client tests. * Set a property (on the commandline) when starting Suites.All (or any other subsuite) that instructs the framework to only run the client tests. * Make it possible to do both of the above. For instructing the framework to use a specific hostname and portnumber instead of the defaults I don't have any good ideas except using properties. The DERBY-1973 report says Ideally this would be through a test decorator and not setting properties on the command line., though. Any suggestions on the best way to solve this? The TestConfiguration class seems to read system properties for framework (embedded or client), hostname and port, but is this code there only to make the tests run under the old harness and intended to be removed when all the tests are converted? Is it likely that all client tests will work with a remote server, or will some tests be written in a way that requires the server(s) to run on the same host as the test framework? Perhaps some tests have to start the networkserver as part of the test, and because of this it has to run on the same host. In this case it would actually be three groups of tests: embedded, local networkserver and remote networkserver. If anyone has given all of this some thought I'd be happy to hear. Vemund
Re: [VOTE] Dyre Tjeldvoll as a committer
Rick Hillegas wrote: Please vote on whether we should make Dyre Tjeldvoll a committer. The vote will close at 5:00 pm San Francisco time on Monday September 17. +1 Vemund
Re: [VOTE] Øystein Grøvlen as a commi tter
Rick Hillegas wrote: Please vote on whether we should make Øystein Grøvlen a committer. The vote will close at 5:00 pm San Francisco time on Tuesday August 28. +1 Vemund
Re: Proposal: default property file
Jørgen Løland wrote: Ole Gunnar Borstad wrote: Siterer Knut Anders Hatlen [EMAIL PROTECTED]: We could also take the idea one step further and create a file with meta-data about the properties, for instance a description, whether it's a database property or a system property, minimum value, maximum value and default value. This could be helpful information both for management/monitoring and perhaps some time for automatically generated documentation. I like this idea, it's pretty much what Derby is lacking wrt. properties; A clean and comprehensible interface. I don't know if this has to be done the hardcore way with XML or something, or if we could just do it with commenting. I think derby.properties uses # for comments. Keeping it simple is a must I guess? Storing default property values in a default.properties file sounds like a very good idea. Personally, I think the way properties are handled in the Derby code is developer-unfriendly with the mixture of hardcoded variables and Property objects. Every now and then, I have to text-search for a property name and see what the value of the alternative is. A properties file in derby.jar would make this much easier. If possible, I think it would be good to use the same format for this file and derby.properties. Since the format of derby.properties has already been defined, I think it would be better to go with that than with xml. Just my $0.02. I have been involved in a similar refactoring for a java based test harness. We had found that properties often where the source of errors both in code and in the use of the system, because of typos and the difficulty of maintaining it over time (removing properties no longer in use, old property files used on a new version of the product, etc.). After having created a separate property file for defaults, much as you have suggested here, we also added several checks to the system: * The contents of the property file supplied by a user was checked for typos and version compatibility by requiring all properties in it to also be present in the default file. This can be a bit strict I guess, but it should at least give a warning I think if you supply a property the system doesn't know about. For us it caught and helped diagnose many problems. * When code tried to get the value of a property that wasn't in the default file, this caused an Exception. This helped avoid developers forgetting to update the default file, and caught typos in property names in the code. * A test can also verify that all properties in the default property file are actually used somewhere in the code. Vemund
Re: PROPOSAL: 10.3.2.x release in mid-September?
Stanley Bradbury wrote: As stated in my comment on the TERRITORY_BASED collation problem at: https://issues.apache.org/jira/browse/DERBY-2896#action_12513395 This problem greatly limits the use of TERRITORY_BASED collation and for the sake of that feature needs to be fixed quickly. Rather than hold up the release along with all the other features and fixes I am proposing a 10.3.2.x release be planned for 4-6 weeks following 10.3.2.1 (assuming this is approved next week). This release would correct DERBY-2896 and any other problems that might be reported and backported in the early days of 10.3. Does this sound like a good plan or would producing a 'fix' release be a lot more work/testing than stopping 10.3.1.2 and waiting for this to be fixed? I figure it best to hash this out now rather than waiting until the day of the vote. I think a 10.3.2 release that includes a number of bugfixes that do not make it for 10.3.1 would be a good thing. The motivation in the community to spend time on fixing these bugs will decide how useful it is, I guess. If a significant number of bugs are fixed, then the work involved in creating a 10.3.2 release will seem worth it. In principle I think the 10.3.1 release should be judged independently of whether there will be a bugfix release shortly after or not. I don't think a future bugfix release should influence whether a bug is a blocker or not for this release. Vemund
Re: Derby metrics
Kathey Marsden wrote: Vemund Ostgaard wrote: Did you plan to do this as a one shot exercise or did you want to do it periodically, like daily or weekly? I think weekly numbers would be good, would be often enough and might mask uninteresting fluctuations that you might see if producing daily numbers. I plan to gather some snapshot metrics now and post them. Ultimately, It would be great to have an automatted way to get weekly or on-demand numbers.I thought first we get an idea on the list of what metrics would be useful, then we make a Wiki page with what is needed, and then folks can post snapshot numbers and perhaps automate. Hope you will be interested in doing some of this work. I dug up this thread which is a year old, as I've been thinking a little more about metrics and a possible framework for storing and presenting them. What I've been thinking about is to have a (Derby) database to store the metrics in, and a (web)interface that would allow users to select which metric to view with various configuration options and then get the result presented as a graph/chart/table of any type that the framework could support. The type of configuration that should be possible at least would be to select which parameters to plot as x and y, being able to plot just a range of the data, being able to filter the data based on the parameters that are not selected as x or y, being able to sort the data ascending or descending, and probably some aggregation functionality (sum, average, etc.). The framework would have to treat data stored as strings, numbers, dates, etc. differently and give the user appropriate options when doing selections, ranges, etc. When the user has selected what he wants plotted, the interface should create an URL for the graph/presentation that can be stored and referred to later to go directly to the same presentation without having to repeat all the configuration. The datasets could be extracted in various ways from Jira, static analysis of the code, testresults, performance measurements, etc. and entered into the database by anyone with the right access. Getting the metrics into the database would be specific for Derby so no existing tools are likely to do exactly that. What I am hoping though is that it would be possible to find some open source, preferably java-based, framework that could handle the other part of the job: reading the datasets from the database (through JDBC) and creating graphs and plots based on user configuration. I've looked a little around and see that there are quite a few open frameworks that can handle the plotting part but I am still looking for something that is open source and also handles the configuration part. If anyone have any suggestions for tools/frameworks to look at, or any other feedback or thoughts on this, please share. Vemund
Re: Regression Test Report - Daily 537472 - Sun DBTG
We've had some trouble with our internal infrastructure, hence the NA results on all platforms. We're looking into the problem and hope to have it fixed ASAP. Vemund [EMAIL PROTECTED] wrote: [Auto-generated mail] *Daily* 537472/2007-05-12 18:00:13 CEST Failed TestsOK Skip Duration Suite --- *Jvm: 1.6* lin NA NA NANA derbyall NA NA NANA suitesAll sles NA NA NANA derbyall NA NA NANA suitesAll sol NA NA NANA derbyall NA NA NANA suitesAll solN+1 NA NA NANA derbyall NA NA NANA suitesAll sparc NA NA NANA derbyall NA NA NANA suitesAll vista NA NA NANA derbyall NA NA NANA suitesAll w2003 NA NA NANA derbyall NA NA NANA suitesAll Details in http://dbtg.thresher.com/derby/test/Daily/jvm1.6/testing/Limited/testSummary-537472.html Attempted failure analysis in http://dbtg.thresher.com/derby/test/Daily/jvm1.6/FailReports/537472.html *Jvm: 1.5* lin NA NA NANA derbyall NA NA NANA suitesAll sles NA NA NANA derbyall NA NA NANA suitesAll sol NA NA NANA derbyall NA NA NANA suitesAll solN+1 NA NA NANA derbyall NA NA NANA suitesAll sparc NA NA NANA derbyall NA NA NANA suitesAll vista NA NA NANA derbyall NA NA NANA suitesAll w2003 NA NA NANA derbyall NA NA NANA suitesAll Details in http://dbtg.thresher.com/derby/test/Daily/jvm1.5/testing/Limited/testSummary-537472.html Attempted failure analysis in http://dbtg.thresher.com/derby/test/Daily/jvm1.5/FailReports/537472.html *Jvm: 1.4* lin NA NA NANA derbyall NA NA NANA suitesAll sles NA NA NANA derbyall NA NA NANA suitesAll sol NA NA NANA derbyall NA NA NANA suitesAll solN+1 NA NA NANA derbyall NA NA NANA suitesAll sparc NA NA NANA derbyall NA NA NANA suitesAll vista NA NA NANA derbyall NA NA NANA suitesAll w2003 NA NA NANA derbyall NA NA NANA suitesAll Details in http://dbtg.thresher.com/derby/test/Daily/jvm1.4/testing/Limited/testSummary-537472.html Attempted failure analysis in http://dbtg.thresher.com/derby/test/Daily/jvm1.4/FailReports/537472.html --- Changes in http://dbtg.thresher.com/derby/test/Daily/UpdateInfo/537472.txt ( All results in http://dbtg.thresher.com/derby/test/ )
Re: [VOTE] 10.2.2.0 release
Rick Hillegas wrote: Please download and test-drive the 10.2.2.0 candidate, then vote on whether to accept it as a Derby release. The candidate lives at: I have tested 10.2.2.0 on HP-UX v1.11 i with HPs SDK 1.5.0.03, without finding any issues. +1 Vemund
Re: [VOTE] Kristian Waagan as a committer
Rick Hillegas wrote: Please vote on whether we should make Kristian Waagan a Derby committer. +1 Vemund
Re: JUnit disagnostice (Was Re: Some JUnit tests now split out from old harness and do not run as part of derbyall)
Daniel John Debrunner wrote: Vemund Ostgaard wrote: When running the top-level suite now, the only output I got was a lot of dots, the time it took to run the suite and the number of OK tests run. Have anyone considered a framework or interface for logging events and information in the junit tests? I think it will be very difficult in the future to analyse test failures only based on the text from an assertion failure or exception. Especially when we start porting multhreaded stress tests and other more complicated parts of the old suite. snip The Swing test runner shows more information about which tests have run and passed. Remember the textui test runner is just a simple test runner. I would encourage investigation of what others have done in this area, see if there are better test runners, rather than invent a new mechanism. I would like that the Derby JUnit tests continue to run as other Junit tests so others can run integrate them up easily. If the default running of Derby's JUnit tests produce a lot of output that will confuse others who are used to the model where a test produces no output if it succeeds. The default behaviour could be to write nothing if that is what we want, while still making it possible to get log from the tests for those who want that. With for instance the logging API that I suggest the logstatements will be in the code but wether the log actually gets printed anywhere (console or file) can be controlled dynamically with properties. Different runners can give more detailed information on the exact tests that were run, but no runners will give us insight into what each test actually did. I very much agree that the tests should run as other JUnit tests and not be confusing when others want to integrate them in junit-compatible environments. I think the type of testsuite you have and the breadth of test execution you do influences the need for logging in the tests. If you have a testuite of unit like tests that just do a single method call into the product (as JUnit was intended for), the exception thrown from a test will probably give you the information you need to be able to analyze what happened. If you have a testsuite of many complex/long/multithreaded tests run within a variety of configurations and decorators, errors can be difficult to analyze and you will more often have to put som debug in the test and rerun to get what you need. Tests are run nightly on several platforms and jvms. When the junit suite grows to a size comparable with the old testsuite and beyond, there will be problems to analyze every night. If these problems are accompanied with a sensible amount of log it will ease the work of the analyzer, who often will not be the original author of the test. The tests will not be perfect, there will be situations that should lead to an exception being thrown that doesn't, but instead create difficulties for following tests. These issues will also be very costly to track down without a better view into what each test actually did. A last issue is that when you make changes to a test, it is good to get some verification that the changes actually had the desired effect. Being able to take a look at the log from the nightly run of tests and verify that the output from your fix got printed is a nice way of doing just that. We all develop on different systems so even when it worked on yours it might not have the same effect on the one its failing on in the nightlies. You may not even be able to reproduce the problem on the machines you have access to and depend on getting more output when it happens, perhaps intermittently) in others testing. If you think about the old harness, how often have you made use of information from the testlogs that you wouldn't have gotten from just an exception thrown? Vemund
Re: Some JUnit tests now split out from old harness and do not run as part of derbyall
Daniel John Debrunner wrote: I would encourage those that regularly run nightly or other testing to add this JUnit test into their framework. I'm sure there will be some issues getting it running on all the platforms but they should be easy to work through. I'm extending the nightly testing with this top-level junit suite, executed with junit.textui.TestRunner, from tonight. If it turns out to run ok on all platforms we will start reporting from this suite as well in the mails from Henri. There is more work to be done for some of the tests in All so that they also run in the client server by themselves, and I'm working on it for the tests in tests.jdbcapi. I'm also planning on laying out some form of a plan or work items that would move us completely over to JUnit. When running the top-level suite now, the only output I got was a lot of dots, the time it took to run the suite and the number of OK tests run. Have anyone considered a framework or interface for logging events and information in the junit tests? I think it will be very difficult in the future to analyse test failures only based on the text from an assertion failure or exception. Especially when we start porting multhreaded stress tests and other more complicated parts of the old suite. It would also give a warm fussy feeling to be able to see that the tests actually did something (in addition to all the dots written by the TestRunner). I've been used to using the java logging API (java.util.logging.*) for similar needs, and think that would cover what is needed for this. Other suggestions or views on this? Vemund
Re: [VOTE] Fernanda Pizzorno as a Derby committer
Daniel John Debrunner wrote: Please vote +1 if you agree Fernanda should be a committer. +1 Vemund
Re: [VOTE] 10.2.1.6 release
I have run derbyall on HP-UX 11.11, and did not see any problems. +1 Vemund
Re: [junit] different configurations at test level Was - Re: [junit] Top-level suites added
Daniel John Debrunner wrote:
Vemund Ostgaard wrote:
This would mean that when executing a single TestCase class (or
suite) with a junit runner, the default behaviour could be to run
with all (supported) frameworks if that is what is most common.
Running with just a selected framework could be possible with a
system property.
snip
B) Have a test's suite method just set up a suite as it sees fit:
public static Test suite() {
TestSuite suite = new TestSuite();
// Embedded tests
suite.add(TimeHandlingTest.class);
suite.add(embeddedTestA);
suite.add(embeddedTestB);
suite.add(embeddedTestC);
// Client tests
TestSuite client = new TestSuite();
client.add(TimeHandlingTest.class);
client.add(clientTestA);
suite.add(new NetworkServerTestSetup(client));
return suite;
}
Downsides are:
possible lack of consistency, but maybe the consistency is use of
the NetworkServerTestSetup decorator.
how to handle remote server testing?
The remote server is a special case of the client framework here. You
want the client tests to be run, but you don't want the
NetworkServerTestSetup to actually start a network server (unless
someone solves the problem of doing this remotely on all platforms). So
the NetworkServerTestSetup needs to handle this by just doing nothing if
the server host is set to a remote machine. The tester will have to
start the server manually before starting the test.
Upside is:
clarity
matches standard JUnit pattern
non-reliance of having the configuration setup at suite setup time.
snip
-
Are there any other ideas? Having written them out, my preference
would be B), seems very clear and straight forward. Before I wrote it
out I was thinking that none of the solutions were good.
When I made the suggestion you quoted above I was thinking along the
lines of B, without thinking in detail or having studied the code. I
think that is the right place to solve the problem, and your code
suggestion is how I see the default behaviour. In addition it should be
possible to override with properties, so that the suite does not add the
embedded tests if the user set a property that he did not want
embedded tests run, only client. For instance, something like:
Testsuite suite = new TestSuite();
if(runEmbedded) {
Testsuite embedded = new TestSuite();
suite.add(embedded);
}
if(runClient) {
Testsuite client = new TestSuite();
suite.add(new NetworkServerTestSetup(client));
}
// etc.
return suite;
}
Or something along those lines?
Thanks,
Vemund
Re: possible JDBC 4 EOD bug??
Daniel John Debrunner wrote: My understanding was that this would be fixed by 6460994, which went into b97. Is there a link for that bug? It's not found at: http://bugs.sun.com/bugdatabase/index.jsp I'm not sure why its not visible there. Anyway, it seems that bug is not relevant to the issue you are seeing anyway, a misunderstanding on my part. I'm making a new bug report that I believe covers the issue you have hit. Vemund
Re: possible JDBC 4 EOD bug??
Daniel John Debrunner wrote:
I'm trying to understand the expected behaviour of the assignment of
columns from an EOD SQL @Select to a data class using JavaBeans style
property accessors. I'm looking at section 19.1.5 User Class of JDBC 4.0.
I have a class similar to the example but a little different. The
difference is the private field name does not match the name of the
JavaBean property, is that allowed?
public class Person
{
private String myName;
public void setName(String name) {
this.myName = name;
}
public String getName() {
return myName;
}
}
If my select returns a column called 'NAME' then it does not map to the
JavaBean property called 'name'. Instead the name of the column needs to
map to the name of the private field, 'myName'. Then the field is set
correctly but the setter is never used. Is this a bug, it seems like it?
If I understand you correctly, I believe it is a known bug that I
stumbled on myself a while ago. I thought it had been fixed in one of
the latest Mustang builds but I haven't verified it myself. Which build
have you been using?
Vemund
Re: possible JDBC 4 EOD bug??
Daniel John Debrunner wrote: Vemund Ostgaard wrote: Daniel John Debrunner wrote: If my select returns a column called 'NAME' then it does not map to the JavaBean property called 'name'. Instead the name of the column needs to map to the name of the private field, 'myName'. Then the field is set correctly but the setter is never used. Is this a bug, it seems like it? If I understand you correctly, I believe it is a known bug that I stumbled on myself a while ago. I thought it had been fixed in one of the latest Mustang builds but I haven't verified it myself. Which build have you been using? Thanks, the description was a bit rushed because I had ony a little time before I needed to leave and I wanted to get the question out there. Do you have the bug number for the issue? My understanding was that this would be fixed by 6460994, which went into b97. I'm using b98. That should indicate that it didn't get fixed as part of that bug report. Vemund
Re: Running a Derby JUnit test using JUnit directly
Daniel John Debrunner wrote: Vemund Ostgaard wrote: For the jobs doing nightly testing on the 10.2 branch we did a short term workaround by copying derbyTesting.jar to its old location, to avoid other changes. So, thats why it hasn't failed in those tests. It's good to report such problems, not just work-around them. That way others get to know and the issue can be fixed quicker. Or maybe I missed the e-mail discussing this? Yes, I agree. The problem we had though was with a tool that we use internally to distribute files to remote machines for testing, and as such didn't seem interesting to the community. We didn't check if the problem hid other problems that were relevant for the community, which evidently was the case. Vemund
Re: Running a Derby JUnit test using JUnit directly
Daniel John Debrunner wrote: Michelle Caisse wrote: I was using 10.2.1.1. I see the same failure as you. It's because the policy file is not setup to handle derbyTesting.jar being in a different folder to derby.jar etc. It's actually nothing to do with JUnit, you will see problems running tests with the old harness. I'm very suprised no-one has seen this before, since I thought there have been several reports of derbyall passing against 10.2.1.1. For the jobs doing nightly testing on the 10.2 branch we did a short term workaround by copying derbyTesting.jar to its old location, to avoid other changes. So, thats why it hasn't failed in those tests. Vemund
Re: [junit] Top-level suites added
Daniel John Debrunner wrote: The current approach has very little magic, one decorator for client and probably one decorator for DB2 client. I strongly agree that tests should not be overly influenced by the suite setup. I think this is a sound principle to keep in mind as work progresses on this new testsuite. It might not be possible to avoid suite influence totally, but hopefully it is kept at a level where it is not regarded as a problem with the testsuite. This is an interesting idea, some thoughts jump out at me: - A single test would potentially run in many configurations: embedded, client, db2 client, encryption DES, encryption AES, ... If the number of configurations grows large, it might pay off to carefully review wether something should be a configuration of its own or rather be tested through a set of more specialized tests. When the master files are gone and we have a new assertion-based testsuite, the tests should only fail on things they are specifically looking for, possibly making them less suited for testing of a large number of configurations? I guess it will be a tradeoff, having many configurations or more tests. - The large number of decorators required per test could factor into large memory use for an entire suite, not sure of the overhead of each TestSetup. - Embedded tests would be running with the network server booted by default, not sure of the impact of this, starting the network server for each individual TestCase would slow an entire suite down too much Are you thinking of the impact of the network server running in the same vm as junit? - It does make running a single TestCase in all configurations easy - The potential requirement to modify all tests when a new configuration is added into the run, e.g. if I add a configuration that only works in 10% of the tests (that's my itch) then I have to modify the other 90% of tests to disable it from running. Assuming the default as you say is to run each test in all possible configurations, I assume this would be enforced in the base test class. Maybe that's the case today with the current approach where a test is ideally self describing. I guess that the default set of configurations that a TestCase supports could be something smaller than the set of all possible configurations. The default runtime configuration would be to run all tests in all possible configurations, but a TestCase with default settings would override this with its smaller (default) set. You would then have to fix your 10% tests by adding the new configuration to their supported set, instead of removing it from the other 90%. Still, that is work, but I would hope we don't end up with a jungle of configurations that anyhow would be a pain to maintain. I do believe, though, that it makes sense that a TestCase itself maintains the information about what configurations/frameworks it supports or requires, and not suite classes located somewhere else. - Maybe there are some configurations that are best handled at the leaf level, e.g. embedded and client, but others are best handled at a suite level, e.g. db2 client, encryption. So a mix of the two approaches would be good. Yes, that may very well be. As I've mentioned above I also think it should be considered if all configurations are really needed or wether it can be tested as well or better with new tests instead. I'm going to continue my current setup approach because it's not opposed to what you approach, the majority of the work will be getting tests converted to JUnit and running in client db2 client mode. Ie. most of the work overlaps anyway. Yes, that is true. The work you are doing now is making it a lot easier to imagine how things may fit together with a testsuite written and run only with junit, and to have opinions on that. Vemund
Re: [junit] Top-level suites added
Daniel John Debrunner wrote:
I added some top-level suites that are intended to be run directly by
JUnit runners, not via the test harness. They are in the package:
org.apache.derbyTesting.functionTests.suites
AllPackages - A suite of all the _Suite suites for the function test
packages
Embedded - All the tests for embedded. Currently just includes
AllPackages.suite(), but future may add more in different
configurations, e.g. encryption.
Client - All the tests for client, runs the AllPackages.suite() with a
decorator that changes the configuration and boots the network server
(using the existing network server decorator)
All - All the tests, currently includes Client.suite() and
Embedded.suite(). (Eventual replacement for derbyall)
It's not all working fine yet, a couple of issues:
1) The network server decorator in jdbcapi._Suite clashes with the one
in Client
I think that adding the network server decorater to a test should just
be a way of saying that this test needs a server running, not knowing
wether it might allready be running. If it is not running, start it. If
it allready runs, leave it be.
2) Some tests fail in the Client suite, need to investigate.
Additional tests were failing due to some test using the old checks to
see if it was running embedded or not, rather than the utilities
provided in the JUnit classes.
I've checked these in for others to see the direction I'm proposing,
with the goal being all tests running under JUnit with no harness.
Example of running these:
java -Dderby.system.home=${PWD} junit.textui.TestRunner
org.apache.derbyTesting.functionTests.suites.Embedded
java -Dderby.system.home=${PWD} junit.textui.TestRunner
org.apache.derbyTesting.functionTests.suites.Client
java -Dderby.system.home=${PWD} junit.textui.TestRunner
org.apache.derbyTesting.functionTests.suites.All
Comments?
I have a suggestion, based on the following view:
* I would avoid creating suites just to handle the various frameworks,
or other configuration related things. This would mean a simpler
structure of suites, easier to understand and maintain. If we want to
have a MATS suite for instance, we would not need an embedded_MATS
suite and a client_MATS suite and so forth to build the structure.
* It would be good if as little as possible (ideally no) magic was
performed on a suite level. Tests should behave the same way when
executed through the lowest level leaf suite as when executed as part
of derbyall, making it easier to reproduce problems.
How could this be accomplished?
Handle everything related to frameworks and configuration on a leaf
level, I believe this could be done in the suite() method in each
TestCase. Just add each Test once for each framework that this run is
configured to use, using the correct decorator. If a test is not
supposed to be run for a given framework, that information should be
maintained in that TestCase in some way, so that test is not added for
that framework. Default would be to run all Tests in all frameworks.
This would mean that when executing a single TestCase class (or suite)
with a junit runner, the default behaviour could be to run with all
(supported) frameworks if that is what is most common. Running with just
a selected framework could be possible with a system property.
Vemund
Re: test harness and jvmoptions
Vemund Ostgaard wrote: Myrna van Lunteren wrote: :-( The jvmflag passing is broken (not sure how well it ever worked). See also DERBY-1091. I intend to work on that this week...But it's a big can of worms. But try without the quotes... I looked a little at this code, and RunTest, RunList etc. Seems to me that RunList places double quotes around the jvmflags, wether you have done so or not, before passing it on to RunTest, where it is eventually executed with Runtime.exec. If Runtime.exec had executed a command shell (which again executed java) I guess the quotes would be removed, but since it executes java directly they are not. I tried removing these added quotes (RunList line 385) and the -Djvmflags option works for me then, when running with Suns jvm 1.5 and using a single flag -d64. I guess the quotes work with the IBM java launcher, or it doesn't make sense that they were ever added. DERBY-1091 raises the issue that the passing of flags doesn't work with more than one flag, but for me it doesn't work with one flag either. Did you get around to looking closer at this Myrna? I would like to remove those quotes, but I guess they are there for a reason... Thanks, Vemund
Re: test harness and jvmoptions
Myrna van Lunteren wrote: On 5/31/06, Vemund Ostgaard [EMAIL PROTECTED] wrote: Hello, I am trying to get the test harness to specify a flag to the jvm when new jvms are started to execute tests/testsuites, but I have got it wrong somehow. I want to supply either -d32 or -d64 option to the Sun jvm (on linux or solaris) to force tests to run in either 32-bit or 64-bit environment. Based on reading the testing readme, I concluded that setting the 'jvmflags' systemproperty was the correct way to do this. What I have tried is: java -d32 -Djvmflags=-d32 org.apache.derbyTesting.functionTests.harness.RunSuite derbytools The result is that all tests fail with: Exception in thread main java.lang.NoClassDefFoundError: -d32 Anyone got any advice on how to correctly get the test harness to use a specific flag/argument when starting new jvms? Thanks, Vemund :-( The jvmflag passing is broken (not sure how well it ever worked). See also DERBY-1091. I intend to work on that this week...But it's a big can of worms. But try without the quotes... With or without quotes (both single and double) seems to behave the same way for me. I see from the harness log that the tests are executed in the same way in any case, with double quotes around the jvmflags value -d32: Top suite: derbytools Suite to run: derbytools:derbytools Now do RunList Now run the suite's tests Run the tests... Execute command: java -DjavaCmd=java -Djvmflags=-d32 -Doutputdir=/export/home/tmp/jagtmp/vo136787derbytools/derbytools_0/log/derbytools/derbytools -Dtopsuitedir=/export/home/tmp/jagtmp/vo136787derbytools/derbytools_0/log/derbytools -Dtopreportdir=/export/home/tmp/jagtmp/vo136787derbytools/derbytools_0/log/derbytools -Drundir=/export/home/tmp/jagtmp/vo136787derbytools/derbytools_0/log -DtestSpecialProps=hostName=localhost -Dsuitename=derbytools:derbytools -Dtopsuitename=derbytools org.apache.derbyTesting.functionTests.harness.RunTest tools/dblook_test.java The description of suns java launcher says: By default, the first non-option argument is the name of the class to be invoked. Since I get a NoClassDefFoundError, it seems as if my -d32 option is considered as the java class to invoke and not as an option, possibly because of the quotes around it? I don't really understand why the harness passes the -d32 option to the jvm with the double quotes around it. Otherwise, if your need is temporary and urgent and you can make (temporary) changes to the source you can see if you can add the flag in the jvm subclass for the jvm you're using (e.g. org/apache/derbyTesting/functionTests/harness/jdk16.java) I looked a little at this code, and RunTest, RunList etc. Seems to me that RunList places double quotes around the jvmflags, wether you have done so or not, before passing it on to RunTest, where it is eventually executed with Runtime.exec. If Runtime.exec had executed a command shell (which again executed java) I guess the quotes would be removed, but since it executes java directly they are not. I thought maybe the reason this didn't work for me was because I used a non-standard option only supported by some jvms/versions, and that other standard options where handled specifically by the harness, but trying this with a regular -D option gave me the same behavior. Vemund
Developer access to Jira
I would like to be granted developer access to Jira. It seems I need this to be able to close a bug I reported. Thanks, Vemund
Re: Derby metrics
Kathey Marsden [W.P.A YELLOW PAGES] wrote: I was thinking about Derby Metrics. I think this is a good initiative Kathey, having metrics like these in place would be useful when discussing releases and quality. 1) Has anyone compiled any of the following information already? 2) Can you think of additonal metrics that might be useful to the community? The size of jar-files might be useful to measure/monitor (when compiled with the right flags and options that are used for releases). The intent would be to have awareness on the small, light-weight image. In addition to measuring lines of code added or total, I've also heard about measuring the number of lines of code that have been changed (added/removed/updated), loosely refered to as measuring code turmoil. With a lot of code turmoil you expect less stability, this number should go down when you get close to a release. Derby lines of code by component. Derby code coverage numbers by component. Lines of code added for 10.2 by component. Derby bugs by component and severity. Derby bug inflow by component and overall. Did you plan to do this as a one shot exercise or did you want to do it periodically, like daily or weekly? I think weekly numbers would be good, would be often enough and might mask uninteresting fluctuations that you might see if producing daily numbers. Vemund
Re: [jira] Commented: (DERBY-1229) sysinfo and sysinfo_withproperties.java fail with java.lang.RuntimePermission getProtectionDomain when db2jcc.jar is in same dir as the derby-jars
Andrew McIntyre (JIRA) wrote: [ http://issues.apache.org/jira/browse/DERBY-1229?page=comments#action_12377975 ] Andrew McIntyre commented on DERBY-1229: Hi Bryan, if you have some time, please take a look at the patch I attached to DERBY-1273, the derby-1273-v2.diff patch. With that patch I attempted to get around the need to have the getProtectionDomain permission at all. I still think that your patch for this issue, which improves the output of the security exceptions on items in the classpath to which access has been restricted, is valuable and should be committed. But my patch attached to DERBY-1273 handles the getProtectionDomain issue in a different way. I was prompted to investigate removing the need for the getProtectionDomain permission due to Kathey's concerns. I'm not sure I've tested all of the many scenarios of classloader/environment/securitymanager/sysinfo-argument execution paths that are possible, and I'm hoping that with a couple more eyes on it we can ensure that we get the maximum useful information out of sysinfo with the fewest possible permissions granted to it. One thing that confused me with the original problem was that I got an exception for db2jcc.jar but not for db2jcc_license_c.jar. When I looked at the code I discovered that they were treated differently. If you are replacing the way getProtectionDomain().getCodeSource() is used to find where db2jcc.jar is, it might be an idea to use the same procedure to find the location of all these jars to get consistent results. Vemund sysinfo and sysinfo_withproperties.java fail with java.lang.RuntimePermission getProtectionDomain when db2jcc.jar is in same dir as the derby-jars -- Key: DERBY-1229 URL: http://issues.apache.org/jira/browse/DERBY-1229 Project: Derby Type: Test Components: Security, Test, Tools Versions: 10.2.0.0 Environment: Solaris 10 x86 Sun JDK java version 1.5.0_06 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode, sharing) Reporter: Vemund Østgaard Assignee: Bryan Pendleton Priority: Minor Attachments: derby-1229.diff, notes.html The problem I've been seeing is that if I run: java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest derbynet/sysinfo.java in a configuration where my db2jcc.jar is located in the same directory as all the other derby-jars, then the test fails with a diff caused by a RuntimeException. If I change nothing else then moving that db2jcc.jar out of that directory (and making the appropriate classpath change) then the test succeeds. The reason is that the test fails to get the permission for doing getProtectionDomain().getCodeSource() on class com.ibm.db2.jcc.DB2Driver. Below follows relevant output from my system that might be useful to debug this: --- [EMAIL PROTECTED]:~/tmp/test env |grep CLAS CLASSPATH=/home/vo136787/derby/tmp/testbuild/lib/derby.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyTesting.jar:/home/vo136787/derby/tmp/testbuild/lib/derbytools.jar:/home/vo136787/derby/tmp/testbuild/lib/derbynet.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyclient.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyrun.jar:/home/vo136787/derby/tmp/testbuild/lib/junit.jar:/home/vo136787/derby/tmp/testbuild/lib/db2jcc.jar:/home/vo136787/derby/tmp/testbuild/lib/db2jcc_license_c.jar:/home/vo136787/derby/tmp/testbuild/lib/jakarta-oro-2.0.8.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_de_DE.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_es.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_fr.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_it.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_ja_JP.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_ko_KR.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_pt_BR.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_zh_CN.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_zh_TW.jar [EMAIL PROTECTED]:~/tmp/test java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest derbynet/sysinfo.java *** Start: sysinfo jdk1.5.0_06 DerbyNetClient 2006-04-19 13:00:33 *** Initialize for framework: DerbyNetClient java -Dderby.system.home=/home/vo136787/tmp/test/DerbyNetClient/sysinfo -Djava.security.manager -Djava.security.policy=/home/vo136787/tmp/test/derby_tests.policy -DderbyTesting.codejar=file:/home/vo136787/derby/tmp/testbuild/lib/ -DderbyTesting.codedir=/home/vo136787/derby/tmp/testbuild/lib -DderbyTesting.serverhost=localhost -DderbyTesting.clienthost=localhost -DderbyTesting.codeclasses=file://unused/ org.apache.derby.drda.NetworkServerControl start -- SecurityManager not installed -- Attempt to
Re: [jira] Commented: (DERBY-1229) sysinfo and sysinfo_withproperties.java fail with java.lang.RuntimePermission getProtectionDomain when db2jcc.jar is in same dir as the derby-jars
Bryan Pendleton (JIRA) wrote: [ http://issues.apache.org/jira/browse/DERBY-1229?page=comments#action_12375657 ] Bryan Pendleton commented on DERBY-1229: Can you confirm that if you add the following line to derby_tests.policy, the problem goes away? The line should be added to all three of the derby.jar, derbynet.jar, and derbytools.jar sections, which is easiest to do if you just search for DERBY-622 and then add this line right after it: permission java.lang.RuntimePermission getProtectionDomain; Yes, this seems to solve the problem in my environment. In some ways, what's confusing about this bug is not that you have to add this line, it's that nobody else has had to add this line. From what I can tell by reading the Sun javadocs, everybody should have needed the getProtectionDomain permission, and certainly when I wrote a simple standalone test program, the program failed unless I gave it the getProtectionDomain permission. So I don't understand why the rest of us have been getting away without having this permission in place. I don't have any good answer to this. I know that when I tested two builds (one with debug and one without) of the same Derby version here I only saw the problem with one of the builds. Possibly it will only happen if you have built Derby with a specific setting, or lacking a specific setting? If you could please try adding the permission to the derby_tests.policy file, and verify that the sysinfo and sysinfo_withproperties tests then pass, that would be nice. Thanks! Yes, both tests pass now. Thank you for looking into this, Bryan. :) Vemund sysinfo and sysinfo_withproperties.java fail with java.lang.RuntimePermission getProtectionDomain when db2jcc.jar is in same dir as the derby-jars -- Key: DERBY-1229 URL: http://issues.apache.org/jira/browse/DERBY-1229 Project: Derby Type: Test Components: Security, Test, Tools Versions: 10.2.0.0 Environment: Solaris 10 x86 Sun JDK java version 1.5.0_06 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode, sharing) Reporter: Vemund Østgaard Priority: Minor The problem I've been seeing is that if I run: java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest derbynet/sysinfo.java in a configuration where my db2jcc.jar is located in the same directory as all the other derby-jars, then the test fails with a diff caused by a RuntimeException. If I change nothing else then moving that db2jcc.jar out of that directory (and making the appropriate classpath change) then the test succeeds. The reason is that the test fails to get the permission for doing getProtectionDomain().getCodeSource() on class com.ibm.db2.jcc.DB2Driver. Below follows relevant output from my system that might be useful to debug this: --- [EMAIL PROTECTED]:~/tmp/test env |grep CLAS CLASSPATH=/home/vo136787/derby/tmp/testbuild/lib/derby.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyTesting.jar:/home/vo136787/derby/tmp/testbuild/lib/derbytools.jar:/home/vo136787/derby/tmp/testbuild/lib/derbynet.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyclient.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyrun.jar:/home/vo136787/derby/tmp/testbuild/lib/junit.jar:/home/vo136787/derby/tmp/testbuild/lib/db2jcc.jar:/home/vo136787/derby/tmp/testbuild/lib/db2jcc_license_c.jar:/home/vo136787/derby/tmp/testbuild/lib/jakarta-oro-2.0.8.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_de_DE.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_es.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_fr.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_it.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_ja_JP.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_ko_KR.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_pt_BR.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_zh_CN.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_zh_TW.jar [EMAIL PROTECTED]:~/tmp/test java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest derbynet/sysinfo.java *** Start: sysinfo jdk1.5.0_06 DerbyNetClient 2006-04-19 13:00:33 *** Initialize for framework: DerbyNetClient java -Dderby.system.home=/home/vo136787/tmp/test/DerbyNetClient/sysinfo -Djava.security.manager -Djava.security.policy=/home/vo136787/tmp/test/derby_tests.policy -DderbyTesting.codejar=file:/home/vo136787/derby/tmp/testbuild/lib/ -DderbyTesting.codedir=/home/vo136787/derby/tmp/testbuild/lib -DderbyTesting.serverhost=localhost -DderbyTesting.clienthost=localhost -DderbyTesting.codeclasses=file://unused/ org.apache.derby.drda.NetworkServerControl start -- SecurityManager not
Re: java.lang.RuntimePermission getProtectionDomain in derbynet/sysinfo.java
Myrna van Lunteren wrote: Hi, I didn't log one, I assumed you were going to, as it appeared you've done more debugging/experimenting... Submitted a new jira for this: DERBY-1229. Vemund
Re: java.lang.RuntimePermission getProtectionDomain in derbynet/sysinfo.java
Bryan Pendleton wrote: Vemund Ostgaard wrote: After quite a bit of reading and testing it seems to me that the tests fail when db2jcc.jar is located in the same directory as the derby*.jar Thanks for taking an interest in this Brian, I've been away on vacation but now I've had a look at this again. I have not succeeded in reproducing this, unfortunately. Here's what I did: 1) copied db2jcc*.jar to trunk/jars/sane, where my other derby jars are 2) Set my classpath to point to the jars in trunk/jars/sane, including db2jcc.jar and db2jcc_license_c.jar 3) java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest derbynet/sysinfo.java And for me, the test still passed. Can you: a) check my steps to see how they might be different from yours? Well, I've not built the jars myself, I just got them from our nightly builds. Basically that means I just have a directory with all the jars in, and no class files or any other files from the derby repository. b) post your exact classpath that you're using c) using that classpath, run - java org.apache.derby.tools.sysinfo - java org.apache.derby.tools.sysinfo -cp and post the results that you get from those two runs? Here follows the relevant output: --- [EMAIL PROTECTED]:~/tmp/test env |grep CLAS CLASSPATH=/home/vo136787/derby/tmp/testbuild/lib/derby.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyTesting.jar:/home/vo136787/derby/tmp/testbuild/lib/derbytools.jar:/home/vo136787/derby/tmp/testbuild/lib/derbynet.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyclient.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyrun.jar:/home/vo136787/derby/tmp/testbuild/lib/junit.jar:/home/vo136787/derby/tmp/testbuild/lib/db2jcc.jar:/home/vo136787/derby/tmp/testbuild/lib/db2jcc_license_c.jar:/home/vo136787/derby/tmp/testbuild/lib/jakarta-oro-2.0.8.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_de_DE.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_es.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_fr.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_it.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_ja_JP.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_ko_KR.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_pt_BR.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_zh_CN.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_zh_TW.jar [EMAIL PROTECTED]:~/tmp/test java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest derbynet/sysinfo.java *** Start: sysinfo jdk1.5.0_06 DerbyNetClient 2006-04-19 13:00:33 *** Initialize for framework: DerbyNetClient java -Dderby.system.home=/home/vo136787/tmp/test/DerbyNetClient/sysinfo -Djava.security.manager -Djava.security.policy=/home/vo136787/tmp/test/derby_tests.policy -DderbyTesting.codejar=file:/home/vo136787/derby/tmp/testbuild/lib/ -DderbyTesting.codedir=/home/vo136787/derby/tmp/testbuild/lib -DderbyTesting.serverhost=localhost -DderbyTesting.clienthost=localhost -DderbyTesting.codeclasses=file://unused/ org.apache.derby.drda.NetworkServerControl start -- SecurityManager not installed -- Attempt to shutdown framework: DerbyNetClient 14a15 [Unable to access Protection Domain or Code Source for class class com.ibm.db2.jcc.DB2Driver: access denied (java.lang.RuntimePermission getProtectionDomain)] 2.4 - (17) 41a43 [Unable to access Protection Domain or Code Source for class class com.ibm.db2.jcc.DB2Driver: access denied (java.lang.RuntimePermission getProtectionDomain)] 2.4 - (17) 69a72 [Unable to access Protection Domain or Code Source for class class com.ibm.db2.jcc.DB2Driver: access denied (java.lang.RuntimePermission getProtectionDomain)] 2.4 - (17) Test Failed. *** End: sysinfo jdk1.5.0_06 DerbyNetClient 2006-04-19 13:00:40 *** [EMAIL PROTECTED]:~/tmp/test java org.apache.derby.tools.sysinfo -- Java Information -- Java Version:1.5.0_06 Java Vendor: Sun Microsystems Inc. Java home: /usr/jdk/instances/jdk1.5.0/jre Java classpath: /home/vo136787/derby/tmp/testbuild/lib/derby.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyTesting.jar:/home/vo136787/derby/tmp/testbuild/lib/derbytools.jar:/home/vo136787/derby/tmp/testbuild/lib/derbynet.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyclient.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyrun.jar:/home/vo136787/derby/tmp/testbuild/lib/junit.jar:/home/vo136787/derby/tmp/testbuild/lib/db2jcc.jar:/home/vo136787/derby/tmp/testbuild/lib/db2jcc_license_c.jar:/home/vo136787/derby/tmp/testbuild/lib/jakarta-oro-2.0.8.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_de_DE.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_es.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_fr.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_it.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_ja_JP.jar:/home/vo136787/derby/tmp/testbuild/lib/derbyLocale_ko_KR.jar:/home/vo136787/derby/tmp
Re: java.lang.RuntimePermission getProtectionDomain in derbynet/sysinfo.java
Bryan Pendleton wrote: Vemund Ostgaard wrote: The following two tests have been failing when I have run derbynetclientmats: derbynetclientmats/derbynetmats/derbynetmats.fail:derbynet/sysinfo.java derbynetclientmats/derbynetmats/derbynetmats.fail:derbynet/sysinfo_withproperties.java The reason is that the test fails to get the permission for doing getProtectionDomain().getCodeSource() on class com.ibm.db2.jcc.DB2Driver: Hmm I see Myrna writes in another mail: With this patch, derbyall on linux passes (except for known diffs in sysinfo* (db2jcc in same path as sane derby.jars) So, I guess this probably is the same issue that I am seeing. Is there a Jira for it? Thanks, Vemund
java.lang.RuntimePermission getProtectionDomain in derbynet/sysinfo.java
The following two tests have been failing when I have run derbynetclientmats: derbynetclientmats/derbynetmats/derbynetmats.fail:derbynet/sysinfo.java derbynetclientmats/derbynetmats/derbynetmats.fail:derbynet/sysinfo_withproperties.java The reason is that the test fails to get the permission for doing getProtectionDomain().getCodeSource() on class com.ibm.db2.jcc.DB2Driver: - Derby Information JRE - JDBC: J2SE 5.0 - JDBC 3.0 [/export/home/tmp/jagtmp/vo136787derbynetclientmats/install/lib/derby.jar] 10.2.0.0 alpha - (392791) [/export/home/tmp/jagtmp/vo136787derbynetclientmats/install/lib/derbyclient.jar] 10.2.0.0 alpha - (392791) [/export/home/tmp/jagtmp/vo136787derbynetclientmats/install/lib/derbynet.jar] 10.2.0.0 alpha - (392791) [/export/home/tmp/jagtmp/vo136787derbynetclientmats/install/lib/derbytools.jar] 10.2.0.0 alpha - (392791) [Unable to access Protection Domain or Code Source for class class com.ibm.db2.jcc.DB2Driver: access denied (java.lang.RuntimePermission getProtectionDomain)] 2.4 - (17) [/export/home/tmp/jagtmp/vo136787derbynetclientmats/install/lib/db2jcc_license_c.jar] 2.4 - (17) After quite a bit of reading and testing it seems to me that the tests fail when db2jcc.jar is located in the same directory as the derby*.jar files, if I move it to a different directory the tests succeed. So, I am wondering, should the tests run ok also when db2jcc.jar is in the same directory as the derby jars? Is there a rule that these files should be put somewhere else? I am a bit surprised about the behaviour/configuration of the securitymanager in these networkclient tests, though I don't know much about it. I am allowed to know that a driver was loaded from a jar file in a different directory but not that it was loaded from a copy of the same jar file in the same directory? Anyone know why this is? Vemund
Re: Questions about sanity test
Myrna van Lunteren wrote: For the most basic check, I run sysinfo. :-) After that, I run the test lang/supersimple.sql. So, I'd do: java org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql When running the test with -Dframework=DerbyNetClient i get a warning at the end that I do not get when running it embedded: *** End: supersimple jdk1.5.0_06 DerbyNetClient 2006-03-07 13:36:58 *** Warning: Cleanup failed on baseDir: ~/derby/testruns/DerbyNetClient/supersimple I see that the DerbyNetClient catalog has an empty catalog named supersimple and a supersimple.pass file. One level above there is a derby_tests.policy file. Any idea why it might be that I get this warning, what is it the test failed to clean up? Vemund
Re: Questions about sanity test
Myrna van Lunteren wrote: Hi, For the most basic check, I run sysinfo. :-) After that, I run the test lang/supersimple.sql. So, I'd do: java org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql The test harness creates the database, and as this is a .sql test, in essence, it uses ij. Running without any framework parameter defaults to embedded, DerbyNetClient runs it with the network client. Thank you, Myrna. I will look closer at the test and the harness to understand how this works. From what you say this sounds close to what I was looking for. Vemund
Questions about sanity test
Hello, I have been reading a readme file explaining the testing in the org.apache.derbyTesting package, and have a couple of questions in that regard. I am looking for a minimal sanity test to verify that an installation of Derby is healthy, that the most basic functionality works. I'm not exactly sure what such a test for Derby ideally should contain, but I am thinking something like: * create database * create table (with index), insert record, update record, read record, delete record, delete table * do this through 'ij', embedded, and with network client. Should try to make the test robust with regards to changes in the product or different vms. Do you have any opinions on what such a test for Derby should or shouldn't do? Does something like this allready exist? If it doesn't exist, would I most likely find the necessary tests or building blocks somewhere within the derbyTesting package, so I could just glue it together as a suite, or will I have to write it from scratch? Would such a test be useful for you (or your customers)? Thanks, Vemund
