Re: {VOTE] Kim Haase as a committer
Rick Hillegas wrote: Please vote on whether we should make Kim Haase a committer. The vote will close at 5:00 pm San Francisco time on Monday March 10. Kim has made an outstanding contribution to Derby's documentation effort. With commit privileges, she will be even more effective. Regards, -Rick +1 -- Jørgen Løland
Re: [jira] Updated: (DERBY-2872) Add Replication functionality to Derby
Hi Wagar, I'm not sure how you should do this, but I'm sure you'll get a good answer if you send this question to the discussion list: [EMAIL PROTECTED] Waqar Hassan wrote: Starting JAVA DB network server using code. please help me how can i start server using coding . On 3/2/08, V.Narayanan (JIRA) <[EMAIL PROTECTED]> wrote: [ https://issues.apache.org/jira/browse/DERBY-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] V.Narayanan updated DERBY-2872: --- Attachment: ReplicationDesign.pdf ReplicationWriteup.txt I started working on a replication wiki over the weekend. I spent the weekend writing down the initial set of details that I thought the wiki should contain and which would be relevant to a new user of the replication feature. I wrote these details in the form of a text file. I would be converting the text file into a replication wiki over the next week. Since this issue deals with the replication design and tries to identify issues that are relevant to this feature I am attaching whatever information I have assimilated to this issue. Note that the attached pdf contains a very simplified block diagram of the main modules involved in replication. I purposely made this simplified to enable the easy understanding of the replication design. More detailed descriptions of the feature can be got by following the JIRA pointers that are mentioned in the explanations of the various blocks of this simplified block diagram, and going through the specific JIRA issues. I have tried to organize the information I have collected or re-collected so that it will be easy for reviewers to point out places in which I have gone wrong. I will be making further updates, improvements and modifications directly to the wiki and will not be attaching it to this issue. Add Replication functionality to Derby -- Key: DERBY-2872 URL: https://issues.apache.org/jira/browse/DERBY-2872 Project: Derby Issue Type: New Feature Components: Replication Affects Versions: 10.4.0.0 Reporter: Jørgen Løland Assignee: Jørgen Løland Attachments: master_classes_1.pdf, poc_master_v2.diff, poc_master_v2.stat, poc_master_v2b.diff, poc_slave_v2.diff, poc_slave_v2.stat, poc_slave_v2b.diff, poc_slave_v2c.diff, proof-of-concept_v2b-howto.txt, proof_of_concept_master.diff, proof_of_concept_master.stat, proof_of_concept_slave.diff, proof_of_concept_slave.stat, replication_funcspec.html, replication_funcspec_v2.html, replication_funcspec_v3.html, replication_funcspec_v4.html, replication_funcspec_v5.html, replication_funcspec_v6.html, replication_funcspec_v7.html, replication_funcspec_v8.html, replication_funcspec_v9.html, replication_script.txt, ReplicationDesign.pdf, ReplicationWriteup.txt, slave_classes_1.pdf It would be nice to have replication functionality to Derby; many potential Derby users seem to want this. The attached functional specification lists some initial thoughts for how this feature may work. Dag Wanvik had a look at this functionality some months ago. He wrote a proof of concept patch that enables replication by copying (using file system copy) and redoing the existing Derby transaction log to the slave (unfortunately, I can not find the mail thread now). DERBY-2852 contains a patch that enables replication by sending dedicated logical log records to the slave through a network connection and redoing these. Replication has been requested and discussed previously in multiple threads, including these: http://mail-archives.apache.org/mod_mbox/db-derby-user/200504.mbox/[EMAIL PROTECTED] http://www.nabble.com/Does-Derby-support-Transaction-Logging---t2626667.html -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. -- Jørgen Løland
Re: [jira] Commented: (DERBY-3523) sql states (X0Y63, X0Y63, X0Y63.S) related to nulls in unique constraints are associated with wrong message texts
Mike Matrigali wrote:
> It does seem like the best case would be for the error message system to
> somehow return a different error message for the same number if it is
> in soft upgrade vs. hard upgrade. Does the error message system support
> such a thing?
Yes, it does. If you want multiple messages with the same SQL state, you
need to add the "..<#>" information to the error state in
SQLState.java. E.g, all these messages have SQL State 08004:
- snip from SQLState.java
String LOGIN_FAILED = "08004";
String NET_CONNECT_AUTH_FAILED = "08004.C.1";
String NET_DATABASE_NOT_FOUND = "08004.C.2";
String AUTH_DATABASE_CONNECTION_REFUSED = "08004.C.3";
String AUTH_SHUTDOWN_NOT_DB_OWNER = "08004.C.4";
String AUTH_ENCRYPT_NOT_DB_OWNER= "08004.C.5";
...
- snip ---
If, e.g., both these messages
'{0}' cannot be a column of a primary key or unique key because it can
contain null values.
'{0}' cannot be a column of a primary key because it can contain null
values.
... should have the same 42831 error code, you would need something like
this in the SQLState.java file:
String LANG_DB2_ADD_UNIQUE_OR_PRIMARY_KEY_ON_NULL_COLS = "42831.S.1";
String LANG_DB2_ADD_PRIMARY_KEY_ON_NULL_COLS = "42831.S.2";
'S' because 42831 is statement severity level (see class javadoc in
SQLState.java), while 1 and 2 are used to tell the messages apart.
messages.xml also needs to be updated, of course.
Re: [VOTE] John H Embretsen as a Derby committer
Daniel John Debrunner wrote: John is actively involved on both the derby-dev and derby-user lists and fully engages in open development. He has had a number of patches committed, most recently taking the stalled JMX work and getting it into a shape where it could be committed to allow others to get involved. Vote closes 2007-04-02 16:00 PDT Dan. +1 -- Jørgen Løland
Re: [VOTE] Thomas Nielsen as a Derby committer
Bryan Pendleton wrote: I propose that Thomas Nielsen shall be a Derby committer. Over the last 6 months, I've worked with Thomas on some Derby features (http://wiki.apache.org/db-derby/OLAPOperations) as well as on some fairly sophisticated bug fixes, including https://issues.apache.org/jira/browse/DERBY-3301 and https://issues.apache.org/jira/browse/DERBY-3321 Thomas has also been active in the development community, reviewing the patches that others propose and suggesting improvements. I think he'd be a valuable committer on the project. Please respond to this thread with your vote. Vote closes 2008-04-04 17:00 PDT. thanks, bryan +1 -- Jørgen Løland
Re: in-memory storage factory?
Jun Yang wrote:
Hi all,
I search in the mail archive and found this:
http://issues.apache.org/jira/browse/DERBY-646
But MemoryStorageFactory is not in the source distribution. Where can I
find it?
Thanks!
Jun
Hi Jun,
The work on in-memory storage was never completed ("status: open") and
is therefore not in Derby.
--
Jørgen Løland
Re: [VOTE] V. Narayanan as a committer
Rick Hillegas wrote: Please vote on whether we should make V. Narayanan a Derby committer. The polls close at 5:00 pm San Francisco time on Thursday April 10. +1! -- Jørgen Løland
Re: [jira] Updated: (DERBY-4186) After failover, test fails when it succeeds in connecting early to failed over slave
Dag,
Thanks for analyzing and fixing this strange issue! Stopping replication
before the startSlave command had completed was never on my mind :-/
I had a look at you patch though, and I think you can fix this bug with
even less code.
From SlaveDatabase.java:86:
/** Set by the database boot thread if it fails before slave mode
* has been started properly (i.e., if inBoot is true). This
* exception will then be reported to the client connection. */
private volatile StandardException bootException;
bootException is only set in one place - SlaveDatabase#handleShutdown.
There you'll also see the reason for the limbo state that made the tests
fail: if an exception makes the slave replication code call
handleShutdown while booting is in progress, the database is supposed to
be shutdown by the client thread when it receives an exception from
SlaveDatabase.boot().
As you already found out, that didn't happen because the bootException
was set during the 500 millis waiting in verifySuccesfulBoot. However,
this should apply to any exception in bootException, not only
DATABASE_SEVERITY ones (although I *think* only DB severity exceptions
will be reported here).
I would go with the same code that is inside the while. Thus, instead of
+if (bootException != null &&
+SQLState.SHUTDOWN_DATABASE.startsWith(
+bootException.getSQLState()) &&
+bootException.getSeverity() ==
ExceptionSeverity.DATABASE_SEVERITY) {
use
+if (bootException != null)
Dag H. Wanvik (JIRA) wrote:
[
https://issues.apache.org/jira/browse/DERBY-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dag H. Wanvik updated DERBY-4186:
-
Attachment: derby-4186-2.stat
derby-4186-2.diff
Respin of this patch, #2, with more comments. I also talked to the author of
this code off-line, Jørgen Løland, and he agreed with my analysis. The new
patch moves the check for the lost exception to inside the method
SlaveDataBase.verifySuccessfulBoot.
Added more explanations in the comments.
After failover, test fails when it succeeds in connecting early to failed over
slave
Key: DERBY-4186
URL: https://issues.apache.org/jira/browse/DERBY-4186
Project: Derby
Issue Type: Bug
Components: Replication, Test
Affects Versions: 10.6.0.0
Reporter: Dag H. Wanvik
Assignee: Dag H. Wanvik
Attachments: bad-slave.txt, derby-4186-2.diff, derby-4186-2.stat,
derby-4186.diff, derby-4186.stat, ok-slave.txt
Occasionally I see this error in ReplicationRun_Local_3_p3:
1)
testReplication_Local_3_p3_StateNegativeTests(org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun_Local_3_p3)junit.framework.AssertionFailedError:
Expected SQLState'08004', but got connection!
at
org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun.waitForSQLState(ReplicationRun.java:332)
at
org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun_Local_3_p3.testReplication_Local_3_p3_StateNegativeTests(ReplicationRun_Local_3_p3.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:105)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
at junit.extensions.TestSetup.run(TestSetup.java:25)
In the code, after a stopMaster is given to the master (should lead to
fail-over),
the tests expects to see CANNOT_CONNECT_TO_DB_IN_SLAVE_MODE (08004.C.7), which
will only succeed if
the tests gets to try to connect before the failover has started. This seems
wrong. If the failover has completed, it should expect a successful
connect (which boots the database, btw, since its shut down after auccessful
failover).
Quote from code:
waitForSQLState("08004", 100L, 20, // 08004.C.7 -
CANNOT_CONNECT_TO_DB_IN_SLAVE_MODE
slaveDatabasePath + FS + slaveDbSubPath + FS + replicatedDb,
slaveServerHost, slaveServerPort); // _failOver above fails...
There is a race between the failover on the slave and the test here I think.
--
Jørgen Løland
Re: [VOTE] Ole Solberg as a committer
Rick Hillegas wrote: Please vote on whether we should make Ole Solberg a committer. The vote closes at 5:00 pm San Francisco time on Thursday June 4. For 4+ years Ole has been a significant contributor to Derby. We all rely heavily on Ole's failure analysis and on the tests and testing infrastructure which Ole continues to extend. Let's make Ole even more productive by minting him as a committer. Regards, -Rick +1! -- Jørgen Løland
Re: [jira] Created: (DERBY-4257) 'Doc error' : master should be specified in slave security manager
Edah-Tally, Did this solve the problem you asked about in email June 1 2009 "Replication - startSlave hangs" to the user list? If this security manager issue is the root of the problem in that email, it would be great if you replied to your own question with the solution. EDAH-TALLY (JIRA) wrote: 'Doc error' : master should be specified in slave security manager -- Key: DERBY-4257 URL: https://issues.apache.org/jira/browse/DERBY-4257 Project: Derby Issue Type: Bug Components: Documentation Affects Versions: 10.5.1.1 Environment: Linux 2.6.29.2-server #1 SMP Sat May 9 22:43:24 CEST 2009 x86_64 AMD Athlon(tm) 64 X2 DualCore Processor 4200+ GNU/Linux java version "1.6.0_06" Java(TM) SE Runtime Environment (build 1.6.0_06-b02) Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode) Reporter: EDAH-TALLY Priority: Minor This concerns the Derby Server And Administration Guide. In the chapter 'Replication and Security', it is specified : <> Add the following permission to the policy file on the slave system: permission java.net.SocketPermission "slaveHost", "accept,resolve"; I could have success with replication only if I use the 'masterHost' in place of 'slaveHost' on the slaves's security policy file. It seems logical to configure the slave host to accept any request from the master host. May be the docs should be updated. Thank you for considering. -- Jørgen Løland
Vacation October 9-10
Hi fellow backupers, I'll be on vacation Thursday and Friday next week. Kindergarten is closed, so I'll play with my son these days. Lars: Manyi is informed and has approved. -- Jørgen Løland
Re: [jira] Updated: (DERBY-1484) Client and embedded behave differently when the table name is null in DatabaseMetaData methods
Myrna van Lunteren wrote: Jørgen, I think a Release note is needed for every change that actually changes behavior - incl. undocumented and incorrect behavior. Release Notes are a help to users to identify those changes in behavior that could impact them when upgrading. Needing a release note for a subtask, or a test, or documentation change is debatable, but I think this one could have a short release note. Please see DERBY-2570 for the template. Thx, Myrna Thank you for commenting on this, Myrna. ReleaseNote text attempt: Summary of Change Tablenames can no longer be null in calls to DatabaseMetaData methods getBestRowIdentifier(), getColumnPrivileges(), getIndexInfo(), getVersionColumns(), getPrimaryKeys(), getImportedKeys() and getExportedKeys(). *Comment: The above is a rather long one-liner, but I think all the information is required* Symptoms Seen by Applications Affected by Change In the previous release, the methods returned information on all tables in the schema if the tablename parameter was null. Derby now raises an SQLException if the tablename is null. Incompatibilities with Previous Release Calls to the specified methods now fail if the tablename parameter is null. Rationale for Change The previous behavior violated the JDBC standard. The new behavior is correct. Application Changes Required Users must recode applications to specify the table names. When information on multiple tables is required, the application first has to get the table names, e.g. by using the DatabaseMetaData method getTables(), and then use the returned table names as input to the method in question.
Re: Regarding DERBY-2124
Ramin Moazeni wrote: Hi In order to fix DERBY-2124, I have to modify the java/engine/org/apache/derby/loc/messages_en.properties file and change the error message associated with the XJ045. is this right? Is there any other files I need to modify? Please advise Thanks, Ramin Moazeni Hi Ramin You may already have noticed that the *english* error message should be modified in messages.xml, not messages_en.properties (see the top of messages_en.properties). The messages.xml file is used to generate messages_en.properties. Non-english messages are modified in the messages_*.properties files as you suggest. Jørgen
Re: query for Primary Key Columns
Hi Ollie, I think you have to use a stored procedure to get the primary key. The call to this stored procedure would be: call sysibm.sqlprimarykeys(,,,); Since there are no catalogs in Derby, the following stored procedure call would provide you with the pk information: call sysibm.sqlprimarykeys(null,'','',''); Alternatively, if you use JDBC to access the database, I would recommend you to have a look at the getPrimaryKeys method in DatabaseMetadata. P.S. Questions like these would normally go to the derby-user mailing list. ollie2308 wrote: Hello, ... new to derby database and want to query the system tables to obtain the column names that make up each tables primary key. Can this be done via a query ? Can find table name, constraint name, constraint type, but can not get the columns that make up the pk Thanks
Compilation problem - head of trunk and JDK6
In the current proof of concept replication code, we boot the slave
database by calling
Database db =
(Database)Monitor.findService(Property.DATABASE_MODULE, dbname);
This requires org.apache.derby.iapi.db.Database to be imported. With the
head of trunk, this import results in compilation errors that seem to be
related to VTIs and probably the new requirements to compile environment
(see further down for compile error). Any ideas for how I should proceed
with this? The problem was probably introduced this or last week.
To get this error, add "import org.apache.derby.iapi.db.Database" to
NetworkServerControlImpl.java and compile with JDK6.
$ ant all
compile:
[javac] Compiling 17 source files to
/export/home/derby/clean1/trunk/classes
[javac]
/export/home/derby/clean1/trunk/java/demo/vtis/java/org/apache/derbyDemo/vtis/core/VTITemplate.java:354:
warning: [deprecation] getUnicodeStream(java.lang.String) in
java.sql.ResultSet has been deprecated
[javac] public java.io.InputStream getUnicodeStream(String
columnName) throws SQLException {
[javac]^
[javac]
/export/home/derby/clean1/trunk/java/demo/vtis/java/org/apache/derbyDemo/vtis/core/VTITemplate.java:300:
warning: [deprecation] getBigDecimal(java.lang.String,int) in
java.sql.ResultSet has been deprecated
[javac] public BigDecimal getBigDecimal(String columnName, int
scale) throws SQLException {
[javac] ^
[javac]
/export/home/derby/clean1/trunk/java/demo/vtis/java/org/apache/derbyDemo/vtis/core/VTITemplate.java:209:
warning: [deprecation] getUnicodeStream(int) in java.sql.ResultSet has
been deprecated
[javac] public java.io.InputStream getUnicodeStream(int
columnIndex) throws SQLException {
[javac]^
[javac]
/export/home/derby/clean1/trunk/java/demo/vtis/java/org/apache/derbyDemo/vtis/core/VTITemplate.java:155:
warning: [deprecation] getBigDecimal(int,int) in java.sql.ResultSet has
been deprecated
[javac] public BigDecimal getBigDecimal(int columnIndex, int
scale) throws SQLException {
[javac] ^
[javac]
/export/home/derby/clean1/trunk/java/demo/vtis/java/org/apache/derbyDemo/vtis/core/StringColumnVTI.java:70:
org.apache.derbyDemo.vtis.core.StringColumnVTI.SimpleBlob is not
abstract and does not override abstract method
getBinaryStream(long,long) in java.sql.Blob
[javac] public static final class SimpleBlob
implements Blob
[javac] ^
[javac]
/export/home/derby/clean1/trunk/java/demo/vtis/java/org/apache/derbyDemo/vtis/core/StringColumnVTI.java:148:
org.apache.derbyDemo.vtis.core.StringColumnVTI.SimpleClob is not
abstract and does not override abstract method
getCharacterStream(long,long) in java.sql.Clob
[javac] public static final class SimpleClob
implements Clob
[javac] ^
[javac]
/export/home/derby/clean1/trunk/java/demo/vtis/java/org/apache/derbyDemo/vtis/core/XmlVTI.java:39:
org.apache.derbyDemo.vtis.core.XmlVTI is not abstract and does not
override abstract method updateNClob(java.lang.String,java.io.Reader) in
java.sql.ResultSet
[javac] public class XmlVTI extends StringColumnVTI
[javac] ^
[javac]
/export/home/derby/clean1/trunk/java/demo/vtis/java/org/apache/derbyDemo/vtis/example/PropertyFileVTI.java:35:
org.apache.derbyDemo.vtis.example.PropertyFileVTI is not abstract and
does not override abstract method
updateNClob(java.lang.String,java.io.Reader) in java.sql.ResultSet
[javac] publicclass PropertyFileVTI extends FlatFileVTI
[javac] ^
[javac]
/export/home/derby/clean1/trunk/java/demo/vtis/java/org/apache/derbyDemo/vtis/example/SubversionLogVTI.java:35:
org.apache.derbyDemo.vtis.example.SubversionLogVTI is not abstract and
does not override abstract method
updateNClob(java.lang.String,java.io.Reader) in java.sql.ResultSet
[javac] publicclass SubversionLogVTI extends FlatFileVTI
[javac] ^
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 5 errors
[javac] 4 warnings
BUILD FAILED
--
Jørgen Løland
Re: Compilation problem - head of trunk and JDK6
Rick Hillegas wrote: Dag H. Wanvik wrote: I think the reason is that the VTI demo result set template presumes JDK1.5, since the JDBC has more methods in JDK 1.6 which are unimplemented in the VTI template. Dag I just checked in submission 587905. Hopefully this fixes the build in Java 6 environments. Please let me know if your build is still broken after syncing to the head of the trunk. Regards, -Rick Seems to work now. Thanks for the quick fix, Rick :) -- Jørgen Løland
Re: Replication in Derby
Laura Stewart wrote: For those of you who are working on Replication in Derby, please remember to open separate issues for the documentation requirements. Replication can be complex and the documentation will need to be clear and detailed :-) Thanks! Hi Laura, Thanks for the reminder and for showing interest in the ongoing replication activities. I created DERBY-3169 as a place keeper for this task. -- Jørgen Løland
Adding a "replication" component to jira
Would it be ok to add a "Replication" component to Jira? I find "Services" to be a too broad category when I'm adding Jira issues. -- Jørgen Løland
Re: Searching the Derby Documentation
Rick Hillegas wrote: Laura Stewart wrote: On 11/14/07, Lance J. Andersen <[EMAIL PROTECTED]> wrote: Perhaps this is an indication that we need to revisit the layout of the manuals to make them easier to use and put this as a high priority for things to address going forward. If the documentation is difficult to navigate, this can be a turnoff to users. Regards lance I think that what we really need to invest in is changes to the code that generates the docs, so that an index can be created with the output. Many index entries are already in the DITA files for the documentation and are not exposed because the code that generates the output is not setup to support index generation... +1 to the idea of generating an index. The old Cloudscape index used to be very helpful and I think that a lot of that work has been preserved in the existing index entry markers. This may be a silly question, but why are we using dita when there are good tools for this? Why aren't we using LaTeX? -- Jørgen Løland
Re: Adding a "replication" component to jira
Andrew McIntyre wrote: On Nov 15, 2007 5:59 AM, Dag H. Wanvik <[EMAIL PROTECTED]> wrote: Jørgen Løland <[EMAIL PROTECTED]> writes: Would it be ok to add a "Replication" component to Jira? I find "Services" to be a too broad category when I'm adding Jira issues. +1 I was thinking this would be a good idea when I saw someone assign an issue concerning replication to 'miscellaneous'. I've gone ahead and added a replication component. andrew Thanks -- Jørgen Løland
Re: [jira] Commented: (DERBY-3064) Implement the LogShipper that will enable the shipping of Log records from the master to the slave
+1 Sounds reasonable to me too. V.Narayanan (JIRA) wrote: [ https://issues.apache.org/jira/browse/DERBY-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550015 ] V.Narayanan commented on DERBY-3064: 1) when you attempt the first connect 2) when a connection failure occurs 1) I believe that if an IOException occurs here we should log the exception and exit. User can handle the cause of the exception here and can restart replication after correcting the problem. We do not know the severity of the cause of the IOException. So it is very difficult to distinguish between when we should stop and re-try based on the cause. So I believe here we should exit when a timeout occurs. 2) Here we should re-try if a timeout or a IOException occurs. This would take care of the case for example when the network connection goes down and comes back again. Also if the log buffer is full and the network is down we fail aborting reconnect attempts. Does this sound OK to you? Implement the LogShipper that will enable the shipping of Log records from the master to the slave -- Key: DERBY-3064 URL: https://issues.apache.org/jira/browse/DERBY-3064 Project: Derby Issue Type: Sub-task Components: Miscellaneous Reporter: V.Narayanan Assignee: V.Narayanan Attachments: LogShipperImpl_v1.diff, LogShipperImpl_v1.stat, LogShipperImpl_v2.diff, LogShipperImpl_v2.stat, LogShipperImpl_v3.diff, LogShipperImpl_v3.stat, LogShipperImpl_v4.diff, LogShipperImpl_v4.stat, LogShipperImpl_v5.diff, LogShipperImpl_v5.stat, LogShipperIntegration_v1.diff, LogShipperIntegration_v1.stat, LogShipperIntegration_v2.diff, LogShipperIntegration_v2.stat, LogShipperIntegration_v3.diff, LogShipperIntegration_v3.stat, LogShipperIntegration_v4.diff, LogShipperIntegration_v4.stat, LogShipperIntegration_v5.diff, LogShipperIntegration_v5.stat -- Jørgen Løland
Re: [jira] Updated: (DERBY-1651) Develop a mechanism to migrate mySQL databases to Derby. Migration tool should include both schema and data migration options.
Andrew McIntyre (JIRA) wrote: [ https://issues.apache.org/jira/browse/DERBY-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew McIntyre updated DERBY-1651: --- Fix Version/s: (was: 10.2.3.0) Unsetting fix version, no movement for over a year. Develop a mechanism to migrate mySQL databases to Derby. Migration tool should include both schema and data migration options. -- Key: DERBY-1651 URL: https://issues.apache.org/jira/browse/DERBY-1651 Project: Derby Issue Type: New Feature Components: Tools Environment: All platforms Reporter: Ramin Moazeni Assignee: Ramin Moazeni Attachments: DBMigration.diff, MigrationTool-MySQLtoDerby.zip Develop a mechanism to migration databases created by other database engines to Derby. While my current interest is to migrate mySQL databases to Derby, the tool could be developed in a way to extend this mechanism to allow migration from other database engines in the future. More details of proposed functionality and implementation strategy can be found at: http://wiki.apache.org/db-derby/MysqlDerbyMigration The plan is to develop and submit patches incrementally. First patch supports migration of tables and views from mySQL database to Derby. Just a thought... wouldn't VTIs be perfect for this purpose? -- Jørgen Løland
Re: Looking towards 10.4
[EMAIL PROTECTED] wrote: Unless someone else volunteers to replace Bernt, I am prepared to take over as the next release manager, but on the condition that we can delay feature freeze to the March 1st time frame, because I have quite a bit of work left on DERBY-3221 and DERBY-3192. Regardless of who will be the next release manager I think it would be good have a discussion on when the feature freeze should be. I encourage everyone who is working on new features to post the feature freeze date they would like to see. Dyre, thanks for volunteering as release manager. March 1st sounds good to me - we have quite a few things left on our replication todo-list. -- Jørgen Løland
Re: Finding object type
Is there a function in the derby source code to find the type of an object.. i.e. if the XXX is a table , it should return 'TABLE' .. if YYY is a view, it should return 'VIEW'.. or anything like that?? please tell where to look for it.. please help asap... Hi Jazarine, Would it be possible to use JDBC in your case? If so, you could e.g. have a look at DatabaseMetaData.getTables (check out the Java API). If that is not an option, you may write a query like the one below to get all non-system tables and views. select tablename, tabletype from sys.systables where tabletype not like 'S'; A tabletype of "T" is a normal table, while "V" means view. These methods should work if you want to find the table type from an application using Derby. If you want to find the tabletype *inside Derby* (because you modify the source code), I would start looking at org.apache.derby.client.am.DatabaseMetaData#getTables and find out what happens there. Hope this helps -- Jørgen Løland
Re: Patch available checkbox on the attach file page?
Kristian Waagan wrote: Hello, When I upload new files to Jira issues, I'm often a bit annoyed by the fact that I have to first upload the file(s) and then go edit the issue to tic the patch available checkbox. It also generates two mails. Would it make sense to include the patch available checkbox on the attach file page/dialog? I have not investigated how this can be done, but if people think it is a useful change I can do a little research. I checked quickly with infra, and based on the feedback I got it seems we would be allowed to make such a change if it is feasible. +1, and if possible also default to "Grant license to ASF...". -- Jørgen Løland
Re: Patch available checkbox on the attach file page?
Thomas Nielsen wrote: Jørgen Løland wrote: Kristian Waagan wrote: Hello, When I upload new files to Jira issues, I'm often a bit annoyed by the fact that I have to first upload the file(s) and then go edit the issue to tic the patch available checkbox. It also generates two mails. Would it make sense to include the patch available checkbox on the attach file page/dialog? I have not investigated how this can be done, but if people think it is a useful change I can do a little research. I checked quickly with infra, and based on the feedback I got it seems we would be allowed to make such a change if it is feasible. +1, and if possible also default to "Grant license to ASF...". +1 from me too on patch available if possible. I have second thoughts over defaulting to "Grant license to ASF ... ". That could lead to someone adding code which is owned by someone else *with* a grant ASF even when that wasn't intentional. I fear that could lead to a very big legal mess (but no I'm not a lawyer so I could be wrong). Forcing the user to click the radio button means they make a conscious choice wrt granting ASF the license? See your point, but maybe we could at least get an error message if *none* of the radio buttons were selected? As it is now, you're not forced to chose any of them. Anyway, this is just a "nice to have" functionality. -- Jørgen Løland
Is it possible to shutdown a service module without shutting down the database?
Most of the replication functionality has been added as services named MasterFactory and SlaveFactory (implemenentations called Master and SlaveController). These are booted from RawStore when replication is started, by using Monitor.bootServiceModule. So far, so good. When replication is stopped, however, there does not seem to be any way to remove these services from the monitor without shutting down the entire database. If I understand this correctly, there will be a reference in Monitor to this service for the entire lifetime of the database. That would also mean that the object cannot be gc'ed, which looks like a (small) memory leak to me. In addition, this would prevent us from creating a new module object of the same type since bootServiceModule would return the reference to the old one. I have tried Monitor.getMonitor().shutdown(slaveFactory_reference), but this method shuts down the entire database. An alternative is to call stop() on the module, which would stop it from doing anything but not remove it's reference in the Monitor. What I'm looking for is basically this (from Monitor class javadoc): "Optionally - an individual module within a service may be shutdown, this will in turn shutdown any modules it started if those module are not in use by other modules within the service. This would be handled by the monitor, not the module itself." Am I missing something, or is this functionality not implemented? -- Jørgen Løland
Opinions please - some questions regarding replication
I have a few questions regarding replication that would be great
to hear the community's opinion on before the 10.4 branching. The
reason for bringing this up before code freeze is that these will
have impact on existing applications if changed later.
1. The master of a replicated database ships log records and
other information to the slave through a network socket. 8001
is currently default port for this communication. From
http://www.iana.org/assignments/port-numbers it seems that
this port is reserved for
vcom-tunnel8001/tcpVCOM Tunnel
vcom-tunnel8001/udpVCOM Tunnel
Should we change port, and maybe even register a port in IANA,
or is it fine as it is? I have no idea how long it will take
to register a port, but chances are it will be longer than the
time we have for 10.4 with the current plan. Another
alternative is to use a port from the range called "dynamic
and/or private ports" (49152 through 65535) BTW: I noticed
that 1527 is registered by Oracle, which may also be a
problem?
tlisrv 1527/tcporacle
tlisrv 1527/udporacle
Note that this is only a question of default port. The port
number may be set explicitly by the client starting
replication.
2. About a week ago, Dibyendu Majumdar raised a question about
why the replication modules are placed under services and not
store. Should replication be moved from
org.apache.derby.{iapi|impl}.services.replication to e.g.
org.apache.derby.{iapi|impl}.store.replication, or is it OK to keep
the current placement?
3. Should replication be shipped in a separate JAR file? Pros and
cons as I see it:
+ Remove 65KB (unjared) of class files from derby.jar
+ Possible to have a different security policy file for replication?
- Yet another JAR file
--
Jørgen Løland
Re: using the EmbedConnection#handleExceptions method in failover
Narayanan wrote: Hi, While working on the failover issue in replication, I have a requirement that I need to shutdown the master database when a failover is successful. I use the logic that throwing an exception with Database severity should shutdown the database. For this to happen from going through the EmbedConnection class I concluded that I need to parse the exception that is thrown with the handleExceptions method that is part of this class. The handleExceptions method basically tests the severity of the exception thrown and performs appropriate action. Can a person who knows about this method better please tell me if I have concluded correctly and if what I think the handleExceptions method does is correct? Narayanan Hi Narayanan, To make an exception shut down a database, there are a few requirements: 1) The exception must be database severity (SQLState x.D) 2) The thread that caused the exception must have the database context. You get this for free as long as the client connection attempt has called tr.startTransaction in EmbedConnection#creator 3) The exception must be a StandardException I had a look at the master side failover code, and you have 1) and 2) covered. However, the StandardException thrown from MasterController is wrapped in an SQLException in BasicDatabase. I think DERBY-3428 can be solved by not wrapping the exception. -- Jørgen Løland
Re: Opinions please - some questions regarding replication
A short summary of the thread so far: 1. Rick and Jørgen thinks it would be good to register a port. We will use 8001 for now unless someone has another suggestion, but will write clearly in the documentation that the default port is likely to change. I will create a jira for this and then contact IANA to get a port. 2. Dan, Dibyendu and Jørgen thinks it would be good to move the replication code to o.a.d.i.store.replication (since replication is integrated with the store). I'll create a jira for this. 3. It is possible to make a separate jar for replication if we solve some sealing issues (Dan, Narayanan), but with only 65K of uncompressed class files it may not be worth it (Rick, Jørgen). *Not concluded yet.* -- Jørgen Løland
Re: 10.4 Feature Freeze
[EMAIL PROTECTED] wrote: Replication On track The 10.4 replication functionality is done, but I would be a lot more confident in this new functionality if we had the time to add a lot of tests. I would prefer that we move code-freeze to end of next week to add tests, but I'll leave that decision to you. -- Jørgen Løland
Re: 10.3 release coming up...fast!
Myrna van Lunteren wrote: Hi! We have now about 7 days before the code complete date of 6/1/07! I now want to talk about 3 things: Hi Myrna, The date sounds good to me. I don't have any new patches on the way, and have only one uncommited patch (2609) which is a test issue and thus not a blocker. --- Jørgen Løland
Re: Updating rows with an open cursor...what is the expected behavior?
Army wrote: The remaining question is whether or not this is correct. I'm hoping Knut Anders is right when he wrote: knut> there's no guarantee (I think) as to when the rows are actually read. knut> But I'll leave it to the scholars to find the chapter and verse in the knut> appropriate spec... ;) That's what I'm looking for :) SQL-2003 states the following for cursors and updates (Chp 4.32.2. Operations on and using cursors): --8<-- If a cursor is open, and the SQL-transaction in which the cursor was opened makes a significant [1] change to SQL-data, then whether that change is visible through that cursor before it is closed is determined as follows: - If the cursor is insensitive, then significant changes are not visible. - If the cursor is sensitive, then significant changes are visible. - If the cursor is asensitive, then the visibility of significant changes is implementation-dependent. -->8--- Comment: [1] significant meaning update performed by other commands than statement: posisioned> that would have had affect if commited before the cursor was opened. Hence, the behavior described by Army is correct iff the cursor is defined as asensitive. Further more: Chp 14.1 (declare cursor): --8<- 5) If is not specified, then ASENSITIVE is implicit. -->8- My understanding is therefore that the differences between index and scan is allowed by SQL 2003. An interesting question is what Derby does if the cursor is specified to be either sensitive or insensitive... Note that I was able to reproduce this behavior with a JDBC program, inlined at the end of this email. That program uses the Connection.createStatement() method, which means that result sets from that statement should default to TYPE_FORWARD_ONLY (as opposed to SCROLL_INSENSITIVE or SCROLL_SENSITIVE). So it seems like the question is: should a "forward only" result set be sensitive to updates to rows that it hasn't read yet? Or is this documented somewhere in JDBC as "not guaranteed" (or whatever the correct term is)? Unless there is doc saying that such a thing leads to undefined behavior (per Knut Anders' suggestion), it seems odd (to me) that the "sensitivity" of a forward-only result set apparently depends on the underlying Derby scan type... As defined in SQL 2003, ASENSITIVE is the default sensitivity. -- Jørgen Løland
Re: Updating rows with an open cursor...what is the expected behavior?
I discussed this with Bernt just now, and found the following in the JDBC 4 spec to be of interest as well: --8<-- The three different ResultSet types are described below. 1. TYPE_FORWARD_ONLY The result set is not scrollable; its cursor moves forward only, from before the first row to after the last row. The rows contained in the result set depend on how the underlying database materializes the results. That is, it contains the rows that satisfy the query at either the time the query is executed or as the rows are retrieved. 2. TYPE_SCROLL_INSENSITIVE The result set is scrollable; its cursor can move both forward and backward relative to the current position, and it can move to an absolute position. The result set is insensitive to changes made to the underlying data source while it is open. It contains the rows that satisfy the query at either the time the query is executed or as the rows are retrieved. 3. TYPE_SCROLL_SENSITIVE The result set is scrollable; its cursor can move both forward and backward relative to the current position, and it can move to an absolute position. The result set reflects changes made to the underlying data source while the result set remains open. The default ResultSet type is TYPE_FORWARD_ONLY. -->8-- It looks like the default behavior (TYPE_FORWARD_ONLY) is more or less the same as SQL ASENSITIVE. "That is, it contains the rows that satisfy the query ... as the rows are retrieved" should allow different results when index and scan are used. -- Jørgen Løland
Re: State of Derby at OSCON
Rick Hillegas wrote: Some Replication (DERBY-2872, it looks like Jørgen and Narayanan are working on this) Confirmed. (Might as well have all the "will be working on for 10.4" messages in one thread). -- Jørgen Løland
Re: Proposal: default property file
Ole Gunnar Borstad wrote: Siterer Knut Anders Hatlen <[EMAIL PROTECTED]>: We could also take the idea one step further and create a file with meta-data about the properties, for instance a description, whether it's a database property or a system property, minimum value, maximum value and default value. This could be helpful information both for management/monitoring and perhaps some time for automatically generated documentation. I like this idea, it's pretty much what Derby is lacking wrt. properties; A clean and comprehensible interface. I don't know if this has to be done the hardcore way with XML or something, or if we could just do it with commenting. I think derby.properties uses # for comments. Keeping it simple is a must I guess? Storing default property values in a default.properties file sounds like a very good idea. Personally, I think the way properties are handled in the Derby code is developer-unfriendly with the mixture of hardcoded variables and Property objects. Every now and then, I have to text-search for a property name and see what the value of the alternative is. A properties file in derby.jar would make this much easier. If possible, I think it would be good to use the same format for this file and derby.properties. Since the format of derby.properties has already been defined, I think it would be better to go with that than with xml. Just my $0.02. -- Jørgen Løland
Re: DatabaseMetaData.getTables() problems for 10.3.1.3 with a 10.3.1.2 database
I see this as well. I created DERBY-2976 for it.
Tim Dudgeon wrote:
I'm seeing an exception for the DatabaseMetaData.getTables() method when
using 10.3.1.3 with a database that has been upgraded to 10.3.1.2.
10.2 databases that have not been upgraded seem to be OK.
Tim
Class.forName("org.apache.derby.jdbc.EmbeddedDriver");
Connection con =
DriverManager.getConnection("jdbc:derby:/path/to/somedb;upgrade=true",
null, null);
DatabaseMetaData meta = con.getMetaData();
ResultSet rs = meta.getTables(null, "APP", null, null);
Exception in thread "main" java.sql.SQLException: The parameter position
'4' is out of range. The number of parameters for this prepared
statement is '3'.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedResultSet.noStateChangeException(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.setString(Unknown Source)
at
org.apache.derby.impl.jdbc.EmbedDatabaseMetaData.getTables(Unknown Source)
at derbytest.Main.main(Main.java:49)
Caused by: ERROR XCL13: The parameter position '4' is out of range. The
number of parameters for this prepared statement is '3'.
at
org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at
org.apache.derby.impl.sql.GenericParameterValueSet.checkPosition(Unknown
Source)
at
org.apache.derby.impl.sql.GenericParameterValueSet.getParameterForSet(Unknown
Source)
... 3 more
--
Jørgen Løland
Re: [VOTE] Øystein Grøvlen as a commi tter
Rick Hillegas wrote: Please vote on whether we should make Øystein Grøvlen a committer. The vote will close at 5:00 pm San Francisco time on Tuesday August 28. +1 -- Jørgen Løland
Re: [jira] Commented: (DERBY-3021) Replication: Add a ReplicationSlave controller that will manage replication on the slave side
Dag H. Wanvik wrote: "Jørgen Løland (JIRA)" <[EMAIL PROTECTED]> writes: Any thoughts on whether the most appropriate error code for this would be: * XRE02.C since it is a replication error message with session severity, or * 0800x (e.g., 08006 - connection failure) since this is a connection error? 08006 seems to be a network client error, wouldn't a new 08004 (the server is there but rejects the connection for some reason...) be more in the spirit of existing allocations? But I am OK with XRE02.C, too. I looked at some other non-"08xxx" which have session severity level in Derby, but am not sure they give much guidance. Maybe we should prefer 08004 when reasonble, since this has standard SQL semantics (not implementation defined). In JDBC 4 this SQLSTATE class has a defined mapping to SQL(Non)TransientConnectionExceptions, too. I must admit I feel a bit out on a limb, here, so it someone has any general guidelines for how to allocate new error codes in Derby, I, for one, would appreciate it. I agree; it makes sense to use existing SQL codes whenever possible and new, custom error codes when the message does not fit anywhere else. Hence, in this case, 08004 seems to be the better choice. -- Jørgen Løland
Re: [VOTE] Dyre Tjeldvoll as a committer
+1 -- Jørgen Løland Rick Hillegas wrote: Please vote on whether we should make Dyre Tjeldvoll a committer. The vote will close at 5:00 pm San Francisco time on Monday September 17. Since 2005 Dyre has submitted many patches and fielded questions on the mailing lists. His contributions range across Derby's Network, Language, and Storage layers and include the following: Performance: 2226, 938, 827, 825, 815 Cleanup and bugs: 2594, 2223, 2191, 2114, 2050, 336, 330, 249, 220, 128, 85 JDBC4: 1380, 1282, 1236, 1235, 1094, 1093, 925, 924 On a personal note, I like how his dry humor leavens our discourse. Regards, -Rick
Re: Debugging Replication
Mayuresh Nirhali wrote: I would like to know if there are any thoughts around debugging the proposed Replication functionality ?? I am not following these discussions closely enough yet! But, I would like to know if for example the logShipper will print out some information in derby.log when the X log is shipped OR there will be a separate machinery like say replicationTrace OR both with different levels of debugging information ? I believe this will be helpful in tracking Replication failures. Please excuse if this question is too early in the development effort planned! Hi Mayuresh, Thanks for a great suggestion. We will definitely take this into account :) Planning has started for junit testing of the replication feature, but as far as I know, no plans have been made for writing messages/tracing to e.g. derby.log. However, I think this is a very good idea. We should at least have messages like: derby.log on master: ReplicationMaster flushTo: 999 ReplicationMaster LogShipper: sent message 1234 to slave, highest log instant: 999 derby.log on slave: ReplicationSlave: received message 1234, highest log instant: 999 ReplicationSlave: appended log instant 555 ReplicationSlave: appended log instant 672 ReplicationSlave: appended log instant 999 It is to early to make a guess for how much and detailed information would be required, but that can be decided upon as the work progresses. The derby.log messages should be turned on by a replication_debug variable being true since a lot of messages would otherwise be generated. -- Jørgen Løland
Re: [jira] Commented: (DERBY-3064) Implement the LogShipper that will enable the shipping of Log records from the master to the slave
2. It seems like you suggest that the shipping thread will continously
check to see whether the timeout has expired. Generally, such
busy-waits should be avoided. I think you should introduce a timer
or something so that the thread can be suspended until it is time
to ship log records.
I have introduced the time concept to overcome the limitation imposed by
DaemonFactory.
The DaemonFactory would call performWork (i.e.) ship log records at its own
convinience. As i understand it we cannot configure DaemonFactory to transmit
at our mentioned time interval.
So, When performWork is called entirely depends on how good DaemonFactory is
feeling :-)
We do not want this to happen, because what if this is too soon for us or what
if it
is too late.
Hence we introduce a transmit interval variable that calculates the difference
between the
last call and the current call of performWork or rather
shipALogChunk(forceFlush also)
If I understand you correctly, this time interval will only be helpful
in the "too soon" case, not the "too late" case.
When I wrote version 2 of the proof of concept patches (see DERBY-2872),
I used a Daemon to read log records at the slave side. I basically had:
performWork {
if (messageWaitingForProcess) processMessage();
}
The waiting interval between each time performWork was called was way
too high for this usage (in the order of 3-10 seconds). This interval is
probably too high for the log shipper as well, so my guess is that
you'll need something similar to an eternal loop with a wait(millis)
somewhere in this package.
--
Jørgen Løland
Re: Bugfixing and 10.3 maintenance release
+1 I also plan to devote some cycles on bugfixing for the maintenance release. I'm currently looking into a few candidates... -- Jørgen Løland Kathey Marsden wrote: I've been thinking quite a bit about the bug backlog and was wondering what it might take to get the community motivated to fix bugs. I decided that what might really help is a maintenance release. We already have some valuable fixes in 10.3 which users have asked for. If each developer were willing to put in a couple more we could have a valuable maintenance release in December.Who would be willing to participate if I were to volunteer to be release manager for a release December 10? By participate I mean would you be willing to make fixes or test the release? If we get enough community enthusiasm for the release I think it is worth doing. Kathey Below are some reports you might find useful: Open Code Bugs [279] https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&&type=1&pid=10594&status=1&status=3&status=4&component=11407&component=11409&component=11690&component=11410&component=11709&component=11411&component=11415&component=11408&component=11412&component=11414&sorter/field=issuekey&sorter/order=DESC Open Product Regressions [8] https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&&pid=10594&status=1&status=3&status=4&customfield_12310090=Regression&sorter/field=issuekey&sorter/order=ASC&sorter/field=updated&sorter/order=DESC&sorter/field=priority&sorter/order=DESC Regression Test Failures [61] (Yikes!) https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&&pid=10594&status=1&status=3&status=4&component=12310664&sorter/field=issuekey&sorter/order=ASC&sorter/field=updated&sorter/order=DESC&sorter/field=priority&sorter/order=DESC Shorter list of high value fix candidates. Please add as you see appropriate http://wiki.apache.org/db-derby/HighValueFixCandidates
Re: deprecation warnings with jdk15 build
Myrna van Lunteren wrote: Hi, Now that we switched to jdk15 as the base level for the trunk, I see a ton of deprecation warnings. My jdk is: Sun's 1.5.0_12-b04, and I've got build.compiler set to modern I find it bothersome and hard to see 'real' build failures like this, but trunk/tools/ant/properties/modern.properties has deprecation set to 'on'. Would it be reasonable to set this to 'off' or am I the only one seeing these warnings? Or is there another trick I need to do? Thx, Myrna Wouldn't it be even better if we fixed these? There is usually a good reason for deprecating methods, e.g. performance. -- Jørgen Løland
