Anders Morken wrote:
Suresh Thalamati (JIRA):
[ http://issues.apache.org/jira/browse/DERBY-801?page=comments#action_12434613 ]
<snip ..>
RAFContainerFactory.java
------------------------
Logic in this new class seems to be deciding whether to load RafContainer.java
or
the RafContainer4.java based on the JVM. I am not sure, if this logic is
necessary here. Did you consider using basic services to load the java classes
specific to a JVM ?
Can you do that? Never occured to me, I'll check it out. Thanks for the
tip. Any pointers and tips would be welcome, but given enough time and
source code perusal I'll probably figure it out on my own as well. From
a quick look at modules.properties it looks pretty straightforward, at
least if adding "J4" to a module name means "load this in Java 1.4 and
up". =)
you are are the right track. I have used this mechanism long time ago, you
might need to something like:
Replace the current data factory module entry
(:derby.module.rawStore.data.generic) with the following entries :
derby.module.rawStore.dataJ1=rg.apache.derby.impl.store.raw.data.BaseDataFileFactory
derby.env.jdk.dataJ1=1
derby.config.rawStore.dataJ1=derby
derby.module.rawStore.dataJ4=org.apache.derby.impl.store.raw.data.DataFileFactoryJava4
derby.env.jdk.dataJ4=4
derby.config.rawStore.dataJ4=derby
DataFileFactoryJava4 will be the new class extended from
BaseDataFileFactory.
I think basic services has support to boot a specific factory implementation
based on the JVM using modules.properties. For example in the current
scenario, one can extend BaseDataFileFactory.java class to
implement newContainerObject(), which will return the RafContainer4( ..). add
the new class to modules.properties to boot only on versions >=jdk14.
4) I noticed, there is new encryption buffer created on every writePage() call,
if the database is encrypted. This may cause jvm peak memory usage increase,
when there is a checkpoint, if there are lot of dirty pages in the cache and
if garbage collection is not happening fast enough. I hope this does not
lead to out of memory errors!
We may need to implement some kind of scheme, that will help in reuse of
the encryption buffers.
This is a (classical) choice between concurrency and resource
consumption. On one hand, object creation and garbage collection from
the eden space is pretty cheap on 1.4 and newer VMs, but we'll force
additional GC runs... This is a complex subject, with many variables. =)
I suppose it could be interesting to do a little throughput testing with
this patch and an encrypted database and see if there is any impact. I
can't really say that I have any idea if recycling a single buffer or
creating lots of buffers is the cheaper way...
I agree, new jvms are smarter. This problem can be addressed later, if
needed. On older jvms I used to run into out of memory problem when I
do some thing like that.
5) I am ok with readPage() and writePage() routines in RafContainer4.java.
just curious , if you considered implementing new read/write..etc calls in
the RafContainer4.java using file channel and just wrapper methods in the
RafContainer.java using the existing random access file, instead of
overriding readPage()/writePage() ...etc.
Uhm, I'm sorry, I'm not sure I understand what you're thinking of here?
Could you elaborate a bit? =)
To avoid dealing with changes to two copies of readPage/writePage
routines in
future fixes; I would have tried to abstract out just the code that is
really
different in jdk14 vs jdk13. For example :
in RafContainer.java , implement :
writeBuffer( .....) {
synchronized(this) {
fileData.seek( ...)
fileData.write( ..)
}
}
and in RafContainer4.java, override that routine :
writeBuffer(....) {
iochannel.write( ...)
}
similarly any other functionality you need. I undestand, it adds an
extra level to the call stack; my guess is that should not have major
performance impact.
For all I know , it may not be that simple ; Different exceptions
..etc. Please feel free to ignore this suggestion, if you think
current approach is good enough.
If we wanted to go even further we could consider using memory mapped IO
- then the parts of Derby that modify data pages could access the memory
mapped file directly and save some overhead, but that probably requires
a bit of a rework of some interfaces and the code that modifies data
pages. And then we have the case of large databases and 32-bit address
spaces... =)
It would be intersting to find, if that improves the performance further.
6) Please file a JIRA to enhance StorageFactory interfaces to support NIO.
There's already DERBY-262 in Jira, but it's a bit unspecific... Anything
in particular you think needs to be added to the interfaces?
DERBY-262 is a very generic NIO entry. I think it would be better to
file a separate JIRA entry that will describe the changes needed in
the storage factory to implement what you are doing in this fix using
the storage factory, instead of limiting the RAFContainer4.java usage
only to storage factory that is based on the java.io.RandomAccessFile.
Thanks
/suresh
