>>>>> "MM" == Mike Matrigali <[EMAIL PROTECTED]> writes:
MM> excellent, I look forward to your work on concurrent I/O. I am likely
MM> to not be on the list much for the next 2 weeks, so won't be able to
MM> help much. In thinking about this issue I was hoping that somehow
MM> the current container cache could be enhanced to support more than
MM> one open container per container. Then one would automatically get
MM> control over the open file resource across all containers, by setting
MM> the currently supported "max" on the container pool.
I have thought about using the current cache implementation for
caching open file descriptors. However, I have not considered using
the container cache for this. Will this work? Do not some of the
code assume that there is only one object for each container? (E.g,
truncation, backup).
MM> The challenge is that this would be a new concept for the basic services
MM> cache implementation. What we want is a cache that supports multiple
MM> objects with the same key, and that returns an available one if another
MM> one is "busy". Also returns a newly opened one, if all are busy. I
MM> am going to start a thread on this, to see if any other help is
MM> available. If possible I like this approach better than having a
MM> queue of open files per container where it hard to control the growth
MM> of one queue vs. the growth in another.
I agree with you that a common cache would be better. However, as you
point out, it is not straight-forward to use the current cache
implementation. My first thought was that it would be too much work
to extend the current caching framework for this use, but I will think
a bit more about this, and follow up in your thread on this.
MM> On the checkpoint issue, I would not have a problem with changes to the
MM> current mechanism to do "rwd" type sync I/O rather than sync at end
MM> (but we will have to support both until we don't have to support older
MM> versions of JVM's). I believe this is as close
MM> to "direct i/o" as we can get from java - if you mean something
MM> different here let me know.
Yes, this was what I was thinking of. Using direct i/o should give
better performance on reads. However, some configurations may see a
performance drop since they depend on the file system buffer for good
performance. However, such applications should be able to reclaim the
performance by increasing the page cache to not depend on file system
buffering.
MM> The benefit is that I believe it will fix the checkpoint
MM> flooding the I/O system problem. The downside is that it will
MM> cause total number of I/O's to increase in cases where the
MM> derby block size is smaller than the filesystem/disk blocksize
MM> -- assuming the OS currently converts our flood of multiple
MM> async writes to the same file to a smaller number of bigger
MM> I/O's.
It will vary from file systems to file systems how this will be done.
It would be nice to do some experiments here. Maybe someone could do
a study on direct i/o on different file systems. (E.g., the
performance impact of different page sizes).
MM> I think this trade off is fine for checkpoints. If
MM> checkpoint efficiency is an issue, there are a number of other
MM> ways to address it in the future.
I agree. Multiple writer threads are one way to address this.
--
Øystein
