I think I found the reseon for the improved performance in derby db:
I downloaded the source files of derby db and searched for 'sync' or
'fsync'.
Searching for 'sync' brought too many results, but 'fsync' was found in only
one file:
engine/org/apache/derby/impl/store/raw/log/LogToFile.java
And here's the comment header of this file:
* Write sync mechanism support is added for performance reasons.
* On commits, logging system has to make sure the log for committed
* transaction is on disk. With out write sync , log is written to
the
* disk and then fsync() is used on commits to make log is written
to the
* disk for sure. On most of the OS , fsync() calls are expensive.
* On heavey commit oriented systems , file sync make the system run
slow.
* This problem is solved by using write sync on preallocated log
file.
* write sync is much faster than doing write and file sync to a
file.
* File should be preallocated for write syncs to perform better
than
* the file sync method. Whenever a new log file is created,
* logSwitchInterval size is preallocated by writing zeros after
file after the header.
And that explains the behaviour of the file creation found in ./log
directory, which is designed to keep the log files.
The files created there are created at the size of 1MB and their size is not
changed during the db's operation. They are full of zeroes (because each run
I stopped with CTRL-C, so the file was allocated but did not have the time
to get full with numbers).
I think this solves the case of the improved performance. Please correct me
if I'm wrong.
Mike Matrigali wrote:
>
> T
>
> yarono wrote:
>> My OS is suse10.
>>
>> my JVM is IBM's version 1.4.2 as detailed in the following link:
>> http://www.novell.com/products/linuxpackages/server10/i386/java-1_4_2-ibm-jdbc.html
>>
>> The app is indeed single-threaded, so group commit is not the issue.
>>
>> The synchronous write was measured in c (not in java).
>>
>> Is there a way to control or configure the synchronization of writes of
>> the
>> JVM?
> not in the JVM, but what we have seen is that there are OS/disk drive
> parameters that can be set which disable actually executing a sync when
> a program uses certain interfaces to do the sync. You don't say what
> specific interface you used in c to do the sync, so I can't comment on
> that.
>
> In windows what we found is that there is the ability on a per disk
> drive to set a flag called "write cache enabled". Basically it says
> when write hit device driver return immediately even if program has
> requested
> no return until write has synced to platter. What is more confusing
> is that in java there are 2 ways to do a sync:
> 1) sync the whole file
> 2) sync every time you do write.
>
> #1 is not affected by "write cache enabled"
> #2 is affected by "write cache enabled"
>
> In jvm's of 1.4.2 and higher Derby uses method 2 to sync the log on
> commit.
>
> I know there is a similar affect on c programs on various OS's but don't
> remember the details, basically we observed the same no syncing on some
> other db products when write cache enabled was set.
>
> My guess is that suse10 has some similar setting, maybe on file system
> rather than device? Here is a random link I came up with about
> write cache's and linux, I have no idea if it is valid but a google
> search on "disable write cache linux" gives lots of hits:
>
> http://www.jasonbrome.com/blog/archives/2004/04/03/writecache_enabled.html
>
>>
>> Mike Matrigali wrote:
>>
>>>is your app single threaded, if so group commit is not the issue.
>>>
>>>What is your OS? What is your JVM? Derby may use different syncing
>>>algorithms depending on JVM version.
>>>
>>>How did you measure synchronous write, ie. did you
>>>write a java program and execute against the same JVM as derby is
>>>running in?
>>>
>>>The disk that that contains the log directory is the one of interest.
>>>Each transaction is made up of a number of log records. From your
>>>description each transaction will have the following:
>>>begin log record
>>>insert log record for row into base table
>>>insert log record for row into primary key index
>>>commit log record
>>>
>>>
>>>
>>>yarono wrote:
>>>
>>>>Hello,
>>>>
>>>>I'm working on a simple db. Each record is composed of 3 long values.
The
>>>>first two are the primary key.
>>>>
>>>>I have to measure the performance of the insertions. Each insertion is
>>>>wrapped in a transaction, which is commited having only one insertion in
>>>>it.
>>>>I've measured both berkeley db performance and postgres and got about
>>>>110-115 insertions per second.
>>>>
>>>>Now in derby db (both in embeded mode and server mode) I get better
>>>>performance: about 250-300 insertions per second. This obviously results
>>>>from some kind of a group commit, although I get these results both when
>>>>auto-commiting or manual-commiting after each insertion.
>>>>
>>>>I've performed a simple test of synchronious writing 24 bytes (3 * 8
>>>>bytes)
>>>>to the disk. It measure 117 writes per second, and I believe this is the
>>>>upper bound of any db performance.
>>>>
>>>>So, I don't understand why I get such good performance, although I
commit
>>>>after each insertion.
>>>>
>>>>I examined the .dat files in both /log and /seg0 folders. None of them
>>>>increase in 24 bytes segments, but rathar bigger segments.
>>>>
>>>>So, my questions are:
>>>>1. Which log file in /log or /seg0 should I examine to analyze the
numebr
>>>>of
>>>>bytes written each write to disk?
>>>>2. How do I disable the group commit or whatever attribute that causes
>>>>this
>>>>communal write? how do I make each transaction be written on its own to
>>>>the
>>>>disk?
>>>>
>>>>Thanks in advance,
>>>>Yaron
>>>
>>>
>>>
>>
>
>
>
--
View this message in context:
http://www.nabble.com/Derby-db---need-to-disable-improved-performance-tf3796921.html#a10954929
Sent from the Apache Derby Users mailing list archive at Nabble.com.
