Option to fsync files

2008-11-19 Thread Mark Miller
What was the reason not to make this toggle-able? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]

Re: Option to fsync files

2008-11-19 Thread Michael McCandless
Well... because it's quite dangerous to turn off. A simple way to disable it entirely is to subclass FSDir and override sync() to be a noop. Mike Mark Miller wrote: What was the reason not to make this toggle-able? - To

Re: Option to fsync files

2008-11-19 Thread Mark Miller
It is dangerous, but probably not any more dangerous than using a consumer hard drive that lies to sync (don't know the numbers, but I have to assume some/many are doing this with Lucene - in which case you pay perf for a false sense of security). Not a real suggestion at this point though. Ju

Re: Option to fsync files

2008-11-19 Thread Michael McCandless
Mark Miller wrote: It is dangerous, but probably not any more dangerous than using a consumer hard drive that lies to sync (don't know the numbers, but I have to assume some/many are doing this with Lucene - in which case you pay perf for a false sense of security). Well, if the consumer

Re: Option to fsync files

2008-11-19 Thread Mark Miller
Michael McCandless wrote: Mark Miller wrote: It is dangerous, but probably not any more dangerous than using a consumer hard drive that lies to sync (don't know the numbers, but I have to assume some/many are doing this with Lucene - in which case you pay perf for a false sense of security).

Re: Option to fsync files

2008-11-19 Thread robert engels
I would really like to see some PROOF of these drives "lying". If that were the case, no database system would ever be reliable on these drives ! And data corruption would be happening all over the place ! On Nov 19, 2008, at 10:56 AM, Mark Miller wrote: Michael McCandless wrote: Mark

Re: Option to fsync files

2008-11-19 Thread Mark Miller
Okay, I'll admit I am trusting some else testing this. One of the java database implementations (hsql or something?) talks about it and tests it it to be the case also with derby and other transaction systems. Corruption didn't appear that hard for him to lure out at all. Also, if you google sl

Re: Option to fsync files

2008-11-19 Thread Mark Miller
/A look at the slashdot article seems to indicate the OS may have been at fault in his case (an update he posted in response to some slashdot flames). Thats interesting. Wasnt there the last time this topic came up. Can't remember what java db had the other info...but looking... /

Re: Option to fsync files

2008-11-19 Thread robert engels
There is an option on some hard drives that is off by default (under windows) called lazy writing. You would need to go into the driver and turn this on. Only very specialized systems would ever do this. On Nov 19, 2008, at 11:13 AM, Mark Miller wrote: Okay, I'll admit I am trusting some el

Re: Option to fsync files

2008-11-19 Thread Jason Rutherglen
http://www.h2database.com/html/advanced.html#durability_problems http://hardware.slashdot.org/article.pl?sid=05/05/13/0529252 On Wed, Nov 19, 2008 at 9:23 AM, Mark Miller <[EMAIL PROTECTED]> wrote: > /A look at the slashdot article seems to indicate the OS may have been at > fault in his case (an

Re: Option to fsync files

2008-11-19 Thread Mark Miller
Thanks Jason...had just found h2 again myself - that guy seems to say even with write caching disabled on the OS, its still a problem (though probably less of a problem than without sync? Jason Rutherglen wrote: http://www.h2database.com/html/advanced.html#durability_problems http://hardware.s

Re: Option to fsync files

2008-11-19 Thread Mark Miller
Ummm...well durability isnt really corruption is it. I guess I'm convert there. Mark Miller wrote: Thanks Jason...had just found h2 again myself - that guy seems to say even with write caching disabled on the OS, its still a problem (though probably less of a problem than without sync? Jason

Re: Option to fsync files

2008-11-19 Thread Mark Miller
robert engels wrote: There is an option on some hard drives that is off by default (under windows) called lazy writing. You would need to go into the driver and turn this on. Only very specialized systems would ever do this. I see evidence that older mac and linux system do do this by default.

Re: Option to fsync files

2008-11-19 Thread robert engels
The "utility" referenced no longer exists... and its no wonder. If is most likely that the tester did not have the drives configured properly. In almost all cases, if the drive did this, you could not run a database system with any resiliency. They would also have problems with shutdown -

Re: Option to fsync files

2008-11-19 Thread robert engels
I don't believe it - unless the older drives had no cache, in which case it wouldn't matter. It is also doubtful at the OS level, as system integrity would be hopelessly compromised... On Nov 19, 2008, at 12:11 PM, Mark Miller wrote: robert engels wrote: There is an option on some hard dr

Re: Option to fsync files

2008-11-19 Thread robert engels
It is not just database by the way, any journaling file system would be pointless... On Nov 19, 2008, at 12:55 PM, robert engels wrote: The "utility" referenced no longer exists... and its no wonder. If is most likely that the tester did not have the drives configured properly. In almost