Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-11 Thread Magnus Hagander
On Wed, Dec 8, 2010 at 15:40, Tom Lane wrote: > Magnus Hagander writes: >> On Wed, Dec 8, 2010 at 02:07, Tom Lane wrote: >>> [ win32.h says ] >>> #define fsync(fd) _commit(fd) > >>> What this means is that switching to a simple preference order >>> "fdatasync, then fsync" will result in choosing

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-09 Thread Greg Smith
Josh Berkus wrote: Did you rerun test_sync with O_DIRECT entabled, using my patch? The figures you had from test_fsync earlier were without O_DIRECT. No--I was just focused on testing the changes that had already been committed. The use of O_DIRECT in the server but not test_fsync could

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-09 Thread Josh Berkus
Greg, > This is interesting, because test_fsync consistently reported a rate of > about half this when using open_datasync instead of the equal > performance I'm getting from the database. I'll see if I can reproduce > that further, but it's no reason to be concerned about the change that's > bee

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-09 Thread Greg Smith
Since any Windows refactoring has been postponed for now (I'll get back to performance checks on that platform later), during my testing time this week instead I did a round of pre-release review of the change Tom has now committed. All looks good to me, including the docs changes. I confirme

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-08 Thread Tom Lane
Given my concerns around exactly what is going on in the Windows code, I'm now afraid to mess with an all-platforms change to fdatasync as the preferred default; if we do that it should probably just be in HEAD not the back branches. So I've come around to the idea that Marti's proposal of a PLATF

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-08 Thread Tom Lane
Magnus Hagander writes: > On Wed, Dec 8, 2010 at 02:07, Tom Lane wrote: >> [ win32.h says ] >> #define fsync(fd) _commit(fd) >> What this means is that switching to a simple preference order >> "fdatasync, then fsync" will result in choosing fsync on Windows (since >> it hasn't got fdatasync), m

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Magnus Hagander
On Wed, Dec 8, 2010 at 02:07, Tom Lane wrote: > Josh Berkus writes: I am unclear as to the reason why there is a test for HAVE_FSYNC_WRITETHROUGH_ONLY in pg_fsync().  Perhaps that is also leftover from a previous vision of how this all works?  Or does an fsync() call actually

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Greg Smith
Tom Lane wrote: So actually, there is no difference between selecting fsync and fsync_writethrough on Windows, this comment and the SGML documentation to the contrary. Both settings result in invoking _commit() and presumably are safe. One wonders why we bothered to invent a separate fsync_writ

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Tom Lane
Josh Berkus writes: >>> I am unclear as to the reason why there is a test for >>> HAVE_FSYNC_WRITETHROUGH_ONLY in pg_fsync(). Perhaps that is also >>> leftover from a previous vision of how this all works? Or does an >>> fsync() call actually fail on Windows? >> No, fsync responds fine. It jus

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Andrew Dunstan
On 12/07/2010 07:32 PM, Joshua D. Drake wrote: On Tue, 2010-12-07 at 19:00 -0500, Chris Browne wrote: x...@thebuild.com (Christophe Pettus) writes: On Dec 7, 2010, at 2:43 PM, Josh Berkus wrote: Because nobody sane uses OSX on the server? The XServe running 10.5 server and 9.0.1 at the othe

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Joshua D. Drake
On Tue, 2010-12-07 at 19:00 -0500, Chris Browne wrote: > x...@thebuild.com (Christophe Pettus) writes: > > On Dec 7, 2010, at 2:43 PM, Josh Berkus wrote: > >> Because nobody sane uses OSX on the server? > > > > The XServe running 10.5 server and 9.0.1 at the other end of the > > office takes your r

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Chris Browne
x...@thebuild.com (Christophe Pettus) writes: > On Dec 7, 2010, at 2:43 PM, Josh Berkus wrote: >> Because nobody sane uses OSX on the server? > > The XServe running 10.5 server and 9.0.1 at the other end of the > office takes your remark personally. :) I'd heard that Apple had cancelled XServe. [

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Tom Lane
Josh Berkus writes: >> Right, which is also an accurate description of its behavior on OS X, >> as well as Linux (if you didn't change hdparm settings). So the real >> question here is what's the point of treating Windows differently. > So, sounds like we should continue treating fsync_writethro

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Josh Berkus
> Right, which is also an accurate description of its behavior on OS X, > as well as Linux (if you didn't change hdparm settings). So the real > question here is what's the point of treating Windows differently. So, sounds like we should continue treating fsync_writethrough the same as we have b

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Tom Lane
Josh Berkus writes: >> I am unclear as to the reason why there is a test for >> HAVE_FSYNC_WRITETHROUGH_ONLY in pg_fsync(). Perhaps that is also >> leftover from a previous vision of how this all works? Or does an >> fsync() call actually fail on Windows? > No, fsync responds fine. It just don

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Christophe Pettus
On Dec 7, 2010, at 2:43 PM, Josh Berkus wrote: > Because nobody sane uses OSX on the server? The XServe running 10.5 server and 9.0.1 at the other end of the office takes your remark personally. :) -- -- Christophe Pettus x...@thebuild.com -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Josh Berkus
> I am unclear as to the reason why there is a test for > HAVE_FSYNC_WRITETHROUGH_ONLY in pg_fsync(). Perhaps that is also > leftover from a previous vision of how this all works? Or does an > fsync() call actually fail on Windows? No, fsync responds fine. It just don't actually sync to disk.

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Tom Lane
"Joshua D. Drake" writes: > On Tue, 2010-12-07 at 18:11 -0500, Tom Lane wrote: >> Josh Berkus writes: >>> Because nobody sane uses OSX on the server? >> Some of us would make the same remark about Windows. But we go out of >> our way to provide a safe default on that platform anyhow. > Not to

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Tom Lane
I wrote: > Some of us would make the same remark about Windows. But we go out of > our way to provide a safe default on that platform anyhow. Oh, wait: that's not the case at all. As of recent releases, we support open_datasync on Windows, and that's the default despite being unsafe. You have to

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Joshua D. Drake
On Tue, 2010-12-07 at 18:11 -0500, Tom Lane wrote: > Josh Berkus writes: > > On 12/7/10 2:28 PM, Tom Lane wrote: > >> Another point here is that it's not clear why we're selecting a > >> known-to-be-insecure default on OS X (where in fact all methods except > >> fsync_writethrough fail to push dat

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Andrew Dunstan
On 12/07/2010 06:11 PM, Tom Lane wrote: Josh Berkus writes: On 12/7/10 2:28 PM, Tom Lane wrote: Another point here is that it's not clear why we're selecting a known-to-be-insecure default on OS X (where in fact all methods except fsync_writethrough fail to push data to disk). We've been ar

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Tom Lane
Josh Berkus writes: > On 12/7/10 2:28 PM, Tom Lane wrote: >> Another point here is that it's not clear why we're selecting a >> known-to-be-insecure default on OS X (where in fact all methods except >> fsync_writethrough fail to push data to disk). We've been around on >> that before, of course,

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Josh Berkus
On 12/7/10 2:28 PM, Tom Lane wrote: > Another point here is that it's not clear why we're selecting a > known-to-be-insecure default on OS X (where in fact all methods except > fsync_writethrough fail to push data to disk). We've been around on > that before, of course, and maybe now is not the ti

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Tom Lane
Josh Berkus writes: > I agree for the backpatch that we should just swap to fdatasync as > default, and should not attempt to add the extra options. I noticed while updating the documentation for this that the current documentation is a flat-out lie. It claims that the preference order for wal_

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Josh Berkus
On 12/07/2010 08:24 AM, Tom Lane wrote: Dropping open_datasync as the first-choice default is something we have to back-patch, but I'm less sure about it being a good idea to back-patch the rearrangement of O_DIRECT management. Somebody who'd explicitly specified open_sync or open_datasync as wa

Re: [HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Andres Freund
On Tuesday 07 December 2010 17:24:14 Tom Lane wrote: > After reviewing the two ongoing threads about fixing the wal_sync_method > fiasco, I think there is general agreement on these two points: > > 1. open_datasync shouldn't be the default choice > 2. O_DIRECT shouldn't be forcibly bundled in with

[HACKERS] Final(?) proposal for wal_sync_method changes

2010-12-07 Thread Tom Lane
After reviewing the two ongoing threads about fixing the wal_sync_method fiasco, I think there is general agreement on these two points: 1. open_datasync shouldn't be the default choice 2. O_DIRECT shouldn't be forcibly bundled in with O_DSYNC/O_SYNC What I suggest we do about the latter is to in