On Wed, Jan 15, 2014 at 08:03:28AM +1300, Gavin Flower wrote: > On 14/01/14 14:09, Dave Chinner wrote: > >On Mon, Jan 13, 2014 at 09:29:02PM +0000, Greg Stark wrote: > >>On Mon, Jan 13, 2014 at 9:12 PM, Andres Freund <and...@2ndquadrant.com> > >>wrote: > [...] > >>The more ambitious and interesting direction is to let Postgres tell > >>the kernel what it needs to know to manage everything. To do that we > >>would need the ability to control when pages are flushed out. This is > >>absolutely necessary to maintain consistency. Postgres would need to > >>be able to mark pages as unflushable until some point in time in the > >>future when the journal is flushed. We discussed various ways that > >>interface could work but it would be tricky to keep it low enough > >>overhead to be workable. > >IMO, the concept of allowing userspace to pin dirty page cache > >pages in memory is just asking for trouble. Apart from the obvious > >memory reclaim and OOM issues, some filesystems won't be able to > >move their journals forward until the data is flushed. i.e. ordered > >mode data writeback on ext3 will have all sorts of deadlock issues > >that result from pinning pages and then issuing fsync() on another > >file which will block waiting for the pinned pages to be flushed. > > > >Indeed, what happens if you do pin_dirty_pages(fd); .... fsync(fd);? > >If fsync() blocks because there are pinned pages, and there's no > >other thread to unpin them, then that code just deadlocked. If > >fsync() doesn't block and skips the pinned pages, then we haven't > >done an fsync() at all, and so violated the expectation that users > >have that after fsync() returns their data is safe on disk. And if > >we return an error to fsync(), then what the hell does the user do > >if it is some other application we don't know about that has pinned > >the pages? And if the kernel unpins them after some time, then we > >just violated the application's consistency guarantees.... > > > [...] > > What if Postgres could tell the kernel how strongly that it wanted > to hold on to the pages?
That doesn't get rid of the problems, it just makes it harder to diagnose them when they occur. :/ Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
