On Thu, Oct 27, 2011 at 7:48 PM, Magnus Hagander <mag...@hagander.net> wrote: > On Thu, Oct 27, 2011 at 12:29, Fujii Masao <masao.fu...@gmail.com> wrote: >> On Thu, Oct 27, 2011 at 6:25 PM, Fujii Masao <masao.fu...@gmail.com> wrote: >>> On Thu, Oct 27, 2011 at 5:18 PM, Magnus Hagander <mag...@hagander.net> >>> wrote: >>>> Not sure I follow. When we arrive at PQgetCopyData() there should be >>>> nothing buffered, and if the end of stream happens there it returns >>>> -1, and we exit, no? So where is the data that's lost? >>>> >>>> I do realize we don't actually fsync() and close() in this case - is >>>> that what you are referring to? But the data should already have been >>>> write()d, so it should still be there, no? >>> >>> Oh, right. Hmm.. xlogdump might be the cause. >>> >>> Though I've not read the code of xlogdump, I wonder if it gives up >>> outputting the contents of WAL file when it finds a partial WAL page... >>> This strikes me that recovery code has the same problem. No? >>> IOW, when a partial WAL page is found during recovery, I'm afraid >>> that page would not be replayed though it contains valid data. >> >> My concern was right. When I performed a recovery using the streamed >> WAL files, the loss of data happened. A partial WAL page was not replayed. >> >> To avoid this problem, I think that we should change pg_receivexlog so >> that it writes WAL data *by the block*, or creates, like walreceiver, WAL >> file >> before writing any data. Otherwise, though pg_receivexlog streams WAL >> data in realtime, the latest WAL data might not be available for recovery. > > Ah, so you were recovering data from the last, partial, file? Not from > a completed file?
Yes. I copied all streamed WAL files to pg_xlog directory and started recovery. > I'm rewriting the handling of partial files per the other thread > started by Heikki. The idea is that there will be an actual .partial > file in there when pg_receivexlog has ended, and you have to deal with > it manually. The typical way would be to pad it with zeroes to the > end. Doing such padding would solve this recovery issue, correct? Yes. But that sounds unuserfriendly. Padding the WAL file manually is easy-to-do for a user? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers