Re: [HACKERS] Synchronous replication, reading WAL for sending

2008-12-24 Thread Fujii Masao
Hi, On Wed, Dec 24, 2008 at 2:34 PM, Pavan Deolasee pavan.deola...@gmail.com wrote: On Tue, Dec 23, 2008 at 9:12 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: As the patch stands, whenever XLOG segment is switched in XLogInsert, we wait for the segment to be sent to the

Re: [HACKERS] Synchronous replication, reading WAL for sending

2008-12-24 Thread Pavan Deolasee
On Wed, Dec 24, 2008 at 1:50 PM, Fujii Masao masao.fu...@gmail.com wrote: And, I'm worried about the situation that the speed to generate xlog on the primary is higher than that to copy them to the standby. We might not be able to start xlog streaming forever. If that's the case, how do you

Re: [HACKERS] Synchronous replication, reading WAL for sending

2008-12-24 Thread Fujii Masao
Hi, On Wed, Dec 24, 2008 at 5:48 PM, Pavan Deolasee pavan.deola...@gmail.com wrote: On Wed, Dec 24, 2008 at 1:50 PM, Fujii Masao masao.fu...@gmail.com wrote: And, I'm worried about the situation that the speed to generate xlog on the primary is higher than that to copy them to the standby.

Re: [HACKERS] Sync Rep: First Thoughts on Code

2008-12-24 Thread Simon Riggs
On Wed, 2008-12-24 at 11:39 +0900, Fujii Masao wrote: We might ask why pg_start_backup() needs to perform checkpoint though, since you have remarked that is a problem also. The answer is that it doesn't really need to, we just need to be certain that archiving has been running since

Re: [HACKERS] Archiving control (a part of synch rep patches)

2008-12-24 Thread Fujii Masao
Hi, On Wed, Dec 24, 2008 at 4:39 PM, Fujii Masao masao.fu...@gmail.com wrote: (3) Revert the arhive status file of the specified file from .done to .ready. This would try to archive the corresponding xlog file again. The archive status of only skipped xlog file can be reverted. We resume

Re: [HACKERS] Synchronous replication, reading WAL for sending

2008-12-24 Thread Simon Riggs
On Wed, 2008-12-24 at 18:31 +0900, Fujii Masao wrote: Frankly, I myself have every doubt that on a relatively high load setup, the standby will not be able keep pace with the primary for two reasons: - Lack of read ahead of data blocks (Suzuki-san's work may help this) - Single

Re: [HACKERS] Synchronous replication, reading WAL for sending

2008-12-24 Thread Pavan Deolasee
On Wed, Dec 24, 2008 at 3:01 PM, Fujii Masao masao.fu...@gmail.com wrote: Good question. If streaming and copying are performed parallelly, such situation doesn't happen because the speed to generate xlog also depends on streaming. This is a price to pay. I think that the serial operations

Re: [HACKERS] Synchronous replication, reading WAL for sending

2008-12-24 Thread Pavan Deolasee
On Wed, Dec 24, 2008 at 3:40 PM, Simon Riggs si...@2ndquadrant.com wrote: If we want to speed up recovery more, I think we'll see the need for an additional process to do WAL CRC checks. Yeah, any such helper process along with other optimizations would certainly help. But I can't still

Re: [HACKERS] Archiving control (a part of synch rep patches)

2008-12-24 Thread Simon Riggs
On Wed, 2008-12-24 at 19:04 +0900, Fujii Masao wrote: Hi, On Wed, Dec 24, 2008 at 4:39 PM, Fujii Masao masao.fu...@gmail.com wrote: (3) Revert the arhive status file of the specified file from .done to .ready. This would try to archive the corresponding xlog file again. The archive

Re: [HACKERS] generic reloptions improvement

2008-12-24 Thread Peter Eisentraut
On Monday 22 December 2008 18:24:53 Alvaro Herrera wrote: Alvaro Herrera alvhe...@commandprompt.com wrote: Here's a patch for improving the general reloptions mechanism. What this patch does is add a table-based option parser. This allows adding new options very easily, and stops the

Re: [HACKERS] Synchronous replication, reading WAL for sending

2008-12-24 Thread Simon Riggs
On Wed, 2008-12-24 at 15:51 +0530, Pavan Deolasee wrote: On Wed, Dec 24, 2008 at 3:40 PM, Simon Riggs si...@2ndquadrant.com wrote: If we want to speed up recovery more, I think we'll see the need for an additional process to do WAL CRC checks. Yeah, any such helper process along

Re: [HACKERS] generic reloptions improvement

2008-12-24 Thread Alvaro Herrera
Peter Eisentraut wrote: I'm not sure how important this is, but if you are enumerating the access methods (RELOPT_KIND_BTREE, etc.), how will this work with user-defined access methods? It is important. I'm intending to have a new routine which would reserve a value at runtime. This value

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Simon Riggs
On Tue, 2008-12-23 at 23:59 -0500, Robert Treat wrote: On Friday 19 December 2008 19:36:42 Simon Riggs wrote: Perhaps we should listen to the people that have said they don't want queries cancelled, even if the alternative is inconsistent answers. That is easily possible yet is not

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Pavan Deolasee
On Wed, Dec 24, 2008 at 4:41 PM, Simon Riggs si...@2ndquadrant.com wrote: Greg and Heikki have highlighted in this thread some aspects of btree garbage collection that will increase the chance of queries being cancelled in various circumstances Even HOT-prune may lead to frequent query

Re: [HACKERS] Archiving control (a part of synch rep patches)

2008-12-24 Thread Fujii Masao
Hi, On Wed, Dec 24, 2008 at 7:26 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2008-12-24 at 19:04 +0900, Fujii Masao wrote: Hi, On Wed, Dec 24, 2008 at 4:39 PM, Fujii Masao masao.fu...@gmail.com wrote: (3) Revert the arhive status file of the specified file from .done to .ready.

Re: [HACKERS] Sync Rep: First Thoughts on Code

2008-12-24 Thread Fujii Masao
Hi, On Wed, Dec 24, 2008 at 6:57 PM, Simon Riggs si...@2ndquadrant.com wrote: Yes, OK. So I think it would only work when full_page_writes = on, and has been on since last checkpoint. So two changes: * We just need a boolean that starts at true every checkpoint and gets set to false anytime

[HACKERS] [idea] a copied relkind in pg_attribute

2008-12-24 Thread KaiGai Kohei
It is an idea to improve the implementation of SE-PostgreSQL. I need a copied relkind in pg_attribute, to help its decision making. When we access on pg_attribute via ALTER TABLE or DML statement directly, SE-PostgreSQL checks privilleges for the fetched tuples. If the relation pointed by its

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Simon Riggs
On Wed, 2008-12-24 at 16:48 +0530, Pavan Deolasee wrote: On Wed, Dec 24, 2008 at 4:41 PM, Simon Riggs si...@2ndquadrant.com wrote: Greg and Heikki have highlighted in this thread some aspects of btree garbage collection that will increase the chance of queries being cancelled in

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Pavan Deolasee
On Wed, Dec 24, 2008 at 5:26 PM, Simon Riggs si...@2ndquadrant.com wrote: The patch does go to some trouble to handle that case, as I'm sure you've seen. Are you saying that part of the patch is ineffective and should be removed, or? Umm.. are you talking about the wait mechanism ? That's

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Simon Riggs
On Wed, 2008-12-24 at 17:56 +0530, Pavan Deolasee wrote: On Wed, Dec 24, 2008 at 5:26 PM, Simon Riggs si...@2ndquadrant.com wrote: The patch does go to some trouble to handle that case, as I'm sure you've seen. Are you saying that part of the patch is ineffective and should be

Re: [HACKERS] Sync Rep: First Thoughts on Code

2008-12-24 Thread Fujii Masao
Hi, On Wed, Dec 24, 2008 at 7:58 PM, Fujii Masao masao.fu...@gmail.com wrote: Hi, On Wed, Dec 24, 2008 at 6:57 PM, Simon Riggs si...@2ndquadrant.com wrote: Yes, OK. So I think it would only work when full_page_writes = on, and has been on since last checkpoint. So two changes: * We just

Re: [HACKERS] Synchronous replication, reading WAL for sending

2008-12-24 Thread Mark Mielke
Fujii Masao wrote: - WALSender reads from WAL buffers and/or WAL files and sends the buffers to WALReceiver. In phase one, we may assume that WALSender can only read from WAL buffers and WAL files in pg_xlog directory. Later on, this can be improved so that WALSender can temporarily restore

Re: [HACKERS] Sync Rep: First Thoughts on Code

2008-12-24 Thread Simon Riggs
On Thu, 2008-12-25 at 00:10 +0900, Fujii Masao wrote: Hi, On Wed, Dec 24, 2008 at 7:58 PM, Fujii Masao masao.fu...@gmail.com wrote: Hi, On Wed, Dec 24, 2008 at 6:57 PM, Simon Riggs si...@2ndquadrant.com wrote: Yes, OK. So I think it would only work when full_page_writes = on, and

Re: [HACKERS] Sync Rep: Second thoughts

2008-12-24 Thread Emmanuel Cecchet
Hi Markus, I didn't have much reliability issues with ensemble, appia or spread, so far. Although, I admit I didn't ever run any of these in production. Performance is certainly an issue, yes. I may suggest another reading even though a bit dates, most of the results still apply:

Re: [Fwd: Re: [HACKERS] Transactions and temp tables]

2008-12-24 Thread Emmanuel Cecchet
Hi Heikki, The point of using temp tables was performance. Using regular tables in our case would hurt performance too much. Well if we cannot get a temporary fix in 8.4, we will maintain a separate patch to get that functionality just for temp tables that are created and dropped in the same

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Simon Riggs
On Wed, 2008-12-24 at 09:59 -0500, Robert Treat wrote: I think the uncertainty comes from peoples experience with typical replication use cases vs a lack of experience with this current implementation. Quite possibly. Publishing user feedback on this will be very important in making

Re: [HACKERS] [idea] a copied relkind in pg_attribute

2008-12-24 Thread Jaime Casanova
On Wed, Dec 24, 2008 at 6:50 AM, KaiGai Kohei kai...@kaigai.gr.jp wrote: The current implementation need to lookup RELOID system cache to identify the relkind of the relation, because pg_attribtue does not have any information about relkind. However, I also think it is not an ideal

Re: [HACKERS] generic reloptions improvement

2008-12-24 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Peter Eisentraut wrote: I'm not sure how important this is, but if you are enumerating the access methods (RELOPT_KIND_BTREE, etc.), how will this work with user-defined access methods? It is important. I'm intending to have a new routine

Re: [HACKERS] Window-functions patch handling of aggregates

2008-12-24 Thread Tom Lane
Gregory Stark st...@enterprisedb.com writes: Tom Lane t...@sss.pgh.pa.us writes: Unless we want to move the goalposts on what an aggregate is allowed to do internally, we're going to have to change this to re-aggregate repeatedly. Neither prospect is appetizing in the least. Does it

Re: [HACKERS] uuids on freebsd

2008-12-24 Thread David Lee Lambert
On Dec 17, 2:30 pm, Andrew Gierth and...@tao11.riddles.org.uk wrote: Has anyone ever managed to get uuid generation working on FreeBSD? [...] ([...] The only solution I could come up with was to knock off a quick uuid-freebsd module that uses the base system uuid functions rather than the

Re: [HACKERS] Sync Rep: First Thoughts on Code

2008-12-24 Thread Fujii Masao
Hi, I fixed some bugs. On Thu, Dec 25, 2008 at 12:31 AM, Simon Riggs si...@2ndquadrant.com wrote: Can we change to IMMEDIATE when it we need the checkpoint? Perhaps yes, though current patch doesn't care about it. I'm not sure if we really need the feature. Yes, as you say, I'd like to also

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Pavan Deolasee
On Wed, Dec 24, 2008 at 7:18 PM, Simon Riggs si...@2ndquadrant.com wrote: With respect, I was hoping you might look in the patch and see if you agree with the way it is handled. No need to remember. The whole latestRemovedXid concept is designed to do help. Well, that's common for all

Re: [HACKERS] incoherent view of serializable transactions

2008-12-24 Thread Kevin Grittner
Emmanuel Cecchet m...@frogthinker.org 12/22/08 11:42 PM If you want to know how to build SERIALIZABLE with a database that provides SI (Snapshot Isolation), read http://portal.acm.org/citation.cfm?doid=1376616.137669 The link didn't seem to work for me, but I think I found the article you

[HACKERS] merry christmas folks

2008-12-24 Thread Grzegorz Jaskiewicz
WITH RECURSIVE tree(b, l, lv) AS ( ( WITH RECURSIVE t(b, l) AS ( select b/11.6, 0 AS l from (select generate_series(0,30)::float8 b union all select generate_series(30,0, -1)::float8 b) ziew UNION ALL select (b*1.06), l+1 FROM t WHERE l 3 ) select b, l,

Re: [HACKERS] DTrace probes patch

2008-12-24 Thread Bruce Momjian
Thanks, applied. --- Robert Lor wrote: Tom Lane wrote: Robert Lor robert@sun.com writes: Tom Lane wrote: I agree. If the probe is meant to track only *some* WAL writes then it needs to be named

Re: [HACKERS] incoherent view of serializable transactions

2008-12-24 Thread Emmanuel Cecchet
Hi Kevin, The link didn't seem to work for me, but I think I found the article you meant: Serializable Isolation for Snapshot Databases by Michael J. Cahill, et al An interesting work. If nothing else, it will help flesh out the documentation of anomalies. If the PostgreSQL community ever

Re: [HACKERS] parallel restore vs. windows

2008-12-24 Thread Jaime Casanova
On Sun, Dec 14, 2008 at 12:13 PM, Andrew Dunstan and...@dunslane.net wrote: This version completes properly on Windows with the regression database. actually, this one doesn't apply cleanly on head -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de

Re: [HACKERS] parallel restore vs. windows

2008-12-24 Thread Andrew Dunstan
Jaime Casanova wrote: On Sun, Dec 14, 2008 at 12:13 PM, Andrew Dunstan and...@dunslane.net wrote: This version completes properly on Windows with the regression database. actually, this one doesn't apply cleanly on head I will have a new patch a day or two after Christmas,

Re: [HACKERS] incoherent view of serializable transactions

2008-12-24 Thread Ron Mayer
Robert Haas wrote: ... serializable transaction ... If we were to construct a database that had one giant lock for the entire database so that only a single query could execute at one time, transactions would be serializable (because they'd in fact be serialized). However, performance would

Re: [HACKERS] incoherent view of serializable transactions

2008-12-24 Thread Robert Haas
On Dec 24, 2008, at 6:46 PM, Ron Mayer rm...@cheapcomplexdevices.com wrote: Robert Haas wrote: ... serializable transaction ... If we were to construct a database that had one giant lock for the entire database so that only a single query could execute at one time, transactions would be

Re: [HACKERS] [idea] a copied relkind in pg_attribute

2008-12-24 Thread KaiGai Kohei
Jaime Casanova wrote: On Wed, Dec 24, 2008 at 6:50 AM, KaiGai Kohei kai...@kaigai.gr.jp wrote: The current implementation need to lookup RELOID system cache to identify the relkind of the relation, because pg_attribtue does not have any information about relkind. However, I also think it is not

Re: [HACKERS] Window-functions patch handling of aggregates

2008-12-24 Thread Hitoshi Harada
2008/12/25 Tom Lane t...@sss.pgh.pa.us: Gregory Stark st...@enterprisedb.com writes: Tom Lane t...@sss.pgh.pa.us writes: Unless we want to move the goalposts on what an aggregate is allowed to do internally, we're going to have to change this to re-aggregate repeatedly. Neither prospect is

[HACKERS] hot standby on mingw

2008-12-24 Thread Jaime Casanova
Hi, i'm getting this error when triyng to build the hot standby (v5 from 20081218 obtained in the wiki) on mingw make[4]: Entering directory `/home/Administrador/pgsql/src/backend/access/heap' gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels