[HACKERS] proposal 9.4 plpgsql: allows access to call stack from GET DIAGNOSTICS statement

2013-02-02 Thread Pavel Stehule
Hello I propose enhancing GET DIAGNOSTICS statement about new field PG_CONTEXT. It is similar to GET STACKED DIAGNOSTICS' PG_EXCEPTION_CONTEXT. Motivation for this proposal is possibility to get call stack for debugging without raising exception. This code is based on cleaned code from Orafce,

[HACKERS] Using indexes for partial index builds

2013-02-02 Thread Paul Norman
Hello, After a discussion on IRC in #postgresql, I had a feature suggestion and it was suggested I write it up here. I have a large (200GB, 1.7b rows) table with a number of columns, but the two of interest here are a hstore column, tags and a postgis geometry column, geom. There is a GIN index

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Pavel Stehule
Hello 2013/2/1 Tom Lane t...@sss.pgh.pa.us: Pavel Stehule pavel.steh...@gmail.com writes: here is patch related to your proposal I looked at this a bit. It's getting there, though I still don't trust the places where you've chosen to clear the prefix setting. (Looking at it, I'm now not

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 1 February 2013 23:56, Tom Lane t...@sss.pgh.pa.us wrote: Well, if we were tracking the latest value in shared memory, we could certainly clamp to that to ensure it didn't go backwards. The problem is where to find storage for a per-DB value. Adding new data columns to catalogs in

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 1 February 2013 23:56, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Feb 1, 2013 at 2:35 PM, Tom Lane t...@sss.pgh.pa.us wrote: In any case, I no longer have much faith in the idea that letting GetOldestXmin go backwards is really safe. That is

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Shigeru Hanada
On Sat, Feb 2, 2013 at 7:30 PM, Pavel Stehule pavel.steh...@gmail.com wrote: possible variants a) don't store NULL values - and remove existing variable when NULL be assigned - it is probably best, but should be confusing for users b) implement flag IS NULL - for variables c) use nullPrint

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-01 15:09:34 -0800, Jeff Janes wrote: On Fri, Feb 1, 2013 at 2:34 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-02-01 14:05:46 -0800, Jeff Janes wrote: As far as I can tell this bug kicks in when your cluster gets to be older than freeze_min_age, and then lasts forever

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-01 16:59:52 -0500, Robert Haas wrote: I don't think I really understand the origin of the formula, so perhaps if someone would try to characterize why it seems to behave reasonably that would be helpful (at least to me). f(deadtuples, relpages, age) = deadtuples/relpages + e

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Andres Freund
On 2013-02-01 19:24:02 -0500, Tom Lane wrote: * There are also replication-related effects: a walsender * process can set its xmin based on transactions that are no longer running * in the master but are still being replayed on the standby, thus possibly * making the GetOldestXmin reading

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Pavel Stehule
2013/2/2 Shigeru Hanada shigeru.han...@gmail.com: On Sat, Feb 2, 2013 at 7:30 PM, Pavel Stehule pavel.steh...@gmail.com wrote: possible variants a) don't store NULL values - and remove existing variable when NULL be assigned - it is probably best, but should be confusing for users b)

Re: [HACKERS] COPY FREEZE has no warning

2013-02-02 Thread Noah Misch
On Fri, Feb 01, 2013 at 12:57:18PM -0500, Bruce Momjian wrote: On Tue, Jan 29, 2013 at 08:34:24PM -0500, Noah Misch wrote: On Fri, Jan 25, 2013 at 11:28:58PM -0500, Bruce Momjian wrote: BEGIN; TRUNCATE vistest; SAVEPOINT s1; COPY vistest FROM stdin CSV FREEZE; ERROR:

Re: [HACKERS] COPY FREEZE has no warning

2013-02-02 Thread Bruce Momjian
On Sat, Feb 2, 2013 at 09:51:13AM -0500, Noah Misch wrote: OK, so, should we change the error message: cannot perform FREEZE because of transaction activity after table creation or truncation to cannot perform FREEZE because the table was not created or

Re: [HACKERS] Turning auto-analyze off (was Re: [GENERAL] Unusually high IO for autovacuum worker)

2013-02-02 Thread Vlad Bailescu
On Fri, Feb 1, 2013 at 5:54 PM, Pavan Deolasee pavan.deola...@gmail.comwrote: There is another problem that I noticed while looking at this case. The analyze took close to 500sec on a fairly good hardware (40GB RAM, 10K rpm disks on RAID10) because many large child tables were scanned at

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Robert Haas
On Fri, Feb 1, 2013 at 7:24 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Having said that, I agree that a fix in GetOldestXmin() would be nice if we could find one, but since the comment describes at least three different ways the value can move backwards,

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Robert Haas
On Fri, Feb 1, 2013 at 6:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: That is admittedly kind of weird behavior, but I think one could equally blame this on CLUSTER. This is hardly the first time we've had to patch CLUSTER's handling of TOAST tables (cf commits

Re: [HACKERS] Suggestion: Issue warning when calling SET TRANSACTION outside transaction block

2013-02-02 Thread Robert Haas
On Fri, Feb 1, 2013 at 12:04 AM, Amit Kapila amit.kap...@huawei.com wrote: I think user should be aware of effect before using SET commands, as these are used at various levels (TRANSACTION, SESSION, ...). Ideally, sure. But these kinds of mistakes are easy to make. That's why LOCK and

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Robert Haas
On Fri, Feb 1, 2013 at 6:09 PM, Jeff Janes jeff.ja...@gmail.com wrote: As an aside, it does seem like log_autovacuum_min_duration=0 should log whether a scan_all was done, and if so what relfrozenxid got set to. That would be nifty. [1] I don't know why it is that a scan_all vacuum with a

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Robert Haas
On Sat, Feb 2, 2013 at 8:41 AM, Andres Freund and...@2ndquadrant.com wrote: - It's probably important to have a formula where we can be sure that the wrap-around term will eventually dominate the dead-tuple term, with enough time to spare to make sure nothing really bad happens; on the other

Re: [HACKERS] COPY FREEZE has no warning

2013-02-02 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: Well, so you are saying that there really isn't any use-visible logic for those messages to be different, No, and in fact the whole block of code is badly written because it conflates two unrelated tests. I guess somebody was trying to save a couple of

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Tom Lane
Shigeru Hanada shigeru.han...@gmail.com writes: On Sat, Feb 2, 2013 at 7:30 PM, Pavel Stehule pavel.steh...@gmail.com wrote: possible variants a) don't store NULL values - and remove existing variable when NULL be assigned - it is probably best, but should be confusing for users b)

Re: [HACKERS] COPY FREEZE has no warning

2013-02-02 Thread Bruce Momjian
On Sat, Feb 2, 2013 at 12:09:05PM -0500, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: Well, so you are saying that there really isn't any use-visible logic for those messages to be different, No, and in fact the whole block of code is badly written because it conflates two

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 2 February 2013 14:24, Andres Freund and...@2ndquadrant.com wrote: b) We don't assign the xmin early enough, we only set it when the first feedback message arrives, but we should set it when walsender starts streaming. That's easy to fix. c) After a disconnect the feedback message will

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 2 February 2013 00:24, Tom Lane t...@sss.pgh.pa.us wrote: * The return value is also adjusted with vacuum_defer_cleanup_age, so * increasing that setting on the fly is another easy way to make * GetOldestXmin() move backwards, with no consequences for data integrity. And as for that,

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 2 February 2013 00:24, Tom Lane t...@sss.pgh.pa.us wrote: * if allDbs is FALSE and there are no transactions running in the current * database, GetOldestXmin() returns latestCompletedXid. If a transaction * begins after that, its xmin will include in-progress transactions in other *

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Andres Freund
On 2013-02-02 18:32:44 +, Simon Riggs wrote: On 2 February 2013 14:24, Andres Freund and...@2ndquadrant.com wrote: b) We don't assign the xmin early enough, we only set it when the first feedback message arrives, but we should set it when walsender starts streaming. That's easy to

Re: [HACKERS] Cascading replication: should we detect/prevent cycles?

2013-02-02 Thread Robert Haas
On Thu, Jan 31, 2013 at 9:48 PM, Josh Berkus j...@agliodbs.com wrote: On 02/01/2013 12:01 PM, Josh Berkus wrote: If we're going to start installing safeguards against doing stupid things, there's a long list of scenarios that happen far more regularly than this ever will and cause far more

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-02 11:25:01 -0500, Robert Haas wrote: On Sat, Feb 2, 2013 at 8:41 AM, Andres Freund and...@2ndquadrant.com wrote: - It's probably important to have a formula where we can be sure that the wrap-around term will eventually dominate the dead-tuple term, with enough time to spare to

Re: [HACKERS] COPY FREEZE has no warning

2013-02-02 Thread Noah Misch
On Sat, Feb 02, 2013 at 10:12:54AM -0500, Bruce Momjian wrote: On Sat, Feb 2, 2013 at 09:51:13AM -0500, Noah Misch wrote: Let's touch on the exception in passing by using the phrase last truncated, giving this wording for both the second and the third COPY FREEZE error sites:

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Robert Haas
On Sat, Feb 2, 2013 at 1:49 PM, Andres Freund and...@2ndquadrant.com wrote: You're right, this doesn't work superbly well, especially for insert-only tables... But imo the place to fix it is not the priorization logic but relation_needs_vacanalyze, since fixing it in priorization won't prevent

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 2 February 2013 18:38, Andres Freund and...@2ndquadrant.com wrote: On 2013-02-02 18:32:44 +, Simon Riggs wrote: On 2 February 2013 14:24, Andres Freund and...@2ndquadrant.com wrote: b) We don't assign the xmin early enough, we only set it when the first feedback message arrives, but

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Pavel Stehule
Hello 2013/2/2 Tom Lane t...@sss.pgh.pa.us: Shigeru Hanada shigeru.han...@gmail.com writes: On Sat, Feb 2, 2013 at 7:30 PM, Pavel Stehule pavel.steh...@gmail.com wrote: possible variants a) don't store NULL values - and remove existing variable when NULL be assigned - it is probably

Re: [HACKERS] PL/Python result object str handler

2013-02-02 Thread Steve Singer
On 13-01-07 09:58 PM, Peter Eisentraut wrote: By implementing a str handler for the result object, it now prints something like PLyResult status=5 nrows=2 rows=[{'foo': 1, 'bar': '11'}, {'foo': 2, 'bar': '22'}] Patch attached for review. Here is a review: This patch adds a function that

Re: [HACKERS] Setting visibility map in VACUUM's second phase

2013-02-02 Thread Jeff Janes
On Sat, Jan 26, 2013 at 11:25 PM, Pavan Deolasee pavan.deola...@gmail.com wrote: On Thu, Jan 24, 2013 at 9:31 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Thu, Jan 24, 2013 at 1:28 AM, Pavan Deolasee pavan.deola...@gmail.com wrote: Good idea. Even though the cost of pinning/unpinning may

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-02 14:54:10 -0500, Robert Haas wrote: On Sat, Feb 2, 2013 at 1:49 PM, Andres Freund and...@2ndquadrant.com wrote: I think scheduling a table for a partial vacuum every min_freeze * 2 xids, even if its insert only, would go a long way of reducing the impact of full-table vacuums.

Re: [HACKERS] logical changeset generation v4 - Heikki's thoughts about the patch state

2013-02-02 Thread Andres Freund
On 2013-01-28 16:55:52 -0500, Steve Singer wrote: If your using non-surragate /natural primary keys this tends to come up occasionally due to data-entry errors or renames. I'm looking at this from the point of view of what do I need to use this as a source for a production replication system

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Tom Lane
Pavel Stehule pavel.steh...@gmail.com writes: 2013/2/2 Tom Lane t...@sss.pgh.pa.us: Shigeru Hanada shigeru.han...@gmail.com writes: +1 for a). If users want to determine whether the result was NULL, or want to use substitute string for NULL result, they can use coalesce in SELECT clause.

Re: PATCH: Split stats file per database WAS: [HACKERS] autovacuum stress-testing our system

2013-02-02 Thread Jeff Janes
On Sat, Jan 5, 2013 at 8:03 PM, Tomas Vondra t...@fuzzy.cz wrote: On 3.1.2013 20:33, Magnus Hagander wrote: Yeah, +1 for a separate directory not in global. OK, I moved the files from global/stat to stat. This has a warning: pgstat.c:5132: warning: 'pgstat_write_statsfile_needed' was used

Re: PATCH: Split stats file per database WAS: [HACKERS] autovacuum stress-testing our system

2013-02-02 Thread Tomáš Vondra
Ok, thanks for the info. I'll look into that and I'll also post info from some of our production systems (we've deployed a 9.1-backpatched version about 2 weeks ago). T. Původní zpráva Od: Jeff Janes jeff.ja...@gmail.com Datum: Komu: Tomas Vondra t...@fuzzy.cz Kopie:

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-01 15:09:34 -0800, Jeff Janes wrote: As an aside, it does seem like log_autovacuum_min_duration=0 should log whether a scan_all was done, and if so what relfrozenxid got set to. But looking at where the log message is generated, I don't know where to retrieve that info. What about

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-03 02:40:04 +0100, Andres Freund wrote: On 2013-02-01 15:09:34 -0800, Jeff Janes wrote: As an aside, it does seem like log_autovacuum_min_duration=0 should log whether a scan_all was done, and if so what relfrozenxid got set to. But looking at where the log message is generated,

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Christopher Browne
On Sat, Feb 2, 2013 at 2:54 PM, Robert Haas robertmh...@gmail.com wrote: On Sat, Feb 2, 2013 at 1:49 PM, Andres Freund and...@2ndquadrant.com wrote: You're right, this doesn't work superbly well, especially for insert-only tables... But imo the place to fix it is not the priorization logic but

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Gavin Flower
On 03/02/13 15:08, Christopher Browne wrote: On Sat, Feb 2, 2013 at 2:54 PM, Robert Haas robertmh...@gmail.com wrote: On Sat, Feb 2, 2013 at 1:49 PM, Andres Freund and...@2ndquadrant.com wrote: You're right, this doesn't work superbly well, especially for insert-only tables... But imo the

Re: [HACKERS] [PATCH] pg_isready (was: [WIP] pg_ping utility)

2013-02-02 Thread Phil Sorber
On Tue, Jan 29, 2013 at 11:43 AM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jan 29, 2013 at 3:12 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Phil Sorber escribió: On Mon, Jan 28, 2013 at 11:20 AM, Fujii Masao masao.fu...@gmail.com wrote: Maybe. But I'm not inclined to add new

[HACKERS] [PATCH] Add PQconninfoParseParams and PQconninfodefaultsMerge to libpq

2013-02-02 Thread Phil Sorber
This patch came up from discussion about pg_isready. PQconninfoParseParams is similar to PQconninfoParse but takes two arrays like PQconnectdbParams. It essentially exposes conninfo_array_parse(). PQconninfodefaultsMerge essentially exposes conninfo_add_defaults(). It allows you to pass a

Re: [HACKERS] PL/Python result object str handler

2013-02-02 Thread Peter Eisentraut
On Sat, 2013-02-02 at 15:43 -0500, Steve Singer wrote: I've looked through the code and everything looks fine. The patch includes no documentation. Adding a few lines to the Utility Functions section of the plpython documentation so people know about this feature would be good. Added

Re: [HACKERS] Setting visibility map in VACUUM's second phase

2013-02-02 Thread Pavan Deolasee
On Sun, Feb 3, 2013 at 2:31 AM, Jeff Janes jeff.ja...@gmail.com wrote: Hi Pavan, I get this warning: vacuumlazy.c:890: warning: passing argument 6 of 'lazy_vacuum_page' makes pointer from integer without a cast and make check then fails. I've added '' to that line, and it now passes make

Re: [HACKERS] [PATCH] Add PQconninfoParseParams and PQconninfodefaultsMerge to libpq

2013-02-02 Thread Magnus Hagander
On Feb 3, 2013 4:16 AM, Phil Sorber p...@omniti.com wrote: This patch came up from discussion about pg_isready. PQconninfoParseParams is similar to PQconninfoParse but takes two arrays like PQconnectdbParams. It essentially exposes conninfo_array_parse(). PQconninfodefaultsMerge

Re: [HACKERS] Suggestion: Issue warning when calling SET TRANSACTION outside transaction block

2013-02-02 Thread Amit kapila
On Saturday, February 02, 2013 9:08 PM Robert Haas wrote: On Fri, Feb 1, 2013 at 12:04 AM, Amit Kapila amit.kap...@huawei.com wrote: I think user should be aware of effect before using SET commands, as these are used at various levels (TRANSACTION, SESSION, ...). Ideally, sure. But these

Re: [HACKERS] [PATCH] Add PQconninfoParseParams and PQconninfodefaultsMerge to libpq

2013-02-02 Thread Phil Sorber
On Sun, Feb 3, 2013 at 1:37 AM, Magnus Hagander mag...@hagander.net wrote: On Feb 3, 2013 4:16 AM, Phil Sorber p...@omniti.com wrote: This patch came up from discussion about pg_isready. PQconninfoParseParams is similar to PQconninfoParse but takes two arrays like PQconnectdbParams. It