Re: [HACKERS] [COMMITTERS] pgsql: Add regression test for bug fixed by recent refactoring.

2013-04-30 Thread Tom Lane
Kevin Grittner writes: > Add regression test for bug fixed by recent refactoring. > Test case by Andres Freund for bug fixed by Tom Lane's refactoring > in commit 5194024d72f33fb209e10f9ab0ada7cc67df45b7 Hm, that actually has got nothing much to do with matviews: regression=# create view vv1 as

Re: [HACKERS] Should pg_upgrade use --quote-all-identifiers?

2013-04-30 Thread Greg Stark
On Wed, May 1, 2013 at 2:23 AM, Tom Lane wrote: > Well, readability of those commands is worth something too, but in any > case the short answer is that pg_dump has only one quote-an-identifier > function, not different ones for server commands and final output. Well for the final output one reas

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Greg Stark
On Tue, Apr 30, 2013 at 10:19 PM, Kevin Grittner wrote: > Clearly we would need to change how we do recovery of unlogged > relations to both allow unlogged matviews and keep the populated > state in pg_class. I don't think we can really make the "two > places" technique work, where the recovery s

Re: [HACKERS] Should pg_upgrade use --quote-all-identifiers?

2013-04-30 Thread Tom Lane
Greg Stark writes: > Huh. I thought you were talking about quoting identifiers in an SQL > dump. But you're not, you're talking about quoting identifiers in sql > being sent to the server during the pg_dump process. Why did pg_dump > ever not quote all such identifiers? Well, readability of those

Re: [HACKERS] Should pg_upgrade use --quote-all-identifiers?

2013-04-30 Thread Greg Stark
On Wed, May 1, 2013 at 12:55 AM, Tom Lane wrote: > Seems like this might be a good idea to avoid the type of failure > exhibited in bug #8128. We don't care too much about the readability > of the dump script created during an upgrade, so it's hard to see a > downside. Huh. I thought you were ta

[HACKERS] Should pg_upgrade use --quote-all-identifiers?

2013-04-30 Thread Tom Lane
Seems like this might be a good idea to avoid the type of failure exhibited in bug #8128. We don't care too much about the readability of the dump script created during an upgrade, so it's hard to see a downside. regards, tom lane -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Andres Freund
On 2013-04-30 18:39:09 -0400, Greg Smith wrote: > The WAL logging of hint bits is where the scary stuff to me for this feature > has always been at. My gut feel is that doing that needed to start being > available as an option anyway. Just this month we've had two customer > issues pop up where w

Re: [HACKERS] Incomplete description of pg_start_backup?

2013-04-30 Thread Jeff Janes
On Tue, Apr 30, 2013 at 3:24 PM, Dmitry Koterov wrote: > I think that at > http://www.postgresql.org/docs/current/static/functions-admin.html and > http://www.postgresql.org/docs/current/static/continuous-archiving.html two > important points on how pg_start_backup() works are missing: > > 1. Aft

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Noah Misch
Orthogonal to this thread, but: On Tue, Apr 30, 2013 at 06:39:09PM -0400, Greg Smith wrote: > The WAL logging of hint bits is where the scary stuff to me for this > feature has always been at. My gut feel is that doing that needed to > start being available as an option anyway. Just this mon

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Greg Smith
On 4/30/13 5:26 PM, Martijn van Oosterhout wrote: I came across this today: Data Integrity Extensions, basically a standard for have an application calculate a checksum of a block and submitting it together with the block so that the disk can verify that the block it is writing matches what the a

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Noah Misch
On Tue, Apr 30, 2013 at 04:47:58AM +0100, Joel Jacobson wrote: > On Tue, Apr 30, 2013 at 12:46 AM, Noah Misch wrote: > > Those existing functions give a mostly-SnapshotNow picture of their objects, > > but an sql-language implementation would give a normally-snapshotted > > picture. > > I assume

[HACKERS] Incomplete description of pg_start_backup?

2013-04-30 Thread Dmitry Koterov
I think that at http://www.postgresql.org/docs/current/static/functions-admin.html and http://www.postgresql.org/docs/current/static/continuous-archiving.html two important points on how pg_start_backup() works are missing: 1. After pg_start_backup() and till pg_stop_backup() VACUUM is denied (e.g

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-30 Thread Jeff Davis
On Tue, 2013-04-30 at 08:34 -0400, Robert Haas wrote: > Uh, wait a minute. I think this is completely wrong. The buffer is > LOCKED for this entire sequence of operations. For a checkpoint to > "happen", it's got to write every buffer, which it will not be able to > do for so long as the buffer

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Martijn van Oosterhout
On Tue, Apr 30, 2013 at 01:05:30PM -0400, Greg Smith wrote: > I re-ran the benchmark that's had me most worried against the > committed code and things look good so far. I've been keeping quiet > because my tests recently have all agreed with what Ants already > described. This is more a confirma

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Kevin Grittner
Josh Berkus wrote: >> That was deemed to be incompatible with unlogged matviews, which >> some didn't want to give up in this initial release. > > Huh?  Unlogged tables don't go in pg_class? Sorry if I wasn't clear.  I try to do incremental development -- get one part working and then go on to t

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Josh Berkus
Kevin, > The reason was that the start of CF4 was deemed too late in the > development cycle to be trying to design what that should look > like. No sooner had you suggested that one column than someone > suggested two others which might also be useful, and it seemed to Yeah, I'm just pointing o

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Kevin Grittner
Andres Freund wrote: > On 2013-04-30 07:33:05 -0700, Kevin Grittner wrote: >> Andres Freund wrote: >>> 1) vacuum can truncate the table to an empty relation already >>>    if there is no data returned by the view's query which then >>>    leads to the wrong scannability state. >>>   So we need

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Kevin Grittner
Andres Freund wrote: > Ah. Tom already fixed the problem in > 5194024d72f33fb209e10f9ab0ada7cc67df45b7. I was working in a > branch based on 3ccae48f44d993351e1f881761bd6c556ebd6638 and it > failed there. Since previous regression tests missed this bug, I've added the test you posted, to make su

Re: [HACKERS] materialized view scannability in other DBs

2013-04-30 Thread Kevin Grittner
Robert Haas wrote: > Oracle has (FWICT) had materialized views since 8i (1999) 8i was when they expanded beyond requiring a manual refresh, and changed the name of the feature from "snapshot" to "materialized view".  I'm not sure how long they had "snapshots" before 8i. > Microsoft SQL Server .

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Robert Haas
On Tue, Apr 30, 2013 at 9:02 AM, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: >> On Mon, Apr 29, 2013 at 7:58 PM, Tom Lane wrote: >> > One of the things that we frequently recommend when doing >> > upgrades is that you do the dump with the newer version's pg_dump, so >> > a

[HACKERS] materialized view scannability in other DBs

2013-04-30 Thread Robert Haas
On Tue, Apr 30, 2013 at 10:40 AM, Kevin Grittner wrote: > "Staleness" is a completely different issue, in my view, from > quietly returning results that are not, and never were, accurate. > Sure we need to implement more refined "scannability" tests than > whether valid data from *some* point in t

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Kevin Grittner
Josh Berkus wrote: > We discussed this around the beginning of CF4.  For some reason > (which I didn't quite understand at the time), the consensus was > that creating a "matview last updated" timestamp was not > implementable for 9.3 and would need to wait for 9.4. The reason was that the start

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Josh Berkus
Robert, > - The data can be used if the matview is fully up-to-date. > - The data can be used if the matview is not out of date by more than > a certain amount of time. > - The data can be used if the matview is out of date with respect to > one of its base tables, but not if it is out of date wit

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Greg Smith
I re-ran the benchmark that's had me most worried against the committed code and things look good so far. I've been keeping quiet because my tests recently have all agreed with what Ants already described. This is more a confirmation summary than new data. The problem case has been Jeff's te

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Andres Freund
On 2013-04-30 08:35:32 -0700, Kevin Grittner wrote: > Andres Freund wrote: > > On 2013-04-30 07:33:05 -0700, Kevin Grittner wrote: > >> Andres Freund wrote: > > >>> 2) Since we don't have a metapage to represent scannability in 9.3 > >>>    we cannot easily use one in 9.4+ without pg_upgrade emp

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Robert Haas
On Tue, Apr 30, 2013 at 10:40 AM, Kevin Grittner wrote: >>> What is a real problem or risk with using this mechanism until we >>> engineer something better? What problems with converting to a >>> later major release does anyone see? >> >> Well, it's a pg_upgrade hazard, if nothing else, isn't it?

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Kevin Grittner
Andres Freund wrote: > On 2013-04-30 07:33:05 -0700, Kevin Grittner wrote: >> Andres Freund wrote: >>> 2) Since we don't have a metapage to represent scannability in 9.3 >>>    we cannot easily use one in 9.4+ without pg_upgrade emptying all >>>    matviews unless we only rely on the catalogs wh

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Andres Freund
On 2013-04-30 15:57:02 +0200, Andres Freund wrote: > On 2013-04-30 05:14:15 +0100, Joel Jacobson wrote: > > On Tue, Apr 30, 2013 at 12:58 AM, Tom Lane wrote: > > > Noah Misch writes: > > >> Note also that minor releases can readily fix bugs in C-language > > >> functions, > > >> but we have no i

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Andres Freund
On 2013-04-30 07:33:05 -0700, Kevin Grittner wrote: > Andres Freund wrote: > > 1) vacuum can truncate the table to an empty relation already if there is > >   no data returned by the view's query which then leads to the wrong > >   scannability state. > > > >   S1: CREATE MATERIALIZED VIEW matview

Re: [HACKERS] Back branches vs. gcc 4.8.0

2013-04-30 Thread Greg Stark
On Fri, Apr 5, 2013 at 11:14 PM, Tom Lane wrote: > Since gcc 4.8 is going to be on a lot of people's machines pretty soon, > I think we need to do something to prevent it from breaking 8.4.x and > 9.0.x. It looks like our choices are (1) teach configure to enable > -fno-aggressive-loop-optimizati

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Simon Riggs
On 30 April 2013 15:51, Tom Lane wrote: > Simon Riggs writes: >> ISTM that we also need this patch to put memory barriers in place >> otherwise the code might be rearranged. > > This is simply silly. You crack me up sometimes. Yes, it is; seem to be having a bad day for thinkos. -- Simon Riggs

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Tom Lane
Simon Riggs writes: > ISTM that we also need this patch to put memory barriers in place > otherwise the code might be rearranged. This is simply silly. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subsc

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Tom Lane
Andrew Dunstan writes: > If all you want is SQL language functions, there is nothing to stop you > from writing them and publishing them today as an extension. It's worth noting also that we actually *have* infrastructure for updating extensions without initdb; unlike the initial contents of pg_

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Kevin Grittner
Robert Haas wrote: > Kevin Grittner wrote: >> Let's look at the "corner" this supposedly paints us into.  If a >> later major release creates a better mechanism, current pg_dump and >> load will already use it, based on the way matviews are created >> empty and REFRESHed by pg_dump.  Worst case,

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Kevin Grittner
Andres Freund wrote: > On 2013-04-30 04:29:26 -0700, Kevin Grittner wrote: >> Let's look at the "corner" this supposedly paints us into.  If a >> later major release creates a better mechanism, current pg_dump and >> load will already use it, based on the way matviews are created >> empty and REF

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Robert Haas
On Tue, Apr 30, 2013 at 7:29 AM, Kevin Grittner wrote: > Let's look at the "corner" this supposedly paints us into. If a > later major release creates a better mechanism, current pg_dump and > load will already use it, based on the way matviews are created > empty and REFRESHed by pg_dump. Worst

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Andres Freund
On 2013-04-30 05:14:15 +0100, Joel Jacobson wrote: > On Tue, Apr 30, 2013 at 12:58 AM, Tom Lane wrote: > > Noah Misch writes: > >> Note also that minor releases can readily fix bugs in C-language functions, > >> but we have no infrastructure to update sql-language functions after > >> initdb. >

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Andres Freund
Could we please stop the ad-hominem stuff from all sides? We want to solve the issue not to make it bigger. On 2013-04-30 04:29:26 -0700, Kevin Grittner wrote: > Let's look at the "corner" this supposedly paints us into.  If a > later major release creates a better mechanism, current pg_dump and >

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Stephen Frost
* Joel Jacobson (j...@trustly.com) wrote: > Thanks for the motivational speech! :) I'll start working on it today! Great, but you should really come up with an initial design and get feedback on it before you start coding up something. :) I've outlined a few very, very high-level ideas about what

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Joel Jacobson
On Tue, Apr 30, 2013 at 2:12 PM, Stephen Frost wrote: > Go for it? :) I don't think you'll find much disagreement here, but > it's no trivial thing to do either.. Perhaps some kind of one-time > startup "script" that gets run? Or maybe an internal identifier that > The above doesn't really see

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Stephen Frost
* Joel Jacobson (j...@trustly.com) wrote: > On Tue, Apr 30, 2013 at 12:58 AM, Tom Lane wrote: > > That alone would probably be sufficient reason why we would never allow > > pg_dump to depend on any such thing (not that I see a compelling > > argument for it to do so anyway...). > > It would be b

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Apr 29, 2013 at 7:58 PM, Tom Lane wrote: > > One of the things that we frequently recommend when doing > > upgrades is that you do the dump with the newer version's pg_dump, so > > as to get the benefits of any bug fixes that are in it. The m

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Stephen Frost
Tom, all, * Tom Lane (t...@sss.pgh.pa.us) wrote: > It may be that the functions Joel proposes are worth having for other > tools to use, but I'm not in favor of making pg_dump use them. I care very little about migrating the pg_dump functionality into server-side functions and a great deal about

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-30 Thread Simon Riggs
On 30 April 2013 13:34, Robert Haas wrote: > On Tue, Apr 30, 2013 at 6:58 AM, Simon Riggs wrote: >> On 9 April 2013 08:36, Jeff Davis wrote: >> >>> 1. I believe that the issue I brought up at the end of this email: >>> >>> http://www.postgresql.org/message-id/1365035537.7580.380.camel@sussancws0

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Stephen Frost
Kevin, * Kevin Grittner (kgri...@ymail.com) wrote: > If there is some > particular problem someone sees, I don't think it has been > expressed yet, which makes it impossible to address, unless you > count the assertion that *if* we had chosen a zero-length heap to > represent a table with valid da

Re: [HACKERS] Graph datatype addition

2013-04-30 Thread Atri Sharma
> actually, i was going to suggest to Atri to take a look at that, > pgrouting is currently in develop of v2.0 which will rewrite some > parts (including some of the algorithms). > > Maybe this datatype could fit better as part of pgrouting Thanks, I will have a look at pgRouting.Although, in a sh

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-30 Thread Robert Haas
On Tue, Apr 30, 2013 at 6:58 AM, Simon Riggs wrote: > On 9 April 2013 08:36, Jeff Davis wrote: > >> 1. I believe that the issue I brought up at the end of this email: >> >> http://www.postgresql.org/message-id/1365035537.7580.380.camel@sussancws0025 >> >> is a real issue. In lazy_vacuum_page(), t

Re: [HACKERS] Graph datatype addition

2013-04-30 Thread Atri Sharma
> 1. We created JSON and XML types as ways of storing data that has a robust > validation system. > > They're still, in a sense, just "plain old text", but it's "plain old text" > that the user can be certain satisfies the respective rules for > representations. Yes, although, I feel that we can

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Andrew Dunstan
On 04/30/2013 07:34 AM, Joel Jacobson wrote: On Tue, Apr 30, 2013 at 11:22 AM, Andrew Dunstan wrote: Minor releases are supposed not to require any such operations. You should normally be able to drop the binaries in place and restart. For C language functions that is indeed all you have to do

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Simon Riggs
On 30 April 2013 12:23, Ants Aasma wrote: >> ISTM that we also need this patch to put memory barriers in place >> otherwise the code might be rearranged. > > The compiler and CPU both have to preserve correctness when > rearranging code, so I don't think we care about it here. It might > matter i

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Joel Jacobson
On Tue, Apr 30, 2013 at 11:22 AM, Andrew Dunstan wrote: > Minor releases are supposed not to require any such operations. You should > normally be able to drop the binaries in place and restart. For C language > functions that is indeed all you have to do, but that's not the case for SQL > languag

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Kevin Grittner
Robert Haas wrote: > Kevin Grittner wrote: >> The hysteria and FUD about using the currently-supported technique >> for unlogged tables to implement unlogged matviews are very >> discouraging.  And the notion that we would release a matview >> feature which allowed false results (in the form of

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Ants Aasma
On Tue, Apr 30, 2013 at 1:55 PM, Simon Riggs wrote: > On 30 April 2013 06:57, Simon Riggs wrote: > >> I'm about to light up the build farm with a trial commit of the >> compiler instructions stuff. > > Amazingly that seemed to work. Thanks for committing. Sorry about missing the .h file from the

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Andres Freund
On 2013-04-30 11:55:29 +0100, Simon Riggs wrote: > ISTM that we also need this patch to put memory barriers in place > otherwise the code might be rearranged. > > -- > Simon Riggs http://www.2ndQuadrant.com/ > PostgreSQL Development, 24x7 Support, Training & Services > --- a/s

Re: [HACKERS] Remaining beta blockers

2013-04-30 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > If you assume that people are going to modify files while the backend > is running, nothing we do anywhere is safe. I agree that it's a bad idea and that people who do such things deserve what they get, but that doesn't mean it won't happen when peopl

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-30 Thread Simon Riggs
On 9 April 2013 08:36, Jeff Davis wrote: > 1. I believe that the issue I brought up at the end of this email: > > http://www.postgresql.org/message-id/1365035537.7580.380.camel@sussancws0025 > > is a real issue. In lazy_vacuum_page(), the following sequence can > happen when checksums are on: > >

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Simon Riggs
On 30 April 2013 06:57, Simon Riggs wrote: > I'm about to light up the build farm with a trial commit of the > compiler instructions stuff. Amazingly that seemed to work. ISTM that we also need this patch to put memory barriers in place otherwise the code might be rearranged. -- Simon Riggs

Re: [HACKERS] The missing pg_get_*def functions

2013-04-30 Thread Andrew Dunstan
On 04/29/2013 11:47 PM, Joel Jacobson wrote: Note also that minor releases can readily fix bugs in C-language functions, but we have no infrastructure to update sql-language functions after initdb. That flexibility is unfortunate to lose, particularly for something that pg_dump depends on. Now,

Re: [HACKERS] Graph datatype addition

2013-04-30 Thread Atri Sharma
> I believe it would be best to largely separate the questions of storage and > access. Partly because of Florian's concern that you'd frequently want only > one representation of the whole graph, but also because the actual storage > interface does NOT have to be user friendly if we have a good ac

Re: [HACKERS] Graph datatype addition

2013-04-30 Thread Atri Sharma
> Have you considered maybe ltree datatype? > > To me all described sounds solveable on pure sql way ( + ltree datatype to > help with indexes and performance as materialised path to avoid recursive > query all the time...) This may build over the existing solutions itself. I will have to check th

Re: [HACKERS] [COMMITTERS] pgsql: Add sql_drop event for event triggers

2013-04-30 Thread Dimitri Fontaine
Tom Lane writes: > Applied with some further hacking. Thanks! > Hmm, that leads me to wonder exactly how extensively the regression > tests test event triggers, because it sure looked to me like there > were multiple bugs left in this version. The first versions of the event triggers patch seri

Re: [HACKERS] Graph datatype addition

2013-04-30 Thread Atri Sharma
> Usually though, you'd be interested a large graphs which include > information for lots of records (e.g., nodes are individual users, > or products, or whatever). A graph datatype is not well suited for > that, because it'd store each graph as a single value, and updating > the graph would mean r