Re: [HACKERS] Perl 5.18 breaks pl/perl regression tests?

2013-06-03 Thread Alvaro Herrera
Tom Lane wrote: It looks quite a bit like somebody's fixed a line-counting bug inside Perl, which may mean that we'll have to maintain two expected-output files or else remove these particular test cases. Which would be annoying. Maybe we can set a $SIG{__WARN__} routine instead, which

Re: [HACKERS] Perl 5.18 breaks pl/perl regression tests?

2013-06-03 Thread Kaare Rasmussen
On 2013-06-03 06:55, Michael Paquier wrote: Just by having a look at the release notes of Perl, there are still nothing describing changes between 1.6.3 and 1.8.0: http://perldoc.perl.org/index-history.html That page is not updated, it seems. In this list

Re: [HACKERS] erroneous restore into pg_catalog schema

2013-06-03 Thread Heikki Linnakangas
On 14.05.2013 15:35, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: I don't disagree, but how is that relevant for fixing the issue at hand? We still need to fix restores that currently target the wrong schema in a backward compatible manner? On this, I agree w/ Tom that

[HACKERS] Time for beta2 ?

2013-06-03 Thread Heikki Linnakangas
We've almost cleared the open items list, and I'm not aware of any other unfixed issues in 9.3beta1. Could we make a beta2 release soon? There have been a bunch of recovery-related fixes since beta1, it would be nice to get those fixes in the hands of testers. - Heikki -- Sent via

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Ants Aasma
On Mon, Jun 3, 2013 at 1:20 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Jun 2, 2013 at 2:44 PM, Jeff Janes jeff.ja...@gmail.com wrote: Do we know why anti-wraparound uses so many resources in the first place? The default settings seem to be quite conservative to me, even for a system

[HACKERS] Implicit rule created for materialized views

2013-06-03 Thread Dave Page
Hi In playing with materialized views, I noticed that they still seem to have an _RETURN rule implicitly created like a regular view. This doesn't seem right to me - is there a reason? viper:~ dpage$ /usr/local/pgsql-9.3/bin/psql -p 5433 -U postgres psql (9.3beta1) Type help for help.

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Craig Ringer
On 06/02/2013 05:56 AM, Robert Haas wrote: On Sat, Jun 1, 2013 at 1:49 PM, Simon Riggs si...@2ndquadrant.com wrote: On 30 May 2013 19:48, Josh Berkus j...@agliodbs.com wrote: There's currently some great ideas bouncing around about eliminating the overhead associated with FREEZE. However, I

Re: [HACKERS] Implicit rule created for materialized views

2013-06-03 Thread Dave Page
On Mon, Jun 3, 2013 at 1:06 PM, Kevin Grittner kevin.gritt...@enterprisedb.com wrote: Yes, that is currently used for REFRESH, and will be used to drive the incremental maintenance when that is added. Without it, CREATE MATERIALIZED VIEW wouldn't be different from CREATE TABLE AS. OK. A

Re: [HACKERS] [COMMITTERS] pgsql: Minor spelling fixes

2013-06-03 Thread Stephen Frost
* Thom Brown (t...@linux.com) wrote: Oh, if you're in the mood for typo-correction... Done, thanks. I wouldn't be against material improvements to the comments in some of the code that's being fixed for typos either, by the way.. Starting with actually *having* some, in some places.

Re: [HACKERS] Running pgindent

2013-06-03 Thread Stephen Frost
* Bruce Momjian (br...@momjian.us) wrote: On Fri, May 31, 2013 at 03:51:30PM -0400, Andrew Dunstan wrote: I spent quite a lot of time trying to make the tool behave the same as the old script. Yes, and I believe we tested running the Perl version to make sure it was the same, so the

Re: [HACKERS] GRANT role_name TO role_name ON database_name

2013-06-03 Thread Stephen Frost
* Clark C. Evans (c...@clarkevans.com) wrote: Yes, if we had per-database roles, it would work. However, I don't think it's necessary. We've already got role permissions specific to a database; so we're most of the way there. PG has two sets of catalogs, per-databases ones and 'shared'

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Kevin Grittner
Craig Ringer cr...@2ndquadrant.com wrote: On 06/02/2013 05:56 AM, Robert Haas wrote: I agree with all that.  I don't have any data either, but I agree that AFAICT it seems to mostly be a problem for large (terabyte-scale) databases, or ones that are dreadfully short of I/O bandwidth.  AWS,

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Andres Freund
On 2013-06-02 11:44:04 -0700, Jeff Janes wrote: Do we know why anti-wraparound uses so many resources in the first place? The default settings seem to be quite conservative to me, even for a system that has only a single 5400 rpm hdd (and even more so for any real production system that would

Re: [HACKERS] Implicit rule created for materialized views

2013-06-03 Thread Tom Lane
Dave Page dp...@pgadmin.org writes: On Mon, Jun 3, 2013 at 1:06 PM, Kevin Grittner kevin.gritt...@enterprisedb.com wrote: A materialized view is pretty much like a view, but with the results materialized. Yeah, I get that, but what is confusing is that this now seems to be a special kind of

Re: [HACKERS] erroneous restore into pg_catalog schema

2013-06-03 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: In the interest of getting the release out, I've reverted commit a475c603. We'll probably want to do something more elegant in the future, but this will do for now. That may be the best short-term answer, but I see no such revert in the repo

Re: [HACKERS] erroneous restore into pg_catalog schema

2013-06-03 Thread Heikki Linnakangas
On 03.06.2013 17:18, Tom Lane wrote: Heikki Linnakangashlinnakan...@vmware.com writes: In the interest of getting the release out, I've reverted commit a475c603. We'll probably want to do something more elegant in the future, but this will do for now. That may be the best short-term answer,

Re: [HACKERS] Perl 5.18 breaks pl/perl regression tests?

2013-06-03 Thread Tom Lane
Kaare Rasmussen ka...@jasonic.dk writes: That page is not updated, it seems. In this list https://metacpan.org/module/RJBS/perl-5.18.0/pod/perldelta.pod is mentioned Line numbers at the end of a string eval are no longer off by one. [perl #114658] Hah. That leads to

[HACKERS] UTF-8 encoding problem w/ libpq

2013-06-03 Thread Martin Schäfer
I try to create database columns with umlauts, using the UTF8 client encoding. However, the server seems to mess up the column names. In particular, it seems to perform a lowercase operation on each byte of the UTF-8 multi-byte sequence. Here is my code: const wchar_t *strName = Lid_äß;

Re: [HACKERS] UTF-8 encoding problem w/ libpq

2013-06-03 Thread k...@rice.edu
On Mon, Jun 03, 2013 at 03:40:14PM +0100, Martin Schäfer wrote: I try to create database columns with umlauts, using the UTF8 client encoding. However, the server seems to mess up the column names. In particular, it seems to perform a lowercase operation on each byte of the UTF-8 multi-byte

Re: [HACKERS] PostgreSQL Process memory architecture

2013-06-03 Thread Ben Zeev, Lior
Hi Stephen, I have some basic question - How do I add this flags CATCACHE_STATS and CATCACHE_FORCE_RELEASE when building postgresql? I added it to src/Makefile.global in this line: CPPFLAGS = -D_GNU_SOURCE -DCATCACHE_STATS -DCATCACHE_FORCE_RELEASE And changed log level to debug2, but it

Re: [HACKERS] Implicit rule created for materialized views

2013-06-03 Thread Kevin Grittner
Dave Page dp...@pgadmin.org wrote: Kevin Grittner kevin.gritt...@enterprisedb.com wrote: Dave Page dp...@pgadmin.org wrote: In playing with materialized views, I noticed that they still seem to have an _RETURN rule implicitly created like a regular view. A materialized view is pretty much

Re: [HACKERS] UTF-8 encoding problem w/ libpq

2013-06-03 Thread Martin Schäfer
-Original Message- From: k...@rice.edu [mailto:k...@rice.edu] Sent: 03 June 2013 16:48 To: Martin Schäfer Cc: pgsql-hackers@postgresql.org Subject: Re: [HACKERS] UTF-8 encoding problem w/ libpq On Mon, Jun 03, 2013 at 03:40:14PM +0100, Martin Schäfer wrote: I try to create

Re: [HACKERS] Running pgindent

2013-06-03 Thread Alvaro Herrera
Stephen Frost escribió: Just to wrap this up- I wanted to say thanks to both you (Bruce) and to Andrew for making pgindent work and the documentation / instructions easy to follow. In the past, pgindent has always seemed to be a black art, where it was difficult to get consistent results

Re: [HACKERS] UTF-8 encoding problem w/ libpq

2013-06-03 Thread k...@rice.edu
On Mon, Jun 03, 2013 at 04:09:29PM +0100, Martin Schäfer wrote: If I change the strCreate query and add double quotes around the column name, then the problem disappears. But the original name is already in lowercase, so I think it should also work without quoting the column name. Am I

Re: [HACKERS] Implicit rule created for materialized views

2013-06-03 Thread Dave Page
On Mon, Jun 3, 2013 at 3:59 PM, Kevin Grittner kgri...@ymail.com wrote: Dave Page dp...@pgadmin.org wrote: Kevin Grittner kevin.gritt...@enterprisedb.com wrote: Dave Page dp...@pgadmin.org wrote: In playing with materialized views, I noticed that they still seem to have an _RETURN rule

Re: [HACKERS] Perl 5.18 breaks pl/perl regression tests?

2013-06-03 Thread David E. Wheeler
On Jun 3, 2013, at 7:31 AM, Tom Lane t...@sss.pgh.pa.us wrote: Hah. That leads to http://perl5.git.perl.org/perl.git/commitdiff/451f421 in which it's said What happens is that eval tacks \n; on to the end of the string if it does not already end with a semicolon. So we could likely hide

Re: [HACKERS] UTF-8 encoding problem w/ libpq

2013-06-03 Thread Heikki Linnakangas
On 03.06.2013 18:27, k...@rice.edu wrote: On Mon, Jun 03, 2013 at 04:09:29PM +0100, Martin Schäfer wrote: If I change the strCreate query and add double quotes around the column name, then the problem disappears. But the original name is already in lowercase, so I think it should also work

Re: [HACKERS] Re: [HACKERS] high io BUT huge amount of free memory

2013-06-03 Thread Merlin Moncure
On Mon, Jun 3, 2013 at 11:08 AM, Миша Тюрин tmih...@bk.ru wrote: Hi all hackers again! Since i had got this topic there many test was done by our team and many papers was seen. And then I noticed that os_page_replacement_algorithm with CLOCK and others features might * interfere / overlap

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Josh Berkus
I agree with all that. I don't have any data either, but I agree that AFAICT it seems to mostly be a problem for large (terabyte-scale) databases, or ones that are dreadfully short of I/O bandwidth. AWS, I'm looking at you. Well, at this point, numerically I'd bet that more than 50% of our

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Josh Berkus
Jeff, Do we know why anti-wraparound uses so many resources in the first place? The default settings seem to be quite conservative to me, even for a system that has only a single 5400 rpm hdd (and even more so for any real production system that would be used for a many-GB database). I

Re: [HACKERS] UTF-8 encoding problem w/ libpq

2013-06-03 Thread Andrew Dunstan
On 06/03/2013 12:22 PM, Heikki Linnakangas wrote: On 03.06.2013 18:27, k...@rice.edu wrote: On Mon, Jun 03, 2013 at 04:09:29PM +0100, Martin Schäfer wrote: If I change the strCreate query and add double quotes around the column name, then the problem disappears. But the original name is

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Robert Haas
On Mon, Jun 3, 2013 at 1:28 PM, Josh Berkus j...@agliodbs.com wrote: Does anti-wraparound vacuum (AWAV) write synchronously? If so, there's a potential whole world of hurt there. Not any moreso than anything else ... although it probably does a very high percentage of FPIs, which might lead to

[HACKERS] Re[2]: [HACKERS] Re: [HACKERS] high io BUT huge amount of free memory

2013-06-03 Thread Миша Тюрин
hm, in that case, wouldn't adding 48gb of physical memory have approximately the same effect? or is something else going on? imho, adding 48gb would have no effects. server already has 376GB memory and still has a lot of unused GB. let me repeat, we added 80GB for files cache by decreasing

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Josh Berkus
Also, locking while it does its work. Eh? Even if we're doing lazy vacuum, we have to lock a few pages at a time of each table. This does result in response time delays on the current workload, which can be quite bad if it's a highly contended table already. -- Josh Berkus PostgreSQL

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Andres Freund
On 2013-06-03 11:00:38 -0700, Josh Berkus wrote: Also, locking while it does its work. Eh? Even if we're doing lazy vacuum, we have to lock a few pages at a time of each table. This does result in response time delays on the current workload, which can be quite bad if it's a highly

Re: [HACKERS] UTF-8 encoding problem w/ libpq

2013-06-03 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: He *is* using UTF-8. Or trying to, anyway :-). The downcasing in the backend is supposed to leave bytes with the high-bit set alone, ie. in UTF-8 encoding, it's supposed to leave ä and ß alone. Well, actually,

Re: [HACKERS] Implicit rule created for materialized views

2013-06-03 Thread Kevin Grittner
Yes, that is currently used for REFRESH, and will be used to drive the incremental maintenance when that is added. Without it, CREATE MATERIALIZED VIEW wouldn't be different from CREATE TABLE AS. A materialized view is pretty much like a view, but with the results materialized. -Kevin On

Re: [HACKERS] Optimising Foreign Key checks

2013-06-03 Thread Jim Nasby
On 6/2/13 4:45 AM, Simon Riggs wrote: Will this add too much cost where it doesn't help? I don't know what to predict there. There's the obvious case of trivial transactions with no more than one referential integrity check per FK, but there's also the case of a transaction with many FK checks

Re: [HACKERS] UTF-8 encoding problem w/ libpq

2013-06-03 Thread Andrew Dunstan
On 06/03/2013 02:28 PM, Tom Lane wrote: . I wonder though if we couldn't just fix this code to not do anything to high-bit-set bytes in multibyte encodings. That's exactly what I suggested back in November. cheers andrew -- Sent via pgsql-hackers mailing list

Re: [HACKERS] UTF-8 encoding problem w/ libpq

2013-06-03 Thread Heikki Linnakangas
On 03.06.2013 21:28, Tom Lane wrote: Heikki Linnakangashlinnakan...@vmware.com writes: He *is* using UTF-8. Or trying to, anyway :-). The downcasing in the backend is supposed to leave bytes with the high-bit set alone, ie. in UTF-8 encoding, it's supposed to leave ä and ß alone. Well,

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Josh Berkus
On 06/03/2013 11:12 AM, Andres Freund wrote: On 2013-06-03 11:00:38 -0700, Josh Berkus wrote: Also, locking while it does its work. Eh? Even if we're doing lazy vacuum, we have to lock a few pages at a time of each table. This does result in response time delays on the current workload,

Re: [HACKERS] Optimising Foreign Key checks

2013-06-03 Thread Simon Riggs
On 3 June 2013 19:41, Jim Nasby j...@nasby.net wrote: On 6/2/13 4:45 AM, Simon Riggs wrote: Will this add too much cost where it doesn't help? I don't know what to predict there. There's the obvious case of trivial transactions with no more than one referential integrity check per FK, but

[HACKERS] Question about storage subsystem of PotgreSQL

2013-06-03 Thread javadi
Hi I want to find some architectural information about the storage subsystem of PostgreSQL especially the modulus which are responsible to loading data from disk to the Shared Buffer. I cannot find any structured and useful information on web. Would you please help me to find such information.

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Martijn van Oosterhout
On Mon, Jun 03, 2013 at 11:27:57AM +0300, Ants Aasma wrote: I can't rule that out. Personally, I've always attributed it to the fact that it's (a) long and (b) I/O-intensive. But it's not impossible there could also be bugs lurking. It could be related to the OS. I have no evidence for

Re: [HACKERS] Question about storage subsystem of PotgreSQL

2013-06-03 Thread Bruce Momjian
On Sat, Jun 1, 2013 at 02:11:13PM +0430, javadi wrote: Hi I want to find some architectural information about the storage subsystem of PostgreSQL especially the modulus which are responsible to loading data from disk to the Shared Buffer. I cannot find any structured and useful

Re: [HACKERS] Question about storage subsystem of PotgreSQL

2013-06-03 Thread Kevin Grittner
javadi seyyedahmad.jav...@gmail.com wrote: I want to find some architectural information about the storage subsystem of PostgreSQL especially the modulus which are responsible to loading data from disk to the Shared Buffer.  I cannot find any structured and useful information on web. Would

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Robert Haas
On Mon, Jun 3, 2013 at 2:56 PM, Josh Berkus j...@agliodbs.com wrote: Really? I though vacuum held onto its locks until it reached vacuum_cost. If it doesn't, then maybe we should adjust the default for vacuum_cost_limit upwards. That would be completely insane. Or in other words, no, it

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Peter Geoghegan
On Mon, Jun 3, 2013 at 5:03 AM, Craig Ringer cr...@2ndquadrant.com wrote: I've seen cases on Stack Overflow and elsewhere in which disk merge sorts perform vastly better than in-memory quicksort, so the user benefited from greatly *lowering* work_mem. I've heard of that happening on Oracle,

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Robert Haas
On Mon, Jun 3, 2013 at 3:48 PM, Martijn van Oosterhout klep...@svana.org wrote: On Mon, Jun 03, 2013 at 11:27:57AM +0300, Ants Aasma wrote: I can't rule that out. Personally, I've always attributed it to the fact that it's (a) long and (b) I/O-intensive. But it's not impossible there

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Martijn van Oosterhout klep...@svana.org wrote: On Mon, Jun 03, 2013 at 11:27:57AM +0300, Ants Aasma wrote: It could be related to the OS. I have no evidence for or against, but it's possible that OS write-out routines defeat the careful cost based

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Peter Geoghegan
On Mon, Jun 3, 2013 at 3:16 PM, Kevin Grittner kgri...@ymail.com wrote: But it seems like the kernel is disposed to cache large amounts of dirty data for an unbounded period of time even if the I/O system is completely idle, It's not unbounded time. Last I heard, the default was 30 seconds.

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Craig Ringer
On 06/04/2013 05:27 AM, Peter Geoghegan wrote: On Mon, Jun 3, 2013 at 5:03 AM, Craig Ringer cr...@2ndquadrant.com wrote: I've seen cases on Stack Overflow and elsewhere in which disk merge sorts perform vastly better than in-memory quicksort, so the user benefited from greatly *lowering*

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Andres Freund
On 2013-06-03 16:41:32 -0700, Peter Geoghegan wrote: On Mon, Jun 3, 2013 at 3:16 PM, Kevin Grittner kgri...@ymail.com wrote: But it seems like the kernel is disposed to cache large amounts of dirty data for an unbounded period of time even if the I/O system is completely idle, It's not

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Josh Berkus
All that has pretty significantly changed - and imo improved! - in the last year or so of kernel development. Unfortunately it will take a while till we commonly see those kernels being used :( ... after being completely broken for 3.2 through 3.5. We're actually using 3.9 in production on

Re: [HACKERS] local_preload_libraries logspam

2013-06-03 Thread Peter Geoghegan
On Mon, May 13, 2013 at 3:22 PM, Peter Geoghegan p...@heroku.com wrote: Attached patch renders all loaded library... messages DEBUG1, regardless of whether local_preload_libraries or shared_preload_libraries is involved, and regardless of EXEC_BACKEND. Can someone take a look at this, please?

Re: [HACKERS] Optimising Foreign Key checks

2013-06-03 Thread Noah Misch
On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: For clarity the 4 problems are 1. SQL execution overhead 2. Memory usage 3. Memory scrolling 4. Locking overhead, specifically FPWs and WAL records from FK checks probably in that order or thereabouts. The above is why I went

[HACKERS] create a git symbolic-ref for REL9_3_STABLE

2013-06-03 Thread Peter Eisentraut
I suppose we'll be branching off 9.3 in a few weeks. That event always creates a service gap in the build farm and similar services, and a race in the NLS service to get everything adjusted to the new branch. It seems to me we could already now create a git symbolic-ref named REL9_3_STABLE that

Re: [HACKERS] create a git symbolic-ref for REL9_3_STABLE

2013-06-03 Thread Andrew Dunstan
On 06/03/2013 09:30 PM, Peter Eisentraut wrote: I suppose we'll be branching off 9.3 in a few weeks. That event always creates a service gap in the build farm and similar services, and a race in the NLS service to get everything adjusted to the new branch. The buildfarm has had a mechanism

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-03 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/02/2013 03:10 PM, Dimitri Fontaine wrote: Tom Lane t...@sss.pgh.pa.us writes: Actually, I believe the answer is just that getSchemaData() is doing things in the wrong order: Indeed Tom, as usual, seems to have the best correct answer :-)

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-03 Thread Tom Lane
Joe Conway m...@joeconway.com writes: I was surprised by a couple of things looking at this code. First, getRules() is written differently than other table subsidiary objects' get functions. Secondly, I would have expected getExtensionMembership() to be recursive -- instead it looks to only

Re: [HACKERS] MVCC catalog access

2013-06-03 Thread Michael Paquier
On Tue, Jun 4, 2013 at 3:57 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, May 30, 2013 at 1:39 AM, Michael Paquier michael.paqu...@gmail.com wrote: +1. Here's a more serious patch for MVCC catalog access. This one involves more data copying than the last one, I think, because the

Re: [HACKERS] PostgreSQL Process memory architecture

2013-06-03 Thread Ben Zeev, Lior
No matter how I try to redesign the schema the indexes consume large amount of memory, About 8KB per index. Is there a way to invalidated this cache? Is there a way to limit the amount of memory and use some kind of LRU/LFU algorithm to clean old cache? -Original Message- From: Atri