[HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-23 Thread Jeff Davis
Starting a new thread to more narrowly address the topic. Attached is my reorganization of Ants's patch here: http://www.postgresql.org/message-id/CA +CSw_vinyf-w45i=M1m__MpJZY=e8s4nt_knnpebtwjtoa...@mail.gmail.com My changes: * wrest the core FNV algorithm from the specific concerns of a data

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-23 Thread Florian Pflug
On Apr23, 2013, at 09:17 , Jeff Davis pg...@j-davis.com wrote: I'd lean toward simplicity and closer adherence to the published version of the algorithm rather than detecting a few more obscure error patterns. It looks like the modification slows down the algorithm, too. The pattern that plain

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-23 Thread Ants Aasma
On Apr 23, 2013 10:17 AM, Jeff Davis pg...@j-davis.com wrote: Attached is my reorganization of Ants's patch here: http://www.postgresql.org/message-id/CA +CSw_vinyf-w45i=M1m__MpJZY=e8s4nt_knnpebtwjtoa...@mail.gmail.com Thanks for your help. Some notes below. My changes: * wrest the core

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-23 Thread Andres Freund
On 2013-04-23 00:17:28 -0700, Jeff Davis wrote: + # important optimization flags for checksum.c + ifeq ($(GCC),yes) + checksum.o: CFLAGS += -msse4.1 -funroll-loops -ftree-vectorize + endif I am pretty sure we can't do those unconditionally: - -funroll-loops and -ftree-vectorize weren't always

Re: [HACKERS] 9.3 Beta1 status report

2013-04-23 Thread Dean Rasheed
On 21 April 2013 06:02, Bruce Momjian br...@momjian.us wrote: I am not sure if Tom shared yet, but we are planning to package 9.3 beta1 on April 29, with a release on May 2. Those dates might change, but that is the current plan. I have completed a draft 9.3 release notes, which you can view

[HACKERS] Couple of issues with pg_xlogdump

2013-04-23 Thread Pavan Deolasee
Hello, I was playing with pg_xlogdump in the HEAD and found a few issues. 1. Tried compiling pg_xlogdump via PGXS mechanism and it fails with the following error: make: *** No rule to make target

Re: [HACKERS] Couple of issues with pg_xlogdump

2013-04-23 Thread Andres Freund
On 2013-04-23 14:51:05 +0530, Pavan Deolasee wrote: Hello, I was playing with pg_xlogdump in the HEAD and found a few issues. 1. Tried compiling pg_xlogdump via PGXS mechanism and it fails with the following error: make: *** No rule to make target

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-23 Thread Ants Aasma
On Tue, Apr 23, 2013 at 11:47 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-04-23 00:17:28 -0700, Jeff Davis wrote: + # important optimization flags for checksum.c + ifeq ($(GCC),yes) + checksum.o: CFLAGS += -msse4.1 -funroll-loops -ftree-vectorize + endif I am pretty sure we

Re: [HACKERS] Couple of issues with pg_xlogdump

2013-04-23 Thread Pavan Deolasee
On Tue, Apr 23, 2013 at 3:00 PM, Andres Freund and...@2ndquadrant.comwrote: On 2013-04-23 14:51:05 +0530, Pavan Deolasee wrote: Hello, I was playing with pg_xlogdump in the HEAD and found a few issues. 1. Tried compiling pg_xlogdump via PGXS mechanism and it fails with the following

Re: [HACKERS] 9.3 Beta1 status report

2013-04-23 Thread Heikki Linnakangas
On 22.04.2013 23:06, Bruce Momjian wrote: On Mon, Apr 22, 2013 at 10:11:48PM +0300, Heikki Linnakangas wrote: E.1.3.2.1. Write-Ahead Log (WAL) Store WAL in a continuous stream, rather than skipping the last 16MB segment every 4GB (Heikki Linnakangas) BACKWARD COMPATIBLE BREAK

Re: [HACKERS] Couple of issues with pg_xlogdump

2013-04-23 Thread Andres Freund
On 2013-04-23 15:16:05 +0530, Pavan Deolasee wrote: Which this confirms. This is likely the current end of wal. If you look at pg_current_xlog_location() after starting the server again, it should show an address nearby? Oh yes, you are right. Again, could there be a better way to

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-23 Thread Jeevan Chalke
Hi Tom, Since we are close to release, we should not have crashes like this. Please have a look. My patch may not be correct as I haven't looked closely. Thanks On Mon, Apr 22, 2013 at 7:28 PM, Jeevan Chalke jeevan.cha...@enterprisedb.com wrote: On Mon, Apr 22, 2013 at 6:41 PM, Andres

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-23 Thread Andres Freund
Hi, On 2013-04-23 17:48:49 +0530, Jeevan Chalke wrote: Hi Tom, Since we are close to release, we should not have crashes like this. Please have a look. My patch may not be correct as I haven't looked closely. Isn't that Kevin's departement? Andres -- Andres Freund

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-23 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: On 2013-04-23 17:48:49 +0530, Jeevan Chalke wrote: Hi Tom, Since we are close to release, we should not have crashes like this. Please have a look. My patch may not be correct as I haven't looked closely. Isn't that Kevin's departement? I'll

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-23 Thread Merlin Moncure
On Tue, Apr 23, 2013 at 7:18 AM, Jeevan Chalke jeevan.cha...@enterprisedb.com wrote: Hi Tom, Since we are close to release, we should not have crashes like this. huh? we are not even in beta yet merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Performance with the new security release?

2013-04-23 Thread Steve Singer
On 13-04-22 11:46 PM, Anne Rosset wrote: Thanks Steve. I have read that a fix has been put in release 9.2.3 for this issue. Is that right? Thanks, Anne No this issue is present in 9.0.13, 9.1.9 and 9.2.4 (as well as 9.2.3). There is talk about fixing this for the next set of minor releases

[HACKERS] putting a bgworker to rest

2013-04-23 Thread Andres Freund
Hi all, I noticed the need to simply stop a bgworker after its work is done but still have it restart in unusual circumstances like a crash. Obviously I can just have it enter a loop where it checks its latch and such, but that seems a bit pointless. Would it make sense to add an extra return

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-23 Thread Jeevan Chalke
On Tue, Apr 23, 2013 at 7:01 PM, Merlin Moncure mmonc...@gmail.com wrote: On Tue, Apr 23, 2013 at 7:18 AM, Jeevan Chalke jeevan.cha...@enterprisedb.com wrote: Hi Tom, Since we are close to release, we should not have crashes like this. huh? we are not even in beta yet I mean, beta

Re: [HACKERS] Performance with the new security release?

2013-04-23 Thread Anne Rosset
Thanks Steve. I found this: http://www.postgresql.org/docs/current/static/release-9-2-3.html Fix performance problems with autovacuum truncation in busy workloads (Jan Wieck) Truncation of empty pages at the end of a table requires exclusive lock, but autovacuum was coded to fail (and release

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-23 Thread Andres Freund
On 2013-04-23 19:33:24 +0530, Jeevan Chalke wrote: On Tue, Apr 23, 2013 at 7:01 PM, Merlin Moncure mmonc...@gmail.com wrote: On Tue, Apr 23, 2013 at 7:18 AM, Jeevan Chalke jeevan.cha...@enterprisedb.com wrote: Hi Tom, Since we are close to release, we should not have crashes like

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-23 Thread Shaun Thomas
On 04/22/2013 05:12 PM, Merlin Moncure wrote: free -g total used free sharedbuffers cached Mem: 378250128 0 0229 -/+ buffers/cache: 20357 This is most likely a NUMA issue. There really

Re: [HACKERS] Enabling Checksums

2013-04-23 Thread Simon Riggs
On 23 April 2013 02:35, Jeff Davis pg...@j-davis.com wrote: On Tue, 2013-04-23 at 01:08 +0300, Ants Aasma wrote: A slight delay, but here it is. I didn't lift the checksum part into a separate file as I didn't have a great idea what I would call it. The code is reasonably compact so I don't

[HACKERS] Proposal to add --single-row to psql

2013-04-23 Thread Christopher Manning
psql currently collects the query result rows in memory before writing them to a file and can cause out of memory problems for large results in low memory environments like ec2. I can't use COPY TO STDOUT or FETCH_COUNT since I'm using Redshift and it doesn't support [writing to STDOUT](

Re: [HACKERS] Performance with the new security release?

2013-04-23 Thread Steve Singer
On 13-04-23 10:04 AM, Anne Rosset wrote: Thanks Steve. I found this: http://www.postgresql.org/docs/current/static/release-9-2-3.html Fix performance problems with autovacuum truncation in busy workloads (Jan Wieck) Truncation of empty pages at the end of a table requires exclusive lock, but

Re: [HACKERS] Performance with the new security release?

2013-04-23 Thread Alvaro Herrera
Anne Rosset wrote: Thanks Steve. I found this: http://www.postgresql.org/docs/current/static/release-9-2-3.html Fix performance problems with autovacuum truncation in busy workloads (Jan Wieck) Truncation of empty pages at the end of a table requires exclusive lock, but autovacuum was

Re: [HACKERS] Couple of issues with pg_xlogdump

2013-04-23 Thread Alvaro Herrera
Andres Freund escribió: On 2013-04-23 15:16:05 +0530, Pavan Deolasee wrote: It works without either if you use explicit options like -s STARTADDR and -p PATH which is frequently useful to just print a few records at the correct point. I am not sure how could put that in there without

Re: [HACKERS] putting a bgworker to rest

2013-04-23 Thread Alvaro Herrera
Andres Freund wrote: Hi all, I noticed the need to simply stop a bgworker after its work is done but still have it restart in unusual circumstances like a crash. Obviously I can just have it enter a loop where it checks its latch and such, but that seems a bit pointless. Would it make

Re: [HACKERS] Proposal to add --single-row to psql

2013-04-23 Thread Pavel Stehule
Hello It is redundant to current FETCH_COUNT implementation, so I don't see sense to use it together. Maybe we can drop FETCH_COUNT and replace it by --single-row mode and probably it can simplify code. Regards Pavel 2013/4/23 Christopher Manning c...@christophermanning.org psql currently

Re: [HACKERS] Proposal to add --single-row to psql

2013-04-23 Thread Tom Lane
Christopher Manning c...@christophermanning.org writes: I'm proposing to add a --single-row option to psql that would allow the result rows of a query to be streamed to a file without collecting them in memory first. Isn't there already a way to set FETCH_COUNT from the command line? (ie, I

Re: [HACKERS] Couple of issues with pg_xlogdump

2013-04-23 Thread Andres Freund
On 2013-04-23 11:51:06 -0300, Alvaro Herrera wrote: Andres Freund escribió: On 2013-04-23 15:16:05 +0530, Pavan Deolasee wrote: It works without either if you use explicit options like -s STARTADDR and -p PATH which is frequently useful to just print a few records at the correct

Re: [HACKERS] putting a bgworker to rest

2013-04-23 Thread Andres Freund
On 2013-04-23 11:59:43 -0300, Alvaro Herrera wrote: Andres Freund wrote: Hi all, I noticed the need to simply stop a bgworker after its work is done but still have it restart in unusual circumstances like a crash. Obviously I can just have it enter a loop where it checks its latch and

[HACKERS] Bug Fix: COLLATE with multiple ORDER BYs in aggregates

2013-04-23 Thread David Fetter
Folks, While testing the upcoming FILTER clause for aggregates, Erik Rijkers uncovered a long-standing bug in $subject, namely that this case wasn't handled. Please find attached a patch by Andrew Gierth and myself which fixes this issue and adds a regression test to ensure it remains fixed.

Re: [HACKERS] putting a bgworker to rest

2013-04-23 Thread Alvaro Herrera
Andres Freund wrote: On 2013-04-23 11:59:43 -0300, Alvaro Herrera wrote: Andres Freund wrote: Hi all, I noticed the need to simply stop a bgworker after its work is done but still have it restart in unusual circumstances like a crash. Obviously I can just have it enter a loop

Re: [HACKERS] putting a bgworker to rest

2013-04-23 Thread Andres Freund
On 2013-04-23 14:11:26 -0300, Alvaro Herrera wrote: Andres Freund wrote: On 2013-04-23 11:59:43 -0300, Alvaro Herrera wrote: Andres Freund wrote: Hi all, I noticed the need to simply stop a bgworker after its work is done but still have it restart in unusual circumstances

Re: [HACKERS] minimizing the target list for foreign data wrappers

2013-04-23 Thread David Gudeman
In case anyone is interested, I tried it and it doesn't seem to work. It looks like some other plan element already has the target-list tuple baked. Now I'm trying to decide whether to give up on FDW. It's a shame because it's such a sweet facility, but at this point, I just don't think that it's

Re: [HACKERS] Couple of issues with pg_xlogdump

2013-04-23 Thread Jeff Janes
On Tue, Apr 23, 2013 at 2:30 AM, Andres Freund and...@2ndquadrant.comwrote: On 2013-04-23 14:51:05 +0530, Pavan Deolasee wrote: [pavan.deolasee@puppetserver pg_xlogdump]$ ./pg_xlogdump ~/db93head/pg_xlog/00010008 pg_xlogdump: FATAL: could not find a valid record after

Re: [HACKERS] minimizing the target list for foreign data wrappers

2013-04-23 Thread David Fetter
David, Please post your patch(es) and some demo of how things broke so others can improve future versions--possibly even 9.3 versions if it turns out you've discovered a bug in the implementation. Thanks very much for your hard work and insights into this. Cheers, David. On Tue, Apr 23, 2013 at

Re: [HACKERS] GSOC Student Project Idea

2013-04-23 Thread Alvaro Herrera
Michael Schuh escribió: http://www.cs.montana.edu/~timothy.wylie/files/bncod13.pdf Um. From the paper I get the impression that there's yet much to learn in order for this indexing method to be really effective? -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] 9.3 Beta1 status report

2013-04-23 Thread Bruce Momjian
On Tue, Apr 23, 2013 at 10:12:58AM +0100, Dean Rasheed wrote: On 21 April 2013 06:02, Bruce Momjian br...@momjian.us wrote: I am not sure if Tom shared yet, but we are planning to package 9.3 beta1 on April 29, with a release on May 2. Those dates might change, but that is the current

Re: [HACKERS] 9.3 Beta1 status report

2013-04-23 Thread Bruce Momjian
On Tue, Apr 23, 2013 at 05:36:03PM +0800, Jov wrote: E.1.3.1.4: Improve performance of the CREATE TABLE ... ON COMMIT DELETE ROWS clause by only issuing delete if the temporary table was accessed (Heikki Linnakangas) should be:              CREATE TEMP TABLE ... ON COMMIT DELETE ROWS or

Re: [HACKERS] 9.3 Beta1 status report

2013-04-23 Thread Erikjan Rijkers
I just spotted some more small stuff: s/IF NOT EXIST /IF NOT EXISTS /g # 2 x It actually had me doubting, but yes that -S should be there... Thanks, Erik Rijkers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] 9.3 Beta1 status report

2013-04-23 Thread Bruce Momjian
On Tue, Apr 23, 2013 at 02:25:08PM +0300, Heikki Linnakangas wrote: On 22.04.2013 23:06, Bruce Momjian wrote: On Mon, Apr 22, 2013 at 10:11:48PM +0300, Heikki Linnakangas wrote: E.1.3.2.1. Write-Ahead Log (WAL) Store WAL in a continuous stream, rather than skipping the last 16MB

Re: [HACKERS] 9.3 Beta1 status report

2013-04-23 Thread Bruce Momjian
On Tue, Apr 23, 2013 at 11:00:31PM +0200, Erikjan Rijkers wrote: I just spotted some more small stuff: s/IF NOT EXIST /IF NOT EXISTS /g # 2 x It actually had me doubting, but yes that -S should be there... Fixed, thanks. -- Bruce Momjian br...@momjian.ushttp://momjian.us

Re: [HACKERS] 9.3 Beta1 status report

2013-04-23 Thread Bruce Momjian
On Tue, Apr 23, 2013 at 05:04:15PM -0400, Bruce Momjian wrote: Do we usually repeat the changes listed in the backwards compatibility section later, in the Changes section? If not, then instead of the first two items above, let's just have these in the backwards-compatibility section:

Re: [HACKERS] GSOC Student Project Idea

2013-04-23 Thread Alexander Korotkov
Michael, On Mon, Apr 22, 2013 at 4:28 PM, Michael Schuh schuh.m...@gmail.com wrote: Although I do not have a lot of experience with PostgreSQL development, I am eager to learn and commit my summer to enabling another fantastic feature for the community. Since iDistance is a non-recursive,

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-23 Thread Kevin Grittner
Jeevan Chalke jeevan.cha...@enterprisedb.com wrote: On Mon, Apr 22, 2013 at 6:41 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-04-22 18:35:04 +0530, Jeevan Chalke wrote: I have observed that following sequence is causing server crash. CREATE MATERIALIZED VIEW temp_class_mv AS   

Re: [HACKERS] 9.3 Beta1 status report

2013-04-23 Thread Alvaro Herrera
Bruce Momjian wrote: On Tue, Apr 23, 2013 at 05:04:15PM -0400, Bruce Momjian wrote: Do we usually repeat the changes listed in the backwards compatibility section later, in the Changes section? If not, then instead of the first two items above, let's just have these in the

Re: [HACKERS] Enabling Checksums

2013-04-23 Thread Jeff Davis
On Tue, 2013-04-23 at 16:28 +0100, Simon Riggs wrote: * make the pg_control.data_checksums field into a version number, for future flexibility... patch attached Commenting on this separately because it's a separate issue. I'd prefer that it was some kind of a checksum ID code -- e.g. 0 for no

[HACKERS] a patch for row-at-a-time execution for table functions

2013-04-23 Thread David Gudeman
The primary change we made to Postgres in order to support our own version of foreign data wrappers was a row-at-a-time execution for table functions. In standard Postgres, when you execute a table function, it gathers all of the rows at once and stuffs them into a buffer in order to support

Re: [HACKERS] a patch for row-at-a-time execution for table functions

2013-04-23 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/23/2013 06:09 PM, David Gudeman wrote: The primary change we made to Postgres in order to support our own version of foreign data wrappers was a row-at-a-time execution for table functions. In standard Postgres, when you execute a table

Re: [HACKERS] 9.3 Beta1 status report

2013-04-23 Thread Bruce Momjian
On Tue, Apr 23, 2013 at 06:56:34PM -0300, Alvaro Herrera wrote: Bruce Momjian wrote: On Tue, Apr 23, 2013 at 05:04:15PM -0400, Bruce Momjian wrote: Do we usually repeat the changes listed in the backwards compatibility section later, in the Changes section? If not, then instead of

[HACKERS] 9.3 release notes suggestions

2013-04-23 Thread Bruce Momjian
Thanks for the many suggestions on improving the 9.3 release notes. There were many ideas I would have never thought of. Please keep the suggestions coming. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's