Re: [HACKERS] How would sorting work with millions of rows in a huge DB with PG?

2011-04-09 Thread tomas
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sun, Apr 10, 2011 at 05:20:02AM +0530, Vaibhav Kaushal wrote: > Hello all, > > I was going through some papers related to sorting and since I am > studying PG code side by side, I wondered how sorting would be done on a > DB with millions of rows o

Re: [HACKERS] pg_upgrade bug found!

2011-04-09 Thread Bruce Momjian
Aidan Van Dyk wrote: > On Sat, Apr 9, 2011 at 7:03 AM, Bruce Momjian wrote: > > Bruce Momjian wrote: > >> Alvaro Herrera wrote: > >> > > >> > Why is it important to have the original pg_clog files around? ?Since > >> > the transactions in question are below the freeze horizon, surely the > >> > tu

Re: [HACKERS] using a lot of maintenance_work_mem

2011-04-09 Thread Stephen Frost
* Greg Stark (gsst...@mit.edu) wrote: > On Sat, Apr 9, 2011 at 6:20 PM, Tom Lane wrote: > > BTW, it sounded like your argument had to do with whether it would use > > HashAgg or not -- that is *not* dependent on the per-palloc limit, and > > never has been. > > His point was he wanted to be allow

Re: [HACKERS] using a lot of maintenance_work_mem

2011-04-09 Thread Joshua D. Drake
On Sun, 2011-04-10 at 03:05 +0100, Greg Stark wrote: > On Sat, Apr 9, 2011 at 6:20 PM, Tom Lane wrote: > > BTW, it sounded like your argument had to do with whether it would use > > HashAgg or not -- that is *not* dependent on the per-palloc limit, and > > never has been. > > > > His point was he

Re: [HACKERS] using a lot of maintenance_work_mem

2011-04-09 Thread Greg Stark
On Sat, Apr 9, 2011 at 6:20 PM, Tom Lane wrote: > BTW, it sounded like your argument had to do with whether it would use > HashAgg or not -- that is *not* dependent on the per-palloc limit, and > never has been. > His point was he wanted to be allowed to set work_mem > 1GB. This is going to becom

[HACKERS] Typed table DDL loose ends

2011-04-09 Thread Noah Misch
While looking at the typed table/pg_upgrade problem, I ran into a few smaller problems in the area. I'm not envisioning a need for much code shift to fix them, but there are a few points of policy. * Table row types used in typed tables vs. ALTER TABLE As previously noted: CREATE TABLE t ();

Re: [HACKERS] Bug in pg_hba.conf or pg_basebackup concerning replication connections

2011-04-09 Thread Josh Berkus
> That's a 2000 line patch that looks like it's out of the question now. > But I think this should fix Josh's immediate problem if we want to do it: I have confirmed that Andrew's patch works. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (

[HACKERS] How would sorting work with millions of rows in a huge DB with PG?

2011-04-09 Thread Vaibhav Kaushal
Hello all, I was going through some papers related to sorting and since I am studying PG code side by side, I wondered how sorting would be done on a DB with millions of rows on disk with GBs of data. Since holding everything in memory would not be the possible solution, how do we actually sort th

Re: [HACKERS] pg_upgrade bug found!

2011-04-09 Thread Noah Misch
On Sat, Apr 09, 2011 at 09:05:42AM -0400, Aidan Van Dyk wrote: > On Sat, Apr 9, 2011 at 7:03 AM, Bruce Momjian wrote: > > Bruce Momjian wrote: > >> Alvaro Herrera wrote: > >> > > >> > Why is it important to have the original pg_clog files around? ?Since > >> > the transactions in question are belo

Re: [HACKERS] Bug in pg_hba.conf or pg_basebackup concerning replication connections

2011-04-09 Thread Andrew Dunstan
On 04/09/2011 07:11 PM, Andrew Dunstan wrote: Incidentally, are walsenders supposed to be able to match any db name other than 'replication'? If not, I think we have a bug in check_db(), which is probably missing an "else return false;" in the amwalsender branch. Sorry, I misread the

Re: [HACKERS] Bug in pg_hba.conf or pg_basebackup concerning replication connections

2011-04-09 Thread Andrew Dunstan
On 04/09/2011 03:18 PM, Brendan Jurd wrote: On 10 April 2011 04:23, Joshua Berkus wrote: If I have the following line in pg_hba.conf: hostreplication replication all md5 pg_basebackup -x -v -P -h master1 -U replication -D $PGDATA pg_basebackup: could

Re: [HACKERS] Teaching regex operators about collations

2011-04-09 Thread David E. Wheeler
On Apr 9, 2011, at 2:40 PM, Tom Lane wrote: > Since ILIKE now responds to collations, it would be nice if the > case-insensitive regex operators did too. The hard part of that is > getting the information from src/backend/utils/adt/regexp.c to > src/backend/regex/regc_locale.c. In principle we c

Re: [HACKERS] Bug in pg_hba.conf or pg_basebackup concerning replication connections

2011-04-09 Thread Josh Berkus
> Welcome to the wonderful world of keywords in hba not being specific > to fields. I encountered this problem myself back in Oct 2010 [1] and > predicted that it would bite other users. You've been kind enough to > validate that prediction. I submitted a WIP patch aimed at fixing it > just ove

Re: [HACKERS] Open issues for collations

2011-04-09 Thread Tom Lane
Peter Eisentraut writes: > On mån, 2011-03-28 at 20:02 -0400, Tom Lane wrote: >> One thing I noticed but didn't push to committing is that the test >> case has a largely-unnecessary assumption about how the local system's >> locale names spell "utf8". We could eliminate that by having it use >>

[HACKERS] Teaching regex operators about collations

2011-04-09 Thread Tom Lane
Since ILIKE now responds to collations, it would be nice if the case-insensitive regex operators did too. The hard part of that is getting the information from src/backend/utils/adt/regexp.c to src/backend/regex/regc_locale.c. In principle we could probably add a field to the data structures carr

Re: [HACKERS] Open issues for collations

2011-04-09 Thread Peter Eisentraut
On mån, 2011-03-28 at 20:02 -0400, Tom Lane wrote: > Peter Eisentraut writes: > > On lör, 2011-03-26 at 00:36 -0400, Tom Lane wrote: > >> * It'd sure be nice if we had some nontrivial test cases that work > >> in encodings besides UTF8. I'm still bothered that the committed > >> patch failed to c

Re: [HACKERS] \dO versus collations for other encodings

2011-04-09 Thread Peter Eisentraut
On fre, 2011-04-08 at 20:14 -0400, Tom Lane wrote: > Given that this display doesn't include any encoding column, I'm > thinking that the intention was to show only relevant collation > entries. > Which we could do by adding a WHERE clause about the encoding. > If the intention was to not restrict

Re: [HACKERS] Bug in pg_hba.conf or pg_basebackup concerning replication connections

2011-04-09 Thread Brendan Jurd
On 10 April 2011 04:23, Joshua Berkus wrote: > If I have the following line in pg_hba.conf: > > host    replication     replication             all                     md5 > > pg_basebackup -x -v -P -h master1 -U replication -D $PGDATA > pg_basebackup: could not connect to server: FATAL:  no pg_hb

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Jesper Krogh
On 2011-04-09 18:54, Tom Lane wrote: I think that would be a positive disimprovement. The current design guarantees that volatile sort expressions are evaluated exactly once, in the order the rows are read from the data source. There would be no guarantees at all, either as to the number of eval

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Jesper Krogh
On 2011-04-09 20:00, David Fetter wrote: Given the horrors query generators perpetrate, it might be worth dropping provably redundant ORDER BYs on the floor at planning time. Well, many people often add a secondary sort-key to their SQL for the only purpose of obtainting a consistent result in t

[HACKERS] Feature request: pg_basebackup --force

2011-04-09 Thread Joshua Berkus
Magnus, all: It seems a bit annoying to have to do an rm -rf * $PGDATA/ before resynching a standby using pg_basebackup. This means that I still need to wrap basebackup in a shell script, instead of having it do everything for me ... especially if I have multiple tablespaces. Couldn't we have

[HACKERS] Bug in pg_hba.conf or pg_basebackup concerning replication connections

2011-04-09 Thread Joshua Berkus
All, If I have the following line in pg_hba.conf: hostreplication replication all md5 pg_basebackup -x -v -P -h master1 -U replication -D $PGDATA pg_basebackup: could not connect to server: FATAL: no pg_hba.conf entry for replication connection from host

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread David Fetter
On Sat, Apr 09, 2011 at 07:24:15PM +0300, Heikki Linnakangas wrote: > On 09.04.2011 19:17, David Fetter wrote: > >On Sat, Apr 09, 2011 at 03:22:14PM +0200, Jesper Krogh wrote: > >>This seems like a place where there is room for improvement. > >> > >>2011-04-09 15:18:08.016 testdb=# select id from t

Re: [HACKERS] pg_hba.conf needs sample replication line, replication user

2011-04-09 Thread Magnus Hagander
On Sat, Apr 9, 2011 at 19:29, Joshua Berkus wrote: > All, > > We left this out of 9.0; let's not leave it out of 9.1.  We need an example > "replication" line in pg_hba.conf, commented out.  e.g. > > # host   replication   all     samenet   md5 > > Also, what happened to having a "replication" us

[HACKERS] pg_hba.conf needs sample replication line, replication user

2011-04-09 Thread Joshua Berkus
All, We left this out of 9.0; let's not leave it out of 9.1. We need an example "replication" line in pg_hba.conf, commented out. e.g. # host replication all samenet md5 Also, what happened to having a "replication" user defined by default? We talked this to death last year, I tho

Re: [HACKERS] using a lot of maintenance_work_mem

2011-04-09 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> If we were actually trying to support such large allocations, >> what I'd be inclined to do is introduce a separate call along the lines >> of MemoryContextAllocLarge() that lacks the safety check. > This sounds like the right ap

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Tom Lane
Martijn van Oosterhout writes: > On Sat, Apr 09, 2011 at 09:17:10AM -0700, David Fetter wrote: >>> It seems strange that there is a need to evaluate testsort(id) at >>> all in this case. >> How would PostgreSQL know that sorting by id leaves no ambiguity for >> the next key to address? > Well, i

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Heikki Linnakangas
On 09.04.2011 19:17, David Fetter wrote: On Sat, Apr 09, 2011 at 03:22:14PM +0200, Jesper Krogh wrote: This seems like a place where there is room for improvement. 2011-04-09 15:18:08.016 testdb=# select id from test1 where id< 3 order by id; id 1 2 (2 rows) Time: 0.328 ms 2011-0

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Martijn van Oosterhout
On Sat, Apr 09, 2011 at 09:17:10AM -0700, David Fetter wrote: > > 2011-04-09 15:18:22.138 testdb=# select id from test1 where id < 3 > > order by id,testsort(id); > > id > > > > 1 > > 2 > > (2 rows) > > > > Time: 3001.896 ms > > > > It seems strange that there is a need to evaluate tes

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Jesper Krogh
> > How would PostgreSQL know that sorting by id leaves no ambiguity for > the next key to address? It wouldn't But it could postpone evaluation until ambiguity was actually met. Jesper > -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subsc

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread David Fetter
On Sat, Apr 09, 2011 at 03:22:14PM +0200, Jesper Krogh wrote: > This seems like a place where there is room for improvement. > > 2011-04-09 15:18:08.016 testdb=# select id from test1 where id < 3 > order by id; > id > > 1 > 2 > (2 rows) > > Time: 0.328 ms > 2011-04-09 15:18:11.936 testd

Re: [HACKERS] getting carriage return character in vacuumo

2011-04-09 Thread Martijn van Oosterhout
On Fri, Apr 08, 2011 at 02:10:56PM -0400, Tom Lane wrote: > Muhammad Usama writes: > > While using the vacuumlo utility I encountered a redundant carriage > > return(\r') character in the output. It is required in any scenario? If not, > > please find attached a tiny patch which will get rid of th

[HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Jesper Krogh
This seems like a place where there is room for improvement. 2011-04-09 15:18:08.016 testdb=# select id from test1 where id < 3 order by id; id 1 2 (2 rows) Time: 0.328 ms 2011-04-09 15:18:11.936 testdb=# CREATE or Replace FUNCTION testsort(id integer) returns integer as $$ BEGIN pe

Re: [HACKERS] [BUGS] Failed assert ((data - start) == data_size) in heaptuple.c

2011-04-09 Thread Brendan Jurd
On 9 April 2011 00:41, Alvaro Herrera wrote: > Excerpts from Brendan Jurd's message of vie abr 08 06:00:22 -0300 2011: >> Memtest didn't report any errors.  I intend to try swapping out the >> RAM tomorrow, but in the meantime we got a *different* assertion >> failure today.  The fact that we are

Re: [HACKERS] pg_upgrade bug found!

2011-04-09 Thread Aidan Van Dyk
On Sat, Apr 9, 2011 at 7:03 AM, Bruce Momjian wrote: > Bruce Momjian wrote: >> Alvaro Herrera wrote: >> > >> > Why is it important to have the original pg_clog files around?  Since >> > the transactions in question are below the freeze horizon, surely the >> > tuples that involve those transaction

Re: [HACKERS] really lazy vacuums?

2011-04-09 Thread Andres Freund
On Thursday, March 24, 2011 06:32:10 PM Jim Nasby wrote: > Is there an equivalent in other OSes? Some have mincore which can be used for that in combination with mmap. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.po

Re: [HACKERS] really lazy vacuums?

2011-04-09 Thread Cédric Villemain
2011/3/24 Jim Nasby : > On Mar 22, 2011, at 11:46 AM, Cédric Villemain wrote: >> 2011/3/22 Greg Stark : >>> On Mon, Mar 21, 2011 at 6:08 PM, Jim Nasby wrote: Has anyone looked at the overhead of measuring how long IO requests to the kernel take? If we did that not only could we get an i

Re: [HACKERS] pg_upgrade bug found!

2011-04-09 Thread Bruce Momjian
Bruce Momjian wrote: > Alvaro Herrera wrote: > > > > Why is it important to have the original pg_clog files around? Since > > the transactions in question are below the freeze horizon, surely the > > tuples that involve those transaction have all been visited by vacuum > > and thus removed if the

Re: [HACKERS] pgindent

2011-04-09 Thread Bruce Momjian
Robert Haas wrote: > On Fri, Apr 8, 2011 at 11:21 PM, Andrew Dunstan wrote: > > We've got more work to do before that works, so I have committed what we > > have. Some symbols have disappeared, some because of code changes and some > > probably because Cygwin has changed the way it does objdump. T

Re: [HACKERS] pg_upgrade bug found!

2011-04-09 Thread Bruce Momjian
Alvaro Herrera wrote: > > Why is it important to have the original pg_clog files around? Since > the transactions in question are below the freeze horizon, surely the > tuples that involve those transaction have all been visited by vacuum > and thus removed if they were leftover from aborted tran

Re: [HACKERS] pg_upgrade bug found!

2011-04-09 Thread Bruce Momjian
bricklen wrote: > On Fri, Apr 8, 2011 at 10:01 PM, bricklen wrote: > > Update on the status of the steps we took, which were: > > - test on a hot standby by bringing it live, running the script, > > determing the missing clog files, copying them into the live (hot > > standby) pg_clog dir > > > >

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-09 Thread Leonardo Francalanci
> I'm pretty sure we wouldn't accept a patch for a feature that would > only work with wal_level=minimal, but it might be a useful starting > point for someone else to keep hacking on. I understand. Reading your post at http://archives.postgresql.org/pgsql-hackers/2011-01/msg00315.php I thoug