[HACKERS] string_to_array eats too much memory?
Hi, I'm playing with GIN to make a full text search system. GIN comes with built-in TEXT[] support and I use string_to_array() to make a TEXT[]. Problem is, if there's large number of array elemets, string_to_array() consumes too much memory. For example, to make ~70k array elements, string_to_array seems to eat several Gig bytes of memory. ~70k array elements means there are same number of words in a document which is not too big in a large text IMO. Comments? -- Tatsuo Ishii SRA OSS, Inc. Japan ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Block B-tree etc. (was Re: Introducing an advanced Frequent
Heikki, On 11/7/06 1:51 AM, "Heikki Linnakangas" <[EMAIL PROTECTED]> wrote: > I don't think Simon's proposal is meant to address that issue, but did > you follow the thread I started in September about Block B-Tree index: > http://archives.postgresql.org/pgsql-hackers/2006-09/msg02041.php Aha - mystery solved - thanks for the clarification. Block Btree ~= Lossy Btree - Luke ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Uncleared result sets in describeOneTableDetails()
On Tue, 2006-11-07 at 17:56 +1100, Brendan Jurd wrote: > Should be just six extra lines (patch attached, untested). Applied to HEAD, with an additional fix: you need to clear "result5" as well. I didn't bother applying it to backbranches, on the grounds that a memory leak in psql is not serious. I think refactoring this function is the right long-term fix, but that will have to wait for 8.2 to branch. -Neil ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [PATCHES] Bug in WAL backup documentation
On Sun, Nov 05, 2006 at 11:49:36 -0500, Tom Lane <[EMAIL PROTECTED]> wrote: > > As already discussed upthread, anyone who wants the path can get it from > `pwd` or local equivalent --- and that mechanism is robust (as long as > the directory move doesn't happen while any particular instance of the > script is running). I don't see why we should go out of our way to > provide a bad substitute for pwd. I think you also still need read access to the intervening directories. If the command works by walking up and matching inode numbers with names, then it will break if it can't read the names. (For example /bin/pwd breaks when it can't read a parent directories filenames.) ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Index ignored with "is not distinct from", 8.2 beta2
I can see that adding null to indexes would allow all cases of "is not distinct from" to use them. The lack of null in indexes would explain why a condition such as "col is not distinct from null" would not pick up an index. But my example was: "col is not distinct from 123" and the equivalent longer form has no problem picking up the index I could restate the problem as: Why can't the planner handle "col is not distinct from 123" as well as "col is not null and 123 is not null and col = 123 or col is null and 123 is null" Jean-Pierre Pelletier From: Martijn van Oosterhout Reply-To: Martijn van Oosterhout To: JEAN-PIERRE PELLETIER <[EMAIL PROTECTED]> CC: [EMAIL PROTECTED], pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Index ignored with "is not distinct from", 8.2 beta2 Date: Tue, 7 Nov 2006 11:03:42 +0100 On Mon, Nov 06, 2006 at 09:10:40PM -0500, JEAN-PIERRE PELLETIER wrote: > I understand that the planner doesn't use indexes for IS NOT DISTINCT FROM, > but it would > be good because "is not distinct from" is very useful when you have nulls > but don't want to use three value logic. The main issue is that currently indexes cannot be used to find NULLs in a table. Patches have been created that cover most index types, but it's not part of the main distribution. Partial indexes can be a solution to the "x IS NULL" clauses. Hope this helps, -- Martijn van Oosterhout http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to litigate. << signature.asc >> ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Index ignored with "is not distinct from", 8.2 beta2
On Tue, Nov 07, 2006 at 02:12:29PM -0600, Jim C. Nasby wrote: > I assume you're referring to > http://archives.postgresql.org/pgsql-patches/2005-09/msg00083.php ? > > I'm curious as to the status of that patch... presumably it never made > it into the queue, but I'm not sure why since you seemed to address > Tom's concerns (other than not indexing IS NOT NULL, which I'm not sure > is all that useful...) There's been work on it. Theodor cleaned it up for HEAD and looked at adding GiST support. I beleive he's waiting for 8.2 to release. Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate. signature.asc Description: Digital signature
Re: [HACKERS] Index ignored with "is not distinct from", 8.2 beta2
On Tue, Nov 07, 2006 at 11:03:42AM +0100, Martijn van Oosterhout wrote: > On Mon, Nov 06, 2006 at 09:10:40PM -0500, JEAN-PIERRE PELLETIER wrote: > > I understand that the planner doesn't use indexes for IS NOT DISTINCT FROM, > > but it would > > be good because "is not distinct from" is very useful when you have nulls > > but don't want to use three value logic. > > The main issue is that currently indexes cannot be used to find NULLs > in a table. Patches have been created that cover most index types, but > it's not part of the main distribution. I assume you're referring to http://archives.postgresql.org/pgsql-patches/2005-09/msg00083.php ? I'm curious as to the status of that patch... presumably it never made it into the queue, but I'm not sure why since you seemed to address Tom's concerns (other than not indexing IS NOT NULL, which I'm not sure is all that useful...) -- Jim Nasby[EMAIL PROTECTED] EnterpriseDB http://enterprisedb.com 512.569.9461 (cell) ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] 8.2 Beta 3 Now Available for Download / Testing ...
Marc G. Fournier wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 As a result of there being two *known* outstanding bugs, we have just bundled up a Beta3, to allow for testing of the recent patch concerning WAL replay ... Please download the latest beta from: ftp.postgresql.org:/pub/source/v8.2beta While the mirrors themselves catch up over the next 24 hours ... What are the bugs? cheers andrew ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
[HACKERS] 8.2 Beta 3 Now Available for Download / Testing ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 As a result of there being two *known* outstanding bugs, we have just bundled up a Beta3, to allow for testing of the recent patch concerning WAL replay ... Please download the latest beta from: ftp.postgresql.org:/pub/source/v8.2beta While the mirrors themselves catch up over the next 24 hours ... - Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . [EMAIL PROTECTED] MSN . [EMAIL PROTECTED] Yahoo . yscrappy Skype: hub.orgICQ . 7615664 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (FreeBSD) iD8DBQFFUM+D4QvfyHIvDvMRAsScAJ9cgzl8AFJDyOoQ+HN43mEe5oWLLgCgn4Wr 5fexUO9hUkMe+5tWKetCyrE= =O1YF -END PGP SIGNATURE- ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Block B-tree etc. (was Re: Introducing an advanced Frequent Update)
Luke, > That's our plan to achieve speedups that other DBMS's have achieved with > Index-Organized-Tables or Clustered Indexes. We're running initial > performance tests of it as we speak, and if all goes well we're hoping > to get that into PostgreSQL 8.3. Also, if the EDB folks weren't clear, the two proposals are meant to compliment each other. -- Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Found a bug
Thanks for the references.Note for future: should have tried VACUUM too.On 11/7/06, Tom Lane <[EMAIL PROTECTED] > wrote:"Gurjeet Singh" < [EMAIL PROTECTED]> writes:> Please refer the following session snippet. It seems that creating and> dropping an index on a table, within a transaction, leaves the table marked> as having an index. This isn't a bug. Refer to catalogs.sgml:relhasindexTrue if this is a table and it has (or recently had) any indexes.This is set by CREATE INDEX, but not cleared immediately by DROP INDEX. VACUUM clears relhasindex if it finds the table has no indexes.The flag is only used as a hint that it's worth looking in pg_index tosee what rows there are for the table.As for "tracking it down", read index_drop(). regards, tom lane-- [EMAIL PROTECTED][EMAIL PROTECTED] gmail | hotmail | yahoo }.com
Re: [HACKERS] Found a bug
"Gurjeet Singh" <[EMAIL PROTECTED]> writes: > Please refer the following session snippet. It seems that creating and > dropping an index on a table, within a transaction, leaves the table marked > as having an index. This isn't a bug. Refer to catalogs.sgml: relhasindex True if this is a table and it has (or recently had) any indexes. This is set by CREATE INDEX, but not cleared immediately by DROP INDEX. VACUUM clears relhasindex if it finds the table has no indexes. The flag is only used as a hint that it's worth looking in pg_index to see what rows there are for the table. As for "tracking it down", read index_drop(). regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Found a bug
Gurjeet Singh wrote: Hi All, Please refer the following session snippet. It seems that creating and dropping an index on a table, within a transaction, leaves the table marked as having an index. Although, I don't think it's a serious bug, since we always retrieve the list using RelationGetIndexList(), and that list turns out to be empty; but, if possible we should track it down. Can't reproduce it here: richardh=# create table t (a int); CREATE TABLE richardh=# begin; BEGIN richardh=# create index t_idx on t (a); CREATE INDEX richardh=# select relhasindex from pg_class where relname = 't'; relhasindex - t (1 row) richardh=# rollback; ROLLBACK richardh=# select relhasindex from pg_class where relname = 't'; relhasindex - f PostgreSQL 8.1.3 on i586-pc-linux-gnu -- Richard Huxton Archonet Ltd ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
[HACKERS] Found a bug
Hi All, Please refer the following session snippet. It seems that creating and dropping an index on a table, within a transaction, leaves the table marked as having an index. Although, I don't think it's a serious bug, since we always retrieve the list using RelationGetIndexList(), and that list turns out to be empty; but, if possible we should track it down. Even restarting the DB cluster doesn't help.postgres=# create table t ( a int ); CREATE TABLEpostgres=# select relhasindex from pg_class where relname = 't'; relhasindex- f(1 row) postgres=# begin;BEGIN postgres=# create index ind_t_a on t(a);CREATE INDEX postgres=# select relhasindex from pg_class where relname = 't'; relhasindex - t(1 row) postgres=# \d t Table "public.t" Column | Type | Modifiers+-+--- a | integer |Indexes: "ind_t_a" btree (a)postgres=# rollback; ROLLBACKpostgres=# \d t Table "public.t" Column | Type | Modifiers +-+--- a | integer |postgres=# select relhasindex from pg_class where relname = 't'; relhasindex - t (1 row)postgres=# Best regards, -- [EMAIL PROTECTED][EMAIL PROTECTED] gmail | hotmail | yahoo }.com
Re: [HACKERS] Uncleared result sets in describeOneTableDetails()
On Tue, 2006-11-07 at 17:56 +1100, Brendan Jurd wrote: > It certainly isn't pretty. It's been a long time since I looked down > the barrel of a 'goto'. I don't think there's anything wrong with using "goto" for error handling in this style. Personally, I think the main stylistic problem is that the function is 600 lines long: it would be a lot cleaner if refactored into smaller functions with smaller individual scopes. -Neil ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Index ignored with "is not distinct from", 8.2 beta2
On Mon, Nov 06, 2006 at 09:10:40PM -0500, JEAN-PIERRE PELLETIER wrote: > I understand that the planner doesn't use indexes for IS NOT DISTINCT FROM, > but it would > be good because "is not distinct from" is very useful when you have nulls > but don't want to use three value logic. The main issue is that currently indexes cannot be used to find NULLs in a table. Patches have been created that cover most index types, but it's not part of the main distribution. Partial indexes can be a solution to the "x IS NULL" clauses. Hope this helps, -- Martijn van Oosterhout http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate. signature.asc Description: Digital signature
[HACKERS] Block B-tree etc. (was Re: Introducing an advanced Frequent Update)
Luke Lonergan wrote: Simon, Bring it on! We at GP have been evaluating various approaches to index organized tables which unify index with heap storage to solve some of the problems you mention. Split index and heap is a big issue in Postgres and we'd all welcome a good solution to it, even for limited circumstances like single index organization or the like. I don't think Simon's proposal is meant to address that issue, but did you follow the thread I started in September about Block B-Tree index: http://archives.postgresql.org/pgsql-hackers/2006-09/msg02041.php That's our plan to achieve speedups that other DBMS's have achieved with Index-Organized-Tables or Clustered Indexes. We're running initial performance tests of it as we speak, and if all goes well we're hoping to get that into PostgreSQL 8.3. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
