Re: [GENERAL] Completely wrong row estimates

2010-04-05 Thread Magnus Hagander
2010/4/4 Björn Lindqvist :
> Subject: Completely wrong row estimates
>
> Hello everybody,
>
> Here is the EXPLAIN ANALYZE output for a simple query in my database
> running on postgres 8.3.9:
>
> EXPLAIN ANALYZE
> SELECT *  FROM word w JOIN video_words vw ON w.id = vw.word_id
> WHERE w.word = 'tagtext';
>
> QUERY PLAN
> ---
>  Nested Loop  (cost=18.89..2711.16 rows=95 width=24) (actual
> time=19.266..131.255 rows=43374 loops=1)
>   ->  Index Scan using word_word_key on word w  (cost=0.00..8.28
> rows=1 width=12) (actual time=0.029..0.034 rows=1 loops=1)
>         Index Cond: ((word)::text = 'tagtext'::text)
>   ->  Bitmap Heap Scan on video_words vw  (cost=18.89..2693.31
> rows=766 width=12) (actual time=19.227..77.809 rows=43374 loops=1)
>         Recheck Cond: (vw.word_id = w.id)
>         ->  Bitmap Index Scan on video_words_word_id_key
> (cost=0.00..18.70 rows=766 width=0) (actual time=12.662..12.662
> rows=43374 loops=1)
>               Index Cond: (vw.word_id = w.id)
>  Total runtime: 154.215 ms
>
> Note how the planner estimates that there are 766 rows in the table
> that matches the word 'tagtext'. In reality 43374 does. I've tried to
> get postgres to refresh the statistics by running with
> enable_statistics_target=100, running VACUUM, VACUUM FULL, VACUUM FULL
> ANALYZE etc but nothing works. Postgres seem stuck with its bad
> statistics and unwilling to change them. There are many other strings
> that also matches tens of thousands of rows in the table which
> postgres only thinks matches 766.

I assume you mean default_statistics_target, not enable_statistics_target.

You should try setting it higher - but obviously just for these
columns. Use something like

ALTER TABLE words ALTER COLUMN word SET STATISTICS 1000

Then you need to run ANALYZE on it. Not vacuum ,certainly not vacuum
full, just analyze.


Oh, and if what you're doing is actually full text search, which is
what it looks like, you should really look at using the native full
text indexing support rather than just stuffing your words in a table.
You'll get better and much faster results.


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Uninstalling the pl/pgsql debugger

2010-04-05 Thread Raymond O'Donnell
Hello all,

I'm probably missing something obvious here I have the pl/pgsql
debugger installed in template1 on my laptop so that it gets installed
automatically in development databases; however, sometimes I need to
remove it from a specific database.

I can't find an uninstall script for it in the contrib directory, though
there seems to be one for all the other contrib modules. It'd be easy to
write one myself, since there aren't many objects to be dropped; before
I do that, however, is there a quick way of uninstalling the debugger?

Thanks,

Ray.


-- 
Raymond O'Donnell :: Galway :: Ireland
r...@iol.ie

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] How to fix set-valued function called in context that cannot accept a set in earlier versions

2010-04-05 Thread Andrus

In 8.3 code below causes exception

ERROR:  set-valued function called in context that cannot accept a set
CONTEXT:  PL/pgSQL function "wordwrap" line 21 at RETURN NEXT

How to fix this so that wordwrap works in any PostgreSql 8.x version or at 
least in

8.3 and 8.4 ?

Andrus

CREATE OR REPLACE FUNCTION wordwrap(line text, linelen integer)
RETURNS SETOF text as $$
DECLARE
 words text[] := string_to_array(line,' ');
 i integer;
 res text:='';

BEGIN
 if trim(line)='' then
   return next '';
   return;
   end if;
for i IN 1 .. array_upper(words,1) LOOP
  if length(res)+length(words[i]) > linelen THEN
return next res;
res := '';
END IF ;
  if res<>'' then
res := res || ' ';
end if;
  res := res || words[i];
  end loop;
return next res;
END
$$ LANGUAGE plpgsql;

select wordwrap('fdgdf',10) 



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] How to fix set-valued function called in context that cannot accept a set in earlier versions

2010-04-05 Thread hubert depesz lubaczewski
On Mon, Apr 05, 2010 at 02:14:33PM +0300, Andrus wrote:
> select wordwrap('fdgdf',10) 

select * from wordwrap('...', ...);

depesz

-- 
Linkedin: http://www.linkedin.com/in/depesz  /  blog: http://www.depesz.com/
jid/gtalk: dep...@depesz.com / aim:depeszhdl / skype:depesz_hdl / gg:6749007

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] How to fix set-valued function called in context that cannot accept a set in earlier versions

2010-04-05 Thread Kenichiro Tanaka

Hello, Andrus

I tested this reproduce case.

8.3.9   return exception
8.3.10 return exception
8.4.0   Crash Server!!
8.4.1   OK
8.4.2   OK
8.4.3   OK

Just for your information,
8.4.0's clash made this call stack(I analyzed core file)

#0  DirectFunctionCall1 (func=0x8095a70 , arg1=16421) at 
fmgr.c:1012

1012result = (*func) (&fcinfo);
(gdb) where
#0  DirectFunctionCall1 (func=0x8095a70 , arg1=16421) at 
fmgr.c:1012

#1  0x082c9eab in CatalogCacheComputeHashValue (cache=0xa091058,
nkeys=, cur_skey=0xbeca1298) at catcache.c:207
#2  0x082cad19 in SearchCatCache (cache=0xa091058, v1=16421, v2=0, v3=0, 
v4=0)

at catcache.c:1137
#3  0x082df1cf in fmgr_info_cxt_security (functionId=16421, 
finfo=0xbeca144c,

mcxt=, ignore_security=0 '\0') at fmgr.c:209
#4  0x082df881 in fmgr_info (functionId=16421, finfo=0xbeca144c) at 
fmgr.c:156
#5  0x082df282 in fmgr_info_cxt_security (functionId=43373, 
finfo=0xb56c6320,

mcxt=, ignore_security=1 '\001') at fmgr.c:406
#6  0x082e0520 in fmgr_security_definer (fcinfo=0xa0e2f34) at fmgr.c:898
#7  0x082e041c in fmgr_security_definer (fcinfo=0xa0e2f34) at fmgr.c:961
#8  0x082e041c in fmgr_security_definer (fcinfo=0xa0e2f34) at fmgr.c:961

I usually see this call stack  in version 8.4.0 and I guess this error 
does not occur in 8.4.1


Thank you.

(2010/04/05 20:14), Andrus wrote:

In 8.3 code below causes exception

ERROR:  set-valued function called in context that cannot accept a set
CONTEXT:  PL/pgSQL function "wordwrap" line 21 at RETURN NEXT

How to fix this so that wordwrap works in any PostgreSql 8.x version 
or at least in

8.3 and 8.4 ?

Andrus

CREATE OR REPLACE FUNCTION wordwrap(line text, linelen integer)
RETURNS SETOF text as $$
DECLARE
 words text[] := string_to_array(line,' ');
 i integer;
 res text:='';

BEGIN
 if trim(line)='' then
   return next '';
   return;
   end if;
for i IN 1 .. array_upper(words,1) LOOP
  if length(res)+length(words[i]) > linelen THEN
return next res;
res := '';
END IF ;
  if res<>'' then
res := res || ' ';
end if;
  res := res || words[i];
  end loop;
return next res;
END
$$ LANGUAGE plpgsql;

select wordwrap('fdgdf',10)




--

Kenichiro Tanaka
K.K.Ashisuto
http://www.ashisuto.co.jp/english/index.html



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] How to fix set-valued function called in context that cannot accept a set in earlier versions

2010-04-05 Thread Merlin Moncure
On Mon, Apr 5, 2010 at 7:28 AM, hubert depesz lubaczewski
 wrote:
> On Mon, Apr 05, 2010 at 02:14:33PM +0300, Andrus wrote:
>> select wordwrap('fdgdf',10)
>
> select * from wordwrap('...', ...);

another workaround is to wrap the pl/pgsql function in sql function.
it's not always easy to re-work the function all into a 'from'
expression.

merlin

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] join two tables without a key

2010-04-05 Thread Igor Neyman
Dino,

Not trying to be a "database purist" here, but...

If table A has no key, then why X,Y, and Z belong in one table?
And, table B has no key, then why P,Q, and R belong in one table? 

And even more so, why are you trying to put X,Y,Z,P,Q,R into one table?
May be, if you tell us, what business entity/rule you are trying to
implement here, it'll be easier to help you.

Igor Neyman

> -Original Message-
> From: Dino Vliet [mailto:dino_vl...@yahoo.com] 
> Sent: Saturday, April 03, 2010 7:32 AM
> To: r...@iol.ie
> Cc: pgsql-general@postgresql.org
> Subject: Re: join two tables without a key
> 
> --- On Sat, 4/3/10, Raymond O'Donnell  wrote:
> 
> 
> 
>   From: Raymond O'Donnell 
>   Subject: Re: [GENERAL] join two tables without a key
>   To: "Dino Vliet" 
>   Cc: pgsql-general@postgresql.org
>   Date: Saturday, April 3, 2010, 1:01 PM
>   
>   
>   On 03/04/2010 11:16, Dino Vliet wrote:
>   
>   > Hi postgresql list, If I have two tables with the 
> same number of rows
>   > but different columns and I want to create one table 
> out of them what
>   > would be the way to do that in postgresql?
>   > 
>   > Table A has N number of rows and columns X,Y,Z and 
> Table B has N 
>   > number of rows and P,Q,R as columns. None of the tables have a
>   > column which can be used as a key.
>   > 
>   > The resulting table should have N number of rows and 
> X,Y,Z,P,Q,R as 
>   > columns.
>   
>   How do the rows in the tables relate to each other? You 
> need to decide
>   first how you match the rows in A and B.
>   
>   Ray.
>   
>   
>   -- 
>   Raymond O'Donnell :: Galway :: Ireland
>   r...@iol.ie
> 
> 
> 
> Hi Ray,
> 
> 
> 
> 
> 
> 
> 
> they don' t. It' s pure randomly generated data. 
> 
> 
> 
> 
> Brgds
> 
> 
> 
> 
> 
> 

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] How to fix set-valued function called in context that cannot accept a set in earlier versions

2010-04-05 Thread Andrus

Thank you.


another workaround is to wrap the pl/pgsql function in sql function.
it's not always easy to re-work the function all into a 'from'
expression.


I tried code below in 8.3 but got error

ERROR:  syntax error at or near "select"
LINE 30: select * from wordwrap(line,linelen);

Andrus.

CREATE OR REPLACE FUNCTION wordwrap(line text, linelen integer)
RETURNS SETOF text as $$
DECLARE
 words text[] := string_to_array(line,' ');
 i integer;
 res text:='';

BEGIN
 if trim(line)='' then
   return next '';
   return;
   end if;
for i IN 1 .. array_upper(words,1) LOOP
  if length(res)+length(words[i]) > linelen THEN
return next res;
res := '';
END IF ;
  if res<>'' then
res := res || ' ';
end if;
  res := res || words[i];
  end loop;
return next res;
END
$$ LANGUAGE plpgsql;

CREATE OR REPLACE FUNCTION wordwrap83(line text, linelen integer)
RETURNS SETOF text as $$
BEGIN
select * from wordwrap(line,linelen);
END
$$ LANGUAGE sql;


select wordwrap83('fdgdf',10)

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] How to fix set-valued function called in context that cannot accept a set in earlier versions

2010-04-05 Thread hubert depesz lubaczewski
On Mon, Apr 05, 2010 at 05:26:39PM +0300, Andrus wrote:
> Thank you.
>
>> another workaround is to wrap the pl/pgsql function in sql function.
>> it's not always easy to re-work the function all into a 'from'
>> expression.
>
> I tried code below in 8.3 but got error
> CREATE OR REPLACE FUNCTION wordwrap83(line text, linelen integer)
> RETURNS SETOF text as $$
> BEGIN
> select * from wordwrap(line,linelen);
> END
> $$ LANGUAGE sql;

remove begin and end from above function.

Best regards,

depesz

-- 
Linkedin: http://www.linkedin.com/in/depesz  /  blog: http://www.depesz.com/
jid/gtalk: dep...@depesz.com / aim:depeszhdl / skype:depesz_hdl / gg:6749007

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] How to fix set-valued function called in contextthat cannot accept a set in earlier versions

2010-04-05 Thread Andrus

Than you.


remove begin and end from above function.


I tried code belwo in 8.3 but got error 


ERROR:  column "line" does not exist
LINE 29: select * from wordwrap(line,linelen);

Andrus.

CREATE OR REPLACE FUNCTION wordwrap(line text, linelen integer)
RETURNS SETOF text as $$
DECLARE
 words text[] := string_to_array(line,' ');
 i integer;
 res text:='';

BEGIN
 if trim(line)='' then
   return next '';
   return;
   end if;
for i IN 1 .. array_upper(words,1) LOOP
  if length(res)+length(words[i]) > linelen THEN
return next res;
res := '';
END IF ;
  if res<>'' then
res := res || ' ';
end if;
  res := res || words[i];
  end loop;
return next res;
END
$$ LANGUAGE plpgsql;

CREATE OR REPLACE FUNCTION wordwrap83(line text, linelen integer)
RETURNS SETOF text as $$
select * from wordwrap(line,linelen);
$$ LANGUAGE sql;


select wordwrap83('fdgdf',10) 



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] How to fix set-valued function called in context that cannot accept a set in earlier versions

2010-04-05 Thread Andrus

Osvaldo,


CREATE OR REPLACE FUNCTION wordwrap83(line text, linelen integer)
RETURNS SETOF text as $$
select * from wordwrap($1,$2);
$$ LANGUAGE sql;


Thank you. It works.


select * FROM wordwrap83('fdgdf',10)
If you have a "RETURN SETOF" function you must call it with "SELECT *
FROM your_function();" not "SELECT your_function()"


select wordwrap83('fdgdf',10)

works in 8.3 and in 8.4

Andrus.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Null vs. Empty String in Postgres 8.3.8

2010-04-05 Thread Tim Landscheidt
Peter Hunsberger  wrote:

>>> I still don't get it.  I do want a zero for the subversion_flags to be 
>>> stored in the table.  But it returned an error because it didn't like 
>>> subversion_flags='' in the UPDATE SQL statement.

>>> subversion_flags | integer       | not null default 0

>> Right. '' is not 0.  the old version of pgsql converted '' to 0 for
>> you, incorrectly.  Now if you want 0 you need to say 0.

> Or, since you have the default, set it to null (Which may be what
> you thought you where doing?)

Setting it to NULL does not set it to the default value. You
have to use the keyword DEFAULT for that.

Tim


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Temporal data storage

2010-04-05 Thread Gerhard Heift
Hello,

I have person to group mapping in which I want to store the time, when this
mapping is valid. I can store the valid time in an array or in seperate rows.

in rows:

create table group_mapping (
  group name not null,
  person integer not null,
  valid_since timestamptz not null,
  valid_until timestamptz not null
);

insert into group_mapping values('test', 1, '2000-1-1', '2001-1-1');
insert into group_mapping values('test', 1, '2002-1-1', '2002-1-1');

or in an array:

create type period as (
  since timestamptz,
  "until" timestamptz
);

create table group_mapping_array (
  group name not null,
  person integer not null,
  valid_time period[] not null
);

insert into group_mapping_array values('test', 1,
  array[
('2000-1-1', '2001-1-1'),
('2002-1-1', '2003-1-1')
  ]::period[]
);

some advantages and disadvantages:

with rows:
+ a gist index already exists (must be modified)
+ can add other attributes to the valid time
- prevent overlapping is very complex
- binary operations like "and", "or" and "not" operates on multiple rows

with array:
+ overlapping can simply done with a constraint and a function
+ binary operations like "and", "or" and "not" are easy to implement
- a gist index must be written nearly from scratch
- adding other attributes is complex

Are there other (dis)advantages I have forgotten?
Which solution is better?
Are there any drawbacks by using arrays in the rows?

Thanks,
  Gerhard


signature.asc
Description: Digital signature


Re: [GENERAL] How to fix set-valued function called in contextthat cannot accept a set in earlier versions

2010-04-05 Thread Alban Hertroys
On 5 Apr 2010, at 16:33, Andrus wrote:

> Than you.
> 
>> remove begin and end from above function.
> 
> I tried code belwo in 8.3 but got error 
> ERROR:  column "line" does not exist
> LINE 29: select * from wordwrap(line,linelen);
> 
> Andrus.

Yes of course, you don't specify where to take line or linelen from. You 
probably meant to put some constant values there or results from another table.

> select wordwrap83('fdgdf',10) 


^^
These values for example.

Alban Hertroys

--
If you can't see the forest for the trees,
cut the trees and you'll see there is no forest.


!DSPAM:737,4bba1f2b10411047147693!



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] count function alternative in postgres

2010-04-05 Thread Merlin Moncure
On Sat, Apr 3, 2010 at 3:02 PM, Scott Bailey  wrote:
> junaid malik wrote:
>>
>> Is there any alternative of mysql function COUNT(DISTINCT expr,
>> [expr...]) in postgres. We get error if we
>>
>> write count like this count(distinct profile.id, profile.name,
>> profile.age) but it works well in mysql.
>>
>> Reference url is given below
>>
>>
>> http://dev.mysql.com/doc/refman/5.1/en/group-by-functions.html#function_count-distinct
>>
>> Thanks
>>
>
> I already answered your question on dbforums.com. But in Postgres you can
> do:
>
> SELECT COUNT( DISTINCT row(col1, col2, col3) ) FROM foo
> or
> SELECT COUNT( DISTINCT (col1, col2, col3) ) FROM foo

very clever!  This is similar to how I use rowtypes to get around the
single column restrictions on function calls in the select field list.

merlin

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] count function alternative in postgres

2010-04-05 Thread Tom Lane
Merlin Moncure  writes:
> On Sat, Apr 3, 2010 at 3:02 PM, Scott Bailey  wrote:
>> SELECT COUNT( DISTINCT row(col1, col2, col3) ) FROM foo

> very clever!  This is similar to how I use rowtypes to get around the
> single column restrictions on function calls in the select field list.

Cute, but note it will only work in 8.4 or later ...

regards, tom lane

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] windows 7 compatiblity?

2010-04-05 Thread Heine Ferreira
Hi

I recently bought Windows 7 32 bit professional.
Unfortunately Postgresql 8.4.3 did not install or work.
I started searching Google and found that a lot of other people have the
same problem.
Will there be a minor release to fix the problem?
Will it be fixed in the upcoming version 9.0?

Thanks

H.F.


Re: [GENERAL] windows 7 compatiblity?

2010-04-05 Thread Pavel Stehule
Hello

2010/4/5 Heine Ferreira :
> Hi
>
> I recently bought Windows 7 32 bit professional.
> Unfortunately Postgresql 8.4.3 did not install or work.
> I started searching Google and found that a lot of other people have the
> same problem.
> Will there be a minor release to fix the problem?
> Will it be fixed in the upcoming version 9.0?

It can depend on win configuration maybe. Last week I did lectures on
Windows 7 with PostgreSQL. There was no problem.

regards
Pavel Stehule

>
> Thanks
>
> H.F.
>
>

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] desktop heap usage in windows

2010-04-05 Thread Heine Ferreira
Hi

In the latest release 8.4.3 is the desktop heap still a problem in Windows?
I found some articles on the web that said on Windows xp the desktop heap is
rather small.
Apparantly under xp it's limited to 45 connections.
They also said that the desktop heap is larger on windows servers and that
gives you about 125 connections on a windows server.
They said it's possible to edit the registry to make the desktop heap larger
but if you make it too large then your machine won't boot.
So how many connections can windows xp and vista handle if you make the
desktop heap as large as possible?
How many connections can a windows server handle if you make the desktop
heap as large as possible?

Thanks

H.F.


Re: [GENERAL] windows 7 compatiblity?

2010-04-05 Thread Frank Heikens



Hi

I recently bought Windows 7 32 bit professional.
Unfortunately Postgresql 8.4.3 did not install or work.


Tell us about your problems, error messages and logs, then we can help  
you to get it working.


I started searching Google and found that a lot of other people have  
the same problem.


Many others have it working, but that's normal and that's why they  
didn't post any message on a forum.



Will there be a minor release to fix the problem?


What problem? Tell us about it!


Will it be fixed in the upcoming version 9.0?


Why waiting? Maybe we can help you now.



Thanks

H.F.





Regards,
Frank


[GENERAL] Fuzzy string matching of product names

2010-04-05 Thread Peter Geoghegan
Hello,

At the moment, users of my application, which runs on 8.4.3, may
search for products in a way that is implemented roughly like this:

SELECT * FROM products WHERE description ILIKE '%%usr_string%%';

This works reasonably well. However, I thought it would be a nice
touch to give my users leeway to spell product names incorrectly when
searching, or to not have to remember if a product is entered as "coca
cola", "CocaCola" or "Coca-cola". At the moment, they don't have to
worry about case sensitivity because I use ILIKE - I'd like to
preserve that. I'd also like to not have it weigh against them heavily
when they don't search for a specific product, but just a common
substring. For example, if they search for "coca-cola", there may be a
number of different coca-cola products: "CocaCola 330ml can",
"Coca-Cola 2 litre bottle", but no actual plain "cocacola". That ought
to not matter too much - all cocacola products should be returned.

This isn't important enough for me to be willing to add a big
dependency to my application. I'd really prefer to limit myself to the
contrib modules. pg_trgm and fuzzystrmatch look very promising, but
it's not obvious how I can use either to achieve what I want.
Postgres's built-in regex support may have a role to play too.

I can live with it not being indexable, because typically there are
only tens of thousands of products in a production system.

Could someone suggest an approach that is reasonably simple and
reasonably generic ?

Thanks,
Peter Geoghegan

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Fuzzy string matching of product names

2010-04-05 Thread Bill Moran
In response to Peter Geoghegan :

> Hello,
> 
> At the moment, users of my application, which runs on 8.4.3, may
> search for products in a way that is implemented roughly like this:
> 
> SELECT * FROM products WHERE description ILIKE '%%usr_string%%';
> 
> This works reasonably well. However, I thought it would be a nice
> touch to give my users leeway to spell product names incorrectly when
> searching, or to not have to remember if a product is entered as "coca
> cola", "CocaCola" or "Coca-cola". At the moment, they don't have to
> worry about case sensitivity because I use ILIKE - I'd like to
> preserve that. I'd also like to not have it weigh against them heavily
> when they don't search for a specific product, but just a common
> substring. For example, if they search for "coca-cola", there may be a
> number of different coca-cola products: "CocaCola 330ml can",
> "Coca-Cola 2 litre bottle", but no actual plain "cocacola". That ought
> to not matter too much - all cocacola products should be returned.
> 
> This isn't important enough for me to be willing to add a big
> dependency to my application. I'd really prefer to limit myself to the
> contrib modules. pg_trgm and fuzzystrmatch look very promising, but
> it's not obvious how I can use either to achieve what I want.
> Postgres's built-in regex support may have a role to play too.
> 
> I can live with it not being indexable, because typically there are
> only tens of thousands of products in a production system.
> 
> Could someone suggest an approach that is reasonably simple and
> reasonably generic ?

http://www.postgresql.org/docs/8.4/static/fuzzystrmatch.html

-- 
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Fuzzy string matching of product names

2010-04-05 Thread Brian Modra
On 05/04/2010, Peter Geoghegan  wrote:
> Hello,
>
> At the moment, users of my application, which runs on 8.4.3, may
> search for products in a way that is implemented roughly like this:
>
> SELECT * FROM products WHERE description ILIKE '%%usr_string%%';
>
> This works reasonably well. However, I thought it would be a nice
> touch to give my users leeway to spell product names incorrectly when
> searching, or to not have to remember if a product is entered as "coca
> cola", "CocaCola" or "Coca-cola". At the moment, they don't have to
> worry about case sensitivity because I use ILIKE - I'd like to
> preserve that. I'd also like to not have it weigh against them heavily
> when they don't search for a specific product, but just a common
> substring. For example, if they search for "coca-cola", there may be a
> number of different coca-cola products: "CocaCola 330ml can",
> "Coca-Cola 2 litre bottle", but no actual plain "cocacola". That ought
> to not matter too much - all cocacola products should be returned.
>
> This isn't important enough for me to be willing to add a big
> dependency to my application. I'd really prefer to limit myself to the
> contrib modules. pg_trgm and fuzzystrmatch look very promising, but
> it's not obvious how I can use either to achieve what I want.
> Postgres's built-in regex support may have a role to play too.
>
> I can live with it not being indexable, because typically there are
> only tens of thousands of products in a production system.
>
> Could someone suggest an approach that is reasonably simple and
> reasonably generic ?

What I do is to create another column that has a simplified version of
the string in it.
(I created a function to simplify strings, and when the source column
is changed or inserted, I also update the "simplified" column.
Then when searching, I use the same function to "simplify" the search
string and use "=" to test against the "simplified" column.

E.g.
if the table has a column called "name" that you want to search, you
create a name_simplified column, and fill it as so:
update your_table set name_simplified=yourSimplifyFunction(name);

Then to search:
select * from your_table where simplified_name =
yourSimplifyFunction('Coca-Cola');

This is really fast, because the match is using the index rather than
a sequential scan.

>
> Thanks,
> Peter Geoghegan
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>


-- 
Brian Modra   Land line: +27 23 5411 462
Mobile: +27 79 69 77 082
5 Jan Louw Str, Prince Albert, 6930
Postal: P.O. Box 2, Prince Albert 6930
South Africa
http://www.zwartberg.com/

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] windows 7 compatiblity?

2010-04-05 Thread Garry Saddington

Frank Heikens wrote:



Hi

I recently bought Windows 7 32 bit professional.
Unfortunately Postgresql 8.4.3 did not install or work.


Tell us about your problems, error messages and logs, then we can help 
you to get it working.


I started searching Google and found that a lot of other people have 
the same problem.


Many others have it working, but that's normal and that's why they 
didn't post any message on a forum.



Will there be a minor release to fix the problem?


What problem? Tell us about it!


Will it be fixed in the upcoming version 9.0?




I had a problem installing my software (which includes Postgresql) on a 
Windows 7 demo laptop last week. It turned out that accounts that are 
designated as administrator do not have enough privileges to install as 
a service. Without wishing to go delving into the config tools and 
wasting any more time I logged out and then back in using the 
Administrator account that came with W7 (no password - security, what 
security?), ran the setup again and everything worked fine. The services 
were then available to all other accounts.

HTH, just my experience.
Garry





--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Fuzzy string matching of product names

2010-04-05 Thread George Silva
The above is true. For geocoding the same idea is used: the metaphone
function is used against street names, and searched to a simples column,
filled with the results of the metaphone function. It works quite well.

George

On Mon, Apr 5, 2010 at 4:23 PM, Brian Modra  wrote:

> On 05/04/2010, Peter Geoghegan  wrote:
> > Hello,
> >
> > At the moment, users of my application, which runs on 8.4.3, may
> > search for products in a way that is implemented roughly like this:
> >
> > SELECT * FROM products WHERE description ILIKE '%%usr_string%%';
> >
> > This works reasonably well. However, I thought it would be a nice
> > touch to give my users leeway to spell product names incorrectly when
> > searching, or to not have to remember if a product is entered as "coca
> > cola", "CocaCola" or "Coca-cola". At the moment, they don't have to
> > worry about case sensitivity because I use ILIKE - I'd like to
> > preserve that. I'd also like to not have it weigh against them heavily
> > when they don't search for a specific product, but just a common
> > substring. For example, if they search for "coca-cola", there may be a
> > number of different coca-cola products: "CocaCola 330ml can",
> > "Coca-Cola 2 litre bottle", but no actual plain "cocacola". That ought
> > to not matter too much - all cocacola products should be returned.
> >
> > This isn't important enough for me to be willing to add a big
> > dependency to my application. I'd really prefer to limit myself to the
> > contrib modules. pg_trgm and fuzzystrmatch look very promising, but
> > it's not obvious how I can use either to achieve what I want.
> > Postgres's built-in regex support may have a role to play too.
> >
> > I can live with it not being indexable, because typically there are
> > only tens of thousands of products in a production system.
> >
> > Could someone suggest an approach that is reasonably simple and
> > reasonably generic ?
>
> What I do is to create another column that has a simplified version of
> the string in it.
> (I created a function to simplify strings, and when the source column
> is changed or inserted, I also update the "simplified" column.
> Then when searching, I use the same function to "simplify" the search
> string and use "=" to test against the "simplified" column.
>
> E.g.
> if the table has a column called "name" that you want to search, you
> create a name_simplified column, and fill it as so:
> update your_table set name_simplified=yourSimplifyFunction(name);
>
> Then to search:
> select * from your_table where simplified_name =
> yourSimplifyFunction('Coca-Cola');
>
> This is really fast, because the match is using the index rather than
> a sequential scan.
>
> >
> > Thanks,
> > Peter Geoghegan
> >
> > --
> > Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> > To make changes to your subscription:
> > http://www.postgresql.org/mailpref/pgsql-general
> >
>
>
> --
> Brian Modra   Land line: +27 23 5411 462
> Mobile: +27 79 69 77 082
> 5 Jan Louw Str, Prince Albert, 6930
> Postal: P.O. Box 2, Prince Albert 6930
> South Africa
> http://www.zwartberg.com/
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>



-- 
George R. C. Silva

Desenvolvimento em GIS
http://blog.geoprocessamento.net


Re: [GENERAL] Fuzzy string matching of product names

2010-04-05 Thread Peter Geoghegan
> http://www.postgresql.org/docs/8.4/static/fuzzystrmatch.html
>
> --
> Bill Moran
> http://www.potentialtech.com
> http://people.collaborativefusion.com/~wmoran/

Fuzzystrmatch is generally used to compare two single words for how
similar they sound. How can that actually be applied to get the
functionality that I've described?

Regards,
Peter Geoghegan

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] windows 7 compatiblity?

2010-04-05 Thread John R Pierce

Garry Saddington wrote:
I had a problem installing my software (which includes Postgresql) on 
a Windows 7 demo laptop last week. It turned out that accounts that 
are designated as administrator do not have enough privileges to 
install as a service. Without wishing to go delving into the config 
tools and wasting any more time I logged out and then back in using 
the Administrator account that came with W7 (no password - security, 
what security?), ran the setup again and everything worked fine. The 
services were then available to all other accounts.



quick and dirty way around that without logging out and back in...

right click the postgres installer, "Run as administrator..." and Yes to 
the UAC popup that ensues, and after this it should install just fine.   
Personally, I would make sure I specified a different directory for the 
data  (its quite possible you *have* to as \program files\ is restricted 
in win7)






--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Fuzzy string matching of product names

2010-04-05 Thread Bill Moran
In response to Peter Geoghegan :

> > http://www.postgresql.org/docs/8.4/static/fuzzystrmatch.html
> >
> > --
> > Bill Moran
> > http://www.potentialtech.com
> > http://people.collaborativefusion.com/~wmoran/
> 
> Fuzzystrmatch is generally used to compare two single words for how
> similar they sound. How can that actually be applied to get the
> functionality that I've described?

Well, it really depends on your particular situation and what you
want to support.  You could break the name down into individual
words and generate metaphones, then use like to match on metaphone:

'The Candlestick Corporation, Limited' -> 'TE CDSK CPRN LMTD'

Searching for "candlestick" -> WHERE metaphone column like '%CDSK%'

Or you could create an array column that has all the metaphones in
it and use an ANY() or ALL() match to find ones that apply.

Exactly how you implement depends on how far you want to go.  Do you
want to support OR matches, AND matches, or both?  Can the words be
out of order?

You could also use Levenshtein as a percentage function to find matches,
even on long strings with multiple words.  Since Levenshtein gives you
the number of alterations between two strings, using that as a percentage
of the total string length gives you a pretty good gauge of how close
they are overall, and would allow you to set a threshold, or possibly even
list results by relevance.

-- 
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Fuzzy string matching of product names

2010-04-05 Thread Leif Biberg Kristensen
On Monday 5. April 2010 22.00.41 Peter Geoghegan wrote:
> similar they sound. How can that actually be applied to get the
> functionality that I've described?

I've got a similar problem in my 18th century research, when clerks usually 
took pride in being able to spell a name in any number of ways. I've landed on 
a solution where I'm sending search strings to SIMILAR TO. I usually get far 
too many hits, but it's much easier to browse through 100 hits than the entire 
dataset which is approaching 60,000 records.

Optimizing the search strings is based upon a lot of experience.

It would probably be better to add a column with normalized names, but the 
amount of work involved with that is staggering. I eventually associate most 
of the records to «persons» with normalized names, but the search process can 
sometimes be very frustrating, and it would really help with some kind of 
fuzzy search.

Just in case anyone should suggest it: Both Soundex and Metaphone are useless 
for Norwegian 18th century names.

regards,
-- 
Leif Biberg Kristensen
http://solumslekt.org/

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] windows 7 compatiblity?

2010-04-05 Thread John R Pierce

John R Pierce wrote:

Garry Saddington wrote:
I had a problem installing my software (which includes Postgresql) on 
a Windows 7 demo laptop last week. It turned out that accounts that 
are designated as administrator do not have enough privileges to 
install as a service. Without wishing to go delving into the config 
tools and wasting any more time I logged out and then back in using 
the Administrator account that came with W7 (no password - security, 
what security?), ran the setup again and everything worked fine. The 
services were then available to all other accounts.



quick and dirty way around that without logging out and back in...

right click the postgres installer, "Run as administrator..." and Yes 
to the UAC popup that ensues, and after this it should install just 
fine.   Personally, I would make sure I specified a different 
directory for the data  (its quite possible you *have* to as \program 
files\ is restricted in win7)



k, you HAVE to change the default dir, I went and installed to 
c:\postgresql\8.4   and its alive and running.


(ok, I'm on win7 64bit professional, but should be same difference).



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] desktop heap usage in windows

2010-04-05 Thread John R Pierce

Heine Ferreira wrote:

Hi

In the latest release 8.4.3 is the desktop heap still a problem in 
Windows?
I found some articles on the web that said on Windows xp the desktop 
heap is rather small.

Apparantly under xp it's limited to 45 connections.


what does the desktop heap (a GUI thing used to manage window handles) 
have to do with network connections?   do you perhaps mean the kernel 
paged and non-paged pools?   

Windows desktop versions (pro, home, etc) are intentionally limited with 
respect to how many network connections can be made as they are sold as 
workstations and not servers.   the biggest limit is on 'half open' 
connections, which impacts how many connections can be initiated at the 
same time.




--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Connection Pooling

2010-04-05 Thread David Kerr
On Sat, Apr 03, 2010 at 09:32:25PM -0400, Merlin Moncure wrote:
- On Fri, Mar 26, 2010 at 5:17 PM, David Kerr  wrote:
- > Howdy all,
- >
- > I have some apps that are connecting to my DB via direct JDBC and I'd like 
to pool their connections.
- >
- > I've been looking at poolers for a while, and pgbouncer and pgpool-ii seem 
to be some of the most popular, so
- > i've started with those.
- >
- >
- > I'm setting up pgbouncer, and i've hit a bit of a snag. Hopefully someone 
can tell me if pgbouncer or pgpool are
- > capable of this (and if so, how to do it) or alternatly a pooler that can...
- >
- > What I'd like to be able to do is this (not using pooler syntax, this is 
just a high level of what i want to achive)
- >
- > Say i set max pool size = 10 connections. and max # of pools = 5.
- >
- > That means that i should have 5 connections to my database covering 50 
connections total.
- >
- > I can't really seem to make that work with pgbouncer without naming the 
pools separetly. (pool1 = dbname = a, pool2 = dbname =a)
- > which means my app is tied to a pool (or has to specifically code to rotate 
pools...) which is not really desireable.
- 
- I have a lot of respect for pgbouncer (haven't used pgpool).  One
- possible way to do what you're thinking is to rotate the pool on user.
-  In bouncer each database role gets its own pool (if you understand
- how transaction mode works you can see why it has to work this way).
- Not sure if this is helpful.  Why are you trying to separate the pools
- like that?
- 
- merlin

Based on a lot of the comments i've gotten here, I'm starting to think that 
I've got the wrong idea about
connection pools and pooling in general. So, let me lay out some of my 
assumptions and my problem and
maybe we can go from there...

My app will have over 10k concurrent users. I have huge servers 32 cores 
(64bit), 64GB ram. RedHat linux.

Those 10k users will all be logging in as one of 5 application users. 

>From following this list, and talking to our PG consultants, I know that for 
>that many connecitons I need 
to have a connection pooler. Because with postgres you shouldn't set 
max_connections much higher than 2000 
(which is pushing it) 

For 4 out of the 5 applications, we're using Geronimo which has it's own pooler 
in the manner that
I've described above. 

For the 5th application, an ETL job that could have as many as 1000 concurrent 
processes/connections,
i don't have a java container. it's just a raw jar file that gets run via java 
.

That's what I'm aiming to pool, and i'd like to do it without modifying the 
code if possible.

Thanks

Dave

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] 'Anneal' <> 'Anneal'

2010-04-05 Thread Rob Richardson
Greetings!
 
I have the following query:
 
select charge.charge, alarm_history.area || '!', bases.area || '!',
alarm_history.area::character varying(32) = bases.area::character
varying(32), alarm_history.area <> bases.area, alarm_history.*
from charge
JOIN bases ON charge.base::text = bases.base::text
inner join alarm_history 
on charge.charge = alarm_history.charge 
or 
(
 alarm_history.alarm_date > charge.fire_date 
 and 
 (
  alarm_history.alarm_date < charge.cold_date 
  or charge.cold_date is null
 )
 and device = 'Global')
where charge.charge = 64489 
 
This query returns 9 records.  For all of them, alarm_history.area is
'Anneal' and bases.area is 'Anneal'.  Yet, the column for the equality
test is always "f" and the column for the inequality test is always "t"!

 
The alarm_history table definition begins with:
 
CREATE TABLE alarm_history
(
  alarm_pkey bigserial NOT NULL,
  area character varying(32), -- shop area

and the bases table definition has:
 
CREATE TABLE bases
(
  area character varying(32) NOT NULL DEFAULT 'All'::character varying,

I have no idea why these two look to be different.  I've tried appending
an exclamation point to both of them to check for extra spaces at the
end.  There aren't any.  I've tried casting them both to text and to
character varying(32), but I got the same results.  
 
I'm running PostgreSQL 8.4 on WinXP Pro.  I get the same results on a
customer's machine running a 64-bit Windows OS, using an identical
database.  
 
Can anyone explain this, please?
 
 
RobR
 


Re: [GENERAL] 'Anneal' <> 'Anneal'

2010-04-05 Thread Rob Richardson
Thank you.  That got it.  It gave the expected result of false for the
inequality check.  I took a second look at the output of the query and
saw the extra space in front of 'Anneal' in the alarm_history table's
field.

RobR 

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] desktop heap usage in windows

2010-04-05 Thread Magnus Hagander
On Mon, Apr 5, 2010 at 22:33, John R Pierce  wrote:
> Heine Ferreira wrote:
>>
>> Hi
>>
>> In the latest release 8.4.3 is the desktop heap still a problem in
>> Windows?
>> I found some articles on the web that said on Windows xp the desktop heap
>> is rather small.
>> Apparantly under xp it's limited to 45 connections.
>
> what does the desktop heap (a GUI thing used to manage window handles) have
> to do with network connections?   do you perhaps mean the kernel paged and
> non-paged pools?
> Windows desktop versions (pro, home, etc) are intentionally limited with
> respect to how many network connections can be made as they are sold as
> workstations and not servers.   the biggest limit is on 'half open'
> connections, which impacts how many connections can be initiated at the same
> time.
>

No, there's a problem with the Desktop Heap getting exhausted because
of system DLLs that allocate from it whenever they are loaded into a
process. It's in the Windows FAQ on wiki.postgresql.org - IIRC it's
right at the bottom.

And yes, we still have the problem - it's a Windows problem, really,
not a PostgreSQL problem. It simply can't deal with many processes the
way the system DLLs are built. The exact limit varies between
installations - likely due to third party DLLs being loaded, like AV
and FW software.

It may be that this is actually not a problem with the 64-bit version
of 9.0. I haven't tried that - probably should.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] desktop heap usage in windows

2010-04-05 Thread John R Pierce

Magnus Hagander wrote:

No, there's a problem with the Desktop Heap getting exhausted because
of system DLLs that allocate from it whenever they are loaded into a
process. It's in the Windows FAQ on wiki.postgresql.org - IIRC it's
right at the bottom.

And yes, we still have the problem - it's a Windows problem, really,
not a PostgreSQL problem. It simply can't deal with many processes the
way the system DLLs are built. The exact limit varies between
installations - likely due to third party DLLs being loaded, like AV
and FW software.

It may be that this is actually not a problem with the 64-bit version
of 9.0. I haven't tried that - probably should.
  


k, so the FAQ entry here [1]  references the MSKB article [2] which, 
under Cause 2, talks about the SharedSection in the registry key [3]


Presumably, tis the 3rd value, the 512, thats used by non-interactive 
sessions, thats the one that needs boosting?


fwiw, on my win7 pro 64bit desktop, that reads...

   %SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows 
SharedSection=1024,20480,768
   Windows=On SubSystemType=Windows ServerDll=basesrv,1 
ServerDll=winsrv:UserServerDllInitialization,3
   ServerDll=winsrv:ConServerDllInitialization,2 ServerDll=sxssrv,4 
ProfileControl=Off

   MaxRequestThreads=16

which is larger (768) than the value shown in the KB article [2] (512)



[1] 
http://wiki.postgresql.org/wiki/Running_%26_Installing_PostgreSQL_On_Native_Windows#I_cannot_run_with_more_than_about_125_connections_at_once.2C_despite_having_capable_hardware


[2] http://support.microsoft.com/kb/184802

[3] HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session 
Manager\SubSystems\Windows




--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] windows 7 compatiblity?

2010-04-05 Thread Dave Page
On 4/5/10, John R Pierce  wrote:
> John R Pierce wrote:
>> Garry Saddington wrote:
>>> I had a problem installing my software (which includes Postgresql) on
>>> a Windows 7 demo laptop last week. It turned out that accounts that
>>> are designated as administrator do not have enough privileges to
>>> install as a service. Without wishing to go delving into the config
>>> tools and wasting any more time I logged out and then back in using
>>> the Administrator account that came with W7 (no password - security,
>>> what security?), ran the setup again and everything worked fine. The
>>> services were then available to all other accounts.
>>
>>
>> quick and dirty way around that without logging out and back in...
>>
>> right click the postgres installer, "Run as administrator..." and Yes
>> to the UAC popup that ensues, and after this it should install just
>> fine.   Personally, I would make sure I specified a different
>> directory for the data  (its quite possible you *have* to as \program
>> files\ is restricted in win7)
>
>
> k, you HAVE to change the default dir, I went and installed to
> c:\postgresql\8.4   and its alive and running.
>

if that were the case, there is no way the installer would have got
through even basic sanity testing, never mind the barrage of tests
every release does go through successfully.

Can we see the installation log please?


-- 
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Connection Pooling

2010-04-05 Thread Merlin Moncure
On Mon, Apr 5, 2010 at 4:36 PM, David Kerr  wrote:
> On Sat, Apr 03, 2010 at 09:32:25PM -0400, Merlin Moncure wrote:
> -
> - I have a lot of respect for pgbouncer (haven't used pgpool).  One
> - possible way to do what you're thinking is to rotate the pool on user.
> -  In bouncer each database role gets its own pool (if you understand
> - how transaction mode works you can see why it has to work this way).
> - Not sure if this is helpful.  Why are you trying to separate the pools
> - like that?
> -
> - merlin
>
> Based on a lot of the comments i've gotten here, I'm starting to think that 
> I've got the wrong idea about
> connection pools and pooling in general. So, let me lay out some of my 
> assumptions and my problem and
> maybe we can go from there...
>
> My app will have over 10k concurrent users. I have huge servers 32 cores 
> (64bit), 64GB ram. RedHat linux.
>
> Those 10k users will all be logging in as one of 5 application users.
>
> From following this list, and talking to our PG consultants, I know that for 
> that many connecitons I need
> to have a connection pooler. Because with postgres you shouldn't set 
> max_connections much higher than 2000
> (which is pushing it)

This is correct.  If you go with pgbouncer, you would want to use
transaction pooling mode obviously.

> For the 5th application, an ETL job that could have as many as 1000 
> concurrent processes/connections,
> i don't have a java container. it's just a raw jar file that gets run via 
> java .
>
> That's what I'm aiming to pool, and i'd like to do it without modifying the 
> code if possible.

pgbouncer is totally transparent.  I manage quite a few databases and
I use it (w/session mode) so I can psql to a single host (localhost),
and bounce between different databases.  Like I said earlier, you have
discreet pools based on role -- otherwise there would be no really
good way to control the role your queries would operate under.  This
will keep your etl from drilling your box, and if you keep your
pool_size under the number of cores you have, you will have some
available if things get really dicey, which is nice.

caveats:
*) no openssl, but stunnel works well (you may have to grab a recent stunnel)
*) transaction mode blocks use of certain database features, like
notifies.  again, doesn't sound too bad for you

doesn't sound like you need openssl though.  If you have the ability
to set up a test environment, I'd set it up and give it a shot.

merlin

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Connection Pooling

2010-04-05 Thread Scott Marlowe
On Mon, Apr 5, 2010 at 2:36 PM, David Kerr  wrote:
> My app will have over 10k concurrent users. I have huge servers 32 cores 
> (64bit), 64GB ram. RedHat linux.
>
> Those 10k users will all be logging in as one of 5 application users.

You should probably also look into memcached to take a lot of the read
load off of your databases.

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general



Re: [GENERAL] "1-Click" installer problems

2010-04-05 Thread Craig Ringer
Nikhil,

Any movement on collecting some system comparision information so we can
start to figure out why the installer works on my systems, but not on yours?

I've just tested on a couple of other Vista systems (at work) without
problems, so there must be something different about your systems and
those of the others you've seen complain on the EDB forums. It'd be good
to find out what it is.

--
Craig Ringer

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] "1-Click" installer problems

2010-04-05 Thread Nikhil G. Daddikar

Hi Craig,

I have a call with EDB folks in an hour. I will post when I have 
something concrete. Meanwhile, if you need anything specific about the 
system, please let me know.


I have Vista Business x64 with Microsoft Security Essentials and UAC 
turned ON. I have tried (at least) the following:


  1. Default install of PG 8.4
  2. Install of PG 8.4 in C:\Postgresql
  3. Disable MS Security Essentials for C:\Postgresql and reinstall
  4. Create a regular (non-admin) posgres user before installation
  5. Delete that user and redo installation
  6. Disable UAC and install in C:\Postgresql as well as in the default
 directory
  7. Enabled the "Administrator" user in Vista, logged in as that user
 and installed

If you want me to try anything else, I can do that too.

Thanks,
Nikhil

On 06-04-2010 09:31, Craig Ringer wrote:

Nikhil,

Any movement on collecting some system comparision information so we can
start to figure out why the installer works on my systems, but not on yours?

I've just tested on a couple of other Vista systems (at work) without
problems, so there must be something different about your systems and
those of the others you've seen complain on the EDB forums. It'd be good
to find out what it is.

--
Craig Ringer

   




Re: [GENERAL] "1-Click" installer problems

2010-04-05 Thread Nikhil G. Daddikar



I have tried earlier versions of 8.4 installer on Vista x32 and they'd 
failed too. But let me try the latest installer again. I will let you know.


-n.

On 06-04-2010 10:09, Craig Ringer wrote:

Nikhil G. Daddikar wrote:
   

  Hi Craig,

I have a call with EDB folks in an hour. I will post when I have
something concrete. Meanwhile, if you need anything specific about the
system, please let me know.

I have Vista Business x64
 

Hmm. Your setup is the same as mine except that all the hosts I've
tested on are 32-bit installs. Interesting.

Unfortunately our friends at Microsoft don't let you simply re-install a
64-bit version, you have to go and buy the OS all over again at absurd
retail prices. I don't have an MSDN subscription at work. So I can't
personally test with x64 versions.

That's certainly something to start looking at, though. I'm sure that if
it was as simple as "the installer is broken on x64" it would've turned
up by now, but perhaps there's something else that only causes a problem
when installs are run on x64 hosts. Something to do with pathname
rewriting - Program Files (x86) for example?

--
Craig Ringer
   



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] "1-Click" installer problems

2010-04-05 Thread Craig Ringer
Nikhil G. Daddikar wrote:
>  Hi Craig,
> 
> I have a call with EDB folks in an hour. I will post when I have
> something concrete. Meanwhile, if you need anything specific about the
> system, please let me know.
> 
> I have Vista Business x64

Hmm. Your setup is the same as mine except that all the hosts I've
tested on are 32-bit installs. Interesting.

Unfortunately our friends at Microsoft don't let you simply re-install a
64-bit version, you have to go and buy the OS all over again at absurd
retail prices. I don't have an MSDN subscription at work. So I can't
personally test with x64 versions.

That's certainly something to start looking at, though. I'm sure that if
it was as simple as "the installer is broken on x64" it would've turned
up by now, but perhaps there's something else that only causes a problem
when installs are run on x64 hosts. Something to do with pathname
rewriting - Program Files (x86) for example?

--
Craig Ringer

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general