Re: [GENERAL] [Skytools-users] WAL Shipping + checkpoint

2009-08-27 Thread Sébastien Lardière

On 27/08/2009 00:18, Mark Kirkwood wrote:

Sébastien Lardière wrote:

On 26/08/2009 04:46, Mark Kirkwood wrote:

Sébastien Lardière wrote:

Hi All,

I've a cluster ( Pg 8.3.7 ) with WAL Shipping, and a few hours ago, 
the master had to restart.


I use walmgr from Skytools, which works very well.

I have already restart the master without any problem, but today, 
the slave doesn't work like I want. The field Time of latest 
checkpoint from the pg_controldata on the slave keep the same 
values, but WAL File are processed correctly.


I try to restart the slave, but, after processed again all the WAL 
between Time of latest checkpoint and, it does nothing else, 
latest checkpoint stay at the same value.


I don't know if it's important ( i think so ), and I can't fix it.

It is normal for it to lag behind somewhat on the slave (depending 
on what your checkpoint timeout etc settings are).


However, I've noticed what you are seeing as well - particularly 
when there are no actual data changes coming through in the logs - 
the slave checkpoint time does not change even tho there have been 
checkpoints on the master (I may have a look in the code to see what 
the story really is...if I have time).




Yes, but the delay between the last checkpoint on the master and the 
slave is very high, now ( 100 000 sec ), because the last checkpoint 
on the slave was yesterday ( as far as pg_controldata is right )


Here a graph from our munin plugin : 
http://seb.ouvaton.org/tmp/bdd-pg_walmgr-week.png


The blue line represent an average between two WAL processed on the 
slave, and the green line, the delai between last checkpoint on the 
master and the slave.


Maybe it's not some good indicator, but the green line let me think 
there is problem.



Do you have archive_timeout set? If so, then what *could* be happening 
is this:


There are actually no real data changes being made on your master 
for some reason. So every time archive_timeout is reached a log full 
of no changes is shipped to your slave and applied - and no checkpoint 
times are changed for reasons I mentioned above.





thanks, but we have not set archive_timeout, and we have a lot of real 
data changes.


That's why i don't understand why checkpoint never happen on the slave.

--
Sébastien Lardière



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Viable alternatives to SQL?

2009-08-27 Thread Kelly Jones
Many sites let you search databases of information, but the search
queries are very limited.

I'm creating a site that'll allow arbitrary SQL queries to my data (I
realize I'll need to handle injection attacks).

Are there other viable ways to query data? I read a little on
Business System 12 (BS12), Tutorial D, and even something called
T-SQL (I think), but they all seem theoretical and not fully
implemented.

I want a query language that non-techies can use easily, but also
supports arbitrarily complex queries. Does such a language exist?

-- 
We're just a Bunch Of Regular Guys, a collective group that's trying
to understand and assimilate technology. We feel that resistance to
new ideas and technology is unwise and ultimately futile.

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Viable alternatives to SQL?

2009-08-27 Thread Martin Gainty

Hibernate is a ORM technology which configures bean classes which will 
construct DB statements based on the method(create,replace,update,delete) from 
the bean class attributes(columns)

to quote
Hibernate allows you to express queries in its own portable SQL
extension (HQL), as well as in native SQL, or with an object-oriented
Criteria and Example API.

Hibernate currently supports the following databases
Microsoft SQL Server 2005/2000OracleMicrosoft AccessFirebirdPostgreSQLDB2 
UDBMySQLSQLite
https://www.hibernate.org/

Martin Gainty 
__ 
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
 
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger 
sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung 
oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem 
Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. 
Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung 
fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le 
destinataire prévu, nous te demandons avec bonté que pour satisfaire informez 
l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est 
interdite. Ce message sert à l'information seulement et n'aura pas n'importe 
quel effet légalement obligatoire. Étant donné que les email peuvent facilement 
être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité 
pour le contenu fourni.




 Date: Thu, 27 Aug 2009 06:43:51 -0700
 Subject: [GENERAL] Viable alternatives to SQL?
 From: kelly.terry.jo...@gmail.com
 To: pgsql-general@postgresql.org
 
 Many sites let you search databases of information, but the search
 queries are very limited.
 
 I'm creating a site that'll allow arbitrary SQL queries to my data (I
 realize I'll need to handle injection attacks).
 
 Are there other viable ways to query data? I read a little on
 Business System 12 (BS12), Tutorial D, and even something called
 T-SQL (I think), but they all seem theoretical and not fully
 implemented.
 
 I want a query language that non-techies can use easily, but also
 supports arbitrarily complex queries. Does such a language exist?
 
 -- 
 We're just a Bunch Of Regular Guys, a collective group that's trying
 to understand and assimilate technology. We feel that resistance to
 new ideas and technology is unwise and ultimately futile.
 
 -- 
 Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-general

_
Windows Live: Keep your friends up to date with what you do online.
http://windowslive.com/Campaign/SocialNetworking?ocid=PID23285::T:WLMTAGL:ON:WL:en-US:SI_SB_online:082009

Re: [GENERAL] Viable alternatives to SQL?

2009-08-27 Thread Peter Hunsberger
On Thu, Aug 27, 2009 at 8:43 AM, Kelly Joneskelly.terry.jo...@gmail.com wrote:
 Many sites let you search databases of information, but the search
 queries are very limited.

 I'm creating a site that'll allow arbitrary SQL queries to my data (I
 realize I'll need to handle injection attacks).

 Are there other viable ways to query data? I read a little on
 Business System 12 (BS12), Tutorial D, and even something called
 T-SQL (I think), but they all seem theoretical and not fully
 implemented.

 I want a query language that non-techies can use easily, but also
 supports arbitrarily complex queries. Does such a language exist?


Yeah, it's called SQL  Seriously, you're two requirements are
rather contradictory.   We have many end users who use SQL, learning
just enough to get the job done when we have this kind of problem.

Having said that, is there a reason why you are not looking at end
user query interfaces such as Hyperion or Cognos?  You might also look
at natural language or controlled vocabulary interfaces; if the
queries are domain specific (eg. medical) then you can often pick off
some low hanging fruit fairly quickly.  If the problem domain is non
specific then I don't think you'll be able to meet these requirements
without throwing a lot of money at the problem (The Cyc guys come to
mind ;-)...

-- 
Peter Hunsberger

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Viable alternatives to SQL?

2009-08-27 Thread Joshua Tolley
On Thu, Aug 27, 2009 at 06:43:51AM -0700, Kelly Jones wrote:
 Are there other viable ways to query data? I read a little on
 Business System 12 (BS12), Tutorial D, and even something called
 T-SQL (I think), but they all seem theoretical and not fully
 implemented.
 
 I want a query language that non-techies can use easily, but also
 supports arbitrarily complex queries. Does such a language exist?

I've never heard of the first two, as far as I can recall. T-SQL is quite
common, however (http://en.wikipedia.org/wiki/Transact-SQL), but although I've
never used it, I seriously doubt it will make things any more user friendly.

Several business intelligence applications try to allow users to generate
their own queries by providing a query builder system of some sort. These are
far from simple drop-in systems, in general. For instance, I'm using Pentaho
(http://www.pentaho.com) in a project, to allow users to write their own
reports. I've had to create a sort of schema metadata thing, which tells
Pentaho what my database looks like, how various tables relate to each other,
etc., and creates a set of objects my users will immediately understand, along
with some method to turn those objects into database queries. Pentaho allows
the users to drag and drop those objects into a report, creating filters and
sorting constraints, different groupings, etc., in a fairly user-friendly way.
When they run the report, Pentaho translates their input into SQL.

Anyway, you might try using a reporting package rather than trying to write
your own query interface and provide for user-friendly error reporting, decent
security constraints, etc.

--
Joshua Tolley / eggyknap
End Point Corporation
http://www.endpoint.com


signature.asc
Description: Digital signature


Re: [GENERAL] ETL software and training

2009-08-27 Thread Tguru

Check Talend for your training needs on an ETL tool.

Go look at the training page: http://www.talend.com/services/training.php



tv-8 wrote:
 
 I guess Talend (Open Studio) might be the right choice. But I do not have
 direct experience with the training.
 
 see www.talend.com
 

 Hi all,

 We are in the process of getting an ETL program. We need it to perform
 some
 basic extract, transform and load jobs.
 But we want to get an open source tool with good training.
 Our team is mainly business oriented, with some computer knowledge. We
 would
 like to have someone come to our company and explain to all the team how
 to
 operate the tool.

 I can operate the tool on my own, but I am not experienced enough to show
 other people how to operate the software.
 So would someone know which open source companies offer ETL tools with
 good
 training and how much time would it take?

 Thanks.
 --
 View this message in context:
 http://www.nabble.com/ETL-software-and-training-tp25131476p25131476.html
 Sent from the PostgreSQL - general mailing list archive at Nabble.com.


 --
 Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-general

 
 
 
 -- 
 Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-general
 
 

-- 
View this message in context: 
http://www.nabble.com/ETL-software-and-training-tp25131476p25167551.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Schema diff tool?

2009-08-27 Thread Sualeh Fatehi
Dave,

The free and open-source SchemaCrawler for PostGresql tool will do
what you need. SchemaCrawler outputs details of your schema (tables,
views, procedures, and more) in a diff-able plain-text format (text,
CSV, or XHTML). SchemaCrawler can also output data (including CLOBs
and BLOBs) in the same plain-text formats. You can use a standard diff
program to diff the current output with a reference version of the
output. SchemaCrawler can be run either from the command line, or as
an ant task. A lot of examples are available with the download to help
you get started.

SchemaCrawler comes with SchemaCrawler Grep, a command line tool that
allows you to search your schema for columns and tables that match a
regular expression. SchemaCrawler can also generate E-R diagrams.

SchemaCrawler is a free, open-source, cross-platform (operating system
and database) tool, written in Java, that is available at
SourceForge:
http://schemacrawler.sourceforge.net/

Sualeh Fatehi.

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] [Skytools-users] WAL Shipping + checkpoint

2009-08-27 Thread Martin Pihlak
 There are actually no real data changes being made on your master
 for some reason. So every time archive_timeout is reached a log full
 of no changes is shipped to your slave and applied - and no checkpoint
 times are changed for reasons I mentioned above.


 
 thanks, but we have not set archive_timeout, and we have a lot of real
 data changes.
 
 That's why i don't understand why checkpoint never happen on the slave.
 

What about the other values in pg_controldata output, do they change at
all? If they remain constant, maybe the logs are just not applied. Also,
are there any restored log file xxx from archive entries in postgres
log?

regards,
Martin


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Viable alternatives to SQL?

2009-08-27 Thread John R Pierce

Joshua Tolley wrote:

I've never heard of the first two, as far as I can recall. T-SQL is quite
common, however (http://en.wikipedia.org/wiki/Transact-SQL), but although I've
never used it, I seriously doubt it will make things any more user friendly.
  


TransactSQL, sometimes called TSQL, is MS SQL Server's equivalent of 
PL/PGSQL.   I doubt this is what the OP was referring to.




--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Data audit trail techniques in postgresql

2009-08-27 Thread Nathaniel Smith
Howdy,

(cross-posted to the pg.sql and pg.general lists)

I'm looking to set up an audit trail system--ie, logging every
INSERT/UPDATE/DELETE on a given table and tracking how the data
changes over time. I know I could do it by hand using triggers, but was
hoping to save a little time using something prepackaged.

I installed and tested the two-year untouched
http://pgfoundry.org/projects/audittrail2/ --the only auditing tool on
pgFoundry with a release--and found that, while it works well, it's
still marked as beta and has an obvious bug in it when used in 8.3.7.

What do others use to accomplish this? Do most pg users just write
triggers by hand? Or is there some nice auditing module that Google
just isn't revealing to me?

Thanks.

Nathaniel Smith
Web Application Developer
Summersault, LLC.
710 E. Main St., Suite 200
Richmond, IN 47374
www.summersault.com


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Data audit trail techniques in postgresql

2009-08-27 Thread Alvaro Herrera
Nathaniel Smith wrote:

 What do others use to accomplish this? Do most pg users just write
 triggers by hand? Or is there some nice auditing module that Google
 just isn't revealing to me?

I think tablelog (to be found in pgfoundry too) is the most commonly
used audit module.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Select data for current week only

2009-08-27 Thread BlackMage

Hey all,

I've been trying to figure this problem out with just using sql but I'm not
sure how too. I have a table that holds events for each week but I only want
to select events happening for the current week(Mon-Sun). So can anyone help
me out with this just using sql? I've accomplished it sorta using php but
only within the current 7 day range(example Thursday-Thursday) but not by
the week.

The field I am using for sorting is a Date type with the format -mm-dd
hh:mm:ss .
-- 
View this message in context: 
http://www.nabble.com/Select-data-for-current-week-only-tp25177178p25177178.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [SQL] [GENERAL] Data audit trail techniques in postgresql

2009-08-27 Thread Rob Sargent

tablelog doesn't appear any more lively than the OPs audittrail2.

Alvaro Herrera wrote:

Nathaniel Smith wrote:

  

What do others use to accomplish this? Do most pg users just write
triggers by hand? Or is there some nice auditing module that Google
just isn't revealing to me?



I think tablelog (to be found in pgfoundry too) is the most commonly
used audit module.

  


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Select data for current week only

2009-08-27 Thread Thomas Kellerer

BlackMage wrote on 27.08.2009 20:09:

I've been trying to figure this problem out with just using sql but I'm not
sure how too. I have a table that holds events for each week but I only want
to select events happening for the current week(Mon-Sun). So can anyone help
me out with this just using sql? I've accomplished it sorta using php but
only within the current 7 day range(example Thursday-Thursday) but not by
the week.


SELECT *
FROM the_table
WHERE extract(week from the_date_column) = extract(date from current_date);


The field I am using for sorting is a Date type with the format -mm-dd
hh:mm:ss .


A date column does not have a format :)

Thomas



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Select data for current week only

2009-08-27 Thread Edwin Plauchu
Postgresql has these functions
http://www.postgresql.org/docs/8.1/static/functions-formatting.html for
formatting data time


2009/8/27 BlackMage dsd7...@uncw.edu


 Hey all,

 I've been trying to figure this problem out with just using sql but I'm not
 sure how too. I have a table that holds events for each week but I only
 want
 to select events happening for the current week(Mon-Sun). So can anyone
 help
 me out with this just using sql? I've accomplished it sorta using php but
 only within the current 7 day range(example Thursday-Thursday) but not by
 the week.

 The field I am using for sorting is a Date type with the format -mm-dd
 hh:mm:ss .
 --
 View this message in context:
 http://www.nabble.com/Select-data-for-current-week-only-tp25177178p25177178.html
 Sent from the PostgreSQL - general mailing list archive at Nabble.com.


 --
 Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-general



Re: [GENERAL] Select data for current week only

2009-08-27 Thread Edwin Plauchu
you need to group dates per number week

2009/8/27 Edwin Plauchu pianodae...@gmail.com

 Postgresql has these functions
 http://www.postgresql.org/docs/8.1/static/functions-formatting.html for
 formatting data time


 2009/8/27 BlackMage dsd7...@uncw.edu


 Hey all,

 I've been trying to figure this problem out with just using sql but I'm
 not
 sure how too. I have a table that holds events for each week but I only
 want
 to select events happening for the current week(Mon-Sun). So can anyone
 help
 me out with this just using sql? I've accomplished it sorta using php but
 only within the current 7 day range(example Thursday-Thursday) but not by
 the week.

 The field I am using for sorting is a Date type with the format -mm-dd
 hh:mm:ss .
 --
 View this message in context:
 http://www.nabble.com/Select-data-for-current-week-only-tp25177178p25177178.html
 Sent from the PostgreSQL - general mailing list archive at Nabble.com.


 --
 Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-general





Re: [GENERAL] Select data for current week only

2009-08-27 Thread Gerhard Heift
On Thu, Aug 27, 2009 at 11:09:36AM -0700, BlackMage wrote:
 
 Hey all,
 
 I've been trying to figure this problem out with just using sql but I'm not
 sure how too. I have a table that holds events for each week but I only want
 to select events happening for the current week(Mon-Sun). So can anyone help
 me out with this just using sql? I've accomplished it sorta using php but
 only within the current 7 day range(example Thursday-Thursday) but not by
 the week.
 
 The field I am using for sorting is a Date type with the format -mm-dd
 hh:mm:ss .

If you use a timestamp and an index on it, the best thing would be
something like this:

SELECT * FROM event where date_trunc('week', now()) = event_date AND
event_date  date_trunc('week', now()) + '1 week'::interval

Regards,
  Gerhard


signature.asc
Description: Digital signature


Re: [GENERAL] Select data for current week only

2009-08-27 Thread Thomas Kellerer

Sam Mason wrote on 27.08.2009 21:06:


The OP leaves it somewhat open, but wouldn't date_trunc be better here?

Otherwise you'll end up getting values for other years as well as the
current one.


Good point, I didn't think of that :)

As an alternative, one could explicitely add the year into the where condition:

SELECT *
FROM the_table
WHERE extract(week from the_date_column) = extract(date from current_date)
AND extract(year from the_date_column) = extract(year from current_date);

but your solution is definitely more elegant ...

Thomas




--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Getting the column to a which a sequence belongs.

2009-08-27 Thread Thomas Kellerer

Hi,

I'm trying to extend the Postgres support in my SQL tool. I'm trying to recreate the SQL for a sequence, and I wonder if there is a way to find out the column to which a sequence belongs. 


I'm talking either about sequences that are created automatically by PG when 
using the serial datatype or sequences that have been changed using ALTER 
SEQUENCE ... OWNED BY ...

I know I can get the sequence that belongs to a column using pg_get_serial_sequence() (although only in 8.4), but I'm looking for the other way: given a sequence find out if it's owned by a column. 


So far debugging psql using the -E option didn't show up anything and I 
couldn't find any hints in the system catalogs documentation.

Is this possible at all? As neither psql nor pgAdmin display this information, 
I suspect it's not.

Regards
Thomas


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Select data for current week only

2009-08-27 Thread Sam Mason
On Thu, Aug 27, 2009 at 08:36:45PM +0200, Thomas Kellerer wrote:
 BlackMage wrote on 27.08.2009 20:09:
 I only want to select events happening for the current week(Mon-Sun).
 
 SELECT *
 FROM the_table
 WHERE extract(week from the_date_column) = extract(date from current_date);

The OP leaves it somewhat open, but wouldn't date_trunc be better here?
Something like:

  SELECT * FROM the_table
  WHERE date_trunc('week',the_date_column) = 
date_trunc('week',CURRENT_TIMESTAMP);

Otherwise you'll end up getting values for other years as well as the
current one.

-- 
  Sam  http://samason.me.uk/

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [SQL] [GENERAL] Data audit trail techniques in postgresql

2009-08-27 Thread Alvaro Herrera
Rob Sargent escribió:
 tablelog doesn't appear any more lively than the OPs audittrail2.

Perhaps, but I have heard of people using it successfully recently,
whereas Nathaniel reported that audittrail2 seems to have obvious bugs.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Select data for current week only

2009-08-27 Thread Sam Mason
On Thu, Aug 27, 2009 at 11:09:36AM -0700, BlackMage wrote:
 The field I am using for sorting is a Date type with the format -mm-dd
 hh:mm:ss .

Values that look like that are normally stored in timestamp columns, not
date column.  You wouldn't get the time part if it was just a date.

-- 
  Sam  http://samason.me.uk/

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Getting the column to a which a sequence belongs.

2009-08-27 Thread Sam Mason
On Thu, Aug 27, 2009 at 09:18:57PM +0200, Thomas Kellerer wrote:
 I'm trying to extend the Postgres support in my SQL tool. I'm trying to 
 recreate the SQL for a sequence, and I wonder if there is a way to find 
 out the column to which a sequence belongs. 

The information is all in the system catalogs; I've not had much
opportunity to fiddle with them so far but the following may be a start
to help get things out for you.

  SELECT c.relname, a.attname, t.relname
  FROM pg_class c, pg_depend d, pg_class t, pg_attribute a
  WHERE c.relkind = 'S'
AND d.objid   = c.oid
AND d.refobjid= t.oid
AND (d.refobjid,d.refobjsubid) = (a.attrelid,a.attnum);

The first reference to pg_class can probably be dropped as you can
convert the names of tables/sequences into their oid by using literals
of type regclass.  For example, to pull out all the column names from
table foo, you can do:

  SELECT attname
  FROM pg_attribute
  WHERE attrelid = 'foo'::regclass;

Have a look here for docs:

  http://www.postgresql.org/docs/current/static/catalogs.html

-- 
  Sam  http://samason.me.uk/

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Getting the column to a which a sequence belongs.

2009-08-27 Thread Thomas Kellerer

Sam Mason wrote on 27.08.2009 21:51:

The information is all in the system catalogs; I've not had much
opportunity to fiddle with them so far but the following may be a start
to help get things out for you.

  SELECT c.relname, a.attname, t.relname
  FROM pg_class c, pg_depend d, pg_class t, pg_attribute a
  WHERE c.relkind = 'S'
AND d.objid   = c.oid
AND d.refobjid= t.oid
AND (d.refobjid,d.refobjsubid) = (a.attrelid,a.attnum);

Ah great, I didn't realize I could use pg_depend for this. 


Thanks, works like a charm! This is exactly what I was looking for.

Thomas


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] query that worked in 8.1 not working in 8.4

2009-08-27 Thread Sean Foreman

We recently upgraded postgres from 8.1 to 8.4.

One of our queries stopped working and after some digging I've narrowed 
the problem down to this:


table structure of interest:

merchant_set
merchant_set_id

merchant
merchant_id
merchant_set_id

customer
customer_id
merchant_set_id

-- failure (count=1)
-- note: merchant_set.merchant_set_id in ...
select
   count(customer.customer_id)
from
   acquire.customer customer
   inner join entity_setup.merchant_set merchant_set on
   (customer.merchant_set_id = merchant_set.merchant_set_id
   and merchant_set.merchant_set_id in (
   select merchant_set_id
   from entity_setup.merchant merchant
   where merchant.merchant_id in (4,8,85,67)))
where
   customer.merchant_set_id = 2;

-- success (count=3562)
-- note: customer.merchant_set_id in ...
select
   count(customer.customer_id)
from
   acquire.customer customer
   inner join entity_setup.merchant_set merchant_set on
   (customer.merchant_set_id = merchant_set.merchant_set_id
   and customer.merchant_set_id in (
   select merchant_set_id
   from entity_setup.merchant merchant
   where merchant.merchant_id in (4,8,85,67)))
where
   customer.merchant_set_id = 2;

Explain for query 1 (failure):
Aggregate  (cost=5.23..5.24 rows=1 width=4) (actual time=0.161..0.161 
rows=1 loops=1)

  Output: count(customer.customer_id)
  -  Nested Loop Semi Join  (cost=3.23..5.22 rows=1 width=4) (actual 
time=0.140..0.153 rows=1 loops=1)

Output: customer.customer_id
-  Seq Scan on merchant_set  (cost=0.00..1.82 rows=1 width=4) 
(actual time=0.034..0.045 rows=1 loops=1)

  Output: merchant_set.merchant_set_id, ...
  Filter: (merchant_set_id = 2)
-  Nested Loop  (cost=3.23..266.07 rows=3562 width=12) (actual 
time=0.101..0.101 rows=1 loops=1)
  Output: customer.customer_id, customer.merchant_set_id, 
merchant.merchant_set_id
  -  HashAggregate  (cost=3.23..3.24 rows=1 width=4) 
(actual time=0.081..0.081 rows=1 loops=1)

Output: merchant.merchant_set_id
-  Seq Scan on merchant  (cost=0.00..3.23 rows=1 
width=4) (actual time=0.039..0.064 rows=2 loops=1)
  Output: merchant.merchant_id, ... , 
merchant.merchant_set_id, ...
  Filter: ((merchant_set_id = 2) AND 
(merchant_id = ANY ('{4,8,85,67}'::integer[])))
  -  Seq Scan on customer  (cost=0.00..227.21 rows=3562 
width=8) (actual time=0.015..0.015 rows=1 loops=1)
Output: customer.customer_id, ... , 
customer.merchant_set_id

Filter: (customer.merchant_set_id = 2)
Total runtime: 0.318 ms

Explain for query 2 (success):
Aggregate  (cost=312.42..312.43 rows=1 width=4) (actual 
time=17.442..17.442 rows=1 loops=1)

  Output: count(customer.customer_id)
  -  Nested Loop  (cost=3.23..303.51 rows=3562 width=4) (actual 
time=0.140..15.179 rows=3562 loops=1)

Output: customer.customer_id
-  Seq Scan on merchant_set  (cost=0.00..1.82 rows=1 width=4) 
(actual time=0.035..0.049 rows=1 loops=1)

  Output: merchant_set.merchant_set_id, ...
  Filter: (merchant_set_id = 2)
-  Nested Loop  (cost=3.23..266.07 rows=3562 width=8) (actual 
time=0.101..11.144 rows=3562 loops=1)

  Output: customer.customer_id, customer.merchant_set_id
  -  HashAggregate  (cost=3.23..3.24 rows=1 width=4) 
(actual time=0.082..0.085 rows=1 loops=1)

Output: merchant.merchant_set_id
-  Seq Scan on merchant  (cost=0.00..3.23 rows=1 
width=4) (actual time=0.038..0.064 rows=2 loops=1)
  Output: merchant.merchant_id, ... , 
merchant.merchant_set_id, ...
  Filter: ((merchant_set_id = 2) AND 
(merchant_id = ANY ('{4,8,85,67}'::integer[])))
  -  Seq Scan on customer  (cost=0.00..227.21 rows=3562 
width=8) (actual time=0.015..6.901 rows=3562 loops=1)
Output: customer.customer_id, ... , 
customer.merchant_set_id

Filter: (customer.merchant_set_id = 2)
Total runtime: 17.610 ms

Notes:
1. The real query gets information from customer  merchant_set so both 
tables are necessary. The query to merchant is a security filter.
2. I have fixed this query by dropping the subquery to merchant, and 
inner joining to merchant directly. This forces me to add a group by so 
customers are not duplicated which isn't as elegant as the original query.


I want to understand why the first version used to work with 8.1 and no 
longer works with 8.4. Is this bad sql and I was getting lucky before or 
is postgres making a bad decision in the latest release?




--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] query that worked in 8.1 not working in 8.4

2009-08-27 Thread Scott Marlowe
On Thu, Aug 27, 2009 at 2:13 PM, Sean
Foremansean.fore...@mpaygateway.com wrote:
 We recently upgraded postgres from 8.1 to 8.4.

So, is there an error message you get back? Or just no data?

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] query that worked in 8.1 not working in 8.4

2009-08-27 Thread Sean Foreman
There is no error message. Posgres likes the query. In this case, the 
query returns a count of 1 instead of 3562.


Scott Marlowe wrote:

On Thu, Aug 27, 2009 at 2:13 PM, Sean
Foremansean.fore...@mpaygateway.com wrote:
  

We recently upgraded postgres from 8.1 to 8.4.



So, is there an error message you get back? Or just no data?
  



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] It looks like transaction, but it isn't transaction

2009-08-27 Thread Sergey Samokhin
Hello, Tom.

On Thu, Aug 27, 2009 at 5:25 AM, Tom Lanet...@sss.pgh.pa.us wrote:
 Yes, if it's submitted as a single Query message, that's what happens.
 skipped

Thanks for the detailed explanation.

-- 
Sergey Samokhin

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Postgres Kickstart/Anaconda auto-install

2009-08-27 Thread Alan McKay
Hey folks,

I realise this is probably more a matter for a kickstart list, but
then again, I have to think that someone else on this list has done
this and can help.  So I'll ask here and there.

I'm dragging our company kicking and screaming into the realm of
Kickstart/Anaconda, and trying to get PG automatically installed via a
kickstart file.

I'm using CentOS 5.3 and have mirrored their repository on my subnet,
and all that part works great.

I've also taken a copy of
http://yum.pgsqlrpms.org/8.4/redhat/rhel-5-i386/ (and the 64 bit one)
and want to install the RPMs from there.  Automatically via Kickstart.

I have this line in my kickstart file :

repo --name=Postgres --baseurl=http://192.168.0.22/ks/postgres/8.4/rhel-5-i386/

and then this :
@examplePG

And within the directory for that repo, I've created a comps.xml file
based on some googling.  Here is my file - it only lists those RPMs
from the repo, that I want

comps
!--  meta --
!-- Meta information will go here eventually --
!--  /meta --
  group
idExamplePG/id
nameExamplePG/name
defaulttrue/default
descriptionExample's Own PostgreSQL Repository/description
uservisibletrue/uservisible
packagelist
  packagereq
type=defaultpostgresql-server-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
  packagereq
type=defaultpostgresql-libs-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
  packagereq
type=defaultpostgresql-plperl-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
  packagereq
type=defaultpostgresql-debuginfo-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
  packagereq
type=defaultpostgresql-devel-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
  packagereq
type=defaultpostgresql-plpython-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
  packagereq
type=defaultpostgresql-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
  packagereq
type=defaultpostgresql-docs-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
  packagereq
type=defaultpostgresql-test-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
  packagereq
type=defaultpostgresql-contrib-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
  packagereq
type=defaultpostgresql-pltcl-8.4.0-1PGDG.rhel5.i386.rpm/packagereq
/packagelist
  /group
/comps

My installation does not complain about the @ExamplePG line, but it
does not install any of these RPMs.  The only PG rpm I get is a
version 8.1 libs RPM from the CentOS install.

As a test I then inserted this into my kickstart file :

@foobar

And sure enough, the install process complains about it and asks if I
want to continue or abort.

Any ideas out there?

thanks,
-Alan

-- 
“Don't eat anything you've ever seen advertised on TV”
 - Michael Pollan, author of In Defense of Food

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [SQL] [GENERAL] Data audit trail techniques in postgresql

2009-08-27 Thread Rob Sargent



Alvaro Herrera wrote:

Rob Sargent escribió:
  

tablelog doesn't appear any more lively than the OPs audittrail2.



Perhaps, but I have heard of people using it successfully recently,
whereas Nathaniel reported that audittrail2 seems to have obvious bugs.

  

Fair enough.



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] query that worked in 8.1 not working in 8.4

2009-08-27 Thread Scott Marlowe
On Thu, Aug 27, 2009 at 2:13 PM, Sean
Foremansean.fore...@mpaygateway.com wrote:
 We recently upgraded postgres from 8.1 to 8.4.

 One of our queries stopped working and after some digging I've narrowed the
 problem down to this:

 table structure of interest:

 merchant_set
 merchant_set_id

 merchant
 merchant_id
 merchant_set_id

 customer
 customer_id
 merchant_set_id

So what data types are these?  I'm guessing you're being bitten by
some auto-cast that got removed in 8.3.  But that's just a guess.

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] query that worked in 8.1 not working in 8.4

2009-08-27 Thread Tom Lane
Sean Foreman sean.fore...@mpaygateway.com writes:
 We recently upgraded postgres from 8.1 to 8.4.
 One of our queries stopped working and after some digging I've narrowed 
 the problem down to this:

 select
 count(customer.customer_id)
 from
 acquire.customer customer
 inner join entity_setup.merchant_set merchant_set on
 (customer.merchant_set_id = merchant_set.merchant_set_id
 and merchant_set.merchant_set_id in (
 select merchant_set_id
 from entity_setup.merchant merchant
 where merchant.merchant_id in (4,8,85,67)))
 where
 customer.merchant_set_id = 2;

There are some bugs in 8.4.0 associated with possibly re-ordering
semijoins (IN joins) incorrectly with respect to other joins.
It looks like you got bit by that.  Are you in a position to try
8.4 branch tip (from CVS or nightly snapshots)?  If not, you'll
have to wait for 8.4.1, but it'd be nice to confirm this case
is fixed before we ship 8.4.1.

regards, tom lane

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] A safe way to upgrade table definitions by using ALTER's

2009-08-27 Thread Sergey Samokhin
Hello.

As I know upgrading database structure from one version to another is
usually done by applying some sql-script with a set of ALTER's that do
all the work.

But how do programmers guarantee that ALTER's they have wrote will
always be applied by administrators to the corresponding version of
the database?

Is there a standard way to store some kind of metainformation in DB
(like version of the current definitions of tables) and then check if
it is too old for being upgraded by a given script?

By database structure I mean definition of tables an application
uses (data types, constraints, modificators etc), stored procedures
etc.

Thanks.

-- 
Sergey Samokhin

P.S. If there is other convinient way to do upgrades which I've missed
- please let me know. I'm still a novice in the PostgreSQL/SQL world.

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] query that worked in 8.1 not working in 8.4

2009-08-27 Thread Sean Foreman

This is not an autocast issue. These are all integers.

The query plan postgres is choosing to execute looks questionable. I 
have provided an example of a working query and non-working query. The 
change is subtle and takes advantage of the questionable query plan to 
get the results I need. The join between customer and merchant looks 
incorrect. It should be joining merchant_set and merchant and then 
hitting customer. I'm not an expert at reading postgres query plans so I 
need some help figuring out if this is a postgres bug or a poorly 
written query and why. I think it may be a bug.

On Thu, Aug 27, 2009 at 2:13 PM, Sean
Foremansean.fore...@mpaygateway.com wrote:
  

We recently upgraded postgres from 8.1 to 8.4.

One of our queries stopped working and after some digging I've narrowed the
problem down to this:

table structure of interest:

merchant_set
merchant_set_id

merchant
merchant_id
merchant_set_id

customer
customer_id
merchant_set_id



So what data types are these?  I'm guessing you're being bitten by
some auto-cast that got removed in 8.3.  But that's just a guess.
  



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] query that worked in 8.1 not working in 8.4

2009-08-27 Thread Sean Foreman
Thanks Tom. That is what I suspected. I will install the branch tip on 
Monday and see if the problem goes away.


Sean

Tom Lane wrote:

Sean Foreman sean.fore...@mpaygateway.com writes:
  

We recently upgraded postgres from 8.1 to 8.4.
One of our queries stopped working and after some digging I've narrowed 
the problem down to this:



  

select
count(customer.customer_id)
from
acquire.customer customer
inner join entity_setup.merchant_set merchant_set on
(customer.merchant_set_id = merchant_set.merchant_set_id
and merchant_set.merchant_set_id in (
select merchant_set_id
from entity_setup.merchant merchant
where merchant.merchant_id in (4,8,85,67)))
where
customer.merchant_set_id = 2;



There are some bugs in 8.4.0 associated with possibly re-ordering
semijoins (IN joins) incorrectly with respect to other joins.
It looks like you got bit by that.  Are you in a position to try
8.4 branch tip (from CVS or nightly snapshots)?  If not, you'll
have to wait for 8.4.1, but it'd be nice to confirm this case
is fixed before we ship 8.4.1.

regards, tom lane
  



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Postgres Kickstart/Anaconda auto-install

2009-08-27 Thread Alan McKay
 And within the directory for that repo, I've created a comps.xml file

And of course re-run createrepo ...

-- 
“Don't eat anything you've ever seen advertised on TV”
 - Michael Pollan, author of In Defense of Food

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] A safe way to upgrade table definitions by using ALTER's

2009-08-27 Thread Steve Atkins


On Aug 27, 2009, at 1:42 PM, Sergey Samokhin wrote:


Hello.

As I know upgrading database structure from one version to another is
usually done by applying some sql-script with a set of ALTER's that do
all the work.

But how do programmers guarantee that ALTER's they have wrote will
always be applied by administrators to the corresponding version of
the database?


In the application is where I do it, though it would be possible to
create a .sql script that errored out if the existing version were
not the one it expected.



Is there a standard way to store some kind of metainformation in DB
(like version of the current definitions of tables) and then check if
it is too old for being upgraded by a given script?


There's no standard way at the database level, though there are
application frameworks that support it in a way that's standard for
that framework.



By database structure I mean definition of tables an application
uses (data types, constraints, modificators etc), stored procedures
etc.


I tend to keep a single row table in the database that contains
the current schema version, then have the application apply
upgrade / downgrade patches as needed (or bail out and
tell the user to do it).

Cheers,
  Steve


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Postgres Kickstart/Anaconda auto-install

2009-08-27 Thread Devrim GÜNDÜZ
On Thu, 2009-08-27 at 16:28 -0400, Alan McKay wrote:
 Any ideas out there?

There is a ks file that I wrote for PostgreSQL Live CD's:

https://projects.commandprompt.com/public/pgcore/browser/livecd/PG84-Fedora-x86.ks

This may guide you. It is for Fedora-11, but you can easily move it to
RHEL 5.

-HTH.

Regards,
-- 
Devrim GÜNDÜZ, RHCE
Command Prompt - http://www.CommandPrompt.com 
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
   http://www.gunduz.org


signature.asc
Description: This is a digitally signed message part


[GENERAL] GUI to edit a table's content

2009-08-27 Thread Gauthier, Dave
Are there any GUI apps out there that can be used to edit (insert, update, 
delete) table data?  SOmething with enough smarts to sniff out constraint 
violations and report accordingly.  Also, ability to eval constraints at commit 
time inside a transaction (option to rollback of course).

Thanks for any pointers.


Re: [GENERAL] GUI to edit a table's content

2009-08-27 Thread Thomas Kellerer

Gauthier, Dave wrote on 27.08.2009 23:17:
Are there any GUI apps out there that can be used to edit (insert, 
update, delete) table data?  SOmething with enough smarts to sniff out 
constraint violations and report accordingly.  Also, ability to eval 
constraints at commit time inside a transaction (option to rollback of 
course).


Check out this list:

http://wiki.postgresql.org/wiki/Community_Guide_to_PostgreSQL_GUI_Tools



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Viable alternatives to SQL?

2009-08-27 Thread Merlin Moncure
On Thu, Aug 27, 2009 at 9:43 AM, Kelly Joneskelly.terry.jo...@gmail.com wrote:
 Many sites let you search databases of information, but the search
 queries are very limited.

 I'm creating a site that'll allow arbitrary SQL queries to my data (I
 realize I'll need to handle injection attacks).

 Are there other viable ways to query data? I read a little on
 Business System 12 (BS12), Tutorial D, and even something called
 T-SQL (I think), but they all seem theoretical and not fully
 implemented.

 I want a query language that non-techies can use easily, but also
 supports arbitrarily complex queries. Does such a language exist?

SQL is your best bet.  It is probably the most natural language for
non-programmers to learn and understand.  It scales in difficulty with
the problem you are trying to solve and is _much_ easier for non
technical user than something like an ORM.  That said, if your site
allows arbitrary queries, sql injection is not your only problem.  It
is trivially easy to write a query that can DOS your server (with SQL
access I could take any PostgreSQL server down in about 1
second)so you need to think about how exactly who your are going
to allow access to your database and how they can access it.

merlin

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Is there a function for Converting a Decimal into BINARY ?

2009-08-27 Thread Ow Mun Heng
Hi Guys,

Searching the net didn't give me much clues as to how to convert a Decimal
number into BINARY.


Eg:
I have a datatype in the DB which needs to be converted.

DEC = 192
BINARY = 1100

DEC = 197
BINARY = 11000101

Which I then need to break down into pairs to do calculations on

11 : 00 : 00 : 00
11 : 00 : 01 : 01

Some of the solutions I've seen on the Internet is based on VB and mainly
userland apps, I would like to do the conversion within PG itself.

Thanks

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Is there a function for Converting a Decimal into BINARY ?

2009-08-27 Thread John R Pierce

Ow Mun Heng wrote:

Hi Guys,

Searching the net didn't give me much clues as to how to convert a Decimal
number into BINARY.
  


well, a decimal number is a fixed point number stored in a modified BCD 
format, which optionally can contain a decimal fractional component.


you likely would want to convert it to a INTEGER or BIGINT first,  then 
cats it to BIT(n) where N is the number of bits you want.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Is there a function for Converting a Decimal into BINARY ?

2009-08-27 Thread Harvey, Allan AC
 Eg:
 I have a datatype in the DB which needs to be converted.
 
 DEC = 192
 BINARY = 1100

How about this

gwmdb= select 192::bit(16);
   bit
--
 1100
(1 row)

Hope that helps

Allan


The material contained in this email may be confidential, privileged or 
copyrighted. If you are not the intended recipient, use, disclosure or copying 
of this information is prohibited. If you have received this document in error, 
please advise the sender and delete the document. Neither OneSteel nor the 
sender accept responsibility for any viruses contained in this email or any 
attachments.

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] How to simulate crashes of PostgreSQL?

2009-08-27 Thread Alvaro Herrera
Vick Khera wrote:
 On Tue, Aug 25, 2009 at 4:55 PM, Tom Lanet...@sss.pgh.pa.us wrote:
  I've always thought that the fd.c layer is more about not having to
  configure the code explicitly for max-files-per-process limits.  Once
  you get into ENFILE conditions, even if Postgres manages to stay up,
  everything else on the box is going to start falling over.  So the
  sysadmin is likely to have to resort to a reboot anyway.
 
 In my case, all sorts of processes were complaining about being unable
 to open files.  Once Pg panicked and closed all its files, everything
 came back to normal.  I didn't have to reboot because most everything
 was written to retry and/or restart itself, and nothing critical like
 sshd croaked.

Hmm.  How many DB connections were there at the time?  Are they normally
long-lived?

I'm wondering if the problem could be caused by too many backends
holding the maximum of open files each.  In my system,
/proc/sys/fs/file-max says ~200k, and per-process limit is 1024, so it
would take about 200 backends with all FDs in use to bring the system to
a near collapse that won't be solved until Postgres is restarted.  This
doesn't sound so far-fetched if the connections are long lived, perhaps
from a pooler.

Maybe we should have another inter-backend signal: when a process gets
ENFILE, signal all other backends and they close a bunch of files each.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] How to simulate crashes of PostgreSQL?

2009-08-27 Thread Greg Stark
On Fri, Aug 28, 2009 at 4:13 AM, Alvaro
Herreraalvhe...@commandprompt.com wrote:
 Maybe we should have another inter-backend signal: when a process gets
 ENFILE, signal all other backends and they close a bunch of files each.

I wonder if this is a new problem due to the FSM and VM using up extra
file handles?


-- 
greg
http://mit.edu/~gsstark/resume.pdf

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] How to simulate crashes of PostgreSQL?

2009-08-27 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes:
 Maybe we should have another inter-backend signal: when a process gets
 ENFILE, signal all other backends and they close a bunch of files each.

I was wondering about that myself, but on balance I think it'd be a lot
of work to achieve not much.  What you would have is that Postgres would
ramp its FD usage up to hit the kernel limit, things outside the
database would fail for some period of time, then a backend would get
ENFILE and we'd cut down our FD usage.  Lather, rinse, repeat, ad
infinitum.  You'd have intermittent hard-to-reproduce failures of every
other service on the box; and you'd *still* be at risk of the DB
crashing, if walwriter or another low-cushion process hit the problem
first.

The only really reliable setup is to have max_connections times
max_files_per_process less than the kernel limit.  If we do something to
mask the problem when it happens, I don't think we're doing the DBA a
service in the long run.

Thought: it's probably possible to find out the kernel limit on many
platforms.  Maybe postmaster startup should try to get that limit, and
print an annoying warning if it's less than max_connections times
max_files_per_process plus some safety factor?

regards, tom lane

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general