Re: [GENERAL] [Skytools-users] WAL Shipping + checkpoint
On 27/08/2009 00:18, Mark Kirkwood wrote: Sébastien Lardière wrote: On 26/08/2009 04:46, Mark Kirkwood wrote: Sébastien Lardière wrote: Hi All, I've a cluster ( Pg 8.3.7 ) with WAL Shipping, and a few hours ago, the master had to restart. I use walmgr from Skytools, which works very well. I have already restart the master without any problem, but today, the slave doesn't work like I want. The field Time of latest checkpoint from the pg_controldata on the slave keep the same values, but WAL File are processed correctly. I try to restart the slave, but, after processed again all the WAL between Time of latest checkpoint and, it does nothing else, latest checkpoint stay at the same value. I don't know if it's important ( i think so ), and I can't fix it. It is normal for it to lag behind somewhat on the slave (depending on what your checkpoint timeout etc settings are). However, I've noticed what you are seeing as well - particularly when there are no actual data changes coming through in the logs - the slave checkpoint time does not change even tho there have been checkpoints on the master (I may have a look in the code to see what the story really is...if I have time). Yes, but the delay between the last checkpoint on the master and the slave is very high, now ( 100 000 sec ), because the last checkpoint on the slave was yesterday ( as far as pg_controldata is right ) Here a graph from our munin plugin : http://seb.ouvaton.org/tmp/bdd-pg_walmgr-week.png The blue line represent an average between two WAL processed on the slave, and the green line, the delai between last checkpoint on the master and the slave. Maybe it's not some good indicator, but the green line let me think there is problem. Do you have archive_timeout set? If so, then what *could* be happening is this: There are actually no real data changes being made on your master for some reason. So every time archive_timeout is reached a log full of no changes is shipped to your slave and applied - and no checkpoint times are changed for reasons I mentioned above. thanks, but we have not set archive_timeout, and we have a lot of real data changes. That's why i don't understand why checkpoint never happen on the slave. -- Sébastien Lardière -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
[GENERAL] Viable alternatives to SQL?
Many sites let you search databases of information, but the search queries are very limited. I'm creating a site that'll allow arbitrary SQL queries to my data (I realize I'll need to handle injection attacks). Are there other viable ways to query data? I read a little on Business System 12 (BS12), Tutorial D, and even something called T-SQL (I think), but they all seem theoretical and not fully implemented. I want a query language that non-techies can use easily, but also supports arbitrarily complex queries. Does such a language exist? -- We're just a Bunch Of Regular Guys, a collective group that's trying to understand and assimilate technology. We feel that resistance to new ideas and technology is unwise and ultimately futile. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Viable alternatives to SQL?
Hibernate is a ORM technology which configures bean classes which will construct DB statements based on the method(create,replace,update,delete) from the bean class attributes(columns) to quote Hibernate allows you to express queries in its own portable SQL extension (HQL), as well as in native SQL, or with an object-oriented Criteria and Example API. Hibernate currently supports the following databases Microsoft SQL Server 2005/2000OracleMicrosoft AccessFirebirdPostgreSQLDB2 UDBMySQLSQLite https://www.hibernate.org/ Martin Gainty __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. Date: Thu, 27 Aug 2009 06:43:51 -0700 Subject: [GENERAL] Viable alternatives to SQL? From: kelly.terry.jo...@gmail.com To: pgsql-general@postgresql.org Many sites let you search databases of information, but the search queries are very limited. I'm creating a site that'll allow arbitrary SQL queries to my data (I realize I'll need to handle injection attacks). Are there other viable ways to query data? I read a little on Business System 12 (BS12), Tutorial D, and even something called T-SQL (I think), but they all seem theoretical and not fully implemented. I want a query language that non-techies can use easily, but also supports arbitrarily complex queries. Does such a language exist? -- We're just a Bunch Of Regular Guys, a collective group that's trying to understand and assimilate technology. We feel that resistance to new ideas and technology is unwise and ultimately futile. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general _ Windows Live: Keep your friends up to date with what you do online. http://windowslive.com/Campaign/SocialNetworking?ocid=PID23285::T:WLMTAGL:ON:WL:en-US:SI_SB_online:082009
Re: [GENERAL] Viable alternatives to SQL?
On Thu, Aug 27, 2009 at 8:43 AM, Kelly Joneskelly.terry.jo...@gmail.com wrote: Many sites let you search databases of information, but the search queries are very limited. I'm creating a site that'll allow arbitrary SQL queries to my data (I realize I'll need to handle injection attacks). Are there other viable ways to query data? I read a little on Business System 12 (BS12), Tutorial D, and even something called T-SQL (I think), but they all seem theoretical and not fully implemented. I want a query language that non-techies can use easily, but also supports arbitrarily complex queries. Does such a language exist? Yeah, it's called SQL Seriously, you're two requirements are rather contradictory. We have many end users who use SQL, learning just enough to get the job done when we have this kind of problem. Having said that, is there a reason why you are not looking at end user query interfaces such as Hyperion or Cognos? You might also look at natural language or controlled vocabulary interfaces; if the queries are domain specific (eg. medical) then you can often pick off some low hanging fruit fairly quickly. If the problem domain is non specific then I don't think you'll be able to meet these requirements without throwing a lot of money at the problem (The Cyc guys come to mind ;-)... -- Peter Hunsberger -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Viable alternatives to SQL?
On Thu, Aug 27, 2009 at 06:43:51AM -0700, Kelly Jones wrote: Are there other viable ways to query data? I read a little on Business System 12 (BS12), Tutorial D, and even something called T-SQL (I think), but they all seem theoretical and not fully implemented. I want a query language that non-techies can use easily, but also supports arbitrarily complex queries. Does such a language exist? I've never heard of the first two, as far as I can recall. T-SQL is quite common, however (http://en.wikipedia.org/wiki/Transact-SQL), but although I've never used it, I seriously doubt it will make things any more user friendly. Several business intelligence applications try to allow users to generate their own queries by providing a query builder system of some sort. These are far from simple drop-in systems, in general. For instance, I'm using Pentaho (http://www.pentaho.com) in a project, to allow users to write their own reports. I've had to create a sort of schema metadata thing, which tells Pentaho what my database looks like, how various tables relate to each other, etc., and creates a set of objects my users will immediately understand, along with some method to turn those objects into database queries. Pentaho allows the users to drag and drop those objects into a report, creating filters and sorting constraints, different groupings, etc., in a fairly user-friendly way. When they run the report, Pentaho translates their input into SQL. Anyway, you might try using a reporting package rather than trying to write your own query interface and provide for user-friendly error reporting, decent security constraints, etc. -- Joshua Tolley / eggyknap End Point Corporation http://www.endpoint.com signature.asc Description: Digital signature
Re: [GENERAL] ETL software and training
Check Talend for your training needs on an ETL tool. Go look at the training page: http://www.talend.com/services/training.php tv-8 wrote: I guess Talend (Open Studio) might be the right choice. But I do not have direct experience with the training. see www.talend.com Hi all, We are in the process of getting an ETL program. We need it to perform some basic extract, transform and load jobs. But we want to get an open source tool with good training. Our team is mainly business oriented, with some computer knowledge. We would like to have someone come to our company and explain to all the team how to operate the tool. I can operate the tool on my own, but I am not experienced enough to show other people how to operate the software. So would someone know which open source companies offer ETL tools with good training and how much time would it take? Thanks. -- View this message in context: http://www.nabble.com/ETL-software-and-training-tp25131476p25131476.html Sent from the PostgreSQL - general mailing list archive at Nabble.com. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general -- View this message in context: http://www.nabble.com/ETL-software-and-training-tp25131476p25167551.html Sent from the PostgreSQL - general mailing list archive at Nabble.com. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Schema diff tool?
Dave, The free and open-source SchemaCrawler for PostGresql tool will do what you need. SchemaCrawler outputs details of your schema (tables, views, procedures, and more) in a diff-able plain-text format (text, CSV, or XHTML). SchemaCrawler can also output data (including CLOBs and BLOBs) in the same plain-text formats. You can use a standard diff program to diff the current output with a reference version of the output. SchemaCrawler can be run either from the command line, or as an ant task. A lot of examples are available with the download to help you get started. SchemaCrawler comes with SchemaCrawler Grep, a command line tool that allows you to search your schema for columns and tables that match a regular expression. SchemaCrawler can also generate E-R diagrams. SchemaCrawler is a free, open-source, cross-platform (operating system and database) tool, written in Java, that is available at SourceForge: http://schemacrawler.sourceforge.net/ Sualeh Fatehi. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] [Skytools-users] WAL Shipping + checkpoint
There are actually no real data changes being made on your master for some reason. So every time archive_timeout is reached a log full of no changes is shipped to your slave and applied - and no checkpoint times are changed for reasons I mentioned above. thanks, but we have not set archive_timeout, and we have a lot of real data changes. That's why i don't understand why checkpoint never happen on the slave. What about the other values in pg_controldata output, do they change at all? If they remain constant, maybe the logs are just not applied. Also, are there any restored log file xxx from archive entries in postgres log? regards, Martin -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Viable alternatives to SQL?
Joshua Tolley wrote: I've never heard of the first two, as far as I can recall. T-SQL is quite common, however (http://en.wikipedia.org/wiki/Transact-SQL), but although I've never used it, I seriously doubt it will make things any more user friendly. TransactSQL, sometimes called TSQL, is MS SQL Server's equivalent of PL/PGSQL. I doubt this is what the OP was referring to. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
[GENERAL] Data audit trail techniques in postgresql
Howdy, (cross-posted to the pg.sql and pg.general lists) I'm looking to set up an audit trail system--ie, logging every INSERT/UPDATE/DELETE on a given table and tracking how the data changes over time. I know I could do it by hand using triggers, but was hoping to save a little time using something prepackaged. I installed and tested the two-year untouched http://pgfoundry.org/projects/audittrail2/ --the only auditing tool on pgFoundry with a release--and found that, while it works well, it's still marked as beta and has an obvious bug in it when used in 8.3.7. What do others use to accomplish this? Do most pg users just write triggers by hand? Or is there some nice auditing module that Google just isn't revealing to me? Thanks. Nathaniel Smith Web Application Developer Summersault, LLC. 710 E. Main St., Suite 200 Richmond, IN 47374 www.summersault.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Data audit trail techniques in postgresql
Nathaniel Smith wrote: What do others use to accomplish this? Do most pg users just write triggers by hand? Or is there some nice auditing module that Google just isn't revealing to me? I think tablelog (to be found in pgfoundry too) is the most commonly used audit module. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
[GENERAL] Select data for current week only
Hey all, I've been trying to figure this problem out with just using sql but I'm not sure how too. I have a table that holds events for each week but I only want to select events happening for the current week(Mon-Sun). So can anyone help me out with this just using sql? I've accomplished it sorta using php but only within the current 7 day range(example Thursday-Thursday) but not by the week. The field I am using for sorting is a Date type with the format -mm-dd hh:mm:ss . -- View this message in context: http://www.nabble.com/Select-data-for-current-week-only-tp25177178p25177178.html Sent from the PostgreSQL - general mailing list archive at Nabble.com. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [SQL] [GENERAL] Data audit trail techniques in postgresql
tablelog doesn't appear any more lively than the OPs audittrail2. Alvaro Herrera wrote: Nathaniel Smith wrote: What do others use to accomplish this? Do most pg users just write triggers by hand? Or is there some nice auditing module that Google just isn't revealing to me? I think tablelog (to be found in pgfoundry too) is the most commonly used audit module. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Select data for current week only
BlackMage wrote on 27.08.2009 20:09: I've been trying to figure this problem out with just using sql but I'm not sure how too. I have a table that holds events for each week but I only want to select events happening for the current week(Mon-Sun). So can anyone help me out with this just using sql? I've accomplished it sorta using php but only within the current 7 day range(example Thursday-Thursday) but not by the week. SELECT * FROM the_table WHERE extract(week from the_date_column) = extract(date from current_date); The field I am using for sorting is a Date type with the format -mm-dd hh:mm:ss . A date column does not have a format :) Thomas -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Select data for current week only
Postgresql has these functions http://www.postgresql.org/docs/8.1/static/functions-formatting.html for formatting data time 2009/8/27 BlackMage dsd7...@uncw.edu Hey all, I've been trying to figure this problem out with just using sql but I'm not sure how too. I have a table that holds events for each week but I only want to select events happening for the current week(Mon-Sun). So can anyone help me out with this just using sql? I've accomplished it sorta using php but only within the current 7 day range(example Thursday-Thursday) but not by the week. The field I am using for sorting is a Date type with the format -mm-dd hh:mm:ss . -- View this message in context: http://www.nabble.com/Select-data-for-current-week-only-tp25177178p25177178.html Sent from the PostgreSQL - general mailing list archive at Nabble.com. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Select data for current week only
you need to group dates per number week 2009/8/27 Edwin Plauchu pianodae...@gmail.com Postgresql has these functions http://www.postgresql.org/docs/8.1/static/functions-formatting.html for formatting data time 2009/8/27 BlackMage dsd7...@uncw.edu Hey all, I've been trying to figure this problem out with just using sql but I'm not sure how too. I have a table that holds events for each week but I only want to select events happening for the current week(Mon-Sun). So can anyone help me out with this just using sql? I've accomplished it sorta using php but only within the current 7 day range(example Thursday-Thursday) but not by the week. The field I am using for sorting is a Date type with the format -mm-dd hh:mm:ss . -- View this message in context: http://www.nabble.com/Select-data-for-current-week-only-tp25177178p25177178.html Sent from the PostgreSQL - general mailing list archive at Nabble.com. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Select data for current week only
On Thu, Aug 27, 2009 at 11:09:36AM -0700, BlackMage wrote: Hey all, I've been trying to figure this problem out with just using sql but I'm not sure how too. I have a table that holds events for each week but I only want to select events happening for the current week(Mon-Sun). So can anyone help me out with this just using sql? I've accomplished it sorta using php but only within the current 7 day range(example Thursday-Thursday) but not by the week. The field I am using for sorting is a Date type with the format -mm-dd hh:mm:ss . If you use a timestamp and an index on it, the best thing would be something like this: SELECT * FROM event where date_trunc('week', now()) = event_date AND event_date date_trunc('week', now()) + '1 week'::interval Regards, Gerhard signature.asc Description: Digital signature
Re: [GENERAL] Select data for current week only
Sam Mason wrote on 27.08.2009 21:06: The OP leaves it somewhat open, but wouldn't date_trunc be better here? Otherwise you'll end up getting values for other years as well as the current one. Good point, I didn't think of that :) As an alternative, one could explicitely add the year into the where condition: SELECT * FROM the_table WHERE extract(week from the_date_column) = extract(date from current_date) AND extract(year from the_date_column) = extract(year from current_date); but your solution is definitely more elegant ... Thomas -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
[GENERAL] Getting the column to a which a sequence belongs.
Hi, I'm trying to extend the Postgres support in my SQL tool. I'm trying to recreate the SQL for a sequence, and I wonder if there is a way to find out the column to which a sequence belongs. I'm talking either about sequences that are created automatically by PG when using the serial datatype or sequences that have been changed using ALTER SEQUENCE ... OWNED BY ... I know I can get the sequence that belongs to a column using pg_get_serial_sequence() (although only in 8.4), but I'm looking for the other way: given a sequence find out if it's owned by a column. So far debugging psql using the -E option didn't show up anything and I couldn't find any hints in the system catalogs documentation. Is this possible at all? As neither psql nor pgAdmin display this information, I suspect it's not. Regards Thomas -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Select data for current week only
On Thu, Aug 27, 2009 at 08:36:45PM +0200, Thomas Kellerer wrote: BlackMage wrote on 27.08.2009 20:09: I only want to select events happening for the current week(Mon-Sun). SELECT * FROM the_table WHERE extract(week from the_date_column) = extract(date from current_date); The OP leaves it somewhat open, but wouldn't date_trunc be better here? Something like: SELECT * FROM the_table WHERE date_trunc('week',the_date_column) = date_trunc('week',CURRENT_TIMESTAMP); Otherwise you'll end up getting values for other years as well as the current one. -- Sam http://samason.me.uk/ -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [SQL] [GENERAL] Data audit trail techniques in postgresql
Rob Sargent escribió: tablelog doesn't appear any more lively than the OPs audittrail2. Perhaps, but I have heard of people using it successfully recently, whereas Nathaniel reported that audittrail2 seems to have obvious bugs. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Select data for current week only
On Thu, Aug 27, 2009 at 11:09:36AM -0700, BlackMage wrote: The field I am using for sorting is a Date type with the format -mm-dd hh:mm:ss . Values that look like that are normally stored in timestamp columns, not date column. You wouldn't get the time part if it was just a date. -- Sam http://samason.me.uk/ -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Getting the column to a which a sequence belongs.
On Thu, Aug 27, 2009 at 09:18:57PM +0200, Thomas Kellerer wrote: I'm trying to extend the Postgres support in my SQL tool. I'm trying to recreate the SQL for a sequence, and I wonder if there is a way to find out the column to which a sequence belongs. The information is all in the system catalogs; I've not had much opportunity to fiddle with them so far but the following may be a start to help get things out for you. SELECT c.relname, a.attname, t.relname FROM pg_class c, pg_depend d, pg_class t, pg_attribute a WHERE c.relkind = 'S' AND d.objid = c.oid AND d.refobjid= t.oid AND (d.refobjid,d.refobjsubid) = (a.attrelid,a.attnum); The first reference to pg_class can probably be dropped as you can convert the names of tables/sequences into their oid by using literals of type regclass. For example, to pull out all the column names from table foo, you can do: SELECT attname FROM pg_attribute WHERE attrelid = 'foo'::regclass; Have a look here for docs: http://www.postgresql.org/docs/current/static/catalogs.html -- Sam http://samason.me.uk/ -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Getting the column to a which a sequence belongs.
Sam Mason wrote on 27.08.2009 21:51: The information is all in the system catalogs; I've not had much opportunity to fiddle with them so far but the following may be a start to help get things out for you. SELECT c.relname, a.attname, t.relname FROM pg_class c, pg_depend d, pg_class t, pg_attribute a WHERE c.relkind = 'S' AND d.objid = c.oid AND d.refobjid= t.oid AND (d.refobjid,d.refobjsubid) = (a.attrelid,a.attnum); Ah great, I didn't realize I could use pg_depend for this. Thanks, works like a charm! This is exactly what I was looking for. Thomas -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
[GENERAL] query that worked in 8.1 not working in 8.4
We recently upgraded postgres from 8.1 to 8.4. One of our queries stopped working and after some digging I've narrowed the problem down to this: table structure of interest: merchant_set merchant_set_id merchant merchant_id merchant_set_id customer customer_id merchant_set_id -- failure (count=1) -- note: merchant_set.merchant_set_id in ... select count(customer.customer_id) from acquire.customer customer inner join entity_setup.merchant_set merchant_set on (customer.merchant_set_id = merchant_set.merchant_set_id and merchant_set.merchant_set_id in ( select merchant_set_id from entity_setup.merchant merchant where merchant.merchant_id in (4,8,85,67))) where customer.merchant_set_id = 2; -- success (count=3562) -- note: customer.merchant_set_id in ... select count(customer.customer_id) from acquire.customer customer inner join entity_setup.merchant_set merchant_set on (customer.merchant_set_id = merchant_set.merchant_set_id and customer.merchant_set_id in ( select merchant_set_id from entity_setup.merchant merchant where merchant.merchant_id in (4,8,85,67))) where customer.merchant_set_id = 2; Explain for query 1 (failure): Aggregate (cost=5.23..5.24 rows=1 width=4) (actual time=0.161..0.161 rows=1 loops=1) Output: count(customer.customer_id) - Nested Loop Semi Join (cost=3.23..5.22 rows=1 width=4) (actual time=0.140..0.153 rows=1 loops=1) Output: customer.customer_id - Seq Scan on merchant_set (cost=0.00..1.82 rows=1 width=4) (actual time=0.034..0.045 rows=1 loops=1) Output: merchant_set.merchant_set_id, ... Filter: (merchant_set_id = 2) - Nested Loop (cost=3.23..266.07 rows=3562 width=12) (actual time=0.101..0.101 rows=1 loops=1) Output: customer.customer_id, customer.merchant_set_id, merchant.merchant_set_id - HashAggregate (cost=3.23..3.24 rows=1 width=4) (actual time=0.081..0.081 rows=1 loops=1) Output: merchant.merchant_set_id - Seq Scan on merchant (cost=0.00..3.23 rows=1 width=4) (actual time=0.039..0.064 rows=2 loops=1) Output: merchant.merchant_id, ... , merchant.merchant_set_id, ... Filter: ((merchant_set_id = 2) AND (merchant_id = ANY ('{4,8,85,67}'::integer[]))) - Seq Scan on customer (cost=0.00..227.21 rows=3562 width=8) (actual time=0.015..0.015 rows=1 loops=1) Output: customer.customer_id, ... , customer.merchant_set_id Filter: (customer.merchant_set_id = 2) Total runtime: 0.318 ms Explain for query 2 (success): Aggregate (cost=312.42..312.43 rows=1 width=4) (actual time=17.442..17.442 rows=1 loops=1) Output: count(customer.customer_id) - Nested Loop (cost=3.23..303.51 rows=3562 width=4) (actual time=0.140..15.179 rows=3562 loops=1) Output: customer.customer_id - Seq Scan on merchant_set (cost=0.00..1.82 rows=1 width=4) (actual time=0.035..0.049 rows=1 loops=1) Output: merchant_set.merchant_set_id, ... Filter: (merchant_set_id = 2) - Nested Loop (cost=3.23..266.07 rows=3562 width=8) (actual time=0.101..11.144 rows=3562 loops=1) Output: customer.customer_id, customer.merchant_set_id - HashAggregate (cost=3.23..3.24 rows=1 width=4) (actual time=0.082..0.085 rows=1 loops=1) Output: merchant.merchant_set_id - Seq Scan on merchant (cost=0.00..3.23 rows=1 width=4) (actual time=0.038..0.064 rows=2 loops=1) Output: merchant.merchant_id, ... , merchant.merchant_set_id, ... Filter: ((merchant_set_id = 2) AND (merchant_id = ANY ('{4,8,85,67}'::integer[]))) - Seq Scan on customer (cost=0.00..227.21 rows=3562 width=8) (actual time=0.015..6.901 rows=3562 loops=1) Output: customer.customer_id, ... , customer.merchant_set_id Filter: (customer.merchant_set_id = 2) Total runtime: 17.610 ms Notes: 1. The real query gets information from customer merchant_set so both tables are necessary. The query to merchant is a security filter. 2. I have fixed this query by dropping the subquery to merchant, and inner joining to merchant directly. This forces me to add a group by so customers are not duplicated which isn't as elegant as the original query. I want to understand why the first version used to work with 8.1 and no longer works with 8.4. Is this bad sql and I was getting lucky before or is postgres making a bad decision in the latest release? -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] query that worked in 8.1 not working in 8.4
On Thu, Aug 27, 2009 at 2:13 PM, Sean Foremansean.fore...@mpaygateway.com wrote: We recently upgraded postgres from 8.1 to 8.4. So, is there an error message you get back? Or just no data? -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] query that worked in 8.1 not working in 8.4
There is no error message. Posgres likes the query. In this case, the query returns a count of 1 instead of 3562. Scott Marlowe wrote: On Thu, Aug 27, 2009 at 2:13 PM, Sean Foremansean.fore...@mpaygateway.com wrote: We recently upgraded postgres from 8.1 to 8.4. So, is there an error message you get back? Or just no data? -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] It looks like transaction, but it isn't transaction
Hello, Tom. On Thu, Aug 27, 2009 at 5:25 AM, Tom Lanet...@sss.pgh.pa.us wrote: Yes, if it's submitted as a single Query message, that's what happens. skipped Thanks for the detailed explanation. -- Sergey Samokhin -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
[GENERAL] Postgres Kickstart/Anaconda auto-install
Hey folks, I realise this is probably more a matter for a kickstart list, but then again, I have to think that someone else on this list has done this and can help. So I'll ask here and there. I'm dragging our company kicking and screaming into the realm of Kickstart/Anaconda, and trying to get PG automatically installed via a kickstart file. I'm using CentOS 5.3 and have mirrored their repository on my subnet, and all that part works great. I've also taken a copy of http://yum.pgsqlrpms.org/8.4/redhat/rhel-5-i386/ (and the 64 bit one) and want to install the RPMs from there. Automatically via Kickstart. I have this line in my kickstart file : repo --name=Postgres --baseurl=http://192.168.0.22/ks/postgres/8.4/rhel-5-i386/ and then this : @examplePG And within the directory for that repo, I've created a comps.xml file based on some googling. Here is my file - it only lists those RPMs from the repo, that I want comps !-- meta -- !-- Meta information will go here eventually -- !-- /meta -- group idExamplePG/id nameExamplePG/name defaulttrue/default descriptionExample's Own PostgreSQL Repository/description uservisibletrue/uservisible packagelist packagereq type=defaultpostgresql-server-8.4.0-1PGDG.rhel5.i386.rpm/packagereq packagereq type=defaultpostgresql-libs-8.4.0-1PGDG.rhel5.i386.rpm/packagereq packagereq type=defaultpostgresql-plperl-8.4.0-1PGDG.rhel5.i386.rpm/packagereq packagereq type=defaultpostgresql-debuginfo-8.4.0-1PGDG.rhel5.i386.rpm/packagereq packagereq type=defaultpostgresql-devel-8.4.0-1PGDG.rhel5.i386.rpm/packagereq packagereq type=defaultpostgresql-plpython-8.4.0-1PGDG.rhel5.i386.rpm/packagereq packagereq type=defaultpostgresql-8.4.0-1PGDG.rhel5.i386.rpm/packagereq packagereq type=defaultpostgresql-docs-8.4.0-1PGDG.rhel5.i386.rpm/packagereq packagereq type=defaultpostgresql-test-8.4.0-1PGDG.rhel5.i386.rpm/packagereq packagereq type=defaultpostgresql-contrib-8.4.0-1PGDG.rhel5.i386.rpm/packagereq packagereq type=defaultpostgresql-pltcl-8.4.0-1PGDG.rhel5.i386.rpm/packagereq /packagelist /group /comps My installation does not complain about the @ExamplePG line, but it does not install any of these RPMs. The only PG rpm I get is a version 8.1 libs RPM from the CentOS install. As a test I then inserted this into my kickstart file : @foobar And sure enough, the install process complains about it and asks if I want to continue or abort. Any ideas out there? thanks, -Alan -- “Don't eat anything you've ever seen advertised on TV” - Michael Pollan, author of In Defense of Food -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [SQL] [GENERAL] Data audit trail techniques in postgresql
Alvaro Herrera wrote: Rob Sargent escribió: tablelog doesn't appear any more lively than the OPs audittrail2. Perhaps, but I have heard of people using it successfully recently, whereas Nathaniel reported that audittrail2 seems to have obvious bugs. Fair enough. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] query that worked in 8.1 not working in 8.4
On Thu, Aug 27, 2009 at 2:13 PM, Sean Foremansean.fore...@mpaygateway.com wrote: We recently upgraded postgres from 8.1 to 8.4. One of our queries stopped working and after some digging I've narrowed the problem down to this: table structure of interest: merchant_set merchant_set_id merchant merchant_id merchant_set_id customer customer_id merchant_set_id So what data types are these? I'm guessing you're being bitten by some auto-cast that got removed in 8.3. But that's just a guess. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] query that worked in 8.1 not working in 8.4
Sean Foreman sean.fore...@mpaygateway.com writes: We recently upgraded postgres from 8.1 to 8.4. One of our queries stopped working and after some digging I've narrowed the problem down to this: select count(customer.customer_id) from acquire.customer customer inner join entity_setup.merchant_set merchant_set on (customer.merchant_set_id = merchant_set.merchant_set_id and merchant_set.merchant_set_id in ( select merchant_set_id from entity_setup.merchant merchant where merchant.merchant_id in (4,8,85,67))) where customer.merchant_set_id = 2; There are some bugs in 8.4.0 associated with possibly re-ordering semijoins (IN joins) incorrectly with respect to other joins. It looks like you got bit by that. Are you in a position to try 8.4 branch tip (from CVS or nightly snapshots)? If not, you'll have to wait for 8.4.1, but it'd be nice to confirm this case is fixed before we ship 8.4.1. regards, tom lane -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
[GENERAL] A safe way to upgrade table definitions by using ALTER's
Hello. As I know upgrading database structure from one version to another is usually done by applying some sql-script with a set of ALTER's that do all the work. But how do programmers guarantee that ALTER's they have wrote will always be applied by administrators to the corresponding version of the database? Is there a standard way to store some kind of metainformation in DB (like version of the current definitions of tables) and then check if it is too old for being upgraded by a given script? By database structure I mean definition of tables an application uses (data types, constraints, modificators etc), stored procedures etc. Thanks. -- Sergey Samokhin P.S. If there is other convinient way to do upgrades which I've missed - please let me know. I'm still a novice in the PostgreSQL/SQL world. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] query that worked in 8.1 not working in 8.4
This is not an autocast issue. These are all integers. The query plan postgres is choosing to execute looks questionable. I have provided an example of a working query and non-working query. The change is subtle and takes advantage of the questionable query plan to get the results I need. The join between customer and merchant looks incorrect. It should be joining merchant_set and merchant and then hitting customer. I'm not an expert at reading postgres query plans so I need some help figuring out if this is a postgres bug or a poorly written query and why. I think it may be a bug. On Thu, Aug 27, 2009 at 2:13 PM, Sean Foremansean.fore...@mpaygateway.com wrote: We recently upgraded postgres from 8.1 to 8.4. One of our queries stopped working and after some digging I've narrowed the problem down to this: table structure of interest: merchant_set merchant_set_id merchant merchant_id merchant_set_id customer customer_id merchant_set_id So what data types are these? I'm guessing you're being bitten by some auto-cast that got removed in 8.3. But that's just a guess. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] query that worked in 8.1 not working in 8.4
Thanks Tom. That is what I suspected. I will install the branch tip on Monday and see if the problem goes away. Sean Tom Lane wrote: Sean Foreman sean.fore...@mpaygateway.com writes: We recently upgraded postgres from 8.1 to 8.4. One of our queries stopped working and after some digging I've narrowed the problem down to this: select count(customer.customer_id) from acquire.customer customer inner join entity_setup.merchant_set merchant_set on (customer.merchant_set_id = merchant_set.merchant_set_id and merchant_set.merchant_set_id in ( select merchant_set_id from entity_setup.merchant merchant where merchant.merchant_id in (4,8,85,67))) where customer.merchant_set_id = 2; There are some bugs in 8.4.0 associated with possibly re-ordering semijoins (IN joins) incorrectly with respect to other joins. It looks like you got bit by that. Are you in a position to try 8.4 branch tip (from CVS or nightly snapshots)? If not, you'll have to wait for 8.4.1, but it'd be nice to confirm this case is fixed before we ship 8.4.1. regards, tom lane -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Postgres Kickstart/Anaconda auto-install
And within the directory for that repo, I've created a comps.xml file And of course re-run createrepo ... -- “Don't eat anything you've ever seen advertised on TV” - Michael Pollan, author of In Defense of Food -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] A safe way to upgrade table definitions by using ALTER's
On Aug 27, 2009, at 1:42 PM, Sergey Samokhin wrote: Hello. As I know upgrading database structure from one version to another is usually done by applying some sql-script with a set of ALTER's that do all the work. But how do programmers guarantee that ALTER's they have wrote will always be applied by administrators to the corresponding version of the database? In the application is where I do it, though it would be possible to create a .sql script that errored out if the existing version were not the one it expected. Is there a standard way to store some kind of metainformation in DB (like version of the current definitions of tables) and then check if it is too old for being upgraded by a given script? There's no standard way at the database level, though there are application frameworks that support it in a way that's standard for that framework. By database structure I mean definition of tables an application uses (data types, constraints, modificators etc), stored procedures etc. I tend to keep a single row table in the database that contains the current schema version, then have the application apply upgrade / downgrade patches as needed (or bail out and tell the user to do it). Cheers, Steve -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Postgres Kickstart/Anaconda auto-install
On Thu, 2009-08-27 at 16:28 -0400, Alan McKay wrote: Any ideas out there? There is a ks file that I wrote for PostgreSQL Live CD's: https://projects.commandprompt.com/public/pgcore/browser/livecd/PG84-Fedora-x86.ks This may guide you. It is for Fedora-11, but you can easily move it to RHEL 5. -HTH. Regards, -- Devrim GÜNDÜZ, RHCE Command Prompt - http://www.CommandPrompt.com devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org signature.asc Description: This is a digitally signed message part
[GENERAL] GUI to edit a table's content
Are there any GUI apps out there that can be used to edit (insert, update, delete) table data? SOmething with enough smarts to sniff out constraint violations and report accordingly. Also, ability to eval constraints at commit time inside a transaction (option to rollback of course). Thanks for any pointers.
Re: [GENERAL] GUI to edit a table's content
Gauthier, Dave wrote on 27.08.2009 23:17: Are there any GUI apps out there that can be used to edit (insert, update, delete) table data? SOmething with enough smarts to sniff out constraint violations and report accordingly. Also, ability to eval constraints at commit time inside a transaction (option to rollback of course). Check out this list: http://wiki.postgresql.org/wiki/Community_Guide_to_PostgreSQL_GUI_Tools -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Viable alternatives to SQL?
On Thu, Aug 27, 2009 at 9:43 AM, Kelly Joneskelly.terry.jo...@gmail.com wrote: Many sites let you search databases of information, but the search queries are very limited. I'm creating a site that'll allow arbitrary SQL queries to my data (I realize I'll need to handle injection attacks). Are there other viable ways to query data? I read a little on Business System 12 (BS12), Tutorial D, and even something called T-SQL (I think), but they all seem theoretical and not fully implemented. I want a query language that non-techies can use easily, but also supports arbitrarily complex queries. Does such a language exist? SQL is your best bet. It is probably the most natural language for non-programmers to learn and understand. It scales in difficulty with the problem you are trying to solve and is _much_ easier for non technical user than something like an ORM. That said, if your site allows arbitrary queries, sql injection is not your only problem. It is trivially easy to write a query that can DOS your server (with SQL access I could take any PostgreSQL server down in about 1 second)so you need to think about how exactly who your are going to allow access to your database and how they can access it. merlin -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
[GENERAL] Is there a function for Converting a Decimal into BINARY ?
Hi Guys, Searching the net didn't give me much clues as to how to convert a Decimal number into BINARY. Eg: I have a datatype in the DB which needs to be converted. DEC = 192 BINARY = 1100 DEC = 197 BINARY = 11000101 Which I then need to break down into pairs to do calculations on 11 : 00 : 00 : 00 11 : 00 : 01 : 01 Some of the solutions I've seen on the Internet is based on VB and mainly userland apps, I would like to do the conversion within PG itself. Thanks -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Is there a function for Converting a Decimal into BINARY ?
Ow Mun Heng wrote: Hi Guys, Searching the net didn't give me much clues as to how to convert a Decimal number into BINARY. well, a decimal number is a fixed point number stored in a modified BCD format, which optionally can contain a decimal fractional component. you likely would want to convert it to a INTEGER or BIGINT first, then cats it to BIT(n) where N is the number of bits you want. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Is there a function for Converting a Decimal into BINARY ?
Eg: I have a datatype in the DB which needs to be converted. DEC = 192 BINARY = 1100 How about this gwmdb= select 192::bit(16); bit -- 1100 (1 row) Hope that helps Allan The material contained in this email may be confidential, privileged or copyrighted. If you are not the intended recipient, use, disclosure or copying of this information is prohibited. If you have received this document in error, please advise the sender and delete the document. Neither OneSteel nor the sender accept responsibility for any viruses contained in this email or any attachments. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] How to simulate crashes of PostgreSQL?
Vick Khera wrote: On Tue, Aug 25, 2009 at 4:55 PM, Tom Lanet...@sss.pgh.pa.us wrote: I've always thought that the fd.c layer is more about not having to configure the code explicitly for max-files-per-process limits. Once you get into ENFILE conditions, even if Postgres manages to stay up, everything else on the box is going to start falling over. So the sysadmin is likely to have to resort to a reboot anyway. In my case, all sorts of processes were complaining about being unable to open files. Once Pg panicked and closed all its files, everything came back to normal. I didn't have to reboot because most everything was written to retry and/or restart itself, and nothing critical like sshd croaked. Hmm. How many DB connections were there at the time? Are they normally long-lived? I'm wondering if the problem could be caused by too many backends holding the maximum of open files each. In my system, /proc/sys/fs/file-max says ~200k, and per-process limit is 1024, so it would take about 200 backends with all FDs in use to bring the system to a near collapse that won't be solved until Postgres is restarted. This doesn't sound so far-fetched if the connections are long lived, perhaps from a pooler. Maybe we should have another inter-backend signal: when a process gets ENFILE, signal all other backends and they close a bunch of files each. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] How to simulate crashes of PostgreSQL?
On Fri, Aug 28, 2009 at 4:13 AM, Alvaro Herreraalvhe...@commandprompt.com wrote: Maybe we should have another inter-backend signal: when a process gets ENFILE, signal all other backends and they close a bunch of files each. I wonder if this is a new problem due to the FSM and VM using up extra file handles? -- greg http://mit.edu/~gsstark/resume.pdf -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] How to simulate crashes of PostgreSQL?
Alvaro Herrera alvhe...@commandprompt.com writes: Maybe we should have another inter-backend signal: when a process gets ENFILE, signal all other backends and they close a bunch of files each. I was wondering about that myself, but on balance I think it'd be a lot of work to achieve not much. What you would have is that Postgres would ramp its FD usage up to hit the kernel limit, things outside the database would fail for some period of time, then a backend would get ENFILE and we'd cut down our FD usage. Lather, rinse, repeat, ad infinitum. You'd have intermittent hard-to-reproduce failures of every other service on the box; and you'd *still* be at risk of the DB crashing, if walwriter or another low-cushion process hit the problem first. The only really reliable setup is to have max_connections times max_files_per_process less than the kernel limit. If we do something to mask the problem when it happens, I don't think we're doing the DBA a service in the long run. Thought: it's probably possible to find out the kernel limit on many platforms. Maybe postmaster startup should try to get that limit, and print an annoying warning if it's less than max_connections times max_files_per_process plus some safety factor? regards, tom lane -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general