[SQL] Re: Search
Thank you!!! It works perfect !! Regards, Sebastian PM <[EMAIL PROTECTED]> schrieb in im Newsbeitrag: 95otrr$hjg$[EMAIL PROTECTED] > force lower case or use the non case-senstitive search e.g. > > lower(column) LIKE lower('%$suchbegriffe[$i]%') > > or > > column ~* '$suchbegriffe[$i]' > (no need for wildcards when using ~* it assumes %value%)
[SQL] Load or Copy ??
Hi What is wrong? $result = pg_Exec($db,"COPY forum FROM {'datei.csv' | stdin} USING DELIMITERS '|';"); I will load datei.csv into forum! This is for MySQL: $result = pg_Exec($db,"LOAD DATA INFILE 'datei.csv' INTO TABLE forum FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'"); And for PostgreSQL: Regards, Sebastian
[SQL] Planner behaviour
Hi, I have a table with emails. I want to search this column with wildcards. To make a wildcard like this possible: "*...@somedomain.com" , I use this query: SELECT * FROM users WHERE lower(reverse_str(email))) LIKE 'moc.niamode...@%' ORDER BY email (I use reverse_str as the index only gets queried when the constant part of the string matched by LIKE is at the beginning of the string) to speed things up I have a index on "lower(reverse_str(email))" Everything works, the index is queried Now the strange part: As soos as I add "LIMIT 10" to the query: SELECT * FROM users WHERE reverse_str(email)) LIKE 'moc.niamode...@%' ORDER BY email LIMIT 10 the database does not use the "reverse_str(email)" index, but just the "email" index, and the query takes endless. Why? What can I do? Plan with "LIMIT" : explain select email FROM book_users WHERE lower(reverse_str(email)) LIKE 'moc.niamode...@%' order by email limit 10; QUERY PLAN - Limit (cost=0.00..8094.69 rows=10 width=23) -> Index Scan using book_users_email_key on book_users (cost=0.00..16868526.16 rows=20839 width=23) Filter: (lower(reverse_str((email)::text)) ~~ 'moc.niamode...@%'::text) (3 rows) Plan without "LIMIT": explain select email FROM book_users WHERE lower(reverse_str(email)) LIKE 'moc.niamode...@%' order by email; QUERY PLAN -- Sort (cost=70639.69..70691.79 rows=20839 width=23) Sort Key: email -> Bitmap Heap Scan on book_users (cost=635.19..69144.81 rows=20839 width=23) Filter: (lower(reverse_str((email)::text)) ~~ 'moc.niamode...@%'::text) -> Bitmap Index Scan on book_users_lower_rev_email_key (cost=0.00..629.98 rows=20839 width=0) Index Cond: ((lower(reverse_str((email)::text)) >= 'moc.niamodemos@'::text) AND (lower(reverse_str((email)::text)) < 'moc.niamodemosA'::text)) (6 rows) With LIMIT it takes endless, without only a fraction of a second. PS: with LIMIT 100 the behavior switches to the same behavior as without limit Thank you very much Sebastian -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql
[SQL] Search
Hi I hope someone can help me My problem: I have make a search machine whit: LIKE '%$suchbegriffe[$i]%' but when I search Test - the search machine shows only entries whit Test. But not test or tESt. (sorry for my bad english) Regards, Sebastian
[SQL] invalid input syntax for integer: ""
Hello :-) got a little prob... so my table create table warengruppen ( kennung char (1), holder int, HauptWarenGruppe int, BezHWG varchar (50), WarenGruppe int, BezWG varchar (50)); the copy syntax copy warengruppen FROM '/usr/pgsql/datanorm.wrg' with DELIMITER ';'; and a smal part of my 'datanorm.wrg' S;;011;Dachsteine+Formst.;;;S;;011;;0111;Dachst.glatt(Biber);S;;011;;0112;Dachst.glatt(Tegal.);S;;011;;0114;Dachst.mulde(Donau);S;;011;;0116;Dachst.symm.(Frankf);S;;011;;0118;Dachst.asym.(Dop.-S);S;;011;;0119;Dachst.Sonstige;S;;012;Dachziegel+Formst.;;;S;;012;;0121;Biberschwanzziegel;S;;012;;0122;Hohlz.+Krempz.;S;;012;;0123;Mnch + Nonne;S;;012;;0124;Strangfalzziegel;S;;012;;0125;Falzz.(Doppelmulde);S;;012;;0126;Flachdachziegel;S;;012;;0127;Verschiebeziegel;S;;012;;0129;Ziegel Sonstige;S;;013;Faserzementplatten;;;S;;013;;0131;Dach+Fassadenplatten;S;;013;;0133;Wellplatten;S;;013;;0135;Tafeln;S;;013;;0137;Elemente;S;;014;Fassadenpl.speziell;;;S;;014;;0141;Asphalt; and the error message ERROR: invalid input syntax for integer: ""CONTEXT: COPY warengruppen, line 1, column holder: "" i know it's hard to fill the integer colume holder with nothing out of csv table.. but it's the same with the colume WarenGruppe so i can't use a char type... so mybe someone know a litte trick to save me ?? don't like to reformat 10MB datasource ... thank's Sebastian
[SQL] Getting points from polygon
Hello, how can I get single xy-points from a polygon like for example: CREATE TABLE "public"."country" ( "country_id" INTEGER NOT NULL, "geo" polygon NOT NULL ) SELECT geo.x, geo.y FROM country WHERE geo.x=5 or SELECT geo[24th point].x FROM country ... Thanks in advance and best regards, Sebastian ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[SQL] Problem with a Pettern Matching Check
Hello hopefully correct List, I was trying to do something that is not working as it supposed to. First I created a table: create table t ( col CHAR (3) CONSTRAINT numonly_col CHECK ( col ~ '^\\d+$' ) ); This check avoids non-numbers like '1a1' and allows '123'. For some reason, I'm unable to find out why, it also avoids things like '1' and '12'. Could someone please give me hint? :) I was trying this one on PostgreSQL 8.0.3 -- Regards Sebastian Siewior ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [SQL] Problem with a Pettern Matching Check
On Mon, 15 Aug 2005 18:37:52 -0400 "Dmitri Bichko" <[EMAIL PROTECTED]> wrote: > I'm guessing it's because char gets padded with spaces to the > specified length. argh. Thank you. > Any reason you are using char(3) instead of varchar(3)? The numbers will have 2 or 3 digits so I tried to save some space :) > And why are you storing numbers as a string, anyway? If you defined > the column as a numeric type, postgres will tell you if you try to > insert something non-numeric. Correct. I will not let the values to be used together with sum/avg/+/... > > Dmitri > -- Regards Sebastian Siewior ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
[SQL] Postgres 8.1 sequences and 'CALL'-syntax
Hello, I use postgres 8.1 and trie to run jboss over sequoia-ha (http://sequoia.continuent.org/HomePage). But i have an problem with sequences. Sequoia claims to support for good reasons and db-independece only "sql-standard(s)". Therefore they DON'T support the postgres-specific "select nextval('seq_name')". Instead they gave me the hint to use the sql-conform "call nexval('seq_name')". But unfortunately i always get only an syntax error on "call" from jdbc-driver or specially postgres as result. How can i use sequences in conjunction with "call"-syntax?? Thx in advance __ Sebastian Schnabl Qualitype AG Quality Assurance Systems |Bioinformatics Moritzburger Weg 67 | 01109 Dresden fon +49.351.8838 0 | fax +49.351.8838 2809 http://www.qualitype.de __ ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [SQL] Database normalization
Hello, > > I have a fairly basic question about database design where im not sure > which approach is considered correct. > > I have two different entities: Clients and Services. Both allow users to > add progressive updates about the two entities. > > The update/message format is exactly the same for both. Should I make two > different tables: > > client_updates and service_updates > or > one table with extra columns : is_client, client_id, service_id, where > either client_id or service_id would be null depending on the is_client > boolean? > > The major problem is simply relating the foreign key in the updates table > back to the correct entity, client or service. > > Regards, > Sebastian > >
Re: [SQL] Database normalization
Thanks for the information. Both tables would be exactly sames apart from the foreign key relation to clients or services. I agree that it seems strange to have one column that is always null. Im using Django as my Object relational Mapper so im pretty sure I can not add a constraint such as : CHECK constraint where !(col1 IS NULL and col2 IS NULL). Another factor ive been considering is that one of the fields in this table(s) definition(s) is free flowing text which could potentially become very large. Should I take this in to consideration when deciding whether to split the tables? In terms of searching speed that is. Kindest regards. Sebastian On 8/28/07, Andrew Sullivan <[EMAIL PROTECTED]> wrote: > > On Tue, Aug 28, 2007 at 12:47:45PM +0100, Sebastian Ritter wrote: > > > The update/message format is exactly the same for both. Should I make > two > > > different tables: > > > > one table with extra columns : is_client, client_id, service_id, > where > > > either client_id or service_id would be null depending on the > is_client > > > boolean? > > Is the rest of the data the same? If so, then one table is right. > If not, then more than one table. In either case, I really hate the > idea of two columns, one of which is always null. But if you're > going to do that, make sure you add a CHECK constraint where !(col1 > IS NULL and col2 IS NULL). > > A > > -- > Andrew Sullivan | [EMAIL PROTECTED] > I remember when computers were frustrating because they *did* exactly what > you told them to. That actually seems sort of quaint now. > --J.D. Baldwin > > ---(end of broadcast)--- > TIP 3: Have you checked our extensive FAQ? > >http://www.postgresql.org/docs/faq >
Re: [SQL] Database normalization
Hi, The views option sounds best in my opinion but sadly i cant play with things like inheritance or views as Im using Django as my ORM and since its relatively new, it doesnt yet support such database operations. Id like the ids for each "entity" to be serial and hence there would be overlap between client_ids and service_ids. Cheers, Sebastian On 8/28/07, Erik Jones <[EMAIL PROTECTED]> wrote: > > On Aug 28, 2007, at 6:47 AM, Sebastian Ritter wrote: > > > Hello, > > > > I have a fairly basic question about database design where im not > > sure which approach is considered correct. > > > > I have two different entities: Clients and Services. Both allow > > users to add progressive updates about the two entities. > > > > The update/message format is exactly the same for both. Should I > > make two different tables: > > > > client_updates and service_updates > > or > > one table with extra columns : is_client, client_id, service_id, > > where either client_id or service_id would be null depending on the > > is_client boolean? > > > > The major problem is simply relating the foreign key in the updates > > table back to the correct entity, client or service. > > Are client_id and service_id artificial ids? If so, is it an actual > requirement that they have distinct id sequences? I.e. is it > necessary that there can be both client_id=1 and service_id=1? If > not, you can use one table, say Entities, lose client_id, service_id, > and is_clent and replace them with entity_id and entity_type. Then > your foreign key in your updates table just needs to reference > entity_id, or possibly (entity_id, entity_type) if you want to be > really strict about things. If you want to make querying the table > simple for either case create Clients and Services views on the > table. This also gives you the ability to add other entity types > where you may to track whatever kind of updates these are. > > Erik Jones > > Software Developer | Emma(r) > [EMAIL PROTECTED] > 800.595.4401 or 615.292.5888 > 615.292.0777 (fax) > > Emma helps organizations everywhere communicate & market in style. > Visit us online at http://www.myemma.com > > >
Re: [SQL] Database normalization
Hi, On 8/28/07, Andrew Sullivan <[EMAIL PROTECTED]> wrote: > > On Tue, Aug 28, 2007 at 03:37:22PM +0100, Sebastian Ritter wrote: > > Thanks for the information. > > > > Both tables would be exactly sames apart from the foreign key relation > to > > clients or services. > > Hmm. Are the services or clients tables different? A useful rule of > thumb is that, to the extent you can sort things into "kinds of > data", then you should have exactly one space for each one. (I hope > that's clear.) The table definition is exactly the same. The only difference is whether the row refers to a client or service. > Another factor ive been considering is that one of the fields in this > > table(s) definition(s) is free flowing text which could potentially > become > > very large. Should I take this in to > > consideration when deciding whether to split the tables? In terms of > > searching speed that is. > > I'd put it in its own table, probably, unless you're going to use it > frequently. Why would frequency of use change whether or not I use one or two tables? Sebastian
Re: [SQL] Database normalization
Thanks guys, Sebastian On 8/28/07, Bart Degryse <[EMAIL PROTECTED]> wrote: > > Im using Django as my Object relational Mapper so im pretty sure I > can not add a constraint such as ... > Then you should seriously consider changing your mapper. > > >>> "Sebastian Ritter" <[EMAIL PROTECTED]> 2007-08-28 16:37 >>> > Thanks for the information. > > Both tables would be exactly sames apart from the foreign key relation to > clients or services. I agree that it seems strange to have one column that > is always null. Im using Django as my Object relational Mapper so im pretty > sure I can not add a constraint such as : CHECK constraint where !(col1 > IS NULL and col2 IS NULL). > > Another factor ive been considering is that one of the fields in this > table(s) definition(s) is free flowing text which could potentially become > very large. Should I take this in to > consideration when deciding whether to split the tables? In terms of > searching speed that is. > > Kindest regards. > Sebastian > > On 8/28/07, Andrew Sullivan <[EMAIL PROTECTED]> wrote: > > > > On Tue, Aug 28, 2007 at 12:47:45PM +0100, Sebastian Ritter wrote: > > > > The update/message format is exactly the same for both. Should I > > make two > > > > different tables: > > > > > > one table with extra columns : is_client, client_id, service_id, > > where > > > > either client_id or service_id would be null depending on the > > is_client > > > > boolean? > > > > Is the rest of the data the same? If so, then one table is right. > > If not, then more than one table. In either case, I really hate the > > idea of two columns, one of which is always null. But if you're > > going to do that, make sure you add a CHECK constraint where !(col1 > > IS NULL and col2 IS NULL). > > > > A > > > > -- > > Andrew Sullivan | [EMAIL PROTECTED] > > I remember when computers were frustrating because they *did* exactly > > what > > you told them to. That actually seems sort of quaint now. > > --J.D. Baldwin > > > > ---(end of broadcast)--- > > TIP 3: Have you checked our extensive FAQ? > > > >http://www.postgresql.org/docs/faq > > > >
Re: [SQL] Was: fetch first rows of grouped data
Sorry my last post isnt complete. This would have to be part of a sub select. It would be more lilke: SELECT * from cities LEFT OUTER JOIN (SELECT c.id as city_id, event.id FROM cities c LEFT OUTER JOIN events e ON (c.id = e.city_id) WHERE c.id = cities.id ORDER BY e.date DESC LIMIT 2) as x ON (cities.id = x.city_id) I think that would work. Seb On 8/28/07, Sebastian Ritter <[EMAIL PROTECTED]> wrote: > > Hi There, > > You can do something like : > > SELECT * FROM cities c LEFT OUTER JOIN events e ON (c.id =e.city_id) ORDER > BY e.date DESC LIMIT 2 > > The left outer join here would ensure that cities with no events are also > added to your result set. > > Seb > > On 8/28/07, Guy Fraser <[EMAIL PROTECTED]> wrote: > > > > Hi there > > > > I have a list of events that take place in a certain city at a > > certain date. Now I would like to have the first two (ordered by > > date) events for each city. > > Is there a way to do this with one query? > > I am using PostgreSQL 7.4. > > > > Thanks for any tips. > > > > Claudia > > > > > > I think I may have come up with a possible solution. > > > > Create a selection that produces a unique identifier for each city > > ordered by date then use array_accum to collect the unique identifiers > > for each city, then match the first two elements of the array with the > > identifiers. > > > > For instance if you had a table : > > > > CREATE TABLE crazy_talk ( > > ct_id bigserial primary key, > > ct_city text, > > ct_date date, > > ct_data text > > ) ; > > > > Then you could use : > > > > SELECT > > ct_id , > > ct_city , > > ct_date , > > ct_data > > FROM > > crazy_talk , > > (SELECT > > ct_city AS city, > > array_accum(ct_id) as match > > FROM > > crazy_talk > > ORDER BY > > ct_city , > > ct_date > > GROUP BY > > ct_city ) AS data_set > > WHERE > > ct_city = city AND > > ct_id IN (match[0],match[1]) > > ORDER BY > > ct_city , > > ct_date > > ; > > > > I hope this helps, I did not try it, but I think it should work. > > > > PS if you don't have array_accum here it is : > > > > CREATE AGGREGATE array_accum ( > > BASETYPE = anyelement, > > SFUNC = array_append, > > STYPE = anyarray, > > INITCOND = '{}' > > ); > > > > > > > > > > ---(end of broadcast)--- > > TIP 5: don't forget to increase your free space map settings > > > >
[SQL] PL/pgSQL question
Hi all, I have a question regarding functions. How can I return zero rows from a function whose return type is a table row? I did the following test and it did not work as expected: CREATE OR REPLACE FUNCTION fn_get_user (integer) RETURNS usertable AS ' DECLARE in_userid ALIAS for $1; resulter usertable%ROWTYPE; BEGIN IF in_userid IS NULL THEN RAISE EXCEPTION ''No user provided''; RETURN null; END IF; SELECT INTO resulter usertable.* FROM usertable WHERE id = in_userid; IF FOUND THEN RETURN resulter; ELSE RETURN null; END IF; END;'LANGUAGE plpgsql; >select * from fn_get_user(-1); id | col1 | col2 | col3| name | email ++-++ || ||| | (1 row) This returns a null row. I am trying to make it behave such that it returns zero rows like a straight select. >select * from usertable where id =-1; id | col1 | col2 | col3| name | email ++-++ (0 rows) Is this possible in anyway? Regards, Sebastian
[SQL] Data layer migration from MSSQL
Hi, I'm working on migrating a data layer from MS SQL Server to PostgreSQL 8.2 and found that some reserved words should only be written between quotes and thus are case sensitive (actually only happened with a table field named "order"). Is there any way to bypass this case sensitivity or at least determine that I am going to use certain reserved words as table fields (or any other possible solutions) ? Thanks, Sebastian
Re: [SQL] Data layer migration from MSSQL
Sorry, I think it was misunderstood. I meant that I used the keyword “Order” as a table property, not as part as the statement In MS SQL I use [Order] and then just query : Select [Table1].[Order] from [Table1] Or Select [Table1].[order] from [Table1] In Postgresql I can’t do : create table Table1 (order varchar(10)) -- because “order” is a keyword I have to do it like : create Table1 (“Order” varchar(10)) And then always do the query like : Select “Order” from Table1 which is different from Select “order” from Table1 Thanks again. De: Medi Montaseri [mailto:[EMAIL PROTECTED] Enviado el: jueves, 21 de febrero de 2008 16:43 Para: Sebastian Rychter CC: pgsql-sql@postgresql.org Asunto: Re: [SQL] Data layer migration from MSSQL I think the grammer should help the parser to determine what you mean when the token ORDER is seen. for example in a select statement... Syntax: SELECT expression [, ...] ... [ FROM from_item [, ...] ] [ WHERE condition ] ... [ ORDER BY expression [ ASC | DESC | USING operator ] keywords (or reserved words) should not be placed in quotes because the parser relies on them to steer ... Cheers Medi On Thu, Feb 21, 2008 at 9:47 AM, Sebastian Rychter <[EMAIL PROTECTED]> wrote: Hi, I'm working on migrating a data layer from MS SQL Server to PostgreSQL 8.2 and found that some reserved words should only be written between quotes and thus are case sensitive (actually only happened with a table field named "order"). Is there any way to bypass this case sensitivity or at least determine that I am going to use certain reserved words as table fields (or any other possible solutions) ? Thanks, Sebastian
[SQL] Extremely Low performance with ODBC
Hi, I'm executing a query through psql ODBC which is taking around 2 minutes to complete. When I run it from PgAdmin it takes less than 3 seconds. The query itself has : . 15 inner joins (from just around 10 different tables - the other inner joins are using different aliases for the same tables) . Select statement returns 1 field. . the testing database is selecting only 1 record. Taking a look at the explain analyze report, I see they are both quite the same and tested the ODBC driver through Visual Foxpro and Vb.NET as well, taking both around 2 minutes to finish. Any idea ? Thanks, Sebastian
Re: [SQL] Extremely Low performance with ODBC
Thanks.. All logs are already off and I'm just sending one query directly from the application (from Visual Foxpro using SQL Pass through and in VB.net using ado.net). I've found that the "problem" is that I'm using implicit joins, and when I change it to explicit Inner Joins the query only takes 3 seconds (as with PgAdmin). I still can't understand if the problem resides on the ODBC driver, the Query Planner or it's just a mix of both. -Mensaje original- De: Richard Broersma [mailto:[EMAIL PROTECTED] Enviado el: Viernes, 23 de Mayo de 2008 01:04 p.m. Para: Sebastian Rychter CC: pgsql-sql@postgresql.org Asunto: Re: [SQL] Extremely Low performance with ODBC On Thu, May 22, 2008 at 11:29 AM, Sebastian Rychter <[EMAIL PROTECTED]> wrote: > Hi, I'm executing a query through psql ODBC which is taking around 2 minutes > to complete. When I run it from PgAdmin it takes less than 3 seconds. > > Any idea ? 1) turn off all ODBC query logging from both the ODBC driver and the ODBC administrator. This feature is a performance killer. Only use this feature if you are trying to debug a problem. 2) Some client programs try to perform the joins on the client side rather than the server side. I would check if the client is actually passing the actual SQL statement back to the server. If it is trying to perform the join on the client, it could be a big performance killer. -- Regards, Richard Broersma Jr. Visit the Los Angles PostgreSQL Users Group (LAPUG) http://pugs.postgresql.org/lapug __ NOD32 3124 (20080522) Information __ This message was checked by NOD32 antivirus system. http://www.eset.com -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql
Re: [SQL] Extremely Low performance with ODBC
-> Nested Loop (cost=39.33..56.20 rows=1 width=43) (actual time=4.597..5.572 rows=1 loops=1) -> Nested Loop (cost=39.33..47.92 rows=1 width=47) (actual time=4.576..5.544 rows=1 loops=1) Join Filter: ("Patient"."DoctorCountry_ID" = "PatientDoctorCountry"."Country_ID") -> Hash Join (cost=39.33..46.87 rows=1 width=51) (actual time=4.551..5.512 rows=1 loops=1) Hash Cond: ("PatientCity"."City_ID" = "Patient"."City_ID") -> Seq Scan on "City" "PatientCity" (cost=0.00..6.57 rows=257 width=4) (actual time=0.020..0.505 rows=257 loops=1) -> Hash (cost=39.32..39.32 rows=1 width=55) (actual time=4.501..4.501 rows=1 loops=1) -> Hash Join (cost=31.77..39.32 rows=1 width=55) (actual time=3.528..4.492 rows=1 loops=1) Hash Cond: ("PatientAltCity"."City_ID" = "Patient"."AltCity_ID") -> Seq Scan on "City" "PatientAltCity" (cost=0.00..6.57 rows=257 width=4) (actual time=0.005..0.486 rows=257 loops=1) -> Hash (cost=31.76..31.76 rows=1 width=59) (actual time=3.495..3.495 rows=1 loops=1) -> Hash Join (cost=24.22..31.76 rows=1 width=59) (actual time=2.515..3.486 rows=1 loops=1) Hash Cond: ("PatientEmployerCity"."City_ID" = "Patient"."EmployerCity_ID") -> Seq Scan on "City" "PatientEmployerCity" (cost=0.00..6.57 rows=257 width=4) (actual time=0.004..0.486 rows=257 loops=1) -> Hash (cost=24.20..24.20 rows=1 width=63) (actual time=2.483..2.483 rows=1 loops=1) -> Hash Join (cost=16.66..24.20 rows=1 width=63) (actual time=1.496..2.475 rows=1 loops=1) Hash Cond: ("PatientDoctorCity"."City_ID" = "Patient"."DoctorCity_ID") -> Seq Scan on "City" "PatientDoctorCity" (cost=0.00..6.57 rows=257 width=4) (actual time=0.004..0.497 rows=257 loops=1) -> Hash (cost=16.65..16.65 rows=1 width=67) (actual time=1.464..1.464 rows=1 loops=1) -> Hash Join (cost=14.57..16.65 rows=1 width=67) (actual time=1.185..1.456 rows=1 loops=1) Hash Cond: ("PatientProvince"."Province_ID" = "Patient"."Province_ID") -> Seq Scan on "Province" "PatientProvince" (cost=0.00..1.78 rows=78 width=4) (actual time=0.008..0.156 rows=78 loops=1) -> Hash (cost=14.55..14.55 rows=1 width=71) (actual time=1.131..1.131 rows=1 loops=1) -> Hash Join (cost=12.47..14.55 rows=1 width=71) (actual time=1.072..1.122 rows=1 loops=1) Hash Cond: ("PatientAltProvince"."Province_ID" = "Patient"."AltProvince_ID") -> Seq Scan on "Province" "PatientAltProvince" (cost=0.00..1.78 rows=78 width=4) (actual time=0.005..0.149 rows=78 loops=1) -> Hash (cost=12.46..12.46 rows=1 width=75) (actual time=0.733..0.733 rows=1 loops=1) -> Hash Join (cost=10.38..12.46 rows=1 width=75) (actual time=0.675..0.725 rows=1 loops=1) Hash Cond: ("PatientEmployerProvince"."Province_ID" = "Patient"."EmployerProvince_ID") -> Seq Scan on "Province" "PatientEmployerProvince" (cost=0.00..1.78 rows=78 width=4) (actual time=0.005..0.147 rows=78 loops=1) -> Hash (cost=10.36..10.36 rows=1 width=79) (actual time=0.404..0.404 rows=1 loops=1) -> Hash Join (cost=8.28..10.36 rows=1 width=79) (actual time=0.345..0.396 rows=1 loops=1) Hash Cond: ("PatientDoctorProvince"."Province_ID" = "Patient"."DoctorProvince_ID") -> Seq Scan on "Province" "PatientDoctorProvince" (cost=0.00..1.78 rows=78 width=4) (actual time=0.005..0.150 rows=78 loops=1) -> Hash (cost=8.27..8.27 rows=1 width=83) (actual time=0.044..0.044 rows=1 loops=1) -> Index Scan using "Patient_pkey" on "Patient" (cost=0.00..8.27 rows=1 width=83) (actual time=0.028..0.032 rows=1 loops=1) Index Cond: ("Patient_ID" = 10450) -> Seq Scan on "Country" "PatientDoctorCountry" (c
Re: [SQL] Extremely Low performance with ODBC
ntEmployerCountry, "Country" PatientDoctorCountry WHERE "Patient"."Family_ID"="PatientFamily"."Family_ID" AND "Patient"."FamilyRole_ID"="PatientFamilyRole"."FamilyRole_ID" AND "Patient"."Title_ID"="Title"."Title_ID" AND "Patient"."City_ID"=PatientCity."City_ID" AND "Patient"."Province_ID"=PatientProvince."Province_ID" AND "Patient"."Country_ID"=PatientCountry."Country_ID" AND "Patient"."AltCity_ID"=PatientAltCity."City_ID" AND "Patient"."AltProvince_ID"=PatientAltProvince."Province_ID" AND "Patient"."AltCountry_ID"=PatientAltCountry."Country_ID" AND "Patient"."EmployerCity_ID"=PatientEmployerCity."City_ID" AND "Patient"."EmployerProvince_ID"=PatientEmployerProvince."Province_ID" AND "Patient"."EmployerCountry_ID"=PatientEmployerCountry."Country_ID" AND "Patient"."DoctorCity_ID"=PatientDoctorCity."City_ID" AND "Patient"."DoctorProvince_ID"=PatientDoctorProvince."Province_ID" AND "Patient"."DoctorCountry_ID"=PatientDoctorCountry."Country_ID" AND "Patient"."Patient_ID"=10450 -Mensaje original- De: Richard Broersma [mailto:[EMAIL PROTECTED] Enviado el: Viernes, 23 de Mayo de 2008 07:00 p.m. Para: Sebastian Rychter CC: pgsql-sql@postgresql.org Asunto: Re: [SQL] Extremely Low performance with ODBC On Fri, May 23, 2008 at 2:45 PM, Sebastian Rychter <[EMAIL PROTECTED]> wrote: > Thanks for your response. I did that in the first place and found that both > explain reports where quite similar. I've got both reports: From pgadmin it > took 7288 ms, and from VFP took 6446 ms (of course, that's the time > calculated within the explain analyze, and not the real elapsed time). actually it looks like 7.288ms and 6.446ms. ;) > That's why I assume the ODBC driver is taking its own time for parsing or > something like that- around 2 minutes actually :) Checking the Postgresql logs would be really helpful here. 1) note the time that you issue the query from your client FE. 2) note the time in the pg server logs that the server gets the passed query string. 2b) check for any disparity between 1) and 2). 3) check that the query is exactly the same as you've hand coded in pg-admin 4) also check the query duration when issued by ODBC in the postgresql log to see if it is really taking 2 minutes. It would be nice to see the snippet of the PG log showing this ODBC query with associate duration times. -- Regards, Richard Broersma Jr. Visit the Los Angles PostgreSQL Users Group (LAPUG) http://pugs.postgresql.org/lapug __ NOD32 3127 (20080523) Information __ This message was checked by NOD32 antivirus system. http://www.eset.com -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql
Re: [SQL] Extremely Low performance with ODBC
I already tried executing them in different order, but that only affects in just a few milliseconds (the one with explicit joins remains much faster). The bad news is that I already compared both Explain Analyze results (from pgAdmin and ODBC) and they follow the same steps. I also tried comparing the Explain Analyze between the Explicit Join and the Implicit Join, and though they are different, the Total time is almost the same. I'll keep on looking for any other differences on different logs.. I think I already searched all the Postgresql forums I know and no one ever reported something like that before. Thanks once again, Sebastian -Mensaje original- De: Richard Broersma [mailto:[EMAIL PROTECTED] Enviado el: Martes, 27 de Mayo de 2008 01:14 a.m. Para: Sebastian Rychter CC: pgsql-sql@postgresql.org Asunto: Re: [SQL] Extremely Low performance with ODBC On Mon, May 26, 2008 at 8:26 PM, Sebastian Rychter <[EMAIL PROTECTED]> wrote: > 2008-05-27 02:55:10 GMT LOG: duration: 112157.000 ms statement: SELECT > 2008-05-27 03:07:29 GMT LOG: duration: 250.000 ms statement: SELECT Wow, that is a big difference. I reformated your querys and diff'ed them. You are correct, they are exactly the same query. My only other guess would be that the first time you execute this query using ODBC, you need to get a fresh read from disk. However, once this dataset is cached in memory it returns much faster when you requery it using PGAdmin. If you run the same query twice conseqitively in ODBC, does the second query perform much better or do both querys perform badly? If they both odbc queries perform badly, the last thing that I would try would be to capture the results of : EXPLAIN ANALYZE SELECT ...(your query), from both ODBC, and PGADMIN. If we compare them It should at least tell us what the server is doing differently that is causing the query mis-behavior. -- Regards, Richard Broersma Jr. Visit the Los Angles PostgreSQL Users Group (LAPUG) http://pugs.postgresql.org/lapug __ NOD32 3133 (20080526) Information __ This message was checked by NOD32 antivirus system. http://www.eset.com -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql
Re: [SQL] Extremely Low performance with ODBC
Thanks. Hiroshi sent me a possible solution (which might be useful for others going through similar situations), which is to toggle off the "Disable Genetic Optimizer" from the ODBC driver configuration. -Mensaje original- De: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] En nombre de Steve Midgley Enviado el: Miércoles, 28 de Mayo de 2008 03:29 a.m. Para: [EMAIL PROTECTED]; pgsql-sql@postgresql.org CC: [EMAIL PROTECTED] Asunto: Re: [SQL] Extremely Low performance with ODBC At 09:20 PM 5/27/2008, [EMAIL PROTECTED] wrote: >Date: Tue, 27 May 2008 09:29:56 -0700 >From: "Richard Broersma" <[EMAIL PROTECTED]> >To: "Sebastian Rychter" <[EMAIL PROTECTED]> >Cc: pgsql-sql@postgresql.org >Subject: Re: Extremely Low performance with ODBC >Message-ID: ><[EMAIL PROTECTED]> > >On Mon, May 26, 2008 at 9:34 PM, Sebastian Rychter ><[EMAIL PROTECTED]> wrote: > > > I'll keep on looking for any other differences on different logs.. > I think I > > already searched all the Postgresql forums I know and no one ever > reported > > something like that before. Apologies for the double post on this - I wonder also if you have tried alternative Pg ODBC drivers? I found this one while googling around: http://uda.openlinksw.com/odbc/st/odbc-progress-st/ But I'd guess there are other alternatives. Also, you might check out their ADO driver and see if it gives you the same problems.. Just some more grist for the solution mill, hopefully! Steve -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql __ NOD32 3137 (20080527) Information __ This message was checked by NOD32 antivirus system. http://www.eset.com -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql
[SQL] reliable lock inside stored procedure
Hi, I have a stored procedure and I need a SHARE ROW EXCLUSIVE lock in this procedure (otherwise data will get corrupted). According to the documentation the LOCK statement is useless and will silently fail if not executed inside a transaction. (btw: this sounds dangerous to me) Also it is not possible to start a transaction inside a stored procedure. How can I make the code of this stored procedure safe? is there a way to let the procedure fail with an error if not executed within a transaction. does adding a SAVEPOINT the trick? documentation says that savepoints can only be established inside transactions, but does it fail fatal enough so that the procedure getss aborted? (more fatal than LOCK does?) thank you very much. /sebastian -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql
Re: [SQL] reliable lock inside stored procedure (SOLVED)
Hi Richard, thank you for your answer! Am 03.11.2008 um 12:06 schrieb Richard Huxton: Sebastian Böhm wrote: Hi, I have a stored procedure and I need a SHARE ROW EXCLUSIVE lock in this procedure (otherwise data will get corrupted). OK. PostgreSQL doesn't have "stored procedures" so I guess you're talking about a function. yes According to the documentation the LOCK statement is useless and will silently fail if not executed inside a transaction. (btw: this sounds dangerous to me) I'm not sure what you mean here, and I don't think you've understood the documentation. It's not possible to have a LOCK statement outside of a transaction. It's just not meaningful to have a transaction that only has a LOCK statement in it. as postgres does not warn you about this, this may lead to not so easy to spot bugs. If you forget to start a transaction and assume that you got a lock while modifieing a table, you can corrupt data. Also it is not possible to start a transaction inside a stored procedure. All functions execute within a transaction. As do all other SELECT, UPDATE, INSERT, DELETE, CREATE TABLE etc. If you (or your client library) do not supply a BEGIN then the command is treated as though BEGIN...COMMIT surrounded it. You can't have nested transactions because the COMMIT of a subtransaction wouldn't be meaningful. You can use SAVEPOINTS to roll back to a known point. so a call to a function is ALLWAYS a transaction ? good then I have no problem... How can I make the code of this stored procedure safe? is there a way to let the procedure fail with an error if not executed within a transaction. You can't execute outside of a transaction. It's not possible. does adding a SAVEPOINT the trick? documentation says that savepoints can only be established inside transactions, but does it fail fatal enough so that the procedure getss aborted? (more fatal than LOCK does?) I'm not sure I understand what you mean here. I assumed that a function can be executed without a transaction, means: every statement in the function is its own transaction. I understood that this is not the case. As SAVEPOINTS failes outside of a transaction I could then be used to detect wether there is a transaction already started or not. Imagine that you have a function in your code (not a postgres- function, but a C function) and this functions issues some statements that may corrupt data if no transaction is already created. You may then use SAVEPOINT in that c-function to make sure that there is a transaction started, as SAVEPOINT creates an exception if no transaction was started. /sebastian -- Richard Huxton Archonet Ltd -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql
Re: [SQL] Query optimizing
Hi Helio, Sorry about the parenthesis - Bad copy/pasting skills! To further discuss your suggestion: Wouldn't adding n_issue=i.id as a where clause filter cause the sub-query to become correlated and thus much less efficient ? I may be wrong, or may have miss-understood your suggestion. Thanks for you help, Sebastian On Mon, Nov 10, 2008 at 11:48 AM, Helio Campos Mello de Andrade < [EMAIL PROTECTED]> wrote: > Hi Sebastian, > > - First of all i think there is an open-parenthesis missing in the query > V2. > Maybe in the V2 version you cold restrict the results in the INNER query a > bit more if you use a restriction clause like "WHERE n_issue = i.id" in > that. It will certainly lower the number of rows returned by it to only 1 > result. > > Regards > > -- > Helio Campos Mello de Andrade > > > > On Mon, Nov 10, 2008 at 8:43 AM, Sebastian Ritter < > [EMAIL PROTECTED]> wrote: > >> Hi all, >> >> I was hoping to receive some advise on a slow running query in our >> business' Issue Tracking System. To shed some light on the below mentioned >> queries, here is a brief summary of how users interact with the system. The >> two main components in the system are a Issues and Followups. An Issue is >> created by our technical support staff when some form of action needs to be >> taken in order to resolve a problem. FollowUps are entries entered against >> an issue depicting the sequence of events taken to resolve the issue. There >> are about 15,000 Issues in the system at present and about 95,000 FollowUps >> in the system. As we need the system to be very responsive, each query >> should ideally run in under 1 second. >> >> A lot of the reports our technical officers submit to us include a listing >> of all actioned issues for a given day along with the last modified followup >> of each said issue. With the number of rows in our database increasing at a >> high rate, these queries are starting to run too slowly. >> >> Here is a condensed version of the two tables: >> >> Issues: >> = >> id - integer >> dt_created - timestamp >> dt_modified - timestamp >> t_title - varchar >> t_description - varchar >> >> FollowUps: >> = >> id - integer >> dt_created - timestamp >> dt_modified - timestamp >> t_description - varchar >> n_issue - foregin key to issues >> >> We have identified that the slowness in our queries is trying to return >> the lastest followup for each actioned issue that day. Without further ado >> here are two variations I have tried within the system (neither of which are >> making the cut): >> >> V1 (correlated subquery - Very bad performance) >> >> (SELECT >> fu.* >> FROM >> manage_followup fu, >> manage_issue i >> WHERE >> i.id = fu.n_issue >> AND >> fu.id = (SELECT >> id >>FROM >> manage_followup >> WHERE >> n_issue = i.id >> ORDER BY >> dt_modified DESC >> LIMIT 1)) AS latestfu, >> >> V2 (Using Group By, "max" aggregate function and distinct- better >> performance, but still bad because of distinct) >> >> >> SELECT DISTINCT ON (fu.n_issue) >> fu.id, >> fu.dt_created, >> fu.dt_modified, >> fu.t_description, >> fu.n_issue as issue_id >> FROM >> manage_followup fu, >> (SELECT >> n_issue, >> max(dt_modified) as dt_modified >> FROM >> manage_followup >> GROUP BY >> n_issue) as max_modified >> WHERE >> max_modified.n_issue = fu.n_issue >> AND >> fu.dt_modified = max_modified.dt_modified) >> AS latestfu ON (latestfu.issue_id = i.id), >> >> We must use distinct here as we sometimes use batch scripts to enter >> followups, which will give them all similar, if not equal, modification >> dates. We also can't use followup ids as an indicator of the latest followup >> because users of the system can retrospectively go back and change older >> followups. >> >> I was hoping some one could provide a solution that does not require a >> corrolated subquery or make use of the distinct keyword. Any help would be >> much appreciated. >> >> Kind regards, >> Sebastian >> >> >> >> >> >>
Re: [SQL] Query optimizing
Cheers for you help guys. Having filtered and then joined has substantially reduced the run time. Much obliged, Sebastian On Mon, Nov 10, 2008 at 12:32 PM, Richard Huxton <[EMAIL PROTECTED]> wrote: > Sebastian Ritter wrote: > > Could it have something > > to do with the fact that it is a subquery and thus the planner can not > > deduce filtering conditions from the outer query against it? My > apologises > > if that made no sense. > > Could make a difference. > > > In summary, what im trying to understand is the following: Will there be > a > > performance difference between filtering query sets first and then > joining > > them together as opposed to joining first and then filtering? Does the > > opitmiser not choose the best course of action either way yielding the > same > > result? > > There obviously is a performance difference between joining all of the > issues table versus joining 1% of it to followups. > > In most cases the planner can push the condition into the subquery, but > not in all cases because: > 1. It's not provably correct to do so > 2. The planner isn't smart enough to figure out that it can > > It's impossible to say which applies to you without knowing the full query. > > -- > Richard Huxton > Archonet Ltd >
[SQL] Query optimizing
Hi all, I was hoping to receive some advise on a slow running query in our business' Issue Tracking System. To shed some light on the below mentioned queries, here is a brief summary of how users interact with the system. The two main components in the system are a Issues and Followups. An Issue is created by our technical support staff when some form of action needs to be taken in order to resolve a problem. FollowUps are entries entered against an issue depicting the sequence of events taken to resolve the issue. There are about 15,000 Issues in the system at present and about 95,000 FollowUps in the system. As we need the system to be very responsive, each query should ideally run in under 1 second. A lot of the reports our technical officers submit to us include a listing of all actioned issues for a given day along with the last modified followup of each said issue. With the number of rows in our database increasing at a high rate, these queries are starting to run too slowly. Here is a condensed version of the two tables: Issues: = id - integer dt_created - timestamp dt_modified - timestamp t_title - varchar t_description - varchar FollowUps: = id - integer dt_created - timestamp dt_modified - timestamp t_description - varchar n_issue - foregin key to issues We have identified that the slowness in our queries is trying to return the lastest followup for each actioned issue that day. Without further ado here are two variations I have tried within the system (neither of which are making the cut): V1 (correlated subquery - Very bad performance) (SELECT fu.* FROM manage_followup fu, manage_issue i WHERE i.id = fu.n_issue AND fu.id = (SELECT id FROM manage_followup WHERE n_issue = i.id ORDER BY dt_modified DESC LIMIT 1)) AS latestfu, V2 (Using Group By, "max" aggregate function and distinct- better performance, but still bad because of distinct) SELECT DISTINCT ON (fu.n_issue) fu.id, fu.dt_created, fu.dt_modified, fu.t_description, fu.n_issue as issue_id FROM manage_followup fu, (SELECT n_issue, max(dt_modified) as dt_modified FROM manage_followup GROUP BY n_issue) as max_modified WHERE max_modified.n_issue = fu.n_issue AND fu.dt_modified = max_modified.dt_modified) AS latestfu ON (latestfu.issue_id = i.id), We must use distinct here as we sometimes use batch scripts to enter followups, which will give them all similar, if not equal, modification dates. We also can't use followup ids as an indicator of the latest followup because users of the system can retrospectively go back and change older followups. I was hoping some one could provide a solution that does not require a corrolated subquery or make use of the distinct keyword. Any help would be much appreciated. Kind regards, Sebastian
Re: [SQL] Query optimizing
Cheers for this Richard. The more I think about it, I believe the join is being made against ALL issues and followups first and then filtered by my where clause conditions afterwards. This would in incur a scan against all 15,000 issues and 95,000 followups. Set theory tells me that I should not use the entire issue table but rather the subset of interest and then join it to the followup table, instead of joining the two tables and then filtering the results. I was under the impression that the postgresql optimizer would have done this logically by itself. Could it have something to do with the fact that it is a subquery and thus the planner can not deduce filtering conditions from the outer query against it? My apologises if that made no sense. In summary, what im trying to understand is the following: Will there be a performance difference between filtering query sets first and then joining them together as opposed to joining first and then filtering? Does the opitmiser not choose the best course of action either way yielding the same result? This might be a stupid question. Sebastian On Mon, Nov 10, 2008 at 12:03 PM, Richard Huxton <[EMAIL PROTECTED]> wrote: > Richard Huxton wrote: > > Do you have an index on (id,dt_modified) for manage_followup? Can you > > provide an EXPLAIN ANALYSE for this? > > > Hi Richard, > > > > Firstly, thank-you very much for your swift reply. To answer your > > question, > > I had not been using an index on dt_modfied. I have added it now and > > ran explain analyse on the function snippet. I am almost too > > embarrassed > to show > > the result > > > > QUERY PLAN > [snip] > > Total runtime: 412464.804 ms > > Something wrong here. I've attacked a small script that generates 10,000 > issues and 10 follow-ups for each. It then pulls off the most recent > follow-ups for all issues occurring on a given date. > > The explain analyse should show both indexes being used and a runtime of > a few milliseconds. > > -- > Richard Huxton > Archonet Ltd >
[SQL] Downgrade database and problem with sequences
I am currently using postgresql 8.3 and whenever I want to get the last inserted ID use : Select currval(pg_get_serial_sequence(table, field)) Then, I had to downgrade a database for certain customer to use postgresql 8.1, and when I execute that same function I found that pg_get_serial_sequence returns nothing. Is there any way to dump my database and then restore it on a previous version and keep using that pg_get_serial_sequence() function ? Thanks, Sebastian -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql
[SQL] subquery question
Hi, I have a table: (date timestamp, id integer, value integer) What Iam trying to do is to get a result that looks like this: day sum_oddsum_even 2009-01-01 656578867 2009-01-02 876785 87667 basically a need to combine these two queries into one: SELECTdate_trunc('day',date) AS day, sum(value) AS sum_odd FROM xyz WHEREid % 2 = 1GROUP BY date_trunc('day',date) SELECTdate_trunc('day',date) AS day, sum(value) AS sum_even FROM xyz WHEREid % 2 = 0GROUP BY date_trunc('day',date) I found various ways to do this via unions or joins, but none of them seem efficient, what is the best way to do that ? thank you very much Sebastian
[SQL] start and end of the week
How do I get the start and end date of the present week? Is this possible? For example this week Start = Sept. 22 End = Sept. 28 Thank you very much. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [SQL] \copy multiline
On Thu, 29 Nov 2012 08:01:31 +, Ben Morrow wrote: > Quoth splu...@gmail.com (Seb): >> I use \copy to output tables into CSV files: >> \copy (SELECT ...) TO 'a.csv' CSV >> but for long and complex SELECT statements, it is cumbersome and >> confusing to write everything in a single line, and multiline >> statements don't seem to be accepted. Is there an alternative, or am >> I missing an continuation-character/option/variable that would allow >> multiline statements in this case? > CREATE TEMPORARY VIEW? Of course, that's perfect. Thanks! -- Seb -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql
Re: [SQL] \copy multiline
On Thu, 29 Nov 2012 10:33:37 +0100, Guillaume Lelarge wrote: > On Wed, 2012-11-28 at 21:21 -0600, Seb wrote: >> Hi, >> I use \copy to output tables into CSV files: >> \copy (SELECT ...) TO 'a.csv' CSV >> but for long and complex SELECT statements, it is cumbersome and >> confusing to write everything in a single line, and multiline >> statements don't seem to be accepted. Is there an alternative, or am >> I missing an continuation-character/option/variable that would allow >> multiline statements in this case? > A simple way to workaround this issue is to create a view with your > query and use the view in the \copy meta-command of psql. Of course, > it means you need to have the permission to create views in the > database. Thanks. Someone also suggested creating a temporary view, which helps keep the schema sane and clean. Cheers, -- Seb -- Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-sql