Fwd: [SQL] COPY to table with array columns (Longish)
Can you provide an example?Thanks,AaronOn 6/11/06, Phillip Smith <[EMAIL PROTECTED] > wrote: Hi All, Hope someone can help me – our main company system runs on Raining Data PICK/D3 (if anyone familiar with it) which stores records in it's "tables" as variable length items. Every item has a unique Primary Key (per table) then each item can have a variable number of fields. These fields are delimited by Char 254, then each field can have sub-values delimited by Char 253, then sub-sub-values delimited by Char 252. Anyway, we are trying to export everything to Postgres for reporting and querying etc (not to actually run the system…. Yet) and hasn't been a problem so far – everything like stock and purchase orders, sales orders etc can pretty easily be turned in to a flat file with standard number of columns and consistent data. We truncate every table each night then import that latest TSV export from D3 using a COPY command. The problem arises with tables like our SYS table which store generic system data, so one record could have 3 fields, but the next could have 300. The only way I can work out how to export multi-valued data like this to Postgres is to use an array column. So the table has 2 columns – the pkey and a data array. How do I get this imported to the truncated table each night? At the moment I think my best option is to modify the export for the SYS table to call PSQL and use standard SQL INSERT statements to directly insert it instead of exporting to a flat file, then import to Postgres. Thanks all, -p For those who are interested, or if it might help, here's a rough comparison of the database structure of D3: Windows = PICK/D3 = Postgres Drive = Account = Database Directory = File = Table File = Item = Row Line in text file = Attribute = Field (none) = Value = Array Element (?) (none) = Sub Value = (none?) Phillip Smith IT Coordinator Weatherbeeta P/L 8 Moncrief Rd Nunawading, VIC, 3131 AUSTRALIA E. [EMAIL PROTECTED]
[SQL] Refactor Query...
hi all, i'm trying to refactor this query: http://www.rafb.net/paste/results/RnkFWZ46.html i've added notes to make its function a bit more clear. i have trie dto refactor it to get rid of the "distinct on" function, but i've had no luck. due to all the WHERE clauses, i can't figure out how to do an inner join with something like this: select max(t_inspect_result.inspect_result_timestamp) as mstamp, t_inspect_result.inspect_id from t_inspect_result group by t_inspect_result.inspect_id ) AS b thanks in advance... __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [SQL] Refactor Query... SOLVED
--- [EMAIL PROTECTED] wrote: > hi all, > > i'm trying to refactor this query: > > http://www.rafb.net/paste/results/RnkFWZ46.html > > i've added notes to make its function a bit more > clear. i have trie dto refactor it to get rid of > the > "distinct on" function, but i've had no luck. > > due to all the WHERE clauses, i can't figure out how > to do an inner join with something like this: > > select > max(t_inspect_result.inspect_result_timestamp) > as mstamp, > t_inspect_result.inspect_id > from t_inspect_result > group by t_inspect_result.inspect_id > ) AS b > > thanks in advance... these queries (i have them all over my app!) are getting complex and i'm being challenged to keep everything in order. this is the final result: http://www.rafb.net/paste/results/pnvLSs30.html here's a special thanks to all who have helped on this and other recent query problems. thank you! i *rally* appreciate the chared insight and guidance. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [SQL] COPY to table with array columns (Longish)
No problem! SYS 'ZKCOST' size = 21 01 2750 This is a simple record in the SYS file. Primary Key is “ZKCOST” and the Data would be 1 element with a value of 2750. SYS 'ZPRECMPL' size = 2069 O 01 928898 02 928899 03 928900 04 928901 05 928902 06 928903 07 928904 08 928907 09 928908 10 928909 11 928910 12 928915 13 928916 14 928917 15 928918 16 928919 17 928920 18 928921 19 928925 20 928926 21 928927 Another SYS record – this is a list of invoices that are waiting to be confirmed and therefore obviously constantly change with additions and deletions. So in the SQL this would need to have a Primary Key of “ZPRECMPL” and 21 elements in the array, each with an invoice number… (well, there’s actually 293 in there at the moment, but I don’t think I need to fill up the e-mail with all of them!) The whole sys file is variable length records like this – they range from 1 to over 17,000 fields per record. Hope this helps, Thanks, -p -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Aaron Bono Sent: Tuesday, 13 June 2006 12:36 AM To: [EMAIL PROTECTED] Cc: pgsql-sql@postgresql.org Subject: Re: [SQL] COPY to table with array columns (Longish) Can you provide an example? Thanks, Aaron On 6/11/06, Phillip Smith <[EMAIL PROTECTED] > wrote: Hi All, Hope someone can help me – our main company system runs on Raining Data PICK/D3 (if anyone familiar with it) which stores records in it's "tables" as variable length items. Every item has a unique Primary Key (per table) then each item can have a variable number of fields. These fields are delimited by Char 254, then each field can have sub-values delimited by Char 253, then sub-sub-values delimited by Char 252. Anyway, we are trying to export everything to Postgres for reporting and querying etc (not to actually run the system…. Yet) and hasn't been a problem so far – everything like stock and purchase orders, sales orders etc can pretty easily be turned in to a flat file with standard number of columns and consistent data. We truncate every table each night then import that latest TSV export from D3 using a COPY command. The problem arises with tables like our SYS table which store generic system data, so one record could have 3 fields, but the next could have 300. The only way I can work out how to export multi-valued data like this to Postgres is to use an array column. So the table has 2 columns – the pkey and a data array. How do I get this imported to the truncated table each night? At the moment I think my best option is to modify the export for the SYS table to call PSQL and use standard SQL INSERT statements to directly insert it instead of exporting to a flat file, then import to Postgres. Thanks all, -p For those who are interested, or if it might help, here's a rough comparison of the database structure of D3: Windows = PICK/D3 = Postgres Drive = Account = Database Directory = File = Table File = Item = Row Line in text file = Attribute = Field (none) = Value = Array Element (?) (none) = Sub Value = (none?) Phillip Smith IT Coordinator Weatherbeeta P/L 8 Moncrief Rd Nunawading, VIC, 3131 AUSTRALIA E. [EMAIL PROTECTED] ***Confidentiality and Privilege Notice*** The material contained in this message is privileged and confidential to the addressee. If you are not the addressee indicated in this message or responsible for delivery of the message to such person, you may not copy or deliver this message to anyone, and you should destroy it and kindly notify the sender by reply email. Information in this message that does not relate to the official business of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta. Weatherbeeta, its employees, contractors or associates shall not be liable for direct, indirect or consequential loss arising from transmission of this message or any attachments
Re: [SQL] COPY to table with array columns (Longish)
"Phillip Smith" <[EMAIL PROTECTED]> writes: > The whole sys file is variable length records like this - they range = > from 1 > to over 17,000 fields per record. 17000? I think you really need to rethink your schema. While you could theoretically drop 17000 elements into a PG array column, you wouldn't like the performance --- it'd be almost unsearchable for instance. I'd think about two tables, one with a single row for each SYS record from the original, and one with one row for each detail item (the invoice numbers in this case). With suitable indexes and a foreign key constraint, this will perform a lot better than an array-based translation. And no, in neither case will you be able to import that file without massaging it first. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [SQL] COPY to table with array columns (Longish)
I agree with Tom. Personally I cannot think of a time I would use an array column over a child table. Maybe someone can enlighten me on when an array column would be a good choice.What language are you using to do the export if I may ask? -AaronOn 6/12/06, Tom Lane <[EMAIL PROTECTED]> wrote: "Phillip Smith" <[EMAIL PROTECTED]> writes:> The whole sys file is variable length records like this - they range => from 1> to over 17,000 fields per record. 17000? I think you really need to rethink your schema. While you couldtheoretically drop 17000 elements into a PG array column, you wouldn'tlike the performance --- it'd be almost unsearchable for instance. I'd think about two tables, one with a single row for each SYS recordfrom the original, and one with one row for each detail item (theinvoice numbers in this case). With suitable indexes and a foreign key constraint, this will perform a lot better than an array-basedtranslation.And no, in neither case will you be able to import that file withoutmassaging it first.regards, tom lane
Re: [SQL] COPY to table with array columns (Longish)
The export from the D3 system is written in PICK BASIC – similar to ‘normal’ BASIC. This currently exports Attribute (field) 1 to 9 of each SYS record with a Primary Key starting with “Z” (‘]’ is a wildcard in the SSELECT statement) 153 EXECUTE 'SSELECT SYS WITH A0 "Z]"' CAPTURING JUNK 154 LOOP 155 READNEXT SYS.ID THEN 156 READ SYS.REC FROM SYS, SYS.ID ELSE 157 SYS.ID = 'XX' 158 END 159 * 160 OUTLINE = SYS.ID 161 FOR SYS.SUB = 1 TO 9 162 OUTLINE = OUTLINE:AM:SYS.REC 163 NEXT SYS.SUB 164 * 165 CONVERT CHAR(252) TO "" IN OUTLINE 166 CONVERT CHAR(92) TO "" IN OUTLINE 167 CONVERT CHAR(34) TO "" IN OUTLINE 168 OUTLINE = OCONV(OUTLINE, "MCU") 169 N=%FPUTS(OUTLINE:NL, (CHAR*)STREAM) 170 END ELSE 171 SYS.ID = 'XX' 172 END 173 490 NULL 174 UNTIL SYS.ID = 'XX' DO REPEAT So you’re suggesting creating a child table for each SYS record? Ie, a table called “ZPRECMPL” etc? Thanks for your input guys, Cheers, -p -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Aaron Bono Sent: Tuesday, 13 June 2006 12:58 PM To: Tom Lane Cc: [EMAIL PROTECTED]; pgsql-sql@postgresql.org Subject: Re: [SQL] COPY to table with array columns (Longish) I agree with Tom. Personally I cannot think of a time I would use an array column over a child table. Maybe someone can enlighten me on when an array column would be a good choice. What language are you using to do the export if I may ask? -Aaron On 6/12/06, Tom Lane <[EMAIL PROTECTED]> wrote: "Phillip Smith" <[EMAIL PROTECTED]> writes: > The whole sys file is variable length records like this - they range = > from 1 > to over 17,000 fields per record. 17000? I think you really need to rethink your schema. While you could theoretically drop 17000 elements into a PG array column, you wouldn't like the performance --- it'd be almost unsearchable for instance. I'd think about two tables, one with a single row for each SYS record from the original, and one with one row for each detail item (the invoice numbers in this case). With suitable indexes and a foreign key constraint, this will perform a lot better than an array-based translation. And no, in neither case will you be able to import that file without massaging it first. regards, tom lane ***Confidentiality and Privilege Notice*** The material contained in this message is privileged and confidential to the addressee. If you are not the addressee indicated in this message or responsible for delivery of the message to such person, you may not copy or deliver this message to anyone, and you should destroy it and kindly notify the sender by reply email. Information in this message that does not relate to the official business of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta. Weatherbeeta, its employees, contractors or associates shall not be liable for direct, indirect or consequential loss arising from transmission of this message or any attachments
Re: [SQL] COPY to table with array columns (Longish)
On Mon, 2006-06-12 at 21:58 -0500, Aaron Bono wrote: > I agree with Tom. Personally I cannot think of a time I would use an > array column over a child table. Maybe someone can enlighten me on > when an array column would be a good choice. Arrays are a good choice when the data comes naturally segmented. I regularly store sentences or sentence fragments as an array of words when I don't care about whitespace (whitespace should be regenerated on the output). Consider meta-keywords for a webpage for example. This is similar process as removing all formatting from phone numbers before storage and reformatting for display to the user again. (123)456-7890 might be stored as '1234567890'. "A\tcat in the\nhat" might be stored as ARRAY['A', 'cat', 'in', 'the', 'hat']. It makes comparisons and uniqueness much easier to deal with if you remove garbage from the data prior to storage. > On 6/12/06, Tom Lane <[EMAIL PROTECTED]> wrote: > "Phillip Smith" <[EMAIL PROTECTED]> writes: > > The whole sys file is variable length records like this - > they range = > > from 1 > > to over 17,000 fields per record. > > 17000? I think you really need to rethink your schema. While > you could > theoretically drop 17000 elements into a PG array column, you > wouldn't > like the performance --- it'd be almost unsearchable for > instance. > > I'd think about two tables, one with a single row for each SYS > record > from the original, and one with one row for each detail item > (the > invoice numbers in this case). With suitable indexes and a > foreign key > constraint, this will perform a lot better than an array-based > translation. > > And no, in neither case will you be able to import that file > without > massaging it first. > > regards, tom lane > -- ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [SQL] COPY to table with array columns (Longish)
> On Mon, 2006-06-12 at 21:58 -0500, Aaron Bono wrote: > > I agree with Tom. Personally I cannot think of a time I would use an > > array column over a child table. Maybe someone can enlighten me on > > when an array column would be a good choice. > > Arrays are a good choice when the data comes naturally segmented. > Also, for a case and point, some of the postgresql system tables use arrays. I suppose that these would be examples were the core develops felt arrays were a good fit. Regards, Richard Broersma Jr. ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [SQL] COPY to table with array columns (Longish)
I think two tables should suffice: ZKCOST and ZPRECMPL. So you would have ZKCOST zkcost_id, zkcost_value and ZPRECMPL zkcost_id, zprecmpl_id, zprecmpl_value where zkcost_id is the primary key for ZKCOST and zkcost_id, zprecmpl_id together are the primary key for ZPRECMPL and zkcost_id is a foreign key from ZPRECMPL to ZKCOST. That will work won't it? -Aaron On 6/12/06, Phillip Smith <[EMAIL PROTECTED]> wrote: So you're suggesting creating a child table for each SYS record? Ie, a table called "ZPRECMPL" etc? ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [SQL] COPY to table with array columns (Longish)
Not quite... ZKCOST and ZPRECMPL are two completely different things. They have no relation except they're both stored in the SYS table in D3. If we put it in a tree: SYS | +- ZKCOST | \- | +- ZPRECMPL | +- | +- | +- | \- or table: SYS +---+-+-+-+-+ | ZKCOST| | | | | | ZPRECMPL | | | | | +---+-+-+-+-+ So other than a variable-element array, the only other way would be to create a table with a column count equal to or greater than the maximum amount of values (call that value 'Y') that any sys item holds then if a particular record (eg, ZKCOST) has less values than Y, fill the rest of the columns with blanks (as above). That's what I've done at the moment, but only for 9 columns, so anything over 9 fields will be truncated past and including field 10: wbau=# \d sys Table "public.sys" Column | Type | Modifiers +--+--- a0 | text | not null a1 | text | a2 | text | a3 | text | a4 | text | a5 | text | a6 | text | a7 | text | a8 | text | a9 | text | Indexes: "id" PRIMARY KEY, btree (a0) a0 = primary key - eg, ZPRECMPL or ZKCOST -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Aaron Bono Sent: Tuesday, 13 June 2006 2:12 PM To: pgsql-sql@postgresql.org Subject: Re: [SQL] COPY to table with array columns (Longish) I think two tables should suffice: ZKCOST and ZPRECMPL. So you would have ZKCOST zkcost_id, zkcost_value and ZPRECMPL zkcost_id, zprecmpl_id, zprecmpl_value where zkcost_id is the primary key for ZKCOST and zkcost_id, zprecmpl_id together are the primary key for ZPRECMPL and zkcost_id is a foreign key from ZPRECMPL to ZKCOST. That will work won't it? -Aaron On 6/12/06, Phillip Smith <[EMAIL PROTECTED]> wrote: > So you're suggesting creating a child table for each SYS record? Ie, a table called "ZPRECMPL" etc? ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq ***Confidentiality and Privilege Notice*** The material contained in this message is privileged and confidential to the addressee. If you are not the addressee indicated in this message or responsible for delivery of the message to such person, you may not copy or deliver this message to anyone, and you should destroy it and kindly notify the sender by reply email. Information in this message that does not relate to the official business of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta. Weatherbeeta, its employees, contractors or associates shall not be liable for direct, indirect or consequential loss arising from transmission of this message or any attachments ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
