Re: [PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-18 Thread Horst Dehmer
Hey Jeff (and others)! First of all: Thanks for your detailed explanations and guide lines. On 17.01.2013, at 18:12, Jeff Janes wrote: > So the theory is that the presence of idx_4 is causing the trigger to > pick a poor plan (i.e. one using idx_4) while its absence removes that > temptation?

Re: [PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-17 Thread Jeff Janes
On Tue, Jan 15, 2013 at 3:44 PM, Horst Dehmer wrote: > idx_4 together with a simple select in the tables on-insert trigger is > slowing things down considerably. So the theory is that the presence of idx_4 is causing the trigger to pick a poor plan (i.e. one using idx_4) while its absence remov

Re: [PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-15 Thread Horst Dehmer
After more testing I have gained some insights: The foreign key constraints are NOT responsible for the low COPY FROM performance in my case. I forgot about the indexes which are created along with the FK constraints. Besides the primary key CONSTRAINT obj_item_loc_pkey PRIMARY KEY (obj_item_i

Re: [PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-12 Thread Claudio Freire
On Sat, Jan 12, 2013 at 7:41 PM, Horst Dehmer wrote: > Since the complete schema (of about 300 tables) is generated, I will just try > char(20) instead of numeric(20) in the next days to see if it makes any > difference. Which I somehow doubt. I think that might just make it worse. Well, maybe

Re: [PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-12 Thread Horst Dehmer
The types referenced by the foreign keys are the same Numeric(20). Since the complete schema (of about 300 tables) is generated, I will just try char(20) instead of numeric(20) in the next days to see if it makes any difference. Which I somehow doubt. But first I'm following the lead of the ta

Re: [PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-12 Thread Tom Lane
Jeff Janes writes: > Will PG allow you to add a FK constraint where there is no usable > index on the referenced side? It will not, because the referenced side must have a unique constraint, ie an index. The standard performance gotcha here is not having an index on the referencing side. But th

Re: [PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-12 Thread Claudio Freire
On Sat, Jan 12, 2013 at 5:16 PM, Horst Dehmer wrote: > Yes, the ids is something I don't like either. > They carry additional semantics, which I cannot make go away. > How are chances char(20) is more time efficient than numeric(20)? > Disk space is no problem here. What are the other tables like

Re: [PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-12 Thread Jeff Janes
On Fri, Jan 11, 2013 at 5:17 PM, Claudio Freire wrote: > On Fri, Jan 11, 2013 at 8:55 PM, Horst Dehmer wrote: >> Except - and that's the wall I'm hitting - for one table which yielded just >> 75 records/second. >> The main 'problem' seem to be the FK constraints. Dropping just them >> restored in

Re: [PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-12 Thread Horst Dehmer
Yes, the ids is something I don't like either. They carry additional semantics, which I cannot make go away. How are chances char(20) is more time efficient than numeric(20)? Disk space is no problem here. On 12.01.2013, at 02:17, Claudio Freire wrote: > On Fri, Jan 11, 2013 at 8:55 PM, Horst D

Re: [PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-11 Thread Claudio Freire
On Fri, Jan 11, 2013 at 8:55 PM, Horst Dehmer wrote: > Except - and that's the wall I'm hitting - for one table which yielded just > 75 records/second. > The main 'problem' seem to be the FK constraints. Dropping just them > restored insert performance for this table to 6k records/s. The table in

[PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-11 Thread Jeff Janes
On Friday, January 11, 2013, Horst Dehmer wrote: > > Except - and that's the wall I'm hitting - for one table which yielded > just 75 records/second. > The main 'problem' seem to be the FK constraints. Dropping just them > restored insert performance for this table to 6k records/s. > It sure soun

[PERFORM] Insert performance for large transaction with multiple COPY FROM

2013-01-11 Thread Horst Dehmer
Hi! I see a massive performance drop when writing a large transaction. I'm writing data for 33 tables with COPY FROM directly from streams in Scala/Java. Over all tables there are 2.2M records which are unevenly distributed from 1 record to 315k records in some tables. For comparison I ran a t