Re: [PERFORM] Fast insert, but slow join and updates for table with 4 billion rows

2016-10-25 Thread Lars Aksel Opsahl
e for sure really fun to work with Postgres/Postgis open source software hold a very high quality. Thanks. Lars Fra: pgsql-performance-ow...@postgresql.org på vegne av Tom Lane Sendt: 24. oktober 2016 14:52 Til: Lars Aksel Opsahl Kopi: pgsql-p

Re: [PERFORM] Fast insert, but slow join and updates for table with 4 billion rows

2016-10-24 Thread Lars Aksel Opsahl
ond: (point_uid_ref = 15) -> Bitmap Index Scan on idx_met_vaer_wisline_nora_bc25_observation_test (cost=0.00..466.30 rows=45430 width=0) (actual time=6.320..6.320 rows=43832 loops=1) Index Cond: (point_uid_ref = 15) Total runtime: 24.767 ms (6 rows) Thanks

Re: [PERFORM] Fast insert, but slow join and updates for table with 4 billion rows

2016-10-24 Thread Lars Aksel Opsahl
? Thanks. Lars EXPLAIN analyze SELECT count(o.*) FROM met_vaer_wisline.nora_bc25_observation o WHERE o.epoch = 128844; -[ RECORD 1

[PERFORM] Fast insert, but slow join and updates for table with 4 billion rows

2016-10-24 Thread Lars Aksel Opsahl
unt(*) from met_vaer_wisline.new_data; count --- 5 (1 row) SELECT count(*) from met_vaer_wisline.nora_bc25_observation ; count 4263866304 Thanks . Lars

Re: [PERFORM] Which Join is better

2011-08-02 Thread lars hofhansl
Unless you use the explicit join syntax: select p.* from A p join B q on (p.id = q.id) and also set  join_collapse_limit= 1 The order of the joins is determined by the planner. Also explain is your friend :) From: Adarsh Sharma To: pgsql-performance@postgresq

Re: [PERFORM] synchronous_commit off

2011-08-02 Thread lars hofhansl
No: The commit has the same guarantees as a synchronous commit w.r.t. data consistency. The commit can only fail (as a whole) due to hardware problems or postgres backend crashes. And yes: The client commit returns, but the server can fail later and not persist the transaction and it will be

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-29 Thread lars hofhansl
cache these optimizations are less important.) -- Lars - Original Message - From: Pavan Deolasee To: Merlin Moncure Cc: lars ; Kevin Grittner ; Ivan Voras ; pgsql-performance@postgresql.org Sent: Wednesday, July 27, 2011 7:15 AM Subject: Re: [PERFORM] UPDATEDs slowing SELECTs in a

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-16 Thread lars
calls. regards, tom lane If you have a patch in mind I'm happy to test it on my setup and report back. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mai

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-15 Thread lars
On 07/14/2011 04:03 PM, Jeff Janes wrote: On Wed, Jul 13, 2011 at 3:41 PM, lars wrote: On 07/13/2011 11:42 AM, Kevin Grittner wrote: So transactions without an XID *are* sensitive to synchronous_commit. That's likely a useful clue. How much did it help the run time of the SELECT

Re: [PERFORM] Statistics and Multi-Column indexes

2011-07-15 Thread lars
many tenants with relatively little data and a few with a lot of data. So the number of tenants is known ahead of time and might be 1000's. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.or

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-13 Thread lars
> set synchronous_commit = on; => update test set created_by = '001' where tenant = '001'; UPDATE 3712 Time: 384.702 ms lars=> select count(*) from test where tenant = '001' and created_date = '2011-6-30'; count ---

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-13 Thread lars
an interesting question -- what happens to the timings if your SELECTs are done with synchronous_commit = off? Just tried that... In that case the WAL is still written (as seen via iostat), but not synchronously by the transaction (as seen by strace). -- Lars -- Sent via pgsql-performan

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-13 Thread lars
.38 rows=3319 width=0) (actua l time=2.589..28.361 rows=3712 loops=1) Recheck Cond: (tenant = '001'::bpchar) Filter: (created_date = '2011-06-30'::date) Buffers: shared hit=2001 -> Bitmap Index Scan on i1 (cost=0.00..90.95 rows=3319 width=0) (actu al time=2.301..2.301 rows=17123 loops=1) Index Cond: (tenant = '001'::bpchar) Buffers: shared hit=107 Total runtime: 30.785 ms (10 rows) There seems to be definitely something funky going on. Since created_by is indexed it shouldn't do any HOT logic. Is there any other information that I can provide? I'm happy to recompile with a patch applied, etc. Thanks. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread Lars
shared_buffers is big enough to hold the entire database, and there is plenty of extra space. (verified with PG_buffercache) So i don't think that is the reason. Tom Lane schrieb: >Jeff Janes writes: >> On 7/12/11, lars wrote: >>> The fact that a select (maybe a big

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread lars
= 0x2897000 gettimeofday({1310512344, 831043}, NULL) = 0 sendto(5, "\2\0\0\0\350\0\0\0\1@\0\0\2\0\0\0\1\0\0\0\0\0\0\0\10\201\0\0?\0\2\0"..., 232, 0, NULL, 0) = 232 sendto(6, "T\0\0\0\36\0\1count\0\0\0\0\0\0\0\0\0\0\24\0\10\377\377\377\377\0\0D"..., 66, 0, NULL, 0) = 66 No writing to the WAL. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread lars
On 07/12/2011 02:51 PM, Kevin Grittner wrote: I ran x a bunch of times to get a baseline, then y once, then x a bunch more times. The results were a bit surprising: cir=> \timing Timing is on. cir=> execute x('001','2011-6-30'); count --- 3456 (1 row) Time: 9.823 ms cir=

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread lars
On 07/12/2011 01:04 PM, lars wrote: On 07/12/2011 12:08 PM, Kevin Grittner wrote: lars wrote: select count(*) from test where tenant = $1 and created_date = $2 Ah, that might be a major clue -- prepared statements. What sort of a plan do you get for that as a prepared statement? (Note, it

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread lars
On 07/12/2011 12:08 PM, Kevin Grittner wrote: lars wrote: select count(*) from test where tenant = $1 and created_date = $2 Ah, that might be a major clue -- prepared statements. What sort of a plan do you get for that as a prepared statement? (Note, it is very likely *not* to be the same

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread lars
On 07/12/2011 08:13 AM, Ivan Voras wrote: On 12/07/2011 02:09, lars wrote: Oh, and iowait hovers around 20% when SELECTs are slow: avg-cpu: %user %nice %system %iowait %steal %idle 1.54 0.00 0.98 18.49 0.07 78.92 When SELECTs are fast it looks like this: avg-cpu: %user %nice %system %iowait

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-11 Thread lars
ing) UPDATEs at all when the database resides in the cache completely. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-11 Thread lars
On 07/11/2011 04:02 PM, lars wrote: On 07/11/2011 02:43 PM, Merlin Moncure wrote: On Mon, Jul 11, 2011 at 2:16 PM, Kevin Grittner wrote: lars wrote: Stopping the UPDATEs, waiting for any CHECKPOINTs to finish, and then running the SELECTs indeed shows a similar slowdown. Interestingly I

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-11 Thread lars
On 07/11/2011 02:43 PM, Merlin Moncure wrote: On Mon, Jul 11, 2011 at 2:16 PM, Kevin Grittner wrote: lars wrote: Stopping the UPDATEs, waiting for any CHECKPOINTs to finish, and then running the SELECTs indeed shows a similar slowdown. Interestingly I see very heavy WAL traffic while

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-11 Thread lars
On 07/11/2011 10:33 AM, Kevin Grittner wrote: lars hofhansl wrote: Yep, I am not seeing the SELECTs slow down (measurably) during checkpoints (i.e. when dirty pages are flushed to disk), but only during writing of the WAL files. How about if you do a whole slew of the UPDATEs and then stop

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-10 Thread lars hofhansl
own drive, and then increased the effect I was seeing. I'll do more research and try to provide more useful details. Thanks for the pg_catalog link, I'll have a look at it. -- Lars - Original Message From: Craig Ringer To: pgsql-performance@postgresql.org Sent: Sun, July

[PERFORM] Statistics and Multi-Column indexes

2011-07-10 Thread lars
istic. Probably only setting n_distinct, i.e. set it "low" if the inner column is not selective within the context of a tenant and "high" otherwise. For various reasons #5 is also not an option. And of course the same set of questions comes up with joins. Thanks. -- L

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-10 Thread lars
y further insights, though :) Thanks. -- Lars On 07/07/2011 04:56 PM, lars wrote: I am doing some research that will hopefully lead to replacing a big Oracle installation with a set PostgreSQL servers. The current Oracle installations consists of multiple of RAC clusters with 8 RAC nodes each

[PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-10 Thread lars
ive answer, which could explain this behavior. Thanks. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] very long updates very small tables

2011-04-04 Thread Lars Feistner
On 03/30/2011 06:54 PM, Kevin Grittner wrote: Lars Feistner wrote: On 03/29/2011 09:28 PM, Kevin Grittner wrote: Lars Feistner wrote: The log tells me that certain update statements take sometimes about 3-10 minutes. But we are talking about updates on tables with 1000 to 1 rows and

Re: [PERFORM] very long updates very small tables

2011-03-30 Thread Lars Feistner
Hello Kevin, On 03/29/2011 09:28 PM, Kevin Grittner wrote: Lars Feistner wrote: The log tells me that certain update statements take sometimes about 3-10 minutes. But we are talking about updates on tables with 1000 to 1 rows and updates that are supposed to update 1 row. The top

[PERFORM] very long updates very small tables

2011-03-29 Thread Lars Feistner
ecute : UPDATE extjs_recentlist SET visible=$1 WHERE recentlist_id=$2 DETAIL: parameters: $1 = 't', $2 = '31' LOG: duration: 124654.000 ms execute : UPDATE extjs_recentlist SET visible=$1 WHERE recentlist_id=$2 DETAIL: parameters: $1 = 't', $2 = '1362

Re: [PERFORM] Migrating to Postgresql and new hardware

2011-01-31 Thread Lars
Interesting. Would have been nice if the test was with a raid-10 setup as raid-5 is not very good for writes... Would you get much of a performance increase with a write-cached ssd even if you got a raid controller with (battery-backed) cache? /Lars -Ursprungligt meddelande- Från

Re: [PERFORM] Migrating to Postgresql and new hardware

2011-01-19 Thread Lars
database servers. There are some shared data that is stored in a special "shared" database. /Lars -Ursprungligt meddelande- Från: mark [mailto:dvlh...@gmail.com] Skickat: den 19 januari 2011 05:10 Till: Lars Kopia: pgsql-performance@postgresql.org Ämne: RE: [PERFORM] Migrating to Po

Re: [PERFORM] Migrating to Postgresql and new hardware

2011-01-19 Thread Lars
provides redundancy in case of disc failure. How do you handle this with fusionIO? Two mirrored cards? /Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Migrating to Postgresql and new hardware

2011-01-19 Thread Lars
then we'd > still have to guess. It's more like 40% read 60% write. > Not only will I not compare apples to oranges, but I really wont compare > apples in Canada to oranges in Japan. :-) Hehe /Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

[PERFORM] Migrating to Postgresql and new hardware

2011-01-18 Thread Lars
ne care to guess how the two alternatives would compare in performance running Postgresql? How would the hardware usage of Postgresql compare to MySqls? Regards /Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http:

Re: [PERFORM] High update activity, PostgreSQL vs BigDBMS

2007-01-02 Thread Lars Heidieker
The performance differences between ordered and meta data only journaling should be very small enyway - -- Viele Grüße, Lars Heidieker [EMAIL PROTECTED] http://paradoxon.info - Mystische Erklärungen. Die mystischen Erklärungen gelten für tief; die Wahrheit ist