Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Kenneth Marshall
On Fri, Jun 09, 2017 at 03:22:35PM +, Frits Jalvingh wrote: > Hi Babu, > > That was all already done, as it is common practice for JDBC. Your > parameter was added to the code that already did all that - and worked > brilliantly there ;) > Hi Frits, What was the parameter? I did not see an E

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Kenneth Marshall
On Fri, Jun 09, 2017 at 02:39:37PM +, Frits Jalvingh wrote: > Hi all, > > Thanks a lot for the many responses! > > About preparing statements: this is done properly in Java, and pgsql does > it by itself. So that cannot be done better ;) > > I tried the copy command, and that indeed works qu

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Kenneth Marshall
On Fri, Jun 09, 2017 at 01:56:58PM +, Frits Jalvingh wrote: > Hi Kenneth, Andreas, > > Thanks for your tips! > > I increased shared_buffers to 8GB but it has no measurable effect at all. I > think that is logical: shared buffers are important for querying but not > for inserting; for that the

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Kenneth Marshall
On Fri, Jun 09, 2017 at 03:24:15PM +0200, Andreas Kretschmer wrote: > > > Am 09.06.2017 um 15:04 schrieb Frits Jalvingh: > >Hi all, > > > >I am trying to improve the runtime of a big data warehouse > >application. One significant bottleneck found was insert > >performance, so I am investigating w

Re: [PERFORM] REINDEX takes half a day (and still not complete!)

2011-04-30 Thread Kenneth Marshall
On Sat, Apr 30, 2011 at 05:26:36PM +0800, Phoenix Kiula wrote: > On Sat, Apr 30, 2011 at 4:07 PM, Greg Smith wrote: > > On 04/23/2011 03:44 PM, Robert Haas wrote: > >> > >> On Apr 17, 2011, at 11:30 AM, Phoenix Kiula > >> ?wrote: > >> > >>> > >>> Postgres is 8.2.9. > >>> > >>> > >> > >> An upgrade

Re: [PERFORM] Time to put theory to the test?

2011-04-26 Thread Kenneth Marshall
On Tue, Apr 26, 2011 at 09:58:49AM -0500, Kevin Grittner wrote: > J Sisson wrote: > > Rob Wultsch wrote: > >> Tip from someone that manages thousands of MySQL servers: Use > >> InnoDB when using MySQL. > > > > Granted, my knowledge of PostgreSQL (and even MSSQL) far surpasses > > my knowledge of

Re: [PERFORM] Bad Query Plan with Range Query

2011-04-15 Thread Kenneth Marshall
On Fri, Apr 15, 2011 at 10:17:32AM -0700, Mark Williams wrote: > We are experiencing a problem with our query plans when using a range query > in Postgresql 8.3. The query we are executing attempts to select the > minimum primary key id after a certain date. Our date columns are bigint's > holdi

Re: [PERFORM] Partial index slower than regular index

2011-04-05 Thread Kenneth Marshall
On Tue, Apr 05, 2011 at 11:35:29PM +0100, Thom Brown wrote: > I'm using 9.1dev. > > Could someone explain the following behaviour? > > -- create a test table > CREATE TABLE indextest (id serial, stuff text); > > -- insert loads of values with intermittent sets of less common values > INSERT INTO

Re: [PERFORM] Updating histogram_bounds after a delete

2011-03-17 Thread Kenneth Marshall
On Thu, Mar 17, 2011 at 09:49:45AM -0500, Kevin Grittner wrote: > Derrick Rice wrote: > > Kevin Grittner > > >> there is a feature to probe the end of an index's range in > >> situations where data skew was often causing less than optimal > >> plans to be chosen. > > > > Was this introduced in

Re: [PERFORM] Help with Query Tuning

2011-03-16 Thread Kenneth Marshall
On Wed, Mar 16, 2011 at 02:43:38PM +0530, Adarsh Sharma wrote: > Dear all, > > I am facing a problem while creating the index to make the below query run > faster. My table size is near about 1065 MB and 428467 rows. > > explain analyze select count(*) from page_content where publishing_date

Re: [PERFORM] ANTI-JOIN needs table, index scan not possible?

2011-03-11 Thread Kenneth Marshall
On Fri, Mar 11, 2011 at 06:54:39PM +0100, hans wulf wrote: > Thanks for the answer. > > so there's no way around this problem? A nice index bitmap merge thing would > be super fast. Big table ANTI JOIN queries with only a few results expected, > are totally broken, if this is true. > > This wa

Re: [PERFORM] Performance issues

2011-03-07 Thread Kenneth Marshall
On Mon, Mar 07, 2011 at 03:17:05PM +0100, Andreas For? Tollefsen wrote: > Thanks, Ken. > > It seems like the tip to turn off synchronous_commit did the trick: > > /usr/lib/postgresql/8.4/bin/pgbench -T 60 test1 > starting vacuum...end. > transaction type: TPC-B (sort of) > scaling factor: 1 > que

Re: [PERFORM] Performance issues

2011-03-07 Thread Kenneth Marshall
On Mon, Mar 07, 2011 at 02:45:03PM +0100, Andreas For? Tollefsen wrote: > Hi, > > I am running Postgresql 8.4.7 with Postgis 2.0 (for raster support). > Server is mainly 1 user for spatial data processing. This involves queries > that can take hours. > > This is running on a ubuntu 10.10 Server w

Re: [PERFORM] Really really slow select count(*)

2011-02-04 Thread Kenneth Marshall
On Fri, Feb 04, 2011 at 05:20:27PM +0100, felix wrote: > reply was meant for the list > > -- Forwarded message -- > From: felix > Date: Fri, Feb 4, 2011 at 4:39 PM > Subject: Re: [PERFORM] Really really slow select count(*) > To: Greg Smith > > > > > On Fri, Feb 4, 2011 at 3:

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Kenneth Marshall
On Thu, Feb 03, 2011 at 09:05:52PM -0500, Robert Haas wrote: > On Thu, Feb 3, 2011 at 8:37 PM, wrote: > > On Thu, 3 Feb 2011, Robert Haas wrote: > > > >> On Thu, Feb 3, 2011 at 7:39 PM, ? wrote: > > Yeah, but you'll be passing the entire table through this separate > process that m

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Kenneth Marshall
On Thu, Feb 03, 2011 at 04:39:12PM -0800, da...@lang.hm wrote: > On Thu, 3 Feb 2011, Robert Haas wrote: > >> On Thu, Feb 3, 2011 at 3:54 PM, wrote: >>> with the current code, this is a completely separate process that knows >>> nothing about the load, so if you kick it off when you start the load

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kenneth Marshall
On Thu, Feb 03, 2011 at 02:11:58AM -0800, da...@lang.hm wrote: > On Thu, 3 Feb 2011, Vitalii Tymchyshyn wrote: > >> 02.02.11 20:32, Robert Haas ???(??): >>> Yeah. Any kind of bulk load into an empty table can be a problem, >>> even if it's not temporary. When you load a bunch of data and then

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-02 Thread Kenneth Marshall
On Wed, Feb 02, 2011 at 03:54:26PM -0500, Mladen Gogala wrote: > Greg Smith wrote: >> Given that even Oracle kicked out the RBO a long time ago, I'm not so sure >> longing for those good old days will go very far. I regularly see queries >> that were tweaked to always use an index run at 1/10 or

Re: [PERFORM] Server Configuration

2011-02-02 Thread Kenneth Marshall
On Wed, Feb 02, 2011 at 03:15:22PM -0300, Cesar Arrieta wrote: > Hi, I have a Server with Fedora Core 11, Tomcat and Postgresql 8.3. > With Hardware: > * 8GB RAM > * 8 processors Intel Xeon E5520 @2.27GHz > * 250GB SATA DISK > > Actually, it serves at most 250 connections. > The problem happends w

Re: [PERFORM] postgres 9 query performance

2011-01-28 Thread Kenneth Marshall
On Fri, Jan 28, 2011 at 09:30:19AM -0800, yazan suleiman wrote: > I am evaluating postgres 9 to migrate away from Oracle. The following query > runs too slow, also please find the explain plan: > > > explain analyze select DISTINCT

Re: [PERFORM] Postgres 9.0 has a bias against indexes

2011-01-27 Thread Kenneth Marshall
On Thu, Jan 27, 2011 at 04:12:53PM -0500, Igor Neyman wrote: > > > > -Original Message- > > From: Scott Marlowe [mailto:scott.marl...@gmail.com] > > Sent: Thursday, January 27, 2011 3:59 PM > > To: Mladen Gogala > > Cc: Igor Neyman; Tom Lane; David

Re: [PERFORM] Postgres 9.0 has a bias against indexes

2011-01-27 Thread Kenneth Marshall
On Thu, Jan 27, 2011 at 10:41:08AM -0500, Mladen Gogala wrote: > I have a table EMP, with 14 rows and a description like this: > scott=> \d+ emp > Table "public.emp" > Column |Type | Modifiers | Storage | > Description > --+-

Re: [PERFORM] Slow query + why bitmap index scan??

2011-01-12 Thread Kenneth Marshall
On Wed, Jan 12, 2011 at 03:21:45PM +0100, Laszlo Nagy wrote: > On 2011-01-12 14:42, Florian Weimer wrote: >> * Laszlo Nagy: >> >>> This query: >>> >>> select hid from product_price_history where id=35547581 >>> >>> Returns 759 rows in 8837 msec! How can this be that slow??? >> If most records are o

Re: [PERFORM] PostgreSQL 9.0 x64 bit pgbench TPC very low question?

2010-12-23 Thread Kenneth Marshall
On Thu, Dec 23, 2010 at 09:20:59PM +0700, tuanhoanganh wrote: > Could you show me what parameter of pgbouncer.ini can do that. I read > pgbouncer and can not make pgbouncer open and keep 200 connect to postgres > (Sorry for my English) > > Thanks you very much. > > Tuan Hoang ANh > You need to

Re: [PERFORM] CPU bound

2010-12-20 Thread Kenneth Marshall
On Mon, Dec 20, 2010 at 10:33:26AM -0500, James Cloos wrote: > > "MG" == Mladen Gogala writes: > > MG> Good time accounting is the most compelling reason for having a wait > MG> event interface, like Oracle. Without the wait event interface, one > MG> cannot really tell where the time is spen

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Kenneth Marshall
On Tue, Dec 07, 2010 at 11:56:51AM -0800, Richard Broersma wrote: > On Tue, Dec 7, 2010 at 11:43 AM, Andy Colson wrote: > > > In PG the first statement you fire off (like an "insert into" for example) > > will start a transaction. ?If you dont commit before you disconnect that > > transaction wil

Re: [PERFORM] Update problem on large table

2010-12-06 Thread Kenneth Marshall
On Mon, Dec 06, 2010 at 03:24:31PM -0500, Josh Kupershmidt wrote: > On Mon, Dec 6, 2010 at 2:48 PM, Jon Nelson wrote: > > On Mon, Dec 6, 2010 at 1:46 PM, bricklen wrote: > >> Not sure if anyone replied about killing your query, but you can do it > >> like so: > >> > >> select pg_cancel_backend(5

Re: [PERFORM] SELECT INTO large FKyed table is slow

2010-12-01 Thread Kenneth Marshall
On Wed, Dec 01, 2010 at 12:15:19PM -0500, Mladen Gogala wrote: > Mario Splivalo wrote: >> I'll try what Pierre suggested, on whole new filesystem. This one did get >> quite filled with thousands of files that I deleted while the database was >> working. >> >> Mario >> > > Yes, that is a g

Re: [PERFORM] MVCC performance issue

2010-11-12 Thread Kenneth Marshall
Ah, this is a very old version. If you can take advantage of a version with HOT support, you should be much, much happier. Cheers, Ken On Fri, Nov 12, 2010 at 06:14:00PM +0200, Kyriacos Kyriacou wrote: > > > We are still using PostgreSQL 8.2.4. We are running a 24x7 system and > database size

Re: [PERFORM] MVCC performance issue

2010-11-12 Thread Kenneth Marshall
On Fri, Nov 12, 2010 at 07:34:36AM -0800, bricklen wrote: > On Fri, Nov 12, 2010 at 5:52 AM, Kenneth Marshall wrote: > > > > I cannot speak to your suggestion, but it sounds like you are not > > vacuuming enough and a lot of the bloat/randomization would be helped > > b

Re: [PERFORM] MVCC performance issue

2010-11-12 Thread Kenneth Marshall
On Fri, Nov 12, 2010 at 03:47:30PM +0200, Kyriacos Kyriacou wrote: > This is my first post in this mailing list and I would like to raise an > issue that in my opinion is causing performance issues of PostgreSQL > especially in a transaction processing environment. In my company we are > using Post

Re: [PERFORM] anti-join chosen even when slower than old plan

2010-11-11 Thread Kenneth Marshall
On Thu, Nov 11, 2010 at 03:56:25PM -0500, gnuo...@rcn.com wrote: > On a thread some time ago, on a similar subject, I opined that I missed the > ability to assign tables to tablespaces and buffers to tablespaces, thus > having the ability to isolate needed tables (perhaps a One True Lookup Table,

Re: [PERFORM] anti-join chosen even when slower than old plan

2010-11-11 Thread Kenneth Marshall
On Thu, Nov 11, 2010 at 09:15:58AM -0500, Mladen Gogala wrote: > Kenneth Marshall wrote: >> I agree with the goal of avoiding the need for a GUC. This needs to >> be as automatic as possible. One idea I had had was computing a value >> for the amount of cache data in the syste

Re: [PERFORM] CREATE INDEX as bottleneck

2010-11-11 Thread Kenneth Marshall
On Thu, Nov 11, 2010 at 02:41:12PM +0100, Marc Mamin wrote: > Hello, > > in the last years, we have successfully manage to cope with our data > growth > using partitioning and splitting large aggregation tasks on multiple > threads. > The partitioning is done logically by our applicationn server,

Re: [PERFORM] anti-join chosen even when slower than old plan

2010-11-11 Thread Kenneth Marshall
On Wed, Nov 10, 2010 at 10:47:21PM -0500, Robert Haas wrote: > On Wed, Nov 10, 2010 at 6:07 PM, Tom Lane wrote: > > "Kevin Grittner" writes: > >> Robert Haas wrote: > >>> Unfortunately, to know how much data we're going to grovel > >>> through, we need to know the plan; and to decide on the righ

Re: [PERFORM] Bufer cache replacement LRU algorithm?

2010-11-03 Thread Kenneth Marshall
Mladen, You would need to check the mailing lists. The release notes have it as being a clock sweep algorithm starting in version 8. Then additional changes were added to eliminate the cache blowout caused by a sequential scan and by vacuum/autovacuum. I do not believe that there are any parameter

Re: [PERFORM] Select count(*), the sequel

2010-10-28 Thread Kenneth Marshall
On Wed, Oct 27, 2010 at 05:49:42PM -0400, Tom Lane wrote: > Kenneth Marshall writes: > > Just keeping the hope alive for faster compression. > > Is there any evidence that that's something we should worry about? > I can't recall ever having seen a code profile that

Re: [PERFORM] Select count(*), the sequel

2010-10-27 Thread Kenneth Marshall
On Wed, Oct 27, 2010 at 09:52:49PM +0200, Pierre C wrote: >> Even if somebody had a >> great idea that would make things smaller without any other penalty, >> which I'm not sure I believe either. > > I'd say that the only things likely to bring an improvement significant > enough to warrant the (q

Re: [PERFORM] Periodically slow inserts

2010-10-21 Thread Kenneth Marshall
Hi, There are a lot of details missing about your system: http://wiki.postgresql.org/wiki/SlowQueryQuestions Cheers, Ken On Thu, Oct 21, 2010 at 02:25:44PM +0200, Gael Le Mignot wrote: > > Hello, > > We are using PostgreSQL for storing data and full-text search indexes > for the webiste of

Re: [PERFORM] Select count(*), the sequel

2010-10-16 Thread Kenneth Marshall
Hi, Interesting data points. The amount of rows that you managed to insert into PostgreSQL before Oracle gave up the ghost is 95% of the rows in the Oracle version of the database. To count 5% fewer rows, it took PostgreSQL 24 seconds longer. Or adjusting for the missing rows, 52 seconds longer fo

Re: [PERFORM] Query much faster with enable_seqscan=0

2010-09-21 Thread Kenneth Marshall
You DB is more than likely cached. You should adjust your page costs to better reflect reality and then the planner can make more accurate estimates and then choose the proper plan. Cheers, Ken On Tue, Sep 21, 2010 at 12:32:01PM -0500, Ogden wrote: > Hello, > > I have received some help from the

Re: [PERFORM] now() gives same time within the session

2010-07-12 Thread Kenneth Marshall
On Mon, Jul 12, 2010 at 06:11:31AM -0700, Rob Wultsch wrote: > On Mon, Jul 12, 2010 at 4:15 AM, A. Kretschmer > wrote: > > In response to atul.g...@globaldatapoint.com : > >> Hi, > >> > >> > >> > >> I need to log the start and end time of the procedures in a table. But the > >> start and end time

Re: [PERFORM] Highly Efficient Custom Sorting

2010-07-07 Thread Kenneth Marshall
Hi Eliot, Would you mind posting your code for reference. It is nice to have working examples when trying to figure out how it all fits together. Regards, Ken On Wed, Jul 07, 2010 at 03:23:12PM -0400, Eliot Gable wrote: > Thanks again for all the input and suggestions from people. I have this >

Re: [PERFORM] Write performance

2010-06-24 Thread Kenneth Marshall
On Thu, Jun 24, 2010 at 02:43:33PM +0200, Janning wrote: > Hi, > > at the moment we encounter some performance problems with our database server. > > We have a 12 GB RAM machine with intel i7-975 and using > 3 disks "Seagate Barracuda 7200.11, ST31500341AS (1.5 GB)" > One disk for the system and

Re: [PERFORM] requested shared memory size overflows size_t

2010-06-18 Thread Kenneth Marshall
On Fri, Jun 18, 2010 at 12:46:11AM +0100, Tom Wilcox wrote: > On 17/06/2010 22:41, Greg Smith wrote: >> Tom Wilcox wrote: >>> Any suggestions for good monitoring software for linux? >> >> By monitoring, do you mean for alerting purposes or for graphing purposes? >> Nagios is the only reasonable c

Re: [PERFORM] slow query performance

2010-06-11 Thread Kenneth Marshall
lue for random_page_cost > via some testing with OS commands. We have several postgres databases > and determining this value on a case by case basis may not be viable > (we may have to go with the defaults) > > On Fri, Jun 11, 2010 at 5:44 AM, Kenneth Marshall wrote: > > Hi

Re: [PERFORM] slow query performance

2010-06-11 Thread Kenneth Marshall
Hi Anj, That is an indication that your system was less correctly modeled with a random_page_cost=2 which means that the system will assume that random I/O is cheaper than it is and will choose plans based on that model. If this is not the case, the plan chosen will almost certainly be slower for

Re: [PERFORM] Need to increase performance of a query

2010-06-10 Thread Kenneth Marshall
On Thu, Jun 10, 2010 at 12:34:07PM -0700, Anne Rosset wrote: > Jochen Erwied wrote: >> Thursday, June 10, 2010, 8:36:08 PM you wrote: >> >> >>> psrdb=# (SELECT >>> psrdb(#MAX(item_rank.rank) AS maxRank >>> psrdb(# FROM >>> psrdb(#item_rank item_rank >>> psrdb(# WHERE >>> psrdb(#

Re: [PERFORM] debugging handle exhaustion and 15 min/ 5mil row delete

2010-05-07 Thread Kenneth Marshall
On Fri, May 07, 2010 at 09:37:42AM -0400, Mark Stosberg wrote: > > Hello, > > We've been a satified user of PostgreSQL for several years, and use it > to power a national pet adoption website: http://www.adoptapet.com/ > > Recently we've had a regularly-timed middle-of-the-night problem where >

Re: [PERFORM] autovacuum strategy / parameters

2010-04-28 Thread Kenneth Marshall
Check out the manual: http://www.postgresql.org/docs/8.4/static/routine-vacuuming.html#AUTOVACUUM Cheers, Ken On Wed, Apr 28, 2010 at 10:37:35AM -0400, akp geek wrote: > Hi - >don't want to side track the discussion. We have 8.4, which of > AUTOVACUUM PARAMETERS can be set to handle

Re: [PERFORM] GiST index performance

2010-03-19 Thread Kenneth Marshall
Hi Yeb, I have not looked at the gist code, but would it be possible to make virtual pages that have a size that is 1/power-of-2 * blocksize. Then the leaf node could be 1/8 or even 1/16 the size of the full pagesize. Regards, Ken On Fri, Mar 19, 2010 at 09:49:30PM +0100, Yeb Havinga wrote: > Ye

Re: [PERFORM] mysql to postgresql, performance questions

2010-03-18 Thread Kenneth Marshall
If you expect this DB to be memory resident, you should update the cpu/disk cost parameters in postgresql.conf. There was a post earlier today with some more reasonable starting values. Certainly your test DB will be memory resident. Ken On Thu, Mar 18, 2010 at 03:31:18PM +0100, Corin wrote: > Hi

Re: [PERFORM] Bad query plan inside EXISTS clause

2010-03-10 Thread Kenneth Marshall
EXISTS matches NULLs too and since they are not indexed a sequential scan is needed to check for them. Try using IN instead. Cheers, Ken On Wed, Mar 10, 2010 at 02:26:20PM +0100, Benoit Delbosc wrote: > Hi all, > > I am trying to understand why inside an EXISTS clause the query planner > does n

Re: [PERFORM] prepared statements and partitioning (partition elimination not working)

2010-03-08 Thread Kenneth Marshall
On Mon, Mar 08, 2010 at 10:24:56AM -0700, Kevin Kempter wrote: > Hi all; > > we've found that partition elimination is not happening for a prepared > statement, however running the same statement in psql manually does give us > partition elimination. > > Is this a known issue? > Yes, see the

Re: [PERFORM] partitioned tables query not using indexes

2010-02-28 Thread Kenneth Marshall
On Sun, Feb 28, 2010 at 12:29:14PM -0800, Josh Berkus wrote: > > > However the same query against the base table when specifying the check > > constraint key in the where clause produces sequential scans: > > Does the "master" table have the same indexes as the slave partitions? > > --Josh Berk

Re: [PERFORM] index usage in not like

2010-02-18 Thread Kenneth Marshall
On Thu, Feb 18, 2010 at 01:18:10PM +0100, A. Kretschmer wrote: > In response to Thom Brown : > > On 18 February 2010 11:55, AI Rumman wrote: > > > "Not like" operation does not use index. > > > > > > select * from vtiger_contactscf where lower(cf_1253) not like > > > lower('Former%') > > > > > > I

Re: [PERFORM] Bad plan choice nestloop vs. hashjoin

2010-01-18 Thread Kenneth Marshall
On Mon, Jan 18, 2010 at 12:13:24PM -0500, Tom Lane wrote: > Kenneth Marshall writes: > > We have just upgraded our monitoring server software and > > now the following query for graphing the data performs > > abysmally with the default settings. Here is the results > >

Re: [PERFORM] Bad plan choice nestloop vs. hashjoin

2010-01-15 Thread Kenneth Marshall
On Fri, Jan 15, 2010 at 04:58:57PM -0600, Kevin Grittner wrote: > Kenneth Marshall wrote: > > > with the default settings > > Do you mean you haven't changed any settings in your postgresql.conf > file from their defaults? > > -Kevin > Sorry, here a

[PERFORM] Bad plan choice nestloop vs. hashjoin

2010-01-15 Thread Kenneth Marshall
Dear performance group: We have just upgraded our monitoring server software and now the following query for graphing the data performs abysmally with the default settings. Here is the results of the EXPLAIN ANALYZE run with nestloops enabled: SET enable_nestloop = 'on'; EXPLAIN SELECT g.graphid

Re: [PERFORM] PG optimization question

2010-01-09 Thread Kenneth Marshall
On Sat, Jan 09, 2010 at 03:42:08PM +0300, Nickolay wrote: > I do not see any way to normalize this table anymore. it's size is 4Gig for > ~4M rows, i.e. 1Kb per row, i think it's ok. > Also there are 2 indexes: by date_time and by a couple of service fields > (total index size is 250Mb now). > I

Re: [PERFORM] Massive table (500M rows) update nightmare

2010-01-08 Thread Kenneth Marshall
On Fri, Jan 08, 2010 at 12:38:46PM -0500, Carlo Stonebanks wrote: >> I thought that post mentioned that the plan >> was one statement in an iteration, and that the cache would have >> been primed by a previous query checking whether there were any rows >> to update. If that was the case, it might

Re: [PERFORM] big select is resulting in a large amount of disk writing by kjournald

2009-12-09 Thread Kenneth Marshall
Hint bit I/O? Ken On Wed, Dec 09, 2009 at 01:29:00PM -0500, Joseph S wrote: > I just installed a shiny new database server with pg 8.4.1 running on > CentOS 5.4. After using slony to replicate over my database I decided to > do some basic performance tests to see how spiffy my shiny new server

Re: [PERFORM] Unexpected sequential scan on an indexed column

2009-11-16 Thread Kenneth Marshall
On Mon, Nov 16, 2009 at 12:45:46PM -0800, Eddy Escardo-Raffo wrote: > Yeah this kind of thing would probably work. Doing this in java with > separate queries would be easy to code but require multiple round trips. > Doing it as a stored procedure would be nicer but I'd have to think a little > more

Re: [PERFORM] FTS performance with the Polish config

2009-11-14 Thread Kenneth Marshall
On Sat, Nov 14, 2009 at 12:25:05PM +0100, Wojciech Knapik wrote: > > Hello > > I just finished implementing a "search engine" for my site and found > ts_headline extremely slow when used with a Polish tsearch configuration, > while fast with English. All of it boils down to a simple testcase, bu

Re: [PERFORM] limiting performance impact of wal archiving.

2009-11-10 Thread Kenneth Marshall
On Tue, Nov 10, 2009 at 12:55:42PM +0100, Laurent Laborde wrote: > Hi ! > We recently had a problem with wal archiving badly impacting the > performance of our postgresql master. > And i discovered "cstream", that can limite the bandwidth of pipe stream. > > Here is our new archive command, FYI, t

Re: [PERFORM] optimizing query with multiple aggregates

2009-10-22 Thread Kenneth Marshall
On Wed, Oct 21, 2009 at 03:51:25PM -0700, Doug Cole wrote: > I have a reporting query that is taking nearly all of it's time in aggregate > functions and I'm trying to figure out how to optimize it. The query takes > approximately 170ms when run with "select *", but when run with all the > aggrega

Re: [PERFORM] Are folks running 8.4 in production environments? and 8.4 and slon 1.2?

2009-10-13 Thread Kenneth Marshall
On Tue, Oct 13, 2009 at 01:03:10AM -0600, Scott Marlowe wrote: > On Mon, Oct 12, 2009 at 1:06 PM, Tory M Blue wrote: > > Any issues, has it baked long enough, is it time for us 8.3 folks to deal > > with the pain and upgrade? > > I am running 8.4.1 for my stats and search databases, and it's work

Re: [PERFORM] Bad performance of SELECT ... where id IN (...)

2009-10-09 Thread Kenneth Marshall
On Fri, Oct 09, 2009 at 08:31:54PM +0800, Xia Qingran wrote: > On Mon, Oct 5, 2009 at 9:58 AM, Omar Kilani wrote: > > Hi Xia, > > > > Try this patch: > > > > http://treehou.se/~omar/postgresql-8.4.1-array_sel_hack.patch > > > > It's a hack, but it works for us. I think you're probably spending > >

Re: [PERFORM] partition queries hitting all partitions even though check key is specified

2009-09-02 Thread Kenneth Marshall
The planner does not yet work as efficiently as it could with child tables. Check the recent mail archives for a long discussion of the same. Regards, Ken On Wed, Sep 02, 2009 at 08:52:30AM -0600, Kevin Kempter wrote: > Hi all; > > I cant figure out why we're scanning all of our partitions. > >

Re: [PERFORM] Best settings to load a fresh database

2009-08-06 Thread Kenneth Marshall
On Thu, Aug 06, 2009 at 01:42:06PM -0500, Campbell, Lance wrote: > PostgreSQL 8.3 > Linux RedHat 4.X > 24G of memory > > When loading a file generated from pg_dumpall is there a key setting in > the configuration file that would allow the load to work faster. > > Thanks, > > Lance Campbell > Pro

Re: [PERFORM] cluster index on a table

2009-06-24 Thread Kenneth Marshall
Clustering reorganizes the layout of a table according to the ordering of a SINGLE index. This will place items that are adjacent in the index adjacent in the heap. So you need to cluster on the index that will help the locality of reference for the queries which will benefit you the most. Executio

Re: [PERFORM] 8.4 COPY performance regression on Solaris

2009-06-19 Thread Kenneth Marshall
Hi, Looking at the XLogInsert() from 8.3 and 8.4, the 8.4 version includes a call to RecoveryInProgress() at the top as well as a call to TRACE_POSTGRESQL_XLOG_INSERT(). Could either of those have caused a context switch or cache flush resulting in worse performance. Cheers, Ken -- Sent via pgs

Re: [PERFORM] Strange performance response for high load times

2009-06-18 Thread Kenneth Marshall
alue can be allocated in an individual query. You can set it on a per query basis to help manage it use, i.e. up it for only the query that needs it. With our systems, which run smaller number of queries we do use 256MB. I hope that this helps. Regards, Ken > On Thu, Jun 18, 2009 at 8:30 PM,

Re: [PERFORM] Strange performance response for high load times

2009-06-18 Thread Kenneth Marshall
On Thu, Jun 18, 2009 at 08:27:02PM +0200, Peter Alban wrote: > Hi All, > > We are having a reasonably powerful machine for supporting about 20 > databases but in total they're not more then 4GB in size. > > The machine is 2 processor 8 core and 8 Gig or ram so I would expect that PG > should cach

Re: [PERFORM] Storing sensor data

2009-05-28 Thread Kenneth Marshall
On Thu, May 28, 2009 at 05:24:33PM +0200, Ivan Voras wrote: > 2009/5/28 Kenneth Marshall : > > > > > One big benefit of partitioning is that you can prune old data with > > minimal impact to the running system. Doing a large bulk delete would > > be extremely I/O imp

Re: [PERFORM] Storing sensor data

2009-05-28 Thread Kenneth Marshall
On Thu, May 28, 2009 at 04:55:34PM +0200, Ivan Voras wrote: > 2009/5/28 Heikki Linnakangas : > > Ivan Voras wrote: > >> > >> I need to store data about sensor readings. There is a known (but > >> configurable) number of sensors which can send update data at any time. > >> The "current" state needs

Re: [PERFORM] Any better plan for this query?..

2009-05-06 Thread Kenneth Marshall
tegrated profiled within a code? or do I > need external tools?.. > > Rgds, > -Dimitri I only suggested it because it might have the effect of changing the sequential scan on the stat table to an indexed scan. Cheers, Ken > > On 5/6/09, Kenneth Marshall wrote: > > On We

Re: [PERFORM] Any better plan for this query?..

2009-05-06 Thread Kenneth Marshall
On Wed, May 06, 2009 at 09:38:59AM +0200, Dimitri wrote: > Hi, > > any idea if there is a more optimal execution plan possible for this query: > > select S.REF as stref, S.NAME as stnm, H.HORDER as hord, H.BEGIN_DATE as hbeg, > H.END_DATE as hend, H.NOTE as hnote > from HISTORY H

Re: [PERFORM] Any better plan for this query?..

2009-05-06 Thread Kenneth Marshall
No. Ken On Wed, May 06, 2009 at 10:04:33PM +0800, Craig Ringer wrote: > Dimitri wrote: > > Hi Chris, > > > > the only problem I see here is it's 2 times slower vs InnoDB, so > > before I'll say myself it's ok I want to be sure there is nothing else > > to do.. :-) > > Can the genetic query optim

Re: [PERFORM] Any better plan for this query?..

2009-05-06 Thread Kenneth Marshall
On Wed, May 06, 2009 at 02:49:23PM +0200, Dimitri wrote: > The story is simple: for the launching of MySQL 5.4 I've done a > testing comparing available on that time variations of InnoDB engines, > and at the end by curiosity started the same test with PostgreSQL > 8.3.7 to see if MySQL performance

Re: [PERFORM] Any better plan for this query?..

2009-05-06 Thread Kenneth Marshall
On Wed, May 06, 2009 at 04:01:03PM +0800, Craig Ringer wrote: > Dimitri wrote: >> Hi, >> any idea if there is a more optimal execution plan possible for this >> query: >> select S.REF as stref, S.NAME as stnm, H.HORDER as hord, H.BEGIN_DATE as >> hbeg, >> H.END_DATE as hend, H.NOTE as hno

Re: [PERFORM] partition question for new server setup

2009-04-28 Thread Kenneth Marshall
On Tue, Apr 28, 2009 at 01:30:59PM -0500, Kevin Grittner wrote: > Craig James wrote: > > > After a reading various articles, I thought that "noop" was the > > right choice when you're using a battery-backed RAID controller. > > The RAID controller is going to cache all data and reschedule the

Re: [PERFORM] partition question for new server setup

2009-04-28 Thread Kenneth Marshall
On Tue, Apr 28, 2009 at 11:56:25AM -0600, Scott Marlowe wrote: > On Tue, Apr 28, 2009 at 11:48 AM, Whit Armstrong > wrote: > > Thanks, Scott. > > > > Just to clarify you said: > > > >> postgres. ?So, my pg_xlog and all OS and logging stuff goes on the > >> RAID-10 and the main store for the db goe

Re: [PERFORM] performance for high-volume log insertion

2009-04-21 Thread Kenneth Marshall
On Tue, Apr 21, 2009 at 11:09:18AM -0700, da...@lang.hm wrote: > On Tue, 21 Apr 2009, Greg Smith wrote: > >> On Mon, 20 Apr 2009, da...@lang.hm wrote: >> >>> while I fully understand the 'benchmark your situation' need, this isn't >>> that simple. in this case we are trying to decide what API/int

Re: [PERFORM] performance for high-volume log insertion

2009-04-21 Thread Kenneth Marshall
ts. Cheers, Ken > On Tue, 21 Apr 2009, Kenneth Marshall wrote: > >> Date: Tue, 21 Apr 2009 08:33:30 -0500 >> From: Kenneth Marshall >> To: Richard Huxton >> Cc: da...@lang.hm, Stephen Frost , >> Greg Smith , pgsql-performance@postgresql.org >> Sub

Re: [PERFORM] performance for high-volume log insertion

2009-04-21 Thread Kenneth Marshall
Hi, I just finished reading this thread. We are currently working on setting up a central log system using rsyslog and PostgreSQL. It works well once we patched the memory leak. We also looked at what could be done to improve the efficiency of the DB interface. On the rsyslog side, moving to prepa

Re: [PERFORM] postgreSQL performance 8.2.6 vs 8.3.3

2009-02-20 Thread Kenneth Marshall
On Fri, Feb 20, 2009 at 04:34:23PM -0500, Battle Mage wrote: > I have a server box that has 4GB of RAM, Quad core CPU AMD Opteron 200.152 > Mhz (1024 KB cache size each) with plenty of hard drive space. > > I installed both postgresql 8.2.6 and 8.3.3 on it. I've created a basic > test db and used

Re: [HACKERS] [PERFORM] GIST versus GIN indexes for intarrays

2009-02-13 Thread Kenneth Marshall
On Fri, Feb 13, 2009 at 04:12:53PM +0300, Teodor Sigaev wrote: >> The short-term workaround for Rusty is probably to create his GIN >> index using the intarray-provided gin__int_ops opclass. But it > Right >> seems to me that we ought to get rid of intarray's @> and <@ operators >> and have the mo

Re: [PERFORM] strange index performance?

2009-01-26 Thread Kenneth Marshall
On Mon, Jan 26, 2009 at 03:49:00PM +0100, Thomas Finneid wrote: > Kenneth Marshall wrote: >> It may be that the smaller index has update contention for the same >> blocks that the larger index does not. > > Is that an assumption based on both indexes existing? if so I might

Re: [PERFORM] strange index performance?

2009-01-26 Thread Kenneth Marshall
On Mon, Jan 26, 2009 at 10:10:13AM +0100, Thomas Finneid wrote: > Scott Marlowe wrote: > >> I'm guessing that you just had more data in the table or something by >> the time you tested that, or some cron job was running in the >> background, or some other issue, not the index. > > It starts from sc

Re: [PERFORM] caching indexes and pages?

2009-01-22 Thread Kenneth Marshall
On Thu, Jan 22, 2009 at 10:58:25PM +0100, Thomas Finneid wrote: > Thomas Markus wrote: > >> try to reorganize your data with CLUSTER and create appropriate indixes >> (dont forget to check statistics). > > One question. Assume I have clustered and new data has been added after > that, according t

Re: [PERFORM] Question about clustering indexes and restores

2009-01-22 Thread Kenneth Marshall
On Thu, Jan 22, 2009 at 02:52:12PM -0500, Harold A. Gim?nez Ch. wrote: > Hi list, > > Clustering my indexes dramatically improves the query performance of many of > my queries. Also, the actual clustering takes a very long time for big > databases, roughly 20 hours. I have two questions about how

Re: [PERFORM] Need help with 8.4 Performance Testing

2008-12-07 Thread Kenneth Marshall
Josh, Since a number of these performance patches use our hash function, would it make sense to apply the last patch to upgrade the hash function mix() to the two function mix()/final()? Since the additional changes increases the performance of the hash function by another 50% or so. My two cents.

Re: [PERFORM] Choosing a filesystem

2008-09-11 Thread Kenneth Marshall
On Thu, Sep 11, 2008 at 06:18:37PM +0100, Matthew Wakeling wrote: > On Thu, 11 Sep 2008, Laszlo Nagy wrote: >> So the basic system will reside on a RAID 1 array, created from two SAS >> disks spinning at 15 000 rpm. I will buy 10 pieces of Seagate Barracuda >> 320GB SATA (ES 7200) disks for the r

Re: [PERFORM] 8.3.1 vs 8.2.X on HP-UX PA-RISC 11.11/11.23

2008-06-12 Thread Kenneth Marshall
Are you using the same locales for both? Ken On Wed, Jun 11, 2008 at 09:40:20PM -0400, Josh Rovero wrote: > We run GCC-compiled postgresql on a number > of HP-UX and Linux boxes. > > Our measurements to date show 8.3.1 > performance to be about 30% *worse* > than 8.2 on HP-UX for the same "drink

Re: [PERFORM] query performance question

2008-06-05 Thread Kenneth Marshall
Dan, Did you try this with 8.3 and its new HOT functionality? Ken On Thu, Jun 05, 2008 at 09:43:06AM -0600, Dan Harris wrote: > [EMAIL PROTECTED] wrote: >> >> 3) Build a table with totals or maybe subtotals, updated by triggers. This >> requires serious changes in application as well as in data

Re: [PERFORM] shared_buffers in 8.3 w/ lots of RAM on dedicated PG machine

2008-02-15 Thread Kenneth Marshall
On Fri, Feb 15, 2008 at 01:35:29PM +0100, Peter Schuller wrote: > Hello, > > my impression has been that in the past, there has been a general > semi-consensus that upping shared_buffers to use the majority of RAM > has not generally been recommended, with reliance on the buffer cache > instead be

Re: [PERFORM] Anyone using a SAN?

2008-02-13 Thread Kenneth Marshall
On Wed, Feb 13, 2008 at 10:56:54AM -0600, Peter Koczan wrote: > Hi all, > > We're considering setting up a SAN where I work. Is there anyone using > a SAN, for postgres or other purposes? If so I have a few questions > for you. > > - Are there any vendors to avoid or ones that are particularly go

Re: [PERFORM] Benchmark Data requested --- pgloader CE design ideas

2008-02-07 Thread Kenneth Marshall
On Thu, Feb 07, 2008 at 12:06:42PM -0500, Greg Smith wrote: > On Thu, 7 Feb 2008, Dimitri Fontaine wrote: > >> I was thinking of not even reading the file content from the controller >> thread, just decide splitting points in bytes (0..ST_SIZE/4 - >> ST_SIZE/4+1..2*ST_SIZE/4 etc) and let the readin

Re: [PERFORM] io storm on checkpoints, postgresql 8.2.4, linux

2007-08-28 Thread Kenneth Marshall
On Wed, Aug 22, 2007 at 07:33:35PM +0400, Dmitry Potapov wrote: > Hello! > > We run a large (~66Gb) web-backend database on Postgresql 8.2.4 on > Linux. The hardware is Dual Xeon 5130 with 16Gb ram, LSI Megaraid U320-2x > scsi controller w/512Mb writeback cache and a BBU. Storage

  1   2   >