Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-10-19 Thread Greg Stark
On Sat, Oct 18, 2014 at 6:01 PM, Tomas Vondra wrote: > Hmmm. I have 0 experience with handling patents and related issues. Any > idea how to address that? Well there's no real way to address it. But to summarize: 1) We should not go searching for patents, knowing that something is patented incre

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-10-17 Thread Greg Stark
On Wed, Oct 15, 2014 at 7:02 PM, Tomas Vondra wrote: > If you know the title of the article, it's usually available elsewhere > on the web - either at the university site, or elsewhere. I found these > two articles about block-based sampling: > > > http://ranger.uta.edu/~gdas/websitepages/preprint

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-10-10 Thread Greg Stark
On Thu, Oct 2, 2014 at 8:56 PM, Josh Berkus wrote: > Yes, it's only intractable if you're wedded to the idea of a tiny, > fixed-size sample. If we're allowed to sample, say, 1% of the table, we > can get a MUCH more accurate n_distinct estimate using multiple > algorithms, of which HLL is one. W

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-09-29 Thread Greg Stark
On Fri, Sep 26, 2014 at 9:06 AM, Simon Riggs wrote: > If we can at least agree it is a problem, we can try to move forwards. Well that's a good question. I don't think we do and I think the reason why is because we haven't actually pinned down exactly what is the problem. The real problem here i

Re: [PERFORM] Yet another abort-early plan disaster on 9.3

2014-09-19 Thread Greg Stark
On 19 Sep 2014 19:40, "Josh Berkus" wrote: > > On 09/19/2014 10:15 AM, Merlin Moncure wrote: > > On Wed, Sep 17, 2014 at 7:11 PM, Josh Berkus wrote: > >> This is the core issue with abort-early plans; they depend on our > >> statistics being extremely accurate, which we know they are not. And if

Re: [PERFORM] 8K recordsize bad on ZFS?

2010-05-10 Thread Greg Stark
On Mon, May 10, 2010 at 8:30 PM, Josh Berkus wrote: > Ivan, > >> Other things could have influenced your result - 260 MB/s vs 300 MB/s is >> close enough to be influenced by data position on (some of) the drives. >> (I'm not saying anything about the original question.) > > You misread my post.  I

Re: [PERFORM] Block at a time ...

2010-03-22 Thread Greg Stark
On Mon, Mar 22, 2010 at 6:47 PM, Scott Carey wrote: > Its fairly easy to break.  Just do a parallel import with say, 16 concurrent > tables being written to at once.  Result?  Fragmented tables. > Fwiw I did do some investigation about this at one point and could not demonstrate any significant

Re: [PERFORM] Block at a time ...

2010-03-17 Thread Greg Stark
On Wed, Mar 17, 2010 at 7:32 AM, Pierre C wrote: >> I was thinking in something like that, except that the factor I'd use >> would be something like 50% or 100% of current size, capped at (say) 1 GB. This turns out to be a bad idea. One of the first thing Oracle DBAs are told to do is change this

Re: [PERFORM] shared_buffers advice

2010-03-16 Thread Greg Stark
On Tue, Mar 16, 2010 at 2:30 PM, Tom Lane wrote: > "Pierre C" writes: >> Does PG issue checkpoint writes in "sorted" order ? > > No.  IIRC, a patch for that was submitted, and rejected because no > significant performance improvement could be demonstrated.  We don't > have enough information abou

Re: [PERFORM] shared_buffers advice

2010-03-16 Thread Greg Stark
On Tue, Mar 16, 2010 at 1:48 PM, Pierre C wrote: > Actually, I meant that in the case of a seq scan, PG will try to use just a > few buffers (a ring) in shared_buffers instead of thrashing the whole > buffers. But if there was actually a lot of free space in shared_buffers, do > the pages stay, or

[PERFORM] Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2010-02-08 Thread Greg Stark
On Mon, Feb 8, 2010 at 4:53 AM, Robert Haas wrote: > On Sun, Feb 7, 2010 at 10:09 PM, Alvaro Herrera >> Yeah, it seems there are two patches here -- one is the addition of >> fsync_fname() and the other is the fsync_prepare stuff. Sorry, I'm just catching up on my mail from FOSDEM this past weeke

[PERFORM] Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2010-02-03 Thread Greg Stark
On Tue, Feb 2, 2010 at 7:45 PM, Robert Haas wrote: > I think you're probably right, but it's not clear what the new name > should be until we have a comment explaining what the function is > responsible for. So I wrote some comments but wasn't going to repost the patch with the unchanged name wit

[PERFORM] Re: use pgsql in a big project, but i found pg has some big problem on concurrency write operation, maybe a joke for myself !

2010-02-02 Thread Greg Stark
2010/2/2 : > UPDATE webpages SET hits = hits + 1 WHERE url ='some url '; > >  when concurrency write transaction on read committed isolation , the hits > may result wrong . That should work fine. All updates for the same url will be serialized. The rest I'm pretty uncertain about what you're de

Re: [PERFORM] Constraint propagating for equal fields

2010-01-29 Thread Greg Stark
2010/1/28 Віталій Тимчишин > > I've always thought that PostgreSQL would propagate constraint from field1 to > field2 if condition says field1=field2, but this does not seem the case: version? -- greg -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make change

[PERFORM] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2010-01-29 Thread Greg Stark
On Tue, Jan 19, 2010 at 3:25 PM, Tom Lane wrote: > That function *seriously* needs documentation, in particular the fact > that it's a no-op on machines without the right kernel call.  The name > you've chosen is very bad for those semantics.  I'd pick something > else myself.  Maybe "pg_start_dat

Re: [PERFORM] ext4 finally doing the right thing

2010-01-21 Thread Greg Stark
Both of those refer to the *drive* cache. greg On 21 Jan 2010 05:58, "Greg Smith" wrote: Greg Stark wrote: > > > That doesn't sound right. The kernel having 10% of memory dirty doesn't mean... Most safe ways ext3 knows how to initiate a write-out on something that

Re: [PERFORM] ext4 finally doing the right thing

2010-01-20 Thread Greg Stark
That doesn't sound right. The kernel having 10% of memory dirty doesn't mean there's a queue you have to jump at all. You don't get into any queue until the kernel initiates write-out which will be based on the usage counters -- basically a lru. fsync and cousins like sync_file_range and posix_fadv

[PERFORM] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2010-01-19 Thread Greg Stark
On Tue, Jan 19, 2010 at 2:52 PM, Greg Stark wrote: > Barring any objections shall I commit it like this? Actually before we get there could someone who demonstrated the speedup verify that this patch still gets that same speedup? -- greg -- Sent via pgsql-performance mailing list (pg

[PERFORM] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2010-01-19 Thread Greg Stark
On Mon, Jan 18, 2010 at 4:35 PM, Greg Stark wrote: > Looking at this patch for the commitfest I have a few questions. So I've touched this patch up a bit: 1) moved the posix_fadvise call to a new fd.c function pg_fsync_start(fd,offset,nbytes) which initiates an fsync without waiti

[PERFORM] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2010-01-18 Thread Greg Stark
Looking at this patch for the commitfest I have a few questions. 1) You said you added an fsync of the new directory -- where is that I don't see it anywhere. 2) Why does the second pass to do the fsyncs read through fromdir to find all the filenames. I find that odd and counterintuitive. It woul

[PERFORM] Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-29 Thread Greg Stark
On Tue, Dec 29, 2009 at 2:05 AM, Andres Freund wrote: >  Reads Completed:        2,        8KiB  Writes Completed:     2362,     > 29672KiB > New: >  Reads Completed:        0,        0KiB  Writes Completed:      550,     > 5960KiB It looks like the new method is only doing 1/6th as much i/o. Do

[PERFORM] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Greg Stark
On Mon, Dec 28, 2009 at 10:54 PM, Andres Freund wrote: > fsync everything in that pass. > Including the directory - which was not done before and actually might be > necessary in some cases. Er. Yes. At least on ext4 this is pretty important. I wish it weren't, but it doesn't look like we're goin

Re: [PERFORM] Idea how to get rid of Bitmap Heap Scan

2009-12-20 Thread Greg Stark
On Sun, Dec 20, 2009 at 2:11 AM, Robert Haas wrote: > On Fri, Dec 18, 2009 at 12:29 PM, Greg Stark wrote: >> A word of warning, in my experience the hardest part for changes like >> this isn't the executor changes (which in this case wouldn't be far >> from e

Re: [PERFORM] Idea how to get rid of Bitmap Heap Scan

2009-12-18 Thread Greg Stark
On Fri, Dec 18, 2009 at 4:18 PM, Michael N. Mikhulya wrote: > Thank you very much. I catch the point why it is done so. > > But I'm curious whether it is still possible to don't fetch data from > files table just because inappropriate ids (e.g. removed ones) will > not produce any wrong effect jus

Re: [PERFORM] Vacuum running out of memory

2009-12-08 Thread Greg Stark
On Tue, Dec 8, 2009 at 4:31 PM, Jonathan Foy wrote: > I was wondering if that was the problem.  So I'm correct in thinking that > the failure occurred when the vacuum tried to pull its 256 MB as defined in > the maintenance_work_mem value, and the system just did not have enough > available... Co

Re: [PERFORM] Cost of sort/order by not estimated by the query planner

2009-12-02 Thread Greg Stark
On Wed, Dec 2, 2009 at 11:13 AM, Laurent Laborde wrote: >>                                           QUERY PLAN >> - >>  Limit  (cost=0.00..2042.87 rows=5 width=1114) >>   ->  Index Scan using _article_

Re: [PERFORM] Cost of sort/order by not estimated by the query planner

2009-12-02 Thread Greg Stark
On Wed, Dec 2, 2009 at 11:13 AM, Laurent Laborde wrote: >>                                             QUERY PLAN >> - >>  Limit  (cost=66114.13..66115.38 rows=500 width=1114) >>   ->  Sort  (cost=6

Re: [PERFORM] OpenMP in PostgreSQL-8.4.0

2009-11-29 Thread Greg Stark
On Sun, Nov 29, 2009 at 1:24 PM, Reydan Cankur wrote: > So I am trying to understand that can anyone rewrite some functions in > postgresql with OpenMP in order to increase performance. > does this work? Well you have to check the code path you're parallelizing for any function calls which might

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-10-31 Thread Greg Stark
On Fri, Oct 30, 2009 at 8:11 PM, Tom Lane wrote: > But having said that, this particular test case is far from compelling. > Any sane text search application is going to try to filter out > common words as stopwords; it's only the failure to do that that's > making this run slow. Well it would be

Re: [PERFORM] database size growing continously

2009-10-30 Thread Greg Stark
On Fri, Oct 30, 2009 at 1:18 PM, Jeremy Harris wrote: > So, on the becoming more intelligent front:  PostgreSQL already does > some operations as background maintenance (autovacuum).  Extending > this to de-bloat indices does not seem conceptually impossible It could be done but it's not easy bec

Re: [PERFORM] database size growing continously

2009-10-30 Thread Greg Stark
On Fri, Oct 30, 2009 at 12:53 PM, Anj Adu wrote: > Any relational database worth its salt has partitioning for a reason. > > 1. Maintenance.  You will need to delete data at some > point.(cleanup)...Partitions are the only way to do it effectively. This is true and it's unavoidably a manual proce

Re: [PERFORM] Indexes on low cardinality columns

2009-10-16 Thread Greg Stark
On Fri, Oct 16, 2009 at 4:36 PM, Vikul Khosla wrote: > In Oracle, we replaced the B-Tree Indexes with Bitmap indexes and saw > performance go > through the roof. I know Postgres does not have Bitmap indexes, > but is there a reasonable alternative to boost performance in situations > where low car

[PERFORM] Re: partition queries hitting all partitions even though check key is specified

2009-09-02 Thread Greg Stark
On Wed, Sep 2, 2009 at 4:05 PM, Kevin Kempter wrote: > explain select * from pwreport.url_hits where "time" > extract('epoch' from > timestamp '2009-08-12 00:00:00')::int4; > Hm. Actually I would have thought this would work. You're using "timestamp" which defaults to without timezone and date_par

Re: [PERFORM] Number of tables

2009-08-31 Thread Greg Stark
On Tue, Sep 1, 2009 at 1:19 AM, Mike Ivanov wrote: >> i am developing a web app for thousands users (1.000/2.000). >> >> Each user have a 2 table of work...I finally have 2.000 (users) x 2 tables >> = 4.000 tables! > > As a someone with a ~50K-table database, I can tell you it's definitely > possib

Re: [PERFORM] What exactly is postgres doing during INSERT/UPDATE ?

2009-08-30 Thread Greg Stark
On Sun, Aug 30, 2009 at 11:56 PM, Merlin Moncure wrote: > 192k written >  raid 10: six writes >  raid 5: four writes, one read (but the read and one of the writes is > same physical location) > > now, by 'same physical' location, that may mean that the drive head > has to move if the data is not in

Re: [PERFORM] What exactly is postgres doing during INSERT/UPDATE ?

2009-08-30 Thread Greg Stark
On Sun, Aug 30, 2009 at 4:40 PM, Merlin Moncure wrote: > For random writes, raid 5 has to write a minimum of two drives, the > data being written and parity.  Raid 10 also has to write two drives > minimum.  A lot of people think parity is a big deal in terms of raid > 5 performance penalty, but I

[PERFORM] Re: PostgreSQL does CAST implicitely between int and a domain derived from int

2009-08-29 Thread Greg Stark
On Sat, Aug 29, 2009 at 10:45 PM, Tom Lane wrote: > Jean-Michel =?ISO-8859-1?Q?Pour=E9?= writes: >> What do you recommend: using normal types and moving constraints in the >> Drupal database? Is PostgreSQL domain broken as it forces casting or is >> this a no-op for performance? > > In principle i

Re: [PERFORM] What exactly is postgres doing during INSERT/UPDATE ?

2009-08-29 Thread Greg Stark
On Sat, Aug 29, 2009 at 5:20 AM, Luke Koops wrote: > Joseph S Wrote >> If I have 14 drives in a RAID 10 to split between data tables >> and indexes what would be the best way to allocate the drives >> for performance? > > RAID-5 can be much faster than RAID-10 for random reads and writes.  It is >

Re: [PERFORM] Performance issues with large amounts of time-series data

2009-08-26 Thread Greg Stark
2009/8/26 Tom Lane : >> How does a float ("REAL") compare in terms of SUM()s ? > > Casting to float or float8 is certainly a useful alternative if you > don't mind the potential for roundoff error.  On any non-ancient > platform those will be considerably faster than numeric.  BTW, > I think that 8

Re: [PERFORM] Number of tables

2009-08-20 Thread Greg Stark
On Fri, Aug 21, 2009 at 1:38 AM, Alvaro Herrera wrote: > Greg Stark wrote: > >> It would be nice to have a solution to that where you could create >> lightweight temporary objects which belong to an "application session" >> which can be picked up by a dif

Re: [PERFORM] Number of tables

2009-08-20 Thread Greg Stark
On Thu, Aug 20, 2009 at 11:18 PM, Craig James wrote: > Greg Stark wrote: >> >> What you want is a multi-column primary key where userid is part of >> the key. You don't want to have a separate table for each user unless >> each user has their own unique set of colum

Re: [PERFORM] Number of tables

2009-08-20 Thread Greg Stark
On Thu, Aug 20, 2009 at 9:16 PM, Craig James wrote: > Fabio La Farcioli wrote: >> >> i am developing a web app for thousands users (1.000/2.000). >> >> Each user have a 2 table of work...I finally have 2.000 (users) x 2 tables >> = 4.000 tables! >> >> Postgres support an elevate number of tables??

Re: [PERFORM] Weird index or sort behaviour

2009-08-18 Thread Greg Stark
On Tue, Aug 18, 2009 at 5:57 PM, Tom Lane wrote: > Hmmm ... actually, after looking at the code, I notice that we only > consider adding a Materialize node to buffer an inner input that is a > Sort node.  The idea was suggested by Greg Stark, if memory serves. > I wonder now if it

Re: [PERFORM] Under the hood of views

2009-08-13 Thread Greg Stark
On Fri, Aug 14, 2009 at 12:04 AM, David Kerr wrote: > On Thu, Aug 13, 2009 at 05:28:01PM +0100, Richard Huxton wrote: > - David Kerr wrote: > - > > - >create view test as > - >select a,b,c,d,e,f,g from testtable; > - > > - >select a from test; > - > > - >(does the engine retrieve b-g?) > - > - Shou

Re: [PERFORM] How to run this in reasonable time:

2009-08-13 Thread Greg Stark
On Thu, Aug 13, 2009 at 3:16 PM, Matthew Wakeling wrote: > Now, I'd like to get this done this side of Christmas, so I was wondering if > there's a neat trick I can use to get it to only consider the rows from s to > e, instead of having to iterate through them all. I tried this, but got an > error

Re: [PERFORM] PG-related ACM Article: "The Pathologies of Big Data"

2009-08-07 Thread Greg Stark
On Fri, Aug 7, 2009 at 9:17 PM, Josh Kupershmidt wrote: > Just stumbled across this recent article published in the > Communications of the ACM: > > http://cacm.acm.org/magazines/2009/8/34493-the-pathologies-of-big-data/fulltext > > The author shares some insights relating to difficulties processin

Re: [PERFORM] CHECK constraint fails when it's not supposed to

2009-08-04 Thread Greg Stark
On Tue, Aug 4, 2009 at 5:49 PM, Richard Yen wrote: > > The data reads: >> >> tii=# select date_start, date_end, term_length, '2009-09-03 >> 05:38:24.030331-07'::timestamptz - date_start AS new_term_length from >> m_class where id = 2652020; >>         date_start          |          date_end        

Re: [PERFORM] GiST, caching, and consistency

2009-08-04 Thread Greg Stark
On Tue, Aug 4, 2009 at 11:56 PM, Robert Haas wrote: > Beats me.  It looks like the first few queries are pulling stuff into > cache, and then after that it settles down, but I'm not sure why it > takes 5 repetitions to do that.  Is the plan changing? Yeah, we're just guessing without the explain a

Re: [PERFORM] SQL select query becomes slow when using limit (with no offset)

2009-07-31 Thread Greg Stark
On Fri, Jul 31, 2009 at 1:11 PM, Kees van Dieren wrote: > It takes 155ms to run this query (returning 2 rows) > > Query plan: without limit: > "Sort  (cost=20169.62..20409.50 rows=95952 width=16)" Could you send the results of EXPLAIN ANALYZE for both queries? Evidently the planner is expecting a

Re: [PERFORM] PostgreSQL 8.4 performance tuning questions

2009-07-30 Thread Greg Stark
On Thu, Jul 30, 2009 at 11:30 PM, Tom Lane wrote: > I did some tracing and verified that pg_dump passes data to deflate() > one table row at a time.  I'm not sure about the performance > implications of that, but it does seem like it might be something to > look into. I suspect if this was a probl

Re: [PERFORM] cluster index on a table

2009-07-16 Thread Greg Stark
On Fri, Jul 17, 2009 at 1:02 AM, Scott Carey wrote: > Indexes would point to a heap page for normal tables and clustered index > pages for clustered tables.  When new versions of data come in, it may point > to new clustered index pages, just like they currently get modified to point > to new heap

Re: [PERFORM] cluster index on a table

2009-07-16 Thread Greg Stark
> Scara Maccai wrote: > >> What am I doing wrong? I didn't exactly follow the full sequence but it sounded like what's happening is that Postgres is noticing all these empty pages from earlier deletes and reusing that space. That's what it's designed to do. As Kevin said, there's no guarantee tha

Re: [PERFORM] cluster index on a table

2009-07-16 Thread Greg Stark
On Thu, Jul 16, 2009 at 9:06 PM, Scott Carey wrote: > Keep the old page around or a copy of it that old transactions reference? > Just more Copy on Write. > How is that different from a nested loop on an index scan/seek currently? > Doesn't an old transaction have to reference an old heap page thro

Re: [PERFORM] cluster index on a table

2009-07-16 Thread Greg Stark
On Thu, Jul 16, 2009 at 8:18 PM, Scott Carey wrote: > " Each index row in the nonclustered index contains the nonclustered key > value and a row locator. This locator points to the data row in the > clustered index or heap having the key value." > > That sort of model should work with MVCC and even

Re: [PERFORM] Repeated Query is much slower in PostgreSQL8.2.4 than DB2 9.1

2009-07-15 Thread Greg Stark
On Wed, Jul 15, 2009 at 9:27 AM, Craig Ringer wrote: > On Wed, 2009-07-15 at 12:10 +0900, ning wrote: > >> First execution: PostgreSQL 0.006277 seconds / DB2 0.009028 seconds >> Second execution: PostgreSQL 0.005932 seconds / DB2 0.000332 seconds > > Actually, on second thoughts that looks a lot li

Re: [PERFORM] Poor query performance

2009-07-15 Thread Greg Stark
On Wed, Jul 15, 2009 at 8:51 AM, Alex wrote: > Also posted this to the list.  Thanks for your answer - still > struggling. Staying on-list is always preferred. >> How is the index  sl_city_etc defined? > >         Index "public.sl_city_etc" >    Column    |            Type > --+--

Re: [PERFORM] Poor query performance

2009-07-14 Thread Greg Stark
On Thu, Jul 9, 2009 at 10:35 PM, Alex wrote: > Forgot to add: > > postg...@ec2-75-101-128-4:~$ psql --version > psql (PostgreSQL) 8.3.5 How is the index sl_city_etc defined? -- greg http://mit.edu/~gsstark/resume.pdf -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.or

Re: [PERFORM] Six PostgreSQL questions from a pokerplayer

2009-07-06 Thread Greg Stark
On Sat, Jul 4, 2009 at 7:51 PM, Patvs wrote: > -4 One a scale from 1 to 10, how significant are the following on > performance increase: > -[ ] Getting a faster harddisk (RAID or a SSD) > -[ ] Getting a faster CPU > -[ ] Upgrading PostgreSQL (8.2 and 8.3) to 8.4 > -[ ] Tweaking PostgreSQL (increasi

Re: [PERFORM] slow DELETE on 12 M row table

2009-06-25 Thread Greg Stark
On Fri, Jun 26, 2009 at 3:33 AM, Janet Jacobsen wrote: > (1) is my interpretation of the posts correct, i.e., if I am deleting > rows from > table1, where the pkey of table 1 is a fkey in table 2, then do I need > to create an > index on the fkey field in table 2? Exactly right. The index on the t

Re: [PERFORM] Nested Loop "Killer" on 8.1

2009-06-25 Thread Greg Stark
On Thu, Jun 25, 2009 at 10:05 PM, Tom Lane wrote: > > Uh, it appears to me the string *does* contain _ characters; perhaps the > OP has neglected to escape those? Sigh. Indeed. -- greg http://mit.edu/~gsstark/resume.pdf -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.o

Re: [PERFORM] Nested Loop "Killer" on 8.1

2009-06-25 Thread Greg Stark
On Wed, Jun 24, 2009 at 1:43 PM, Dave North wrote: > Essentially, we're seeing a query plan that is taking 95 secs with a nested > loop execution plan and 1 sec with a merge join plan.  We've tried > increasing the default_statistics_target to 1000 and re-analyzed but the > same query plan is retu

Re: [PERFORM] How would you store read/unread topic status?

2009-06-23 Thread Greg Stark
On Tue, Jun 23, 2009 at 9:04 PM, Mathieu Nebra wrote: > We have indexes on them, so we can SELECT every topic WHERE the user has > written. Is it the good way of doing this? I'm kind of skeptical that a simple index on userid,topic isn't sufficient to handle this case. But you would have to test i

Re: [PERFORM] How would you store read/unread topic status?

2009-06-23 Thread Greg Stark
All the other comments are accurate, though it does seem like something the database ought to be able to handle. The other thing which hasn't been mentioned is that you have a lot of indexes. Updates require maintaining all those indexes. Are all of these indexes really necessary? Do you have rout

Re: [PERFORM] select max() much slower than select min()

2009-06-19 Thread Greg Stark
On Fri, Jun 19, 2009 at 3:26 PM, Tom Lane wrote: > > That's the problem then.  Notice what the query plan is doing: it's > scanning the table in order by ts_id, looking for the first row that > falls within the ts_interval_start_time range.  Evidently this > particular range is associated with smal

Re: [PERFORM] PostgreSQL with PostGIS on embedded hardware

2009-05-13 Thread Greg Stark
On Mon, May 11, 2009 at 5:05 PM, Stefan Kaltenbrunner wrote: >> Good to know!!! I imagine that on a PS3 it would be _really_ fast... :-) > > well not really - while it is fairly easy to get postgresql running on a PS3 > it is not a fast platform. While the main CPU there is a pretty fast Power > b

Re: [PERFORM] Any better plan for this query?..

2009-05-12 Thread Greg Stark
On Tue, May 12, 2009 at 5:49 PM, Tom Lane wrote: > See previous discussions.  IIRC, there are two killer points: > > 1. There is no (portable) way to pass the connection from the postmaster > to another pre-existing process. The Apache model is to have all the backends call accept. So incoming co

Re: [PERFORM] high shared buffer and swap

2009-05-04 Thread Greg Stark
Sorry for top-posting - the iphone mail client sucks. I think what's happening is that the sytem is seeing that some pages of shared memory haven't been used recently and because there's more shared memory than filesystem cache less recently than the filesystem cache pages. So it pages out

Re: [PERFORM] Need help with 8.4 Performance Testing

2008-12-09 Thread Greg Stark
On Sun, Dec 7, 2008 at 7:38 PM, Josh Berkus <[EMAIL PROTECTED]> wrote: > > Also, the following patches currently still have bugs, but when the bugs are > fixed I'll be looking for performance testers, so please either watch the > wiki or watch this space: >... > -- posix_fadvise (Gregory Stark) Eh

Re: [PERFORM] Increasing GROUP BY CHAR columns speed

2008-11-30 Thread Greg Stark
On Sat, Nov 29, 2008 at 6:43 PM, Andrus <[EMAIL PROTECTED]> wrote: >> I'm still not sure why the planner chose to sort rather than hash with >> oversized work_mem (is there an implied order in the query results I >> missed?). > > Group by contains decimal column exchrate. Maybe pg is not capable to

Re: [PERFORM] Anyone using a SAN?

2008-02-14 Thread Greg Stark
Tobias Brox wrote: [Peter Koczan - Wed at 10:56:54AM -0600] The consensus on this list seemed to be that running postgres on SAN is not cost efficiently - one would get better performance for a lower cost if the database host is connected directly to the disks - and also, buying the wrong SAN

Re: [PERFORM] size of pg_dump files containing bytea values

2006-07-13 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > "Steve McWilliams" <[EMAIL PROTECTED]> writes: > > I notice that non-printables in bytea values are being spit out by pg_dump > > using escaped octet sequences even when the "-Fc" option is present > > specifying use of the custom binary output format rather

Re: [PERFORM] Optimizer internals

2006-06-16 Thread Greg Stark
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > Now, if we're considering UPDATES (the worst case for PostgreSQL's > current MVCC architecture), then this is (IMHO) a true statement. > There aren't many *successful* commercial databases that incur the > additional overhead of creating another vers

Re: [PERFORM] SAN performance mystery

2006-06-16 Thread Greg Stark
"Alex Turner" <[EMAIL PROTECTED]> writes: > Given the fact that most SATA drives have only an 8MB cache, and your RAID > controller should have at least 64MB, I would argue that the system with the > RAID controller should always be faster. If it's not, you're getting > short-changed somewhere,

Re: [PERFORM] Optimizer internals

2006-06-16 Thread Greg Stark
Mark Lewis <[EMAIL PROTECTED]> writes: > On Thu, 2006-06-15 at 14:05 -0400, John Vincent wrote: > > Now I've been told by our DBA that we should have been able to wholy > > satisfy that query via the indexes. > > DB2 can satisfy the query using only indexes because DB2 doesn't do > MVCC. Well it

Re: [PERFORM] Postgres fsync off (not needed) with NetApp

2006-06-14 Thread Greg Stark
Mark Lewis <[EMAIL PROTECTED]> writes: > On Wed, 2006-06-14 at 14:48 -0700, Dan Gorman wrote: > > > > However, if I have it attached to a NetApp that ensures data writes > > to via the NVRAM can I safely turn fsync off to gain additional > > performance? > > No. You need fsync on in order to

Re: [PERFORM] OT - select + must have from - sql standard syntax?

2006-06-13 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > The spec does require a FROM clause in SELECT (at least as of SQL99, did > not check SQL2003). However, it's clearly mighty useful to allow FROM > to be omitted for simple compute-this-scalar-result problems. You > should respond to the Oracle guy that "SE

Re: [PERFORM] function not called if part of aggregate

2006-06-11 Thread Greg Stark
"Craig A. James" <[EMAIL PROTECTED]> writes: > This doesn't seem right to me -- how can the optimizer possibly know that a > function doesn't have a side effect, as in my case? Functions could do all > sorts of things, such as logging activity, filling in other tables, etc, etc. The optimizer ca

Re: [PERFORM] Some queries starting to hang

2006-06-06 Thread Greg Stark
Simon Riggs <[EMAIL PROTECTED]> writes: > The idea I just had was: why do we need EXPLAIN ANALYZE to run to > completion? In severe cases like this thread, we might be able to > discover the root cause by a *partial* execution of the plan, as long as > it was properly instrumented. That way, the O

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-31 Thread Greg Stark
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > I tried duplicating this but couldn't. What's the data in the tables? Sorry, I had intended to include the definition and data: stark=> create table t1 (a integer primary key, b integer); NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-31 Thread Greg Stark
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > On Sun, May 28, 2006 at 07:20:59PM -0400, Greg Stark wrote: > > Brendan Duddridge <[EMAIL PROTECTED]> writes: > > > > > We do have foreign keys on other tables that reference the product table. > >

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-28 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Greg Stark <[EMAIL PROTECTED]> writes: > > > Except I don't think this is taking an exclusive lock at all. The original > > post had the deadlock detection fire on a SharedLock. > > Yeah, but it was a ShareLock on

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-28 Thread Greg Stark
Brendan Duddridge <[EMAIL PROTECTED]> writes: > We do have foreign keys on other tables that reference the product table. > Also, there will be updates going on at the same time as this update. When > anyone clicks on a product details link, we issue an update statement to > increment the click

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-28 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Greg Stark <[EMAIL PROTECTED]> writes: > > What queries are those two processes executing? And what foreign keys do you > > have on the product table or elsewhere referring to the product table? And > > what indexes do yo

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-28 Thread Greg Stark
Brendan Duddridge <[EMAIL PROTECTED]> writes: > Further to my issue, the update never did finish. I received the following > message in psql: > > ssprod=# update product set is_hungry = 'true' where date_modified > > current_date - 10; > ERROR: deadlock detected > DETAIL: Process 18778 waits

Re: [PERFORM] [HACKERS] Big IN() clauses etc : feature proposal

2006-05-11 Thread Greg Stark
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > Perhaps it would be worth creating a class of temporary tables that used > a tuplestore, although that would greatly limit what could be done with > that temp table. I can say that I've seen plenty of instances where the ability to create temporary ta

Re: [PERFORM] [GENERAL] Arguments Pro/Contra Software Raid

2006-05-09 Thread Greg Stark
Steve Atkins <[EMAIL PROTECTED]> writes: > On May 9, 2006, at 2:16 AM, Hannes Dorbath wrote: > > > Hi, > > > > I've just had some discussion with colleagues regarding the usage of > > hardware or software raid 1/10 for our linux based database servers. > > > > I myself can't see much reason to

Re: [PERFORM] [GENERAL] Arguments Pro/Contra Software Raid

2006-05-09 Thread Greg Stark
Douglas McNaught <[EMAIL PROTECTED]> writes: > Vivek Khera <[EMAIL PROTECTED]> writes: > > > On May 9, 2006, at 11:51 AM, Joshua D. Drake wrote: > > > >> And dollar for dollar, SCSI will NOT be faster nor have the hard > >> drive capacity that you will get with SATA. > > > > Does this hold true s

Re: [PERFORM] [HACKERS] Big IN() clauses etc : feature proposal

2006-05-09 Thread Greg Stark
PFC <[EMAIL PROTECTED]> writes: > > I really like this. It's clean, efficient, and easy to use. > > This would be a lot faster than using temp tables. > Creating cursors is very fast so we can create two, and avoid doing > twice the same work (ie. hashing the ids from the res

Re: [PERFORM] pgmemcache

2006-04-13 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Christian Storm <[EMAIL PROTECTED]> writes: > > Not sure if I follow why this is a problem. Seems like it would be > > beneficial to have both BEFORE and AFTER COMMIT triggers. > > With the BEFORE COMMIT trigger you would have the ability to 'un- > > comm

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Greg Stark
Gavin Hamill <[EMAIL PROTECTED]> writes: > This is one hell of a moving target and I can't help but think I'm just > missing something that's right in front of my nose, too close to see. I'm assuming you compiled postgres yourself? Do you have the output from the configure script? I'm wondering

Re: [PERFORM] Bad row estimates

2006-03-08 Thread Greg Stark
Alex Adriaanse <[EMAIL PROTECTED]> writes: > Its row estimates are still way off. As a matter of fact, it almost seems as > if the index doesn't affect row estimates at all. Indexes normally don't affect estimates. Expression indexes do effectively create a new column to generate stats for, but

Re: [PERFORM] Bad row estimates

2006-03-04 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Otherwise I think you really need a special datatype for time > intervals and a GIST or r-tree index on it :-(. You could actually take short cuts using expression indexes to do this. If it works out well then you might want to implement a real data type to

Re: [PERFORM] Bad row estimates

2006-03-04 Thread Greg Stark
Greg Stark <[EMAIL PROTECTED]> writes: > The "now() BETWEEN test_table_1.start_ts AND test_table_1.end_ts can't be > answered completely using a btree index. You could try using a GIST index here > but I'm not clear how much it would help you (or how much work i

Re: [PERFORM] Bad row estimates

2006-03-03 Thread Greg Stark
Alex Adriaanse <[EMAIL PROTECTED]> writes: > SELECT count(*) FROM test_table_1 > INNER JOIN test_table_2 ON > (test_table_2.s_id = 13300613 AND test_table_1.id = test_table_2.n_id) > WHERE now() BETWEEN test_table_1.start_ts AND test_table_1.end_ts > AND test_table_1.id = test_

Re: [PERFORM] Like 'name%' is not using index

2006-03-02 Thread Greg Stark
"Jozsef Szalay" <[EMAIL PROTECTED]> writes: > One would > think that Postgres will use the index to look up the matches, but > apparently that is not the case. It performs a full table scan. My > query looks something like this: > > SELECT * FROM table WHERE name LIKE 'smith%'; There are two p

Re: [PERFORM] Bad plan on a view

2006-03-01 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > PFC <[EMAIL PROTECTED]> writes: > > So, in order to speed up requests which need a full table scan, I wanted > > to put the text fields in another table, and use a view to make it look > > like nothing happened. Also, the small table used for searching is

Re: [PERFORM] Good News re count(*) in 8.1

2006-02-22 Thread Greg Stark
"Kevin Grittner" <[EMAIL PROTECTED]> writes: > There have been several times that I have run a SELECT COUNT(*) on an entire > table on all central machines. On identical hardware, with identical data, > and equivalent query loads, the PostgreSQL databases have responded with a > count in 50% to 7

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index

2006-02-16 Thread Greg Stark
Markus Schaber <[EMAIL PROTECTED]> writes: > Hmm, to remove redundancy, I'd change the <= to a < and define: > > if a==b then f(a)==f(b) > if a > > Data types which could probably provide a useful function for f would be > > int2, int4, oid, and possibly int8 and text (at least for SQL_ASCII).

Re: [PERFORM] Reliability recommendations

2006-02-15 Thread Greg Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > Jeremy Haile wrote: > > We are a small company looking to put together the most cost effective > > solution for our production database environment. Currently in > > production Postgres 8.1 is running on this machine: > > > > Dell 2850 > > 2 x 3.0

Re: [PERFORM] Help with optimizing a sql statement

2006-02-09 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Are you sure this WHERE clause really expresses your intent? It seems > awfully oddly constructed. Removing the redundant parens and clarifying > the layout, I get ... > That next-to-last major AND clause seems a rather unholy mix of join and > restriction

  1   2   3   4   >