Re: [PERFORM] linux deadline i/o elevator tuning

2009-04-09 Thread Matthew Wakeling
longer to seek to the next real request that it will for the program to issue its next request, which is broken on SSDs. Anticipatory attempts to increase performance by being unfair - it is essentially the opposite of CFQ. Matthew -- Now you see why I said that the first seven minutes

Re: [PERFORM] plpgsql arrays

2009-04-08 Thread Matthew Wakeling
On Tue, 7 Apr 2009, Tom Lane wrote: Subsequent discussion showed that the problem was Matthew hadn't found that page. I guess that at least the DECLARE CURSOR reference page ought to have something like if you are trying to use cursors in plpgsql, see link. Matthew, where *were* you looking

Re: [PERFORM] plpgsql arrays

2009-04-07 Thread Matthew Wakeling
be the syntax for putting a single row from a cursor into a variable? I have tried: FETCH INTO left left_cursor; which says syntax error, and left = FETCH left_cursor; which gives the error 'ERROR: missing datatype declaration at or near =' Matthew -- I've run DOOM more in the last few days than I

Re: [PERFORM] plpgsql arrays

2009-04-07 Thread Matthew Wakeling
, gene WHERE location.id = gene.id ORDER BY objectid, start, end; left = FETCH left_cursor; END; $$ LANGUAGE plpgsql; Matthew -- Prove to thyself that all circuits that radiateth and upon which thou worketh are grounded, lest they lift thee to high-frequency potential and cause thee to radiate

Re: [PERFORM] plpgsql arrays

2009-04-07 Thread Matthew Wakeling
On Tue, 7 Apr 2009, Tom Lane wrote: Matthew Wakeling matt...@flymine.org writes: CREATE OR REPLACE FUNCTION overlap_gene_primer() RETURNS SETOF RECORD AS $$ DECLARE left location; retval RECORD; BEGIN DECLARE left_cursor NO SCROLL CURSOR FOR SELECT location FROM location, gene

Re: [PERFORM] plpgsql arrays

2009-04-06 Thread Matthew Wakeling
putting the finishing touches on a plpgsql implementation - in order to perform the join on a asymmetric set of ranges, you just need to keep two separate history lists as you sweep through the two incoming streams. This would be sufficient for range constraints. Matthew -- Surely the value

Re: [PERFORM] difficulties with time based queries

2009-04-06 Thread Matthew Wakeling
hardware in the world, but I would expect better than that for linear queries. Might you have an unbalanced index tree? Reindexing would also solve that problem. Matthew -- There are only two kinds of programming languages: those people always bitch about and those nobody uses. (Bjarne

Re: [PERFORM] plpgsql arrays

2009-04-06 Thread Matthew Wakeling
On Fri, 3 Apr 2009, Simon Riggs wrote: On Fri, 2009-04-03 at 10:04 -0400, Tom Lane wrote: Matthew Wakeling matt...@flymine.org writes: On Fri, 3 Apr 2009, Robert Haas wrote: Why not just use SQL to do the join? Because the merge condition is: WHERE l1.start = l2.end AND l2.start = l1.end

[PERFORM] plpgsql arrays

2009-04-03 Thread Matthew Wakeling
locations', genes_size; For 200,000 rows it takes 40 minutes. So, is there a way to dump the results of a query into an array quickly in plpgsql, or alternatively is there a way to read two results streams simultaneously? Matthew -- I would like to think that in this day and age people would know

Re: [PERFORM] Rewriting using rules for performance

2009-04-03 Thread Matthew Wakeling
thing for me. Matthew -- The email of the species is more deadly than the mail. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] plpgsql arrays

2009-04-03 Thread Matthew Wakeling
On Fri, 3 Apr 2009, Robert Haas wrote: On Fri, Apr 3, 2009 at 9:32 AM, Matthew Wakeling matt...@flymine.org wrote: I'm writing a plpgsql function that effectively does a merge join on the results of two queries. Why not just use SQL to do the join? Because the merge condition is: WHERE l1

Re: [PERFORM] plpgsql arrays

2009-04-03 Thread Matthew Wakeling
to choose a sort order that works. Sorting by the start field would be sufficient in this case. Matthew -- For those of you who are into writing programs that are as obscure and complicated as possible, there are opportunities for... real fun here -- Computer

Re: [PERFORM] plpgsql arrays

2009-04-03 Thread Matthew Wakeling
On Fri, 3 Apr 2009, Matthew Wakeling wrote: On Fri, 3 Apr 2009, Tom Lane wrote: Not unless you have sorted the inputs in some way that has more knowledge than the equal operator represents. Otherwise you can have elements drop out that might still be needed to match to a later left-hand

Re: [PERFORM] plpgsql arrays

2009-04-03 Thread Matthew Wakeling
they aren't. Our no it isn't messages obviously crossed on the wire. Matthew -- Some people, when confronted with a problem, think I know, I'll use regular expressions. Now they have two problems. -- Jamie Zawinski -- Sent via pgsql-performance mailing list (pgsql-performance

Re: [PERFORM] plpgsql arrays

2009-04-03 Thread Matthew Wakeling
the merge, make sure to use () noation: if (genes[x]).field something then How is that different to genes[x].field? Matthew -- And the lexer will say Oh look, there's a null string. Oooh, there's another. And another., and will fall over spectacularly when it realises there are actually

Re: [PERFORM] Very specialised query

2009-04-02 Thread Matthew Wakeling
Trying to rewrite a plpgsql function in C. How do I do the equivalent of: FOR loc IN SELECT * FROM location ORDER BY objectid, intermine_start, intermine_end LOOP END LOOP; in a C function? Matthew -- I wouldn't be so paranoid if you weren't all out to get me!! -- Sent via pgsql

Re: [PERFORM] self join revisited

2009-04-01 Thread Matthew Wakeling
be gained from a self-join algorithm. For instance, if given some rather evil cleverness, it could be adapted to calculate overlaps very quickly. However, a self-join is very similar to a merge join, and the benefit over a standard merge join would be small. Matthew -- We did a risk management

Re: [PERFORM] Raid 10 chunksize

2009-04-01 Thread Matthew Wakeling
. Computers crash. Hardware fails. Relying on un-backed-up RAM to keep your data safe does not work. Matthew -- Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far

Re: [PERFORM] Raid 10 chunksize

2009-04-01 Thread Matthew Wakeling
from turning write-cache 'on' and have everything else in place, well, that seems like a 'no-brainer' to me, at least ;) In that case, buying a battery-backed-up cache in the RAID controller would be even more of a no-brainer. Matthew -- If pro is the opposite of con, what is the opposite

Re: [PERFORM] Very specialised query

2009-04-01 Thread Matthew Wakeling
.end == l2.end. That can be easily filtered by adding where n=1 or l1.start != l2.start or l1.end != l2.end to outer select. Close - duplicates will occur when l1.start == l2.start, so you filter them out by adding where n = 1 OR l1.start l2.start. Matthew -- Lord grant me patience, and I

Re: [PERFORM] Very specialised query

2009-04-01 Thread Matthew Wakeling
-+--+-+--+-- 1 | 61544858 | 1 | 61545105 | 21512431 (1 row) Matthew -- I suppose some of you have done a Continuous Maths course. Yes? Continuous Maths? menacing stares from audience Whoah, it was like that, was it! -- Computer Science Lecturer -- Sent via pgsql

Re: [PERFORM] Raid 10 chunksize

2009-04-01 Thread Matthew Wakeling
to spec anymore, and you're done. Most decent servers have dual power supplies, and they should really be connected to two independent UPS units. You can test them one by one without much risk of bringing down your server. Matthew -- Okay, I'm weird! But I'm saving up to be eccentric. -- Sent via

Re: [PERFORM] Very specialised query

2009-04-01 Thread Matthew Wakeling
On Wed, 1 Apr 2009, Matthew Wakeling wrote: So, I have written a plpgsql function to calculate overlaps. It works reasonably quickly where there aren't that many overlaps. However, it seems to go very slowly when there are a large number of rows to return. In plpgsql, what happens about

Re: [PERFORM] Very specialised query

2009-03-31 Thread Matthew Wakeling
reasonably fast. Matthew -- If you're thinking Oh no, this lecturer thinks Turing Machines are a feasible method of computation, where's the door?, then you are in luck. There are some there, there, and by the side there. Oxygen masks will not drop from the ceiling

Re: [PERFORM] Very specialised query

2009-03-31 Thread Matthew Wakeling
, there's also the great big sort and unique, but I think I can get rid of that. Matthew -- Contrary to popular belief, Unix is user friendly. It just happens to be very selective about who its friends are. -- Kyle Hearn -- Sent via pgsql-performance mailing list (pgsql-performance

Re: [PERFORM] Very specialised query

2009-03-30 Thread Matthew Wakeling
://www.postgresql.org/docs/8.3/interactive/sql-createaggregate.html Can you return multiple rows from an aggregate function? Matthew -- -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Very specialised query

2009-03-30 Thread Matthew Wakeling
to not be that fast. The problem is that OR there, which causes a bitmap index scan, as the leaf of a nested loop join, which can be rather slow. Matthew -- I'd try being be a pessimist, but it probably wouldn't work anyway. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org

Re: [PERFORM] Very specialised query

2009-03-30 Thread Matthew Wakeling
. Therefore, I would expect to only have to take any notice of l2.start when working out selectivity on a range check for this particular plan. When there is an index scan on a different field, then try and match that one up instead. Matthew -- -- Sent via pgsql-performance mailing list (pgsql

Re: [PERFORM] Very specialised query

2009-03-30 Thread Matthew Wakeling
AND l1.end l2.start Unless this particular issue is improved in the planner, I don't think this particular style of query is going to work for us. I know that the database could in theory return a result in about 1400ms. I'll see how close to that I can get with plpgsql. Matthew -- First law

Re: [PERFORM] Very specialised query

2009-03-30 Thread Matthew Wakeling
the two respective forms apart from the CPU cycles required to rewrite the first one into the second one internally. Matthew -- Don't worry! The world can't end today because it's already tomorrow in Australia. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make

Re: [PERFORM] Very specialised query

2009-03-30 Thread Matthew Wakeling
. It is currently only choosing the correct plan now by chance, and some time later it may by chance switch to one that takes 40 minutes. I'll certainly add it to my list of possibilities. Matthew -- Jadzia: Don't forget the 34th rule of acquisition: Peace is good for business. Quark: That's the 35th

Re: [PERFORM] Very specialised query

2009-03-27 Thread Matthew Wakeling
for the data type. Is there an operator class for integer for gist indexes that I can use? Matthew -- And why do I do it that way? Because I wish to remain sane. Um, actually, maybe I should just say I don't want to be any worse than I already am. - Computer Science Lecturer -- Sent via pgsql

Re: [PERFORM] Very specialised query

2009-03-27 Thread Matthew Wakeling
have a cost of (cost=0.00..123.10 rows=4809 width=12) Postgres estimates these two index scans to be equivalent in cost, where they are actually vastly different in real cost. Shouldn't Postgres favour a between index scan over an open-ended one? Matthew -- [About NP-completeness

[PERFORM] Very specialised query

2009-03-26 Thread Matthew Wakeling
to a few seconds. Any chances of it running in Postgres, or any other tips? Matthew -- Hi! You have reached 555-0129. None of us are here to answer the phone and the cat doesn't have opposing thumbs, so his messages are illegible. Please leave your name and message after the beep ... -- Sent

Re: [PERFORM] Very specialised query

2009-03-26 Thread Matthew Wakeling
a simple bio_seg index lookup is the fastest way). Is there a way to define these three methods of generating the results and get the planner to choose the fastest one? Matthew -- Beware of bugs in the above code; I have only proved it correct, not tried

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-20 Thread Matthew Wakeling
be implemented as a lock-free structure. But I don't know what the actual structure is, so I could be talking through my hat. Matthew -- So, given 'D' is undeclared too, with a default of zero, C++ is equal to D. mnw21, commenting on the Surely the value of C++ is zero, but C is now 1 response

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-18 Thread Matthew Wakeling
a snapshot and queueing all newer locks forces fairness in the locking strategy, and avoids one of the sides getting starved. Matthew -- I've run DOOM more in the last few days than I have the last few months. I just love debugging ;-) -- Linus Torvalds - Sent via pgsql-performance mailing list

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-18 Thread Matthew Wakeling
lock on a page is a pretty easy way to wake up at the right time. However, is there not some way to wait for a notify? I'm no C expert, but in Java that's one of the most fundamental features of a lock. Matthew -- A bus station is where buses stop. A train station is where trains stop. On my

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-18 Thread Matthew Wakeling
On Wed, 18 Mar 2009, Simon Riggs wrote: On Wed, 2009-03-18 at 11:45 +, Matthew Wakeling wrote: The problem with making all other locks welcome is that there is a possibility of starvation. Imagine a case where there is a constant stream of shared locks - the exclusive locks may never

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-18 Thread Matthew Wakeling
is the obvious way of doing this. It would make most operations really trivial. Just wake everything in the shared queue at once, and you can throw it away wholesale and allocate a new queue. It avoids a whole lot of queue manipulation. Matthew -- Software suppliers are trying to make their software

Re: [PERFORM] Proposal of tunable fix for scalability of 8.4

2009-03-16 Thread Matthew Wakeling
be. (It may be fewer - I don't know what the average WAL write size is.) Matthew -- What goes up must come down. Ask any system administrator. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [PERFORM] When does sequential performance matter in PG?

2009-03-10 Thread Matthew Wakeling
be spread over the disc array. Matthew -- [About NP-completeness] These are the problems that make efficient use of the Fairy Godmother.-- Computer Science Lecturer -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription

Re: [PERFORM] SSD performance

2009-02-20 Thread Matthew Wakeling
the device into little bits. Especially under database-like access patterns. Matthew -- I quite understand I'm doing algebra on the blackboard and the usual response is to throw objects... If you're going to freak out... wait until party time and invite me along -- Computer Science

Re: [PERFORM] suggestions for postgresql setup on Dell 2950 , PERC6i controller

2009-02-17 Thread Matthew Wakeling
tests on the same partition one after the other instead. Matthew -- We did a risk management review. We concluded that there was no risk of any management.-- Hugo Mills h...@carfax.nildram.co.uk -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make

Re: [PERFORM] Query composite index range in an efficient way

2009-02-17 Thread Matthew Wakeling
Postgres use a sequential scan then? How many rows does your query return? If it's more than about 10% of the total rows in the table, then a sequential scan is probably the fastest method. Matthew -- Matthew: That's one of things about Cambridge - all the roads keep changing names as you

Re: [PERFORM] dissimilar drives in Raid10 , does it make difference ?

2009-02-13 Thread Matthew Wakeling
, you don't lose all your data. Matthew -- Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning. -- Rich Cook -- Sent via pgsql-performance

Re: [PERFORM] explanation of some configs

2009-02-10 Thread Matthew Wakeling
On Mon, 9 Feb 2009, justin wrote: Well then we have conflicting instructions in places on wiki.postgresql.org which links to this http://www.varlena.com/GeneralBits/Tidbits/annotated_conf_e.html Could you be a little more specific as to which sections conflict? Matthew -- The only secure

Re: [PERFORM] explanation of some configs

2009-02-09 Thread Matthew Wakeling
writes until the copying has freed up the checkpoint segments again. Matthew -- Don't worry! The world can't end today because it's already tomorrow in Australia. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http

Re: [PERFORM] [PERFORMANCE] Buying hardware

2009-01-26 Thread Matthew Wakeling
might be okay. If your controller supports it. Matthew -- The third years are wandering about all worried at the moment because they have to hand in their final projects. Please be sympathetic to them, say things like ha-ha-ha, but in a sympathetic tone of voice

Re: [PERFORM] SSD performance

2009-01-23 Thread Matthew Wakeling
happening. Except VMS maybe. Not arguing that your method is faster though. Matthew -- Finger to spiritual emptiness underlying everything. -- How a foreign C manual referred to a pointer to void. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes

Re: [PERFORM] SSD performance

2009-01-23 Thread Matthew Wakeling
is the performance, rather than the data quantity. The thing is, it isn't just a matter of storage heirarchy. There's the volatility matter there as well. What you have in these SSDs is a device which is non-volatile, like a disc, but fast, like RAM. Matthew -- Anyone who goes

Re: [PERFORM] understanding postgres issues/bottlenecks

2009-01-12 Thread Matthew Wakeling
for interesting pieces of hardware, like graphics cards and (unfortunately) some high-performance RAID cards. Matthew -- A good programmer is one who looks both ways before crossing a one-way street. Considering the quality and quantity of one-way streets in Cambridge, it should be no surprise

Re: [PERFORM] Fwd: Casting issue!!

2009-01-07 Thread Matthew Wakeling
using pl/pgsql. I mean i dont want to write value::dataType. I dont want to use explicit type cast. Maybe change something in the config files? to make it work like 8.2 on tha regard(cast values). What does that have to do with performance? Matthew -- Illiteracy - I don't know the meaning

Re: [PERFORM] Experience with HP Smart Array P400 and SATA drives?

2008-12-11 Thread Matthew Wakeling
ahci controller standard, which is useful. Matthew -- The best way to accelerate a Microsoft product is at 9.8 metres per second per second. - Anonymous -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http

Re: [PERFORM] Need help with 8.4 Performance Testing

2008-12-10 Thread Matthew Wakeling
. The X-25M is basically a RAID controller in its own right, connected to ten flash devices. Matthew -- Don't worry! The world can't end today because it's already tomorrow in Australia. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your

Re: [PERFORM] Experience with HP Smart Array P400 and SATA drives?

2008-12-10 Thread Matthew Wakeling
status? Also, is it possible to set the drives in a hardware RAID into auto-spindown mode? I'm going to be putting a SSD in as the main system drive, with the RAID array to hold my large stuff which I only work on part of the time. Matthew -- Failure is not an option. It comes bundled

Re: [PERFORM] Need help with 8.4 Performance Testing

2008-12-09 Thread Matthew Wakeling
Postgres can scale that well. Matthew -- Those who do not understand Unix are condemned to reinvent it, poorly. -- Henry Spencer -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [PERFORM] Postgres using more memory than it should

2008-12-04 Thread Matthew Wakeling
agree it would be nice to fix this, but I'm not sure how at the moment. Matthew -- To most people, solutions mean finding the answers. But to chemists, solutions are things that are still all mixed up. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes

Re: [PERFORM] Postgres using more memory than it should

2008-12-04 Thread Matthew Wakeling
On Thu, 4 Dec 2008, Matthew Wakeling wrote: http://archives.postgresql.org/pgsql-bugs/2008-11/msg00180.php Thanks, that does explain everything. Oh right, yes. It explains everything *except* the fact that the backend is still holding onto all the RAM after the query is finished. Could

[PERFORM] Postgres using more memory than it should

2008-12-03 Thread Matthew Wakeling
] The query that is being run is an INSERT INTO table SELECT a fairly complex query. Any ideas why this is going so badly, and what I can do to solve it? Matthew -- First law of computing: Anything can go wro sig: Segmentation fault. core dumped. -- Sent via pgsql-performance mailing list (pgsql

Re: [PERFORM] Postgres using more memory than it should

2008-12-03 Thread Matthew Wakeling
that the query has been stopped, the backend process is still hanging onto the RAM. Matthew -- Failure is not an option. It comes bundled with your Microsoft product. -- Ferenc Mantfeld -- Sent via pgsql-performance mailing list (pgsql-performance

Re: [PERFORM] Postgres using more memory than it should

2008-12-03 Thread Matthew Wakeling
in the release notes. This is one of the only machines we have that has not been upgraded, and it is on our schedule. Because it is running a slightly old version of RedHat Fedora, upgrading involves more horribleness than our sysadmin is willing to do on the fly with the server up. Matthew -- The email

Re: [PERFORM] Performance and IN clauses

2008-11-18 Thread Matthew Wakeling
this was to operate on groups of a bit less than a thousand values, and issue one query per group. Of course, Postgres may have improved since then, so I'll let more knowledgable people cover that for me. Matthew -- Heat is work, and work's a curse. All the heat in the universe, it's going to cool

Re: [PERFORM] Slow SQL query (14-15 seconds)

2008-11-13 Thread Matthew Wakeling
table, in one of the subqueries. Perhaps you could run each of the subqueries individually, and send us the one that takes loads of time as a simpler problem to solve. Matthew -- Those who do not understand Unix are condemned to reinvent it, poorly. -- Henry Spencer -- Sent via

Re: [PERFORM] Using index for IS NULL query

2008-11-12 Thread Matthew Wakeling
On Tue, 11 Nov 2008, Tom Lane wrote: Index is not used for is null How to fix ? Update to something newer than 8.1 (specifically, you'll need 8.3). Oooh, that's useful to know. We can get rid of all our extra nulls indexes. Thanks. Matthew -- As you approach the airport, you see a sign

Re: [PERFORM] Increasing select max(datecol) from bilkaib where datecol=date'2008-11-01' and (cr='00' or db='00') speed

2008-11-12 Thread Matthew Wakeling
Firstly, please upgrade to Postgres 8.3 if possible. On Wed, 12 Nov 2008, Andrus wrote: There are columns kuupaev date, cr char(10), db char(10) and regular indexes for all those fields. Create a single index on (cr, db, datecol). Matthew -- Those who do not understand Unix are condemned

Re: [PERFORM] Increasing select max(datecol) from bilkaib where datecol=date'2008-11-01' and (cr='00' or db='00') speed

2008-11-12 Thread Matthew Wakeling
on (db, bilkaib) then you will be able to use other values in the query too. Matthew -- Contrary to popular belief, Unix is user friendly. It just happens to be very selective about who its friends are. -- Kyle Hearn -- Sent via pgsql-performance mailing list (pgsql-performance

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-11 Thread Matthew Wakeling
for a response. So fsync should always work right, unless the system is horribly broken, on all systems going back many years. Matthew -- I'd try being be a pessimist, but it probably wouldn't work anyway. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make

Re: [PERFORM] Backup strategies

2008-10-15 Thread Matthew Wakeling
? The files may change, but it doesn't matter, because there is enough information in the xlog to correct it all. Matthew -- Of course it's your fault. Everything here's your fault - it says so in your contract.- Quark -- Sent via pgsql-performance mailing list

Re: [PERFORM] Disc space usage

2008-10-09 Thread Matthew Wakeling
are the issue. Can't do that until next time it happens, because we don't have the logs from when it did happen any more. Matthew -- Jadzia: Don't forget the 34th rule of acquisition: Peace is good for business. Quark: That's the 35th. Jadzia: Oh yes, that's right. What's the 34th again? Quark

[PERFORM] Disc space usage

2008-10-08 Thread Matthew Wakeling
this happening again? Matthew -- An ant doesn't have a lot of processing power available to it. I'm not trying to be speciesist - I wouldn't want to detract you from such a wonderful creature, but, well, there isn't a lot there, is there? -- Computer Science

Re: [PERFORM] Disc space usage

2008-10-08 Thread Matthew Wakeling
been done a few times. Yes, we do copy large databases quite often, and drop them again. The database cluster was initialised back in March. What PG version is this, anyway? Postgres 8.3.0 Matthew -- Unfortunately, university regulations probably prohibit me from eating small children

Re: [PERFORM] Disc space usage

2008-10-08 Thread Matthew Wakeling
always have some quite heavy write traffic. Is it possible that the changes that we just wrote haven't been checkpointed properly yet, resulting in some of those files being missing from the template database, and therefore the createdb to fail? Matthew -- Now, you would have thought

Re: [PERFORM] Disc space usage

2008-10-08 Thread Matthew Wakeling
? You insult me, sir! ;) No, it's Linux. Matthew -- For every complex problem, there is a solution that is simple, neat, and wrong. -- H. L. Mencken -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes

Re: [PERFORM] Disc space usage

2008-10-08 Thread Matthew Wakeling
. Unfortunately my colleagues just wrote the script so that it retries, so we don't have a decent log of the failures, which were a while back. However, I have now altered the script so that it fails with a message saying Report this to Matthew, so if it happens again I'll be able to give you some more

Re: [PERFORM] Confusing Query Performance

2008-10-01 Thread Matthew Wakeling
. Matthew -- Existence is a convenient concept to designate all of the files that an executable program can potentially process. -- Fortran77 standard -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

[PERFORM] Slow index

2008-09-25 Thread Matthew Wakeling
queries in a millisecond or less. Any ideas welcome. Also, the mailing list server doesn't seem to be able to cope with image attachments. Matthew -- import oz.wizards.Magic; if (Magic.guessRight())... -- Computer Science Lecturer -- Sent via pgsql-performance mailing list (pgsql

Re: [PERFORM] Slow index

2008-09-25 Thread Matthew Wakeling
On Thu, 25 Sep 2008, Tom Lane wrote: Matthew Wakeling [EMAIL PROTECTED] writes: Hi all. I'm having an interesting time performance-wise with a set of indexes. Any clues as to what is going on or tips to fix it would be appreciated. Are the indexed columns all the same datatype? (And which

Re: [PERFORM] Different execution plan

2008-09-24 Thread Matthew Wakeling
. Matthew -- I suppose some of you have done a Continuous Maths course. Yes? Continuous Maths? menacing stares from audience Whoah, it was like that, was it! -- Computer Science Lecturer -- Sent via pgsql-performance mailing list (pgsql-performance

Re: [PERFORM] RAID arrays and performance

2008-09-19 Thread Matthew Wakeling
be quite large. I'm considering if I can parallelise things a little though. Alternately, if you'd like to join in on testing this all out more help would certainly be welcome. How would you like me to help? Matthew -- What goes up must come down. Ask any system administrator. -- Sent via

Re: [PERFORM] RAID arrays and performance

2008-09-19 Thread Matthew Wakeling
at the moment, so it probably wouldn't make a massive improvement. It would also unfortunately require changing a lot of our code. Worth doing at some point. Matthew -- Interwoven alignment preambles are not allowed. If you have been so devious as to get this message, you will understand it, and you

Re: [PERFORM] RAID arrays and performance

2008-09-18 Thread Matthew Wakeling
of bitmap index scans at the moment, and it'd help to be able to spread them across the 16 discs in the RAID array. It's the bottleneck in our system at the moment. Matthew -- The email of the species is more deadly than the mail. -- Sent via pgsql-performance mailing list (pgsql-performance

Re: [PERFORM] Effects of setting linux block device readahead size

2008-09-15 Thread Matthew Wakeling
  |  38  | What io scheduler are you using? The anticipatory scheduler is meant to prevent this slowdown with multiple concurrent reads. Matthew -- And the lexer will say Oh look, there's a null string. Oooh, there's another. And another., and will fall over spectacularly when it realises

Re: [PERFORM] Choosing a filesystem

2008-09-11 Thread Matthew Wakeling
. It'll be pretty good. Put the OS and the WAL on the pair, and the database on the large array. However, one of the biggest things that will improve your performance (especially in OLTP) is to use a proper RAID controller with a battery-backed-up cache. Matthew -- X's book explains this very

Re: [PERFORM] too many clog files

2008-09-05 Thread Matthew Wakeling
On Fri, 5 Sep 2008, Duan Ligong wrote: Well, we could not wait so long and just moved the old clog files. Congratulations. You have probably just destroyed your database. Matthew -- Lord grant me patience, and I want it NOW! -- Sent via pgsql-performance mailing list (pgsql-performance

Re: [PERFORM] limit clause breaks query planner?

2008-09-04 Thread Matthew Wakeling
, you could behave as currently, and if the correlation is 1, then you know (from the histogram) where in the table the values are. Matthew -- X's book explains this very well, but, poor bloke, he did the Cambridge Maths Tripos... -- Computer Science Lecturer

Re: [PERFORM] limit clause breaks query planner?

2008-09-04 Thread Matthew Wakeling
a sequential scan backwards, or even a scan from the middle of the table onwards. This improvement of course only actually helps if the query has a LIMIT clause, and presumably would muck up simultaneous sequential scans. Matthew -- Picard: I was just paid a visit from Q. Riker: Q! Any idea what he's

Re: [PERFORM] slow update of index during insert/copy

2008-09-02 Thread Matthew Wakeling
. Matthew -- Contrary to popular belief, Unix is user friendly. It just happens to be very selective about who its friends are. -- Kyle Hearn -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [PERFORM] select on 22 GB table causes An I/O error occured while sending to the backend. exception

2008-08-29 Thread Matthew Wakeling
listen on it. The postmaster opens a socket for listening. Only one process can do that. When an incoming connection is received, postmaster passes that connection on to a child backend process. The child then has a socket, but it is a connected socket, not a listening socket. Matthew

Re: [PERFORM] select on 22 GB table causes An I/O error occured while sending to the backend. exception

2008-08-29 Thread Matthew Wakeling
. Are there any other modern programs that allocate lots of RAM and never use it? Matthew -- Nog: Look! They've made me into an ensign! O'Brien: I didn't know things were going so badly. Nog: Frightening, isn't it? -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org

Re: [PERFORM] select on 22 GB table causes An I/O error occured while sending to the backend. exception

2008-08-28 Thread Matthew Wakeling
just a consequence of the amount of memory needed being unpredictable. Probably the best solution is to just tell the kernel somehow to never kill the postmaster. Matthew -- Taking apron off And now you can say honestly that you have been to a lecture where you watched paint dry

Re: [PERFORM] select on 22 GB table causes An I/O error occured while sending to the backend. exception

2008-08-28 Thread Matthew Wakeling
On Thu, 28 Aug 2008, Steve Atkins wrote: Probably the best solution is to just tell the kernel somehow to never kill the postmaster. Or configure adequate swap space? Oh yes, that's very important. However, that gives the machine the opportunity to thrash. Matthew -- The early bird gets

Re: [PERFORM] select on 22 GB table causes An I/O error occured while sending to the backend. exception

2008-08-28 Thread Matthew Wakeling
just flat out broken. Exactly. a cost-benefit model would work well here. Work out how much RAM would be freed by killing a process, and use that when choosing which process to kill. Matthew -- You will see this is a 3-blackboard lecture. This is the closest you are going to get from me to high

Re: [PERFORM] select on 22 GB table causes An I/O error occured while sending to the backend. exception

2008-08-28 Thread Matthew Wakeling
of backends. Matthew -- Of course it's your fault. Everything here's your fault - it says so in your contract.- Quark -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [PERFORM] select on 22 GB table causes An I/O error occured while sending to the backend. exception

2008-08-28 Thread Matthew Dennis
On Thu, Aug 28, 2008 at 8:11 PM, Scott Marlowe [EMAIL PROTECTED]wrote: wait a min here, postgres is supposed to be able to survive a complete box failure without corrupting the database, if killing a process can corrupt the database it sounds like a major problem. Yes it is a major

Re: [PERFORM] Is there a way to SubPartition?

2008-08-27 Thread Matthew Wakeling
contains historic data that is no longer updated. Once a day, transfer the data between the partitions, and the historic data partition will not need vacuuming. Some changes to your code will be needed however. Matthew -- Vacuums are nothings. We only mention them to let them know we know they're

Re: [PERFORM] Large number of tables slow insert

2008-08-26 Thread Matthew Wakeling
. Matthew -- And why do I do it that way? Because I wish to remain sane. Um, actually, maybe I should just say I don't want to be any worse than I already am. - Computer Science Lecturer -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your

Re: [PERFORM] PostgreSQL+Hibernate Performance

2008-08-20 Thread Matthew Wakeling
. However, if you have index1, there is no point in having index2 or index5. Matthew -- Isn't Microsoft Works something of a contradiction? -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [PERFORM] Slow query with a lot of data

2008-08-19 Thread Matthew Wakeling
attname = 'domain'; It took 50 minutes to run this query for 280 users (and user IN ([280 ids])), 78000 rows were returned and stored in a table. Is this reasonable? Sounds like an awfully long time to me. Also, I think restricting it to 280 users is probably not making it any faster. Matthew

Re: [PERFORM] Slow query with a lot of data

2008-08-19 Thread Matthew Wakeling
~350days... What makes you say that? Perhaps you could post EXPLAINs of both of the queries. Matthew -- What goes up must come down. Ask any system administrator. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http

<    1   2   3   4   5   6   >