Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-10 Thread Andy Colson
Floris Bos / Maxnet wrote: The following settings differ from the defaults: -- shared_buffers=3500MB maintenance_work_mem = 128MB fsync = off synchronous_commit = off checkpoint_segments = 25 -- == Table layout == -- Table public.posts_index Column |

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-10 Thread Josh Berkus
Andy, 6) To the -hackers: I write the records and then refind them in the exact same order, would it be a better test to search for records in a more random order? would it make a difference? Would searching for some but not all make a difference? Are you on OpenSolaris? Can you give your

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-10 Thread Andy Colson
Josh Berkus wrote: Andy, 6) To the -hackers: I write the records and then refind them in the exact same order, would it be a better test to search for records in a more random order? would it make a difference? Would searching for some but not all make a difference? Are you on OpenSolaris?

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-10 Thread Josh Berkus
Floris, One more question set: what version of OpenSolaris, and what filesystem are you using? Does the OS have any non-default tuning settings? How did you install or compile PostgreSQL? -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-10 Thread Tom Lane
Floris Bos / Maxnet b...@je-eigen-domein.nl writes: Hi, Tom Lane wrote: Floris Bos / Maxnet b...@je-eigen-domein.nl writes: postg...@db:/data$ /opt/postgres/8.4-beta/bin/64/initdb -E SQL_ASCII -X /data/pg_xlog /data/db The database cluster will be initialized with locale en_US.UTF-8.

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Richard Huxton
Floris Bos / Maxnet wrote: I am having the problem that some queries are unable to find rows when using the index. When I force a sequential scan, by doing set enable_indexscan=false; set enable_bitmapscan=false;, the same queries work fine. Not a hacker myself, but I can tell you that the

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Greg Stark
And can you post an explain plan for the incorrect scan? In particular is it using a bitmap index scan or a regular index scan? Or does it happen with either? -- Greg On 9 Jun 2009, at 09:43, Richard Huxton d...@archonet.com wrote: Floris Bos / Maxnet wrote: I am having the problem

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Floris Bos / Maxnet
Hi, Richard Huxton wrote: Not a hacker myself, but I can tell you that the first question you'll be asked is can you produce a test case? If you can generate the problem from a test table+generated data that will let people figure out the problem for you. Unfortunately, I have not been able

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Tom Lane
Floris Bos / Maxnet b...@je-eigen-domein.nl writes: Richard Huxton wrote: Not a hacker myself, but I can tell you that the first question you'll be asked is can you produce a test case? If you can generate the problem from a test table+generated data that will let people figure out the

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Simon Riggs
On Tue, 2009-06-09 at 13:40 +0200, Floris Bos / Maxnet wrote: fsync = off That's a bad plan if you care about your database. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Florian Weimer
* Simon Riggs: On Tue, 2009-06-09 at 13:40 +0200, Floris Bos / Maxnet wrote: fsync = off That's a bad plan if you care about your database. It shouldn't introduce this type of corruption, though. -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Floris Bos / Maxnet
Hi, Simon Riggs wrote: On Tue, 2009-06-09 at 13:40 +0200, Floris Bos / Maxnet wrote: fsync = off That's a bad plan if you care about your database. I am aware of the risk of dataloss in case of power failure, etc. However fsync=on is simply too slow for my purpose, and it concerns data

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Greg Stark
Well sure it could -- once. It wouldn't be reproducible in a freshly rebuilt index unless he's crashing his machine every time. -- Greg On 9 Jun 2009, at 17:12, Florian Weimer fwei...@bfk.de wrote: * Simon Riggs: On Tue, 2009-06-09 at 13:40 +0200, Floris Bos / Maxnet wrote: fsync =

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Josh Berkus
Floris, The website this setup is for has been running various previous versions of PostgreSQL with fsync=off since 2005. So I still expect it to work. You've been lucky, that's all. Our documentation has been clear, back to version 7.0, that turning fsync=off carries the risk that you

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Florian Weimer
* Josh Berkus: Our documentation has been clear, back to version 7.0, that turning fsync=off carries the risk that you will have to recreate your entire database in the event of unexpected shutdown. That's not new. The documentation does not say this. Instead, there's the following rather

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Alvaro Herrera
Florian Weimer wrote: * Josh Berkus: Our documentation has been clear, back to version 7.0, that turning fsync=off carries the risk that you will have to recreate your entire database in the event of unexpected shutdown. That's not new. The documentation does not say this. Instead,

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Kevin Grittner
Alvaro Herrera alvhe...@commandprompt.com wrote: Yeah, AFAICT the writes are handed off to the operating system (just not synced), so if it flushes its caches sanely at all there shouldn't be a problem. I would certainly *hope* that's the case. We sometimes use fsync=off for conversions,

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Josh Berkus
Alvaro, Kevin, Yeah, AFAICT the writes are handed off to the operating system (just not synced), so if it flushes its caches sanely at all there shouldn't be a problem. I would certainly *hope* that's the case. We sometimes use fsync=off for conversions, where we plan to just start over if

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: It doesn't. But what I don't trust, and the *first* place I'd look for problems, is whether the OS flushes *all* dirty buffers to disk in the event the application gets killed. Why wouldn't you trust it? The sort of thing you seem to be thinking about

Re: [HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-09 Thread Floris Bos / Maxnet
Hi, Josh Berkus wrote: It doesn't. But what I don't trust, and the *first* place I'd look for problems, is whether the OS flushes *all* dirty buffers to disk in the event the application gets killed. That's why I want more information on Floris' case. Was 8.4 killed or shut down with -m

[HACKERS] Multicolumn index corruption on 8.4 beta 2

2009-06-08 Thread Floris Bos / Maxnet
Hi, I pgdump'ed a 8.3.7 database and loaded the dump to a different server running PostgreSQL 8.4 beta 2 (compiled from source) under Opensolaris. One of the tables has about 6 million records, and a Btree index that spans 3 columns. I am having the problem that some queries are unable to