Re: [HACKERS] [PATCHES] update i386 spinlock for hyperthreading

2003-12-30 Thread Jan Wieck
Manfred Spraul wrote: > Bruce Momjian wrote: > >>>Anyone see an attack path here? >>> >>> >> >>Should we have one lock per hash bucket rather than one for the entire >>hash? >> >> > That's the simple part. The problem is the aging strategy: we need a > strategy that doesn't rely on a global

Re: [HACKERS] [PATCHES] update i386 spinlock for hyperthreading

2003-12-29 Thread Jan Wieck
Tom Lane wrote: Bruce Momjian <[EMAIL PROTECTED]> writes: Manfred Spraul wrote: I remember there were patches that tried other algorithms instead of the simple LRU for the buffer manager. Has anyone tried to change the locking of the buffer manager? CVS HEAD already has an Adap

Re: [HACKERS] IEEE 754

2003-12-29 Thread Jan Wieck
Sai Hertz And Control Systems wrote: Dear all , I would like to share my concerns about the IEEE 754 specification and floating point handling by PostgreSQL . Also I would like to learn how professional users of PostgreSQL work with rounding of monetary terms . For all monetary values the P

Re: [HACKERS] PostgreSQL port to pure Java?

2003-12-25 Thread Jan Wieck
Ivelin Ivanov wrote: That was uncalled for. Statements like this do not make the Postgres community any healthier. You don't have any benefit of pushing back Java users. On the other hand, can you imagine the reaction of the Java camp to an idea like "why not rewrite JBoss in Tcl and PL/Tcl so th

Re: [HACKERS] Bug in new buffer freelist code

2003-12-23 Thread Jan Wieck
Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: It seems to me that buffers that are thrown away via StrategyInvalidateBuffer() do not get their relnode and blocknum cleaned out. Mmmm. They definitely should be; if you look at the prior version of buf_table.c, BufTableDelete di

Re: [HACKERS] Bug in new buffer freelist code

2003-12-23 Thread Jan Wieck
Tom Lane wrote: BTW, I just managed to reproduce the hang, after a whole afternoon of trying ... only it was with a non-debug build. Sigh. Anyway, it seems my HP machine has a significantly higher probability of showing the problem than my Linux machine --- I have been unable to see the problem

Re: [HACKERS] Bug in new buffer freelist code

2003-12-23 Thread Jan Wieck
Tom Lane wrote: I just had the parallel regression tests hang up due to what appears to be a bug in the new ARC code. The CLUSTER test gets into an infinite loop trying to do "CLUSTER clstr_1;". The loop is in StrategyInvalidateBuffer's check that the buffer is already in the freelist; it isn't,

Re: [HACKERS] Resurrecting pg_upgrade

2003-12-16 Thread Jan Wieck
Tom Lane wrote: "Marc G. Fournier" <[EMAIL PROTECTED]> writes: Then again, in the case of pg_upgrade, wouldn't just disabling access from anywhere except localhost prevent others from getting in? Not if your normal operating mode includes connections from clients running locally. I really don't s

Re: [HACKERS] pljava revisited

2003-12-10 Thread Jan Wieck
Andrew Rawnsley wrote: Other pl* (perl, python, tcl) languages have vanilla C glue code. Might be better to stick to this. If you aren't using advanced C++ features that shouldn't be too hard - well structured C can be just as readable as well structured C++. At the very lowest level, about the

Re: [HACKERS] ERROR: Index pg_toast_8443892_index is not a btree

2003-12-10 Thread Jan Wieck
Tom Lane wrote: strk <[EMAIL PROTECTED]> writes: The question now is: what does that message mean ? It means that the "magic number" that should be on the first page of the btree index isn't right. We can deduce that something has clobbered the first page of the index, but guessing what and how r

Re: [HACKERS] ERROR: Index pg_toast_8443892_index is not a btree

2003-12-09 Thread Jan Wieck
Christopher Kings-Lynne wrote: I couldn't agree more. Look at this very instance. He now found the right reindex command and the corrupted file is gone. We don't have the slightest clue what happened to that file. Was it truncated? Did some other process scribble around in the shared memory? Ho

Re: [HACKERS] ERROR: Index pg_toast_8443892_index is not a btree

2003-12-09 Thread Jan Wieck
Tom Lane wrote: Christopher Kings-Lynne <[EMAIL PROTECTED]> writes: Is there anything stopping us going through the code and finding all ereports that can be fixed by a REINDEX, and issue a HINT with all of them saying that they should REINDEX the broken index? How would you know which ones corre

Re: [HACKERS] Minor (very) feature request...

2003-12-08 Thread Jan Wieck
Steve Wampler wrote: Would it be (is it?) possible to add timestamp to the log messages put out by postgresql? I've got several databases running in an environment where users have this annoying habit of coming up to me with ("Oh yes, three days ago around 4pm our instrument had trouble writing t

Re: [HACKERS] CVS HEAD compile failure

2003-12-08 Thread Jan Wieck
Tom Lane wrote: Kurt Roeckx <[EMAIL PROTECTED]> writes: I can compile current cvs with gcc 2.95.3, openssl 0.9.7b and zlib 1.2.1. "current CVS" meaning "since I fixed the include order" ? Good question. Using my cvsup tree here, which I did sup today already. So what -D would trigger the failure?

Re: [HACKERS] CVS HEAD compile failure

2003-12-08 Thread Jan Wieck
Kurt Roeckx wrote: On Mon, Dec 08, 2003 at 01:27:35PM -0500, Tom Lane wrote: Andrew Dunstan <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> I've been able to reproduce this on one of my machines, and it's nasty. > In that case I'm confused about why this code compiles on my machine: What compile

Re: [HACKERS] ERROR: Index pg_toast_8443892_index is not a btree

2003-12-08 Thread Jan Wieck
strk wrote: JanWieck wrote: strk wrote: > I get the following error when vacuuming a db or inserting > a big value in a column of a toastable datatype (GEOMETRY). > > ERROR: Index pg_toast_8443892_index is not a btree > > My last action has been killing a psql that was getting > mad about rec

Re: [HACKERS] ERROR: Index pg_toast_8443892_index is not a btree

2003-12-08 Thread Jan Wieck
strk wrote: I get the following error when vacuuming a db or inserting a big value in a column of a toastable datatype (GEOMETRY). ERROR: Index pg_toast_8443892_index is not a btree My last action has been killing a psql that was getting mad about receiving too much input and beeping as hell (r

Re: [HACKERS] pg_restore and create FK without verification check

2003-11-28 Thread Jan Wieck
Oli Sennhauser wrote: People might be more interested in debating this topic with you if we hadn't discussed it at length just a couple months back. There wasn't consensus then that we had to offer an escape hatch, and you've not offered any argument that wasn't made before. I'm simply presenti

Re: [HACKERS] pg_restore and create FK without verification check

2003-11-27 Thread Jan Wieck
ow wrote: --- Tom Lane <[EMAIL PROTECTED]> wrote: People might be more interested in debating this topic with you if we hadn't discussed it at length just a couple months back. There wasn't consensus then that we had to offer an escape hatch, and you've not offered any argument that wasn't made b

Re: [HACKERS] Release cycle length

2003-11-21 Thread Jan Wieck
Alvaro Herrera wrote: On Fri, Nov 21, 2003 at 09:38:50AM +0800, Christopher Kings-Lynne wrote: >Yeah, I think the main issue in all this is that for real production >sites, upgrading Postgres across major releases is *painful*. We have >to find a solution to that before it makes sense to speed up

Re: [HACKERS] Background writer committed

2003-11-20 Thread Jan Wieck
Shridhar Daithankar wrote: Jan Wieck wrote: I committed the first part of the background writer process. We had a consensus on attempting to avoid write() calls from regular backends, but did no come to any conclusions what to do to force the kernel to actually do some IO. Consequently, this

[HACKERS] Background writer committed

2003-11-19 Thread Jan Wieck
I committed the first part of the background writer process. We had a consensus on attempting to avoid write() calls from regular backends, but did no come to any conclusions what to do to force the kernel to actually do some IO. Consequently, this patch is a separate process launched by postma

Re: [HACKERS] Release cycle length

2003-11-19 Thread Jan Wieck
Marc G. Fournier wrote: On Tue, 18 Nov 2003, Peter Eisentraut wrote: The time from release 7.3 to release 7.4 was 355 days, an all-time high. We really need to shorten that. We already have a number of significant improvements in 7.5 now, and several good ones coming up in the next few weeks. W

Re: [HACKERS] cvs head? initdb?

2003-11-16 Thread Jan Wieck
Robert Treat wrote: but how do you test this if you cant run them both against each other to compare? (initally running vs 7.4 does tell you something, but even now, 7.5 improved cross datatype index improvments could skew the results of any comparisons) Right. But with the current two candidat

Re: [HACKERS] cvs head? initdb?

2003-11-16 Thread Jan Wieck
Bruce Momjian wrote: I figured it should begin with debug_ or log_, maybe: debug_shared_buffers = 10 # seconds If it's just that and since nobody else seemed to care ... changed. Jan -- #==# # It's easier to get forgiveness f

Re: [HACKERS] cvs head? initdb?

2003-11-15 Thread Jan Wieck
Jan --- Robert Treat wrote: On Friday 14 November 2003 14:23, Neil Conway wrote: > Jan Wieck <[EMAIL PROTECTED]> writes: > > Robert Treat wrote: > >> people would always want to have those choices (especially for d

Re: [HACKERS] cvs head? initdb?

2003-11-14 Thread Jan Wieck
Bruce Momjian wrote: Jan Wieck wrote: >> Yeah, there was a problem with *extreme* sharing ... the code tried to >> use the same buffer for multiple disk blocks at the same time, and >> somehow the backends did not agree on the correct content. But it's >> fixed

Re: [HACKERS] cvs head? initdb?

2003-11-14 Thread Jan Wieck
Robert Treat wrote: On Fri, 2003-11-14 at 10:32, Jan Wieck wrote: Bruce Momjian wrote: > Jan Wieck wrote: >> Christopher Browne wrote: >> >> > [EMAIL PROTECTED] (elein) writes: >> >> What is the status of CVS head? Isn't it in sync with 7.4.RC2? I >&

Re: [HACKERS] Background writer process

2003-11-14 Thread Jan Wieck
Shridhar Daithankar wrote: On Friday 14 November 2003 03:05, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the algorithm how much and how often to flush can be significantly improved. But after all, this does not change

Re: [HACKERS] cvs head? initdb?

2003-11-14 Thread Jan Wieck
Bruce Momjian wrote: Jan Wieck wrote: Christopher Browne wrote: > [EMAIL PROTECTED] (elein) writes: >> What is the status of CVS head? Isn't it in sync with 7.4.RC2? I >> just upgraded from CVS and rebuilt clean and initdb now gives this >> lovely informative initdb

Re: [HACKERS] cvs head? initdb?

2003-11-13 Thread Jan Wieck
Christopher Browne wrote: [EMAIL PROTECTED] (elein) writes: What is the status of CVS head? Isn't it in sync with 7.4.RC2? I just upgraded from CVS and rebuilt clean and initdb now gives this lovely informative initdb failed message. No, I believe that they have started applying 7.5 patches, not

Re: [HACKERS] Background writer process

2003-11-13 Thread Jan Wieck
Kurt Roeckx wrote: On Thu, Nov 13, 2003 at 05:39:32PM -0500, Bruce Momjian wrote: Jan Wieck wrote: > Bruce Momjian wrote: > > He found that write() itself didn't encourage the kernel to write the > > buffers to disk fast enough. I think the final solution will be to use

Re: [HACKERS] Background writer process

2003-11-13 Thread Jan Wieck
Bruce Momjian wrote: Kurt Roeckx wrote: On Thu, Nov 13, 2003 at 04:35:31PM -0500, Jan Wieck wrote: > For sure the sync() needs to be replaced by the discussed fsync() of > recently written files. And I think the algorithm how much and how often > to flush can be significantly impr

[HACKERS] Background writer process

2003-11-13 Thread Jan Wieck
The attached diff is another attempt for distributing the write IO. It is a separate background process much like the checkpointer. It's purpose is to keep the number of dirty blocks in the buffer cache at a reasonable level and try that the buffers returned by the strategy for replacement are

Re: [HACKERS] ARC buffer strategy committed

2003-11-13 Thread Jan Wieck
Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: Yeah, adding a buffer multiple times to the list of unused buffers ensures that it later on gets used for multiple contents simultaneously. Hm. Looking at the patch last night, I was wondering why you had removed all the guard logi

Re: [HACKERS] Proposal for a cascaded master-slave replication system

2003-11-13 Thread Jan Wieck
Bruce Momjian wrote: Hans-J?rgen Sch?nig wrote: Meanwhile we seem to be in a situation where PostgreSQL is rather competing against Oracle than against MySQL. In our case there are more people asking for Oracle -> Pg migration than for MySQL -> Pg. MySQL does not seem to be the great enemy beca

Re: [HACKERS] ARC buffer strategy committed

2003-11-13 Thread Jan Wieck
Jan Wieck wrote: Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: That's strange, if I reverse apply my patch I have buffer leak's and all kinds of crap. Can't even initdb. Who else applied patches tonight? FWIW, I can initdb --- that seems fine --- but the regressi

Re: [HACKERS] ARC buffer strategy committed

2003-11-12 Thread Jan Wieck
Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: That's strange, if I reverse apply my patch I have buffer leak's and all kinds of crap. Can't even initdb. Who else applied patches tonight? FWIW, I can initdb --- that seems fine --- but the regression tests spew lo

Re: [HACKERS] ARC buffer strategy committed

2003-11-12 Thread Jan Wieck
Jan Wieck wrote: Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: I have just committed the implementation of ARC into the 7.5devel tree. I'm seeing a whole bunch of regression test failures that weren't there half an hour ago ... regards, tom lane Oh ... hmmm ... w

Re: [HACKERS] ARC buffer strategy committed

2003-11-12 Thread Jan Wieck
Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: I have just committed the implementation of ARC into the 7.5devel tree. I'm seeing a whole bunch of regression test failures that weren't there half an hour ago ... regards, tom lane Oh ... hmmm ...

[HACKERS] ARC buffer strategy committed

2003-11-12 Thread Jan Wieck
I have just committed the implementation of ARC into the 7.5devel tree. This doesn't contain the vacuum page delay, or the test implementation of the background writer based on the checkpoint process. These two need some more discussion and bouncing around. Jan -- #

Re: [HACKERS] Proposal for a cascaded master-slave replication system

2003-11-12 Thread Jan Wieck
omething I allways wanted to have. I think your right on by not thinking smaller!! Thanks Jan Jordan Henderson On Wednesday 12 November 2003 10:45, Jan Wieck wrote: Hans-Jürgen Schönig wrote: > Jan, > > First of all we really appreciate that this is going to be an Open > Source project

Re: [HACKERS] Proposal for a cascaded master-slave replication system

2003-11-12 Thread Jan Wieck
Hans-Jürgen Schönig wrote: Jan, First of all we really appreciate that this is going to be an Open Source project. There is something I wanted to add from a marketing point of view: I have done many public talks in the 2 years or so. There is one question people keep asking me: "How about the

Re: [HACKERS] 7.4 hot backup capabilities?

2003-11-12 Thread Jan Wieck
Austin Gonyou wrote: On Wed, 2003-11-12 at 01:00, Shridhar Daithankar wrote: Austin Gonyou wrote: > What facilities are/will be available for hot(online) backups with the > 7.4 release? PITR, something else? TIA. pg_dump? Did you mean hot failover or hot backup? Postgresql does hot backup for a

Re: [HACKERS] [GENERAL] Proposal for a cascaded master-slave replication system

2003-11-11 Thread Jan Wieck
Joe Conway wrote: Jan Wieck wrote: http://developer.postgresql.org/~wieck/slony1.html Very interesting read. Nice work! We want to build this system as a community project. The plan was from the beginning to release the product under the BSD license. And we think it is best to start it as such

Re: [HACKERS] bugzilla (Was: What do you want me to do?)

2003-11-11 Thread Jan Wieck
Marc G. Fournier wrote: On Tue, 11 Nov 2003, Shridhar Daithankar wrote: --- Hello, My name is <> and I work with postgresql group. Recently we/postgresql advocay/general group have decided to form a group of people who would help other projects w.r.t postgresql. So feel free to ask me que

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-11 Thread Jan Wieck
Shridhar Daithankar wrote: On Tuesday 11 November 2003 18:55, Jan Wieck wrote: And how does a newly mmap'ed segment propagate into a running backend? It wouldn't. Just like we allocate fixed amount of shared memory at startup now, we would do same for mmaped segments. Alloca

[HACKERS] Proposal for a cascaded master-slave replication system

2003-11-11 Thread Jan Wieck
Dear community, for some reason the post I sent yesterday night still did not show up on the mailing lists. I have set up some links on the developers side under http://developer.postgresql.org/~wieck/slony1.html The concept will be the base for some of my work as a Software Engineer here at Af

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-11 Thread Jan Wieck
Shridhar Daithankar wrote: On Tuesday 11 November 2003 00:50, Neil Conway wrote: Jan Wieck <[EMAIL PROTECTED]> writes: > We can't resize shared memory because we allocate the whole thing in > one big hump - which causes the shmmax problem BTW. If we allocate > that in chunks of

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
Andrew Sullivan wrote: On Sun, Nov 09, 2003 at 08:54:25PM -0800, Joe Conway wrote: two servers, mounted to the same data volume, and some kind of coordination between the writer processes. Anyone know if this is similar to how Oracle handles RAC? It is similar, yes, but there's some mighty power

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
Bruce Momjian wrote: Jan Wieck wrote: >> > If the background cleaner has to not just write() but write/fsync or >> > write/O_SYNC, it isn't going to be able to clean them fast enough. It >> > creates a bottleneck where we didn't have one before. >> >

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
Bruce Momjian wrote: Jan Wieck wrote: Zeugswetter Andreas SB SD wrote: >> > One problem with O_SYNC would be, that the OS does not group writes any >> > more. So the code would need to eighter do it's own sorting and grouping >> > (256k) or use aio, or you won&

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
Bruce Momjian wrote: Jan Wieck wrote: Bruce Momjian wrote: > Jan Wieck wrote: >> Bruce Momjian wrote: >> >> > Now, O_SYNC is going to force every write to the disk. If we have a >> > transaction that has to write lots of buffers (has to write them to >&g

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
Zeugswetter Andreas SB SD wrote: > One problem with O_SYNC would be, that the OS does not group writes any > more. So the code would need to eighter do it's own sorting and grouping > (256k) or use aio, or you won't be able to get the maximum out of the disks. Or just run multiple writer process

Re: [HACKERS] Question for the developers.

2003-11-10 Thread Jan Wieck
[EMAIL PROTECTED] wrote: Hi, I am Suchindra Katageri and am working as a Software Engineer at Linuxlabs, Atlanta, GA. I am presently working on developing libraries to make postgresql run on a cluster. I was wondering if it was possible to force Database writes to stable storage, without m

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
Tom Lane wrote: Bruce Momjian <[EMAIL PROTECTED]> writes: Now, if we are sure that writes will happen only in the checkpoint process, O_SYNC would be OK, I guess, but will we ever be sure of that? This is a performance issue, not a correctness issue. It's okay for backends to wait for writes as l

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
Bruce Momjian wrote: Jan Wieck wrote: Bruce Momjian wrote: > I would be interested to know if you have the background write process > writing old dirty buffers to kernel buffers continually if the sync() > load is diminished. What this does is to push more dirty buffers into > the

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
Bruce Momjian wrote: Jan Wieck wrote: Bruce Momjian wrote: > Now, O_SYNC is going to force every write to the disk. If we have a > transaction that has to write lots of buffers (has to write them to > reuse the shared buffer) So make the background writer/checkpointer keeping the LRU h

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
Bruce Momjian wrote: Tom Lane wrote: Andrew Sullivan <[EMAIL PROTECTED]> writes: > On Sun, Nov 02, 2003 at 01:00:35PM -0500, Tom Lane wrote: >> real traction we'd have to go back to the "take over most of RAM for >> shared buffers" approach, which we already know to have a bunch of >> severe disad

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
before it got released. Did any PostgreSQL developer other than you ever look at the SRA code? Jan Bruce Momjian wrote: scott.marlowe wrote: On Tue, 4 Nov 2003, Tom Lane wrote: > Jan Wieck <[EMAIL PROTECTED]> writes: > > What still needs to be addressed is the IO storm cause by

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
Bruce Momjian wrote: Now, O_SYNC is going to force every write to the disk. If we have a transaction that has to write lots of buffers (has to write them to reuse the shared buffer) So make the background writer/checkpointer keeping the LRU head clean. I explained that 3 times now. Jan -- #===

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-10 Thread Jan Wieck
ds modify? Jan ------- Jan Wieck wrote: Tom Lane wrote: > Jan Wieck <[EMAIL PROTECTED]> writes: > >> How I can see the background writer operating is that he's keeping the >> buffers in the order of the LRU chain(s) clean, because those are the >> buffers that

Re: [HACKERS] Performance features the 4th

2003-11-09 Thread Jan Wieck
Matthew T. O'Connor wrote: - Original Message - From: "Jan Wieck" <[EMAIL PROTECTED]> Tom Lane wrote: > Gaetano and a couple of other people did experiments that seemed to show > it was useful. I think we'd want to change the shape of the knob per > l

Re: [HACKERS] Performance features the 4th

2003-11-09 Thread Jan Wieck
scott.marlowe wrote: On Fri, 7 Nov 2003, Matthew T. O'Connor wrote: - Original Message - From: "Jan Wieck" <[EMAIL PROTECTED]> > Tom Lane wrote: > > Gaetano and a couple of other people did experiments that seemed to show > > it was useful. I think

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-09 Thread Jan Wieck
Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: How I can see the background writer operating is that he's keeping the buffers in the order of the LRU chain(s) clean, because those are the buffers that most likely get replaced soon. In my experimental ARC code it would trav

Re: [HACKERS] stats collector causes shared-memory-block leakage

2003-11-07 Thread Jan Wieck
Tom Lane wrote: While investigating Scott Goodwin's recent report of trouble on Mac OS X, I have realized that we have an unpleasant little misbehavior in our last several releases. After a backend crash, the postmaster will attempt to recycle (delete and recreate) the old shared memory segment.

Re: [HACKERS] Performance features the 4th

2003-11-07 Thread Jan Wieck
Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: However, I have not seen much evidence yet that the vacuum delay alone does that much. Gaetano and a couple of other people did experiments that seemed to show it was useful. I think we'd want to change the shape of the kno

Re: [HACKERS] Experimental ARC implementation

2003-11-07 Thread Jan Wieck
Bruce Momjian wrote: Jan Wieck wrote: What doing frequent fdatasync/fsync during a constant ongoing checkpoint will cause is to significantly lower the physical write storm happening at the sync(), which is causing huge problems right now. I don't see that frankly because sync() is sy

Re: [HACKERS] Performance features the 4th

2003-11-07 Thread Jan Wieck
Christopher Browne wrote: A long time ago, in a galaxy far, far away, [EMAIL PROTECTED] (Bruce Momjian) wrote: Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: > As a matter of fact, people who have performance problems are likely to > be the same who have upgrade problems. An

Re: [HACKERS] Experimental ARC implementation

2003-11-07 Thread Jan Wieck
Bruce Momjian wrote: Jan Wieck wrote: If the system is write-bound, the checkpointer will find that many dirty blocks that he has no time to nap and will burst them out as fast as possible anyway. Well, at least that's the theory. PostgreSQL with the non-overwriting storage concept can

Re: [HACKERS] Experimental ARC implementation

2003-11-07 Thread Jan Wieck
Bruce Momjian wrote: Greg Stark wrote: Bruce Momjian <[EMAIL PROTECTED]> writes: > Have you considered having the background writer check the pages it is > about to write to see if they can be added to the FSM, thereby reducing > the need for vacuum? Seems we would need to add a statistics param

Re: [HACKERS] Changes to Contributor List

2003-11-05 Thread Jan Wieck
i.e.: A) What contributors are missing from the list? B) What contributors are listed under Major Developers who haven't contributed any code since 7.1.0? C) Who needs their e-mail address updated? D) Who needs their description updated? (Text, please ... I won't write it) J

Re: [HACKERS] Schema boggle...

2003-11-05 Thread Jan Wieck
Chris Bowlby wrote: Hi All, I work with Marc Fournier, for those who don't know, and have been working extensively with the schemas feature for Hub.Org's new Account Management package. Each client's data is stored in a seperate schema set asside just for them (though they will never have direc

Re: [HACKERS] Performance features the 4th

2003-11-05 Thread Jan Wieck
Andrew Dunstan wrote: Jan Wieck wrote: How portable is getrusage()? Could the postmaster issue that frequently for RUSAGE_CHILDREN and leave the result somewhere in the shared memory for whoever is concerned? SVr4, BSD4.3, SUS2 and POSIX1003.1, I believe. I also believe there is a M$ dll

Re: [HACKERS] Performance features the 4th

2003-11-05 Thread Jan Wieck
Neil Conway wrote: Jan Wieck <[EMAIL PROTECTED]> writes: This patch contains the "still not yet ready" performance improvements discussed over the couple last days. Cool stuff! The buffer replacement strategy is a slightly modified version of ARC. BTW Jan, I got your message abo

Re: [HACKERS] Performance features the 4th

2003-11-05 Thread Jan Wieck
Manfred Spraul wrote: Jan Wieck wrote: _Vacuum page delay_: Tom Lane's napping during vacuums with another tuning option. I replaced the usleep() call with a PG_DELAY(msec) macro in miscadmin.h, which does use select(2) instead. That should address the possible portability problems.

Re: [HACKERS] Experimental ARC implementation

2003-11-05 Thread Jan Wieck
Zeugswetter Andreas SB SD wrote: > Why not use the checkpointer itself inbetween checkpoints ? > use a min and a max dirty setting like Informix. Start writing > when more than max are dirty stop when at min. This avoids writing > single pages (which is slow, since it cannot be grouped together > b

[HACKERS] Performance features the 4th

2003-11-05 Thread Jan Wieck
I've just uploaded http://developer.postgresql.org/~wieck/all_performance.v4.74.diff.gz This patch contains the "still not yet ready" performance improvements discussed over the couple last days. _Shared buffer replacement_: The buffer replacement strategy is a slightly modified version of ARC

Re: [HACKERS] Experimental ARC implementation

2003-11-05 Thread Jan Wieck
Zeugswetter Andreas SB SD wrote: My plan is to create another background process very similar to the checkpointer and to let that run forever basically looping over that BufferSync() with a bool telling that it's the bg_writer. Why not use the checkpointer itself inbetween checkpoints ? use a mi

Re: [HACKERS] Experimental ARC implementation

2003-11-04 Thread Jan Wieck
Jan Wieck wrote: Jan Wieck wrote: Jan Wieck wrote: I will follow up shortly with an approach that integrates Tom's delay mechanism plus my first READ_BY_VACUUM hack into one combined experiement. Okay, the attached patch contains the 3 already discussed and one additional change. O

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-04 Thread Jan Wieck
Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: Tom Lane wrote: I have never been happy with the fact that we use sync(2) at all. Sure does it do too much. But together with the other layer of indirection, the virtual file descriptor pool, what is the exact guaranteed behavi

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-04 Thread Jan Wieck
Tom Lane wrote: Jan Wieck <[EMAIL PROTECTED]> writes: What still needs to be addressed is the IO storm cause by checkpoints. I see it much relaxed when stretching out the BufferSync() over most of the time until the next one should occur. But the kernel sync at it's end still pushes

Re: [HACKERS] 7.4RC1 tag'd, branched and bundled ...

2003-11-04 Thread Jan Wieck
Gaetano Mendola wrote: Tom Lane wrote: "Marc G. Fournier" <[EMAIL PROTECTED]> writes: On Tue, 4 Nov 2003, Gaetano Mendola wrote: Is in the last Tom's patch about Vacuum sleep between pages ? that won't be in v7.4, to the best of my knowledge ... Definitely not. It's a very experimental pat

Re: [HACKERS] bufmgr code question

2003-11-04 Thread Jan Wieck
Neil Conway wrote: In the BufferDesc struct, there seem to be two ways to mark a buffer page as dirty: setting the BM_DIRTY bit mask in the 'flags' field of the struct, and setting the 'cntxDirty' field to true. What is the difference between these two indications of a page's dirtiness? I don't see

Re: [HACKERS] Experimental ARC implementation

2003-11-04 Thread Jan Wieck
Jan Wieck wrote: Jan Wieck wrote: I will follow up shortly with an approach that integrates Tom's delay mechanism plus my first READ_BY_VACUUM hack into one combined experiement. Okay, the attached patch contains the 3 already discussed and one additional change. Ooopsy the B1/B2

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-03 Thread Jan Wieck
Ang Chin Han wrote: Christopher Browne wrote: Centuries ago, Nostradamus foresaw when "Stephen" <[EMAIL PROTECTED]> would write: As it turns out. With vacuum_page_delay = 0, VACUUM took 1m20s (80s) to complete, with vacuum_page_delay = 1 and vacuum_page_delay = 10, both VACUUMs completed in 18m3s

Re: [HACKERS] 7.4RC1 failed to build on Linux

2003-11-03 Thread Jan Wieck
Marc G. Fournier wrote: what version of tcl do you have installed? Looks like 8.0. As per recent discussion we are actually require 8.1 exactly because of those missing functions in all versions prior to that. Jan On Tue, 4 Nov 2003, Kenji Sugita wrote: Failed to build on Linux. $ uname -m

Re: [HACKERS] Experimental ARC implementation

2003-11-03 Thread Jan Wieck
Jan Wieck wrote: I will follow up shortly with an approach that integrates Tom's delay mechanism plus my first READ_BY_VACUUM hack into one combined experiement. Okay, the attached patch contains the 3 already discussed and one additional change. I also made a few changes. 1) ARC policy

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-03 Thread Jan Wieck
Christopher Browne wrote: The world rejoiced as [EMAIL PROTECTED] (Hannu Krosing) wrote: Christopher Browne kirjutas E, 03.11.2003 kell 02:15: Well, actually, the case where it _would_ be troublesome would be where there was a combination of huge tables needing vacuuming and smaller ones that are

[HACKERS] Experimental ARC implementation

2003-11-03 Thread Jan Wieck
Attached is a first trial implementation of the Adaptive Replacement Cache (ARC). The patch is against CVS HEAD of 7.4. The algorithm is based on what's described in these papers: http://www.almaden.ibm.com/StorageSystems/autonomic_storage/ARC/arcfast.pdf http://www.almaden.ibm.com/StorageSystem

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-11-01 Thread Jan Wieck
Stephen wrote: I tried the Tom Lane's patch on PostgreSQL 7.4-BETA-5 and it works fantastically! Running a few short tests show a significant improvement in responsiveness on my RedHat 9 Linux 2.4-20-8 (IDE 120GB 7200RPM UDMA5). I am currently looking at implementing ARC as a replacement strategy.

Re: [HACKERS] Experimental patch for inter-page delay in VACUUM

2003-10-31 Thread Jan Wieck
Bruce Momjian wrote: Tom Lane wrote: "Matthew T. O'Connor" <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> 2. I only bothered to insert delays in the processing loops of plain >> VACUUM and btree index cleanup. VACUUM FULL and cleanup of non-btree >> indexes aren't done yet. >> > I thought we d

Re: [HACKERS] 7.4RC1 planned for Monday

2003-10-31 Thread Jan Wieck
Stephan Szabo wrote: On Thu, 30 Oct 2003, Tom Lane wrote: Stephan Szabo <[EMAIL PROTECTED]> writes: > On Thu, 30 Oct 2003, Tom Lane wrote: >> rule/foreign key interaction reported by Michele Bendazzoli > In the interests of disclosure, if the case in question for the rule > fails, almost certainly

Re: [HACKERS] Rule regression failure freebsd?

2003-10-31 Thread Jan Wieck
Jan Wieck wrote: Tom Lane wrote: Christopher Kings-Lynne <[EMAIL PROTECTED]> writes: See attached regression.diffs. Looks like Jan forgot to update this expected file to match his changes. regards, tom lane Not exactly, I didn't run the parallel regression test and thus missed tha

Re: [HACKERS] Rule regression failure freebsd?

2003-10-31 Thread Jan Wieck
Tom Lane wrote: Christopher Kings-Lynne <[EMAIL PROTECTED]> writes: See attached regression.diffs. Looks like Jan forgot to update this expected file to match his changes. regards, tom lane Not exactly, I didn't run the parallel regression test and thus missed that rules and foreign_key are i

Re: [HACKERS] 7.4RC1 planned for Monday

2003-10-30 Thread Jan Wieck
Joshua D. Drake wrote: Sooner or later you have to say "this release is done, let's ship it". It's way too late to go back into invention mode for 7.4. I agree with the argument. It is just that the Vacuum one... well is very tempting. Since improving the buffer cache policy will not change an

Re: [HACKERS] 7.4 and 7.3.5 showstopper

2003-10-30 Thread Jan Wieck
Jan Wieck wrote: Confirmed, that's a bug - pgsql-hackers CC'd and scipt for full reproduction attached. Assumptions where correct, bug fixed in REL7_3_STABLE and HEAD. I also added a slightly modified version of the script that reproduced the bug to the foreign_key regression test.

Re: [HACKERS] CREATE TYPE for case insensitive text and varchar

2003-10-30 Thread Jan Wieck
Tom Lane wrote: Daniel Migowski <[EMAIL PROTECTED]> writes: I miss the possibility to code something like a userdifined varchar(n). You're out of luck on that. The data types that can have precision parameters attached to them are hard-wired into the parser. Maybe you don't need to invent a whole

Re: [HACKERS] pg_user

2003-10-30 Thread Jan Wieck
. It never striked me as a good idea in general that we only have the one, shared pg_shadow catalog and all databases share all users. So I think what I try to say is ... back to the drawing board, because your initial solution is not acceptable. Jan On Mon, 27 Oct 2003, Jan Wieck wrote: ivan wr

[HACKERS] 7.4 and 7.3.5 showstopper (was: Re: [SQL] Bug in Rule+Foreing key constrain?)

2003-10-30 Thread Jan Wieck
ORE INSERT trigger in PL/pgSQL that tries to UPDATE the row and if GET DIAGNOSTICS tells it it succeeded, returns NULL to suppress the INSERT. That should work around the bug for the time being. Jan Michele Bendazzoli wrote: On Thu, 2003-10-30 at 18:29, Jan Wieck wrote: Not entirely. On which ta

<    1   2   3   4   5   6   7   8   9   10   >