Re: [HACKERS] bogus aset link
Thanks! Your advice helped me find what caused the problem immediately. It was a wrong 'palloc' as you suspected (and as usual for me ;-) ). Ntinos Katsaros Tom Lane writes: [EMAIL PROTECTED] writes: While writting some code for the backend (some SPI-like functions for a project) I saw this message: WARNING: problem in alloc set ExecutorState: bogus aset link in block 0x8301270, chunk 0x8304458 I think there is something wrong with some of the memory allocations I do, but this message is not helping me much in finding the exact error. The most likely bet is that your code wrote past the end of a memory chunk it had palloc'd, and thereby clobbered the bookkeeping info for the next physically adjacent chunk. You could home in on the location of the clobber by sprinkling MemoryContextCheck(TopMemoryContext) calls through your code. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] CVS is very slow
On Mon, 3 Jan 2005, Bruce Momjian wrote: Tonight regular CVS is very slow to generate a diff. Does anyone know why? Load average is only 2.0. if load avg is that low, then I'd have to say there is some network lag somewhere ... if load avg was higher, then I'd say a flood of messages to majordomo ... one thing to note that altho there is a flood of messages to majordomo, it doesn't mean a flood to the lists themselves, as majordomo itself processes 'bounce messages', keeping statistics on, and auto-unsubscribes if appropriate ... so 10 messages to the lists could translate into 1000 messages flooding into majordomo itself if 100 of the recipients bounce ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] [ANNOUNCE] PostgreSQL 8.0.0 Release Candidate 3
On Sun, 2005-01-02 at 09:56, Marc G. Fournier wrote:
As was anticipated, time between Release Candidate 2 and 3 was nice and
short, with more changes being made now to Documentation vs Code.
A current list of *known* supported platforms can be found at:
http://developer.postgresql.org/supported-platforms.html
Are you sure this list is complete? I don't see the ps2, xbox, or
gamecube listed under supported platforms :-)
Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
Re: [HACKERS] [ANNOUNCE] PostgreSQL 8.0.0 Release Candidate 3
I also don't see MIPSEL and ARM on the list, both running debian sarge (in the
build farm).
Jim
-- Original Message ---
From: Robert Treat [EMAIL PROTECTED]
To: pgsql-hackers@postgresql.org
Sent: 03 Jan 2005 08:35:19 -0500
Subject: Re: [HACKERS] [ANNOUNCE] PostgreSQL 8.0.0 Release Candidate 3
On Sun, 2005-01-02 at 09:56, Marc G. Fournier wrote:
As was anticipated, time between Release Candidate 2 and 3 was nice and
short, with more changes being made now to Documentation vs Code.
A current list of *known* supported platforms can be found at:
http://developer.postgresql.org/supported-platforms.html
Are you sure this list is complete? I don't see the ps2, xbox, or
gamecube listed under supported platforms :-)
Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
--- End of Original Message ---
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] TSearch still in contrib?
On Friday 31 December 2004 14:37, Joshua D. Drake wrote:
Hello,
Is there any reason why TSearch (not TSearch2) is still in
contrib? TSearch is a 7.3.x module...
I believe it was left in for backwards compatibility and afaik it still builds
in the 8.0 tree. If you'd like to make the argument that 8.0 is a good time
to remove it go ahead. The first couple lines of README.tsearch might help:
This module is deprecated in 7.4 version of PostgreSQL and will be
obsoleted in 8.0. Please, use new tsearch2 contrib module.
--
Robert Treat
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
[HACKERS] [PERFORM] query rewrite using materialized views
Hi, are there any plans for rewriting queries to preexisting materialized views? I mean, rewrite a query (within the optimizer) to use a materialized view instead of the originating table? Regards, Yann ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] 'COPY ... FROM' inserts to btree, blocks on buffer writeout
Simon Riggs [EMAIL PROTECTED] writes: The situation where another backend requests the block immediately before the I/O is fairly common AFAICS, especially since StrategyGetBuffer ignores the BM_DIRTY flag in selecting victims. How do you figure that? StrategyGetBuffer won't return the same buffer again (because dirty or not, it'll be pinned by the time anyone else gets to run StrategyGetBuffer). The case we are interested in is where someone suddenly wants the original page again --- that is, a page that was just about to fall off the back end of the freelist is wanted again. I don't see that that case is common, especially not with a reasonably large shared_buffer setting, and most especially not when the bgwriter is doing its job and keeping the back end of the freelist clean. ISTM making the code deadlock-safe will effect cases where there never would have been a deadlock, slowing both backends down while waiting for the I/O to complete. The other backend will have to wait for the I/O to complete before he can gain an exclusive lock on the page ... but so what? If he'd come along a microsecond later, he'd have had to wait, too. Basically we are eliminating a very narrow window by causing it to behave the same as what happens in the larger window where the I/O is occurring. (BTW, I/O in this case actually just represents transferring the data to kernel buffers, so the amount of delay involved is likely not to be all that large...) regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] uptime() for postmaster
Hi Tom, Am 31.12.2004 um 20:18 schrieb Tom Lane: Matthias Schmidt [EMAIL PROTECTED] writes: a) is the name uptime() OK? Probably should use pg_uptime(), or something else starting with pg_. What about 'pg_starttime()' since it is not a period but a point-in-time? b) is the return-type 'Interval' OK? It might be better to return the actual postmaster start time (as timestamptz) and let the user do whatever arithmetic he wants. With an interval, there's immediately a question of interpretation --- what current timestamp did you use in the computation? I'm not dead set on this, but it feels cleaner. you're right. Let's go for timestamptz and let the users decide ... c) does it make sense (... fit in the scheme?) to place the code here: src/backend/utils/misc/uptime.c No. This sort of stuff should go into utils/adt/. I'd be inclined to drop the function into one of the existing timestamp-related files rather than make a whole new file just for it. Someplace near the now() function would make sense, for instance. yep - so the stuff goes to: utils/adt/timestamp.c, where now() and many other time-related functions are. d) Can I piggy-back on 'BackendParameters' to get postmasters start-time to the backends? AFAICS you have no other choice. regards, tom lane cheers, Matthias -- Matthias Schmidt Viehtriftstr. 49 67346 Speyer Tel.: +49 6232 4867 Fax.: +49 6232 640089 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] PostgreSQL 8.0.0 Release Candidate 3
Hello Marc, Marc G. Fournier schrieb am 02.01.2005 15:56: As was anticipated, time between Release Candidate 2 and 3 was nice and short, with more changes being made now to Documentation vs Code. A current list of *known* supported platforms can be found at: http://developer.postgresql.org/supported-platforms.html We're always looking to improve that list, so we encourage anyone that is running a platform not listed to please report on any success or failures with Release Candidate 3. Baring *any* coding changes (documentation != code) over the next week or so, we *hope* that this will the final Release Candidate before Full Release, with that being aimed for the 15th (or earlier). As always, this release is available on all mirrors, as listed at: http://www.postgresql.org/mirrors-ftp.html 1. this URL doesn't exist any more. I only found the following: http://wwwmaster.postgresql.org/download/mirrors-ftp 2. I visited some of the mirrors as well as ftp.postgresql.org and there the source tarballs of rc3 have file date 31.12.2004 22:39 while Tom Lane posted last corrections related to copyright messages on 01.01.2005 23:14, and Bruce Momjian posted Updates related to UTF-8 on 01.01.2005 17:36 So my question: are there really the correct rc3 tarballs around? With best regards, Roland ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] 'COPY ... FROM' inserts to btree, blocks on buffer writeout
I wrote: I think that it would work for BufferAlloc to share-lock the victim buffer before calling FlushBuffer; we'd have to add a bool parameter to FlushBuffer telling it the lock was already acquired. I've applied a patch for this. BTW, it looks to me like this deadlock potential has existed at least since 7.0. I seem to recall one or two reports of unexplainable apparent deadlocks, which perhaps are now explained. On closer investigation the deadlock does not seem to exist in 7.4 and before, because BufferReplace didn't acquire the buffer sharelock. (There is a comment in the 7.4 code claiming that we didn't need to, but I'm unconvinced that it's correct...) regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] TSearch still in contrib?
Robert Treat wrote: On Friday 31 December 2004 14:37, Joshua D. Drake wrote: Hello, Is there any reason why TSearch (not TSearch2) is still in contrib? TSearch is a 7.3.x module... I believe it was left in for backwards compatibility and afaik it still builds in the 8.0 tree. If you'd like to make the argument that 8.0 is a good time to remove it go ahead. The first couple lines of README.tsearch might help: This module is deprecated in 7.4 version of PostgreSQL and will be obsoleted in 8.0. Please, use new tsearch2 contrib module. Compiling and working are different things :). To be fair I haven't tested TSearch on 8.0 but since it is considered outdated... perhaps we should remove it. Sincerely, Joshua D. Drake -- Command Prompt, Inc., home of PostgreSQL Replication, and plPHP. Postgresql support, programming shared hosting and dedicated hosting. +1-503-667-4564 - [EMAIL PROTECTED] - http://www.commandprompt.com Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL begin:vcard fn:Joshua D. Drake n:Drake;Joshua D. org:Command Prompt, Inc. adr:;;PO Box 215;Cascade Locks;Oregon;97014;USA email;internet:[EMAIL PROTECTED] title:Consultant tel;work:503-667-4564 tel;fax:503-210-0334 note:Command Prompt, Inc. is the largest and oldest US based commercial PostgreSQL support provider. We provide the only commercially viable integrated PostgreSQL replication solution, but also custom programming, and support. We authored the book Practical PostgreSQL, the procedural language plPHP, and adding trigger capability to plPerl. x-mozilla-html:FALSE url:http://www.commandprompt.com/ version:2.1 end:vcard ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] Bgwriter behavior
OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote on the patch. Is it: * too late for 8.0 * not the right improvement * to be applied with doc/default additions Comments? --- Simon Riggs wrote: On Sat, 2005-01-01 at 17:47, Simon Riggs wrote: On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote: Simon Riggs wrote: Well, I think we're saying: its not in 8.0 now, and we take our time to consider patches for 8.1 and accept the situation that the parameter names/meaning will change in next release. I have no problem doing something for 8.0 if we can find something that meets all the items I mentioned. One idea would be to just remove bgwriter_percent. Beta/RC users would still have it in their postgresql.conf, but it is commented out so it should be OK. If they uncomment it their server would not start but we could just tell testers to remove it. I see that as better than having conflicting parameters. Can't say I like that at first thought. I'll think some more though... Another idea is to have bgwriter_percent be the percent of the buffer it will scan. Hmmmwell that was my original suggestion (bg2.patch on 12 Dec) (...though with a bug, as Neil pointed out) We could default that to 50% or 100%, but we then need to make sure all beta/RC users update their postgresql.conf with the new default because the commented-out default will not be correct. ...we just differ/ed on what the default should be... At this point I see these as our only two viable options, aside from doing nothing. I realize our current behavior requires a full scan of the buffer cache, but how often is the bgwriter_maxpages limit met? If it is not a full scan is done anyway, right? Well, if you heavy a very heavy read workload then that would be a problem. I was more worried about concurrency in a heavy write situation, but I can see your point, and agree. (Idea #1 still suffers from this, so we should rule it out...) It seems the only way to really add functionality is to change bgwriter_precent to control how much of the buffer is scanned. OK. I think you've persuaded me on idea #2, if I understand you right: bgwriter_percent = 50 (default) bgwriter_maxpages = 100 (default) percent is the number of shared_buffers we scan, limited by maxpages. (I'll code it up in a couple of hours when the kids are in bed) Here's the basic patch - no changes to current default values or docs. Not sure if this is still interesting or not... -- Best Regards, Simon Riggs [ Attachment, skipping... ] ---(end of broadcast)--- TIP 8: explain analyze is your friend -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
[HACKERS] [Fwd: Re: postgres+tcl on cygwin]
I have not had time to test this. cheers andrew Original Message Subject:Re: postgres+tcl on cygwin Date: Mon, 3 Jan 2005 03:50:24 -0800 (PST) From: Patrick Samson [EMAIL PROTECTED] To: Andrew Dunstan [EMAIL PROTECTED] I succeeded in the build with the TCL support this way: - As I always did, I took the source tarball from the postgresql site. It doesn't mean it's not possible with the source package available on the cygwin site. Just that I always took the original source and it was fine with it OOTB. Version: 8.0.0rc2 OS: Win98SE - In /lib/tclConfig.sh, change to: TCL_LIB_SPEC='-L/usr/lib -ltcl84' - In src/pl/tcl/pltcl.c, permute the order of two includes, to read like that: #include tcl.h #include postgres.h It will produce a warning about a DLLIMPORT redefinition. Ignore it. - ./configure --with-tcl - make Have fun, and let me know your result. Patrick ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Implementing and Experimenting with a Full Disjunctions Operator.
Hi All, As part of my thesis I need to implement a new algorithm for Full-Disjunctions and 2 older ones. A short explanation of what Full-Disjunction is, is that it comes to solve what A natural outer join usually can't do when you have more than 2 relations. The main goal is to retrieve maximal answers from a set of relations. The natural representation of the operator that is usually depicted in the literature is FD(r1,...,rN). (r=relation/table) Ullman's algorithm uses several natural outer joins so there is no problem there but our algorithm must be run internally at the server since it uses no existing operators but it is also not limited to the gamma-acyclic restriction of Ullman's algorithm. I have already read most of the development documentations, faqs, presentations, listened on this mailing list, etc... I already compiled a dynamically loaded library with a function and ran it successfully. The research part of implementing the algorithm is theoretical and experimentation. After looking around in the code and seeing how the SPI generally works I have several concerns (the first one is the most crucial to me): 1) As part of the experimentation I need to know exactly how many blocks have been read when the algorithm ran. I need complete control over the process to run my simulations. I see that there are functions like heap heap_getnext() heap_fetch() SearchSysCache(). Our algorithm usually read sequentially the relations and I don't see how to read complete blocks and count these blocks. In addition temporary queues that must be held in memory will be needed to be dumped to disk at various times (because of their size) and fetched. Is there a way to control this process with accuracy and calculate the exact disk writes? 2) As part of the theoretical work and experimentation we want to load blocks of relation rows to the main memory and cache them using our techniques. Is there a way to control the memory blocks so they won't be swapped. In addition, is there a way to get a specific size of memory so we can plan our operator running path. I see that palloc return's to me a chunck of memory but I don't know in advance how much is available to me (aside from polling for it). 3) When outputting the results as a set of records, I cannot know in advance the type of temporary table that will come out just like a subquery like (select * from relationA,relationB); Is there a problem outputting this kind of table? 4) When inputting the various tables to the operator I understand that the function is limited to a fixed number of arguments. Is there a way to circumvent that or would I need to use an ugly ARRAY construct. Obviously there are better ways than a dynamically loaded library function, so after we study the algorithm I don't think there should be any problem integrating it to postgreSQL, of course if it will be good enough :) Thank you. Regards, tzahi. * - * - * Itzhak Fadida MSc Student Information System Engineering Area Faculty of Industrial Engineering Management Technion - Israel Institute of Technology Technion City, Haifa, Israel 32000 Technion Email: [EMAIL PROTECTED] Alternative Email: [EMAIL PROTECTED] * - * - * - * - * - * - * - * - * - * - * - * - * - * - * WARNING TO SPAMMERS: see at http://members.lycos.co.uk/my2nis/spamwarning.html ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] Bgwriter behavior
Bruce Momjian pgman@candle.pha.pa.us writes: OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote on the patch. Is it: * too late for 8.0 * not the right improvement * to be applied with doc/default additions My vote: too late for 8.0. There is no hard evidence that this is a useful improvement, and no time for such evidence to be obtained. regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
[HACKERS] possible bug in case comparison on index scan.
I've uncovered a possible bug on that may be related to the other locale
relation issues for win32 being discussed right now. Basically, I have
a situation where a query is pulling up the wrong record based on string
case. So far, I have not been able to reproduce this in linux.
Here is the situation:
Database is initialized to the default locale which is English_United
States.1252 (as reported by pg_controldata.exe). Databases initialized
to the 'C' locale don't have this problem. If this is just a locale
issue, I'm not concerned (just use the C locale), but just want to make
sure a btree related issue is not slipping through.
Observe:
esp=# select version();
version
-
PostgreSQL 8.0.0rc2 on i686-pc-mingw32, compiled by GCC gcc.exe (GCC)
3.4.1 (mingw special)
(1 row)
esp=# prepare data9_start_nl_item_base_file_5 (character varying,
character varying, int4)
esp-# as select 1::int4, ib_sales_description_1 from
data9.item_base_file
esp-# where ib_sales_description_1 = $1 and
esp-# (ib_sales_description_1 $1 or ib_sales_description_2
= $2)
esp-# order by ib_sales_description_1, ib_sales_description_2, id
esp-# limit $3;
PREPARE
esp=# execute data9_start_nl_item_base_file_5('SAT', '', 1 );
int4 | ib_sales_description_1
--+
1 | satellite
(1 row)
esp=# select 'sat' = 'SAT';
?column?
--
f
(1 row)
esp=# execute data9_start_nl_item_base_file_5('SAT', '', 2 );
int4 |ib_sales_description_1
--+---
1 | satellite
1 | SATELLITE (current) AUTO DUAL
(2 rows)
Merlin
---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match
Re: [HACKERS] TODO item: make world safe for spaces in build/install paths
Tom Lane wrote: Pursuant to Theodore Petrosky's recent trouble report, I thought I would see what happens if you try to build Postgres in a directory whose path contains spaces, The last I heard in the autotools community on this issue was forget it. Not that that means it's impossible, but you're going to fight a hard battle with make, which has no quoting mechanism at all. or if the install prefix contains spaces. This should be achievable, with a few thousand quotes in the right places. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [PATCHES] [HACKERS] Bgwriter behavior
On Mon, 3 Jan 2005, Bruce Momjian wrote: OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote on the patch. Is it: * too late for 8.0 Too late by at least 3 RCs ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] [ANNOUNCE] PostgreSQL 8.0.0 Release Candidate 3
Jim Buttafuoco wrote: I also don't see MIPSEL and ARM on the list, both running debian sarge (in the build farm). The mips entry is actually a mipsel, but uname identifies them the same. I don't see any arm machine in the build farm. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] 'COPY ... FROM' inserts to btree, blocks on buffer
On Mon, 2005-01-03 at 17:14, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: The situation where another backend requests the block immediately before the I/O is fairly common AFAICS, especially since StrategyGetBuffer ignores the BM_DIRTY flag in selecting victims. How do you figure that? StrategyGetBuffer won't return the same buffer again (because dirty or not, it'll be pinned by the time anyone else gets to run StrategyGetBuffer). The case we are interested in is where someone suddenly wants the original page again --- that is, a page that was just about to fall off the back end of the freelist is wanted again. I don't see that that case is common, especially not with a reasonably large shared_buffer setting, and most especially not when the bgwriter is doing its job and keeping the back end of the freelist clean. Yes, what I was effectively arguing for was to tune for the case where shared_buffers is still at the default...which is of course fairly pointless, since the way to tune is just to increase shared_buffers. ...Fully agree with your original suggestion now. -- Best Regards, Simon Riggs ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [PATCHES] [HACKERS] Bgwriter behavior
On Mon, 2005-01-03 at 20:09, Bruce Momjian wrote: OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote on the patch. Is it: * too late for 8.0 * not the right improvement * to be applied with doc/default additions Comments? --- Simon Riggs wrote: On Sat, 2005-01-01 at 17:47, Simon Riggs wrote: On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote: Simon Riggs wrote: Well, I think we're saying: its not in 8.0 now, and we take our time to consider patches for 8.1 and accept the situation that the parameter names/meaning will change in next release. I hear veto ... so the above situation stands then: 8.1 it is. Not unhappy...I want this thing released as much as the next man... -- Best Regards, Simon Riggs ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] Bgwriter behavior
Simon Riggs wrote: On Mon, 2005-01-03 at 20:09, Bruce Momjian wrote: OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote on the patch. Is it: * too late for 8.0 * not the right improvement * to be applied with doc/default additions Comments? --- Simon Riggs wrote: On Sat, 2005-01-01 at 17:47, Simon Riggs wrote: On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote: Simon Riggs wrote: Well, I think we're saying: its not in 8.0 now, and we take our time to consider patches for 8.1 and accept the situation that the parameter names/meaning will change in next release. I hear veto ... so the above situation stands then: 8.1 it is. Not unhappy...I want this thing released as much as the next man... Well, we went through the process and that's the best we can do. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [PATCHES] [HACKERS] Bgwriter behavior
On Mon, 2005-01-03 at 23:03, Bruce Momjian wrote:
Simon Riggs wrote:
On Mon, 2005-01-03 at 20:09, Bruce Momjian wrote:
OK, we have a submitted patch that attempts to improve bgwriter by
making bgwriter_percent control what percentage of the buffer is
scanned.
The patch still needs doc changes and a change to the default value but
at this point we need a vote on the patch. Is it:
* too late for 8.0
* not the right improvement
* to be applied with doc/default additions
Comments?
---
Simon Riggs wrote:
On Sat, 2005-01-01 at 17:47, Simon Riggs wrote:
On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote:
Simon Riggs wrote:
Well, I think we're saying: its not in 8.0 now, and we take our
time to
consider patches for 8.1 and accept the situation that the
parameter
names/meaning will change in next release.
I hear veto ... so the above situation stands then: 8.1 it is.
Not unhappy...I want this thing released as much as the next man...
Well, we went through the process and that's the best we can do.
Here's my bgwriter instrumentation patch, which gives info that could
allow the bgwriter settings to be tuned.
--
Best Regards, Simon Riggs
Index: src/backend/storage/buffer/bufmgr.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/storage/buffer/bufmgr.c,v
retrieving revision 1.182
diff -d -c -r1.182 bufmgr.c
*** src/backend/storage/buffer/bufmgr.c 24 Nov 2004 02:56:17 - 1.182
--- src/backend/storage/buffer/bufmgr.c 4 Jan 2005 00:04:18 -
***
*** 440,445
--- 440,446
UnpinBuffer(buf, true);
inProgress = FALSE;
buf = NULL;
+ StrategyBufferStatWastedIO();
}
}
} while (buf == NULL);
***
*** 682,687
--- 683,689
BufferDesc **dirty_buffers;
BufferTag *buftags;
int num_buffer_dirty;
+ int num_buffer_cleaned = 0;
int i;
/* If either limit is zero then we are disabled from doing anything... */
***
*** 770,775
--- 772,778
TerminateBufferIO(bufHdr, 0);
UnpinBuffer(bufHdr, true);
+ num_buffer_cleaned++;
}
LWLockRelease(BufMgrLock);
***
*** 777,782
--- 780,787
pfree(dirty_buffers);
pfree(buftags);
+ StrategyBufferStatCleaned(num_buffer_cleaned);
+
return num_buffer_dirty;
}
Index: src/backend/storage/buffer/freelist.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/storage/buffer/freelist.c,v
retrieving revision 1.48
diff -d -c -r1.48 freelist.c
*** src/backend/storage/buffer/freelist.c 16 Sep 2004 16:58:31 - 1.48
--- src/backend/storage/buffer/freelist.c 4 Jan 2005 00:04:18 -
***
*** 115,120
--- 115,133
} while(0)
+ void
+ StrategyBufferStatWastedIO(void)
+ {
+ StrategyControl-num_wasted++;
+ }
+
+ void
+ StrategyBufferStatCleaned(long num_cleaned)
+ {
+ StrategyControl-num_cleaned += num_cleaned;
+ }
+
+
/*
* Printout for use when DebugSharedBuffers is enabled
*/
***
*** 130,159
t1_hit,
t2_hit,
b2_hit;
- int id,
- t1_clean,
- t2_clean;
ErrorContextCallback *errcxtold;
- id = StrategyControl-listHead[STRAT_LIST_T1];
- t1_clean = 0;
- while (id = 0)
- {
- if (BufferDescriptors[StrategyCDB[id].buf_id].flags BM_DIRTY)
- break;
- t1_clean++;
- id = StrategyCDB[id].next;
- }
- id = StrategyControl-listHead[STRAT_LIST_T2];
- t2_clean = 0;
- while (id = 0)
- {
- if (BufferDescriptors[StrategyCDB[id].buf_id].flags BM_DIRTY)
- break;
- t2_clean++;
- id = StrategyCDB[id].next;
- }
-
if (StrategyControl-num_lookup == 0)
all_hit = b1_hit = t1_hit = t2_hit = b2_hit = 0;
else
--- 143,150
***
*** 166,185
StrategyControl-num_lookup);
b2_hit = (StrategyControl-num_hit[STRAT_LIST_B2] * 100 /
StrategyControl-num_lookup);
! all_hit = b1_hit + t1_hit + t2_hit + b2_hit;
}
errcxtold = error_context_stack;
error_context_stack = NULL;
elog(DEBUG1, ARC T1target=%5d B1len=%5d T1len=%5d T2len=%5d B2len=%5d,
T1_TARGET, B1_LENGTH, T1_LENGTH, T2_LENGTH, B2_LENGTH);
! elog(DEBUG1, ARC total =%4ld%% B1hit=%4ld%% T1hit=%4ld%% T2hit=%4ld%% B2hit=%4ld%%,
all_hit, b1_hit, t1_hit, t2_hit, b2_hit);
! elog(DEBUG1, ARC clean buffers at LRU T1= %5d T2= %5d,
! t1_clean, t2_clean);
! error_context_stack = errcxtold;
StrategyControl-num_lookup = 0;
StrategyControl-num_hit[STRAT_LIST_B1] = 0;
StrategyControl-num_hit[STRAT_LIST_T1] = 0;
StrategyControl-num_hit[STRAT_LIST_T2] = 0;
--- 157,188
StrategyControl-num_lookup);
Re: [PATCHES] [HACKERS] Bgwriter behavior
Simon Riggs wrote: Here's my bgwriter instrumentation patch, which gives info that could allow the bgwriter settings to be tuned. Uh, what does this do exactly? Add additional logging output? -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] uptime() for postmaster
Matthias Schmidt wrote: Hi Tom, Am 31.12.2004 um 20:18 schrieb Tom Lane: Matthias Schmidt [EMAIL PROTECTED] writes: a) is the name uptime() OK? Probably should use pg_uptime(), or something else starting with pg_. What about 'pg_starttime()' since it is not a period but a point-in-time? b) is the return-type 'Interval' OK? It might be better to return the actual postmaster start time (as timestamptz) and let the user do whatever arithmetic he wants. With an interval, there's immediately a question of interpretation --- what current timestamp did you use in the computation? I'm not dead set on this, but it feels cleaner. you're right. Let's go for timestamptz and let the users decide ... Well, the unix guys have the abit to have the uptime as an interval, I'm inclined to have boths: pg_uptime ( interval ) and pg_starttime ( timestamptz ) Regards Gaetano Mendola ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] oldish libpq bug still in RC2
This item still seems open. Is it a TODO? --- Hannu Krosing wrote: It seems that this bug is still lurking in libpq: http://search.postgresql.org/pgsql-hackers/2004-09/msg00703.php Is anybody working on it, or should I try something myself, perhaps just replacing the lone recv() with pqsecure_read() ? -- Hannu Krosing [EMAIL PROTECTED] ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] race condition for drop schema cascade?
Did this get resolved as an OS file system issue? --- Andrew Dunstan wrote: Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: You're right - my query was not sufficiently specific. There have in fact been 4 failures: pgbuildfarm=# select sysname, snapshot, stage, branch from build_status where log ~ 'tablespace testspace is not empty.*tablespace testspace is not empty' and not log ~ 'No space left'; sysname | snapshot |stage | branch +-+--+ hare| 2004-12-09 05:15:05 | Check| HEAD otter | 2004-12-11 15:50:09 | Check| HEAD otter | 2004-12-15 15:50:10 | Check| HEAD gibbon | 2004-12-28 23:55:05 | InstallCheck | HEAD Why does the last show as an install failure? We run the standard regression suite twice - the failure on Gibbon occurred on the second of these. Clearly this is very transient. Anyway, given the small number of machines involved, I'm once again wondering what filesystem they are using. They wouldn't be running the check over NFS, by any chance, for instance? The theory that is in my mind is that the bgwriter could have written out a page for the table in the test tablespace, and thereby be holding an open file pointer for it. On standard Unix filesystems this would not disrupt the backend's ability to unlink the table at the DROP stage, but I'm wondering about nonstandard filesystems ... Jim Buttafuoco reported on December 16th that he had rebuilt the filesystem on his MIPS box - I assume this means that he isn't using NFS. In any case, we have not seen the problem since then. His Alpha box has not been reporting buildfarm results since before then. The Cygwin box is running on NTFS - and we know we've encountered plenty of problems with unlinking on Windows. I know it's not much to go on. cheers andrew ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] TODO item: make world safe for spaces in build/install
Peter Eisentraut wrote: Tom Lane wrote: Pursuant to Theodore Petrosky's recent trouble report, I thought I would see what happens if you try to build Postgres in a directory whose path contains spaces, The last I heard in the autotools community on this issue was forget it. Not that that means it's impossible, but you're going to fight a hard battle with make, which has no quoting mechanism at all. or if the install prefix contains spaces. This should be achievable, with a few thousand quotes in the right places. Interesting. TODO updated: * Allow building with directories containing spaces There are two capabilities here, first the ability to build from a source directory that contains spaces, and second the ability to install into a directory that contains spaces. The first is probably not possible because 'gmake' and other compiler tools do not fully support spaces in path names. The second is possible with proper quoting in the makefiles. Because PostgreSQL supports relocatable installs, it is possible to install into a directory that doesn't contain spaces and then copy the install to a directory with spaces. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] [HACKERS] Bgwriter behavior
This has been saved for the 8.1 release: http:/momjian.postgresql.org/cgi-bin/pgpatches2 --- Simon Riggs wrote: On Sat, 2005-01-01 at 17:47, Simon Riggs wrote: On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote: Simon Riggs wrote: Well, I think we're saying: its not in 8.0 now, and we take our time to consider patches for 8.1 and accept the situation that the parameter names/meaning will change in next release. I have no problem doing something for 8.0 if we can find something that meets all the items I mentioned. One idea would be to just remove bgwriter_percent. Beta/RC users would still have it in their postgresql.conf, but it is commented out so it should be OK. If they uncomment it their server would not start but we could just tell testers to remove it. I see that as better than having conflicting parameters. Can't say I like that at first thought. I'll think some more though... Another idea is to have bgwriter_percent be the percent of the buffer it will scan. Hmmmwell that was my original suggestion (bg2.patch on 12 Dec) (...though with a bug, as Neil pointed out) We could default that to 50% or 100%, but we then need to make sure all beta/RC users update their postgresql.conf with the new default because the commented-out default will not be correct. ...we just differ/ed on what the default should be... At this point I see these as our only two viable options, aside from doing nothing. I realize our current behavior requires a full scan of the buffer cache, but how often is the bgwriter_maxpages limit met? If it is not a full scan is done anyway, right? Well, if you heavy a very heavy read workload then that would be a problem. I was more worried about concurrency in a heavy write situation, but I can see your point, and agree. (Idea #1 still suffers from this, so we should rule it out...) It seems the only way to really add functionality is to change bgwriter_precent to control how much of the buffer is scanned. OK. I think you've persuaded me on idea #2, if I understand you right: bgwriter_percent = 50 (default) bgwriter_maxpages = 100 (default) percent is the number of shared_buffers we scan, limited by maxpages. (I'll code it up in a couple of hours when the kids are in bed) Here's the basic patch - no changes to current default values or docs. Not sure if this is still interesting or not... -- Best Regards, Simon Riggs [ Attachment, skipping... ] ---(end of broadcast)--- TIP 8: explain analyze is your friend -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] uptime() for postmaster
Gaetano Mendola [EMAIL PROTECTED] writes: Well, the unix guys have the abit to have the uptime as an interval, I'm inclined to have boths: pg_uptime ( interval ) and pg_starttime ( timestamptz ) Well for the OS these are not redundant values. The clock could have been adjusted at any time. So you can't just calculate uptime by subtracting the current time from the start time. I suppose this argument is true for Postgres as well. But I'm not sure Postgres can really make the distinction as easily as the kernel. To return the actual uptime without being deceived by clock changes it would need to store not the wall clock time on startup, but the system uptime. And then calculate the difference in the current system uptime. I'm not sure if there is a portable interface to get a system uptime. -- greg ---(end of broadcast)--- TIP 8: explain analyze is your friend
