Re: [HACKERS] postmaster.exe vs postgres.exe (was: CVS HEAD busted on

2006-06-22 Thread Bruce Momjian
Tom Lane wrote: > "Dave Page" writes: > >>> though - Magnus & > >>> I were wondering if Peter's change means we no longer need to ship > >>> postmaster.exe and postgres.exe with pgInstaller. Presumably > >>> we can just use postgres.exe for everything now? > > >> Won't we still need to know if w

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Heikki Linnakangas
On Thu, 22 Jun 2006, Jim Nasby wrote: DB2 switched to MVCC in version 8. ... Um, no it didn't. - Heikki ---(end of broadcast)--- TIP 6: explain analyze is your friend

[HACKERS] Full Disjunction

2006-06-22 Thread Christopher Kings-Lynne
What IS this full disjunction business? User Pgstudy wrote: Log Message: --- Some more deformed tuple structures consolidation works. Now even faster but still half way done. Modified Files: -- fd: algutils.c (r1.9 -> r1.10) (http://cvs.pgfoundry.org/cg

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Alvaro Herrera
Jonah H. Harris wrote: > On 6/22/06, Gavin Sherry <[EMAIL PROTECTED]> wrote: > >When an update occurs, the existing row version is > >copied to te UNDO file > > Not in all systems. A few now perform in-memory UNDO and only write > it to disk if and when it is required. How does that work? If t

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Gavin Sherry
On Thu, 22 Jun 2006, Jonah H. Harris wrote: > Not in all systems. A few now perform in-memory UNDO and only write > it to disk if and when it is required. Interesting... > > > Overwriting MVCC comes with its own baggage. Ask any Oracle user about > > error ORA-01555[1]. There's also the added c

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Gavin Sherry <[EMAIL PROTECTED]> wrote: There seems to be a bit of confusion about what MVCC is. Yes, PostgreSQL implements MVTO-style concurrency, Oracle implements MVRC, ... Let's not go into theory here, because there's plenty of papers and books on the subject. The other MVCC

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Christopher Browne
The world rejoiced as [EMAIL PROTECTED] (Tom Lane) wrote: > Lukas Smith <[EMAIL PROTECTED]> writes: >> Jochem van Dieten wrote: >>> make the session handler smarter? And if you can't do that, put some >>> logic in the session table that turns an update without changes into a >>> no-op? > >> err isn

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Gavin Sherry
On Thu, 22 Jun 2006, Agent M wrote: > > On Jun 22, 2006, at 9:56 PM, Christopher Kings-Lynne wrote: > > >> The example is a very active web site, the flow is this: > >> query for session information > >> process HTTP request > >> update session information > >> This happens for EVERY http request.

Row comparison for tables (was Re: [HACKERS] vacuum, performance, and MVCC)

2006-06-22 Thread Tom Lane
I wrote: > Jim Nasby <[EMAIL PROTECTED]> writes: >> What would be nice to add is the ability to perform that check more >> easily. As of 8.1... >> ... >> if NEW=OLD then >> ... >> ERROR: operator does not exist: test = test >> HINT: No operator matches the given name and argument type(s). You

[HACKERS] Small overhead run time memory trace (Was Re: shall we have a TRACE_MEMORY mode)

2006-06-22 Thread Qingqing Zhou
"Tom Lane" <[EMAIL PROTECTED]> wrote > > One idea that comes to mind is to have a compile time option to record > the palloc __FILE__ and _LINE__ in every AllocChunk header. Then it > would not be so hard to identify the culprit while trawling through > memory. The overhead costs would be so hig

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Steve Atkins
On Jun 22, 2006, at 6:56 PM, Agent M wrote: On Jun 22, 2006, at 9:56 PM, Christopher Kings-Lynne wrote: The example is a very active web site, the flow is this: query for session information process HTTP request update session information This happens for EVERY http request. Chances are that

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Agent M
On Jun 22, 2006, at 9:56 PM, Christopher Kings-Lynne wrote: The example is a very active web site, the flow is this: query for session information process HTTP request update session information This happens for EVERY http request. Chances are that you won't have concurrent requests for the sam

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Mark Kirkwood
Tom Lane wrote: Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: Tom Lane wrote: It'd be interesting to compare 8.1 and HEAD for the no-overhead case; I don't think you need to redo all four cases, but I'd like to see that one. 8.1:50,50,49 HEAD: 49,48,49 OK, so that seems comparable

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Thomas Lockhart writes: >> This code seems to have been inserted by Tom Lockhart on 1997-07-29 >> (geo_ops.c rev 1.13). Tom, any info on the copyright status? > None, beyond the info you already resurrected. I vaguely recall that I > did take the LJ letter as an invitation to reuse algorithms.

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Christopher Kings-Lynne
The example is a very active web site, the flow is this: query for session information process HTTP request update session information This happens for EVERY http request. Chances are that you won't have concurrent requests for the same row, but you may have well over 100 HTTP server processes/t

Re: [HACKERS] Going for "all green" buildfarm results

2006-06-22 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Mark Wong wrote: >> Flex is 2.5.33 on both systems. I'm assuming that's too modern so >> I'll go ahead and stop building 7.3 for those systems. > You could be lucky the others build. I believe our supported version is > still 2.5.4, which is what all

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Thomas Lockhart
1) Is any John Franks code really in this file? Possibly, maybe probably. I don't remember the details (9 years is a long time!) but almost certainly any code or algorithms were specifically for the "inside" or "outside" routines. 2) Did John provide a separate license for PostgreSQL to li

Re: [HACKERS] Interval aggregate regression failure (expected seems

2006-06-22 Thread Michael Glaesemann
On Jun 23, 2006, at 9:47 , Michael Glaesemann wrote: It also changes the result of the aggregate test for intervals, but I think that's to be expected. My goodness. Of course it changes the aggregate test results. That was what brought this up in the first place. (*kicks self for not rea

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> The patch to c.h certainly had no compunction about possibly changing >> the behavior for Cygwin: > Maybe we need to divorce Cygwin and Win32. That seems like probably an overreaction. The impression I got was that this patch had ac

Re: [HACKERS] Interval aggregate regression failure (expected seems

2006-06-22 Thread Michael Glaesemann
Tom Lane wrote: I've also confirmed that the problem is in interval_div; you can reproduce the failure with select '41 years 1 mon 11 days'::interval / 10; which should give '4 years 1 mon 9 days 26:24:00', but when timestamp.o is compiled with "-mcpu=pentium4 -march=pentium4", you ge

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan <[EMAIL PROTECTED]> writes: Meanwhile, I'd like to know how to fix the Cygwin build on HEAD. I suspect it was the MSVC "improvements" that did it. The patch to c.h certainly had no compunction about possibly changing the behavior for Cygwin: [sni

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > Seeing stats_command_string with almost zero overhead is great news! > Should we remove that setting and just have it enabled all > the time? If you don't need it, you shouldn't have to pay any overhead for it, I think. One could make an argument now fo

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Meanwhile, I'd like to know how to fix the Cygwin build on HEAD. I > suspect it was the MSVC "improvements" that did it. The patch to c.h certainly had no compunction about possibly changing the behavior for Cygwin: *** *** 82,94 #e

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Andrew Dunstan
Tom Lane wrote: Meanwhile, I'd like to know how to fix the Cygwin build on HEAD. I suspect it was the MSVC "improvements" that did it. Probably. This is the commit: 2006-06-07 18:24 momjian [file list snipped] Prepare code to be built by MSVC:

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > On REL8_1_STABLE sources commenting out the DLLINIT definition in > Makefile.cygwin works just fine. Same goes for > Win32/HEAD/Makefile.win32. I just did complete (unreported) buildfarm > run with these changes made, so I think ripping that out sho

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Andrew Dunstan
Andrew Dunstan wrote: Tom Lane wrote: OK, so let's yank the file altogether and see what happens. I can make a cut at fixing the makefiles based on removing references to DLLINIT, but it might be better if someone who's in a position to test the results on Windows did the patch ...

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Alvaro Herrera
Tom Lane wrote: > Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: > > Tom Lane wrote: > >> It'd be interesting to compare 8.1 and HEAD for the no-overhead case; > >> I don't think you need to redo all four cases, but I'd like to see that > >> one. > > > 8.1:50,50,49 > > HEAD: 49,48

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Andrew Dunstan
Tom Lane wrote: OK, so let's yank the file altogether and see what happens. I can make a cut at fixing the makefiles based on removing references to DLLINIT, but it might be better if someone who's in a position to test the results on Windows did the patch ... Something has br

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Larry Rosenman
Josh Berkus wrote: > Tom, > >> adddepend >> dbase >> dbmirror >> fulltextindex >> mSQL-interface >> mac >> oracle >> tips >> userlock > > I think you're right. I will do this before I leave town on the 30th. before anyone asks, the files I wrote in contrib/mac are free to be licensed any way th

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Alvaro Herrera
Tom Lane wrote: > Charles Comiskey <[EMAIL PROTECTED]> writes: > > item #3: Carsten Wolff copyright in informix.c file > > The file informix.c contains a copyright from Carsten Wolff. Did Carsten > > directly contribute this file to the PostgreSQL project? Wow, I see what mess we would be into

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Tom Lane
Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> It'd be interesting to compare 8.1 and HEAD for the no-overhead case; >> I don't think you need to redo all four cases, but I'd like to see that one. > 8.1: 50,50,49 > HEAD: 49,48,49 OK, so that seems comparable to my results

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Stefan Kaltenbrunner
Tom Lane wrote: > Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: >> Tom Lane wrote: >>> Or do you mean that you have stats_row_level and/or stats_block_level on >>> in all four cases? > >> yes - stats_row_level and stats_block_level on in all cases (sorry for >> the confusion) - I can easily red

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Tom Lane
Jim Nasby <[EMAIL PROTECTED]> writes: > What would be nice to add is the ability to perform that check more > easily. As of 8.1... > ... > if NEW=OLD then > ... > ERROR: operator does not exist: test = test > HINT: No operator matches the given name and argument type(s). You > may need to add

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Bort, Paul wrote: >>> so presumably this is only needed for old Cygwin versions. Can anyone >>> say how old "1001" is and whether we still ought to care about it? >> >> IIRC, I've been on 1.5.x for at least three years. 1.0/1.1 seems to be >> around 20

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Tom Lane
Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> Or do you mean that you have stats_row_level and/or stats_block_level on >> in all four cases? > yes - stats_row_level and stats_block_level on in all cases (sorry for > the confusion) - I can easily redo the tests without those

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Tom Lane
Jim Nasby <[EMAIL PROTECTED]> writes: > Question: do we currently create new index entries even if the index > key hasn't changed? Yes. > If so, what's the purpose of storing the CTID of > the next version in the old version of the row? So that UPDATE can always find the newest version of th

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Stefan Kaltenbrunner
Tom Lane wrote: > Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: >> This is what I get on a fast AMD Dual Opteron box(Running Debian >> Sarge/AMD64): > >>8.1.4 HEAD >> 100 SELECT 1;74,74,7377,76,77 >> stats_command_string=1;

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Tom Lane
Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: > This is what I get on a fast AMD Dual Opteron box(Running Debian > Sarge/AMD64): > 8.1.4 HEAD > 100 SELECT 1; 74,74,7377,76,77 > stats_command_string=1; 105,99,106

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jim Nasby
On Jun 22, 2006, at 1:09 PM, Tom Lane wrote: Lukas Smith <[EMAIL PROTECTED]> writes: Jochem van Dieten wrote: make the session handler smarter? And if you can't do that, put some logic in the session table that turns an update without changes into a no-op? err isnt that one the job of the

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jim Nasby
On Jun 22, 2006, at 2:00 PM, Mark Woodward wrote: I actually have a good number of years of experience in this topic, and memcached or file system files are NOT the best solutions for a server farm. What's wrong with memcached for session data? -- Jim C. Nasby, Sr. Engineering Consultant

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jim Nasby
On Jun 22, 2006, at 12:56 PM, Greg Stark wrote: Just for the record, if i understood correctly -- this was all a bit black magicky -- Oracle found the data in the rollback segment by storing a pointer to it in the block header where the updated data is. Ie, it could jump straight to the right

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread David Fetter
On Thu, Jun 22, 2006 at 07:01:38PM +0200, Lukas Smith wrote: > Jochem van Dieten wrote: > > >make the session handler smarter? And if you can't do that, put > >some logic in the session table that turns an update without > >changes into a no-op? > > err isnt that one the job of the database? By

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Stefan Kaltenbrunner
Tom Lane wrote: > The bad news is that except in the stats_command_string cases, HEAD > is noticeably slower than 8.1 on the machine with slow gettimeofday. > In the single-transaction test this might be blamed on the addition > of statement_timestamp support (which requires a gettimeofday per > s

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Andrew Dunstan
PFC wrote: What you seem not to grasp at this point is a large web-farm, about 10 or more servers running PHP, Java, ASP, or even perl. The database is usually the most convenient and, aside from the particular issue we are talking about, best suited. The answer is sticky sessions

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> >> What you seem not to grasp at this point is a large web-farm, about 10 >> or >> more servers running PHP, Java, ASP, or even perl. The database is >> usually >> the most convenient and, aside from the particular issue we are talking >> about, best suited. > > The answer is sticky session

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Andrew Dunstan
Bort, Paul wrote: so presumably this is only needed for old Cygwin versions. Can anyone say how old "1001" is and whether we still ought to care about it? IIRC, I've been on 1.5.x for at least three years. 1.0/1.1 seems to be around 2000/2001, based on a quick Google. So it's definite

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Tom Lane <[EMAIL PROTECTED]> wrote: Jonah, I've been working with this system for years, and it's not that easy to "handle the differences with a few macros". True, it is harder than just that. I didn't mean to make light of it at all, just that a good amount of design upfront woul

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Stefan Kaltenbrunner
Tom Lane wrote: > Josh Berkus writes: >> Yeah, thanks for reminding me. Will do before feature freeze. As soon as >> I can figure out how to generate a patch that removes directories. > > Don't worry about that; CVS never deletes directories. But anyway, > I can easily handle removing the co

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread PFC
What you seem not to grasp at this point is a large web-farm, about 10 or more servers running PHP, Java, ASP, or even perl. The database is usually the most convenient and, aside from the particular issue we are talking about, best suited. The answer is sticky sessions : each user is assi

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Josh Berkus
Tom, > adddepend > dbase > dbmirror > fulltextindex > mSQL-interface > mac > oracle > tips > userlock I think you're right. I will do this before I leave town on the 30th. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Josh Berkus writes: > Yeah, thanks for reminding me. Will do before feature freeze. As soon as > I can figure out how to generate a patch that removes directories. Don't worry about that; CVS never deletes directories. But anyway, I can easily handle removing the code. I just want someone e

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Yeah, you should be able to find the older version easily enough, if you > arrived at the newer version and realized you needed to visit the older > version. But this fails in scenarios where you are searching on a > column that's been updated --- the index

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Bort, Paul
> > so presumably this is only needed for old Cygwin versions. Can anyone > say how old "1001" is and whether we still ought to care about it? > IIRC, I've been on 1.5.x for at least three years. 1.0/1.1 seems to be around 2000/2001, based on a quick Google. So it's definitely older than PG 7.3

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> >> As you can see, in about a minute at high load, this very simple table >> lost about 10% of its performance, and I've seen worse based on update >> frequency. Before you say this is an obscure problem, I can tell you it >> isn't. I have worked with more than a few projects that had to switch

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Josh Berkus
Tom, > [ shrug... ] I'm not planning to panic; we've still got explicit GPL > code that's not been cleaned out of contrib/ yet. (Um, weren't you on > the hook to move those modules to pgfoundry projects?) Yeah, thanks for reminding me. Will do before feature freeze. As soon as I can figure

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Alvaro Herrera
Adding back pgsql-hackers. Mark Woodward wrote: > > Mark Woodward wrote: > > > >> Hmm, OK, then the problem is more serious than I suspected. > >> This means that every index on a row has to be updated on every > >> transaction that modifies that row. Is that correct? > > > > Add an index entry, y

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Josh Berkus writes: > Augh. Does this mean that we need to "backpatch" earlier versions to remove > the possible GPL links? [ shrug... ] I'm not planning to panic; we've still got explicit GPL code that's not been cleaned out of contrib/ yet. (Um, weren't you on the hook to move those modules

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread PFC
As you can see, in about a minute at high load, this very simple table lost about 10% of its performance, and I've seen worse based on update frequency. Before you say this is an obscure problem, I can tell you it isn't. I have worked with more than a few projects that had to switch away fro

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Rod Taylor
On Thu, 2006-06-22 at 13:56 -0400, Mark Woodward wrote: > >> > You mean systems that are designed so exactly, that they can't take > >> 10% > >> > performance change ? > >> > >> No, that's not really the point, performance degrades over time, in one > >> minute it degraded 10%. > >> > >> The update

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
"Magnus Hagander" <[EMAIL PROTECTED]> writes: >>> item #2: Is dllinit.c GPL code? > I don't think it's needed on Win32. It's not included in my VC++ build, > because I forgot it :-), and it works just fine. > The point is that as long as we don't do anything in it (which we > don't), the runtime s

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Diogo Biazus
Diogo, are you working from my old xlogdump hack?  If so what version?I can send you the latest off-list.  I add stuff to it periodically when I need it, and I don't think I've published it lately.Yup, I've got a version that was posted here some time ago. If you could send me the latest version I

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Tom Lane
Greg Stark <[EMAIL PROTECTED]> writes: > Tom Lane <[EMAIL PROTECTED]> writes: >> The Oracle design has got other drawbacks: if you need to access a row >> version other than than the very latest, you need to go searching in the >> rollback segments for it. This is slow (no index help) > Just for

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Josh Berkus
Tom, Augh. Does this mean that we need to "backpatch" earlier versions to remove the possible GPL links? -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--- TIP 4: Have you searched our list archives? ht

Re: [HACKERS] let's meet

2006-06-22 Thread A.M.
On Thu, June 22, 2006 1:42 pm, Josh Berkus wrote: > Marc, > > >> Sorry folks, my fault ... hit the 'accept' button too fast >> > > So, was "Wilbur" really as attractive as she claimed? In all seriousness, it's actually a pretty clever spam ploy. 1) Spam mailing lists of nerdy, desperate guys with

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Tom Lane
Lukas Smith <[EMAIL PROTECTED]> writes: > Jochem van Dieten wrote: >> make the session handler smarter? And if you can't do that, put some >> logic in the session table that turns an update without changes into a >> no-op? > err isnt that one the job of the database? No. That idea has been sugge

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Lukas Smith
Greg Stark wrote: There are other solutions too. I never used DB2 but I was led to believe they used their transaction log to retrieve old versions of the records. Someone else here claimed DB2 didn't implement MVCC at all so perhaps that's wrong though. I would be suprised giving this paper:

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Diogo Biazus
Agree, the project must choose one path as the starting point. But the two options can be given in the long run.I still think that as a starting point the functions inside the database are a good option.The reasons are: - using SQL to agregate and transform data in any way from the logs.- it's eas

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Tom Lane
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > On 6/22/06, Tom Lane <[EMAIL PROTECTED]> wrote: >> Yes it would. The most obvious point is that memory management and >> error handling conventions inside the backend are quite different from >> what you'd expect to employ in a standalone program. >

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > The Oracle design has got other drawbacks: if you need to access a row > version other than than the very latest, you need to go searching in the > rollback segments for it. This is slow (no index help) Just for the record, if i understood correctly -- th

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Heikki Linnakangas
On Thu, 22 Jun 2006, Jonah H. Harris wrote: On 6/22/06, Rod Taylor <[EMAIL PROTECTED]> wrote: If you INSERT into multiple partitions (by time -- say one per minute) and TRUNCATE periodically (30 minute old partitions for 30 minute expiry) it works much better. Expiring the session is quite fast

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Magnus Hagander
> > item #2: Is dllinit.c GPL code? > > The file dllinit.c, located in the src/utils directory > documents the > > author as Mumit Khan. Did Mumit Khan contribute this code > and did he > > contribute it for distribution under the PostgreSQL license? If I > > read correctly, the name stamp i

Re: [HACKERS] let's meet

2006-06-22 Thread Marc G. Fournier
On Thu, 22 Jun 2006, Josh Berkus wrote: Marc, Sorry folks, my fault ... hit the 'accept' button too fast So, was "Wilbur" really as attractive as she claimed? I don't know, Bruce mentioned to me that he wanted to meet "him", so we'll have to wait until they have their get-together >:) -

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Rod Taylor
On Thu, 2006-06-22 at 13:42 -0400, Jonah H. Harris wrote: > On 6/22/06, Rod Taylor <[EMAIL PROTECTED]> wrote: > > If you INSERT into multiple partitions (by time -- say one per minute) > > and TRUNCATE periodically (30 minute old partitions for 30 minute > > expiry) it works much better. Expiring t

Re: [HACKERS] Going for "all green" buildfarm results

2006-06-22 Thread Tom Lane
Neil Conway <[EMAIL PROTECTED]> writes: > On Thu, 2006-06-22 at 12:52 -0400, Andrew Dunstan wrote: >> I believe our supported version is still 2.5.4, which is >> what all my linux systems have. > Its not clear to me why some people have such antipathy toward recent > flex releases, but if our only

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Rod Taylor <[EMAIL PROTECTED]> wrote: If you INSERT into multiple partitions (by time -- say one per minute) and TRUNCATE periodically (30 minute old partitions for 30 minute expiry) it works much better. Expiring the session is quite fast as well since they'll go away with the trunca

Re: [HACKERS] let's meet

2006-06-22 Thread Josh Berkus
Marc, > Sorry folks, my fault ... hit the 'accept' button too fast So, was "Wilbur" really as attractive as she claimed? -- Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--- TIP 6: explain analyze is your friend

Re: [HACKERS] Going for "all green" buildfarm results

2006-06-22 Thread Neil Conway
On Thu, 2006-06-22 at 12:52 -0400, Andrew Dunstan wrote: > I believe our supported version is still 2.5.4, which is > what all my linux systems have. Its not clear to me why some people have such antipathy toward recent flex releases, but if our only supported flex version is 2.5.4, I think this s

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
>> > You mean systems that are designed so exactly, that they can't take >> 10% >> > performance change ? >> >> No, that's not really the point, performance degrades over time, in one >> minute it degraded 10%. >> >> The update to session ratio has a HUGE impact on PostgreSQL. If you have >> a >> t

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Tom Lane <[EMAIL PROTECTED]> wrote: Yes it would. The most obvious point is that memory management and error handling conventions inside the backend are quite different from what you'd expect to employ in a standalone program. No, this wouldn't really be that hard, especially if he

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Rod Taylor
> > You mean systems that are designed so exactly, that they can't take 10% > > performance change ? > > No, that's not really the point, performance degrades over time, in one > minute it degraded 10%. > > The update to session ratio has a HUGE impact on PostgreSQL. If you have a > thousand acti

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> Christopher Browne <[EMAIL PROTECTED]> writes: >> After a long battle with technology, [EMAIL PROTECTED] ("Mark >> Woodward"), an earthling, wrote: >>> Not true. Oracle does not seem to exhibit this problem. > >> Oracle suffers a problem in this regard that PostgreSQL doesn't; in >> Oracle, rollb

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Tom Lane
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > I think it should certainly be able to run on it's own, but it > wouldn't be that hard to extend the functions so that they were usable > from within the database or vice-versa. Yes it would. The most obvious point is that memory management and erro

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Rod Taylor
> > Here we have for example some tables which are frequently updated but > > contain >100 million rows. Vacuuming that takes hours. And the dead row > > candidates are the ones which are updated again and again and looked up > > frequently... > > This demonstrates that "archival" material and "ac

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread D'Arcy J.M. Cain
On Thu, 22 Jun 2006 19:01:38 +0200 Lukas Smith <[EMAIL PROTECTED]> wrote: > Jochem van Dieten wrote: > > > make the session handler smarter? And if you can't do that, put some > > logic in the session table that turns an update without changes into a > > no-op? > > err isnt that one the job of t

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Tom Lane <[EMAIL PROTECTED]> wrote: The Oracle design has got other drawbacks: if you need to access a row version other than than the very latest, you need to go searching in the rollback segments for it. There are ways to implement this functionality without implementing it exactl

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Chris Browne
[EMAIL PROTECTED] (Csaba Nagy) writes: >> > [...] >> > There has to be a more linear way of handling this scenario. >> >> So vacuum the table often. > > Good advice, except if the table is huge :-) ... Then the table shouldn't be designed to be huge. That represents a design error. > Here we h

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Lukas Smith
Jochem van Dieten wrote: make the session handler smarter? And if you can't do that, put some logic in the session table that turns an update without changes into a no-op? err isnt that one the job of the database? regards, Lukas ---(end of broadcast)-

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Tom Lane <[EMAIL PROTECTED]> wrote: it'd make it impossible to use the viewer to work on extracting data from a failed cluster; which is, at least in my mind, one of the primary use-cases for the thing. While I too see this as something which could be used for this outside the datab

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Lukas Smith
Tom Lane wrote: Basically there's no free lunch: if you want the benefits of MVCC it's going to cost you somewhere. In the Postgres design you pay by having to do VACUUM pretty often for heavily-updated tables. I don't think that decision is fundamentally wrong --- the attractive thing about i

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jochem van Dieten
On 6/22/06, Mark Woodward wrote: (..) thousand active sessions (..) If an active user causes a session update once a second (..) Generally speaking, sessions aren't updated when they change, they are usually updated per HTTP request. The data in a session may not change, but the session hand

Re: [HACKERS] Going for "all green" buildfarm results

2006-06-22 Thread Andrew Dunstan
Mark Wong wrote: Now why are we failing on 7.3? What version of flex do you have? If it's too modern we'll just need to take 7.3 out of the cobra and stoat rotations - we'd really only make supercritical fixes on that branch these days. Flex is 2.5.33 on both systems. I'm assuming that

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Tom Lane
Christopher Browne <[EMAIL PROTECTED]> writes: > After a long battle with technology, [EMAIL PROTECTED] ("Mark Woodward"), an > earthling, wrote: >> Not true. Oracle does not seem to exhibit this problem. > Oracle suffers a problem in this regard that PostgreSQL doesn't; in > Oracle, rollbacks ar

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Tom Lane
"Diogo Biazus" <[EMAIL PROTECTED]> writes: > The idea I've been discussing with Simon Riggs is to create a set of > functions that can be called from within the database. I'd question that at the very start. I don't see any strong reason to do it that way, and as you admit further down it'd make

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> Ühel kenal päeval, N, 2006-06-22 kell 10:20, kirjutas Jonah H. Harris: >> On 6/22/06, Alvaro Herrera <[EMAIL PROTECTED]> wrote: >> > > Hmm, OK, then the problem is more serious than I suspected. >> > > This means that every index on a row has to be updated on every >> > > transaction that modif

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Hannu Krosing <[EMAIL PROTECTED]> wrote: I guess that MySQL on its original storage does that, but they allow only one concurrent update per table and no transactions. More like practically every commercial database. As ~97% of transactions commit (yes, some can argue that number),

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> Ühel kenal päeval, N, 2006-06-22 kell 09:59, kirjutas Mark Woodward: >> > After a long battle with technology, [EMAIL PROTECTED] ("Mark >> > Woodward"), an earthling, wrote: >> >>> Clinging to sanity, [EMAIL PROTECTED] ("Mark Woodward") mumbled >> into >> > It pointed to *ALL* the versions. >>

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Hannu Krosing
Ühel kenal päeval, N, 2006-06-22 kell 10:20, kirjutas Jonah H. Harris: > On 6/22/06, Alvaro Herrera <[EMAIL PROTECTED]> wrote: > > > Hmm, OK, then the problem is more serious than I suspected. > > > This means that every index on a row has to be updated on every > > > transaction that modifies that

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Hannu Krosing
Ühel kenal päeval, N, 2006-06-22 kell 09:59, kirjutas Mark Woodward: > > After a long battle with technology, [EMAIL PROTECTED] ("Mark > > Woodward"), an earthling, wrote: > >>> Clinging to sanity, [EMAIL PROTECTED] ("Mark Woodward") mumbled into > > It pointed to *ALL* the versions. > > Hmm, OK,

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
[ redirecting to -hackers, as I see no need for this to be a core issue ] Charles Comiskey <[EMAIL PROTECTED]> writes: > Hello, > I've recently looked through the PostgreSQL code and a couple of questions > surfaced. I was hoping someone here may be able to answer them. Two have > links to pos

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mario Weilguni
Am Donnerstag, 22. Juni 2006 16:09 schrieb Csaba Nagy: > > > [...] > > > There has to be a more linear way of handling this scenario. > > > > So vacuum the table often. > > Good advice, except if the table is huge :-) > > Here we have for example some tables which are frequently updated but > conta

  1   2   >