Re: [HACKERS] GSSAPI and V2 protocol
On Tue, 5 Feb 2008, Tom Lane wrote: The problem seems to be that AuthenticationGSSContinue messages carry a variable-length payload, and the V2 protocol doesn't really cope with that because it doesn't have a message length word. 1. If the GSSContinue payload is self-identifying about its length, qwe could teach fe-connect.c how to determine that. The GSS data is supposed to be opaque to the caller, so this doesn't seem likely or a good idea. 2. We could retroactively redefine the contents of AuthenticationGSSContinue as carrying a length word after the authentication type code, but only in V2 protocol (so as not to break existing working cases). This is pretty ugly but certainly possible. I see no harm in doing this. What's there now can't work and the change is self contained. Is there any problem with the password message taking a String datatype instead of Byte[n] with a null byte? Kris Jurka ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Re: [COMMITTERS] pgsql: configure tag'd 8.3.0 and built witih autoconf 2.59
Marc G. Fournier wrote: Actually, branch in one to two weeks has been the status quo almost since day one ... not that I'm against branch on release, I'm only saying that we've followed this same procedure on branching since ... forever. That is incorrect. See earlier in this thread. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
[HACKERS] PostgreSQL 8.4 development plan
Hackers, As you know we've finally released PostgreSQL 8.3, after a development cycle that lasted well over a year despite our original plans for a 6 month cycle. The core team are aware that there are a number of factors that contributed to this slippage: - Lack of prompt and early review of patches. - A significant rise in the number and complexity of patches submitted. - Prioritising completion of incomplete patches over meeting the timetable. In the 8.4 development cycle we would like to try a new style of development, designed to keep the patch queue to a limited size and to provide timely feedback to developers on the work they submit. To do this we will replace the traditional 'feature freeze' with a series of 'commit fests' throughout the development cycle. The idea of commit fests was discussed last October in -hackers, and it seemed to meet with general approval. Whenever a commit fest is in progress, the focus will shift from development to review, feedback and commit of patches. Each fest will continue until all patches in the queue have either been committed to the CVS repository, returned to the author for additional work, or rejected outright, and until that has happened, no new patches will be considered. Of course, individual developers are free to continue working on their patches throughout the fest, but we encourage everyone to do what they can to help work through the patch queue. We feel that this idea can only be successful if the whole development community is willing to focus on patch review during the commit fests, in the same way that everyone is expected to focus on testing during beta period. The proposed timetable for the cycle is as follows: 1st March 2008 - commit fest begins 1st May 2008 - commit fest begins 1st July 2008 - commit fest begins 1st September 2008 - commit fest begins 1st November 2008 - final commit fest begins 1st January 2009 - beta 1 1st March 2009 - 8.4.0 release Note the lack of any 'feature freeze' date as such. However, any significant feature patches not submitted by 1st November will clearly not make it into 8.4. The hope here is that we will not have enormous, previously unreviewed patches landing on us at the end of October --- if that happens, we'll be back in the same position we were in at 8.3 feature freeze. Although this schedule allows for the final commit fest to take a good deal of time, we'll reserve the right to reject patches that are too large to be reviewed in a timely fashion. We want to encourage people to do development of large features in an incremental fashion, with a new increment landing during each commit fest. Regards, Dave (on behalf of the core team) -- Dave Page PostgreSQL Core Team ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] GSSAPI and V2 protocol
On Wed, Feb 06, 2008 at 02:57:39AM -0500, Kris Jurka wrote: On Tue, 5 Feb 2008, Tom Lane wrote: The problem seems to be that AuthenticationGSSContinue messages carry a variable-length payload, and the V2 protocol doesn't really cope with that because it doesn't have a message length word. 1. If the GSSContinue payload is self-identifying about its length, qwe could teach fe-connect.c how to determine that. The GSS data is supposed to be opaque to the caller, so this doesn't seem likely or a good idea. Yeah, agreed, that seems like a very fragile idea. 2. We could retroactively redefine the contents of AuthenticationGSSContinue as carrying a length word after the authentication type code, but only in V2 protocol (so as not to break existing working cases). This is pretty ugly but certainly possible. I see no harm in doing this. What's there now can't work and the change is self contained. Is there any problem with the password message taking a String datatype instead of Byte[n] with a null byte? I agree that this is probabliy the best way, if we can do it. But you do raise a good point - the message that goes the other way can certainly contain embedded NULLs. //Magnus ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] PostgreSQL 8.4 development plan
On Wed, 2008-02-06 at 08:56 +, Dave Page wrote: Hackers, +1 Very much in favour. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [BUGS] BUG #3909: src\tools\msvc\clean.bat clears parse.h file
On Tue, Feb 05, 2008 at 09:47:16PM +, Dave Page wrote: On Feb 5, 2008 5:56 PM, Magnus Hagander [EMAIL PROTECTED] wrote: Dave Page wrote: On Feb 5, 2008 3:24 PM, Magnus Hagander [EMAIL PROTECTED] wrote: On Mon, Jan 28, 2008 at 06:27:05PM +, Pavel Golub wrote: I think a better solution is to add a parameter to clean.bat to make it work like make clean does. So you'd to clean when you mean make clean, and clean dist when you mean make distclean. Thoughts on this? Pretty sure I griped at you before about this, because when it removes it, it fails to rebuild it the next time round, even on in an env like mine which can build from CVS perfectly well. More than once I've had to unpack the tarball again having run a clean. Uh, that would be a different issue, and I thought that one was fixed. I certainly clean/rebuild a lot, and it works just fine with the stuff coming out of cvs. The problem comes from if you remove *one* of the files but not *all* of them. We only trigger on one of them (unlike the makefile which triggers on all). But as long as they are all removed, it should be ok. Can you confirm if you actually still have that problem with 8.3.0? Yup - run clean.bat, and then build.bat and the resulting build fails because parser/parse.h is missing. Thats with the 8.3.0 tarball, which built just fine prior to running clean. Bison and flex are in the path. I've managed to reproduce this now, and it's way more broken than I thought. In fact, clean.bat simply does not work. It seems the batch processor falls over *badly* when dealing with so many call statements. I put in some echo 1, echo 2 etc to see in which order i runs the bat file. I put them in from top to bottom, but the execution order came out as 0,5,6,7,8,1,5,6,7,8,7,8,2,3,4,5,6,7,8. Which obviously is less than perfect. I've redone a clean.bat now that inlines the deletes, and it seems to be working better. I'll also add the clean dist stuff so it's possible to clean/rebuild without bison/flex, and then commit this. //Magnus ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] PostgreSQL 8.4 development plan
On Wed, Feb 06, 2008 at 08:56:51AM +, Dave Page wrote: Hackers, Looks great and well-thought through. Let's hope it works out! I assume you'll be committing this info to the developer section on the website? //Magnus ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
[HACKERS] HeapTupleSatisfies micro tuning
I note that in HeapTupleSatisfies... we call TransactionIdIsCurrentTransactionId() explicitly and then call it again in TransactionIdIsInProgress(). Which means we also end up checking twice whether each xid is normal also. It would seem easier to make a check TransactionIdIsRecent() early and if true then check TransactionIdIsCurrentTransactionId() and then check RecentTransactionIdIsInProgress(). That would avoid the multiple checks, as well as save a few cycles since the IsCurrent check can be expensive if we have many subtransactions. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Why are we waiting?
On Mon, 2008-02-04 at 19:46, Simon Riggs wrote: We've got various facilities now for looking at LWLock waits, but I'd like to have more information about the *reasons* for lock waits. I know its possible to get backtraces in Dtrace at various tracepoints but that can be fairly hard to interpret. Simon, A couple of guys here at Sun have started looking at trying to improve the scaling on Sun's T2000 Niagara servers (single core, 32 HW-threads). We mostly use various DTrace scripts to monitor locking usage. We'd be happy to share results, probes and DTrace scripts for monitoring if you like. So far we've found lock contention (especially for the ProcArrayLock) to be the likely source for our inability to saturate the CPU with a TPC-C like OLTP load with 1000 users. The following shows average time spent in and waiting for exclusive ProcArrayLock vs the time used in the transaction through commit, (i.e., up until the exclusive ProcArrayLock acquire to update the PGPROC setting transaction no longer running): # ./pg-lwlock-procarray.d $(pgrep -n postgres) postgres`LWLockAcquire postgres`ProcArrayEndTransaction+0x10 postgres`CommitTransaction+0xf0 postgres`CommitTransactionCommand+0x90 postgres`finish_xact_command+0x60 postgres`exec_execute_message+0x3b4 postgres`PostgresMain+0x13a0 postgres`BackendRun+0x27c postgres`BackendStartup+0xe0 postgres`ServerLoop+0x1a0 postgres`PostmasterMain+0xcbc postgres`main+0x1d8 postgres`_start+0x108 23 avg lwlock acquire service time [ns] 193051989 transaction-commit [count] 23 transaction-start [count] 23 avg transaction time [ns] 12763079 The stack trace shows that the only time the lock is acquired exclusively is from the call to ProcArrayEndTransaction() in CommitTransaction(). Also, an interesting observation is that the hot locks seem to have changed from v8.2 to v8.3, making the ProcArrayLock more contended. See the following outputs: PostgreSQL 8.2 (32-bit): -bash-3.00# ./825_lwlock_wait.d Lock IdMode Count FirstLockMgrLock Shared 2 ProcArrayLock Shared 209 XidGenLock Exclusive1030 XidGenLock Shared1215 WALInsertLock Exclusive3942 CLogControlLock Shared4113 ProcArrayLock Exclusive6929 WALWriteLock Exclusive 17155 CLogControlLock Exclusive 128182 Lock Id Combined Time (ns) FirstLockMgrLock24705 WALInsertLock 79644210 XidGenLock179886846 ProcArrayLock 1486950738 WALWriteLock 18425400504 CLogControlLock1507388036453 PostgreSQL 8.3 (64-bit): -bash-3.00# ./83_lwlock_wait.d Lock IdMode Count SInvalLock Exclusive 1 WALWriteLock Exclusive 1 SInvalLock Shared 20 CLogControlLock Shared 534 CLogControlLock Exclusive 845 XidGenLock Exclusive1140 ProcArrayLock Shared1821 WALInsertLock Exclusive 17043 ProcArrayLock Exclusive 49762 Lock IdMode Combined Time (ns) SInvalLock Exclusive17216 SInvalLock Shared 531129 WALWriteLock Exclusive 2003064 CLogControlLock Shared 61325819 CLogControlLock Exclusive 73509195 XidGenLock Exclusive929477929 WALInsertLock Exclusive 17941476941 ProcArrayLock Shared 31114676303 ProcArrayLock Exclusive 888356047549 Regards, Staale Smedseng Database Technology Group, Sun Microsystems ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] PostgreSQL 8.4 development plan
On Feb 6, 2008 1:49 PM, Magnus Hagander [EMAIL PROTECTED] wrote: On Wed, Feb 06, 2008 at 08:56:51AM +, Dave Page wrote: Hackers, Looks great and well-thought through. Let's hope it works out! I assume you'll be committing this info to the developer section on the website? It's on the developer wiki. /D ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] PostgreSQL 8.4 development plan
On Wed, Feb 06, 2008 at 02:42:35PM +, Dave Page wrote: On Feb 6, 2008 1:49 PM, Magnus Hagander [EMAIL PROTECTED] wrote: On Wed, Feb 06, 2008 at 08:56:51AM +, Dave Page wrote: Hackers, Looks great and well-thought through. Let's hope it works out! I assume you'll be committing this info to the developer section on the website? It's on the developer wiki. Good start. /me thinks it should be on the website. We've usually announced our feature freeze dates there... (in less details, sure, but something there) //Magnus ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Why are we waiting?
On Wed, 2008-02-06 at 15:30 +0100, Staale Smedseng wrote: On Mon, 2008-02-04 at 19:46, Simon Riggs wrote: We've got various facilities now for looking at LWLock waits, but I'd like to have more information about the *reasons* for lock waits. I know its possible to get backtraces in Dtrace at various tracepoints but that can be fairly hard to interpret. Thanks for this Staale. Lock IdMode Count ProcArrayLock Shared1821 ProcArrayLock Exclusive 49762 Lock IdMode Combined Time (ns) ProcArrayLock Shared 31114676303 ProcArrayLock Exclusive 888356047549 Which looks like a mean service time of 17ms for X lock waiters. What is the frequency distribution of lock wait time on ProcArrayLock? Does the distribution vary over time? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] PostgreSQL 8.4 development plan
On Feb 6, 2008 2:44 PM, Magnus Hagander [EMAIL PROTECTED] wrote: Good start. /me thinks it should be on the website. We've usually announced our feature freeze dates there... (in less details, sure, but something there) Feel free - you've been hacking that recently! /D ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Why are we waiting?
Staale Smedseng [EMAIL PROTECTED] writes: The stack trace shows that the only time the lock is acquired exclusively is from the call to ProcArrayEndTransaction() in CommitTransaction(). I'm not sure but I think that's only true in 8.3. As I understood it in 8.2 transaction start also needed the exclusive lock. Also, an interesting observation is that the hot locks seem to have changed from v8.2 to v8.3, making the ProcArrayLock more contended. See the following outputs: PostgreSQL 8.2 (32-bit): ... PostgreSQL 8.3 (64-bit): ... I'm not sure 32-bit and 64-bit cases are going to be directly comparable. We could have a problem with cache line aliasing on only one or the other for example. But that is a pretty striking difference. Does the 8.3 run complete more transactions in that time? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support! ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
[HACKERS] pg_dump additional options for performance
pg_dump allows you to specify -s --schema-only, or -a --data-only. The -s option creates the table, as well as creating constraints and indexes. These objects need to be dropped prior to loading, if we are to follow the performance recommendations in the docs. But the only way to do that is to manually edit the script to produce a cut down script. So it would be good if we could dump objects in 3 groups 1. all commands required to re-create table 2. data 3. all commands required to complete table after data load My proposal is to provide two additional modes: --schema-pre-load corresponding to (1) above --schema-post-load corresponding to (3) above This would then allow this sequence of commands pg_dump --schema-pre-load pg_dump --data-only pg_dump --schema-post-load to be logically equivalent, but faster than pg_dump --schema-only pg_dump --data-only both forms of which are equivalent to just pg_dump [Assuming data isn't changing between invocations...] -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] [BUGS] BUG #3909: src\tools\msvc\clean.bat clears parse.h file
On Tue, Feb 05, 2008 at 09:47:16PM +, Dave Page wrote: On Feb 5, 2008 5:56 PM, Magnus Hagander [EMAIL PROTECTED] wrote: Dave Page wrote: On Feb 5, 2008 3:24 PM, Magnus Hagander [EMAIL PROTECTED] wrote: On Mon, Jan 28, 2008 at 06:27:05PM +, Pavel Golub wrote: I think a better solution is to add a parameter to clean.bat to make it work like make clean does. So you'd to clean when you mean make clean, and clean dist when you mean make distclean. Thoughts on this? Pretty sure I griped at you before about this, because when it removes it, it fails to rebuild it the next time round, even on in an env like mine which can build from CVS perfectly well. More than once I've had to unpack the tarball again having run a clean. Uh, that would be a different issue, and I thought that one was fixed. I certainly clean/rebuild a lot, and it works just fine with the stuff coming out of cvs. The problem comes from if you remove *one* of the files but not *all* of them. We only trigger on one of them (unlike the makefile which triggers on all). But as long as they are all removed, it should be ok. Can you confirm if you actually still have that problem with 8.3.0? Yup - run clean.bat, and then build.bat and the resulting build fails because parser/parse.h is missing. Thats with the 8.3.0 tarball, which built just fine prior to running clean. Bison and flex are in the path. Fixed version applied to HEAD. //Magnus ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
[HACKERS] Segment Visibility Map for VACUUM
Earlier, I proposed a Segment Visibility Map as part of my design of new partitioning feature proposals. This idea has been spun-off into a completely **separate** proposal because SVM has merit enough to be worthy of implementation on its own: http://developer.postgresql.org/index.php/Segment_Visibility_Map (full text of this is copied below also) This then allows it to be compared directly with other proposals - Dead Space Map (Itagaki) - Visibility Maps (Heikki) and everybody else down the years with a variant on this idea ISTM that SVM would offer many of the advantages of the other approaches, with a less invasive, lower contention design. The implementation is fairly straightforward also. I'd like to get this sorted out as one of the first things in 8.4 development, so that all of the other *potential* uses of the visibility information can also be considered and possibly implemented in the same release. SEGMENT VISIBILITY MAP = Use Case = Larger tables frequently need VACUUMing, but only over a small range of blocks. It is common for much a table to be read-only, so if we can identify which parts are read-only then we can improve performance of VACUUM, as we as other utilities. = Segment Visibility Map = We would keep a dynamic visibility map at *segment* level, showing which segments have all rows as 100% visible. This is known as the Segment Visibility Map (SVM). We have at least two bits for each segment in the table, rather than each block. The two bits allow us to represent four states: * read_write * read_only * read_only_pending (explained later) * read_only_frozen Some additional states that have been discussed are * explicitly marked read only * clustered * compressed * off-line (indicates high cost to access) We should allow for 1 byte (8 bits) per segment. That's possibly small enough to store directly on pg_class, even for fairly large tables. However we want to avoid bloat when the SVM changes, plus we may need to handle some very big tables. So we would store SVM as a single column, in a table pg_svm with storage attribute MAIN, so it is inline, compressible. A segment is a range of blocks, size of which can be altered as required. Initial suggestion is that a segment would be 1GB in size, same as a physical data segment. We might allow this to be user settable, or we might automatically adjust it to minimise the overhead and maximise the usefulness. It would be possible to change the SVM segment size dynamically, recalculating the bit positions accordingly. If that was desirable to optimise the overhead and benefit from this approach. = Access to the SVM = Since the map is very small and hardly ever changes for a table we can cache it easily on each backend. If the map does change we can perform a relcache invalidation to make everybody re-read the map. (Few improvements needed there also, but not relevant here). No dynamic shared memory cache is required because any concurrent changes to the table would be ignored by a scan anyway, so it doesn't matter if an INSERT, UPDATE or DELETE occurs while we are scanning. Any new scans that start will attempt to lock the table and then perform a rel cache check before continuing. So the visibility will be set correctly for *that* scan at least. In most cases the visibility map can be summarised as a single boolean to show whether *any* 100% visible segments exist. That makes accessing the map very cheap in the common, unset case. This would be an additional boolean entry on the Relation struct. The check to see if the rel cache has changed is essentially free, since we do it already when we grab a lock. Since INSERT, UPDATE and DELETE only spoil the cache, they don't need to have a completely up-to-date picture. (That aspect may change). If we want backends running non-utility commands to see the SVM then we would need to check the rel cache each time we access the table, not just each time we lock a table. We need not do that in all cases, only those that would benefit from knowing SVM information. = UnSetting the Segment Visibility Map = The SVM will be unset, i.e. set to read_write if an INSERT, UPDATE or DELETE occurs within a segment marked read_only or read_only_pending. This catalog change can be handled with a non-transactional overwrite - the bit always exists already, so the overwritten data is always same size. That's pessimistic, since it resets the state even if the UPDATE/DELETE aborts, but its better than holding the row locked until the transaction completes, which might prevent other things from happening. Or maybe we could do that at transaction commit. DML commands would first check the boolean rel.has_read_only_segments to see if any non read_write segments exist. If so, then we would calculate the segment being written to by the DML operation then set the state accordingly. The relcache invalidation caused by making a segment read_only_pending can flush the rd_targBlock for the
Re: [HACKERS] pg_dump additional options for performance
Simon Riggs wrote: My proposal is to provide two additional modes: --schema-pre-load corresponding to (1) above --schema-post-load corresponding to (3) above This would then allow this sequence of commands pg_dump --schema-pre-load pg_dump --data-only pg_dump --schema-post-load to be logically equivalent, but faster than pg_dump --schema-only pg_dump --data-only It would actually be better, as well as faster, because it will be guaranteed to work :-) There are known cases where schema-only followed by data-only fails. One more note: we need to make sure the corresponding new modes are also added to pg_restore. cheers andrew ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] pg_dump additional options for performance
On Wed, 2008-02-06 at 16:41 +0100, Magnus Hagander wrote: Where would you load primary keys and such contrants? Pre- or post dump? I think the case could be made for either one... Post dump. If the constraint was successfully in place when we performed the dump then it should work successfully after the load. I like your syntax/idea as well. I think that would be worth doing in addition for when you want fine grained control. (I'd add the ability to dump any named object, allowing you to dump individual indexes, functions, etc..) I don't think the two syntaxes compete. I want to be able to say everything before, data and everything after without having to remember to specify --what=tables,views,sequences,kitchensink etc. Especially since you probably won't notice you've missed out an object type (e.g. sequences) until you try to run an application. Too late! -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] pg_dump additional options for performance
Simon wrote: My proposal is to provide two additional modes: --schema-pre-load corresponding to (1) above --schema-post-load corresponding to (3) above Sounds nice. For a large schema we might rather want one switch that dumps 2 files, no ? Probably also better from a mvcc perspective. Andreas ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] PostgreSQL 8.4 development plan
Am Mittwoch, 6. Februar 2008 schrieb Andrew Dunstan: I would like to see this tied down some more. The time for the commit fests is too open ended. I think we should say something like All commit fests will run no more than two weeks, except for the final commit fest which can run for one month. Something along those lines was discussed, but we feel that because we have no experience with how commit fests will run, it is unwise to specify that much detail already. It is quite possible that as we gain experience with the process the timeline will be clarified. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] PostgreSQL 8.4 development plan
This all sounds very promising. On Feb 6, 2008 7:56 PM, Dave Page [EMAIL PROTECTED] wrote: Each fest will continue until all patches in the queue have either been committed to the CVS repository, returned to the author for additional work, or rejected outright, and until that has happened, no new patches will be considered. So does this mean we will have a new patches awaiting the next review cycle queue alongside the patches awaiting review queue? Just thinking that we'll need somewhere to park the new patches which roll in during a commit fest. Or, you know, start using an actual development tracker =) Cheers BJ ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] PostgreSQL 8.4 development plan
Dave Page wrote: On Feb 6, 2008 3:57 PM, Andrew Dunstan [EMAIL PROTECTED] wrote: I would like to see this tied down some more. The time for the commit fests is too open ended. I think we should say something like All commit fests will run no more than two weeks, except for the final commit fest which can run for one month. I think thats one of the problems - without knowing what patches are going to come in, or how many there will be, we have no way of knowing how long each fest will take. What this does mean though is that we continuously feedback to developers and keep the patch queue down - kinda like checkpoint smoothing I guess. I would rather set a target and modify it if necessary based on experience than have none at all. The danger of not doing so is that we'll be in almost constant 'commit fest' mode. cheers andrew ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] pg_dump additional options for performance
Simon Riggs wrote: On Wed, 2008-02-06 at 16:41 +0100, Magnus Hagander wrote: I don't think the two syntaxes compete. I want to be able to say everything before, data and everything after without having to remember to specify --what=tables,views,sequences,kitchensink etc. Especially since you probably won't notice you've missed out an object type (e.g. sequences) until you try to run an application. Too late! Is this not a job for the -l / -L mode of pg_restore, but ported to pg_dump? With a bit of tweaking ti the format you could do something like: pg_dump -l mydb | grep BEFORE obj_list.txt pg_dump -L obj_list.txt mydb mydb.before.schema -- Richard Huxton Archonet Ltd ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] PostgreSQL 8.4 development plan
Dave Page wrote: Hackers, As you know we've finally released PostgreSQL 8.3, after a development cycle that lasted well over a year despite our original plans for a 6 month cycle. The core team are aware that there are a number of factors that contributed to this slippage: - Lack of prompt and early review of patches. - A significant rise in the number and complexity of patches submitted. - Prioritising completion of incomplete patches over meeting the timetable. In the 8.4 development cycle we would like to try a new style of development, designed to keep the patch queue to a limited size and to provide timely feedback to developers on the work they submit. To do this we will replace the traditional 'feature freeze' with a series of 'commit fests' throughout the development cycle. The idea of commit fests was discussed last October in -hackers, and it seemed to meet with general approval. Whenever a commit fest is in progress, the focus will shift from development to review, feedback and commit of patches. Each fest will continue until all patches in the queue have either been committed to the CVS repository, returned to the author for additional work, or rejected outright, and until that has happened, no new patches will be considered. Of course, individual developers are free to continue working on their patches throughout the fest, but we encourage everyone to do what they can to help work through the patch queue. We feel that this idea can only be successful if the whole development community is willing to focus on patch review during the commit fests, in the same way that everyone is expected to focus on testing during beta period. The proposed timetable for the cycle is as follows: 1st March 2008 - commit fest begins 1st May 2008 - commit fest begins 1st July 2008 - commit fest begins 1st September 2008 - commit fest begins 1st November 2008 - final commit fest begins 1st January 2009 - beta 1 1st March 2009 - 8.4.0 release Note the lack of any 'feature freeze' date as such. However, any significant feature patches not submitted by 1st November will clearly not make it into 8.4. The hope here is that we will not have enormous, previously unreviewed patches landing on us at the end of October --- if that happens, we'll be back in the same position we were in at 8.3 feature freeze. Although this schedule allows for the final commit fest to take a good deal of time, we'll reserve the right to reject patches that are too large to be reviewed in a timely fashion. We want to encourage people to do development of large features in an incremental fashion, with a new increment landing during each commit fest. Regards, Dave (on behalf of the core team) I would like to see this tied down some more. The time for the commit fests is too open ended. I think we should say something like All commit fests will run no more than two weeks, except for the final commit fest which can run for one month. If we can't make that work then the whole idea is probably in trouble anyway. Another possibility which might help allocating reviewers to projects (especially large projects) earlier in the process. cheers andrew ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] PostgreSQL 8.4 development plan
On Feb 6, 2008 3:57 PM, Andrew Dunstan [EMAIL PROTECTED] wrote: I would like to see this tied down some more. The time for the commit fests is too open ended. I think we should say something like All commit fests will run no more than two weeks, except for the final commit fest which can run for one month. I think thats one of the problems - without knowing what patches are going to come in, or how many there will be, we have no way of knowing how long each fest will take. What this does mean though is that we continuously feedback to developers and keep the patch queue down - kinda like checkpoint smoothing I guess. /D ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] pg_dump additional options for performance
On Wed, Feb 06, 2008 at 03:13:24PM +, Simon Riggs wrote: pg_dump allows you to specify -s --schema-only, or -a --data-only. The -s option creates the table, as well as creating constraints and indexes. These objects need to be dropped prior to loading, if we are to follow the performance recommendations in the docs. But the only way to do that is to manually edit the script to produce a cut down script. So it would be good if we could dump objects in 3 groups 1. all commands required to re-create table 2. data 3. all commands required to complete table after data load My proposal is to provide two additional modes: --schema-pre-load corresponding to (1) above --schema-post-load corresponding to (3) above This would then allow this sequence of commands pg_dump --schema-pre-load pg_dump --data-only pg_dump --schema-post-load to be logically equivalent, but faster than pg_dump --schema-only pg_dump --data-only both forms of which are equivalent to just pg_dump [Assuming data isn't changing between invocations...] I've been considering just this. Another otpion I came up with was a more generic switch where you'd have: pg_dump --what=tables pg_dump --what=indexes,constraints or something like that. Would give more flexibility, but I'm not sure if that's worthwhile. Having the ability to just this filtering that you're talking about would be very handy - I've needed it more than once. Where would you load primary keys and such contrants? Pre- or post dump? I think the case could be made for either one... //Magnus ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Page-at-a-time Locking Considerations
Gregory Stark napsal(a): Simon Riggs [EMAIL PROTECTED] writes: I wonder how hard it would be to shove the clog into regular shared memory pages and let the clock sweep take care of adjusting the percentage of shared mem allocated to the clog versus data pages. I tried to use memory mapped files (mmap) for clog and I think it should be also possible way. I got about 2% better performance, but it needs more testing. Zdenek ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] PostgreSQL 8.4 development plan
Andrew Dunstan [EMAIL PROTECTED] writes: I would rather set a target and modify it if necessary based on experience than have none at all. We felt that we'd like to get a couple of fests under our belts before trying to nail down very many rules. The process will get more formalized later, no doubt, but let's see what the actual problems are before guessing about how to fix them. The original draft listed the first commit fest as being in May, but we added a March fest in part to have a practice run without too much stuff being on the plate. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] PostgreSQL 8.4 development plan
Brendan Jurd [EMAIL PROTECTED] writes: Just thinking that we'll need somewhere to park the new patches which roll in during a commit fest. Bruce has always kept two patch queues, one for the current version and one for the stuff held for the next version. This won't change anything except the labels on the queues. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] PostgreSQL 8.4 development plan
On Feb 6, 2008 4:24 PM, Andrew Dunstan [EMAIL PROTECTED] wrote: I would rather set a target and modify it if necessary based on experience than have none at all. The danger of not doing so is that we'll be in almost constant 'commit fest' mode. Yes, that is something we discussed, and the reason why we used the wording 'proposed timetable for the cycle'. We will adjust the timing if need be, but wanted to start out on a confident note :-) /D ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] PostgreSQL 8.4 development plan
Dave Page wrote: On Feb 6, 2008 4:24 PM, Andrew Dunstan [EMAIL PROTECTED] wrote: I would rather set a target and modify it if necessary based on experience than have none at all. The danger of not doing so is that we'll be in almost constant 'commit fest' mode. Yes, that is something we discussed, and the reason why we used the wording 'proposed timetable for the cycle'. We will adjust the timing if need be, but wanted to start out on a confident note :-) Sometimes I wish we could decide if we want to be wishy or washy ;-) cheers andrew ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] PostgreSQL 8.4 development plan
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: I would rather set a target and modify it if necessary based on experience than have none at all. We felt that we'd like to get a couple of fests under our belts before trying to nail down very many rules. The process will get more formalized later, no doubt, but let's see what the actual problems are before guessing about how to fix them. The original draft listed the first commit fest as being in May, but we added a March fest in part to have a practice run without too much stuff being on the plate. OK, that makes some sense, although I don't know about the not much stuff on the plate. We presumably have quite a lot of stuff in the queue from the last 7 months or so. cheers andrew ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] build environment: a different makefile
Currently trying to enhance the way we can make binaries that run on Solaris. One thing I found was a scalability bottleneck in the use of the ProcArrayLock. (this one has also been reported by a couple of my colleagues). One big user of this lock is GetSnapshotData. After it has taken this lock it does its work and releases it again. While it is holding the lock it is not doing any system calls and the lock holding process is barely preempted. The only way to make this code faster is making the code use less CPU cycles to achieve its goal. One way is having the compiler do some strong code in-lining. The SunStudio compiler we are using fortunately has an option for this. Unfortunately there are restrictions. One restriction I face is its inability to deal with ld -rs. These are used in the build environment to create all the SUBSYS.o object files. I was hoping someone in the community already has a makefile that just creates object files from C-sources directly that I can use to try out the effect of in-lining to the performance of postgres. Any other hints to achieve my goal are welcome too, of-course. Please note that in-lining is done in both the compiler and the linker. Thanks, Paul - Paul van den Bogaard [EMAIL PROTECTED] ISV-E -- ISV Engineering, Opensource Engineering group Sun Microsystems, Inc phone:+31 334 515 918 Saturnus 1 extentsion: x (70)15918 3824 ME Amersfoort mobile: +31 651 913 354 The Netherlands fax:+31 334 515 001 ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] PostgreSQL 8.4 development plan
Andrew Dunstan [EMAIL PROTECTED] writes: Tom Lane wrote: we added a March fest in part to have a practice run without too much stuff being on the plate. OK, that makes some sense, although I don't know about the not much stuff on the plate. We presumably have quite a lot of stuff in the queue from the last 7 months or so. There is, although I think a large fraction of it will get bounced as needs more work, which should reduce the pressure. We'll just be trying to give feedback to let the patch authors move forward, which will not take as much time as actually committing would take. The current queue is http://momjian.postgresql.org/cgi-bin/pgpatches_hold Note that a lot of the bulk is discussion of things that aren't anywhere near committable anyway. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Page-at-a-time Locking Considerations
Zdenek Kotala [EMAIL PROTECTED] writes: I tried to use memory mapped files (mmap) for clog and I think it should be also possible way. I got about 2% better performance, but it needs more testing. If you only got 2% out of it, it's not even worth thinking about how to fix the serious bugs that approach would create (primarily, lack of control over when pages can get flushed to disk). regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Why are we waiting?
What is the frequency distribution of lock wait time on ProcArrayLock? See below for wait time distributions for ProcArrayLock (both shared and exclusive). The time measured is from entry into LWLockAcquire() until return. I've recorded the same data in two different resolutions (ms, and us for the lower part of the distribution). The DTrace script is at the bottom. These results are for 1000 TPC-C like clients, and measured over the 1000 PostgreSQL processes over a period of 10 seconds. Does the distribution vary over time? Hmm... I will have to get back to you on that one. Staale CPU IDFUNCTION:NAME 6 71245 :tick-10sec Total LW_EXCLUSIVE25178 Total Transaction Starts 25679 Total LW_SHARED 107211 LW_SHARED [ms] value - Distribution - count 0 | 0 0 |@@ 100565 10 | 1 20 | 0 30 | 0 40 | 0 50 | 0 60 | 0 70 | 6 80 | 304 90 |@1370 100 |@2685 110 |@1731 120 | 307 130 | 13 140 | 0 150 | 0 160 | 5 170 | 4 180 | 26 190 | 36 200 | 24 210 | 61 220 | 49 230 | 15 240 | 0 250 | 0 260 | 0 270 | 0 280 | 2 290 | 2 300 | 0 310 | 2 320 | 1 330 | 1 340 | 0 350 | 0 360 | 0 370 | 0 380 | 0 390 | 0 400 | 1 410 | 0 LW_EXCLUSIVE [ms] value - Distribution - count 0 | 0 0 |@@ 1565 10 | 0 20 | 0 30 | 0 40 | 0 50 | 0 60 | 0 70 | 16 80 |@894 90 |@@@ 4108 100 |@8090 110 | 4863
Re: [HACKERS] Why are we waiting?
On Wed, 2008-02-06 at 18:44 +0100, Staale Smedseng wrote: What is the frequency distribution of lock wait time on ProcArrayLock? See below for wait time distributions for ProcArrayLock (both shared and exclusive). The time measured is from entry into LWLockAcquire() until return. I've recorded the same data in two different resolutions (ms, and us for the lower part of the distribution). The DTrace script is at the bottom. These results are for 1000 TPC-C like clients, and measured over the 1000 PostgreSQL processes over a period of 10 seconds. Thanks! Interesting resonance pattern. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] PostgreSQL 8.4 development plan
On Wednesday 06 February 2008 09:09, Tom Lane wrote: Brendan Jurd [EMAIL PROTECTED] writes: Just thinking that we'll need somewhere to park the new patches which roll in during a commit fest. Bruce has always kept two patch queues, one for the current version and one for the stuff held for the next version. This won't change anything except the labels on the queues. I think we might want to do something along the lines of what Stefan set up (at least I think it was he) for the end of 8.4 on developer.postgresql.org. Bruce's patch list is easy to update, but hard to read. I'll put some effort into it. -- Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Why are we waiting?
I'm not sure 32-bit and 64-bit cases are going to be directly comparable. We could have a problem with cache line aliasing on only one or the other for example. Agreed, this is likely comparing apples and oranges. I'll see if I can get a one-to-one comparison done (these were the numbers I had close by when writing the email). But that is a pretty striking difference. Does the 8.3 run complete more transactions in that time? I'll make sure to include those numbers as well. :) Staale ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] GSSAPI and V2 protocol
Magnus Hagander [EMAIL PROTECTED] writes: On Wed, Feb 06, 2008 at 02:57:39AM -0500, Kris Jurka wrote: On Tue, 5 Feb 2008, Tom Lane wrote: 2. We could retroactively redefine the contents of AuthenticationGSSContinue as carrying a length word after the authentication type code, but only in V2 protocol (so as not to break existing working cases). This is pretty ugly but certainly possible. I see no harm in doing this. What's there now can't work and the change is self contained. Is there any problem with the password message taking a String datatype instead of Byte[n] with a null byte? I agree that this is probabliy the best way, if we can do it. But you do raise a good point - the message that goes the other way can certainly contain embedded NULLs. I hadn't thought about the response side of the problem, but yeah, it is equally broken. To fix that would have to mean that V2 has two different password message formats for GSS vs other cases, which I think is starting to exceed my threshold of ugliness --- we are now talking at least four places needing weird special cases for V2 vs V3 protocol, two each in the server and (each) client library. I also quite dislike the idea that a password message couldn't even be parsed without context knowledge about which auth method it was for. In retrospect it was a serious error to use the PasswordMessage format for GSS responses, but with 8.3 already out the door I'm afraid we are stuck with that decision. I vote we just decide that GSS isn't going to be supported on protocol V2, and put a suitable error message into the server for that. It doesn't seem to me that this combination is worth the amount of contortions it would require to support. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] PostgreSQL 8.4 development plan
The plan looks great. I am +1 -Original Message- From: [EMAIL PROTECTED] ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] crash / data recovery issues
I'm trying to do some data recovery on an 8.1.9 system. The brief history is
the system crashed, attempted to do xlog replay but that failed. I did a
pg_resetxlog to get something that would startup, and it looks as if the
indexes on pg_class have become corrupt. (ie. reindex claimes duplicate rows,
which do not show up when doing count() manipulations on the data). As it
turns out, I can't drop these indexes either (system refuses with message
indexes are needed by the system). This has kind of let the system in an
unworkable state.
I've tried to do a pg_dump, but get schema with OID 96568 does not exist
error. The database has a number (~100) temp schemas in it, so I was
suspecting that the problem was with some object referencing a temp schema
with broken dependencies, but I looked through pg_depend for any referencing
objects but found none. I also looked through pg_type, pg_proc, pg_class,
pg_constraint, pg_operator, pg_opclass, pg_conversion at their respective
*namespace fields and also found no matches. Any suggestions on what else
might cause this, or how to get past it?
I also did some digging to find the original error on xlog replay and it
was failed to re-find parent key in 763769 for split pages 21032/21033.
I'm wondering if this is actually something you can push past with
pg_resetxlog, or if I need to do a pg_resetxlog and pass in values prior to
that error point (i guess essentially letting pg_resetxlog do a lookup)...
thoughts?
--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 6: explain analyze is your friend
Re: [HACKERS] PostgreSQL 8.4 development plan
Josh Berkus escribió: I think we might want to do something along the lines of what Stefan set up (at least I think it was he) for the end of 8.4 on developer.postgresql.org. Bruce's patch list is easy to update, but hard to read. I'll put some effort into it. Easy to update for Bruce -- for anyone else it is impossible to update AFAIK. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Why are we waiting?
Gregory Stark [EMAIL PROTECTED] writes: Staale Smedseng [EMAIL PROTECTED] writes: Also, an interesting observation is that the hot locks seem to have changed from v8.2 to v8.3, making the ProcArrayLock more contended. See the following outputs: PostgreSQL 8.2 (32-bit): ... PostgreSQL 8.3 (64-bit): ... I'm not sure 32-bit and 64-bit cases are going to be directly comparable. We could have a problem with cache line aliasing on only one or the other for example. Yeah, I find these numbers highly dubious. AFAIR we didn't do anything that would have reduced CLogControlLock contention, and we definitely did work to reduce ProcArrayLock contention, so the claimed results seem directly opposite to expectation. I am wondering if the waits are being attributed to the right locks --- I remember such an error in a previous set of dtrace results, and some of the other details such as claiming shared lock delays but no exclusive lock delays for FirstLockMgrLock seem less than credible as well. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] crash / data recovery issues
Robert Treat wrote: I'm trying to do some data recovery on an 8.1.9 system. The brief history is the system crashed, attempted to do xlog replay but that failed. I did a pg_resetxlog to get something that would startup, and it looks as if the indexes on pg_class have become corrupt. (ie. reindex claimes duplicate rows, which do not show up when doing count() manipulations on the data). As it turns out, I can't drop these indexes either (system refuses with message indexes are needed by the system). This has kind of let the system in an unworkable state. You can work out of it by starting a standalone server with system indexes disabled (postgres -O -P, I think) and do a REINDEX on it (the form of it that reindexes all system indexes -- I think it's REINDEX DATABASE). I also did some digging to find the original error on xlog replay and it was failed to re-find parent key in 763769 for split pages 21032/21033. I'm wondering if this is actually something you can push past with pg_resetxlog, or if I need to do a pg_resetxlog and pass in values prior to that error point (i guess essentially letting pg_resetxlog do a lookup)... thoughts? You should be able to get out of that by reindexing that index. (Actually, after you do a pg_resetxlog I think the best is to pg_dump the whole thing and reload it. That gives you at least the assurance that your FKs are not b0rked) -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Why are we waiting?
On Wed, 2008-02-06 at 13:55 -0500, Tom Lane wrote: Gregory Stark [EMAIL PROTECTED] writes: Staale Smedseng [EMAIL PROTECTED] writes: Also, an interesting observation is that the hot locks seem to have changed from v8.2 to v8.3, making the ProcArrayLock more contended. See the following outputs: PostgreSQL 8.2 (32-bit): ... PostgreSQL 8.3 (64-bit): ... I'm not sure 32-bit and 64-bit cases are going to be directly comparable. We could have a problem with cache line aliasing on only one or the other for example. Yeah, I find these numbers highly dubious. AFAIR we didn't do anything that would have reduced CLogControlLock contention, and we definitely did work to reduce ProcArrayLock contention, so the claimed results seem directly opposite to expectation. I am wondering if the waits are being attributed to the right locks --- I remember such an error in a previous set of dtrace results, and some of the other details such as claiming shared lock delays but no exclusive lock delays for FirstLockMgrLock seem less than credible as well. There were only 2 lock delays for FirstLockMgrLock in SHARED mode, so it seems believable that there were 0 lock delays in EXCLUSIVE mode. I assumed that Staale is running with clog buffers tweaked? Can you confirm Staale? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] crash / data recovery issues
Robert Treat [EMAIL PROTECTED] writes: I'm trying to do some data recovery on an 8.1.9 system. ... I also did some digging to find the original error on xlog replay and it was failed to re-find parent key in 763769 for split pages 21032/21033. Hmm, the only known cause of that was fixed in 8.1.6. Don't suppose you made a copy of everything before destroying the evidence with pg_resetxlog? If you did, any chance I could get access to it? regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] crash / data recovery issues
On Wednesday 06 February 2008 13:56, Alvaro Herrera wrote:
Robert Treat wrote:
it looks as if the indexes on pg_class have become corrupt. (ie. reindex
claimes duplicate rows, which do not show up when doing count()
manipulations on the data). As it turns out, I can't drop these indexes
either (system refuses with message indexes are needed by the system).
This has kind of let the system in an unworkable state.
You can work out of it by starting a standalone server with system
indexes disabled (postgres -O -P, I think) and do a REINDEX on it (the
form of it that reindexes all system indexes -- I think it's REINDEX
DATABASE).
Sorry, I should have mentioned I tried the above was under postgres -d
1 -P -O -D /path/to/data, but the reindex complains (doing reindex directly
on the pg_class indexes, or doing reindex system).
Personally I was surprised to find out it wouldn't let me drop the indexes
under this mode, but thats a different story. Oh, probably worth noting I
am able to reindex other system tables this way, just not pg_class.
--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Page-at-a-time Locking Considerations
Tom Lane wrote: Zdenek Kotala [EMAIL PROTECTED] writes: I tried to use memory mapped files (mmap) for clog and I think it should be also possible way. I got about 2% better performance, but it needs more testing. If you only got 2% out of it, it's not even worth thinking about how to fix the serious bugs that approach would create (primarily, lack of control over when pages can get flushed to disk). You can flush a pages by msync() function which writes dirty pages on disk. I don't see any other problem. Originally I tried to fix problem with a lot of parallel issues reported by Jignesh. However, it needs more testing if it really helps. Zdenek ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Why are we waiting?
Simon Riggs [EMAIL PROTECTED] writes: There were only 2 lock delays for FirstLockMgrLock in SHARED mode, so it seems believable that there were 0 lock delays in EXCLUSIVE mode. Not really, considering the extremely limited use of LW_SHARED in lock.c (GetLockConflicts is used only by CREATE INDEX CONCURRENTLY, and GetLockStatusData only by the pg_locks view). For the type of benchmark that I gather this is, there should be *zero* LW_SHARED acquisitions at all. And even if there are some, they could only be blocking against the (undoubtedly much more frequent) LW_EXCLUSIVE acquisitions; it's not very credible that there is zero contention among the LW_EXCLUSIVE locks yet a few shared acquirers manage to get burnt. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Why are we waiting?
On Wed, 2008-02-06 at 14:42 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: There were only 2 lock delays for FirstLockMgrLock in SHARED mode, so it seems believable that there were 0 lock delays in EXCLUSIVE mode. Not really, considering the extremely limited use of LW_SHARED in lock.c (GetLockConflicts is used only by CREATE INDEX CONCURRENTLY, and GetLockStatusData only by the pg_locks view). For the type of benchmark that I gather this is, there should be *zero* LW_SHARED acquisitions at all. And even if there are some, they could only be blocking against the (undoubtedly much more frequent) LW_EXCLUSIVE acquisitions; it's not very credible that there is zero contention among the LW_EXCLUSIVE locks yet a few shared acquirers manage to get burnt. ...but the total wait time on those lock waits was 24 microseconds. I hardly call that burnt. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] GSSAPI and V2 protocol
Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: On Wed, Feb 06, 2008 at 02:57:39AM -0500, Kris Jurka wrote: On Tue, 5 Feb 2008, Tom Lane wrote: 2. We could retroactively redefine the contents of AuthenticationGSSContinue as carrying a length word after the authentication type code, but only in V2 protocol (so as not to break existing working cases). This is pretty ugly but certainly possible. I see no harm in doing this. What's there now can't work and the change is self contained. Is there any problem with the password message taking a String datatype instead of Byte[n] with a null byte? I agree that this is probabliy the best way, if we can do it. But you do raise a good point - the message that goes the other way can certainly contain embedded NULLs. I hadn't thought about the response side of the problem, but yeah, it is equally broken. To fix that would have to mean that V2 has two different password message formats for GSS vs other cases, which I think is starting to exceed my threshold of ugliness --- we are now talking at least four places needing weird special cases for V2 vs V3 protocol, two each in the server and (each) client library. I also quite dislike the idea that a password message couldn't even be parsed without context knowledge about which auth method it was for. In retrospect it was a serious error to use the PasswordMessage format for GSS responses, but with 8.3 already out the door I'm afraid we are stuck with that decision. I vote we just decide that GSS isn't going to be supported on protocol V2, and put a suitable error message into the server for that. It doesn't seem to me that this combination is worth the amount of contortions it would require to support. Agreed. The cost is rapidly becoming too high. But we certainly can't change the protocol for the stuff that actually does work. //Magnus ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Page-at-a-time Locking Considerations
Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane wrote: If you only got 2% out of it, it's not even worth thinking about how to fix the serious bugs that approach would create (primarily, lack of control over when pages can get flushed to disk). You can flush a pages by msync() function which writes dirty pages on disk. I don't see any other problem. Then you need to learn more. The side of the problem that is hard to fix is that sometimes we need to prevent pages from being flushed to disk until some other data (typically WAL entries) has reached disk. With mmap'd data we have no control over early writes. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Why are we waiting?
Simon Riggs [EMAIL PROTECTED] writes: On Wed, 2008-02-06 at 14:42 -0500, Tom Lane wrote: Not really, considering the extremely limited use of LW_SHARED in lock.c (GetLockConflicts is used only by CREATE INDEX CONCURRENTLY, and GetLockStatusData only by the pg_locks view). For the type of benchmark that I gather this is, there should be *zero* LW_SHARED acquisitions at all. And even if there are some, they could only be blocking against the (undoubtedly much more frequent) LW_EXCLUSIVE acquisitions; it's not very credible that there is zero contention among the LW_EXCLUSIVE locks yet a few shared acquirers manage to get burnt. ...but the total wait time on those lock waits was 24 microseconds. I hardly call that burnt. What you are failing to grasp is that the data is simply not credible (unless perhaps Staale fesses up that his benchmark includes a whole lot of pg_locks monitoring, in which case I'd want to see it redone without anyway). regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] PostgreSQL 8.4 development plan
Alvaro Herrera wrote: Josh Berkus escribió: I think we might want to do something along the lines of what Stefan set up (at least I think it was he) for the end of 8.4 on developer.postgresql.org. Bruce's patch list is easy to update, but hard to read. I'll put some effort into it. Easy to update for Bruce -- for anyone else it is impossible to update AFAIK. Yes, I feel we could use a group writeable patch queue of some sort. Perhaps an IMAP server setup could do the job. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] PostgreSQL 8.4 development plan
On Feb 6, 2008 9:56 AM, Dave Page [EMAIL PROTECTED] wrote: Whenever a commit fest is in progress, the focus will shift from development to review, feedback and commit of patches. Each fest will continue until all patches in the queue have either been committed to the CVS repository, returned to the author for additional work, or rejected outright, and until that has happened, no new patches will be considered. If we don't have a bench farm anytime soon, I think we should consider planning a set of benchmarks after each commit fest to prevent performance regressions on different workloads. I don't expect them to be comprehensive but it could allow us to prevent the most obvious regressions. -- Guillaume ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] PostgreSQL 8.4 development plan
Peter Eisentraut [EMAIL PROTECTED] writes: Josh Berkus escribió: I think we might want to do something along the lines of what Stefan set up (at least I think it was he) for the end of 8.4 on developer.postgresql.org. Bruce's patch list is easy to update, but hard to read. I'll put some effort into it. Yes, I feel we could use a group writeable patch queue of some sort. Perhaps an IMAP server setup could do the job. Seems like a wiki page with links to pgsql-patches archive entries would be easy. But an issue for any of this is who has permissions to edit the queue? I concur that Bruce only is the wrong answer, but I'm not sure anyone with a wiki account is the right answer. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] PostgreSQL 8.4 development plan
On Wed, 06 Feb 2008 15:46:22 -0500 Tom Lane [EMAIL PROTECTED] wrote: Seems like a wiki page with links to pgsql-patches archive entries would be easy. But an issue for any of this is who has permissions to edit the queue? I concur that Bruce only is the wrong answer, but I'm not sure anyone with a wiki account is the right answer. The Wiki accounts are controlled to a degree. If the page gets abused we remove the privilege. Remember we can always rollback changes. This is no different than email moderation imo. Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit signature.asc Description: PGP signature
Re: [HACKERS] PostgreSQL 8.4 development plan
Joshua D. Drake wrote: On Wed, 06 Feb 2008 15:46:22 -0500 Tom Lane [EMAIL PROTECTED] wrote: Seems like a wiki page with links to pgsql-patches archive entries would be easy. But an issue for any of this is who has permissions to edit the queue? I concur that Bruce only is the wrong answer, but I'm not sure anyone with a wiki account is the right answer. The Wiki accounts are controlled to a degree. If the page gets abused we remove the privilege. Remember we can always rollback changes. This is no different than email moderation imo. Is it technically possible to set permissions on a per-page basis? //Magnus ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Page-at-a-time Locking Considerations
Tom Lane wrote: Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane wrote: If you only got 2% out of it, it's not even worth thinking about how to fix the serious bugs that approach would create (primarily, lack of control over when pages can get flushed to disk). You can flush a pages by msync() function which writes dirty pages on disk. I don't see any other problem. Then you need to learn more. The side of the problem that is hard to fix is that sometimes we need to prevent pages from being flushed to disk until some other data (typically WAL entries) has reached disk. With mmap'd data we have no control over early writes. I see. Thanks for explanation. Zdenek ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] PostgreSQL 8.4 development plan
Le Wednesday 06 February 2008 21:35:54 Peter Eisentraut, vous avez écrit : Alvaro Herrera wrote: Easy to update for Bruce -- for anyone else it is impossible to update AFAIK. Yes, I feel we could use a group writeable patch queue of some sort. Perhaps an IMAP server setup could do the job. I've read some developers appreciating the way review board works: http://review-board.org/ http://code.google.com/p/reviewboard/ http://code.google.com/p/reviewboard/wiki/UserBasics This last link present the expected workflow when using the tool, and maybe you'll find this matches the project needs... I don't know if such a tool can help the project, but mentioning its existence certainly can't arm... Hope this helps, -- dim signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 8.4 development plan
On Wed, 06 Feb 2008 22:07:06 +0100 Magnus Hagander [EMAIL PROTECTED] wrote: Joshua D. Drake wrote: On Wed, 06 Feb 2008 15:46:22 -0500 Tom Lane [EMAIL PROTECTED] wrote: Seems like a wiki page with links to pgsql-patches archive entries would be easy. But an issue for any of this is who has permissions to edit the queue? I concur that Bruce only is the wrong answer, but I'm not sure anyone with a wiki account is the right answer. The Wiki accounts are controlled to a degree. If the page gets abused we remove the privilege. Remember we can always rollback changes. This is no different than email moderation imo. Is it technically possible to set permissions on a per-page basis? That I can't answer but consider that the people that are putting info on the wiki are mostly people we trust. We just make it clear that if you are a jack ass you will have your account removed. IMO, we are putting entirely too much energy into controlling flow of text here. You have to log in to change the text, the text is revertible as is the ability to log in in the first place. Sincerely, Joshua D. Drake //Magnus -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit signature.asc Description: PGP signature
Re: [HACKERS] PostgreSQL 8.4 development plan
Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: Josh Berkus escribió: I think we might want to do something along the lines of what Stefan set up (at least I think it was he) for the end of 8.4 on developer.postgresql.org. Bruce's patch list is easy to update, but hard to read. I'll put some effort into it. Yes, I feel we could use a group writeable patch queue of some sort. Perhaps an IMAP server setup could do the job. Seems like a wiki page with links to pgsql-patches archive entries would be easy. But an issue for any of this is who has permissions to edit the queue? I concur that Bruce only is the wrong answer, but I'm not sure anyone with a wiki account is the right answer. this is basically what I did during the 8.3 cycle on the wiki - I would be willing to maintain a similiar thing for 8.4 if people think it is a good idea. Stefan ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] NULL OR ZERO
On Feb 3, 2008 7:41 PM, Jaime Casanova [EMAIL PROTECTED] wrote: On Feb 3, 2008 7:26 PM, Omar Bettin [EMAIL PROTECTED] wrote: Probably I am on the wrong place but for me NULL on numbers means 0 or ZERO. I know about standards... NULL means unknown value, ZERO is a known value NULL represents absence of a value. You get the UNKNOWN truth value if you compare NULL with any data value or another NULL (damn you, 3VL!). ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] build environment: a different makefile
Paul van den Bogaard wrote: The SunStudio compiler we are using fortunately has an option for this. Unfortunately there are restrictions. One restriction I face is its inability to deal with ld -rs. These are used in the build environment to create all the SUBSYS.o object files. I was hoping someone in the community already has a makefile that just creates object files from C-sources directly that I can use to try out the effect of in-lining to the performance of postgres. I don't know if anyone has a makefile for it, but the following seems to work for me: pgsql/src/backend$ cc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -g -L../../src/port -Wl,-rpath,'/home/peter/devel/pg83/pg-install/lib' -Wl,-E $(find -name *.o | grep -v SUBSYS | grep -v conversion_procs) ../../src/timezone/SUBSYS.o ../../src/port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres If you find that the optimizations you are hoping for are useful, I'm sure we could put an option of that sort somewhere in the makefiles. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] PostgreSQL 8.4 development plan
On Wed, 6 Feb 2008, Magnus Hagander wrote: Is it technically possible to set permissions on a per-page basis? Technically possible? Of course. It's sure not easy to do, though; the Mediawiki team considers having any real ACL structure added onto their code a non-feature and last time I checked you had to patch the source. I'd say it's really more trouble than it's worth here. It's not like the developer site is open to the whole world or something. The number of people capable of noticing and reverting bad changes in a critical, popular page vastly outnumbers those likely to do something stupid (with all the stuff I've done on the developer's wiki I don't think I've had to revert a change bigger than a grammatical error so far). -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] build environment: a different makefile
Peter Eisentraut [EMAIL PROTECTED] writes: I don't know if anyone has a makefile for it, but the following seems to work for me: pgsql/src/backend$ cc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -g -L../../src/port -Wl,-rpath,'/home/peter/devel/pg83/pg-install/lib' -Wl,-E $(find -name *.o | grep -v SUBSYS | grep -v conversion_procs) ../../src/timezone/SUBSYS.o ../../src/port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres If you find that the optimizations you are hoping for are useful, I'm sure we could put an option of that sort somewhere in the makefiles. I've sometimes wondered whether the SUBSYS.o files really offer any advantage compared to just linking all the individual .o files. They certainly eat disk space, but perhaps they save some time ... or perhaps not, especially in a one-off build. I suppose that we might fall foul of command line length limits on some platforms :-(. The output of your find command amounts to nearly 11000 characters in HEAD. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] PostgreSQL 8.4 development plan
Dimitri Fontaine [EMAIL PROTECTED] writes: Le Wednesday 06 February 2008 21:35:54 Peter Eisentraut, vous avez écrit : Yes, I feel we could use a group writeable patch queue of some sort. Perhaps an IMAP server setup could do the job. I've read some developers appreciating the way review board works: http://review-board.org/ http://code.google.com/p/reviewboard/ http://code.google.com/p/reviewboard/wiki/UserBasics Hmm, the info on that last page might be out of date, but what it says is that the only SCMS they really support 100% is SVN. The other ones they claim support for don't work [well/at all] with the post-review tool. It looks interesting though, and would alleviate a few of the problems people have mentioned with reviewing stuff that's posted as diffs. Has anyone here got any direct experience with it? regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] PostgreSQL 8.4 development plan
On Wed, 06 Feb 2008 18:50:34 -0500 Tom Lane [EMAIL PROTECTED] wrote: I've read some developers appreciating the way review board works: http://review-board.org/ http://code.google.com/p/reviewboard/ http://code.google.com/p/reviewboard/wiki/UserBasics Hmm, the info on that last page might be out of date, but what it says is that the only SCMS they really support 100% is SVN. The O.k. I am not too interested in starting a whole war here (again) but for the record, we have what appears to be a perfectly working capability to move from cvs to svn. So *if* review board is something we really like, the SCM should not be the barrier. Sincerely, Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit signature.asc Description: PGP signature
Re: [HACKERS] Page-at-a-time Locking Considerations
Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane wrote: Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane wrote: If you only got 2% out of it, it's not even worth thinking about how to fix the serious bugs that approach would create (primarily, lack of control over when pages can get flushed to disk). You can flush a pages by msync() function which writes dirty pages on disk. I don't see any other problem. Then you need to learn more. The side of the problem that is hard to fix is that sometimes we need to prevent pages from being flushed to disk until some other data (typically WAL entries) has reached disk. With mmap'd data we have no control over early writes. I see. Thanks for explanation. In theory mlock() ought to provide that facility. The kernel people know it's used by crypto software to avoid having disk copies of sensitive keys, so there's at least a fighting chance it actually works for this too. But I wouldn't put too much money on it working this purpose on every platform that has it. It's entirely conceivably that some platforms have mlock avoid swapping out pages but not avoid syncing them but leaving them in RAM. Or that some might sync mlocked pages when the process which had the page locked dies, especially if it crashes. Or that some versions of some OSes are simply buggy. It's not like it's a case that would ever be tested or even noticed if it failed. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support! ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] PostgreSQL 8.4 development plan
Joshua D. Drake [EMAIL PROTECTED] writes: O.k. I am not too interested in starting a whole war here (again) but for the record, we have what appears to be a perfectly working capability to move from cvs to svn. So *if* review board is something we really like, the SCM should not be the barrier. I believe the compromise that's been reached for the moment is that the core SCM will remain CVS, because everybody's favorite other SCM can import from CVS but not necessarily from somebody else's favorite other SCM. So a diff tool that doesn't work with CVS isn't going to be especially useful for us. I would imagine that the problem is mostly a lack of round tuits, and that if we really fell in love with review board we could probably teach it to handle diffs against CVS (especially seeing that the rest of it besides post-review already works with CVS, supposedly). So, again, the question is has anyone really used it? Is it the best thing since sliced bread, or not so much? regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] PostgreSQL 8.4 development plan
Tom Lane wrote: Joshua D. Drake [EMAIL PROTECTED] writes: O.k. I am not too interested in starting a whole war here (again) but for the record, we have what appears to be a perfectly working capability to move from cvs to svn. So *if* review board is something we really like, the SCM should not be the barrier. I believe the compromise that's been reached for the moment is that the core SCM will remain CVS, because everybody's favorite other SCM can import from CVS but not necessarily from somebody else's favorite other SCM. So a diff tool that doesn't work with CVS isn't going to be especially useful for us. I would imagine that the problem is mostly a lack of round tuits, and that if we really fell in love with review board we could probably teach it to handle diffs against CVS (especially seeing that the rest of it besides post-review already works with CVS, supposedly). So, again, the question is has anyone really used it? Is it the best thing since sliced bread, or not so much? regards, tom lane My official role at my place of work is configuration management software architect. We primarily use ClearCase and I am responsible for the software side of the tooling around it. We have several thousands users and terrabytes of data stored from millions of change sets. Not that roles or anything matter, but where your job is PostgreSQL, my job is SCM. Probably because I am spoiled - I don't understand how your teams get along so well with CVS. From my perspective, nearly everything available is better than CVS. If it works good for you, and you don't ever have merging problems, or history tracking problems, then great - any move is going to be a hassle and will cause pain wasting at least some time in the next development cycle. If you do want to see the benefit of change - here is my experience with Subversion: I have been playing with Subversion for just almost two years now in a small group of people with 3 people on a small project. While working on the main branch (trunk) submissions were generally smooth. Conflict resolution is poor without graphical tool support, but with only three people and co-ordinated work this was rarely an issue. Atomic submissions were a pleasant relief and performance was adequate. Commits are not at the level of functionality that I am accustomed to though. First, commits are not registered until a person is complete their work and the work is submitted. Second, merging of commits is very weak in every production version of Subversion available today (1.4 and before) because Subversion does not perform merge tracking. As soon as one begins using multiple branches, it becomes very difficult to keep track of where things are, and the people who support Subversion are satisfied writing commit numbers in their comments to keep track of completed merges. Finally, because the concept of directories, branches, and tags have all been blurred into one muddle, horrible things happen if you try to do anything clever. In my case, I had a web project that I intended to break into web, lib, and source. I renamed trunk to trunk/www and created trunk/lib, and trunk/source. For this point forwards, I was completely unable to merge changes from other branches to trunk. Subversion became completely confused. It was at this point that my frustration acceptance level was passed, and I switched to GIT. This was last December. Subversion 1.5 was supposed to be out to address many of these issues, but it was a hollow promise as it was still not released the last time I checked, and a review of their discussions on the matter show that many of the promises they made were likely premature. Since then, I have been consistently impressed with GIT. I have completed complex merges and extensive parallel development that would have been painful or impossible with Subversion. I am not a fan of de-centralization as most GIT supporters are - but I am a fan of full feature change sets. In GIT I can merge a change set back and forth between branches and it will track it. I can rebase the change set to a later baseline and continue manipulating it. I can save my work space aside, or use the same work space to switch to another branch and have my uncommitted work automatically three-way merged to the new context. Our team on this outside work project is now up to 5 people and everybody likes GIT better than Subversion. My story is that Subversion is cute - but it does not scale in terms of flexible parallel development models, nor does it provide sufficient functionality over CVS to be considered the best thing since sliced bread. It is an improvement over CVS, but it is not a great tool. If you are going to go through the effort of migrating to another system, I would seriously consider the benefits of other systems out there before believing that Subversion is the answer to all problems. GIT is one good
Re: [HACKERS] PostgreSQL 8.4 development plan
Tom Lane wrote: Brendan Jurd [EMAIL PROTECTED] writes: Just thinking that we'll need somewhere to park the new patches which roll in during a commit fest. Bruce has always kept two patch queues, one for the current version and one for the stuff held for the next version. This won't change anything except the labels on the queues. Sure, I can do that. One is already called the _hold_ patch queue. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Page-at-a-time Locking Considerations
Zdenek Kotala wrote: Tom Lane wrote: Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane wrote: If you only got 2% out of it, it's not even worth thinking about how to fix the serious bugs that approach would create (primarily, lack of control over when pages can get flushed to disk). You can flush a pages by msync() function which writes dirty pages on disk. I don't see any other problem. Then you need to learn more. The side of the problem that is hard to fix is that sometimes we need to prevent pages from being flushed to disk until some other data (typically WAL entries) has reached disk. With mmap'd data we have no control over early writes. I see. Thanks for explanation. This is mentioned in the TODO list: * Consider mmap()'ing files into a backend? Doing I/O to large tables would consume a lot of address space or require frequent mapping/unmapping. Extending the file also causes mapping problems that might require mapping only individual pages, leading to thousands of mappings. Another problem is that there is no way to _prevent_ I/O to disk from the dirty shared buffers so changes could hit disk before WAL is written. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] patch queue needs update was:(PostgreSQL 8.4 development plan)
On Feb 6, 2008 1:52 PM, Alvaro Herrera [EMAIL PROTECTED] wrote: Josh Berkus escribió: I think we might want to do something along the lines of what Stefan set up (at least I think it was he) for the end of 8.4 on developer.postgresql.org. Bruce's patch list is easy to update, but hard to read. I'll put some effort into it. Easy to update for Bruce -- for anyone else it is impossible to update AFAIK. and, of course, will be things that escape from Bruce's eyes... for example, the Add GUC temp_tablespaces toprovide a default location for patch was actually committed in 8.3 and it's still on the patch queue http://momjian.us/mhonarc/patches_hold/msg0.html i think a depuration will be needed to keep the first commit fests simple... -- regards, Jaime Casanova Programming today is a race between software engineers striving to build bigger and better idiot-proof programs and the universe trying to produce bigger and better idiots. So far, the universe is winning. Richard Cook ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
[HACKERS] 8.3 / 8.2.6 restore comparison
Hello, I have been testing a migration for a week now trying to get it into a reasonable state. This is what we have: Restore file 220G 8.2.6 and 8.3.0 are configured identically: shared_buffers = 8000MB work_mem = 32MB maintenance_work_mem = 512MB fsync = off full_page_writes = off checkpoint_segments = 300 synchronous_commit = off (8.3) wal_writer_delay = off (8.3) autovacuum = off 8.2.6 after 2 hours has restored 41GB. 8.3.0 after 2.5 hours had restored 38GB. Originally I was thinking that 8.2.6 was stomping 8.3. However I am thinking that the reduction in the tuple header sizes for 8.3 means that yes I restored 38GB, it is actually *more* data than 8.2.6. Does that seem accurate to everyone else? If so what can we do to speed this up? We are certainly *not* saturating the disk (16 spindles SCSI). I am thinking the way we are going to need to do this is to have an extended outage and write a custom script to do a concurrent dump and load. (no in this case slony is not an option). Sincerely, Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit signature.asc Description: PGP signature
Re: [HACKERS] PostgreSQL 8.4 development plan
Tom Lane wrote: There is, although I think a large fraction of it will get bounced as needs more work, which should reduce the pressure. We'll just be trying to give feedback to let the patch authors move forward, which will not take as much time as actually committing would take. The current queue is http://momjian.postgresql.org/cgi-bin/pgpatches_hold Note that a lot of the bulk is discussion of things that aren't anywhere near committable anyway. Wouldn't it make sense try to catch up a bit and fix as much of this as is feasible (including return and resubmit) for things where the desire is uncontroversial, even if the implementation is flawed, and accept other things that are fully acceptable in the time it takes to do that, and then call it a wrap? The curre nt *plan* is for a 14 month cycle. And it will probably slip. Some of the queued items are going to be very old by the time you go to 8.4 on this program, which seems a shame. That sort of plan looks to me more like a 'major refactoring to get to 9.0' sort of plan, than an incremental release. James ---(end of broadcast)--- TIP 6: explain analyze is your friend
