Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended trans action ID module to contrib as
> > (1) we've always been laxer about contrib than the core code, > > While that appears to be true, I think > > (a) there is no technical reason allowing us to do that, and > (b) most people don't seem to like it. > I will even go so far as to say there are technical reasons not to do it. I beleive that contrib is currently included in most if not all our packages. It certainly is on win32 and I think I've seen the RPMs. It may not be loaded by default but it's there. And that will have users expecting the same code quality as for the rest of the PGDG code. If we can't (or won't) do that, well,that's why we have pgfoundry. /Magnus ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended trans action ID module to contrib as
> > Right. My thought is still that if it isn't good enough for core, it > > shouldn't be in contrib. If it *is* good enough, and we want it, we > > should accept that it came in long after freeze and put it in core > > anyway. If it *isn't*, then it should be on pgfoundry and be moved into > > core when it's ready - for 8.4 or so. > > The long and the short of it was that the patch wasn't ready. So if the patch wasn' ready, why did it get accepted for /contrib? > To put it > in core for 8.3, we'd have either had to delay the beta yet more, or > force initdb post-beta1, neither of which would have flown. So it should've been saved for 8.4. > > The whole contrib thing confuses a lot of users. > > To me, contrib exists mostly as a forcing function to ensure that we > keep the extension-module system working. Ok. But if that's what it's mainly for then we *really* shouldn't put things that we expect our users to rely heavily on. And if this thing will go deep into replication systems, that's exactly what it is. > Contrib also has a role to play as a repository of code examples that > people can crib from when developing new extension modules. I would > not want to claim that it's all "best practice" code --- a lot of it > definitely isn't --- but it stands a lot better chance of representing > current good practice if it's maintained with the core code than if it's > out on pgfoundry. On pgfoundry, it won't get included in the global- > search-and-replace patches that we do so many of, and it'll most likely > accumulate a lot of cruft from trying to be compatible with multiple > core releases. Same comment applies here. And it's certainly far from best practice if it "breaks the rules"... /Magnus ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Latest ecpg patch broke MSVC build
> In hindsight, all these ecpg changes should have been made between beta1 > and beta2 when we have time to deal with the fallout, not right before > beta1. > Or considered new features and held back for 8.4. Not picking on Michael, but the resemblance to the /contrib discussion is striking. Ecpg is another part of core PostgreSQL that lives by slightly different rules. /Magnus ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Skytools committed without hackers discussion/review
Bruce Momjian <[EMAIL PROTECTED]> writes: > I don't see how timing has anything to do with this. You could have > added it between beta1 and beta2 after sufficient hackers discussion. Uh, it *was* after beta1. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Money type clarity
Joshua D. Drake wrote: > The docs since 7.3 have declared the money type deprecated.. that is an > awful long time. Can we get some clarity on the issue? > IMHO it's not but it certainly need some more work on the storage (numeric?) and locale part as already discussed. -- Euler Taveira de Oliveira http://www.timbira.com/ ---(end of broadcast)--- TIP 6: explain analyze is your friend
[HACKERS] Skytools committed without hackers discussion/review
Jan Wieck wrote: > On 10/8/2007 1:34 PM, Tom Lane wrote: > > Magnus Hagander <[EMAIL PROTECTED]> writes: > >> Marko Kreen wrote: > >>> Because of the bad timing it would have been -core call anyway > >>> whether it gets in or not so Jan asked -core directly. That's > >>> my explanation about what happened, obviously Jan and Tom have > >>> their own opinion. > > > >> Right. I can see your point, but it's my understanding that -hackers is > >> really the ones supposed to decide on this. > > > > It would ultimately have been core's decision, but the discussion should > > have happened on -hackers. There was no reason for it to be private. > > That blame certainly belongs to me and I apologize for jumping that and > adding it to contrib without any -hackers discussion. > > It is definitely a timing issue since I write this very email from JFK, > boarding a flight to Hong Kong in less than an hour and will be mostly > offline for the rest of the week. I don't see how timing has anything to do with this. You could have added it between beta1 and beta2 after sufficient hackers discussion. Doing it the way you did with no warning, right before beta, and then leaving is the worse of all times. I am surprised we are not backing out the patch and requiring that the patch go through the formal review process. This is not the first time you have had trouble with patches. There was an issue with your patch of February, 2007: http://archives.postgresql.org/pgsql-hackers/2007-02/msg00385.php (In summary, you had to be coaxed to explain your patch to the community.) Basically, I am not sure you understand the process that has to be followed, or feel you are somehow immune from following it. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Tatsuo Ishii wrote: > > Tatsuo Ishii wrote: > > > > > More question. Who is in charge of updating HISTORY? I see no commit > > > > > messages for this. > > > > > > > > It is a generated file. > > > > > > I mean release.sgml. > > > > Tom and others made commits to this and we will update it again before > > final. > > Did it? I see nothing for txid in relesase.sgml. Right. release.sgml will be updated in batches as we near final release. We don't update for individual commits. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
> Tatsuo Ishii wrote: > > > > More question. Who is in charge of updating HISTORY? I see no commit > > > > messages for this. > > > > > > It is a generated file. > > > > I mean release.sgml. > > Tom and others made commits to this and we will update it again before > final. Did it? I see nothing for txid in relesase.sgml. -- Tatsuo Ishii SRA OSS, Inc. Japan ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Tatsuo Ishii wrote: > > > More question. Who is in charge of updating HISTORY? I see no commit > > > messages for this. > > > > It is a generated file. > > I mean release.sgml. Tom and others made commits to this and we will update it again before final. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[HACKERS] Timezone database changes
I had a thought a week ago. If we update the time zone database for future dates, and you have a future date/time stored, doesn't the time change when the time zone database changes. For example if I schedule an appointment in New Zealand for 10:00a and we change the time zone database so that date is now daylight savings, doesn't the time change to display as 9 or 11am? That seems pretty bad. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] type money causes unrestorable dump
On Mon, 8 Oct 2007 20:02:56 -0700 "Joshua D. Drake" <[EMAIL PROTECTED]> wrote: > On Mon, 8 Oct 2007 22:52:23 -0400 (EDT) > Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > Joshua D. Drake wrote: > > -- Start of PGP signed section. > > > On Mon, 8 Oct 2007 22:42:57 -0400 (EDT) > > > Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > > > > > Joshua D. Drake wrote: > > > > -- Start of PGP signed section. > > > > > On Mon, 8 Oct 2007 22:31:31 -0400 (EDT) > > > > > Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > > > > > > >dering money is deprecated, is this really needed? > > > > > > > > We have other MONEY TODO items too. I am not sure if it is > > > > depricated or in need of major work. > > > > > > Right from the docs :) > > > > > > Note: The money type is deprecated. Use numeric or decimal > > > instead, in combination with the to_char function. > > > > It will be un-depricated if it is improved. > > The money data type has been deprecated for years. It is completely > non standard and essentially duplicative of numeric/decimal. What is > the point? Please ignore this. I have asked on -hackers for clarity on the issue. If it is not deprecated I will submit a patch to -docs. Joshua D. Drake > > Joshua D. Drake > > > > > > > > -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ signature.asc Description: PGP signature
[HACKERS] Money type clarity
Hello, The docs since 7.3 have declared the money type deprecated.. that is an awful long time. Can we get some clarity on the issue? If it isn't deprecated cool, I will submit a patch to docs. Sincerely, Joshua D. Drake -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ signature.asc Description: PGP signature
Re: [HACKERS] type money causes unrestorable dump
On Mon, 8 Oct 2007 22:52:23 -0400 (EDT) Bruce Momjian <[EMAIL PROTECTED]> wrote: > Joshua D. Drake wrote: > -- Start of PGP signed section. > > On Mon, 8 Oct 2007 22:42:57 -0400 (EDT) > > Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > > > Joshua D. Drake wrote: > > > -- Start of PGP signed section. > > > > On Mon, 8 Oct 2007 22:31:31 -0400 (EDT) > > > > Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > > > > >dering money is deprecated, is this really needed? > > > > > > We have other MONEY TODO items too. I am not sure if it is > > > depricated or in need of major work. > > > > Right from the docs :) > > > > Note: The money type is deprecated. Use numeric or decimal > > instead, in combination with the to_char function. > > It will be un-depricated if it is improved. The money data type has been deprecated for years. It is completely non standard and essentially duplicative of numeric/decimal. What is the point? Joshua D. Drake > -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ signature.asc Description: PGP signature
Re: [HACKERS] type money causes unrestorable dump
On Mon, 8 Oct 2007 22:42:57 -0400 (EDT) Bruce Momjian <[EMAIL PROTECTED]> wrote: > Joshua D. Drake wrote: > -- Start of PGP signed section. > > On Mon, 8 Oct 2007 22:31:31 -0400 (EDT) > > Bruce Momjian <[EMAIL PROTECTED]> wrote: > > >dering money is deprecated, is this really needed? > > We have other MONEY TODO items too. I am not sure if it is depricated > or in need of major work. Right from the docs :) Note: The money type is deprecated. Use numeric or decimal instead, in combination with the to_char function. Sincerely, Joshua D. Drake -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ signature.asc Description: PGP signature
Re: [HACKERS] type money causes unrestorable dump
Joshua D. Drake wrote: -- Start of PGP signed section. > On Mon, 8 Oct 2007 22:42:57 -0400 (EDT) > Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > Joshua D. Drake wrote: > > -- Start of PGP signed section. > > > On Mon, 8 Oct 2007 22:31:31 -0400 (EDT) > > > Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > > >dering money is deprecated, is this really needed? > > > > We have other MONEY TODO items too. I am not sure if it is depricated > > or in need of major work. > > Right from the docs :) > > Note: The money type is deprecated. Use numeric or decimal instead, in > combination with the to_char function. It will be un-depricated if it is improved. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] type money causes unrestorable dump
On Mon, 8 Oct 2007 22:31:31 -0400 (EDT) Bruce Momjian <[EMAIL PROTECTED]> wrote: > Tom Lane wrote: > > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > > I noticed that if you create a dump on a database containing a > > > money column and a certain locale, this dump is not restorable on > > > a database with a different locale. > > > > We've been through this, no? If money doesn't print that way, > > there's no obvious reason to have the type at all. Use numeric if > > you don't want something with locale-specific behavior. > > Added to TODO: > > * MONEY dumps in a locale-specific format making it difficult to > restore to a system with a different locale Considering money is deprecated, is this really needed? Joshua D. Drake > -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ signature.asc Description: PGP signature
Re: [HACKERS] type money causes unrestorable dump
Joshua D. Drake wrote: -- Start of PGP signed section. > On Mon, 8 Oct 2007 22:31:31 -0400 (EDT) > Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > Tom Lane wrote: > > > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > > > I noticed that if you create a dump on a database containing a > > > > money column and a certain locale, this dump is not restorable on > > > > a database with a different locale. > > > > > > We've been through this, no? If money doesn't print that way, > > > there's no obvious reason to have the type at all. Use numeric if > > > you don't want something with locale-specific behavior. > > > > Added to TODO: > > > > * MONEY dumps in a locale-specific format making it difficult to > > restore to a system with a different locale > > Considering money is deprecated, is this really needed? We have other MONEY TODO items too. I am not sure if it is depricated or in need of major work. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Release Notes Overview
Gregory Stark wrote: > > "Simon Riggs" <[EMAIL PROTECTED]> writes: > > > Asynchronous Commit allows some transactions to commit faster than > > others, offering a trade-off between performance and durability for > > specific transaction types only > > A lot of users will be confused about what asynchronous commit does. I think > it's important to be consistently precise when describing it. > > It doesn't allow commits to be any faster, what it does is "allow clients to > start a new transaction and continue working without waiting for their > previous commit to complete". Saying something like "This allows high volumes > of short transactions such as typical web sites to run more efficiently and > with fewer connections" might also help clarify the use case it helps. Well, logically the commit does happen faster in that your transaction and others see the commit. It is just durability that is delayed. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] type money causes unrestorable dump
Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > I noticed that if you create a dump on a database containing a money > > column and a certain locale, this dump is not restorable on a database > > with a different locale. > > We've been through this, no? If money doesn't print that way, there's > no obvious reason to have the type at all. Use numeric if you don't > want something with locale-specific behavior. Added to TODO: * MONEY dumps in a locale-specific format making it difficult to restore to a system with a different locale -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Use of "postmaster"
Patch applied. Thanks. Your documentation changes can be viewed on our web site shortly. --- Brendan Jurd wrote: > On 10/4/07, Tom Lane <[EMAIL PROTECTED]> wrote: > > "Brendan Jurd" <[EMAIL PROTECTED]> writes: > > > Now that we've renamed the server binary to "postgres", what is the > > > status on use of the name "postmaster"? Is it now deprecated? And if > > > not, is there any point in keeping it around? > > > > We should replace it by terms like "server" in contexts where it's > > not actually important to the reader which process is involved, > > but I think Peter's hit most of them already ... > > Looks like Peter got the sgml sources pretty well cleaned up, but > didn't touch the FAQs. > > The attached patch replaces some more references to "postmaster" in > the FAQs. Per Tom's guidance, I only replaced those references where > I felt a distinction between the postmaster and its children wasn't > important to the reader. > > Thanks for your time, > BJ [ Attachment, skipping... ] > > ---(end of broadcast)--- > TIP 6: explain analyze is your friend -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Latest ecpg patch broke MSVC build
In hindsight, all these ecpg changes should have been made between beta1 and beta2 when we have time to deal with the fallout, not right before beta1. --- Tom Lane wrote: > This morning's ecpg patch certainly seems to have been snake-bit. > Although the Windows gcc buildfarm members seem happy, the MSVC ones > are all failing with > > Linking... >Creating library Release\libecpg\libecpg.lib and object > Release\libecpg\libecpg.exp > libecpg.exp : error LNK2001: unresolved external symbol [EMAIL > PROTECTED] > .\Release\libecpg\libecpg.dll : fatal error LNK1120: 1 unresolved > externals > > I see that DllMain() got added to misc.c, so it's not obvious what's > wrong here. Some adjustment needed in the MSVC build scripts maybe? > > regards, tom lane > > ---(end of broadcast)--- > TIP 3: Have you checked our extensive FAQ? > >http://www.postgresql.org/docs/faq -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
> > More question. Who is in charge of updating HISTORY? I see no commit > > messages for this. > > It is a generated file. I mean release.sgml. -- Tatsuo Ishii SRA OSS, Inc. Japan ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Tatsuo Ishii wrote: > More question. Who is in charge of updating HISTORY? I see no commit > messages for this. It is a generated file. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
> On Sun, Oct 07, 2007 at 11:32:19PM +, Jan Wieck wrote: > > Log Message: > > --- > > Added the Skytools extended transaction ID module to contrib as discussed > > on CORE previously. > > > > This module offers transaction ID's containing the original XID and the > > transaction epoch as a bigint value to the user level. It also provides > > a special txid_snapshot data type that contains an entire transactions > > visibility snapshot information, which is useful to determine if a > > particular txid was visible to a transaction or not. > > > > The module has been tested by porting Slony-I from using its original > > xxid data type. > > > > Jan > > A couple of questions on this. I'm not objecting to the tech stuff itself, > but on procedure: > > 1) Why was this added without any discussion, or even notification, on > -hackers before it was done? Last I checked, -core claim not to deal with > such technicalities, but defer to -hackers (or other lists as needed). I > certainly > trust -core to make the right call no these things, but it's not the > procedure that we claim to have. > > If that procedure is changing (I've been getting a sneaky feeling that > it's tilting a bit in that direction before this one), that's fine, but it > should be communicated to the community so everybody knows how it works. > > > 2) How can this go in *months* after feature freeze, and even after we > tagged and bundled beta1? This makes such discission even more important, > IMHO. > > 3) I thought the general agreement was to cut down on contrib, and move > things either to pgfoundry or to core. Why are we adding more? I'm sure > there's motivation for this as discussed on -core, but the rest of us would > like to know that as well... Like why we're not trying to make it a real > feature, if it's something that's important enough to break the rules as in > #2 above. > > > Or I could've missed the discussion on -hackers that actually took place - > in that case, just discard this message. but the only one I recall is > someone asking for pl/proxy to go in. More question. Who is in charge of updating HISTORY? I see no commit messages for this. -- Tatsuo Ishii SRA OSS, Inc. Japan ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Tom Lane wrote: So I have no interest in trying to "retire" contrib. I think there's room for debate about exactly which modules to include, of course. I dont' think there's much call for retiring it. I think there is interest in renaming it, as "contrib" is a wholly misleading name, and rearranging the modules somewhat, and above all in documenting it properly. One of the original goals of the buildfarm was to lessen the extent to which contrib was a second class citizen by making sure it was tested on the same schedule as the rest of Postgres. cheers andrew ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
A Dimarts 09 Octubre 2007, Joshua D. Drake va escriure: > > Contrib is just a dead zone for the user populous. Most people consider > it unsupported *stuff* with no particular purpose (regardless of the > real meaning). > I think no visible documentation is the reason for this misconception and 8.3 will provide contrib documentation together with core docs. Let's see what happens then... -- Albert Cervera i Areny http://www.NaN-tic.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Magnus Hagander <[EMAIL PROTECTED]> writes: > Right. My thought is still that if it isn't good enough for core, it > shouldn't be in contrib. If it *is* good enough, and we want it, we > should accept that it came in long after freeze and put it in core > anyway. If it *isn't*, then it should be on pgfoundry and be moved into > core when it's ready - for 8.4 or so. The long and the short of it was that the patch wasn't ready. To put it in core for 8.3, we'd have either had to delay the beta yet more, or force initdb post-beta1, neither of which would have flown. > The whole contrib thing confuses a lot of users. To me, contrib exists mostly as a forcing function to ensure that we keep the extension-module system working. If we got rid of it entirely, as some here propose, we'd be more likely to break things that we'd not find out about until much later (like when some pgfoundry project tried to update their code, which almost certainly wouldn't be till the next beta cycle). Contrib also has a role to play as a repository of code examples that people can crib from when developing new extension modules. I would not want to claim that it's all "best practice" code --- a lot of it definitely isn't --- but it stands a lot better chance of representing current good practice if it's maintained with the core code than if it's out on pgfoundry. On pgfoundry, it won't get included in the global- search-and-replace patches that we do so many of, and it'll most likely accumulate a lot of cruft from trying to be compatible with multiple core releases. So I have no interest in trying to "retire" contrib. I think there's room for debate about exactly which modules to include, of course. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
On Mon, 08 Oct 2007 22:32:55 +0200 Magnus Hagander <[EMAIL PROTECTED]> wrote: > Tom Lane wrote: > > Magnus Hagander <[EMAIL PROTECTED]> writes: > >> Marko Kreen wrote: > >>> Because of the bad timing it would have been -core call anyway > >>> whether it gets in or not so Jan asked -core directly. That's > >>> my explanation about what happened, obviously Jan and Tom have > >>> their own opinion. > > > >> Right. I can see your point, but it's my understanding that > >> -hackers is really the ones supposed to decide on this. > > > > It would ultimately have been core's decision, but the discussion > > should have happened on -hackers. There was no reason for it to be > > private. > > Hmm. I thought that -core doesn't decide on things like these, they > just "vote" on -hackers and have no special powers (other than being > very respected community members that we all listen to, of course). That was my understanding as well. I quote from Tom Lane on August 30th of this year: I think you're overstating the amount of power the core committee has. Core sets release schedules, but beyond that has no more power than any other committer. The pool of committers is quite a bit bigger than core. In practice, of course, core has quite a lot of political power because folk are often willing to follow our lead. But we'd lose that real quick if we tried to lead in the wrong direction. Source: http://people.planetpostgresql.org/xzilla/index.php?/archives/317-PostgreSQL-is-Not-a-Democracy.html Sincerely, Joshua D. Drake -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ signature.asc Description: PGP signature
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
On Mon, 08 Oct 2007 17:38:48 -0400 Robert Treat <[EMAIL PROTECTED]> wrote: > On Monday 08 October 2007 16:29, Magnus Hagander wrote: > > The whole contrib thing confuses a lot of users. Is it included, or > > isn't it? IMHO, that distinction need to be clear, and I thought we > > were working (if not actively then at least passively) to "retire" > > contrib, moving things either to core or to pgFoundry. Adding yet > > another important feature that's "just in contrib" is making things > > worse, not better. > > IMHO, of course ;-) > > > > +1. I felt the same way about pg_standby, which would have been far > more accessible for 8.2 users had it lived on pg_foundry. Well pg_standby is certainly an excellent example :). We have had this similar discussion before, this exact discussion is one of the reasons Tsearch2 got pushed into core. Contrib is just a dead zone for the user populous. Most people consider it unsupported *stuff* with no particular purpose (regardless of the real meaning). If you look at contrib in other projects they are always user contributed modules that are cool, but your mileage may vary. Personally, I would like to see contrib completely removed. Sincerely, Joshua D. Drake > -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ signature.asc Description: PGP signature
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Tom Lane wrote: > (1) we've always been laxer about contrib than the core code, While that appears to be true, I think (a) there is no technical reason allowing us to do that, and (b) most people don't seem to like it. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
On Monday 08 October 2007 16:29, Magnus Hagander wrote:
> The whole contrib thing confuses a lot of users. Is it included, or
> isn't it? IMHO, that distinction need to be clear, and I thought we
> were working (if not actively then at least passively) to "retire"
> contrib, moving things either to core or to pgFoundry. Adding yet
> another important feature that's "just in contrib" is making things
> worse, not better.
> IMHO, of course ;-)
>
+1. I felt the same way about pg_standby, which would have been far more
accessible for 8.2 users had it lived on pg_foundry.
--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
Re: [HACKERS] PG on NFS may be just a bad idea
On Mon, 2007-10-08 at 16:50 -0400, Alvaro Herrera wrote: > palloc uses malloc underneath. My thought is to replace that with > sbrk, mmap or something like that. Not very portable though, a lot of > work, and most likely not nearly enough benefits. Yeah, I agree this isn't likely to be a win in the general case. However, it would be interesting to explore a specialized allocator for short-lived memory contexts, where we don't care about having an effective pfree(). If the context is going to be reset or deleted shortly anyway, we could probably optimize and simplify palloc() by skipping free space accounting and then make pfree() a no-op. I recall Tom mentioning something to this effect a few months back... -Neil ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
"Magnus Hagander" <[EMAIL PROTECTED]> writes: > The whole contrib thing confuses a lot of users. Is it included, or > isn't it? IMHO, that distinction need to be clear, and I thought we > were working (if not actively then at least passively) to "retire" > contrib, moving things either to core or to pgFoundry. Adding yet > another important feature that's "just in contrib" is making things > worse, not better. > IMHO, of course ;-) I disagree with this principle. I think contrib has a role to play and there are modules which belong in contrib for now and forever. The key distinction isn't code quality -- I think that's an effect rather than a cause. The key distinction is the intended audience. Contrib is for things which are integral parts of the system and as such would be harder to maintain in pgfoundry, but have a very limited audience, especially things that a typical admin wouldn't necessarily want in his install for security or safety reasons. So pageinspect, adminpack, pg_buffercache, pg_standby, etc, these are all things which are tightly tied to the system. They often need to be modified when making unrelated changes to the core and can't be maintained as a separate add-on by a different group of maintainers. But they're not appropriate to install by default because they have a limited audience. Some of the modules in there are legacy modules which today would probably be done as pgfoundry modules. The GIST data types, earthdistance, isn, etc. We did actively prune out a lot of modules like that which were poorly maintained and bitrotting. The ones which remain are in reasonably good shape and have wide enough user-bases that moving them to pgfoundry would cause more upgrade pain than it would help. But that doesn't mean we're phasing contrib out entirely. The question remains whether txid is more like pageinspect/pg_standby/etc or more like earthdistance/isn. It does sound like the whole point of it is to provide an interface to core that pgfoundry modules can use, so presumably it's dealing with the nitty gritty stuff that pgfoundry modules would have trouble maintaining. Also, we only want *one* official such module. I think pushing it to pgfoundry doesn't make any sense. Does it make sense to put it in core? it has a limited audience in that only skype and slony users need it. On the other hand there's not much reason an admin wouldn't want it installed I don't think. What happens if we put it in core now and then the replication folk add more to the interface and include things that not all admins would want installed? And is the interface mature enough that users should be building applications depending on this interface directly? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] [HACKERS] Add function for quote_qualified_identifier?
Bruce Momjian escribió: > Brendan Jurd wrote: > > On 9/29/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > I think we need more than one person's request to add this function. > > > > Well, I don't expect it would get requested. Most DBAs would likely > > look for the function in the docs, see it's not there and then just > > implement it themselves. Obviously it's not critical. But > > anticipating those little requirements and providing for them is one > > of the things that makes a piece of software a pleasure to use. > > "Batteries included" and all that. > > I was just looking for someone else to say "Yea, I would like that too". Probably pgsql-hackers is not the best place to ask. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] PG on NFS may be just a bad idea
Bruce Momjian wrote: > Alvaro Herrera wrote: > > Tom Lane wrote: > > > Simon Riggs <[EMAIL PROTECTED]> writes: > > > > http://blogs.netapp.com/dave/2007/08/oracle-optimize.html > > > > > > Not a whole lot of technical content there, but pretty interesting > > > nonetheless. I *think* that the issues we're seeing are largely in the > > > NFS client-side kernel code, so bypassing that stack as Oracle is doing > > > might eliminate the problem. Of course, there's a sizable amount of > > > code to be written to do that ... > > > > Yeah. Next step we will be writing our own malloc. > > I assume there should be a ;-) in there because we already have our own > malloc (palloc). Yeah, some sort of smiley should be there. But what I'm talking about is rewriting the underlying memory allocation mechanism, just like we would be rewriting the NFS client. palloc uses malloc underneath. My thought is to replace that with sbrk, mmap or something like that. Not very portable though, a lot of work, and most likely not nearly enough benefits. -- Alvaro Herrera http://www.PlanetPostgreSQL.org/ "Nadie esta tan esclavizado como el que se cree libre no siendolo" (Goethe) ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] PG on NFS may be just a bad idea
Alvaro Herrera wrote: > Tom Lane wrote: > > Simon Riggs <[EMAIL PROTECTED]> writes: > > > http://blogs.netapp.com/dave/2007/08/oracle-optimize.html > > > > Not a whole lot of technical content there, but pretty interesting > > nonetheless. I *think* that the issues we're seeing are largely in the > > NFS client-side kernel code, so bypassing that stack as Oracle is doing > > might eliminate the problem. Of course, there's a sizable amount of > > code to be written to do that ... > > Yeah. Next step we will be writing our own malloc. I assume there should be a ;-) in there because we already have our own malloc (palloc). -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Tom Lane wrote: > Magnus Hagander <[EMAIL PROTECTED]> writes: >> Marko Kreen wrote: >>> Because of the bad timing it would have been -core call anyway >>> whether it gets in or not so Jan asked -core directly. That's >>> my explanation about what happened, obviously Jan and Tom have >>> their own opinion. > >> Right. I can see your point, but it's my understanding that -hackers is >> really the ones supposed to decide on this. > > It would ultimately have been core's decision, but the discussion should > have happened on -hackers. There was no reason for it to be private. Hmm. I thought that -core doesn't decide on things like these, they just "vote" on -hackers and have no special powers (other than being very respected community members that we all listen to, of course). I seem to recall hearing all the time (most often from people on core, but I'm certainly one of the people who relay that information further) that core specifically *doesn't* decide on things like that (being direct technical issues, or just the talk about the name-change that's been flooding -advocacy), but that core are only there for "dealing with companies that don't want to deal in public", and for making decisions "if -hackers can't agree", security sensitive stuff, and things like that. It may be that it just was like that before, and isn't anymore, and my information is outdated. I don't mind, really, because I certainly trust -core to make good decisions. But if that's so, then at least I have to change what I tell people that ask... //Magnus ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Jan Wieck wrote: > On 10/8/2007 1:34 PM, Tom Lane wrote: >> Magnus Hagander <[EMAIL PROTECTED]> writes: >>> Marko Kreen wrote: Because of the bad timing it would have been -core call anyway whether it gets in or not so Jan asked -core directly. That's my explanation about what happened, obviously Jan and Tom have their own opinion. >> >>> Right. I can see your point, but it's my understanding that -hackers is >>> really the ones supposed to decide on this. >> >> It would ultimately have been core's decision, but the discussion should >> have happened on -hackers. There was no reason for it to be private. > > That blame certainly belongs to me and I apologize for jumping that and > adding it to contrib without any -hackers discussion. > > It is definitely a timing issue since I write this very email from JFK, > boarding a flight to Hong Kong in less than an hour and will be mostly > offline for the rest of the week. > > I agree with the technical issue Tom brought up. Slony itself doesn't > rely on strtoull() either and this slipped through. I will see that I > fix that by using Slony's int64 scanning code. I can work on it during > the flight and commit the fix when I arrive in the hotel. Good. Win32 is pretty much dead right now, so we can't proceed with things like testing the buildfarm with msvc/ecpg. > To Magnus: It certainly would have been cool to have this in core, but > two weeks ago we didn't know if we can get the code into shape for that > before BETA (as it is right now I would say it still isn't). So we shot > for the next best target, which was contrib, where post BETA changes > aren't as critical. Right. My thought is still that if it isn't good enough for core, it shouldn't be in contrib. If it *is* good enough, and we want it, we should accept that it came in long after freeze and put it in core anyway. If it *isn't*, then it should be on pgfoundry and be moved into core when it's ready - for 8.4 or so. The whole contrib thing confuses a lot of users. Is it included, or isn't it? IMHO, that distinction need to be clear, and I thought we were working (if not actively then at least passively) to "retire" contrib, moving things either to core or to pgFoundry. Adding yet another important feature that's "just in contrib" is making things worse, not better. IMHO, of course ;-) //Magnus ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended trans actionID module to contrib as
> --- Original Message --- > From: Andrew Dunstan <[EMAIL PROTECTED]> > To: Marko Kreen <[EMAIL PROTECTED]> > Sent: 08/10/07, 18:05:57 > Subject: Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended > transactionID module to contrib as > > > Surely there should at the very least have been a patch posted, core > approval or not. There was no 'core approval', just a couple of people saying they didn't think it would be a problem to include. Regards, Dave ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
On 10/8/07, Magnus Hagander <[EMAIL PROTECTED]> wrote: > Marko Kreen wrote: > > On 10/8/07, Magnus Hagander <[EMAIL PROTECTED]> wrote: > >> On Sun, Oct 07, 2007 at 11:32:19PM +, Jan Wieck wrote: > >>> Log Message: > >>> --- > >>> Added the Skytools extended transaction ID module to contrib as discussed > >>> on CORE previously. > > > > To explain the situation, the public discussion about the current > > submission happened here: > > > > > > http://lists.pgfoundry.org/pipermail/skytools-users/2007-September/000245.html > > > > and here: > > > > http://lists.slony.info/pipermail/slony1-hackers/2007-September/57.html > > Ok. That certainly explains it - it did sound weird to have that go in > without any public discussion at all - but none of those lists are > pgsql-hackers ;-) Yes, sorry about that. Just the discussion started very hypetetically, with us probing each other opinion, and there was nothing to discuss with -hackers. When we saw a concrete plan for the module, then it was too late to do a regular -hackers submission, due to the beta timeline we needed -core opinion immidiately. Now, after -core gave a nod, then yes, the patch should have been to -hackers with a notice that it is on fast-path. [btw, this is me guessing Jan's thinking, but I would have acted same way.] > > Now as you can read from recent disussion we had, we found out > > that it would be *really* *really* cool if it would appear > > in 8.3... Talk about last moment... > > Well, if it's really really cool to have, why do we put it in /contrib? > If it's that cool, it should be in core, no? Main cool thing came from fact that this is the last moment Slony could do such big conversion of it's codebase. > That's not just making comments, I really *do* think that it should be > in core if it's interesting enough to be added to contrib at this time. Yes, it is cool enough to be in core and I think that's the goal. But asking for it to be accepted to core a week before beta and couple months after feature freeze would be asking bit too much, don't you think? ;) -- marko ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
On 10/8/2007 1:34 PM, Tom Lane wrote: Magnus Hagander <[EMAIL PROTECTED]> writes: Marko Kreen wrote: Because of the bad timing it would have been -core call anyway whether it gets in or not so Jan asked -core directly. That's my explanation about what happened, obviously Jan and Tom have their own opinion. Right. I can see your point, but it's my understanding that -hackers is really the ones supposed to decide on this. It would ultimately have been core's decision, but the discussion should have happened on -hackers. There was no reason for it to be private. That blame certainly belongs to me and I apologize for jumping that and adding it to contrib without any -hackers discussion. It is definitely a timing issue since I write this very email from JFK, boarding a flight to Hong Kong in less than an hour and will be mostly offline for the rest of the week. I agree with the technical issue Tom brought up. Slony itself doesn't rely on strtoull() either and this slipped through. I will see that I fix that by using Slony's int64 scanning code. I can work on it during the flight and commit the fix when I arrive in the hotel. To Magnus: It certainly would have been cool to have this in core, but two weeks ago we didn't know if we can get the code into shape for that before BETA (as it is right now I would say it still isn't). So we shot for the next best target, which was contrib, where post BETA changes aren't as critical. Jan -- #==# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #== [EMAIL PROTECTED] # ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting to 8.3 beta1
Alvaro Herrera wrote: > > Greg Smith wrote: > > > On Thu, 27 Sep 2007, Tom Lane wrote: > > > > > >> Also, I spent a dreary two or three hours this afternoon examining the > > >> CVS commit logs since 8.3 branched...I tried to post that info to > > >> pgsql-docs but it broke the list's message size limits (even gzipped, > > >> it's about 90K). > > > > Bruce you mentioned in a post earlier this week that you were going to > > jump back on the release notes. Earlier last week, I said I would jump > > on them as did Greg. > > > > Greg went through the effort to do the below... shall we use it together? > > Too late. Tom already did a lot of the work. See > > http://developer.postgresql.org/cvsweb.cgi/pgsql/doc/src/sgml/release.sgml?r1=1.508&r2=1.509 > > or the HTML version at > http://momjian.us/main/writings/pgsql/sgml/release-8-3.html Sorry I dropped the ball on the release notes. My wife had minor surgery last Monday and I hoped to be freed up earlier but the recovery was longer than expected. I am back to reading email now. Thanks to Tom for doing the release notes, and to others for helping. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] [HACKERS] Add function for quote_qualified_identifier?
Brendan Jurd wrote: > On 9/29/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > I think we need more than one person's request to add this function. > > Well, I don't expect it would get requested. Most DBAs would likely > look for the function in the docs, see it's not there and then just > implement it themselves. Obviously it's not critical. But > anticipating those little requirements and providing for them is one > of the things that makes a piece of software a pleasure to use. > "Batteries included" and all that. I was just looking for someone else to say "Yea, I would like that too". > Anyway, I seem to be flogging a horse which, if not dead, is surely > mortally wounded. If quote_qualified_ident isn't desired, perhaps you > can still use the regression test I included for quote_ident in the > patch. The test is functional as a standalone item, and seems to fill > a gap. Well, we don't test everything and I don't remember problems in quoting in the past, at least not enough to add another test. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Magnus Hagander <[EMAIL PROTECTED]> writes: > Marko Kreen wrote: >> Because of the bad timing it would have been -core call anyway >> whether it gets in or not so Jan asked -core directly. That's >> my explanation about what happened, obviously Jan and Tom have >> their own opinion. > Right. I can see your point, but it's my understanding that -hackers is > really the ones supposed to decide on this. It would ultimately have been core's decision, but the discussion should have happened on -hackers. There was no reason for it to be private. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Including Snapshot Info with Indexes
Gokulakannan Somasundaram wrote: > I am always slightly late in understanding things. Let me try > to understand the use of DSM. It is a bitmap index on whether all the tuples > in a particular block is visible to all the backends, whether a particular > block contains tuples which are invisible to everyone. But i think this will > get subjected to the same limitations of Bitmap index. Even Oracle suggests > the use of Bitmap index for only data warehousing tables, where the Bitmap > indexes will be dropped and recreated after every bulk load. This is not a > viable alternative for OLTP transactions. Well, it's not quite the same as a bitmap index, though both use a bitmap. You didn't quite get into details on what the limitations are and why it wouldn't be suitable for OLTP, but I don't see any significant problems. > But i think i am late in the game > as i haven't participated in those discussions Better late than never :). >One Bitmap index block usually maps to lot of blocks in the heap. > So locking of one page to update the DSM for update/delete/insert would hit > the concurrency. But again all these are my observation w.r.t oracle bitmap > indexes. May be i am missing something in DSM. Yeah, the DSM page could become a contention bottleneck. My current thinking is that we'd have a flag in the heap page header, that would be set together with the bit in the DSM. When the flag in the page header is set, you don't need to lock and update the DSM because you know the bit is already set. Vacuum would have to clear both the DSM bit and the flag. > Let's take up Retail Vacuuming again. The User defined function > which would return different values at different time can be classified as > non-deterministic functions. We can say that this index cannot be created > on a non-deterministic function. This is the way it is implemented in > Oracle. What they have done is they have classified certain built-in > operators and functions as deterministic. Similarly they have classified a > few as non-deterministic operators and functions. Can we follow a similar > approach? We already do. A function must be marked as IMMUTABLE in order to use it in an index expression. But we can't enforce that the user defined function really behaves like an immutable function should. If someone creates a user-defined function in C that calls the C random() function, we can't stop it. As I said earlier, using an index like that will of course lead to bogus results. But it won't currently cause any server crashes or more serious corruption. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Marko Kreen wrote: > On 10/8/07, Magnus Hagander <[EMAIL PROTECTED]> wrote: >> On Sun, Oct 07, 2007 at 11:32:19PM +, Jan Wieck wrote: >>> Log Message: >>> --- >>> Added the Skytools extended transaction ID module to contrib as discussed >>> on CORE previously. > > To explain the situation, the public discussion about the current > submission happened here: > > > http://lists.pgfoundry.org/pipermail/skytools-users/2007-September/000245.html > > and here: > > http://lists.slony.info/pipermail/slony1-hackers/2007-September/57.html Ok. That certainly explains it - it did sound weird to have that go in without any public discussion at all - but none of those lists are pgsql-hackers ;-) > And ofcourse, the original submission was at 2006-07 to _8.2_: > > http://archives.postgresql.org/pgsql-patches/2006-07/msg00157.php Ah. I only searched for this year, since I only considered submissions for 8.3. But still, it wasn't AFAIK on any of the patch lists etc. > It was rejected then mostly on 3 reasons (from my POV): > > - it was messy and contained unnecesary cruft. > - it was submitted to core not /contrib > - slony was not interested in it at that moment > > Now as you can read from recent disussion we had, we found out > that it would be *really* *really* cool if it would appear > in 8.3... Talk about last moment... Well, if it's really really cool to have, why do we put it in /contrib? If it's that cool, it should be in core, no? That's not just making comments, I really *do* think that it should be in core if it's interesting enough to be added to contrib at this time. > Because of the bad timing it would have been -core call anyway > whether it gets in or not so Jan asked -core directly. That's > my explanation about what happened, obviously Jan and Tom have > their own opinion. Right. I can see your point, but it's my understanding that -hackers is really the ones supposed to decide on this. //Magnus ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Including Snapshot Info with Indexes
Gokulakannan Somasundaram wrote: Hi Heikki, I am always slightly late in understanding things. Let me try to understand the use of DSM. It is a bitmap index on whether all the tuples in a particular block is visible to all the backends, whether a particular block contains tuples which are invisible to everyone. But i think this will get subjected to the same limitations of Bitmap index. Even Oracle suggests the use of Bitmap index for only data warehousing tables, where the Bitmap indexes will be dropped and recreated after every bulk load. This is not a viable alternative for OLTP transactions. But i think i am late in the game as i haven't participated in those discussions While the DSM might be similar in spirit to a bitmap index, the actual implementation has a lot more freedome I'd say, since you can tailor it exactly to the need of tracking some summarized visibility info. So not all shortcomings of bitmap indices must necessarily apply to the DSM also. But of course thats mostly handwavering... One Bitmap index block usually maps to lot of blocks in the heap. So locking of one page to update the DSM for update/delete/insert would hit the concurrency. But again all these are my observation w.r.t oracle bitmap indexes. May be i am missing something in DSM. A simple DSM would probably contain a bit per page that says "all xmin < GlobalXmin, and all xmax unset or aborted". That bit would only get SET during VACUUM, and only unset during INSERT/UPDATE/DELETE. If setting it is protected by a VACUUM-grade lock on the page, we might get away with no locking during the unset, making the locking overhead pretty small. I couldn't get that piece of discussion in the archive, which discusses the design of Retail Vacuum. So please advise me again here. Let's take up Retail Vacuuming again. The User defined function which would return different values at different time can be classified as non-deterministic functions. We can say that this index cannot be created on a non-deterministic function. This is the way it is implemented in Oracle. What they have done is they have classified certain built-in operators and functions as deterministic. Similarly they have classified a few as non-deterministic operators and functions. Can we follow a similar approach? Postgres already distinguishes VOLATILE,STABLE and IMMUTABLE functions. It doesn't, however, risk physical data corruption, even if you get that classification wrong. The worst that happens AFAIK are wrong query results - but fixing your function, followed by a REINDEX always corrects the problme. If you start poking holes into that safety net, there'll be a lot of pushback I believe - and IMHO rightly so, because people do, and always will, get such classifications wrong. greetings, Florian Pflug ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Marko Kreen wrote: Now as you can read from recent disussion we had, we found out that it would be *really* *really* cool if it would appear in 8.3... Talk about last moment... That discussion didn't happen on any list I read, so to me it just came as a bolt from the blue. Surely there should at the very least have been a patch posted, core approval or not. cheers andrew ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
On 10/8/07, Magnus Hagander <[EMAIL PROTECTED]> wrote: > On Sun, Oct 07, 2007 at 11:32:19PM +, Jan Wieck wrote: > > Log Message: > > --- > > Added the Skytools extended transaction ID module to contrib as discussed > > on CORE previously. To explain the situation, the public discussion about the current submission happened here: http://lists.pgfoundry.org/pipermail/skytools-users/2007-September/000245.html and here: http://lists.slony.info/pipermail/slony1-hackers/2007-September/57.html And ofcourse, the original submission was at 2006-07 to _8.2_: http://archives.postgresql.org/pgsql-patches/2006-07/msg00157.php It was rejected then mostly on 3 reasons (from my POV): - it was messy and contained unnecesary cruft. - it was submitted to core not /contrib - slony was not interested in it at that moment Now as you can read from recent disussion we had, we found out that it would be *really* *really* cool if it would appear in 8.3... Talk about last moment... Because of the bad timing it would have been -core call anyway whether it gets in or not so Jan asked -core directly. That's my explanation about what happened, obviously Jan and Tom have their own opinion. -- marko ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Including Snapshot Info with Indexes
Hi Heikki, I am always slightly late in understanding things. Let me try to understand the use of DSM. It is a bitmap index on whether all the tuples in a particular block is visible to all the backends, whether a particular block contains tuples which are invisible to everyone. But i think this will get subjected to the same limitations of Bitmap index. Even Oracle suggests the use of Bitmap index for only data warehousing tables, where the Bitmap indexes will be dropped and recreated after every bulk load. This is not a viable alternative for OLTP transactions. But i think i am late in the game as i haven't participated in those discussions One Bitmap index block usually maps to lot of blocks in the heap. So locking of one page to update the DSM for update/delete/insert would hit the concurrency. But again all these are my observation w.r.t oracle bitmap indexes. May be i am missing something in DSM. I couldn't get that piece of discussion in the archive, which discusses the design of Retail Vacuum. So please advise me again here. Let's take up Retail Vacuuming again. The User defined function which would return different values at different time can be classified as non-deterministic functions. We can say that this index cannot be created on a non-deterministic function. This is the way it is implemented in Oracle. What they have done is they have classified certain built-in operators and functions as deterministic. Similarly they have classified a few as non-deterministic operators and functions. Can we follow a similar approach? Thanks, Gokul. On 10/8/07, Heikki Linnakangas <[EMAIL PROTECTED]> wrote: > > Gokulakannan Somasundaram wrote: > > On 10/8/07, Heikki Linnakangas <[EMAIL PROTECTED]> wrote: > >> IMO, the most promising approach to achieving index-only-scans at the > >> moment is the Dead Space Map, as discussed in the 8.3 dev cycle. > > > > Index only scans means that in order to get certain results, we may not > > goto the table at all. For example, if you have an index on columns a > and b, > > and if there is a query like "select b from table where a between a1 and > > a2", then the explain plan need not goto the table. I can't understand > how > > dead space map will provide such a functionality. > > Please read the discussions in the archives. The dead space map holds > visibility information in a condensed form. For index-only-scans, we > need to know if all tuples on page are are visible to us. If the dead > space map is designed with index-only-scans in mind, we can store a bit > there indicating "all tuples on this page are visible to everyone". > Pages that have that bit set don't need to be visited to check visibility. > > What information exactly is going to be stored in the dead space map is > still debated. For vacuuming, we need to know which pages contain dead > tuples that are worth vacuuming, which isn't the same thing as "all > tuples are visible to everyone", but it's quite close. > > Heap pages that do have dead or recently modified rows do need to be > visited, so the executor needs to always be prepared to check visibility > from the heap. However, on a table that's not very frequently updated, > most bits will be set and the effect will be the same as genuine > index-only-scans. On a table that is frequently updated, you would > suffer a big hit in update performance with the "duplicate visibility > information in all indexes" scheme as well, as the updates would need to > update the indexes as well, so the performance characteristics are > roughly the same. > > > That's true. But i am not asking to replace the current index > > implementation, but to provide an extra option while indexing. Say if a > > particular database setup doesn't do much deletes and updates(imagine > tables > > with partitioning, where the partitions/tables are dropped instead of > > deletes. They can have an option to "create index .. with snapshot" > > A nice property of utilizing the dead space map for index-only-scans is > that it doesn't require any action from the DBA. It will "just work". It > also adapts well to tables that have parts that are frequently updated, > and other parts that are not. The frequently updated parts will behave > like what we have now, index-only-scans are not possible because, but > deletes/updates are cheap. But the less frequently updated parts will > eventually have the bits set in the map, and we can do index-only-scans > for those parts. > > > Imagine the Index Vacuum also will do lesser Random I/Os > > Index vacuum doesn't do random I/Os as it is. > > > Also, the full visibility information would need 12 bytes of space per > >> tuple. An index tuple on an int4 key currently takes 12 bytes, so that > >> would double the index size. Storage size has a big impact on > >> performance. More bytes means more I/O, less data fits in cache, and > >> more WAL traffic. > > > > I am thinking of certain optimizations her
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Andrew Dunstan <[EMAIL PROTECTED]> writes: > I share your other concerns. This process certainly seems to have been > less than transparent. FWIW, Jan asked -core about a week ago whether it'd be okay to add this code post-beta, and we concluded it would be okay on the grounds that (1) we've always been laxer about contrib than the core code, (2) we'd already granted similar permission to Oleg and Teodor to add some example tsearch dictionaries to contrib post-beta. But I was expecting some -hackers discussion and/or a proposed patch on -patches next. I agree that Jan skipped a step. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
Magnus Hagander wrote: Or I could've missed the discussion on -hackers that actually took place - in that case, just discard this message. but the only one I recall is someone asking for pl/proxy to go in. There was some discussion (subject: Provide 8-byte transaction IDs to user level), but that itself happened after feature freeze and didn't seem too conclusive. I share your other concerns. This process certainly seems to have been less than transparent. cheers andrew ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Improving the Performance of Full Table Updates
"Tom Lane" <[EMAIL PROTECTED]> writes: > "Heikki Linnakangas" <[EMAIL PROTECTED]> writes: >> * mmap the WAL segments, instead of using the slru buffers. > > This one's almost certainly a nonstarter, for lack of any portable way > to control when writes happen. I think mlock and msync(MS_SYNC) ought to be enough. I'm not sure every OS implements them as specified though. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Improving the Performance of Full Table Updates
"Heikki Linnakangas" <[EMAIL PROTECTED]> writes: > We mustn't write data pages before the WAL hits the disk, but we don't > have any such limitation for WAL. Um, you're right, I was remembering discussions about trying to mmap data files. Still, googling "msync performance" leaves me full of concern about whether this would be a portable improvement. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] [COMMITTERS] pgsql: Added the Skytools extended transaction ID module to contrib as
On Sun, Oct 07, 2007 at 11:32:19PM +, Jan Wieck wrote: > Log Message: > --- > Added the Skytools extended transaction ID module to contrib as discussed > on CORE previously. > > This module offers transaction ID's containing the original XID and the > transaction epoch as a bigint value to the user level. It also provides > a special txid_snapshot data type that contains an entire transactions > visibility snapshot information, which is useful to determine if a > particular txid was visible to a transaction or not. > > The module has been tested by porting Slony-I from using its original > xxid data type. > > Jan A couple of questions on this. I'm not objecting to the tech stuff itself, but on procedure: 1) Why was this added without any discussion, or even notification, on -hackers before it was done? Last I checked, -core claim not to deal with such technicalities, but defer to -hackers (or other lists as needed). I certainly trust -core to make the right call no these things, but it's not the procedure that we claim to have. If that procedure is changing (I've been getting a sneaky feeling that it's tilting a bit in that direction before this one), that's fine, but it should be communicated to the community so everybody knows how it works. 2) How can this go in *months* after feature freeze, and even after we tagged and bundled beta1? This makes such discission even more important, IMHO. 3) I thought the general agreement was to cut down on contrib, and move things either to pgfoundry or to core. Why are we adding more? I'm sure there's motivation for this as discussed on -core, but the rest of us would like to know that as well... Like why we're not trying to make it a real feature, if it's something that's important enough to break the rules as in #2 above. Or I could've missed the discussion on -hackers that actually took place - in that case, just discard this message. but the only one I recall is someone asking for pl/proxy to go in. //Magnus ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Improving the Performance of Full Table Updates
Tom Lane wrote: > "Heikki Linnakangas" <[EMAIL PROTECTED]> writes: >> * mmap the WAL segments, instead of using the slru buffers. > > This one's almost certainly a nonstarter, for lack of any portable way > to control when writes happen. For flushing, there's msync, which I believe is quite portable. We mustn't write data pages before the WAL hits the disk, but we don't have any such limitation for WAL. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Improving the Performance of Full Table Updates
"Heikki Linnakangas" <[EMAIL PROTECTED]> writes: > * mmap the WAL segments, instead of using the slru buffers. This one's almost certainly a nonstarter, for lack of any portable way to control when writes happen. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Another Idea: Try Including snapshot with TOAS (was: Including Snapshot Info with Indexes)
On Mon, 2007-10-08 at 14:58 +0300, Hannu Krosing wrote: > 1. get rid of indexes for TOAST tables > > instead of TOAST tuple id store CTID of first TOAST block directly, and > use something like skip lists inside the TOAST block headers to access > next TOAST tuples. It should be possible to optimise TOAST for when there is just a single chunk that is toasted. Since we often (and by default) compress data stored in TOAST tables this would be a frequently used optimisation. Instead of storing the TOAST OID, which is then looked-up in the index to find the TOAST tid, we can just store the tid directly in the toast pointer on the main heap. That way we'll get faster read and write access for a good proportion of rows by avoiding the toast index and going straight to the toast heap. We'd need a different kind of toast pointer which would be 2 bytes longer than the normal pointer. I think we have a spare flag bit to indicate this. That's just a rough sketch, I've not checked the details on this. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] proposal casting from XML[] to int[], numeric[], text[]
Peter Eisentraut wrote: Am Freitag, 28. September 2007 schrieb Nikolay Samokhvalov: what should be returned for XML like "PostgreSQL is a powerful, open source relational database system" if user requests for text under "em" node? In XML world, the correct answer is "PostgreSQL is a powerful, open source relational database system" -- concatenation of all strings from the node itself and all its descendants, in the correct order. Will be this expected for RDBMS users?). Well, if that is the defined behavior for XPath, then that's what we should do. The xpath string value of a single node is the concatentation of the text children of the node and all its children in document order, IIRC. But that's not what we're dealing with here. xpath() doesn't return a single node but a node set (or so say the docs). The string value of a node set is in effect the string value of its first member, which seems less than useful in this context, or at least no great guide for us. I think there's probably a good case for a cast from xml[] to text[] if we don't have one. cheers andrew ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[HACKERS] Another Idea: Try Including snapshot with TOAS (was: Including Snapshot Info with Indexes)
Ühel kenal päeval, E, 2007-10-08 kell 12:12, kirjutas Gokulakannan Somasundaram: > Hi, > Currently The index implementation in Postgresql does not store > the Snapshot information in the Index. .. > Please reply back with your comments. I think you got enoght "search for previous discussion" answers on this aone already ;) So I propose a few another ideas to investigate 1. get rid of indexes for TOAST tables instead of TOAST tuple id store CTID of first TOAST block directly, and use something like skip lists inside the TOAST block headers to access next TOAST tuples. 2. store visibility info in TOAST tables if you store visibility info in TOAST tuples, it becomes possible to update just the small part of the tuple in the main heap and keep the bulk of toasted data in place. both of these ideas are much more complicated to implement than it appears from my simple description, but should have big benefits for a sizable number of scenarios which use TOAST. - Hannu ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Including Snapshot Info with Indexes
Ühel kenal päeval, E, 2007-10-08 kell 11:41, kirjutas Heikki Linnakangas: > The dead space map holds > visibility information in a condensed form. For index-only-scans, we > need to know if all tuples on page are are visible to us. If the dead > space map is designed with index-only-scans in mind, we can store a bit > there indicating "all tuples on this page are visible to everyone". > Pages that have that bit set don't need to be visited to check visibility. > > What information exactly is going to be stored in the dead space map is > still debated. For vacuuming, we need to know which pages contain dead > tuples that are worth vacuuming, which isn't the same thing as "all > tuples are visible to everyone", but it's quite close. I would prefer a separate MVC visibility heap (aka. extended "dead space map") which would duplicate whole visibility info from heap pages, just in compressed form. After a few releases with duplicated visibility info, we could remove it from the data heap. If the whole visibility info (cmin, cmax, tmin, tmax, flags, (+ size for DSM uses)) for tuples, were in a separate heap, it would allow for a lot of on-the-fly compression. for example we could: * get rid of both tmin and tmax for all completed transactions * reduce any deleted tuple to just flags * reduce any tuple produced by aborted transaction to just flags * reduce any tuple visible to all backends to just flags * RRL compress (runs of) pages produced by same transaction * RRL compress (runs of) pages with all tuples visible * RRL compress (runs of) pages with all tuples deleted depending on distribution of Inserts/Updates/Deletes it will make visibility info a little or a lot smaller than it is currently, greatly enchancing chances that it stays in cache (even for OLAP loads, because data for these are usually produced by bulk inserts and thus their visibility info is highly compressable) It also makes VACUUM more efficient, as it's initial scan (find vacuumable tuples) will need to do lot less IO. And it allows for more intelligent choices for new tuple placement , especially if we want to preserve clustering. And of course it gives you kind of index-only scans (mostly read index + check in vis.heap) - Hannu ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] proposal casting from XML[] to int[], numeric[], text[]
Am Freitag, 28. September 2007 schrieb Nikolay Samokhvalov: > what should be returned for XML like "PostgreSQL > is a powerful, open source relational database system" if user > requests for text under "em" node? In XML world, the correct answer is > "PostgreSQL is a powerful, open source relational database system" -- > concatenation of all strings from the node itself and all its > descendants, in the correct order. Will be this expected for RDBMS > users?). Well, if that is the defined behavior for XPath, then that's what we should do. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Including Snapshot Info with Indexes
Gokulakannan Somasundaram wrote: > On 10/8/07, Heikki Linnakangas <[EMAIL PROTECTED]> wrote: >> IMO, the most promising approach to achieving index-only-scans at the >> moment is the Dead Space Map, as discussed in the 8.3 dev cycle. > > Index only scans means that in order to get certain results, we may not > goto the table at all. For example, if you have an index on columns a and b, > and if there is a query like "select b from table where a between a1 and > a2", then the explain plan need not goto the table. I can't understand how > dead space map will provide such a functionality. Please read the discussions in the archives. The dead space map holds visibility information in a condensed form. For index-only-scans, we need to know if all tuples on page are are visible to us. If the dead space map is designed with index-only-scans in mind, we can store a bit there indicating "all tuples on this page are visible to everyone". Pages that have that bit set don't need to be visited to check visibility. What information exactly is going to be stored in the dead space map is still debated. For vacuuming, we need to know which pages contain dead tuples that are worth vacuuming, which isn't the same thing as "all tuples are visible to everyone", but it's quite close. Heap pages that do have dead or recently modified rows do need to be visited, so the executor needs to always be prepared to check visibility from the heap. However, on a table that's not very frequently updated, most bits will be set and the effect will be the same as genuine index-only-scans. On a table that is frequently updated, you would suffer a big hit in update performance with the "duplicate visibility information in all indexes" scheme as well, as the updates would need to update the indexes as well, so the performance characteristics are roughly the same. > That's true. But i am not asking to replace the current index > implementation, but to provide an extra option while indexing. Say if a > particular database setup doesn't do much deletes and updates(imagine tables > with partitioning, where the partitions/tables are dropped instead of > deletes. They can have an option to "create index .. with snapshot" A nice property of utilizing the dead space map for index-only-scans is that it doesn't require any action from the DBA. It will "just work". It also adapts well to tables that have parts that are frequently updated, and other parts that are not. The frequently updated parts will behave like what we have now, index-only-scans are not possible because, but deletes/updates are cheap. But the less frequently updated parts will eventually have the bits set in the map, and we can do index-only-scans for those parts. > Imagine the Index Vacuum also will do lesser Random I/Os Index vacuum doesn't do random I/Os as it is. > Also, the full visibility information would need 12 bytes of space per >> tuple. An index tuple on an int4 key currently takes 12 bytes, so that >> would double the index size. Storage size has a big impact on >> performance. More bytes means more I/O, less data fits in cache, and >> more WAL traffic. > > I am thinking of certain optimizations here. we have a bit unused in > indextuple structure. If a particular tuple is not deleted, then we can > signify that using that bit and save 6 bytes of saving the xmax and cmax. We > are trading of this space efficiency in place of Random I/Os, which is not a > bad trade-off , i suppose. Again this is going to optional for the user. If > users have an option to create Bitmap index/ Binary index, why can't they > have this option as well? Because we have to maintain it? :) Speaking of bitmap indexes, that would be nice to have. It looks like Gavin dropped the ball on the patch, so if you want to continue that work, that would be great. > There's non-trivial implementation issues involved as well. You'd need a >> way to reliably find all the index pointers for a given heap tuple >> (search the archives for "retail vacuum" for the issues involved in >> that. Broken user defined functions are a problem for example). And >> you'd need to keep them all locked at the same time to modify them all >> atomically, which is prone to deadlocks. > > I think Vacuum need not goto the table, as the visibility information is > present in the index itself. Vacuum isn't the problem here. The problem is: given heap tuple X, how do you find the the corresponding index tuples? The obvious solution is to calculate the index keys from the heap tuple, and use them to look up. But what if you have an expression index on a user-defined function, and that user-defined function is broken so that it returns a different value than when the index tuple was inserted? You won't find the index tuples in that case, so you won't be able to update the visibility information. Granted, you've got a broken user-defined-function in that case, and you're going to get bogus query results anyway. But not f
Re: [HACKERS] Encoding and i18n
Am Samstag, 6. Oktober 2007 schrieb Tom Lane: > It's not real clear to me whether, on a Unix machine, there is even > supposed to be any difference between setting LC_TIME=es_ES.iso88591 and > setting it to es_ES.utf8. Since nl_langinfo(CODESET) is supposedly > determined only by LC_CTYPE, you could argue that strftime's results > should be in that encoding regardless, and that the codeset component of > other LC_ variables should be ignored. Some experimentation suggests > that at least in glibc it doesn't work that way, and that there is in > fact no principled way for you to find out what encoding strftime is > giving you :-(. It might be useful to research whether that behavior is following the spec (POSIX or whatever). -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Encoding and i18n
Am Sonntag, 7. Oktober 2007 schrieb Gregory Stark:
> "Tom Lane" <[EMAIL PROTECTED]> writes:
> > Since nl_langinfo(CODESET) is supposedly determined only by LC_CTYPE, you
> > could argue that strftime's results should be in that encoding
> > regardless,
>
> It seems to me we aren't actually using strftime any more in any case. We
> seem to be using things like _("Monday") instead.
I seem to recall that we don't use strftime *yet*, exactly because of this
sort of issue. This was discussed before the 8.2 release.
--
Peter Eisentraut
http://developer.postgresql.org/~petere/
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
Re: [HACKERS] Including Snapshot Info with Indexes
On 10/8/07, Heikki Linnakangas <[EMAIL PROTECTED]> wrote: > > Csaba Nagy wrote: > > On Mon, 2007-10-08 at 09:40 +0100, Heikki Linnakangas wrote: > >> This idea has been discussed to death many times before. Please search > >> the archives. > > > > Somewhat related to the "visibility in index" thing: would it be > > possible to have a kind of index-table ? We do have here some tables > > which have 2-4 fields, and the combination of them forms the primary key > > of the table. There are usually no other indexes on the table, and the > > net result is that the PK index duplicates the heap. So it would be cool > > if the index would be THE heap somehow... > > The clustered index patch I worked on for 8.3, but didn't make it in, > would have helped that use case a lot. > > A column-store kind of mechanism would be nice. Some columns could be > stored in index-like structures, while other would be in the heap. I > haven't seen any practical proposal on how to do it though. I think it more resembles the Oracle's IOT with overflow. I think my proposal, once implemented can be easily extended to incorporate IOT/Clustered indexes. One thing is for sure. Without storing Visibility info, Unique Secondary indexes(Indexes on IOT/Clustered indexed tables) is not possible, as it is not possible to re-locate the same entry in a b-tree, if we are storing the Primary key in place of tuple-id. -- > Heikki Linnakangas > EnterpriseDB http://www.enterprisedb.com >
Re: [HACKERS] Including Snapshot Info with Indexes
On 10/8/07, Heikki Linnakangas <[EMAIL PROTECTED]> wrote: > > Gokulakannan Somasundaram wrote: > > Currently The index implementation in Postgresql does not store the > > Snapshot information in the Index. If we add the snapshot information > into > > the indexing structure, we will have the following advantages. > > This idea has been discussed to death many times before. Please search > the archives. > > > a) There can be index only scans like Oracle > > IMO, the most promising approach to achieving index-only-scans at the > moment is the Dead Space Map, as discussed in the 8.3 dev cycle. Index only scans means that in order to get certain results, we may not goto the table at all. For example, if you have an index on columns a and b, and if there is a query like "select b from table where a between a1 and a2", then the explain plan need not goto the table. I can't understand how dead space map will provide such a functionality. In short each index will act like an Index Organized Table, if the all the columns of the query are present in the index. > b) Unique indexes will become less costly, as older index tuples can be > > found out. > > Doesn't seem like a big benefit, considering that in most cases there > won't be any tuples in the index with a duplicate key. A common > exception to that is (non-HOT) updating a row. But in that case, the > page containing the old tuple is already in cache, so the lookup of the > visibility from the heap is cheap. Its not a big benefit. agreed. > c) Even the index scans will get faster, since some of the index tuples > > won't translate into HeapScans. > > That's the same as doing an index-only-scan, right? No here if you have an index on a(say). If there is a query like select * form table where a between a1 and a2, currently the scan goes to the table to verify the visibility. Of course if the tuple satisfies vacuum, then it is marked in the index, which is an optimization. This is not index-only scan. This is a normal index scan, which can skip certain random I/Os. > d) Deletes and Updates will become slightly costly, as they have to update > > these indexes. > > I think you're grossly underestimating the cost of that. For example, on > a table with 3 indexes. a delete currently requires one index lookup + > one heap lookup. With visibility in the indexes, that would require 3 > index lookups + one heap lookup. That's 4 vs. 2 page accesses, not > taking into account the non-leaf b-tree pages. The real impact will > depend on what's in cache, but the cost can be very high. That's true. But i am not asking to replace the current index implementation, but to provide an extra option while indexing. Say if a particular database setup doesn't do much deletes and updates(imagine tables with partitioning, where the partitions/tables are dropped instead of deletes. They can have an option to "create index .. with snapshot" Imagine the Index Vacuum also will do lesser Random I/Os Also, the full visibility information would need 12 bytes of space per > tuple. An index tuple on an int4 key currently takes 12 bytes, so that > would double the index size. Storage size has a big impact on > performance. More bytes means more I/O, less data fits in cache, and > more WAL traffic. I am thinking of certain optimizations here. we have a bit unused in indextuple structure. If a particular tuple is not deleted, then we can signify that using that bit and save 6 bytes of saving the xmax and cmax. We are trading of this space efficiency in place of Random I/Os, which is not a bad trade-off , i suppose. Again this is going to optional for the user. If users have an option to create Bitmap index/ Binary index, why can't they have this option as well? There's non-trivial implementation issues involved as well. You'd need a > way to reliably find all the index pointers for a given heap tuple > (search the archives for "retail vacuum" for the issues involved in > that. Broken user defined functions are a problem for example). And > you'd need to keep them all locked at the same time to modify them all > atomically, which is prone to deadlocks. I think Vacuum need not goto the table, as the visibility information is present in the index itself. I don't know whether i have given the correct answer here. Expecting your reply.. Thanks, Gokul.
Re: [HACKERS] Improving the Performance of Full Table Updates
Gokulakannan Somasundaram wrote: >Finally i found some time to look more into the CRC code. The > time is mostly taken when there is a back-up block in the XLOG structure. > when it calculates the CRC for the entire block(there is some optimization > already for the holes), the time is spent on the CRC macro. I tried doing > some small optimization like changing the int into uint8, thinking that the > exponentiation operation might get slightly faster, with no success. I don't > know whether changing the CRC algorithm is a good option. Yeah, I believe the current implementation is quite well-optimized, so it's unlikely that you'll find any micro-optimizations like that that make any significant difference. I'm afraid you're going to need bigger changes to make it faster. Some ideas that I've had in the past that might be worth investigating: * Use a much bigger WAL block size, to avoid splitting each big record, like one with a backup block. One block / one WAL segment might not be a bad idea. * mmap the WAL segments, instead of using the slru buffers. Even if it didn't help with performance per se, we could get rid of the wal_buffers setting and let the OS manage the caching. * Use a cheaper CRC algorithm. * Calculate CRC when flushing, instead of inserting, the WAL. This would allow the WAL writer to pick up much the of the CRC work, increasing throughput on multi-core systems, particularly on bulk loads. * Cover multiple WAL records with the CRC. Presumably calculating the CRC of a large contiguous chunk cheaper than n smaller chunks. * Allow buffering of multiple WAL segments, to avoid the WAL flush on every xlog segment switch. * Reorder WAL records on the fly, so that if you have something like "full page image + update + update + update + update" all happening on a single page, we could skip writing the individual update records, and take a full page image of the page after the last update instead. (getting all the locking right is quite tricky...) * Change the locking so that you don't need to hold the WALInsertLock while memcpying the record to the WAL block, but just to allocate the slot for it. Wouldn't increase single-process performance, but would make it more scalable. I haven't pursued any of these further, so some or all of them might turn out to be not helpful or not feasible. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Including Snapshot Info with Indexes
Csaba Nagy wrote: > On Mon, 2007-10-08 at 09:40 +0100, Heikki Linnakangas wrote: >> This idea has been discussed to death many times before. Please search >> the archives. > > Somewhat related to the "visibility in index" thing: would it be > possible to have a kind of index-table ? We do have here some tables > which have 2-4 fields, and the combination of them forms the primary key > of the table. There are usually no other indexes on the table, and the > net result is that the PK index duplicates the heap. So it would be cool > if the index would be THE heap somehow... The clustered index patch I worked on for 8.3, but didn't make it in, would have helped that use case a lot. A column-store kind of mechanism would be nice. Some columns could be stored in index-like structures, while other would be in the heap. I haven't seen any practical proposal on how to do it though. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Including Snapshot Info with Indexes
On Mon, 2007-10-08 at 09:40 +0100, Heikki Linnakangas wrote: > This idea has been discussed to death many times before. Please search > the archives. Somewhat related to the "visibility in index" thing: would it be possible to have a kind of index-table ? We do have here some tables which have 2-4 fields, and the combination of them forms the primary key of the table. There are usually no other indexes on the table, and the net result is that the PK index duplicates the heap. So it would be cool if the index would be THE heap somehow... The most prominent example of this in our DBs is this table: db> \d table_a Table "public.table_a" Column | Type | Modifiers ---++--- id_1| bigint | not null id_2| bigint | not null Indexes: "pk_table_a" PRIMARY KEY, btree (id_1, id_2) db> select reltuples::bigint, relpages from pg_class where relname='table_a'; reltuples | relpages ---+-- 445286464 | 710090 (1 row) db> select reltuples::bigint, relpages from pg_class where relname='pk_table_a'; reltuples | relpages ---+-- 445291072 | 599848 (1 row) This postgres install is compiled with 32K page size (for the ones who wonder about the page counts). In any case, it is clear that the index basically duplicates the heap... Cheers, Csaba. ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] 8.4 TODO item: make src/port support libpq and ecpg directly
On Thu, Oct 04, 2007 at 05:33:43PM -0400, Tom Lane wrote: > This business with having libpq and ecpg pull in src/port modules > manually is getting unmaintainable. I wonder whether we could persuade > src/port to generate three versions of libpgport.a --- backend, > frontend, and frontend-shlib-ready --- and then just -l the appropriate > one in libpq and ecpg. This'd waste a few cycles building modules that > would never be used, but on the other hand we'd buy some of that back > by not building the same object files three or four times. With so few and small files, I really don't think we need to consider the effects on build time. It's going to be "fast enough" either way. Going with the most maintainable way is much more important. If it actually put the code in the binaries that'd be worse, but the linker should strip that out, no? Or is that different on 'nix thatn win32? Because if it does, why do you need a separate one for frontend-shlib-ready? FWIW, the MSVC port already does this. The only downside I've seen is that unless you define proper dependencies libpq won't build without a manual build of libpgport first. But with proper dependenceis set, that's not an issue. //Magnus ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Including Snapshot Info with Indexes
Gokulakannan Somasundaram wrote: > Currently The index implementation in Postgresql does not store the > Snapshot information in the Index. If we add the snapshot information into > the indexing structure, we will have the following advantages. This idea has been discussed to death many times before. Please search the archives. > a) There can be index only scans like Oracle IMO, the most promising approach to achieving index-only-scans at the moment is the Dead Space Map, as discussed in the 8.3 dev cycle. > b) Unique indexes will become less costly, as older index tuples can be > found out. Doesn't seem like a big benefit, considering that in most cases there won't be any tuples in the index with a duplicate key. A common exception to that is (non-HOT) updating a row. But in that case, the page containing the old tuple is already in cache, so the lookup of the visibility from the heap is cheap. > c) Even the index scans will get faster, since some of the index tuples > won't translate into HeapScans. That's the same as doing an index-only-scan, right? > d) Deletes and Updates will become slightly costly, as they have to update > these indexes. I think you're grossly underestimating the cost of that. For example, on a table with 3 indexes. a delete currently requires one index lookup + one heap lookup. With visibility in the indexes, that would require 3 index lookups + one heap lookup. That's 4 vs. 2 page accesses, not taking into account the non-leaf b-tree pages. The real impact will depend on what's in cache, but the cost can be very high. Also, the full visibility information would need 12 bytes of space per tuple. An index tuple on an int4 key currently takes 12 bytes, so that would double the index size. Storage size has a big impact on performance. More bytes means more I/O, less data fits in cache, and more WAL traffic. There's non-trivial implementation issues involved as well. You'd need a way to reliably find all the index pointers for a given heap tuple (search the archives for "retail vacuum" for the issues involved in that. Broken user defined functions are a problem for example). And you'd need to keep them all locked at the same time to modify them all atomically, which is prone to deadlocks. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
