Re: [HACKERS] Database Kernels and O_DIRECT
On 10/14/03 8:26 PM, "Greg Stark" <[EMAIL PROTECTED]> wrote: > > All the more reason Postgres's view of the world should maybe be represented > there. As it turns out Linus seems unsympathetic to the O_DIRECT approach and > seems more interested in building a better kernel interface to control caching > and i/o scheduling. Something that fits better with postgres's design than > Oracle's. This would certainly help Postgres as currently written, but it won't have the theoretical performance headroom of what Oracle wants. A practical kernel API is too narrow to be fully aware of and exploit database state. And then there is the portability issue... The way you want these kinds of things implemented in an operating system kernel are somewhat orthogonal to how you want them implemented from the perspective of a database kernel. Typical resource use cases for an operating system and a database engine make pretty different assumptions and the best you'll get is a compromise that doesn't optimize either. Making additional optimizations to the OS kernel works great for Postgres (on Linux, at least) because currently very little is optimized in this regard. Basically Linus is doing some design optimization work for us. An improvement, but kind of a mediocre one in the big scheme of things and not terribly portable. If we suddenly wanted to optimize Postgres for performance the way Oracle does, we would be a lot more keen on the O_DIRECT approach. > Actually I think it would be useful for the WAL. As I understand it there's no > point caching the WAL and every write is going to get synced anyways so > there's no point in buffering it either. The sooner the process can find out > it's been synced the better. But I'm not really 100% up on the way the WAL is > used so I could be wrong. Aye, I think you may be correct. > Bah. So Oracle has to live with whatever OS features VMS had 20 years ago. It > has to reimplement whatever I/O scheduling or other strategies it wants. > Rather than being the escape from the "lowest common denominator" it is in > fact precisely the cause of it. You appear to have completely missed the point. The point of the abstraction layer is so they can optimize the hell out of the database for every single platform they support without having to rewrite a bunch of the database every time. The database kernel API is BETTER AND MORE OPTIMAL than the operating system API. It allows them to use whatever memory management scheme, I/O scheme, etc is the best for every single platform. If "the best" happens to going to the native OS service, then that is what they do, but most of the code doesn't need to know this if the abstraction layer is well-designed. Most of the code in a DBMS does not care where memory comes from, how its managed, what the file system actually looks like, or how I/O is done. As long as the behavior is the same from the database kernel API it is writing to, it is all good. What this means from a practical standpoint is that you don't *have* to use SysV IPC on every platform, or POSIX, or mmap, or whatever. You can use whatever that particular platform likes as long it can be mapped into the database kernel API, which tends to be at a high enough level that just about *any* reasonable implementation of an OS API can be mapped into it with quite a bit of optimization. > You describe Postgres as if abstraction is a foreign concept to it. Much > better to have well designed minimal abstractions for each of the resources > needed, rather than trying to turn every OS you meet into the first one you > met. You have a serious misconception of what a database kernel is and looks like. A database kernel doesn't look like the OS kernel that is mapped to it. You write a database kernel API that is idealized for database usage and provides services specifically designed for the needs of a database. It is a high-level API, not a mirror copy of standard OS APIs; if you did that, you wouldn't have any room to do the database kernel implementation. You then build an implementation of the API on the local system using whatever operating system interfaces suit your fancy. The API is simple enough and small enough that this isn't particularly difficult to do in a typical case. And you can write a default kernel that is portable "as is" to most operating systems. There is some abstraction in Postgres and the database is well-written, but it isn't written in a manner that makes it easy to swap out operating system or API models. It is written to be portable at all levels. A database kernel isn't necessarily required to be portable at the very lowest level, but it is vastly more optimizable because you aren't forced into a narrow set of choices for interfacing with the operating system. Operating system APIs are not particularly well-suited for databases, and if you force a database to adhere to operating system APIs directly, you end up with a suboptimal situati
Re: [HACKERS] postgres --help-config
Tom Lane writes: > It'd be better if we could get it right the first time, with the > understanding that the output format is not very negotiable at this > late hour. But as best I can tell, most of the unhappiness is with the > design of the switch set, which is not something I want to defend in > detail. There's a lot there that isn't needed for the RHDB tool as I > understand it, and I think that altering the switches used to get the > output that the tool does need would still be a feasible change from the > tool's point of view. I have some more questions: - When the set of GUC properties (when to set, how to set, etc.) change, what is the upgrade path? Remember that we change those a lot. - Who is going to maintain the descriptions in this very special "GNU trick" format? I can happily agree if we had a short description that is shown in an overview list, and an long description that is shown when the option is opened up in its own window, but I don't agree with with the current format. At least not in the way it was explained to me, maybe I'm misunderstanding. > I would be in favor of simplifying the supported switch set to the > minimum needed by Red Hat's tool (the equivalent of -G -M if I > understood Fernando correctly), and re-adding complexity in future > when and if it's shown to be needed. But we need to make a decision > about this now. Preferably yesterday. I propose we rip out everything except --help-config -m that shows the information in the "machine-readable" tab separated format without headers. If someone can answer the two questions above. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] postgres --help-config
Bruce Momjian writes: > The problem is how that affects Red Hat. What do they do with their > tool? They could use the prototype version of this feature that implemented a separate program (pg_guc) that provided this information. That way they can generate any output they want for as long as they want without affecting anyone. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] pg_dump oid representation
Manuel Sugawara <[EMAIL PROTECTED]> writes: > I did the es translation of pg_dump and while looking at the code I > found that in several places the char * representation of Oid's is > used instead of the oid itself. Any one knows why is done this > way? AFAIR pg_dump invariably stores OIDs as strings rather than numbers. This is arguably inefficient, but no one's felt like expending the effort to change it. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] postgres --help-config
Bruce Momjian <[EMAIL PROTECTED]> writes: > ... Will Red Hat be upset if we > leave it unchanged for 7.4.X and rip this out and redo it in 7.5? It'd be better if we could get it right the first time, with the understanding that the output format is not very negotiable at this late hour. But as best I can tell, most of the unhappiness is with the design of the switch set, which is not something I want to defend in detail. There's a lot there that isn't needed for the RHDB tool as I understand it, and I think that altering the switches used to get the output that the tool does need would still be a feasible change from the tool's point of view. I would be in favor of simplifying the supported switch set to the minimum needed by Red Hat's tool (the equivalent of -G -M if I understood Fernando correctly), and re-adding complexity in future when and if it's shown to be needed. But we need to make a decision about this now. Preferably yesterday. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Database Kernels and O_DIRECT
James Rogers <[EMAIL PROTECTED]> writes: > > > > Someone from Oracle is on there explaining what Oracle's needs are. Perhaps > > someone more knowledgable than myself could explain what would most help > > postgres in this area. > > > There is an important difference between Oracle and Postgres that makes > discussions of this complicated because the assumptions are different. All the more reason Postgres's view of the world should maybe be represented there. As it turns out Linus seems unsympathetic to the O_DIRECT approach and seems more interested in building a better kernel interface to control caching and i/o scheduling. Something that fits better with postgres's design than Oracle's. > the former case, it is very useful and conducive to better performance > to have O_DIRECT and direct control of the I/O in general -- the more, > the better. In the latter case (e.g. Postgres), it is more of a > nuisance and difficult to exploit well. Actually I think it would be useful for the WAL. As I understand it there's no point caching the WAL and every write is going to get synced anyways so there's no point in buffering it either. The sooner the process can find out it's been synced the better. But I'm not really 100% up on the way the WAL is used so I could be wrong. > The point of having a database kernel underneath the DBMS is two-fold. > > First, it improves portability by acting as an operating system > abstraction layer, replacing OS kernel services with its own equivalents Bah. So Oracle has to live with whatever OS features VMS had 20 years ago. It has to reimplement whatever I/O scheduling or other strategies it wants. Rather than being the escape from the "lowest common denominator" it is in fact precisely the cause of it. You describe Postgres as if abstraction is a foreign concept to it. Much better to have well designed minimal abstractions for each of the resources needed, rather than trying to turn every OS you meet into the first one you met. -- greg ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Stupid index idea...
There is no backward link from a heap tuple to it's index entries. So if you have 3 indexes on a table and do an update, you need at least 2 more index lookups just to set that bit, if you somehow manage to remember by what index you found this heap tuple in the first place. On update-heavy tables this will degrade performance quite a bit, wouldn't it? And I don't know offhand into what concurrency problems we run to actually clear that bit. Who is responsible for clearing it and what are the criteria for doing so anyway? Jan Mario Weilguni wrote: Hi, probably it's just a stupid idea, but what do you think of this:currently, most if not all queries with aggregates (count(), sum()) make seq scans when there are no conditions at all. especially count() is a classic question on any postgres list. the reason is - at least that's what I got from this list - is the multi-version system postgres is using. Tom Lane once said that resolving this to use index scans would mean adding 8 bytes to every index tuples, which is no way to go. but what if 1 bit could be added to index items, indication that the heap tuples MAY be modified? in any of my database applications updated tuples in the tables are seldom, because of vacuum. I've a table containing 10 million of tuples, and a count(1) on them takes several minutes. however, the table is quite static and changes seldom, and only a few items change. so an extra bit in the index could indicate: this is a candiate, but the heap must be checked. in most of my applications this would mean that for 95% of all tuples the index can be used for aggregates, only the remaining 5% of all tuples that MIGHT be modified or deleted have to be checked via heap. and even those 5% are often too much, in my case it's typically below 1%. The statistic collector could even tell the optimizer if an index scan is the way to go... Would that work? Best regards, Mario Weilguni ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED]) -- #==# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #== [EMAIL PROTECTED] # ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] postgres --help-config
Tom Lane wrote: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > Peter Eisentraut wrote: > >> I'm beginning to think that we should scrap it and start with a real > >> design for 7.5. I know that's radical, but I don't think we're going to > >> arrive at anything that anyone's going to like by the time we want to > >> release. > > > The problem is how that affects Red Hat. What do they do with their > > tool? > > I believe said tool is within a couple days of code freeze for initial > release, in fact. We could probably still cope with a change in the > spelling of the switches, but redesigning the output format or removing > the functionality altogether would be a very nasty surprise. > > > > Let me be clear on this --- your tools is not part of the PostgreSQL > > > community. We are not required to allow any of this functionality > > > unless the community decides they want it. > > I'm really having a hard time responding to this line of argument > politely. Where were all these complaints when the patch was proposed > and accepted? If there's not time now to redesign the feature to your > liking, it is *NOT* Red Hat's fault, it is *YOURS*. Yanking the rug out > from under someone else's project just because you didn't review the > patch adequately at the time is not my idea of how a community should act. I thought you might have an emotional reaction to this issue. I did not review these changes thoroughly because: o There was no proposal on the switches and their usage. o The commit message didn't mention any switches other than --help-config. o There are no docs to show the new flags. o You were handling it, and I trusted your style, so I didn't see a reason to study it more thoroughly. Let's imagine how this would have worked for an outside project/company: o Project leader comes to us and says they want to make a PostgreSQL admin tool. o They explain their needs and we agree on how to implement it. o We implement the feature as discussed. Would we have agreed to adding all those flags? I don't think so. We would have given them a clean output, and asked them to handle the functionality in their code, which is probably the correct approach. This procedure is in our developer's FAQ: The usual process for source additions is: o Review the TODO list. o Discuss hackers the desirability of the fix/feature. o How should it behave in complex circumstances? o How should it be implemented? o Submit the patch to the patches list. o Answer email questions. o Wait for the patch to be applied. Now, we have Red Hat having you add a patch on July 4 (posted for review June 30), very near feature freeze, but it meets a discussed need (--help-config), so it goes in. I only learned about it when Peter saw the C code handling the new flags and asked questions about it. I do see the patch submitted, with clear illustration of the flags: http://archives.postgresql.org/pgsql-patches/2003-06/msg00420.php I guess I thought those flags were for Red Hat's tool or a separate utility, but it clearly states it is part of the postgres binary, so that was my fault. It looks like this all became visible on September 29: http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&threadm=29402.1065021420%40sss.pgh.pa.us&rnum=2&prev=/groups%3Fq%3Dhelp-config%2Bgroup:comp.databases.postgresql.*%26hl%3Den%26lr%3D%26ie%3DUTF-8%26group%3Dcomp.databases.postgresql.*%26selm%3D29402.1065021420%2540sss.pgh.pa.us%26rnum%3D2 I think I am agreeing with Peter's comment in that email thread: I'm quite unhappy about the --help-config option. It was developed without discussion, it was installed hastily, we don't have any information about that interactive configuration application it's supposed to target, it's not documented, it's full of unfinished business, it certainly doesn't make the code easier to maintain because all the documenation is duplicated, but not one-to-one. At this point, I wouldn't spend a lot of time trying to make sense of it. We can revisit it again in the next release and investigate how we can eliminate the duplication of effort between the documentation and the code. We certainly want to keep Red Hat happy. Will Red Hat be upset if we leave it unchanged for 7.4.X and rip this out and redo it in 7.5? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
[HACKERS] pg_dump oid representation
I did the es translation of pg_dump and while looking at the code I found that in several places the char * representation of Oid's is used instead of the oid itself. Any one knows why is done this way? Regards, Manuel. ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] postgres --help-config
Bruce Momjian <[EMAIL PROTECTED]> writes: > Peter Eisentraut wrote: >> I'm beginning to think that we should scrap it and start with a real >> design for 7.5. I know that's radical, but I don't think we're going to >> arrive at anything that anyone's going to like by the time we want to >> release. > The problem is how that affects Red Hat. What do they do with their > tool? I believe said tool is within a couple days of code freeze for initial release, in fact. We could probably still cope with a change in the spelling of the switches, but redesigning the output format or removing the functionality altogether would be a very nasty surprise. > > Let me be clear on this --- your tools is not part of the PostgreSQL > > community. We are not required to allow any of this functionality > > unless the community decides they want it. I'm really having a hard time responding to this line of argument politely. Where were all these complaints when the patch was proposed and accepted? If there's not time now to redesign the feature to your liking, it is *NOT* Red Hat's fault, it is *YOURS*. Yanking the rug out from under someone else's project just because you didn't review the patch adequately at the time is not my idea of how a community should act. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] postgres --help-config
Peter Eisentraut wrote: > Bruce Momjian writes: > > > Let me be clear on this --- your tools is not part of the PostgreSQL > > community. We are not required to allow any of this functionality > > unless the community decides they want it. The major argument for > > keeping it, in my mind, is to be helpful to Red Hat. > > > > My current idea is to keep --help-config as readable output, add > > --help-config-raw as machine-readable output, document those, and remove > > all the additional flags. > > I'm beginning to think that we should scrap it and start with a real > design for 7.5. I know that's radical, but I don't think we're going to > arrive at anything that anyone's going to like by the time we want to > release. The problem is how that affects Red Hat. What do they do with their tool? I was hoping we could get a minimal setup for them to access those variables so they could move forward. Let me add I know these admin tools are very popular with support companies. I know SRA has an admin tool that can control postgresql.conf variables. My guess is that they did it by hacking their version of PostgreSQL. Looks like they have new URL showing their Win32 and enhanced PostgreSQL products (English auto-translation): http://www.excite.co.jp/world/url/body?wb_url=http%3A%2F%2Fpowergres.sra.co.jp%2F&wb_lp=JAEN&wb_dis=2 -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] postgres --help-config
Bruce Momjian writes: > Let me be clear on this --- your tools is not part of the PostgreSQL > community. We are not required to allow any of this functionality > unless the community decides they want it. The major argument for > keeping it, in my mind, is to be helpful to Red Hat. > > My current idea is to keep --help-config as readable output, add > --help-config-raw as machine-readable output, document those, and remove > all the additional flags. I'm beginning to think that we should scrap it and start with a real design for 7.5. I know that's radical, but I don't think we're going to arrive at anything that anyone's going to like by the time we want to release. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] postgres --help-config
On Tue, 14 Oct 2003, Rod Taylor wrote: > > > I wouldn't want the whole diff on the mail, but a link to the relevant > > > diffs in cvsweb would be most useful (one for each changed file -- not ideal, > > > but much better than nothing). You're not the first one to suggest it ... > > > > I agree, it would be very useful. Marc, would it be possible to set this > > up? > > I volunteer to provide a patch (after 7.4 release) to CVSMODULES to do > this for emails and the $PostgreSQL$ bit. > > It will be based upon the freely available FreeBSD code line. FreeBSD's CVS scripts are the ones I've used too, and they work well. Perhaps it would make sense to have a separate mailing list so people who only want to see commit notices and not full diffs will be happy? Personally I don't find commit notices without diffs very useful -- if forced to choose, I'd rather have diffs without notices. :) But I can understand people not wanting to deal with the huge increase in message sizes if they're on a slow dial-up link or whatever. Jon ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [PERFORM] Sun performance - Major discovery!
Marko Karppinen writes: > I'm not aware of any Darwin-specific "workarounds" in the tree > right now; the only thing close to that is the support for Apple's > two-level namespaces feature. And while you can argue the relative > merits of Apple's approach, the reason for its existence isn't > sloppiness and the support for it that was implemented by Tom > most certainly isn't a workaround. PostgreSQL is only part of the deal; in other projects, people have to fight with different kinds of problems. Let me just point out the broken precompiler, the namespace level thing (which might be a fine feature, but the way it was shoved in was not), using zsh as the default "Bourne" shell, using different file types for loadable modules and linkable shared libraries, standard system paths with spaces in them, and there may be more that I don't remember now. In my experience, the whole system just has been very unpleasant to develop portable software for since the day it appeared. You're not at fault for that, but please understand that, considering all this, the last thing I want to spend time on is improving the user response mechanics for a "don't do that then" problem. > The fact of the matter is that Mac OS X has about ten million active > users, and when one of these people is looking for an RDBMS, he's > gonna go for one that compiles and works great on his system, rather > worrying if his platform is optimal for running PostgreSQL. Supporting > this platform well is absolutely crucial to the overall adoption of pg, > and even if you consider yourself to be above such pedestrian > concerns, many people who have to make the business case for putting > money into PostgreSQL development most definitely think otherwise. Everyone shall be happy if they don't use compiler switches that are known to create broken code. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] postgres --help-config
> > I wouldn't want the whole diff on the mail, but a link to the relevant > > diffs in cvsweb would be most useful (one for each changed file -- not ideal, > > but much better than nothing). You're not the first one to suggest it ... > > I agree, it would be very useful. Marc, would it be possible to set this > up? I volunteer to provide a patch (after 7.4 release) to CVSMODULES to do this for emails and the $PostgreSQL$ bit. It will be based upon the freely available FreeBSD code line. signature.asc Description: This is a digitally signed message part
Re: [HACKERS] postgres --help-config
On Tue, 2003-10-14 at 15:57, Alvaro Herrera Munoz wrote: > On Tue, Oct 14, 2003 at 07:52:55PM +, Jon Jensen wrote: > > Other projects I've worked on have such a list, and each commit message is > > followed by a complete diff (usually with -u for readability) so even > > non-committers can do a code read right then and there while the changes > > are still fresh in the author's mind and can be discussed in detail. > > I wouldn't want the whole diff on the mail, but a link to the relevant > diffs in cvsweb would be most useful (one for each changed file -- not ideal, > but much better than nothing). You're not the first one to suggest it ... I agree, it would be very useful. Marc, would it be possible to set this up? -Neil ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] pg_ctl reload - is it safe?
wscott.marlowe wrote: > > If you use "intr" then this type of thing can happen. Lots of programs assume > > the unix semantics for disk accesses. You can get all kinds of bugs when > > they're violated. > > > > If you use "soft" then the consequences can be much much worse. If your > > fileserver were to reboot you could silently lose disk writes corrupting your > > database. > > What if the WAL was local on disk, and the data was going to nfs storage, > would that be safe, or saferer? :-) Not sure --- we do a sync() on the entire machine before recycling the WAL logs. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Database Kernels and O_DIRECT
On Sun, 2003-10-12 at 15:13, Greg Stark wrote: > There's an interesting thread on linux-kernel right now about O_DIRECT and the > kernel i/o APIs databases need. I noticed a connection between what they were > discussing and the earlier discussions here and the pining for an interface to > avoid having vacuum preempt other disk i/o. > > Someone from Oracle is on there explaining what Oracle's needs are. Perhaps > someone more knowledgable than myself could explain what would most help > postgres in this area. There is an important difference between Oracle and Postgres that makes discussions of this complicated because the assumptions are different. Oracle runs on top of a database kernel, whereas Postgres does not. In the former case, it is very useful and conducive to better performance to have O_DIRECT and direct control of the I/O in general -- the more, the better. In the latter case (e.g. Postgres), it is more of a nuisance and difficult to exploit well. The point of having a database kernel underneath the DBMS is two-fold. First, it improves portability by acting as an operating system abstraction layer, replacing OS kernel services with its own equivalents (which may map to any number of mechanisms underneath). It is the reason Oracle is easily supported on so many operating systems; to port to a new OS, they only have to modify the database kernel, and they probably have a highly portable generic version to start with that they can then optimize for a given platform at their leisure. All the rest of Oracle's code only has to compile against and run on the virtual operating system that is their database kernel. Second, where possible, the database kernel bypasses the OS kernel internally (e.g. O_DIRECT) and implements its own versions of the OS kernel services that are highly-tuned for database purposes. This often has significant performance benefits. While it kind of looks like an OS on top of an OS, well-written database kernels often tend to exist almost parallel the system kernel in certain respects, only using the system kernel where it is convenient or for future capabilities that have been stubbed out in the database kernel. Writing DBMS code to a database kernel almost always produces a more scalable system than writing to portable OS APIs because it eliminates the "lowest common denominator" effect. Having a database kernel isn't really important unless you are a performance junkie or have to address really scalable database systems. Some more advanced DBMS features are easier to implement on a database kernel as a pragmatic concern, because the system model being implemented for is more database friendly. It lets the database take advantage of the more advanced features and optimizations of whatever operating system it is running on without the vast majority of the DBMS code base being aware of these significant differences. I'd like to see Postgres move to a database kernel eventually for a lot of reasons, but it would a relatively significant change. Maybe v8? :-) Cheers, -James Rogers [EMAIL PROTECTED] ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] postgres --help-config
> -Original Message- > From: Jon Jensen [mailto:[EMAIL PROTECTED] > Sent: 14 October 2003 20:53 > To: [EMAIL PROTECTED] > Subject: Re: [HACKERS] postgres --help-config > > Is there a mailing list somewhere that all the CVS commits > get sent to? > Other projects I've worked on have such a list, and each > commit message is followed by a complete diff (usually with > -u for readability) so even non-committers can do a code read > right then and there while the changes are still fresh in the > author's mind and can be discussed in detail. > > If such a list already exists, where do I subscribe? If not, > don't you think it would be helpful in spreading around > fuller knowledge of changes to the codebase? Please see: http://archives.postgresql.org/pgsql-committers/ Regards, Dave. ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] postgres --help-config
On Tue, Oct 14, 2003 at 07:52:55PM +, Jon Jensen wrote: > Is there a mailing list somewhere that all the CVS commits get sent to? Yes, pgsql-committers. > Other projects I've worked on have such a list, and each commit message is > followed by a complete diff (usually with -u for readability) so even > non-committers can do a code read right then and there while the changes > are still fresh in the author's mind and can be discussed in detail. I wouldn't want the whole diff on the mail, but a link to the relevant diffs in cvsweb would be most useful (one for each changed file -- not ideal, but much better than nothing). You're not the first one to suggest it ... -- Alvaro Herrera (<[EMAIL PROTECTED]>) "El destino baraja y nosotros jugamos" (A. Schopenhauer) ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] postgres --help-config
On Tue, 14 Oct 2003, Bruce Momjian wrote: > I knew you were adding --help-config, but I didn't realize the extent of > the "features". The commit message is: > > revision 1.1 > date: 2003/07/04 16:41:21; author: tgl; state: Exp; > Add --help-config facility to dump information about GUC parameters > without needing a running backend. Reorder postgresql.conf.sample > to match new layout of runtime.sgml. This commit re-adds work lost > in Wednesday's crash. > > which I thought was a single option, which we all knew was needed, not > six additional options for output format. Also, with no documention, or > a posted list of the flags you wanted to add (at least I never saw it), > it was easy to miss. Is there a mailing list somewhere that all the CVS commits get sent to? Other projects I've worked on have such a list, and each commit message is followed by a complete diff (usually with -u for readability) so even non-committers can do a code read right then and there while the changes are still fresh in the author's mind and can be discussed in detail. If such a list already exists, where do I subscribe? If not, don't you think it would be helpful in spreading around fuller knowledge of changes to the codebase? Jon ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] pg_ctl reload - is it safe?
On 14 Oct 2003, Greg Stark wrote: > > Michael Brusser <[EMAIL PROTECTED]> writes: > > > > Michael Brusser <[EMAIL PROTECTED]> writes: > > > > 2003-10-10 22:37:05 ERROR: cannot read block 0 of s_noteimportlinks: > > > > Interrupted system call > > > > > > Hmm. I found this hard to believe at first, but indeed my local man > > > pages for read() and write() say they can return EINTR if interrupted > > > by a signal. This may depend on the filesystem in use (are you using > > > NFS?) > > The traditional unix semantics are the read/write my return EINTR if > interrupted -- but that that would only EVER happen for network connections. > The traditional semantics are that it would NEVER happen on disk i/o. BSD > kernels at least, and probably all unix kernels, do an uninterruptible sleep > on disk accesses, hence the dreaded "D" in ps listings. > > > Yes, we use NFS. Many of our customers use it as well. > > Normally NFS guarantees the traditional unix semantics. > Unless you're using either "soft" or "intr" options. > > If you are, well, stop. > > If you use "intr" then this type of thing can happen. Lots of programs assume > the unix semantics for disk accesses. You can get all kinds of bugs when > they're violated. > > If you use "soft" then the consequences can be much much worse. If your > fileserver were to reboot you could silently lose disk writes corrupting your > database. What if the WAL was local on disk, and the data was going to nfs storage, would that be safe, or saferer? :-) ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] pg_ctl reload - is it safe?
Larry Rosenman wrote: -- Start of PGP signed section. > > > --On Tuesday, October 14, 2003 15:31:42 -0400 Bruce Momjian > <[EMAIL PROTECTED]> wrote: > > > Tom Lane wrote: > >> Michael Brusser <[EMAIL PROTECTED]> writes: > >> > Yes, we use NFS. Many of our customers use it as well. > >> > >> You are of course aware that this is not real safe... > > > > Maybe we should throw a "stop using NFS" if we get an EINTR from > > read()/write(), or explain what NFS options they should avoid. > Err, some of us use NetApp filers as NFS servers for our PG data, and I > believe that that negates at least some of the risk. I don't want to see > us (PG) not support something just because it *MAY* be unsafe. True. I was going by the guy who said that only unsafe NFS flags will cause this behavior --- and of course, 1/2 of it was a joke, and the other 1/2 was just an idea thrown out. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] pg_ctl reload - is it safe?
--On Tuesday, October 14, 2003 15:31:42 -0400 Bruce Momjian <[EMAIL PROTECTED]> wrote: Tom Lane wrote: Michael Brusser <[EMAIL PROTECTED]> writes: > Yes, we use NFS. Many of our customers use it as well. You are of course aware that this is not real safe... Maybe we should throw a "stop using NFS" if we get an EINTR from read()/write(), or explain what NFS options they should avoid. Err, some of us use NetApp filers as NFS servers for our PG data, and I believe that that negates at least some of the risk. I don't want to see us (PG) not support something just because it *MAY* be unsafe. LER -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED] US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749 pgp0.pgp Description: PGP signature
Re: [HACKERS] pg_ctl reload - is it safe?
Tom Lane wrote: > Michael Brusser <[EMAIL PROTECTED]> writes: > > Yes, we use NFS. Many of our customers use it as well. > > You are of course aware that this is not real safe... Maybe we should throw a "stop using NFS" if we get an EINTR from read()/write(), or explain what NFS options they should avoid. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] postgres --help-config
Fernando Nasser wrote:
> Bruce,
>
> Before I comment on your suggestions, I would like to mention that many of the
> things below were added on request by the few people who cared to comment on it.
> Aizaz spent most of his time changing here and there to accommodate these
> requests. Anyway, we know we can't satisfy all, but I wish people would be more
> timely when criticizing things. Aizaz is away until May and left thinking
> everything was under control. And we developed a very nice tool that depends on
> this feature confident that we could count on it.
I understand this is not ideal timing. However, open source is
certainly very fluid and it is hard when someone adds something and then
isn't available for later adjustments.
I knew you were adding --help-config, but I didn't realize the extent of
the "features". The commit message is:
revision 1.1
date: 2003/07/04 16:41:21; author: tgl; state: Exp;
Add --help-config facility to dump information about GUC parameters
without needing a running backend. Reorder postgresql.conf.sample
to match new layout of runtime.sgml. This commit re-adds work lost
in Wednesday's crash.
which I thought was a single option, which we all knew was needed, not
six additional options for output format. Also, with no documention, or
a posted list of the flags you wanted to add (at least I never saw it),
it was easy to miss.
Imagine SRA wanted some features added to work better with their admin
tool --- if all the companies did that, PostgreSQL could become a
bloated mess.
> Bruce Momjian wrote:> Tom Lane wrote:
> >
> > First, the default output should have the sorted field first. It
> > appears the default is to sort by context:
> >
> > postgres --help-config
> >
> > That should be the first printed field. However, I see you can sort by
> > Group, so maybe group should always be first --- that seems most
> > logical.
> >
>
> Whatever you do to the human-readable output, please don't change the -M/-m
> format. It makes it harder to write code to parse these things if the fields
> keep changing positions.
>
> Someone is supposed to write an utility to create the sample postgresql.conf
> file from the output as well. I believe this tool would use it without the -G
> so that is why we have the -G option. Also, it was deemed better for the user
> readable output to have it grouped (it is easier to check with the documentation
> as well when it is being reviewed).
We don't normally add things until someone is ready to do the actual
coding --- only then do we know what they will really want.
> > The documentation I see is the command-line help in the C file:
> >
> > printf(gettext("Usage:\n %s --help-config [OPTION]... [NAME]\n\n"), progname);
> > printf(gettext("General Options:\n"));
> > printf(gettext(" NAME output information about parameters matching this
> > name\n"));
> > printf(gettext(" -g GROUP output information about parameters matching this
> > group\n"));
> > printf(gettext(" -llist available parameter groups\n"));
> > printf(gettext(" -hshow this help, then exit\n"));
>
> We don't use the ones above.
>
> > printf(gettext("\nOutput Options:\n"));
> > printf(gettext(" -G do not group by category\n"));
> > printf(gettext(" -m machine-friendly format: tab separated fields\n"));
> > printf(gettext(" -M same as -m, but header with column names is
> > suppressed\n"));
> >
>
> We do use these.
>
> > I don't see any value to most of these parameters, and the "add them in
> > case they might need them" has never been our philosophy, so I don't see
> > why we should add them at this point. Can we just get minimal
> > functionality and see what people ask for --- they might ask for
> > something completely different from what we have spec'ed out here, and
> > if they do, we will have to remove existing options to meet their needs,
> > or maintain a bloat of options. You know I have no problems adding
> > options to make common tasks easier for our users, but we have no usage
> > cases yet to know what they want.
> >
>
> Our GUI tool uses only -G and -M (we may need to resort to -m soon, so we would
> like that one to be preserved as well).
>
> The remaining ones are related to its use as a command line utility and meant to
> produce readable output. I guess some options were indeed requested to limit
> the amount of info printed. BTW, it was a separate utility at that time:
> pg_guc. Aizaz was asked to transform it into a postgres --help-config option.
I think --help-config needs to be a postgres flag, but the output format
flags should be in a separate utility, perhaps not one we ship with
PostgreSQL. I would not have a problem with a separate tool, perhaps in
the src/tools directory, that would parse the output into all sorts of
formats. I am just learning Perl, but I bet I could do it. T
Re: [HACKERS] postgres --help-config
Alvaro Herrera <[EMAIL PROTECTED]> writes: > On Tue, Oct 14, 2003 at 11:34:14AM -0400, Fernando Nasser wrote: >> And we developed a very nice tool that depends on this feature >> confident that we could count on it. > Is this tool going to be released somehow? Certainly. Keep an eye on http://sources.redhat.com/rhdb/ I'm not sure what the release schedule is for that particular tool, but it's soon. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] postgres --help-config
On Tue, Oct 14, 2003 at 11:34:14AM -0400, Fernando Nasser wrote: Unrelated question, > And we developed a very nice tool that depends on this feature > confident that we could count on it. Is this tool going to be released somehow? -- Alvaro Herrera () "I dream about dreams about dreams", sang the nightingale under the pale moon (Sandman) ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] create database that already exists.
Alvaro Herrera wrote: So, how did you know the database already existed? The objects you are seeing may be well part of template1 ... You are right, I'm an idiot...sorry :) Merlin ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] create database that already exists.
On Tue, Oct 14, 2003 at 01:58:24PM -0400, Merlin Moncure wrote: > Here is a verbatim readout of my typing commands into the database > shell. I issued a 'create database' to pg 7.4b2 and it did not fail > despite an already existing database with the same name. Just thought > I'd pass it along, I haven't been able to reproduce it. So, how did you know the database already existed? The objects you are seeing may be well part of template1 ... -- Alvaro Herrera () "El realista sabe lo que quiere; el idealista quiere lo que sabe" (Anónimo) ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] pg_ctl reload - is it safe?
Michael Brusser <[EMAIL PROTECTED]> writes: > > Michael Brusser <[EMAIL PROTECTED]> writes: > > > 2003-10-10 22:37:05 ERROR: cannot read block 0 of s_noteimportlinks: > > > Interrupted system call > > > > Hmm. I found this hard to believe at first, but indeed my local man > > pages for read() and write() say they can return EINTR if interrupted > > by a signal. This may depend on the filesystem in use (are you using > > NFS?) The traditional unix semantics are the read/write my return EINTR if interrupted -- but that that would only EVER happen for network connections. The traditional semantics are that it would NEVER happen on disk i/o. BSD kernels at least, and probably all unix kernels, do an uninterruptible sleep on disk accesses, hence the dreaded "D" in ps listings. > Yes, we use NFS. Many of our customers use it as well. Normally NFS guarantees the traditional unix semantics. Unless you're using either "soft" or "intr" options. If you are, well, stop. If you use "intr" then this type of thing can happen. Lots of programs assume the unix semantics for disk accesses. You can get all kinds of bugs when they're violated. If you use "soft" then the consequences can be much much worse. If your fileserver were to reboot you could silently lose disk writes corrupting your database. -- greg ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Hacking PostgreSQL to work in Mac OS X 10.3 (Panther 7B85)
James Wilson <[EMAIL PROTECTED]> writes: > make -C ecpglib all > gcc -no-cpp-precomp -g -O2 -Wall -Wmissing-prototypes > -Wmissing-declarations -bundle execute.o typename.o descriptor.o > data.o error.o prepare.o memory.o connect.o misc.o > -L../../../../src/port -L/opt/local/lib -L../pgtypeslib -lpgtypes > -L../../../../src/interfaces/libpq -lpq -lm -o libecpg.so.4.0 > ld: Undefined symbols: > _PQfreemem > _PQresultErrorField > _PQsetNoticeReceiver > _last_path_separator I'll bet you have an old libpq in /opt/local/lib. Why is that getting included in the link anyway? regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] [PERFORM] Sun performance - Major discovery!
On 14.10.2003, at 19:52, Tom Lane wrote: This means that relaxing the check would require (a) finding out which of the sub-flags break our code and which don't; (b) finding out how the answer to (a) has varied with gcc release; and (c) finding out how we can test whether a given sub-flag is set --- are there #defines for each of them in gcc 3? Okay, I can see how that makes this unpractical to implement. Thanks. The current error message is "do not put -ffast-math in CFLAGS"; does someone have an idea for a better text that doesn't imply that you actually /have/ --ffast-math in CFLAGS? It'd be good to acknowledge that it can be set implicitly, too. And on the same subject: On 14.10.2003, at 18:13, Peter Eisentraut wrote: That sounds perfectly reasonable to me. Why should we develop elaborate workarounds for compiler flags that are known to create broken code? I also want to point out that I'm getting kind of tired of developing more and more workarounds for sloppy Apple engineering. Peter, you are free to consider your current environment to be the peak of perfection, but that doesn't mean that the only reason for differences between your system and others' is the sloppiness of their engineering. I'm not aware of any Darwin-specific "workarounds" in the tree right now; the only thing close to that is the support for Apple's two-level namespaces feature. And while you can argue the relative merits of Apple's approach, the reason for its existence isn't sloppiness and the support for it that was implemented by Tom most certainly isn't a workaround. The fact of the matter is that Mac OS X has about ten million active users, and when one of these people is looking for an RDBMS, he's gonna go for one that compiles and works great on his system, rather worrying if his platform is optimal for running PostgreSQL. Supporting this platform well is absolutely crucial to the overall adoption of pg, and even if you consider yourself to be above such pedestrian concerns, many people who have to make the business case for putting money into PostgreSQL development most definitely think otherwise. mk ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] create database that already exists.
Here is a verbatim readout of my typing commands into the database shell. I issued a 'create database' to pg 7.4b2 and it did not fail despite an already existing database with the same name. Just thought I'd pass it along, I haven't been able to reproduce it. Just thought I'd pass it along... Merlin [shell log follows] C:\Projects\dbmanager>psql -h linux_server esp Welcome to psql 7.3.2, the PostgreSQL interactive terminal. Type: \copyright for distribution terms \h for help with SQL commands \? for help on internal slash commands \g or terminate with semicolon to execute query \q to quit esp=# create database test; CREATE DATABASE esp=# \c test You are now connected to database test. test=# create table t1(id int, f1 text, f2 text); CREATE TABLE test=# create table t2(id int, f1 text, f2 text); CREATE TABLE test=# \d List of relations Schema |Name| Type | Owner ++--+ public | cuid_sequence | sequence | postgres public | t1 | table| Merlin.Moncure public | t2 | table| Merlin.Moncure public | xfd_field | table| postgres public | xfd_field_key_link | table| postgres public | xfd_file | table| postgres public | xfd_key| table| postgres (7 rows) test=# create database test; ERROR: database "test" already exists test=# \c esp You are now connected to database esp. esp=# create database test; ERROR: database "test" already exists esp=# select version(); version --- PostgreSQL 7.4beta2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.2 (Mandrake Linux 9.0 3.2-1md k) (1 row) esp=# ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] Stupid index idea...
Hi, probably it's just a stupid idea, but what do you think of this:currently, most if not all queries with aggregates (count(), sum()) make seq scans when there are no conditions at all. especially count() is a classic question on any postgres list. the reason is - at least that's what I got from this list - is the multi-version system postgres is using. Tom Lane once said that resolving this to use index scans would mean adding 8 bytes to every index tuples, which is no way to go. but what if 1 bit could be added to index items, indication that the heap tuples MAY be modified? in any of my database applications updated tuples in the tables are seldom, because of vacuum. I've a table containing 10 million of tuples, and a count(1) on them takes several minutes. however, the table is quite static and changes seldom, and only a few items change. so an extra bit in the index could indicate: this is a candiate, but the heap must be checked. in most of my applications this would mean that for 95% of all tuples the index can be used for aggregates, only the remaining 5% of all tuples that MIGHT be modified or deleted have to be checked via heap. and even those 5% are often too much, in my case it's typically below 1%. The statistic collector could even tell the optimizer if an index scan is the way to go... Would that work? Best regards, Mario Weilguni ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Parallel postgresql
Hans-Jürgen Schönig wrote: > Bruce Momjian wrote: > > Martin Rusoff wrote: > > > >>I was just contemplating how to make postgres parallel (for DSS > >>applications)... Has anyone done work on this? It looks to me like there > >>are a couple of obvious places to add parallel operation: > >> > >>Stage 1) I/O , perhaps through MPIO - would improve tablescanning and > >>load/unload operations. One (or more) Postgresql servers would use > >>MPIO/ROMIO to access a parallel file system like PVFS or GPFS(IBM). > >> > >>Stage 2) Parallel Postgres Servers, with the postmaster spawning off the > >>server on a different node (possibly borrowing some code from GNU queue) > >>and doing any buffer twiddling with RPC for that connection, The client > >>connection would still be through the proxy on the postmaster node? (kind > >>of like MOSIX) > > > > > > One idea would be to throw parts of the executor (like a table sort) to > > different machines or to different processors on the same machine, > > perhaps via dblink. You could use threads to send several requests and > > wait for their results. > > > > Threading the entire backend would be hard, but we could thread some > > parts of it by having slave backends doing some of the work in parallel. > > > > This would be nice - especially for huge queries needed in warehouses. > Maybe it could even make sense to do things in par. if there is just one > machine (e.g. computing a function while a sort process is waiting for > I/O or so). > > Which operations can run in par.? What do you think? > I guess implementing something like that means 20 years more work on the > planner ... I would think a very expensive function call could already be done in this way, though you can't do SQL in the function because the visiblity rules and commit/abort handling aren't pass down to the child --- that would severely limit what could be done in a child --- the only logical thing would be some function that calls an external program to send email or something. We could implement something to pass the parent pid down to the child, and the child could use that for visibility rules and maybe commit/abort if we used the parent xid to stamp any rows modified by the child. Of course, anything I/O bound wouldn't benefit from this. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] [DOCS] Request for Info.
Srikanth, HACKERS and DOCS are not the appropriate mailing lists for your request. If you need follow-up help, please join NOVICE or GENERAL. >I am an Oracle DBA having 4 +years of experience and want to > migrate to postgreSQL . > Please kindly Assist me for Documents to convert Oracle Application to > PostgreSQL . > And also send me the links to open source code for postgreSQL. See the articles at: http://techdocs.postgresql.org The source code is what you download when you download PostgreSQL; go to our home page, http://www.postgresql.org -- Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] pg_ctl reload - is it safe?
Michael Brusser <[EMAIL PROTECTED]> writes: > Yes, we use NFS. Many of our customers use it as well. You are of course aware that this is not real safe... regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] [PERFORM] Sun performance - Major discovery!
Marko Karppinen <[EMAIL PROTECTED]> writes: > At least the --fast-math part causes problems, seeing that PostgreSQL > actually checks for the __FAST_MATH__ macro to make sure that it isn't > turned on. There might be other problems with Apple's flags, but I > think that the __FAST_MATH__ check should be altered. Removing the check is not acceptable --- we spent far too much time fighting bug reports that turned out to trace to -ffast-math. See for example http://archives.postgresql.org/pgsql-bugs/2002-09/msg00169.php > As you know, setting --fast-math in GCC is the equivalent of setting > -fno-math-errno, -funsafe-math-optimizations, -fno-trapping-math, > -ffinite-math-only and -fno-signaling-nans. I suspect that -funsafe-math-optimizations is the only one of those that really affects the datetime code, but I would be quite worried about the side-effects of any of them on the float8 arithmetic routines. Also I think the behavior of -ffast-math has changed over time; in the gcc 2.95.3 manual I see none of the above and only the description `-ffast-math' This option allows GCC to violate some ANSI or IEEE rules and/or specifications in the interest of optimizing code for speed. For example, it allows the compiler to assume arguments to the `sqrt' function are non-negative numbers and that no floating-point values are NaNs. Since we certainly do use NaNs, it would be very bad to allow -ffast-math in gcc 2.95. gcc 3.2 has some but not all of the sub-flags you list above, so apparently the behavior changed again as of gcc 3.3. This means that relaxing the check would require (a) finding out which of the sub-flags break our code and which don't; (b) finding out how the answer to (a) has varied with gcc release; and (c) finding out how we can test whether a given sub-flag is set --- are there #defines for each of them in gcc 3? This does not sound real practical to me... > This would allow people to use CFLAGS="-fast" on their G5s, beat some > Xeon speed records, and not worry about esoteric IEEE math standards. In the words of the sage, "I can make this code *arbitrarily* fast ... if it doesn't have to give the right answer." Those "esoteric" standards make the difference between printing 5:00:00 and printing 4:59:60. regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Dreaming About Redesigning SQL
While I definitely agree that the mathematics of the data persistence mechanism is not as important to me as whether it works or not, as a former mathematician, I have done a little study related to the mathematics of non-relational approaches, such as PICK (the one both Wol and I have been know to advocate on behalf of). These models tend to start with language rather than mathematics. So, what started out as my attempt to show such things as the fact that a PICK file is more like a mathematical RELATION than an RDBMS table, I ended up studying the mathematics of language for a short time - one can see that the mathematics of language, which is what we are storing when working with text-based objects, is much more complex than simple relations. By the way, in case you are wondering how PICK files are more like mathematical relations -- they do have a numbered position for each domain (in other words, there is a location for each column within a row as there is a location in a PICK ITEM/RECORD); they do not by default request a constraint on the length of values in a given domain (a quite unnecessary database constraint); and they permit relations as elements within a relation -- there is no mathematical requirement that a relation be in first normal form, for example. I do tire of the thought that a database premised on the relational model is somehow more mathematically accurate than those premised on a language model. PICK, like XML, was used to make it easy to think about storing and retrieving text. If you set aside the need for storing other objects for now and focus on text-based data persistence, it is simply a means to persist propositions. If one were to normalize your sentences before you said them, you might guess that people would have a harder time figuring out what you were saying. Similarly, normalizing data before persisting it tends to make it difficult to retrieve the original propositions, reconstructing language from normalized data. It's time to move on from the relational model -- it was a good academic exercise, but has not proven a very agile means for persisting and retrieving propositions, methinks. smiles. --dawn "Anthony W. Youngman" <[EMAIL PROTECTED]> wrote in message news:<[EMAIL PROTECTED]>... > In article <[EMAIL PROTECTED]>, Lee Fesperman > <[EMAIL PROTECTED]> writes > >If you don't care for mathematical principles, there's always ad-hoc database > >models. > >Check out Pick, OO and XML databases. They're interested in what works and > >ignore > >elegance and mathematical principles. > > Mathematical principles? You mean like Euclidean Geometry and Newtonian > Mechanics? They're perfectly solid, good, mathematically correct. Shame > they don't actually WORK all the time in the real world. > > That's what I feel about relational, too ... > > Cheers, > Wol ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] Dreaming About Redesigning SQL
"Dawn M. Wolthuis" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Thank you, Seun, for asking your question with a bit of logic and not > gut-reaction emotional baggage (and for also asking a question of me > off-list so I could ramble). I'll try to make this more suscinct. With all due respect, Dawn, you are an idiot. That statement has nothing to do with my viscera or with my emotions; I write it with complete dispassion. You simply are an idiot, and Seun simply lacks sufficient education to recognize that fact from what you have written. ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Dreaming About Redesigning SQL
Bob Badour" <[EMAIL PROTECTED]> wrote in message news:<[EMAIL PROTECTED]>... > "Dawn M. Wolthuis" <[EMAIL PROTECTED]> wrote in message > news:[EMAIL PROTECTED] > > Thank you, Seun, for asking your question with a bit of logic and not > > gut-reaction emotional baggage (and for also asking a question of me > > off-list so I could ramble). I'll try to make this more suscinct. > > With all due respect, Dawn, you are an idiot. That statement has nothing to > do with my viscera or with my emotions; I write it with complete dispassion. > You simply are an idiot, and Seun simply lacks sufficient education to > recognize that fact from what you have written. Thanks for the clarification, Bob. What seemed like an emotional outburst from me was actually a logically-derived statement regarding my capacity then, right? If I were you, I might be just a tad bit concerned that I have taught many engineers calculus and you are flying on planes or driving over bridges they designed. Let's hope they learned it well in spite of me, eh? smiles. --dawn Dawn M. Wolthuis ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] 2-phase commit
Why would you spent time on implementing a mechanism whose ultimate benefit is supposed to be increasing reliability and performance, when you already realize that it will have to lock up at the slightest sight of trouble? There are better mechanisms out there that you can use instead. If you want cross-server transactions, what other methods are there that are more reliable? It seems network unreliability is going to be a problem no matter what method you use. I guess we need something like PITR to make this work because otherwise I cannot see a way to get in sync again. Maybe I should call the desired mechanism "Entire cluster back to transaction X recovery". Did anybody hear about PITR recently? How else would you recover from any kind of problem? No matter what you are doing network reliability will be a problem so we have to live with it. Having some "going back to something consistent" is necessary anyway. People might argue now that committed transactions might be lost. If people knew which ones, its ok. 90% of all people will understand that in case of a crash something evil might happen. Hans -- Cybertec Geschwinde u Schoenig Ludo-Hartmannplatz 1/14, A-1160 Vienna, Austria Tel: +43/2952/30706 or +43/660/816 40 77 www.cybertec.at, www.postgresql.at, kernel.cybertec.at ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Dreaming About Redesigning SQL
Thank you, Seun, for asking your question with a bit of logic and not gut-reaction emotional baggage (and for also asking a question of me off-list so I could ramble). I'll try to make this more suscinct. First of all, I have read Codd's 1970 & 1974 ACM papers, as well as his "The Relational Model for Database Management, Version 2" book. I have also read several books by Chris Date and by Fabian Pascal. I held a dialog (that reads more like a monologue) with Pascal and it is reproduced in total if you scroll to the bottom of the http://store.tincat-group.com page and click on the Dick Pick / Ted Codd Blue Brothers parody picture. I had not been reading this news group until lately, but it strikes me that this is a group that might be very entertained by that dialog. I have a master's degree in mathematics and my father is a linguist. I find the language of mathematics and the mathematics of language both fascinating. My experience, however, is that I have run IT project teams working with a variety of databases (and languages) and have never seen any environment that is as agile for develoeprs (both productive from the start and easy to maintain) than the teams I have led that worked with the UniData database. I have worked with SQL as well as both older and newer database languages. So, nope, I'm not trolling. I've been doing some research the past couple of years and I'm convinced that it is time to do something new (and yet old) with data persistence. I favor using Java for a variety of reasons, but am comfortable with other languages as well, and think that using Java both for the software application and for the constraints on the data, rather than encoding constraints in some other language within a database, makes for both a more agile development approach AND, surprisingly enough, tends to make for better data integrity, although a lousy software developer can certainly mess up either environment. Separating the DBA from the software developer has definitely had a negative affect on the speed with which software is developed and maintained, but my experience (and intuition -- I don't, yet, have scientific evidence) tells me that the benefits purported by the approach of having a dba work on some centralized constraints on the persisted data outside of the context of the use of that data have not really come to fruition and/or are not worth the costs of using this approach (more on that some other time). So, while some might classify me as an idiot (men can be so emotional sometimes ;-), I have several graduate classes in logic to my credit and believe that I am approaching this topic quite logically, even if my summaries skip some of the logical steps in the process. I have thought about how to prove my points and since my point is really about agility and quality in application software development and maintenance, a competition to see what tools and techniques and what data persistence approaches win such a competition might be the best proof. The current industry benchmarks for databases tend to be SQL-based and highly political, so let's put different approaches to the test. Thanks for asking your question and not just assuming I'm a nut because I disagree with the current state of the industry on this topic. I'm sure there are gaps in my thinking and I know some of my opinions are based on intuition that arises from my experience, but I do hope to have more proof in the future. I am also very willing to adjust my opinions with convincing arguments and evidence and trust that there are some on this list who work similarly. --dawn [EMAIL PROTECTED] (Seun Osewa) wrote in message news:<[EMAIL PROTECTED]>... > [EMAIL PROTECTED] (Dawn M. Wolthuis) wrote in message news:<[EMAIL PROTECTED]>... > > I would suggest ditching the entire relational model (as both overly > > simplistic in its theory and overly complex in its implementation) and > > start with English (that is one of the other names for the GIRLS > > language). Note that language is also the starting point for putting > > data in XML documents, but it sure doesn't seem to be the starting > > point for XQuery, eh? > > > > --dawn > > Dawn M. Wolthuis > > www.tincat-group.com > > Please explain further. What do you really mean? Its natural for > everyone here to think every word in that post was a troll unless you > explain your views more clearly. You could not have expressed a more > unpopular/unsupportable combination of ideas! Exactly how would we go > about using language as a query tool? Is this AI? What would the > underlying model be knowing how redundant and imprecise language can > be? Tell what we may have missed. > > Seun Osewa ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
[HACKERS] [Linus Torvalds ] Re: statfs() / statvfs() syscall ballsup...
There's an interesting thread on linux-kernel right now about O_DIRECT and the
kernel i/o APIs databases need. I noticed a connection between what they were
discussing and the earlier discussions here and the pining for an interface to
avoid having vacuum preempt other disk i/o.
Someone from Oracle is on there explaining what Oracle's needs are. Perhaps
someone more knowledgable than myself could explain what would most help
postgres in this area.
There was another thread I commented on that touched on another postgres
wishlist item. A way to sync IDE disks reliably without disabling write
caching entirely. There was some inkling that newer drives might provide for
such a possibility. Perhaps that too could be worth advocating for on
postgres's behalf.
--- Begin Message ---
On 12 Oct 2003, Greg Stark wrote:
>
> There are other reasons databases want to control their own cache. The
> application knows more about the usage and the future usage of the data than
> the kernel does.
But this again is not an argument for not using the page cache - it's only
an argument for _telling_ the kernel about its use.
> However on busy servers whenever it's run it causes lots of pain because the
> kernel flushes all the cached data in favour of the data this job touches.
Yes. But this is actually pretty easy to avoid in-kernel, since all of the
LRU logic is pretty localized.
It could be done on a per-process thing ("this process should not pollute
the active list") or on a per-fd thing ("accesses through this particular
open are not to pollute the active list").
>And
> worse, there's no way to indicate that the i/o it's doing is lower priority,
> so i/o bound servers get hit dramatically.
IO priorities are pretty much worthless. It doesn't _matter_ if other
processes get preferred treatment - what is costly is the latency cost of
seeking. What you want is not priorities, but batching.
Linus
--- End Message ---
--
greg
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
Re: [HACKERS] Hacking PostgreSQL to work in Mac OS X 10.3 (Panther 7B85)
Hi Marko, I've done a checkout from CVS and performed a build under OS X Panther 7B85 with the non-hacked header files. The problem with the param.h header file seems to be fixed, however the build is failing with undefined symbols in libs: make -C ecpglib all gcc -no-cpp-precomp -g -O2 -Wall -Wmissing-prototypes -Wmissing-declarations -bundle execute.o typename.o descriptor.o data.o error.o prepare.o memory.o connect.o misc.o -L../../../../src/port -L/opt/local/lib -L../pgtypeslib -lpgtypes -L../../../../src/interfaces/libpq -lpq -lm -o libecpg.so.4.0 ld: Undefined symbols: _PQfreemem _PQresultErrorField _PQsetNoticeReceiver _last_path_separator I noticed it's not linking with the "-undefined suppress" flag which I've found it required when building 'bundles' for darwin. Cheers, On 12/10/2003, at 3:28 AM, Marko Karppinen wrote: James, we've spent some time lately with Mac OS X related changes to the code base. Please try this again with a fresh CVS copy of PostgreSQL and let us know if there are any further changes you think we'd need to work properly with Panther. Thanks, mk On 11.10.2003, at 04:32, James Wilson wrote: After toying with a few builds of the up coming OS X 10.3 (Panther) release, I've written a doc on how to get postgresql to build using Panther build 7B85 and Xcode build 7B85 (which are rumored to be the gold master and the builds that will be shipping later this month). --- James Wilson CCIE #6662 LithiumCorp Pty Ltd mobile: 0422 22 3742 email: [EMAIL PROTECTED] ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] Dreaming About Redesigning SQL
[EMAIL PROTECTED] (Dawn M. Wolthuis) wrote in message news:<[EMAIL PROTECTED]>... > Bob Badour" <[EMAIL PROTECTED]> wrote in message news:<[EMAIL PROTECTED]>... > > "Dawn M. Wolthuis" <[EMAIL PROTECTED]> wrote in message > > news:[EMAIL PROTECTED] > > > Thank you, Seun, for asking your question with a bit of logic and not > > > gut-reaction emotional baggage (and for also asking a question of me > > > off-list so I could ramble). I'll try to make this more suscinct. > > > > With all due respect, Dawn, you are an idiot. That statement has nothing to > > do with my viscera or with my emotions; I write it with complete dispassion. > > You simply are an idiot, and Seun simply lacks sufficient education to > > recognize that fact from what you have written. > > Thanks for the clarification, Bob. What seemed like an emotional > outburst from me was actually a logically-derived statement regarding > my capacity then, right? If I were you, I might be just a tad bit > concerned that I have taught many engineers calculus and you are > flying on planes or driving over bridges they designed. Let's hope > they learned it well in spite of me, eh? smiles. > --dawn > > Dawn M. Wolthuis Dawn: I'm sure you're not an idiot. Bob: Do pigs really like people to wrestle in mud with them? ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Dreaming About Redesigning SQL
"Mike Preece" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > [EMAIL PROTECTED] (Dawn M. Wolthuis) wrote in message news:<[EMAIL PROTECTED]>... > > Bob Badour" <[EMAIL PROTECTED]> wrote in message news:<[EMAIL PROTECTED]>... > > > "Dawn M. Wolthuis" <[EMAIL PROTECTED]> wrote in message > > > news:[EMAIL PROTECTED] > > > > Thank you, Seun, for asking your question with a bit of logic and not > > > > gut-reaction emotional baggage (and for also asking a question of me > > > > off-list so I could ramble). I'll try to make this more suscinct. > > > > > > With all due respect, Dawn, you are an idiot. That statement has nothing to > > > do with my viscera or with my emotions; I write it with complete dispassion. > > > You simply are an idiot, and Seun simply lacks sufficient education to > > > recognize that fact from what you have written. > > > > Thanks for the clarification, Bob. What seemed like an emotional > > outburst from me was actually a logically-derived statement regarding > > my capacity then, right? If I were you, I might be just a tad bit > > concerned that I have taught many engineers calculus and you are > > flying on planes or driving over bridges they designed. Let's hope > > they learned it well in spite of me, eh? smiles. > > --dawn > > > > Dawn M. Wolthuis > > Dawn: I'm sure you're not an idiot. Mike, you have expressed certainty contradicting the patently obvious before. Your opinion is only a useful gauge insofar as you are wrong so much more frequently than you are correct. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 2-phase commit
On Thu, 9 Oct 2003, Bruce Momjian wrote: > Agreed. Let's get it into 7.5 and see it in action. If we need to > adjust it, we can, but right now, we need something for distributed > transactions, and this seems like the logical direction. I've started working on two-phase commits last week, and the very basic stuff is now working. Still a lot of bugs though. I posted the stuff I've put together to patches-list. I'd appreciate any comments. - Heikki ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] 2-phase commit
I'm tired of this kind of "2PC is too slow" arguments. I think Satoshi, the only guy who made a trial implementation of 2PC for PostgreSQL, has already showed that 2PC is not that slow. Where does Satoshi's implementation sit right now? Will it patch to v7.4? Can it provide us with a base to work from, or is it complete? It is not ready yet. You can find it at ... http://snaga.org/pgsql/ It is based on 7.3 * the 2-phase commit protocol (precommit and commit) * the multi-master replication using 2PC * distributed transaction (distributed query) current work * restarting (from 2nd phase) when the session is disconnected in 2nd phase (XLOG stuffs) * XA compliance future work * hot failover and recovery in PostgreSQL cluster * data partitioning on different servers I have compiled it a while ago. Seems to be pretty nice :). Hans -- Cybertec Geschwinde u Schoenig Ludo-Hartmannplatz 1/14, A-1160 Vienna, Austria Tel: +43/2952/30706 or +43/660/816 40 77 www.cybertec.at, www.postgresql.at, kernel.cybertec.at ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] [JDBC] Using JDBC and libpq in the same process
Nicola, No, the driver does not use libpq it uses it's own socket code. Dave On Thu, 2003-10-09 at 07:30, Nicola Pero wrote: > Hi, > > I have to work on some software which is using JDBC and libpq (via JNI) in > the same process to connect to the same Postgres database. The software > is heavily multithreaded. > > When running under heavy loads and accessing the database under JDBC and > libpq at the same time, we get obscure JVM crashes. > > Before I even start debugging this, I wanted to ask - is there any known > fundamental reason why JDBC and libpq should *not* be used at the same > time in the same process ? > > > ---(end of broadcast)--- > TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] > -- Dave Cramer <[EMAIL PROTECTED]> ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Dreaming About Redesigning SQL
[EMAIL PROTECTED] (Dawn M. Wolthuis) wrote in message news:<[EMAIL PROTECTED]>... > I would suggest ditching the entire relational model (as both overly > simplistic in its theory and overly complex in its implementation) and > start with English (that is one of the other names for the GIRLS > language). Note that language is also the starting point for putting > data in XML documents, but it sure doesn't seem to be the starting > point for XQuery, eh? > > --dawn > Dawn M. Wolthuis > www.tincat-group.com Please explain further. What do you really mean? Its natural for everyone here to think every word in that post was a troll unless you explain your views more clearly. You could not have expressed a more unpopular/unsupportable combination of ideas! Exactly how would we go about using language as a query tool? Is this AI? What would the underlying model be knowing how redundant and imprecise language can be? Tell what we may have missed. Seun Osewa ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] Dreaming About Redesigning SQL
Good question. Although I would want to move away from relational databases too, if there is an RDBMS and one wants to query it, what would I aim for? If you look at XQuery, you will see an example of what I would definitely NOT aim for. Although the user of such a language might very well be a technical person, instead of starting with mathematics (relational calculus, relational algebra) I would suggest starting with language. The mathematics of language is more complex than the mathematics of relations, particularly simple relations (such as 1NF tables). If you look at the history of data persistence prior to Codd's 1970 ACM paper, you will see several attempts at this. One I have studied of late is GIRLS (Generalized Information Retrieval Language and System), specified by Don Nelson and implemented by several folks with the most famous being Dick Pick. This GIRLS language was specified a full 40 years ago and lives today in many IT shops under a variety of about 10 different names, including IBM's UniQuery and Retrieve (for UniData and Universe respectively). This language is flawed, as are all, but so very close to what I would think would be a good approach. It was written at TRW in order to make it so that the military in Viet Nam could query their data without technical folks in the field. It went into production in 1969 with the US Army. Prior to the end of the cold war, it was used by the CIA to track (the associated database) and query about Russion spies in the US. I would suggest ditching the entire relational model (as both overly simplistic in its theory and overly complex in its implementation) and start with English (that is one of the other names for the GIRLS language). Note that language is also the starting point for putting data in XML documents, but it sure doesn't seem to be the starting point for XQuery, eh? --dawn Dawn M. Wolthuis www.tincat-group.com ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Dreaming About Redesigning SQL
With all due respect, Dawn, you are an idiot. "Dawn M. Wolthuis" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Good question. Although I would want to move away from relational > databases too, if there is an RDBMS and one wants to query it, what > would I aim for? If you look at XQuery, you will see an example of > what I would definitely NOT aim for. Although the user of such a > language might very well be a technical person, instead of starting > with mathematics (relational calculus, relational algebra) I would > suggest starting with language. The mathematics of language is more > complex than the mathematics of relations, particularly simple > relations (such as 1NF tables). > > If you look at the history of data persistence prior to Codd's 1970 > ACM paper, you will see several attempts at this. One I have studied > of late is GIRLS (Generalized Information Retrieval Language and > System), specified by Don Nelson and implemented by several folks with > the most famous being Dick Pick. This GIRLS language was specified a > full 40 years ago and lives today in many IT shops under a variety of > about 10 different names, including IBM's UniQuery and Retrieve (for > UniData and Universe respectively). This language is flawed, as are > all, but so very close to what I would think would be a good approach. > It was written at TRW in order to make it so that the military in > Viet Nam could query their data without technical folks in the field. > It went into production in 1969 with the US Army. Prior to the end of > the cold war, it was used by the CIA to track (the associated > database) and query about Russion spies in the US. > > I would suggest ditching the entire relational model (as both overly > simplistic in its theory and overly complex in its implementation) and > start with English (that is one of the other names for the GIRLS > language). Note that language is also the starting point for putting > data in XML documents, but it sure doesn't seem to be the starting > point for XQuery, eh? > > --dawn > Dawn M. Wolthuis > www.tincat-group.com ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Dreaming About Redesigning SQL
[EMAIL PROTECTED] (Seun Osewa) wrote in message news:<[EMAIL PROTECTED]>... > I have tried, twice, to download the evaluation version of the alphora > product for testing and it doesn't work. Guess there would be a lot > to learn from playing with it; the product is more than a RDBMS Aw, that's unfortunate. It took me a while to get working. It is infact an integrated application development environment where you can define a great part of your application in a declarative fashion. regards, Lauri Pietarinen ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] Using JDBC and libpq in the same process
Hi, I have to work on some software which is using JDBC and libpq (via JNI) in the same process to connect to the same Postgres database. The software is heavily multithreaded. When running under heavy loads and accessing the database under JDBC and libpq at the same time, we get obscure JVM crashes. Before I even start debugging this, I wanted to ask - is there any known fundamental reason why JDBC and libpq should *not* be used at the same time in the same process ? ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Dreaming About Redesigning SQL
>> The mathematics of language is more complex than the mathematics of relations, particularly simple relations (such as 1NF tables). << Are you sure, you know what you are talking about? >> I would suggest ditching the entire relational model (as both overly simplistic in its theory and overly complex in its implementation.. << Incredible! How about reading some books on the subject? -- -- Anith ( Please reply to newsgroups only ) ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] [PORTS] [COMMITTERS] pgsql-server/src/template bsdi freebsd
At 10:45 AM -0400 10/9/03, Bruce Momjian wrote: Neil Conway wrote: On Thu, 2003-10-09 at 09:35, Bruce Momjian wrote: > I only put back what was already there --- not sure why others don't use > it. You want it enabled on Linux? Well, why do we have it enabled at all? If it's to speed compilation, we may as well enable it on other platforms where -pipe works, of which Linux is one. My gcc 2.95.3 manual says: -pipe Use pipes rather than temporary files for communi- cation between the various stages of compilation. This fails to work on some systems where the assem- bler cannot read from a pipe; but the GNU assembler has no trouble. so it looks like we can't use it on all platforms without testing. I will enable it for linux. Do people want to test other platforms? It should work on any platform that uses the GNU tools, so that means *BSD is in the same boat as Linux. Does it really speed compilation though? I saw somewhere that it didn't make much difference and might even hurt sometimes. -- The opinions expressed in this message are mine, not those of Caltech, JPL, NASA, or the US Government. [EMAIL PROTECTED], or [EMAIL PROTECTED] ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Parallel postgresql
Bruce Momjian wrote: Martin Rusoff wrote: I was just contemplating how to make postgres parallel (for DSS applications)... Has anyone done work on this? It looks to me like there are a couple of obvious places to add parallel operation: Stage 1) I/O , perhaps through MPIO - would improve tablescanning and load/unload operations. One (or more) Postgresql servers would use MPIO/ROMIO to access a parallel file system like PVFS or GPFS(IBM). Stage 2) Parallel Postgres Servers, with the postmaster spawning off the server on a different node (possibly borrowing some code from GNU queue) and doing any buffer twiddling with RPC for that connection, The client connection would still be through the proxy on the postmaster node? (kind of like MOSIX) One idea would be to throw parts of the executor (like a table sort) to different machines or to different processors on the same machine, perhaps via dblink. You could use threads to send several requests and wait for their results. Threading the entire backend would be hard, but we could thread some parts of it by having slave backends doing some of the work in parallel. This would be nice - especially for huge queries needed in warehouses. Maybe it could even make sense to do things in par. if there is just one machine (e.g. computing a function while a sort process is waiting for I/O or so). Which operations can run in par.? What do you think? I guess implementing something like that means 20 years more work on the planner ... By the way: NCR has a quite nice solution for problems like that. Teradata has been designed to run everything on multiple nodes (they call it AMPs). Teradata has been designed for A LOT OF data and reporting purposes. There are just three problems: - not Open Source - ~$70k / node - runs on Windows and NCR's UNIX implementation. Is anybody familiar with Teradata? Hans -- Cybertec Geschwinde u Schoenig Ludo-Hartmannplatz 1/14, A-1160 Vienna, Austria Tel: +43/2952/30706 or +43/660/816 40 77 www.cybertec.at, www.postgresql.at, kernel.cybertec.at ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Separate shared_buffer management process
This would be a good idea I think. DB2 has a page-cleaner background process that periodically writes out dirty pages to disk. Reduces checkpoint I/O. I don't see much point in serializing all bufferpool I/O through a separate dedicated backend. Informix uses something like this. -- Pip-pip Sailesh http://www.cs.berkeley.edu/~sailesh Ph: (510) 642-8072 - Original Message - From: Bruce Momjian <[EMAIL PROTECTED]> Date: Wednesday, October 8, 2003 12:33 pm Subject: Re: [HACKERS] Separate shared_buffer management process > > Added to TODO: > > * Use background process to write dirty shared buffers to disk > > --- > > > Tom Lane wrote: > > Bruce Momjian <[EMAIL PROTECTED]> writes: > > > Would it be a good idea to have a separate shared buffer > process to > > > manage the cache? Could such a process take workload off of > the main > > > backend and improve their performance? > > > > > Just an idea? > > > > I can't recall if this has been discussed on the list, but I > know I've > > thought about the idea of a background "buffer writer" process that > > would simply cycle through the buffer cache and write out dirty > buffers> in some low-priority fashion. > > > > The idea is this would reduce the I/O crunch at checkpoint > times, as > > well as reducing the odds that any foreground backend process > would have > > to block waiting for I/O before it could recycle a buffer slot > to read > > in a page it needs. (Perhaps the background writer could be > tuned to > > preferentially write dirty buffers that are near the tail of the LRU > > queue, and thus are likely to get recycled soon.) > > > > In the WAL world, you cannot "write a dirty buffer" until you have > > written *and synced* the WAL log as least as far as the LSN of the > > buffer you want to write. So a background buffer writer would have > > to write WAL buffers as well, and in that context it could find > itself> blocking foreground processes. I'm not sure what this > does to the > > notion of "background I/O". Maybe only buffers whose changes are > > already synced in WAL should be eligible for background write. > > It needs some thought anyway. > > > > regards, tom lane > > > > -- > Bruce Momjian| http://candle.pha.pa.us > [EMAIL PROTECTED] | (610) 359-1001 > + If your life is a hard drive, | 13 Roberts Road > + Christ can be your backup.| Newtown Square, > Pennsylvania 19073 > > ---(end of broadcast)-- > - > TIP 8: explain analyze is your friend > ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Dreaming About Redesigning SQL
[EMAIL PROTECTED] ("Dann Corbit") wrote in message news:<[EMAIL PROTECTED]>...
> > -Original Message-
> > From: Seun Osewa [mailto:[EMAIL PROTECTED]
> > Sent: Friday, October 03, 2003 11:52 AM
> > To: [EMAIL PROTECTED]
> > Subject: [HACKERS] Dreaming About Redesigning SQL
> >
> >
> > Hi,
> >
> > This is for relational database theory experts on one hand
> > and imlementers of real-world alications on the other hand.
> > If there was a chance to start again and design SQL afresh,
> > for best cleaness/power/performance what changes would you
> > make? What would your query language (and the underlying
> > database concept) look like?
> >
> > Seun Osewa
> > PS: I should want to post my ideas too for review but more
> > experienced/qualified people should come first
>
> I imagine you have read the 3rd database manifesto by Codd. I think
> he's gone off the deep end a bit.
Dann, you are showing your ignorance. While Dr. Codd recently died, if
you think he wrote a third database manifesto, you have definitely
gone off the deep end yourself.
> You don't just throw away a trillion
> dollars worth of effort and tools to make things mathematically
> orthogonal.
Again, you are showing your ignorance. Nobody has ever suggested
anything even remotely resembling the above.
> However, on some things he is clearly right. For instance, null values
> are evil.
Dr. Codd believed we need two NULLs. You ascribe correctness to the
one thing I think he clearly got wrong.
> Programmers understand it
That's an absurd assertion.
> Therefore, his idea of using default values instead and never using null
> is a good one.
That is not his idea.
> If SQL vendors would follow the ANSI/ISO standard to the letter, and
> implement the latest iteration, that would solve all of the problems
> that SQL tool users have to face.
Upon what do you base this ridiculous opinion?
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] Request for Info.
Hi, I am an Oracle DBA having 4 +years of experience and want to migrate to postgreSQL . Please kindly Assist me for Documents to convert Oracle Application to PostgreSQL . And also send me the links to open source code for postgreSQL. Thanks& Regards, Srikanth Pentakota
Re: [HACKERS] pg_ctl reload - is it safe?
Yes, we use NFS. Many of our customers use it as well. Mike. > -Original Message- > From: Tom Lane [mailto:[EMAIL PROTECTED] ... ... > Michael Brusser <[EMAIL PROTECTED]> writes: > > 2003-10-10 22:37:05 ERROR: cannot read block 0 of s_noteimportlinks: > > Interrupted system call > > Hmm. I found this hard to believe at first, but indeed my local man > pages for read() and write() say they can return EINTR if interrupted > by a signal. This may depend on the filesystem in use (are you using > NFS?) > > We had probably better fix fd.c to retry on EINTR. > > regards, tom lane > ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] feature request
I wonder if this is feasible to enhance create trigger so I could say 'create or replace' Thanks, Mike. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] pg_ctl reload - is it safe?
Michael Brusser <[EMAIL PROTECTED]> writes: > 2003-10-10 22:37:05 ERROR: cannot read block 0 of s_noteimportlinks: > Interrupted system call Hmm. I found this hard to believe at first, but indeed my local man pages for read() and write() say they can return EINTR if interrupted by a signal. This may depend on the filesystem in use (are you using NFS?) We had probably better fix fd.c to retry on EINTR. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] postgres --help-config
Bruce,
Before I comment on your suggestions, I would like to mention that many of the
things below were added on request by the few people who cared to comment on it.
Aizaz spent most of his time changing here and there to accommodate these
requests. Anyway, we know we can't satisfy all, but I wish people would be more
timely when criticizing things. Aizaz is away until May and left thinking
everything was under control. And we developed a very nice tool that depends on
this feature confident that we could count on it.
Bruce Momjian wrote:> Tom Lane wrote:
First, the default output should have the sorted field first. It
appears the default is to sort by context:
postgres --help-config
That should be the first printed field. However, I see you can sort by
Group, so maybe group should always be first --- that seems most
logical.
Whatever you do to the human-readable output, please don't change the -M/-m
format. It makes it harder to write code to parse these things if the fields
keep changing positions.
Someone is supposed to write an utility to create the sample postgresql.conf
file from the output as well. I believe this tool would use it without the -G
so that is why we have the -G option. Also, it was deemed better for the user
readable output to have it grouped (it is easier to check with the documentation
as well when it is being reviewed).
The documentation I see is the command-line help in the C file:
printf(gettext("Usage:\n %s --help-config [OPTION]... [NAME]\n\n"), progname);
printf(gettext("General Options:\n"));
printf(gettext(" NAME output information about parameters matching this
name\n"));
printf(gettext(" -g GROUP output information about parameters matching this
group\n"));
printf(gettext(" -llist available parameter groups\n"));
printf(gettext(" -hshow this help, then exit\n"));
We don't use the ones above.
printf(gettext("\nOutput Options:\n"));
printf(gettext(" -G do not group by category\n"));
printf(gettext(" -m machine-friendly format: tab separated fields\n"));
printf(gettext(" -M same as -m, but header with column names is suppressed\n"));
We do use these.
I don't see any value to most of these parameters, and the "add them in
case they might need them" has never been our philosophy, so I don't see
why we should add them at this point. Can we just get minimal
functionality and see what people ask for --- they might ask for
something completely different from what we have spec'ed out here, and
if they do, we will have to remove existing options to meet their needs,
or maintain a bloat of options. You know I have no problems adding
options to make common tasks easier for our users, but we have no usage
cases yet to know what they want.
Our GUI tool uses only -G and -M (we may need to resort to -m soon, so we would
like that one to be preserved as well).
The remaining ones are related to its use as a command line utility and meant to
produce readable output. I guess some options were indeed requested to limit
the amount of info printed. BTW, it was a separate utility at that time:
pg_guc. Aizaz was asked to transform it into a postgres --help-config option.
What would make sense is to always output sorted by Group/Name, and see
how that works for folks.
Please don't do it for -M/-m. It may unnecessarily increase the response time
of GUI tools and add to the server load. The client can always sort or store it
the way it wants.
I also didn't think the dashed lines were
needed --- they seemed to clutter the output:
This seems to be a question of personal preference. But as I said, we only use
the -M/-m output in our tool.
Name: preload_libraries
Context: postmaster
Group: Resource Usage / Kernel Resources
Type: string
Default value:
Description: shared libraries to preload into server
Name: fsync
Context: sighup
Group: Write-Ahead Log / Settings
Type: Boolean
Default value: true
Description: force synchronization of updates to disk
The server will use the fsync() system call in several places to make sure
that updates are physically written to disk. This insures that a database cluster will
recover to a consistent state after an operating system or hardware crash.
vs.
Name: preload_libraries
Context: postmaster
Group: Resource Usage / Kernel Resources
Type: string
Default value:
Description: shared libraries to preload into server
Name: fsync
Conte
Re: [HACKERS] comments on casts
Christopher Kings-Lynne writes: > Also, with the pg_largeobject table, there seems to be no 'owner' > concept on lobs at all. So is there no problem with any random gumby > commenting on anyone else's large object? Not any worse that any random gumby reading or writing anyone else's large object. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Heading to final release
Bruce Momjian writes: > What to do about exposing the list of possible SQLSTATE error codes I say we put the list in the documentatio and that's it. Exposing the list in C header files wouldn't really be an ultimate solution, because not everyone uses C. > Freeze message strings That is old news. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [PERFORM] Sun performance - Major discovery!
Marko Karppinen writes: > GCC sets __FAST_MATH__ even if you counter a -ffast-math with the > negating flags above. This means that it is not currently possible to > use the -fast flag when compiling PostgreSQL at all. Instead, you have > to go through all the flags Apple is setting and only pass on those > that don't break pg. That sounds perfectly reasonable to me. Why should we develop elaborate workarounds for compiler flags that are known to create broken code? I also want to point out that I'm getting kind of tired of developing more and more workarounds for sloppy Apple engineering. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] CREATE USER bug
Neil Conway <[EMAIL PROTECTED]> writes: > On Mon, 2003-10-06 at 16:21, Joshua D. Drake wrote: >> postgres=# create user with encrypted password '98wq7912a'; >> CREATE USER >> postgres=# create user with encrypted password '98wq7912a'; >> ERROR: CREATE USER: user name "with" already exists > So, what are we doing about this? If we're considering it a bug, one way > to fix it is to move WITH from unreserved_keywords to reserved_keywords. > Any other suggestions? It's not a bug. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Heading to final release
On Mon, Oct 13, 2003 at 11:51:52PM -0400, Jan Wieck wrote: > You can build more secure systems as long as you want, evolution will > develop the better idiot. As long as you create safer cars with more Sure, but I think all Rod is asking for is something like the ability to add the -w switch to perl in order to see things that he might have overlooked. I can't see any principled objection to that, especially if, as Bruce suggested, it defaults to off. (Of course, if the switch isn't ready, I dunno whether it's worth waiting for. Sounds like a feature to me.) A -- Andrew Sullivan 204-4141 Yonge Street Afilias CanadaToronto, Ontario Canada <[EMAIL PROTECTED]> M2P 2A8 +1 416 646 3304 x110 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] comments on casts
Christopher Kings-Lynne <[EMAIL PROTECTED]> writes: > Also, with the pg_largeobject table, there seems to be no 'owner' > concept on lobs at all. So is there no problem with any random gumby > commenting on anyone else's large object? We don't have a lot of choice. I suppose ideally LOs should have owners, but I don't really want to put that much effort (and breakage of existing apps, no doubt) into them. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Heading to final release
> >> Some dumb-user/fat-finger/ooops protection is surely welcome, but there > >> is a limit. A system console has to be behind a locked door instead of > >> the single-user boot being root-password protected. As soon as people > > > > Unfortunately, as more and more companies start to outsource their > > server administration these are the people who will be interacting with > > the database more in this role -- in fact, for most it is the only time > > they'll ever be on the database box. > You can build more secure systems as long as you want, evolution will > develop the better idiot. As long as you create safer cars with more Consider it like shipping. You can assume that UPS, FedEx or whomever will be nice and gentle to the package marked 'Fragile' and has a 'This Side Up' sticker OR you can double box it and use plenty of tightly packed peanuts. One of those 2 options is bound to have busted up contents by the time it reaches the other side nearly every time -- but they could still run a fork lift through the thing. Yes, if you're going to drive the package to the destination yourself, then all of the extra packaging would just get in the way -- but shipping has been outsourced by your company to save funds. If the product breaks when it gets to the client, it isn't going to be the shipping companies fault. It's the same reason PostgreSQL will not load when the blocksize has changed and why RESTRICT / CASCADE options exist for inter-object enforcement. Anyway, add the option if you like BUT can we start protecting these things with something more than superuser access? You require superuser to do daily maintenance tasks with PostgreSQL but for the most part these are exactly the wrong people to be making decisions about whether it is safe to do action X or Y at the time. Anyway, one of the local Nuclear power plants has safety courses. At the safest plant in Canada the operators have an accuracy rate of close to 99.9%. That is, they make the correct choice or complete the correct action for 99.9% of the choices in their day. This means at that plant there are 50 potentially fatal decision made every month. I simply want to remove the junior electricians ability to pick the wrong panel at the datacentre by ensuring someone else has given them the key. I want to remove my 'super users' ability to make a bad decision (even though they're 99.9% accurate in their decision making) by granting or revoking their ability to do so. signature.asc Description: This is a digitally signed message part
Re: [HACKERS] 2-phase commit
Jan Wieck wrote: > 2PC is not too slow in normal operations when everything is purring > like little kittens and you're just wasting your excess bandwidth on > it. The point is that it behaves horrible and like a dirty backstreet > cat at the time when things go wrong ... basically it's a neat thing > to have, but from the second you need it it becomes useless. I can't see anyone being forced to use it once it maybe/is supported. Like many tools, "ouch!" is a good reaction when used untrained/incorrectly. Peter ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
