Re: [HACKERS] Postgresql and multithreading
I have updated the developers FAQ item 1.9 to address this: http://developer.postgresql.org/readtext.php?src/FAQ/FAQ_DEV.html+Developers-FAQ --- Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 01:51:28AM -0400, Bruce Momjian wrote: > > > > Let me add one more thing on this "thread". This is one email in a > > long list of "Oh, gee, you aren't using that wizz-bang new > > sync/thread/aio/raid/raw feature" discussion where someone shows up > > and wants to know why. Does anyone know how to address these, > > efficiently? > > If somebody pops up asks such dumb questions without even looking at the > FAQ, it is bad, if not idiotic, because it takes useful time away from > the developers. > > But my question was not `why don't you implement this feature?`, but `do > you have plans to implement this feature in the future?', and in the > open source spirit of `if something is not there, go implement it > yourself - without troubling developers' ;) > > Also, I have read the section 1.9 of the developers FAQ (Why don't we > use threads in the backend?) long, long ago. > > > If we discuss it, it ends up causing a lot of effort on our part for > > the requestor to finally say, "Oh, gee, I didn't realize that." > > Please don't. See the "NB" at end of my first mail of this thread. > > Anuradha > > -- > > Debian GNU/Linux (kernel 2.4.18-xfs-1.1) > > QOTD: > "I'll listen to reason when it comes out on CD." > > > ---(end of broadcast)--- > TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] > -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Postgresql and multithreading
This in many ways is a bogus argument in that 1) postgresql runs on more then just Linux and 2) amount of memmory that can be addressed by a process is tunable up to the point that it reaches a hardware limitation. It also should be noted that when a process reaches such a size that it better have a good reason. Now let us do a gedanken experiment and say you do have a good reason - fork a couple of these and your machine will thrash like nothing else ... also that whole hardware limitation will come into play more sooner then later ... On 21 Oct 2002, Doug McNaught wrote: > "Steve Wolfe" <[EMAIL PROTECTED]> writes: > > > On the recurring debate of threading vs. forking, I was giving it a fwe > > thoughts a few days ago, particularly with concern to Linux's memory model. > > > > On IA32 platforms with over 4 gigs of memory, any one process can only > > "see" up to 3 or 4 gigs of that. Having each postmaster fork off as a new > > process obviously would allow a person to utilize very copious quantities of > > memory, assuming that (a) they were dealing with concurrent PG sessions, and > > (b) PG had reason to use the memory. > > > > I'm not entirely clear on threading in Linux - would it provide the same > > benefits, or would it suddenly lock you into a 3-gig memory space? > > Linux threads are basically processes that share the same VM space, so > you'd be limited to 3GB or whatever, since that's what a VM space can > "see". > > -Doug > > ---(end of broadcast)--- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to [EMAIL PROTECTED] so that your > message can get through to the mailing list cleanly > -- //\\ || D. Hageman<[EMAIL PROTECTED]> || \\// ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Postgresql and multithreading
"Steve Wolfe" <[EMAIL PROTECTED]> writes: > On the recurring debate of threading vs. forking, I was giving it a fwe > thoughts a few days ago, particularly with concern to Linux's memory model. > > On IA32 platforms with over 4 gigs of memory, any one process can only > "see" up to 3 or 4 gigs of that. Having each postmaster fork off as a new > process obviously would allow a person to utilize very copious quantities of > memory, assuming that (a) they were dealing with concurrent PG sessions, and > (b) PG had reason to use the memory. > > I'm not entirely clear on threading in Linux - would it provide the same > benefits, or would it suddenly lock you into a 3-gig memory space? Linux threads are basically processes that share the same VM space, so you'd be limited to 3GB or whatever, since that's what a VM space can "see". -Doug ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Postgresql and multithreading
"D. Hageman" <[EMAIL PROTECTED]> writes: > This in many ways is a bogus argument in that 1) postgresql runs on more > then just Linux and 2) amount of memmory that can be addressed by a > process is tunable up to the point that it reaches a hardware limitation. 1) The OP specifically asked about Linux threads. 2) True up to a point--Linux (and most other Unices) reserve some part of the VM address space for the kernel. On 64-bit this is a non-issue, on 32-bit it's quite important now that you can put 4+GB in a machine. > It also should be noted that when a process reaches such a size that it > better have a good reason. Now let us do a gedanken experiment and say > you do have a good reason - fork a couple of these and your machine will > thrash like nothing else ... also that whole hardware limitation will come > into play more sooner then later ... True enough. The only real use I can see for gobs of memory on a 32-bit PAE machine with PG is to give each process its own big hunk of 'sortmem' for doing large sorts. If you have 64 GB in the machine setting 'sortmem' to 1GB or so starts to look reasonable... -Doug ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: Antw: Re: [HACKERS] Postgresql and multithreading
On Sat, 19 Oct 2002, Shridhar Daithankar wrote: > On 18 Oct 2002 at 18:10, Ulrich Neumann wrote: > > > I'm still VERY interested in giving those changes back to the > > community, but i think that my changes are still not wanted. Okay, can you submit such changes in pieces / steps? For instance, I know there was alot of talk about how to fix/improve the global variable situation, but nobody appeared to have picked that up and done anything with it ... if you have, is it something you could submit as a patch for review? Note that as the source matures, we've become more critical of patches, especially those that affect the *whole* server, so if you can submit patches in manageable chunks, it becomes easier to review and get those changes in before the next round of changes ... ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Postgresql and multithreading
Anuradha Ratnaweera <[EMAIL PROTECTED]> writes: > Let me explain my posting which started this `thread': > - The developer's FAQ section 1.9 explains why PostgreSQL doesn't use > threads (and many times it has been discussed on the list). > - The TODO list has an item `Experiment with multi-threaded backend' and > points to a mailing list discussion about the implementation by Myron > Scott. His final comment is that he didn't `gain much performance' > and `ended up with some pretty unmanagable code'. He also says that > he wouldn't `personally try this again ... but there probably was a > better way'. > - I was going through the TODO list, and was wondering if I should try > on this. But before doing that, naturally, I wanted to figure out if > any of the core developers themselves have any plans of doing it. > Now, I am trying hard to figure out why this `are you going to do this? > otherwise I can try it', type posting was not differentiated from > numerous `why don't YOU implement this feature' type postings ;) Well, if you'd actually said the above, we'd probably have replied to the effect of "we still think it's an unpromising project, but try it if you like". By my reading, your earliest postings in this thread showed no sign of any familiarity at all with the history: http://archives.postgresql.org/pgsql-hackers/2002-10/msg00704.php http://archives.postgresql.org/pgsql-hackers/2002-10/msg00707.php http://archives.postgresql.org/pgsql-hackers/2002-10/msg00711.php and so you got the sort of response that's usually given to clueless newbies... regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Postgresql and multithreading
On Fri, Oct 18, 2002 at 10:28:38AM -0400, Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > > On Thu, 2002-10-17 at 22:20, Tom Lane wrote: > >> Simple: respond to 'em all with a one-line answer: "convince us why we > >> should use it". The burden of proof always seems to fall on the wrong > >> end in these discussions. > > > ... Now, it seems, that > > people don't want to answer questions at all as it's bothering the > > developers. > > Not at all. But rehashing issues that have been talked out repeatedly > is starting to bug some of us ;-). Perhaps the correct "standard > answer" is more like "this has been discussed before, please read the > list archives". Let me explain my posting which started this `thread': - The developer's FAQ section 1.9 explains why PostgreSQL doesn't use threads (and many times it has been discussed on the list). - The TODO list has an item `Experiment with multi-threaded backend' and points to a mailing list discussion about the implementation by Myron Scott. His final comment is that he didn't `gain much performance' and `ended up with some pretty unmanagable code'. He also says that he wouldn't `personally try this again ... but there probably was a better way'. - I was going through the TODO list, and was wondering if I should try on this. But before doing that, naturally, I wanted to figure out if any of the core developers themselves have any plans of doing it. Now, I am trying hard to figure out why this `are you going to do this? otherwise I can try it', type posting was not differentiated from numerous `why don't YOU implement this feature' type postings ;) Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) "Life is too important to take seriously." -- Corky Siegel ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: Antw: Re: [HACKERS] Postgresql and multithreading
On 18 Oct 2002 at 18:10, Ulrich Neumann wrote: > Marc, > > not only the global variables are a problem. PostgreSQL doesn't clean > up > all the memory before a process terminates and you must deal with > signals between threads. OK, first of all let me say this. I am interested in seeing postgresql mutlithreaded but I don't think it's a good idea to provide a thread per connection. On slick unices like freebsd/linux, threads and processes are almost equally heavy. Besides if connection opening time for a database is an issue, application should cache the pool of connections. Databases are not web servers which should cater to thousands of connections per minute. I am interested in seeing postgresql multithreaded so that it performs better on SMP machine. Right now I am building/designing a *huge* database which is going to use at least 4 way machines. With number of connections very small, around 10-15 and being persistent, I am afraid that postgresql would not use all the available horsepower. > I've modified PostgreSQL that it is completely thread based and nearly > clean > with allocating/deallocating memory and I'm using signals between > threads. I am sure it must have been massive task. > I'm still VERY interested in giving those changes back to the > community, > but i think that my changes are still not wanted. Well, I understand your feeling but I am sure it's not the cases. People here have stated that there are other optimisation areas which needs to be addresses first. Threading is not ruled out but it's not on current radar either. I would be more than happy to assist you to make this happen. I have some experience in threads(pthreads on linux) but absolute no experience in postgresql code. Can we make a patch that converts a single module/operation to multithreaded? Let's say sequential table scan. I propose that such a conversion should be done under conditional compilation with default threaded behaviour as off. This would not disturb existing setup and behaviour where threads are not good enough. Besides since we are distributing almost exclusive work to threads, there should be little syncronisation issues. I request postgresql hackers to look up to this proposal favourably. This would be a good and gradual way to get threading in postgresql. Awaiting your feedback.. Bye Shridhar -- The Consultant's Curse: When the customer has beaten upon you long enough, give him what he asks for, instead of what he needs. This is very strong medicine, and is normally only required once. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Antw: Re: [HACKERS] Postgresql and multithreading
Marc, not only the global variables are a problem. PostgreSQL doesn't clean up all the memory before a process terminates and you must deal with signals between threads. I've modified PostgreSQL that it is completely thread based and nearly clean with allocating/deallocating memory and I'm using signals between threads. I've written the code that it's easy to combine the changes with new releases of PostgreSQL but it has some problems that require a decrease of the number of global variables. This is very hard if you don't want to drift away from the main sourcecode of PostgreSQL. Actually I have a table that holds all the globals that needs to be addressed and I can tell you that there are a lot of global variables. However. I've tried to give the changes with the memory freeing back to the main source tree but my feeling was that they don't like it because it doesn't matter on UNIX if you're only process based. The same with several tools like PSQL. After this I wanted to look closer at the global variables. If this may change in the future you would be very happy if you don't have to search for unreleased memory because it is solved. I'm still VERY interested in giving those changes back to the community, but i think that my changes are still not wanted. Another problem I have is that the build tools i'm using are not GNU and the project file is different in this case. The developers from PostgreSQL don't want this too. Currently i'm working on a project file that uses GNU make but i'm not very happy with this because it is dramatically slower compared to the Metrowerks Tools. With Codewarrior I can build PostgreSQL in around 2 Minutes with all libraires and so on and GNU make needs much much longer. But I still hope that I'll find a way how I can handle all this. PostgreSQL itslef is running very good with "only threads" and the database itself is a VERY GOOD product. In the future I'll make some tests to find out information about the performance and how threads affect the performance on Linux. I'm also working on several additions to PostgreSQL for remote realtime Monitoring and configuration, LDAP authentication and more. Ulrich >>> "Marc G. Fournier" <[EMAIL PROTECTED]> 18.10.2002 05:21:59 >>> On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > > > > > ... what I want to know is whether multithreading is likely to get > > > into in postgresql, say somewhere in 8.x, or even in 9.x? > > > > It may be optional some day, most likely for Win32 at first, but we see > > little value to it on most other platforms; of course, we may be wrong. > > In that case, I wonder if it is worth folking a new project to add > threading support to the backend? Of course, keeping in sync with the > original would be lot of work. Actually, if you go through the archives, there has been talk about what would have to be done to the main source tree towards getting threading included ... as well as a lengthy discussion about the steps involved. the first and foremost issue that needs to be addressed is cleaning up the global variables that are used throughout ... ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html -- This e-mail is virus scanned Diese e-mail ist virusgeprueft ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Postgresql and multithreading
On Fri, 2002-10-18 at 09:28, Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > > On Thu, 2002-10-17 at 22:20, Tom Lane wrote: > >> Simple: respond to 'em all with a one-line answer: "convince us why we > >> should use it". The burden of proof always seems to fall on the wrong > >> end in these discussions. > > > ... Now, it seems, that > > people don't want to answer questions at all as it's bothering the > > developers. > > Not at all. But rehashing issues that have been talked out repeatedly > is starting to bug some of us ;-). Perhaps the correct "standard > answer" is more like "this has been discussed before, please read the > list archives". I agree. That sounds like a much more reasonable response. In fact, if you were to simply let the fledglings respond, it would completely take you guys out of the loop. Perhaps something like a Wiki or FAQ-O-Matic can be added whereby, the user base can help maintain it? That would seemingly help take some load off of Bruce too. Greg ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Postgresql and multithreading
Tom Lane wrote: > Greg Copeland <[EMAIL PROTECTED]> writes: > > On Thu, 2002-10-17 at 22:20, Tom Lane wrote: > >> Simple: respond to 'em all with a one-line answer: "convince us why we > >> should use it". The burden of proof always seems to fall on the wrong > >> end in these discussions. > > > ... Now, it seems, that > > people don't want to answer questions at all as it's bothering the > > developers. > > Not at all. But rehashing issues that have been talked out repeatedly > is starting to bug some of us ;-). Perhaps the correct "standard > answer" is more like "this has been discussed before, please read the > list archives". I need to add something to the developers FAQ on this. I will do it soon. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Postgresql and multithreading
Greg Copeland <[EMAIL PROTECTED]> writes: > On Thu, 2002-10-17 at 22:20, Tom Lane wrote: >> Simple: respond to 'em all with a one-line answer: "convince us why we >> should use it". The burden of proof always seems to fall on the wrong >> end in these discussions. > ... Now, it seems, that > people don't want to answer questions at all as it's bothering the > developers. Not at all. But rehashing issues that have been talked out repeatedly is starting to bug some of us ;-). Perhaps the correct "standard answer" is more like "this has been discussed before, please read the list archives". regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Postgresql and multithreading
On Thu, 2002-10-17 at 22:20, Tom Lane wrote: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > Let me add one more thing on this "thread". This is one email in a long > > list of "Oh, gee, you aren't using that wizz-bang new > > sync/thread/aio/raid/raw feature" discussion where someone shows up and > > wants to know why. Does anyone know how to address these, efficiently? > > Simple: respond to 'em all with a one-line answer: "convince us why we > should use it". The burden of proof always seems to fall on the wrong > end in these discussions. > > regards, tom lane That may be easier said that done. If you don't know what the objections are, it's hard to argue your case. If you do know and understand the objections, chances are you already know the code very well and/or have the mailing lists for a very long time. This basically means, you don't want to hear from anyone unless they are "one" with the code. That seems and sounds very anti-open source. After it's all said and done, I think you guys are barking up the wrong tree. Open Source is all about sharing ideas. Many times I've seen ideas expressed here that were not exact hits yet help facilitate discussion, understanding on the topics in general and in some cases may even spur other ideas or associated code fixes/improvements. When I first started on this list, I was scolded rather harshly for not asking all of my questions on the list. Originally, I was told to ask reasonable questions so that everyone can learn. Now, it seems, that people don't want to answer questions at all as it's bothering the developers. Commonly asked items, such as threading, seems like they are being addressed rather well without core developer participation. Right now, I'm not seeing any down sides to what's currently in place. If the core developers still feel like they are spending more time then they like, then perhaps those that following the mailing list can step forward a little more to address general questions and defer when needed. The topic, such as threading, was previously addressed yet people still followed up on the topic. Perhaps those that don't want to be bothered should allow more time for others to address the topic and leave it alone once it has been addressed. That alone seems like it would be a huge time saver for the developers and a better use of resources. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Postgresql and multithreading
On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > > > > > ... what I want to know is whether multithreading is likely to get > > > into in postgresql, say somewhere in 8.x, or even in 9.x? > > > > It may be optional some day, most likely for Win32 at first, but we see > > little value to it on most other platforms; of course, we may be wrong. > > In that case, I wonder if it is worth folking a new project to add > threading support to the backend? Of course, keeping in sync with the > original would be lot of work. Actually, if you go through the archives, there has been talk about what would have to be done to the main source tree towards getting threading included ... as well as a lengthy discussion about the steps involved. the first and foremost issue that needs to be addressed is cleaning up the global variables that are used throughout ... ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Postgresql and multithreading
Bruce Momjian <[EMAIL PROTECTED]> writes: > Let me add one more thing on this "thread". This is one email in a long > list of "Oh, gee, you aren't using that wizz-bang new > sync/thread/aio/raid/raw feature" discussion where someone shows up and > wants to know why. Does anyone know how to address these, efficiently? Simple: respond to 'em all with a one-line answer: "convince us why we should use it". The burden of proof always seems to fall on the wrong end in these discussions. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Postgresql and multithreading
On Wed, 16 Oct 2002, Bruce Momjian wrote: > It may be optional some day, most likely for Win32 at first, but we see > little value to it on most other platforms; of course, we may be wrong. > I am also not sure if it is a big win on Apache either; I think the > jury is still out on that one, hence the slow adoption of 2.X, and we > don't want to add threads and make a mess of the code or slow it down, > which does often happen. Actually, Apache2 is both multi-process and multi-thread, even in the 'multi-thread' model ... I've played a bit with it, and it works at a sort of 'half way point' taking advantage of advantages of both ... in fact, I really wish someone would look at it seriously, since it mimics alot of what we do to start with ... old apache - one parent process (ie. postmaster) with child process (ie. postgres) actually handling the work new apache - one parent process (ie. postmaster) with child processes (ie. postgres) actually handling the work, with a twist .. each child process can handle x threaded processes so, in a heavily loaded web server, you might see 10 httpd processes running, each of which handling 15 threaded connections ... even getting away from multiple db connections per child process, I could see some other areas where multi-threading could be useful, assuming that my limited knowleddge of threading is remotely correct ... a big/cool one could be: distributed/clustered databases a database could be setup on multiple servers, where the tables are created as 'CREATE TABLE newtable ON SERVER serverb', so that when you connect to that table, the child process knows to auto-matically establish connections to the remote servers to pull data in this would also work for inter-database queries, that several ppl in the past have asked for ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Postgresql and multithreading
On Wed, Oct 16, 2002 at 02:08:21PM -0400, Curtis Faith wrote: > > 2) Including the pros and cons of the feature/implementation and how close > the group is to deciding whether something would be worth doing. - I can > also do this. The pros and cons of many such features have been discussed over the lists as well as on the FAQs. But the second matter, the group's likehood their implementation cannot always be deduced from those communications or from docs. Therefore suggested material into the FAQs are going to be extremely useful to like-to-be developers. They also would hopefully reduce unnecessary traffic on the list. Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) I have found little that is good about human beings. In my experience most of them are trash. -- Sigmund Freud ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Postgresql and multithreading
On Wed, 2002-10-16 at 23:08, Curtis Faith wrote: > Bruce Momjian wrote: > I tried to prepare as best I could before bringing anything forward to > HACKERS. In particular, I read the last two years of archives with anything > to do with the WAL log and looked at the current code, read the TODOs, read > a fair amount of discussions about aio. etc. So I was attempting to comply > with my interpretation of the process. Yet, despite these efforts, you no > doubt consider me guilty of creating unnecessary work, an outcome neither > of us desired. But that "unneccessary work" resulted in Tom finding and fixing an unintended behaviour (aka a performance bug) that prevented postgres from ever doing more than 1 commit per disk revolution on non-lieing SCSI disks ;) > I'm undeterred in my desire to come up with something meaningful and am > working on some interesting tests. I do, however, now know that the level > of scepticism and the standards of architectural purity are high. I > consider this good all around. I still have big expectations for use of aio, especially considering that at least for free OSes one is not forced to stop at the DB/OS boundary, but are free to go and improve the os side implementation as well if it is needed. But still some empirical tests are probably needed - if we can keep IO occupied for 99% in a meaningful way withou aio, then the time is probably better spent on something else ;) Hannu ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Postgresql and multithreading
Bruce Momjian wrote: > Let me add one more thing on this "thread". This is one email in a long > list of "Oh, gee, you aren't using that wizz-bang new > sync/thread/aio/raid/raw feature" discussion where someone shows up and > wants to know why. Does anyone know how to address these, efficiently? You've brought up good points here. I'm sure that you consider me guilty of this with regard to my aio discussions. So I might offer a few specific suggestions. 1) Someone's taking the time to generate a summary of the current thinking with regard to a particular "whiz-bang" feature. - I can do this as a first pass for aio, if you think it's a good idea. 2) Including the pros and cons of the feature/implementation and how close the group is to deciding whether something would be worth doing. - I can also do this. 3) A set of criteria that would need to be met or demonstrated before a feature would be considered good enough for inclusion into the main code. If there was a separate section of the document "Developer's FAQ" that handled this, this would help. 4) A development/feature philosophy section. Maybe three or four paragraphs with one specific example: perhaps multi-threading. 5) I'd also suggest changing some specfics in the FAQ Section 1.2 where there is currently: * The usual process for source additions is: * * Review the TODO list. * Discuss hackers the desirability of the fix/feature. * How should it behave in complex circumstances? Add here that a check should be made to the new section in the FAQ described above. Also, section 1.1 has: * Discussion on the patch typically happens here. If the patch adds a * major feature, it would be a good idea to talk about it first on * the HACKERS list, in order to increase the chances of it being * accepted, as well as toavoid duplication of effort. We should perhaps add here a section describing the phenomenon you describe regarding "whiz-bang" features. I tried to prepare as best I could before bringing anything forward to HACKERS. In particular, I read the last two years of archives with anything to do with the WAL log and looked at the current code, read the TODOs, read a fair amount of discussions about aio. etc. So I was attempting to comply with my interpretation of the process. Yet, despite these efforts, you no doubt consider me guilty of creating unnecessary work, an outcome neither of us desired. I'm undeterred in my desire to come up with something meaningful and am working on some interesting tests. I do, however, now know that the level of scepticism and the standards of architectural purity are high. I consider this good all around. - Curtis ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Postgresql and multithreading
On Wed, 2002-10-16 at 16:37, Robert Treat wrote: > I'm pretty sure BSD allows you to relicense derived code as you see fit. > However, any derived project that was released GPL would have a hell of > a time ever getting put back into the main source (short of > relicensing). Exactly. This is one of the reasons why BSD license is very much liked by proprietary software vendors (think MSFT), unlike the GPL which doesn't allow someone to change and redistribute their work with restrictive licenses. Cheers, Tycho (BTW: I'm not asking to change the license of Postgresql, I know the - dogmatic - answer to that one. So please don't misunderstand my mail) -- Tycho Fruru [EMAIL PROTECTED] "Prediction is extremely difficult. Especially about the future." - Niels Bohr ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Postgresql and multithreading
On Wed, 2002-10-16 at 04:34, Justin Clift wrote: > Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > > > > Nope. To keep the `original' code licence as it is and to release the > > > changes GPL? Is the question sane at first place? > > > > That would be a pretty big mess, I think. People would add your patch > > to our BSD code and it would be GPL. It could be done, of course. > > Don't think so. The patches would be "derived code" that only exist > because of the BSD licensed PostgreSQL base. > > Being "derived code" they'd have to be released as BSD and GPL wouldn't > enter the picture, regardless if they're released separately as add-on > patches or not. > I'm pretty sure BSD allows you to relicense derived code as you see fit. However, any derived project that was released GPL would have a hell of a time ever getting put back into the main source (short of relicensing). Robert Treat ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Postgresql and multithreading
On Wed, 2002-10-16 at 01:27, Shridhar Daithankar wrote: > Well, slow adoption rate is attributed to 'apache 1.3.x is good enough for us' > syndrome, as far as I can see from news. Once linux distros start shipping with > apache 2.x series *only*, the upgrade cycle will start rolling, I guess. I think that's part of it. I think the other part is that by the time you're getting to huge r/s numbers, typical web site bandwidth is already used up. So, what's the point in adding more breathing room when you don't have the bandwidth to use it anyways. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Postgresql and multithreading
Bruce Momjian wrote: > Anuradha Ratnaweera wrote: > > Nope. To keep the `original' code licence as it is and to release the > > changes GPL? Is the question sane at first place? > > That would be a pretty big mess, I think. People would add your patch > to our BSD code and it would be GPL. It could be done, of course. Don't think so. The patches would be "derived code" that only exist because of the BSD licensed PostgreSQL base. Being "derived code" they'd have to be released as BSD and GPL wouldn't enter the picture, regardless if they're released separately as add-on patches or not. :-) Regards and best wishes, Justin Clift > -- > Bruce Momjian| http://candle.pha.pa.us > [EMAIL PROTECTED] | (610) 359-1001 > + If your life is a hard drive, | 13 Roberts Road > + Christ can be your backup.| Newtown Square, Pennsylvania 19073 -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Postgresql and multithreading
On 16 Oct 2002 at 15:40, Gavin Sherry wrote: > > In that case, I wonder if it is worth folking a new project to add > > threading support to the backend? Of course, keeping in sync with the > > original would be lot of work. > > http://sourceforge.net/projects/mtpgsql Last discussion that happened for multithreading was not to add threads per connection like mysql, but to split tasks between threads so that IO blocking and efficiently using SMP abilities would be taken care off IIRC.. One thread per connection isn't going to change much, at least for mainstream postgresql. For embedded, it might be a necessity.. Bye Shridhar -- QOTD: "You're so dumb you don't even have wisdom teeth." ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Postgresql and multithreading
On 16 Oct 2002 at 1:25, Bruce Momjian wrote: > Anuradha Ratnaweera wrote: > > Thanks, Bruce. But what I want to know is whether multithreading is > > likely to get into in postgresql, say somewhere in 8.x, or even in 9.x? > > (as they did with Apache). Are there any plans to do so, or is postgres > > going to remain rather a multi-process application? > It may be optional some day, most likely for Win32 at first, but we see > little value to it on most other platforms; of course, we may be wrong. > I am also not sure if it is a big win on Apache either; I think the Well, I have done some stress testing on 1.3.26 and 2.0.39. Under same hardware and network setup and same test case, 1.3.26 maxed at 475-500 requests/sec and 2.0.39 gave flat 800 requests/sec. Yes, under light load, there is hardly any difference. But Apache2 series is definitely an improvement. > jury is still out on that one, hence the slow adoption of 2.X, and we > don't want to add threads and make a mess of the code or slow it down, > which does often happen. Well, slow adoption rate is attributed to 'apache 1.3.x is good enough for us' syndrome, as far as I can see from news. Once linux distros start shipping with apache 2.x series *only*, the upgrade cycle will start rolling, I guess. Bye Shridhar -- Programming Department: Mistakes made while you wait. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Postgresql and multithreading
On Wed, Oct 16, 2002 at 01:51:28AM -0400, Bruce Momjian wrote: > > Let me add one more thing on this "thread". This is one email in a > long list of "Oh, gee, you aren't using that wizz-bang new > sync/thread/aio/raid/raw feature" discussion where someone shows up > and wants to know why. Does anyone know how to address these, > efficiently? If somebody pops up asks such dumb questions without even looking at the FAQ, it is bad, if not idiotic, because it takes useful time away from the developers. But my question was not `why don't you implement this feature?`, but `do you have plans to implement this feature in the future?', and in the open source spirit of `if something is not there, go implement it yourself - without troubling developers' ;) Also, I have read the section 1.9 of the developers FAQ (Why don't we use threads in the backend?) long, long ago. > If we discuss it, it ends up causing a lot of effort on our part for > the requestor to finally say, "Oh, gee, I didn't realize that." Please don't. See the "NB" at end of my first mail of this thread. Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) QOTD: "I'll listen to reason when it comes out on CD." ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Postgresql and multithreading
Let me add one more thing on this "thread". This is one email in a long list of "Oh, gee, you aren't using that wizz-bang new sync/thread/aio/raid/raw feature" discussion where someone shows up and wants to know why. Does anyone know how to address these, efficiently? If we discuss it, it ends up causing a lot of effort on our part for the requestor to finally say, "Oh, gee, I didn't realize that." It isn't our job to explain that wizz-bang isn't always great. Maybe that should be our reply, "Wizz-bang isn't always great" and leave it at that. Of course, some will leave thinking we are just idiots, but then again, it takes one to know one. It is sort of like walking into a chess match and asking Bobby Fisher why he didn't move that pawn. :-) --- Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote: > > On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > > > > > And a minor question is wheter it is legal to keep the _changes_ in such > > > a project GPL? > > > > Do you mean 'relicence the forked copy'? > > Nope. To keep the `original' code licence as it is and to release the > changes GPL? Is the question sane at first place? > > Anuradha > > -- > > Debian GNU/Linux (kernel 2.4.18-xfs-1.1) > > You got to be very careful if you don't know where you're going, > because you might not get there. > -- Yogi Berra > > > ---(end of broadcast)--- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org > -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Postgresql and multithreading
Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote: > > On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > > > > > And a minor question is wheter it is legal to keep the _changes_ in such > > > a project GPL? > > > > Do you mean 'relicence the forked copy'? > > Nope. To keep the `original' code licence as it is and to release the > changes GPL? Is the question sane at first place? That would be a pretty big mess, I think. People would add your patch to our BSD code and it would be GPL. It could be done, of course. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Postgresql and multithreading
On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote: > On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > > > And a minor question is wheter it is legal to keep the _changes_ in such > > a project GPL? > > Do you mean 'relicence the forked copy'? Nope. To keep the `original' code licence as it is and to release the changes GPL? Is the question sane at first place? Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) You got to be very careful if you don't know where you're going, because you might not get there. -- Yogi Berra ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Postgresql and multithreading
On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > > > > > ... what I want to know is whether multithreading is likely to get > > > into in postgresql, say somewhere in 8.x, or even in 9.x? > > > > It may be optional some day, most likely for Win32 at first, but we see > > little value to it on most other platforms; of course, we may be wrong. > > In that case, I wonder if it is worth folking a new project to add > threading support to the backend? Of course, keeping in sync with the > original would be lot of work. http://sourceforge.net/projects/mtpgsql > And a minor question is wheter it is legal to keep the _changes_ in such > a project GPL? Do you mean 'relicence the forked copy'? Gavin ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Postgresql and multithreading
Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > > > > > ... what I want to know is whether multithreading is likely to get > > > into in postgresql, say somewhere in 8.x, or even in 9.x? > > > > It may be optional some day, most likely for Win32 at first, but we see > > little value to it on most other platforms; of course, we may be wrong. > > In that case, I wonder if it is worth folking a new project to add > threading support to the backend? Of course, keeping in sync with the > original would be lot of work. Probably not, but you can try. > In that way, one should be able to test the hypothesis (whether threads > improve things, or the other way round - if one likes it it that way :)) > without messing around with stable postgres code, as they did and do > with postgresql-R. I guess. > And a minor question is wheter it is legal to keep the _changes_ in such > a project GPL? We don't think we change the license, and we are happy with BSD. It certainly will never be merged in with a GPL, I can say that for sure. > > I am also not sure if it is a big win on Apache either; I think the > > jury is still out on that one, hence the slow adoption of 2.X, > > As far as we are concened, it is the stability, rather than speed which > still keeps us in 1.3. You could easily lose stability with threads -- don't think they are a free ride --- they aren't, and no, I don't feel like regurgitating what is already a 'thread' link on the TODO list. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Postgresql and multithreading
On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote: > Anuradha Ratnaweera wrote: > > > ... what I want to know is whether multithreading is likely to get > > into in postgresql, say somewhere in 8.x, or even in 9.x? > > It may be optional some day, most likely for Win32 at first, but we see > little value to it on most other platforms; of course, we may be wrong. In that case, I wonder if it is worth folking a new project to add threading support to the backend? Of course, keeping in sync with the original would be lot of work. In that way, one should be able to test the hypothesis (whether threads improve things, or the other way round - if one likes it it that way :)) without messing around with stable postgres code, as they did and do with postgresql-R. And a minor question is wheter it is legal to keep the _changes_ in such a project GPL? > I am also not sure if it is a big win on Apache either; I think the > jury is still out on that one, hence the slow adoption of 2.X, As far as we are concened, it is the stability, rather than speed which still keeps us in 1.3. > and we don't want to add threads and make a mess of the code or slow > it down, which does often happen. Fully agreed. Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) Equality is not when a female Einstein gets promoted to assistant professor; equality is when a female schlemiel moves ahead as fast as a male schlemiel. -- Ewald Nyquist ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Postgresql and multithreading
Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 12:59:57AM -0400, Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > > > > > > Is there any plans to make postgresql multithreading? > > > > We don't think it is needed, except perhaps for Win32 and Solaris, which > > have slow process creation times. > > Thanks, Bruce. But what I want to know is whether multithreading is > likely to get into in postgresql, say somewhere in 8.x, or even in 9.x? > (as they did with Apache). Are there any plans to do so, or is postgres > going to remain rather a multi-process application? It may be optional some day, most likely for Win32 at first, but we see little value to it on most other platforms; of course, we may be wrong. I am also not sure if it is a big win on Apache either; I think the jury is still out on that one, hence the slow adoption of 2.X, and we don't want to add threads and make a mess of the code or slow it down, which does often happen. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Postgresql and multithreading
On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > > Is there any plans to make postgresql multithreading? > > Thanks in advance (and also for all who commented to my question > regarding replication.) > > Anuradha > > NB: please don't open fire to declare war on whether multithreading is > needed for PGSql or not. I am just expecting a black and white answer > from the `authorities' ;) This has been discussed at length in the past. The answer has always been no. Consult the archives for extensive heated discussion :-). Gavin ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Postgresql and multithreading
On Wed, Oct 16, 2002 at 12:59:57AM -0400, Bruce Momjian wrote: > Anuradha Ratnaweera wrote: > > > > Is there any plans to make postgresql multithreading? > > We don't think it is needed, except perhaps for Win32 and Solaris, which > have slow process creation times. Thanks, Bruce. But what I want to know is whether multithreading is likely to get into in postgresql, say somewhere in 8.x, or even in 9.x? (as they did with Apache). Are there any plans to do so, or is postgres going to remain rather a multi-process application? Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) One nice thing about egotists: they don't talk about other people. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Postgresql and multithreading
Anuradha Ratnaweera wrote: > > Is there any plans to make postgresql multithreading? > > Thanks in advance (and also for all who commented to my question > regarding replication.) > > Anuradha > > NB: please don't open fire to declare war on whether multithreading is > needed for PGSql or not. I am just expecting a black and white answer > from the `authorities' ;) We don't think it is needed, except perhaps for Win32 and Solaris, which have slow process creation times. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] Postgresql and multithreading
Is there any plans to make postgresql multithreading? Thanks in advance (and also for all who commented to my question regarding replication.) Anuradha NB: please don't open fire to declare war on whether multithreading is needed for PGSql or not. I am just expecting a black and white answer from the `authorities' ;) -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) If you can survive death, you can probably survive anything. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
