Re: [HACKERS] no universally correct setting for fsync
On 8/05/2010 1:56 AM, Josh Berkus wrote: I never meant to suggest any statement in that section is factually wrong; it's just all too rosy, leading people to believe it's no big deal to turn it off. Yeah, that section is overdue for an update. I'll write some new text and post it to pgsql-docs. It's probably worth mentioning that people who want to turn off fsync to gain a performance boost should instead look at a RAID controller with a BBU so they can safely enable write-back caching, getting most of the benefits of fsync=off safely. -- Craig Ringer -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Thu, 2010-05-06 at 12:03 -0700, Josh Berkus wrote: So changing to a lock-based mechanism or designing a plugin interface are really not at all realistic at this date. I agree that changing to a lock-based mechanism is too much at this stage of development. However, putting in a plugin is trivial. We could do it if we choose, without instability or risk. It is as easy a change as option (1). It's not complex to design because it would use the exact same API as the internal conflict resolution module already does; we can just move the current conflict code straight into a contrib module. This can be done bug-free in about 3 hours work. There is no performance issue associated with that either. Plugins would allow all of the various mechanisms requested on list over 18 months, nor would they prevent including some of those options within the core at a later date. Without meaning to cause further contention, it is very clear that putting in contrib modules isn't bad after all, so there is no material argument against the plugin approach. I recognise that plugins for some reason ignite unstated fears, by observation that there is always an argument every time I mention them. I invite an explanation of that off-list. Realistically, we have two options at this point: 1) reduce max_standby_delay to a boolean. 2) have a delay option (based either on WAL glob start time or on system time) like the current max_standby_delay, preferably with some bugs fixed. With a plugin option, I would not object to option 1. If option 1 was taken, without plugins, it's clear that would be against consensus. Having said that, I'll confirm now there will not be an extreme reaction from me if option (1) was forced, nor do I counsel that from others. I said it before and I'll say it again: release early, release often. None of this needs to delay release. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] beta to release
On Fri, 2010-05-07 at 18:26 -0400, Robert Haas wrote: On Fri, May 7, 2010 at 5:35 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: [ argues, in effect, for starting 9.1 development right now ] I can't stop you from spending your time as you please. My development time for at least the next month or two is going to be spent on code-reading the HS/SR code and fixing bugs as they come in. I don't foresee having any time to work on my own 9.1 projects, let alone review anyone else's. I'm really making an effort to be a good community member. There are a couple of reasons I don't think that I can spend ALL of my PG time over the next few months on release prep: 1. I'm not really sure what I would spend that much time doing. 2. My employer has things they want done for 9.1. That having been said, I am perfectly happy to devote a substantial amount of time to helping with 9.0, but I think it needs a little more organization to make it productive. I am not the first person to make this observation and I'm sure I won't be the last. I've often faced the issue you describe. I think its difficult for everybody to help at this stage. In many ways it is a serialization and it's good that Tom holds the gate tighter than normal at this point. The main thing I've tried to do was map out plans for my time so you know which features will be worked on, in which order. Most of that planning needs to happen quietly because its not really possible to say I intend to work on X without it becoming a discussion about what X should look like, which is distracting to the release process. Having said that, I've often found that discussing things off-list with other hackers leads to later grief. I think Bruce wrote a good piece on that once. What 2 or 3 people think is a good way forwards is seldom shared by others, and many ideas fall foul of complete blockers, or simply easier or better ideas. Last year we both worked on the same issues. I'd ask that if you intend to work on anything you know myself or other hackers are working on, please ask so we can avoid duplicating any efforts. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] beta to release
On Sat, May 8, 2010 at 12:04 AM, Marc G. Fournier scra...@hub.org wrote: IMHO, there is nothing wrong with you (or any other developer) spending time working on v9.1 features if said person feels that they have satisfied themselves that v9.0 is ready for release (ie. I think the best test anyone can run, espeecially those whose employer uses PostgreSQL, is to run tests using their own applications / environment ... regressions are great and all, but real world always finds something new) ... Tom's employer requires *as clean* a release as possible, so for him, his priority is to go through and test everything and anything he can think of ... and that includes doing review of the code that got added ... but, that is what *his* employer is paying his time to do ... Again, IMHO, the critical thing throughout beta is that if a bug is reported, or an oddiity, that any 'development for 9.1' gets drop'd fast and teh bug report is jumped on / fixed ASAP ... To me, beta is ... we're ready for release, we're not throwing in any new code .. it is a time for more 'end users' to start testing real world applications (even if they won't run 9.0, but will wait for 9.0.1) to start evaluating, which inevitable will generate bug reports to be fixed ... That all sounds reasonable to me, and is about how I feel about it myself. Thinking about Tom's comments a little more, I think there are probably a few things that I could/should go back over in a little more detail, and I'm going to make time to do that, but I'm also going to work on 9.1 features as time permits. I will also take the time to respond to design proposals for proposed 9.1 projects, especially from anyone who similarly takes the time to respond to mine. I probably will not look at actual patches very much just yet. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] beta to release
On Sat, May 8, 2010 at 6:34 AM, Simon Riggs si...@2ndquadrant.com wrote: I've often faced the issue you describe. I think its difficult for everybody to help at this stage. In many ways it is a serialization and it's good that Tom holds the gate tighter than normal at this point. The main thing I've tried to do was map out plans for my time so you know which features will be worked on, in which order. Most of that planning needs to happen quietly because its not really possible to say I intend to work on X without it becoming a discussion about what X should look like, which is distracting to the release process. Having said that, I've often found that discussing things off-list with other hackers leads to later grief. I think Bruce wrote a good piece on that once. What 2 or 3 people think is a good way forwards is seldom shared by others, and many ideas fall foul of complete blockers, or simply easier or better ideas. Yeah, I think we need to start having those discussions on-list. Trying to develop things quietly so it doesn't become a distraction can result in wasting a lot of time going down a dead end. Tom and Alvaro saved me a ton of time on the temporary-relations-get-different-filenames patch I submitted earlier this week, and I really appreciate that. My guess is that it didn't take them much time to respond to my emails, either. Even if neither of them actually read for several months, I have a lot more confidence that the basic approach is sound than I would otherwise, and I was able to develop it much more quickly than would have been possible in a complete vacuum. Of course, as you say, we don't want to go nuts and get into long arguments about things, but I think that high-level design discussions should be viewed as in order. This is important (1) to avoid duplication, (2) to enable people to get done the work their employers are willing to fund at the time their employers are willing to fund it, and (3) to create a reasonable hope of some of the big patches landing earlier in the cycle. Last year we both worked on the same issues. I'd ask that if you intend to work on anything you know myself or other hackers are working on, please ask so we can avoid duplicating any efforts. I will try to do that. Currently, the things that I know I will be spending some effort on for 9.1 are (1) global temporary and/or unlogged tables, (2) inner join removal, and (3) mentoring the GSoC projects on matviews, json, and merge. Everything else is pretty amorphous at this point, but I typically send out design emails fairly early on in the process. Please also share your plans for 9.1. Come to think of it, I think we should start a page on the wiki for developers to list major projects they plan to work on for 9.1, maybe also with a space to indicate whether the project is possible or definite. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] beta to release
On Sat, 2010-05-08 at 08:12 -0400, Robert Haas wrote: (3) mentoring the GSoC projects on matviews, json, and merge. Everything else is pretty amorphous at this point, That's good you volunteered. I'm sorry to say that I'm really surprised to hear anyone thinks MERGE or matviews are suitable student projects. I regard both as long and hard projects and feel we shouldn't be encouraging people to do things like that as their first project. (Not aimed at you, or any individual.) Please also share your plans for 9.1. Will do. Synch rep. is one, there are others, but not as much thought gone into this as usual yet. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Lightning Talks for PgCon! Submit yours today.
Hi! We're having Lightning Talks again at PgCon - scheduled for 5:30pm on May 20th in Ottawa! Do you have a talk or idea you'd like to share? Lightning Talks are one of the most highly attended sessions because they are fast, fun, and useful (but not always). Slides are not required. If you use them, you'll have to operate them as PDFs. Please send your 5-minute talk idea to li...@pgcon.org. Slots fill up fast, so get them in now! We'll accept submissions until May 16 via email, and after that, you'll need to find me (Selena) at the conference if you'd like to be added. We can only accept 11 talks in the time allowed. Selection is generally first-come, first-served. I will not determine the order of the talks until the time of the session. More details are at: http://www.pgcon.org/2010/schedule/events/267.en.html -- http://chesnok.com/daily - me http://endpoint.com - work -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] psql weird behaviour with charset encodings
hernan gonzalez hgonza...@gmail.com writes:
Sorry about a error in my previous example (mixed width and precision).
But the conclusion is the same - it works on bytes:
This example works like that because it's running in C locale always.
Try something like this:
#includestdio.h
#includelocale.h
int main () {
char s[] = ni\xc3qo; /* 5 bytes , not valid utf8 */
setlocale(LC_ALL, );
printf(|%.*s|\n,3,s);
return 0;
}
I get different (and undesirable) effects depending on LANG.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] possible memory leak with SRFs
Nikhil Sontakke nikhil.sonta...@enterprisedb.com writes: Ok thanks. So if someone uses a really long-running srf with argument expression evaluations thrown in, then running into out of memory issues should be expected and then in those cases they are better off using multiple srf calls to get the same effect if they can.. I believe this is only an issue for SRFs called in a query targetlist, which is a usage fraught with semantic problems anyway. Hopefully we can get LATERAL done soon (I'm planning to look at it for 9.1) and then deprecate this whole mess. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] possible memory leak with SRFs
On 05/07/2010 09:06 PM, Nikhil Sontakke wrote: Yeah this is my basic confusion. But wouldn't the arguments be evaluated afresh on the subsequent call for this SRF? No, see ExecMakeFunctionResult(). If we did that we'd have serious problems with volatile functions, ie srf(random()). Ok thanks. So if someone uses a really long-running srf with argument expression evaluations thrown in, then running into out of memory issues should be expected and then in those cases they are better off using multiple srf calls to get the same effect if they can.. I've very recently looked into this exact case myself for someone, and came to the conclusion that there is no simple fix for this. If you want to see a concrete example of a query that fails, apply your patch and then run the regression tests -- the misc test will fail. I think this is an example of why we still need to implement a real SFRM_ValuePerCall mode that allows results to be pipelined. Yes, ValuePerCall sort of works from the targetlist, but it is pretty much useless for the use cases where people really want to use it. Or would a FROM clause ValuePerCall suffer the same issue? Joe signature.asc Description: OpenPGP digital signature
Re: [HACKERS] possible memory leak with SRFs
Joe Conway m...@joeconway.com writes: I think this is an example of why we still need to implement a real SFRM_ValuePerCall mode that allows results to be pipelined. Yes, ValuePerCall sort of works from the targetlist, but it is pretty much useless for the use cases where people really want to use it. Or would a FROM clause ValuePerCall suffer the same issue? I don't think it'd be a big problem. We could use the technique suggested in the comments in ExecMakeTableFunctionResult: use a separate memory context for evaluating the arguments than for evaluating the function itself. This will work in FROM because we can insist the SRF be at top level. The problem with SRFs in tlists is that they can be anywhere and there can be more than one, so it's too hard to keep track of what to reset when. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] possible memory leak with SRFs
On 05/08/2010 09:12 AM, Tom Lane wrote: Joe Conway m...@joeconway.com writes: I think this is an example of why we still need to implement a real SFRM_ValuePerCall mode that allows results to be pipelined. Yes, ValuePerCall sort of works from the targetlist, but it is pretty much useless for the use cases where people really want to use it. Or would a FROM clause ValuePerCall suffer the same issue? I don't think it'd be a big problem. We could use the technique suggested in the comments in ExecMakeTableFunctionResult: use a separate memory context for evaluating the arguments than for evaluating the function itself. This will work in FROM because we can insist the SRF be at top level. The problem with SRFs in tlists is that they can be anywhere and there can be more than one, so it's too hard to keep track of what to reset when. That's what I was thinking. I saw your other email about LATERAL for 9.1 -- would it be helpful for me to work on this issue for 9.1? After all, about 7 years ago I said I'd do it ;-). Or do you think it will be an integral part of the LATERAL work? Joe signature.asc Description: OpenPGP digital signature
Re: [HACKERS] possible memory leak with SRFs
Joe Conway m...@joeconway.com writes: That's what I was thinking. I saw your other email about LATERAL for 9.1 -- would it be helpful for me to work on this issue for 9.1? After all, about 7 years ago I said I'd do it ;-). Or do you think it will be an integral part of the LATERAL work? Dunno. What I was actually wanting to focus on is the problem of generalizing nestloop inner indexscans so that they can use parameters coming from more than one join level up. Robert suggested that LATERAL might fall out of that fairly easily, but I haven't thought much about it yet. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] psql weird behaviour with charset encodings
Wow, you are right, this is bizarre... And it's not that glibc intends to compute the length in unicode chars, it actually counts bytes (c plain chars) -as it should- for computing field widths... But, for some strange reason, when there is some width calculation involved it tries so parse the char[] using the locale encoding (when there's no point in doing it!) and if it fails, it truncates (silently) the printf output. So it seems more a glib bug to me than an interpretion issue (bytes vs chars). I posted some details in stackoverflow: http://stackoverflow.com/questions/2792567/printf-field-width-bytes-or-chars BTW, I understand that postgresql uses locale semantics in the server code. But is this really necessary/appropiate in the client (psql) side? Couldnt we stick with C locale here? -- Hernán J. González http://hjg.com.ar/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Simon Riggs wrote: With a plugin option, I would not object to option 1. If option 1 was taken, without plugins, it's clear that would be against consensus. Having said that, I'll confirm now there will not be an extreme reaction from me if option (1) was forced, nor do I counsel that from others. I found this email amusing. You phrase it like the community is supposed to be worried by an objection from you or an extreme reaction; I certainly am not. You have been in the community long enough to not use such phrasing. This is not the first time I have complained about this. I have no idea why an objection from you should mean more than an objection from anyone else in the community, and I have no idea what an extreme reaction means, or why anyone should care. Do you think the community is negotiting with you? I think the concensus is to change this setting to a boolean. If you don't want to do it, I am sure we can find someone who will. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Sat, May 8, 2010 at 2:48 PM, Bruce Momjian br...@momjian.us wrote: I think the concensus is to change this setting to a boolean. If you don't want to do it, I am sure we can find someone who will. I still think we should revert to Tom's original proposal. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Robert Haas wrote: On Sat, May 8, 2010 at 2:48 PM, Bruce Momjian br...@momjian.us wrote: I think the concensus is to change this setting to a boolean. ?If you don't want to do it, I am sure we can find someone who will. I still think we should revert to Tom's original proposal. And Tom's proposal was to do it on WAL slave arrival time? If we could get agreement from everyone that that is the proper direction, fine, but I am hearing things like plugins, and other complexity that makes it seem we are not getting closer to an agreed solution, and without agreement, the simplest approach seems to be just to remove the part we can't agree upon. I think the big question is whether this issue is significant enough that we should ignore our policy of no feature design during beta. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Sat, May 8, 2010 at 3:40 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Sat, May 8, 2010 at 2:48 PM, Bruce Momjian br...@momjian.us wrote: I think the concensus is to change this setting to a boolean. ?If you don't want to do it, I am sure we can find someone who will. I still think we should revert to Tom's original proposal. And Tom's proposal was to do it on WAL slave arrival time? If we could get agreement from everyone that that is the proper direction, fine, but I am hearing things like plugins, and other complexity that makes it seem we are not getting closer to an agreed solution, and without agreement, the simplest approach seems to be just to remove the part we can't agree upon. I think the big question is whether this issue is significant enough that we should ignore our policy of no feature design during beta. Tom's proposal was basically to define recovery_process_lock_timeout. The recovery process would wait X seconds for a lock, then kill whoever held it. It's not the greatest knob in the world for the reasons already pointed out, but I think it's still better than a boolean and will be useful to some users. And it's pretty simple. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] plpython3
On Feb 1, 2010, at 12:18 PM, James William Pye wrote: Right now, I'm trying to trim some of the easy issues[1] and getting a project web page up. I expect to be able to make a release soon, and I'll follow-up to this thread when I do. Well, I ended up doing some others things at that point in time, but I have managed to get a release out: http://python.projects.postgresql.org/backend/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Bruce Momjian br...@momjian.us writes: I have no idea why an objection from you should mean more than an objection from anyone else in the community, and I have no idea what an extreme reaction means, or why anyone should care. Maybe I shouldn't say anything here. But clearly while you're spot on that Simon's objection is worth just as much as any other contributor's, I disagree that we shouldn't care about the way those people feel about being a member of our community. I appreciate your efforts to avoid having anyone here use such a wording but I can't help to dislike your argument for it. I hope that's simply a localisation issue (l10n is so much harder than i18n). Anyway, I so much hate reading such exchanges here that I couldn't help ranting about it. Back to suitable -hackers content. I think the concensus is to change this setting to a boolean. If you don't want to do it, I am sure we can find someone who will. I don't think so. I understand the current state to be: a. this problem is not blocking beta, but a must fix before release b. we either have to change the API or the behavior c. only one behavior change has been proposed, by Tom d. proposed behavior would favor queries rather than availability e. API change 1 is boolean + explicit pause/resume command f. API change 2 is boolean + plugin facility, with a contrib for current behavior. g. API change 3 is boolean only I don't remember reading any mail on this thread bearing consensus on the choices above, but rather either one of us pushing for his vision or people defending the current situation, complaining about it or asking that a reasonable choice is made soon. If we have to choose between reasonable and soon, soon won't be my vote. Beta is meant to last more or less 3 months after all. Each party's standing is clear. Decision remains to be made, and I guess that the one writing the code will have a much louder voice. Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] psql weird behaviour with charset encodings
Well, I finally found some related -rather old- issues in Bugzilla (glib) http://sources.redhat.com/bugzilla/show_bug.cgi?id=6530 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=208308 http://sources.redhat.com/bugzilla/show_bug.cgi?id=649 The last explains why they do not consider it a bug: ISO C99 requires for %.*s to only write complete characters that fit below the precision number of bytes. If you are using say UTF-8 locale, but ISO-8859-1 characters as shown in the input file you provided, some of the strings are not valid UTF-8 strings, therefore sprintf fails with -1 because of the encoding error. That's not a bug in glibc. It's clear, though it's also rather ugly, from a specification point of view (we must count raw bytes for the width field, but also must decode the utf8 chars for finding character boundaries). I guess we must live with that. Hernán J. González
Re: [HACKERS] max_standby_delay considered harmful
Robert Haas wrote: On Sat, May 8, 2010 at 3:40 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Sat, May 8, 2010 at 2:48 PM, Bruce Momjian br...@momjian.us wrote: I think the concensus is to change this setting to a boolean. ?If you don't want to do it, I am sure we can find someone who will. I still think we should revert to Tom's original proposal. And Tom's proposal was to do it on WAL slave arrival time? ?If we could get agreement from everyone that that is the proper direction, fine, but I am hearing things like plugins, and other complexity that makes it seem we are not getting closer to an agreed solution, and without agreement, the simplest approach seems to be just to remove the part we can't agree upon. I think the big question is whether this issue is significant enough that we should ignore our policy of no feature design during beta. Tom's proposal was basically to define recovery_process_lock_timeout. The recovery process would wait X seconds for a lock, then kill whoever held it. It's not the greatest knob in the world for the reasons already pointed out, but I think it's still better than a boolean and will be useful to some users. And it's pretty simple. I thought there was concern about lock stacking causing unpredictable/unbounded delays. I am not sure boolean has a majority vote, but I am suggesting that because it is the _minimal_ feature set, and when we can't agree during beta, the minimal feature set seems like the best choice. Clearly, anything is more feature-full than boolean --- the big question is whether Tom's proposal is significantly better than boolean that we should spend the time designing and implementing it, with the possibility it will all be changed in 9.1. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Bruce Momjian wrote: I think the big question is whether this issue is significant enough that we should ignore our policy of no feature design during beta The idea that you're considering removal of a feature that we already have people using in beta and making plans around is a policy violation too you know. A freeze should include not cutting things just because their UI or implementation is not ideal yet. And you've been using the word consensus here when there is no such thing. At best there's barely a majority here among people who have stated an opinion, and consensus means something much stronger even than that; that means something closer to unanimity. I thought the summary of where the project is at Josh wrote at http://archives.postgresql.org/message-id/4be31279.7040...@agliodbs.com was excellent, both from a technical and a process commentary standpoint. I'd be completely happy to follow that plan, and then we'd be at a consensus--with no one left arguing. It was very clear back in February that if SR didn't hit the feature set to make HS less troublesome out of the box, there would be some limitations here, and that set of concerns hasn't changed much since then. I thought the backup plan if we didn't get things like xid feedback was to keep the capability as written anyway, knowing that it's still much better than no control over cancellation timing available at all. Keep improving documentation around its issues, and continue to hack away at them in user space and in the field. Then we do better for 9.1. You seem bent on removing the feedback part of that cycle. The full statement of the ESR bit Josh was quoting is Release early. Release often. And listen to your customers.[1] My customers include some of whom believed the PostgreSQL community process enough to contribute toward the HS development that's been completed and donated to the project. They have a pretty clear view on this I'm relaying when I talk about what I'd like to see happen. They are saying they cannot completely ignore their requirements for HA failover, but would be willing to loosen them just a bit (increasing failover time slightly) if it reduces the odds of query cancellation, and therefore improves how much load they can expect to push toward the standby. max_standby_delay is a currently available mechanism that does that. I'm not going to be their nanny and say no, that's not perfectly predictable, you might get a query canceled sometimes when you don't expect it anyway. Instead, I was hoping to let them deploy 9.0 with this option available (but certainly not the default), informed of the potential risks, see how that goes. We can confirm whether the userland workarounds we believe will be effective here really are. If so, then we can solider forward directly incorporating them into the server code, knowing that works. If not, switch to one of the safer modes, see if there's something better to use altogether in 9.1, and perhaps this whole approach gets removed. That's healthy development progress either way. Upthread Bruce expressed some concern that this was going to live forever once deployed. There is no way I'm going to let this behavior continue to be available in 9.1 if field tests say the workarounds aren't good enough. That's going to torture all of us who do customer deployments of this technology every day if that turns out to be the case, and nobody is going to feel the heat from that worse than 2ndQuadrant. I did a round once of removing GUCs that didn't do what they were expected to in the field before, based on real-world tests showing regular misuse, and I'll do it again if this falls into that same category. We've already exposed this release to a whole stack of risk from work during its development cycle, risk that doesn't really drop much just from cutting this one bit. I'd at least like to get all the reward possible from that risk, which I expected to include feedback in this area. Circumventing the planned development process by dropping this now will ruin how I expected the project to feel out the right thing on the user side, and we'll all be left with little more insight for what to do in 9.1 than we have now. And I'm not looking forward to explaining to people why a feature they've been seeing and planning to deploy for months has now been cut only after what was supposed to be a freeze for beta. [1] http://catb.org/esr/writings/homesteading/cathedral-bazaar/ar01s04.html , and this particular bit is quite relevant here: Linus was keeping his hacker/users constantly stimulated and rewarded—stimulated by the prospect of having an ego-satisfying piece of the action, rewarded by the sight of constant (even daily) improvement in their work. Linus was directly aiming to maximize the number of person-hours thrown at debugging and development, even at the possible cost of instability in the code and user-base burnout
Re: [HACKERS] max_standby_delay considered harmful
Greg Smith wrote: Bruce Momjian wrote: I think the big question is whether this issue is significant enough that we should ignore our policy of no feature design during beta The idea that you're considering removal of a feature that we already have people using in beta and making plans around is a policy violation too you know. A freeze should include not cutting things just because their UI or implementation is not ideal yet. And you've been using the word consensus here when there is no such thing. At best there's barely a majority here among people who have stated an opinion, and consensus means something much stronger even than that; that means something closer to unanimity. I thought the summary of where the project is at Josh wrote at http://archives.postgresql.org/message-id/4be31279.7040...@agliodbs.com was excellent, both from a technical and a process commentary standpoint. I'd be completely happy to follow that plan, and then we'd be at a consensus--with no one left arguing. I can't argue with anything you have said in your email. The big question is whether designing during beta is worth it in this case, and whether we can get something that is useful and gives us useful feedback for 9.1, and is it worth spending the time to figure this out during beta? If we can, great, let's do it, but I have not seen that yet, and I am unclear how long we should keep trying to find it. I think everyone agrees the current code is unusable, per Heikki's comment about a WAL file arriving after a period of no WAL activity, and look how long it took our group to even understand why that fails so badly. I thought Tom's idea had problems, and there were ideas of how to improve it. It just seems like we are drifting around on something that has no easy solution, and not something that we are likely to hit during beta where we should be focusing on the release. Saying we have three months to fix this during beta seems like a recipe for delaying the final release, and this feature is not worth that. What we could do is to convert max_standby_delay to a boolean, 'ifdef' out the code that was handling non-boolean cases, and then if someone wants to work on a patch in a corner and propose something in a month that improves this, we can judge the patch on its own merits, and apply it if it is a great benefit, because basically that is what we are doing now if we fix this --- adding a new patch/feature during beta. (Frankly, because we are not requiring an initdb during beta, I am unclear how we are going to rename max_standby_delay to behave as a boolean.) It is great if we can get a working max_standby_delay, but I fear drifting/distraction at this stage. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Sunday 09 May 2010 01:34:18 Bruce Momjian wrote: I think everyone agrees the current code is unusable, per Heikki's comment about a WAL file arriving after a period of no WAL activity, and look how long it took our group to even understand why that fails so badly. To be honest its not *that* hard to simply make sure generating wal regularly to combat that. While it surely aint a nice workaround its not much of a problem either. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Andres Freund and...@anarazel.de writes: On Sunday 09 May 2010 01:34:18 Bruce Momjian wrote: I think everyone agrees the current code is unusable, per Heikki's comment about a WAL file arriving after a period of no WAL activity, and look how long it took our group to even understand why that fails so badly. To be honest its not *that* hard to simply make sure generating wal regularly to combat that. While it surely aint a nice workaround its not much of a problem either. Well, that's dumping a kluge onto users; but really that isn't the point. What we have here is a badly designed and badly implemented feature, and we need to not ship it like this so as to not institutionalize a bad design. I like the proposal of a boolean because it provides only the minimal feature set of two cases that are both clearly needed and easily implementable. Whatever we do later is certain to provide a superset of those two cases. If we do something else (and that includes my own proposal of a straight lock timeout), we'll be implementing something we might wish to take back later. Taking out features after they've been in a release is very hard, even if we realize they're badly designed. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] psql weird behaviour with charset encodings
hgonza...@gmail.com writes: http://sources.redhat.com/bugzilla/show_bug.cgi?id=649 The last explains why they do not consider it a bug: ISO C99 requires for %.*s to only write complete characters that fit below the precision number of bytes. If you are using say UTF-8 locale, but ISO-8859-1 characters as shown in the input file you provided, some of the strings are not valid UTF-8 strings, therefore sprintf fails with -1 because of the encoding error. That's not a bug in glibc. Yeah, that was about the position I thought they'd take. So the bottom line here is that we're best off to avoid %.*s because it may fail if the string contains data that isn't validly encoded according to libc's idea of the prevailing encoding. I think that means the patch I committed earlier is still a good idea, but the comments need a bit of adjustment. Will fix. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] psql weird behaviour with charset encodings
hernan gonzalez hgonza...@gmail.com writes: BTW, I understand that postgresql uses locale semantics in the server code. But is this really necessary/appropiate in the client (psql) side? Couldnt we stick with C locale here? As far as that goes, I think we have to turn on that machinery in order to have gettext() work (ie, to have localized error messages). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Bruce Momjian wrote: Simon Riggs wrote: On Fri, 2010-04-30 at 12:22 -0400, Bruce Momjian wrote: (wal_keep_segments can be changed without restarting, right?) Should we allow -1 to mean keep all segments? Why is that not called max_wal_segments? wal_keep_segments sounds like its been through Google translate. LOL, good one. I assume it was done so it would start with 'wal', but I see 'max_wal_senders', which doesn't start with 'wal' and would match your suggestion exactly. I think we should either rename 'wal_keep_segments' or 'max_wal_senders'. Uh, did we decide that 'wal_keep_segments' was the best name for this GUC setting? I know we shipped beta1 using that name. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Bruce Momjian br...@momjian.us writes: Uh, did we decide that 'wal_keep_segments' was the best name for this GUC setting? I know we shipped beta1 using that name. I thought min_wal_segments was a reasonable proposal, but it wasn't clear if there was consensus or not. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Tom Lane wrote: Taking out features after they've been in a release is very hard, even if we realize they're badly designed. It doesn't have to be; that's the problem the release often part takes care of. If a release has only been out a year, and a new one comes out saying oh, that thing we released for the first time in the last version, it didn't work as well as we'd hoped in the field; you should try to avoid that and use this new implementation that works better instead once you can upgrade, that's not only not hard, it's exactly what people using a X.0 release expect to happen. I've read the message from you that started off this thread several times now. Your low-level code implementation details shared later obviously need to be addressed. But all of the fundamental and fatal issues you mentioned at the start continue to strike me as either situations where you don't agree with the use case this was designed for, or spots where you feel the userland workarounds required to make it work right are too onerous. Bruce's objections seem to fall mainly into the latter category. I've been wandering around talking to people about that exact subject--what do people want and expect from Hot Standby, and what would they do to gain its benefits--for over six months now, independently of Simon's work which did a lot of that before me too. The use cases are covered as best they can be without better support from expected future SR features like heartbeats and XID loopback. As for the workarounds required to make things work, the responses I get match what we just saw from Andres. When the required details are explained, people say that's annoying but I can do that, and off we go. There are significant documentation issues I know need to be cleaned up here, and I've already said I'll take care of that as soon as freeze is really here and I have a stable target. (That this discussion is still going on says that's not yet) What I fail to see are problems significant enough to not ship the parts of this feature that are done, so that it can be used by those it is appropriate for, allow feedback, and make it easy to test individual improvements upon what's already there. I can't make you prioritize based on what people are telling me. All I can do is suggest you reconsider handing control over the decision to use this feature or not to the users of the software, so they can make their own choice. I'm tired of arguing about this instead of doing productive work, and I've done all I can here to try and work within the development process of the community. If talk of removing the max_standby_delay feature clears up, I'll happily provide my promised round of documentation updates, to make its limitations and associated workarounds as clear as they can be, within a week of being told go on that. If instead this capability goes away, making those moot, I'll maintain my own release for the 2ndQuadrant customers who have insisted they need this capability if I have to. That would be really unfortunate, because the only bucket I can pull time out of for that is the one I currently allocate to answering questions on the mailing lists here most days. I'd rather spend that helping out the PostgreSQL community, but we do need to deliver what our customers want too. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Greg Smith wrote: Tom Lane wrote: Taking out features after they've been in a release is very hard, even if we realize they're badly designed. It doesn't have to be; that's the problem the release often part takes care of. If a release has only been out a year, and a new one comes out saying oh, that thing we released for the first time in the last version, it didn't work as well as we'd hoped in the field; you should try to avoid that and use this new implementation that works better instead once you can upgrade, that's not only not hard, it's exactly what people using a X.0 release expect to happen. I think this is the crux of the issue. Tom and I are saying that historically we have shipped only complete features, or as complete as reasonable, and have removed items during beta that we found didn't meet this criteria, in an attempt to reduce the amount of feature set churn in Postgres. A database is complex, so modifying the API between major releases is something we only do when we find a significant benefit. In this case, if we keep max_standby_delay as non-boolean, we know it will have to be redesigned in 9.1, and it is unclear to me what additional knowledge we will gain by shipping it in 9.0, except to have to tell people that it doesn't work well or requires complex work-arounds, and that doesn't thrill any of us. (I already suggested that statement_timeout might supply a reasonable and predictable workaround for non-boolean usage of max_standby_delay.) -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Sat, May 8, 2010 at 6:51 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Sat, May 8, 2010 at 3:40 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Sat, May 8, 2010 at 2:48 PM, Bruce Momjian br...@momjian.us wrote: I think the concensus is to change this setting to a boolean. ?If you don't want to do it, I am sure we can find someone who will. I still think we should revert to Tom's original proposal. And Tom's proposal was to do it on WAL slave arrival time? ?If we could get agreement from everyone that that is the proper direction, fine, but I am hearing things like plugins, and other complexity that makes it seem we are not getting closer to an agreed solution, and without agreement, the simplest approach seems to be just to remove the part we can't agree upon. I think the big question is whether this issue is significant enough that we should ignore our policy of no feature design during beta. Tom's proposal was basically to define recovery_process_lock_timeout. The recovery process would wait X seconds for a lock, then kill whoever held it. It's not the greatest knob in the world for the reasons already pointed out, but I think it's still better than a boolean and will be useful to some users. And it's pretty simple. I thought there was concern about lock stacking causing unpredictable/unbounded delays. I am not sure boolean has a majority vote, but I am suggesting that because it is the _minimal_ feature set, and when we can't agree during beta, the minimal feature set seems like the best choice. Clearly, anything is more feature-full than boolean --- the big question is whether Tom's proposal is significantly better than boolean that we should spend the time designing and implementing it, with the possibility it will all be changed in 9.1. I doubt it's likely to be thrown out completely. We might decide to fine-tune it in some way. My fear is that if we ship this with only a boolean, we're shipping crippleware. If that fear turns out to be unfounded, I will of course be happy, but that's my concern, and I don't believe that it's entirely unfounded. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Sat, May 8, 2010 at 10:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: Uh, did we decide that 'wal_keep_segments' was the best name for this GUC setting? I know we shipped beta1 using that name. I thought min_wal_segments was a reasonable proposal, but it wasn't clear if there was consensus or not. I think most people thought it was another reasonable choice, but I think the consensus position is probably something like it's about the same rather than it's definitely better. We had one or two people with stronger opinions than that on either side, I believe. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Robert Haas wrote: Clearly, anything is more feature-full than boolean --- the big question is whether Tom's proposal is significantly better than boolean that we should spend the time designing and implementing it, with the possibility it will all be changed in 9.1. I doubt it's likely to be thrown out completely. We might decide to fine-tune it in some way. My fear is that if we ship this with only a boolean, we're shipping crippleware. If that fear turns out to be unfounded, I will of course be happy, but that's my concern, and I don't believe that it's entirely unfounded. Well, historically, we have been willing to not ship features if we can't get it right. No one has ever accused us of crippleware, but our hesitancy has caused slower user adoption, though long-term, it has helped us grow a dedicated user base that trusts us. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Sun, May 9, 2010 at 12:08 AM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: Clearly, anything is more feature-full than boolean --- the big question is whether Tom's proposal is significantly better than boolean that we should spend the time designing and implementing it, with the possibility it will all be changed in 9.1. I doubt it's likely to be thrown out completely. We might decide to fine-tune it in some way. My fear is that if we ship this with only a boolean, we're shipping crippleware. If that fear turns out to be unfounded, I will of course be happy, but that's my concern, and I don't believe that it's entirely unfounded. Well, historically, we have been willing to not ship features if we can't get it right. No one has ever accused us of crippleware, but our hesitancy has caused slower user adoption, though long-term, it has helped us grow a dedicated user base that trusts us. We can make the decision to not ship the feature if the feature is max_standby_delay. But I think the feature is Hot Standby, which I think we've pretty much committed to shipping. And I am concerned that if the only mechanism for controlling query cancellation vs. recovery lag is a boolean, people feel that we didn't get Hot Standby right. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
