Re: [HACKERS] Listen/notify across clusters
On 10 Jul 2013, at 19:26, Josh Berkus wrote: > > > Huh? LISTEN/NOTIFY across replication has been a desired feature since > we introduced streaming replication. We want it, there's just no > obvious way to do it. > > Your email kinda implies that it's not desirable. Thanks Josh. I was under impression that it is desirable. It certainly makes sense to be used with the existing database infrastructure. In terms of features, apart from separating LISTEN so that it can be actually used on Standbys, wouldn't it be a matter of including the notifications in the WAL stream, as simple packets ? This would guarantee same behaviour as on the master. -- GJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Listen/notify across clusters
Hi masters of PostgreSQL, I recently got asked about possibility of listening to notifications on warm standby. So question, how hard would that be to implement ? Is it even possible without major changes to the architecture ? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unused code in float8_to_char , formatting.c ?
On 7 Apr 2013, at 05:14, Robert Haas wrote: > On Thu, Apr 4, 2013 at 6:47 PM, Greg Jaskiewicz wrote: >> Looking around the code Today, one of my helpful tools detected this dead >> code. >> As far as I can see, it is actually unused call to strlen() in formatting.c, >> float8_to_char(). > > I poked at this a little and suggest the following somewhat more > extensive cleanup. > > It seems to me that there are a bunch of these functions where len is > unconditionally initialized in NUM_TOCHAR_prepare and then used there. > Similarly in NUM_TOCHAR_cleanup. And then there's a chunk of each > individual function that does it a third time. Rather than use the > same variable in all three places, I've moved the variable > declarations to the innermost possible scope. Doing that revealed a > bunch of other, similar places where we can get rid of strlen() calls. > > Does this version seem like a good idea? Looks more extensive :-) On the quick glance, without a lot of testing it looks ok. But the lack of test cases stressing all different cases in that file, makes it impossible to say that there's no regressions. Personally I always feel uneasy making extensive changes in complicated code like this, without any integrated test case(s). -- GJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] unused code in float8_to_char , formatting.c ?
Hi Guys, Looking around the code Today, one of my helpful tools detected this dead code. As far as I can see, it is actually unused call to strlen() in formatting.c, float8_to_char(). Diff attached. formatting_dead_code.diff Description: Binary data -- GJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Proof of concept: using genetic algorithm to come up with most optimal PostgreSQL.conf
(Resending, I think google mail failed delivering it first time). Hi folks, I've always been fascinated with genetic algorithms. Having had a chance to implement it once before, to solve real life issue - I knew they can be brilliant at searching for right solutions in multi dimensional space. Thinking about just the postgresql.conf and number of possible options there to satisfy performance needs - I thought, this does sound like a good example of problem that can be solved using genetic algorithm. So I sat down after work for few days, and came up with a simple proof of concept. It generates random population of postgresql configuration files, and runs simple pgbench test on each one of them. It takes the average TPS for 3 consecutive runs as the score that then is applied to each 'guy'. Then I run a typical - I suppose - cross over operation and slight mutation of each new chromosome - to come up with new population, and so on and so forth. Running this for 2 days - I came up to conclusion that it does indeed seem to work, although default pgbench 'test cases' are not really stressing the database enough for it to generate diverse enough populations each time. Also, ideally this sort of thing should be run on two or more different hosts. One (master) that just generates new configurations, saves, restores, manages the whole operation - and 'slave' host(s) that run the actual tests. One benefit of that would be the fact that genetic algorithms are highly parallelizable. I did reboot my machines after tests couple times, to test configuration files and to see if the results were in fact repeatable (as much as they can be) - and I have to say, to my surprise - they were. I.e. the configuration files with poor results were still obviously slower then the best ones. I did include my sample results for everyone to see, including nice spreadsheet with graphs (everyone loves graphs) showing the scores across all populations. The tests were ran on my mac laptops (don’t have access to bunch of servers that I can test things like that on for couple days, sorry). The project, including readme file is available to look at:https://github.com/waniek/genpostgresql.conf Things I know so far: * I probably need to take into account more configuration options; * pgbench with its default test case is not the right characterization suite for this exercise, I need something more life like. I suppose we all have some sort of a characterization suite that could be used here; * Code needs a lot work on it, if this was to be used professionally; * Just restarting postgresql with different configuration file doesn't really constitute fully proper way to test new configuration files, but it seem to work; I don't expect much out of this - after all this is just a proof of concept. But if there are people out there thinking this can be in any way useful - please give us a shout. Also, if you know something more about genetic algorithms then I do - and can suggest improvement - let me know. Lastly, I'm looking for some more sophisticated pgbench test cases that I could throw in at it. I think in general pgbench as a project could use some more sophisticated benchmarks that should be included with the project, for everyone to see. Perhaps even to run some automated regression tests against git head. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] [proof of concept] Evolving postgresql.conf using genetic algorithm
Hi folks, I've always been fascinated with genetic algorithms. Having had a chance to implement it once before, to solve real life issue - I knew they can be brilliant at searching for right solutions in multi dimensional space. Thinking about just the postgresql.conf and number of possible options there to satisfy performance needs - I thought, this does sound like a good example of problem that can be solved using genetic algorithm. So I sat down after work for few days, and came up with a simple proof of concept. It generates random population of postgresql configuration files, and runs simple pgbench test on each one of them. It takes the average TPS for 3 consecutive runs as the score that then is applied to each 'guy'. Then I run a typical - I suppose - cross over operation and slight mutation of each new chromosome - to come up with new population, and so on and so forth. Running this for 2 days - I came up to conclusion that it does indeed seem to work, although default pgbench 'test cases' are not really stressing the database enough for it to generate diverse enough populations each time. Also, ideally this sort of thing should be run on two or more different hosts. One (master) that just generates new configurations, saves, restores, manages the whole operation - and 'slave' host(s) that run the actual tests. One benefit of that would be the fact that genetic algorithms are highly parallelizable. I did reboot my machines after tests couple times, to test configuration files and to see if the results were in fact repeatable (as much as they can be) - and I have to say, to my surprise - they were. I.e. the configuration files with poor results were still obviously slower then the best ones. I did include my sample results for everyone to see, including nice spreadsheet with graphs (everyone loves graphs) showing the scores across all populations. The tests were ran on my mac laptops (don’t have access to bunch of servers that I can test things like that on for couple days, sorry). The project, including readme file is available to look at: https://github.com/waniek/genpostgresql.conf Things I know so far: * I probably need to take into account more configuration options; * pgbench with its default test case is not the right characterization suite for this exercise, I need something more life like. I suppose we all have some sort of a characterization suite that could be used here; * Code needs a lot work on it, if this was to be used professionally; * Just restarting postgresql with different configuration file doesn't really constitute fully proper way to test new configuration files, but it seem to work; I don't expect much out of this - after all this is just a proof of concept. But if there are people out there thinking this can be in any way useful - please give us a shout. Also, if you know something more about genetic algorithms then I do - and can suggest improvement - let me know. Lastly, I'm looking for some more sophisticated pgbench test cases that I could throw in at it. I think in general pgbench as a project could use some more sophisticated benchmarks that should be included with the project, for everyone to see. Perhaps even to run some automated regression tests against git head. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Wasn't this more of an issue in de-coupling compression from encryption ? On 25 Jun 2012, at 16:36, Euler Taveira wrote: > On 24-06-2012 23:04, Robert Haas wrote: >> So I think we really >> need someone to try this both ways and compare. Right now it seems >> like we're mostly speculating on how well either approach would work, >> which is not as good as having some experimental results. >> > Not a problem. That's what I'm thinking too but I would like to make sure that > others don't object to general idea. Let me give it a try in both ideas... > > > -- > Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ > PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers > -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] autovacuum locks
On 2 Mar 2012, at 15:28, Robert Haas wrote: > On Fri, Mar 2, 2012 at 6:22 AM, Gregg Jaskiewicz wrote: >> Looking at the system bit more now, it look like 'waiting' states are >> changing for both the query and autovacuum in pg_stat_activity. >> But very slowly. It looks like they both got into that sort of state >> that keeps them on the edge of starvation. >> >> So this isn't really a deadlock, but rather very poor performance in >> this corner case. > > Are you making any use of cursors? nope. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Inlining comparators as a performance optimisation
On 28 Nov 2011, at 02:15, Peter Geoghegan wrote: > Attached are the results from performing a similar process to the > prior benchmark, but on Greg Smith's high-end server, and with an > orderlines table that has been "doubled-up" until it is 1538 MB, > making the same old query perform a quicksort that's over 3GB. Short > version: HEAD is 20468.0ms, with my patch it's 13689.698ms, so these > gains hold-up very well for very large in-memory sorts, possibly even > perfectly. > This is some really good stuff. Has to be said, that there must be quite few other places where inlining like that could have quite a positive benefit. But - I also have to say that both template functions and template classes in C++ give you pretty much the same speed improvement, with much better clarity and readability of the code. (I've tested it with the example Peter presented, if someone is interested in the code). One more reason why PostgreSQL project should look into the future and get some of the bits simplified and optimised by switching to C++. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Configuration include directory
On 16 Nov 2011, at 04:53, Greg Smith wrote: > > -Called by specifying "includedir ". No changes to the shipped > postgresql.conf yet. > -Takes an input directory name Very useful idea. What will happen if I specify: includedir './' Ie, what about potential cyclic dependency. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] const correctness
On 9 Nov 2011, at 15:33, Peter Geoghegan wrote: > On 9 November 2011 15:24, Tom Lane wrote:. >> If you go down this road you soon start needing duplicate functions >> for no other reason than that one takes/returns "const" and one doesn't. > > Why would you have to do that? > > To my mind, the fact that const "spreads" is a feature, not a deficiency. > +1. I would go as far as compiling most of my stuff using C++ compiler, because it is much more strict about const-correctness. (but then I have rule about making source files small). C compilers (and standard) allows you to do silly things like : char *foo = "barbar"; foo[1] = '4'; Not an option in C++ and if you use const correctness. I had few bugs like that in the past, where pointer was passed around (in C code), and one of the pointers was pointing to const string - but since compiler was fine with it... You know what happened. And that was on an embedded platform which made it even harder to trace down. The point is, const correctness is deeply unappreciated. Added bonus is the fact that compiler can make extra optimisations based on the const keyword. Kind of like read-only transactions in the database. Probably the most extreme and tedious way of introducing full const correctness in PostgreSQL would be to rename all files to .cpp, and let c++ compiler tell you what you need to fix. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] new warning
On 9 Nov 2011, at 16:16, Robert Haas wrote: > On Wed, Nov 9, 2011 at 11:12 AM, Kevin Grittner > wrote: >> This commit adds a new warning on my machine: >> >> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d326d9e8ea1d690cf6d968000efaa5121206d231 >> >> copy.c: In function *DoCopy*: >> copy.c:1861:14: warning: *bufferedTuples* may be used uninitialized >> in this function > > Yeah, I was just noticing that. It looks like it's just a case of the > compiler failing to see that there can't be a problem in real life, so > I pushed a fix to initialize it to NULL. > Yes, but one day someone will add some more code there - and will forget to initialise it to NULL.. Kabboom. ;) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.1.2 ?
On 9 Nov 2011, at 05:06, Magnus Hagander wrote: > I definitely think they are important enough to trigger a release. But as you > say, I think we need confirmation that they actually fix the problem... > Would you consider it a blocker for a rollout on production system ?
Re: [HACKERS] Materialized views
On 8 Nov 2011, at 21:23, Kevin Grittner wrote: > This is the time of year when the Wisconsin Courts formalize their > annual plan for where people will be spending the bulk of their time > in the coming year. Two years ago at this time, managers decided > that serializable transactions were a big enough issue to justify > assigning about half of my 2011 time to working on PostgreSQL > enhancements for that. This year our big database issue is > materialized views. +1 for that too. Too many hacked together MVs here, and more coming. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] 9.1.2 ?
Given the amount of fixes that went into the branch, and importance of them - when can we expect 9.1.2 to be released officially ? 9.1.1 was stamped on 22nd of September. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git trunk doesn't build
On 7 Nov 2011, at 15:44, Heikki Linnakangas wrote: > On 07.11.2011 14:01, Gregg Jaskiewicz wrote: >> On 7 November 2011 11:57, Heikki Linnakangas >> wrote: >> >>> Looks like the range types patch was broken for float timestamps. I'll go >>> fix that. Thanks for the report! >> >> yup, no probs. > > Fixed. Thank You. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug in walsender when calling out to do_pg_stop_backup (and others?)
On 19 Oct 2011, at 18:28, Florian Pflug wrote: > All the other flags which indicate cancellation reasons are set from signal > handers, I believe. We could of course mark as ClientConnectionLostPending as > volatile just to be consistent. Not sure whether that's a good idea, or not. > It might prevent a mistake should we ever add code to detect lost connections > asynchronously (i.e., from somewhere else than pq_flush). And the cost is > probably negligible, because CHECK_FOR_INTERRUPTS tests for InterruptPending > before calling ProcessInterrupts(), so we only pay the cost of volatile if > there's actually an interrupt pending. But I still think it's better to add > qualifies such a volatile only when really necessary. A comment about why it > *isn't* volatile is probably in order, though, so I'll add that in the next > version of the patch. > Makes sense. I had to ask, because it sticks out. And indeed there is a possibility that someone will one day will try to use from signal handler, etc. > best regards, > Florian Pflug > > PS: Thanks for the review. It's very much appreciated! No problem, Got inspired by the talk I attended here at the pgconf.eu. Seems like a good idea to do these things, I have years of experience as developer and doing (relatively) well thanks to PostgreSQL - makes sense to contribute something back. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug in walsender when calling out to do_pg_stop_backup (and others?)
On 15 Oct 2011, at 11:31, Florian Pflug wrote: > > Ok, here's a first cut. So I looked at the patch, and first thing that pops out, is lack of the volatile keyword before the ClientConnectionLostPending variable is defined. Is that done on purpose ? Is that on purpose ? Otherwise the patch itself looks ok. I haven't tested the code, just reviewed the patch itself. And it obviously needs testing, should be easy to follow your original problem description. Btw, I just tried to do it through commitfest.postgresql.org , but before I get my head around on how to add myself to the reviewer list there - I thought I'll just send this response here. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug in walsender when calling out to do_pg_stop_backup (and others?)
On 19 Oct 2011, at 17:54, Florian Pflug wrote: > On Oct19, 2011, at 17:47 , Greg Jaskiewicz wrote: >> On 15 Oct 2011, at 11:31, Florian Pflug wrote: >>> >>> Ok, here's a first cut. >> >> So I looked at the patch, and first thing that pops out, >> is lack of the volatile keyword before the ClientConnectionLostPending >> variable is defined. Is that done on purpose ? Is that on purpose ? > > That's on purpose. volatile is only necessary for variables which are either > accessed from within signal handlers or which live in shared memory. Neither > is true for ClientConnectionLostPending, so non-volatile should be fine. Ok, cool. I'm aware of the reasons behind volatile, just noticed that some other flags used in similar way are marked as such. At the end of the day, this is just a hint to the compiler anyway. > >> Otherwise the patch itself looks ok. >> I haven't tested the code, just reviewed the patch itself. And it obviously >> needs testing, should be easy to follow your original problem description. > > Yeah, further testing is on my todo list. The interesting case is probably > what happens if the connection is dropped while there's already a > cancellation request pending. And also the other way around - a cancellation > request arriving after we've already discovered that the connection is gone. > >> Btw, I just tried to do it through commitfest.postgresql.org , but before I >> get my head around on how to add myself to the reviewer list there - I >> thought I'll just send this response here. > > You just need to click "Edit Patch" and put your name into the Reviewer > field. You do need a postgres community account to edit patches, but the > signup process is quite quick and painless AFAIR. Ok, clicking 'edit patch' sounds a bit big. Probably better would be, to just be able to click on some sort of "I'm in" button/checkbox type of thing. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
