Re: [HACKERS] Changing pg_dump default file format
On 11/07/2013 10:00 AM, Josh Berkus wrote: On 11/07/2013 08:26 AM, Tom Lane wrote: Magnus Hagander mag...@hagander.net writes: On Thu, Nov 7, 2013 at 5:02 PM, Joshua D. Drake j...@commandprompt.com wrote: I humbly request on behalf of those who manage production postgresql instances that we change the default backup file format from -Fp to -Fc. I'm a bit worried about it breaking peoples scripts as well, if they pipe the output. Whether they pipe the output or not, this will break people's scripts. I agree it's a legacy behavior that we wouldn't choose again, but you could say that about an awful lot of things including much of the SQL standard. I think it's too late to consider this now. I don't agree that it's too late, personally. However, I do agree that it would be problematic to change the default format since the default is to send to STDOUT. If we wanted to change the default format, we would need to make filename a required parameter. If we wanted to change the defaults, I think it would be easier to create a separate bin name (e.g. pg_backup) than to change the existing parameters for pg_dump. I am not opposed to that. Allow pg_dump to be what it is, and create a pg_backup? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Changing pg_dump default file format
On 11/07/2013 11:01 AM, Alvaro Herrera wrote: Maybe we could provide a master controller program (pg_backup or whatever name we agree on), which could receive commands much like pg_ctl. $ pg_backup --help pg_backup is a backup handler program for PostgreSQL. Usage: pg_backup backup [-p PATTERN] DATABASE [...] pg_backup restore [-f PATH] pg_backup backup_globals ... The pattern stuff would let you specify many databases in the command line and have it dump each DB on a separate file, and perhaps do strftime-style replacements, etc. Initially we could have this just call pg_dump underneath, but eventually it might grow superpowers of its own. +1 -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Changing pg_dump default file format
On 11/07/2013 10:54 AM, Kevin Grittner wrote: That is who I am thinking of. A DBA team may have hundreds of databases to manage, each with many scripts which have been running nicely for years. A change like this is bound to break some of those crontab scripts they may not even remember they are running -- like ones which dump a key table to INSERT statements to feed into some other database product, which will now choke when it gets a custom format file instead of the text file it has been getting for years. Requiring the DBA team to track all these scripts down to add -Fp would be annoying, to put it mildly. Only because they don't take the time to properly document or put into some form of automation/tracking/configfile management mechanism. Don't get me wrong, I would be in the same boat but I don't think it is realistic to manage the communities expectations with the lackluster operations performance of fellow DBAs. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Changing pg_dump default file format
On 11/07/2013 12:42 PM, Josh Berkus wrote: All, I'm amused at how this has become a let's pile on everything which has ever been missing in pg_dump into one thread. Well it has been broken longer than most of our utilities. Sorry... not broken but certainly not complete. It is to be expected. Agree with Tom that if we're going to create a new program name, we should fix the pg_dumpall issue as well. All I want to start is this simple fix. I don't know who is going to step up to work on pg_dump/all considering it is the bastard step child of the project. We only work on it when we absolutely have to (and CMD is just as guilty as the rest). JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Commitfest II CLosed
On 10/21/2013 08:11 AM, Robert Haas wrote: Supposedly, we have a policy that for each patch you submit, you ought to review a patch. That right there ought to provide enough reviewers for all the patches, but clearly it didn't. And I'm pretty sure that some people (like me) looked at a lot MORE patches than they themselves submitted. I think auditing who is not contributing in that area and finding tactful ways to encourage them to contribute would be a very useful service to the project. What if as part of the patch submission process you had to pick the patch you were going to review? If there are no patches to review, then we obviously don't have a problem. If there are patches to review then we are all set. I guess there is the problem of there only being patches that a submitter is not qualified to review but I find that miniscule as every person on this list (myself included) can do a cursory review (patch applies, docs are good, indentation is appropriate, works as advertised). The commitfest app would have to be modified for this but what do people think? Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Auto-tuning work_mem and maintenance_work_mem
On 10/17/2013 08:55 AM, Kevin Grittner wrote: Robert Haas robertmh...@gmail.com wrote: I still think my previous proposal of increasing the defaults for work_mem and maintenance_work_mem by 4X would serve many more people well than it would serve poorly. I haven't heard anyone disagree with that notion. Does anyone disagree? Should we do it? I think that it makes sense to do that. Those are still reasonable defaults for a machine with 2GB of RAM, maybe even with less. We're talking about putting this only in a release that will come out in 2014. How many machines used for a database server that new will have less than that? A lot. A whole lot, more than what most people have in production with more than that. You are forgetting a very large segment of the population who run... VMs. Why don't we just have 3 default config files: 2GB memory 4GB memory 8GB memory Have initdb detect how much memory is available on the machine in TOTAL and pick the most appropriate. Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Auto-tuning work_mem and maintenance_work_mem
On 10/17/2013 09:49 AM, Robert Haas wrote: A lot. A whole lot, more than what most people have in production with more than that. You are forgetting a very large segment of the population who run... VMs. That's true, but are you actually arguing for keeping work_mem at 1MB? Even on a VM with only 1GB of RAM, work_mem=4MB is not going to cause any problems unless you're also trying to service a large number of simultaneous connections. And if you're doing that, you probably need to rethink something anyway. No. I am arguing for the multiple config file option. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Auto-tuning work_mem and maintenance_work_mem
On 10/17/2013 10:33 AM, Jeff Janes wrote: A lot. A whole lot, more than what most people have in production with more than that. You are forgetting a very large segment of the population who run... VMs. Why don't we just have 3 default config files: 2GB memory 4GB memory 8GB memory But what would go in each of those files? Once we agree on what would be in them, why not just have a continuous knob that does that same thing? Because we should set defaults, not optimized parameters. Workloads vary and we can reasonably say this is what we want BY DEFAULT for something but we can not reasonably say, this is what will suit your needs. Once you get above 8GB of memory you are dealing with workloads that vary widely and will almost always need some kind of indvidual attention. However, 8GB and below, we can set reasonable defaults that allow a user to likely but possibly not worry about changing the conf. Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Auto-tuning work_mem and maintenance_work_mem
On 10/09/2013 07:58 AM, Bruce Momjian wrote: But it still is an independent parameter. I am just changing the default. maintenance_work_mem can depend on work_mem ~ work_mem * 1 * max_connection / 4 That is kind of hard to do because we would have to figure out if the old maintenance_work_mem was set from a default computation or by the user. FYI, this auto-tuning is not for us, who understand the parameters and how they interact, but for the 90% of our users who would benefit from better defaults. It is true that there might now be cases where you would need to _reduce_ work_mem from its default, but I think the new computed default will be better for most users. Just to step in here as a consultant. Bruce is right on here. Autotuning has nothing to do with us, it has to do with Rails developers who deploy PostgreSQL and known nothing of it except what ActiveRecord tells them (I am not being rude here). We could argue all day what the best equation is for this, the key is to pick something reasonable, not perfect. Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Could ANALYZE estimate bloat?
On 09/20/2013 11:59 AM, Josh Berkus wrote: Hackers, I've been tinkering with a number of table bloat checks, and it occurred to me that the problm is that these are all approximations based on overall gross statistics, and as such highly inaccurate. It seems like would could have ANALYZE, while sampling from the table, also check the amount of dead space per page and use that as an estimate of the % of dead space overall. We'd still need something else for indexes, but this seems like it would be a good start. No? I think this is a great idea. -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] get rid of SQL_ASCII?
On 09/05/2013 09:42 AM, Josh Berkus wrote: Peter, Other ideas? Are there legitimate uses for SQL_ASCII? Migrating from MySQL. We've had some projects where we couldn't fix MySQL's non-enforcement text garbage, and had to use SQL_ASCII on the receiving side. If it hadn't been available, the user would have given up on Postgres. iconv? -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Kudos for Reviewers -- wrapping it up
On 08/07/2013 10:35 AM, Bruce Momjian wrote: Actually, for me, motiving reviewers seems like the Lemon-Soaked Paper Napkins, as it requires unbounded effort and its importance is not being balanced with other priorities. Ignoring the non-productive part of this thread, I would like to mention that motivating reviewers is not necessarily complicated. We just have to ask ourselves what motivates a person: The feeling that their work is worthwhile, productive, will be appreciated and that they will receive recognition for the effort. Right now, we do not publicly outside of the dome that is -hackers provide those incentives. Give reviewers the just recognition they deserve and I believe we will see more reviewing effort. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Autovacuum different in 9.2.4?
Hello, I seem to recall autovacuum changes landing for 9.2.4. Can someone please describe what those changes were and how they could affect usage? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] don't own lock of type?
Hello, What exactly causes this? WARNING: you don't own a lock of type ExclusiveLock Does this mean the user calling the lock doesn't own the object? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Autovacuum different in 9.2.4?
On 08/05/2013 12:13 PM, Jeff Janes wrote: On Mon, Aug 5, 2013 at 11:16 AM, Joshua D. Drake j...@commandprompt.com wrote: Hello, I seem to recall autovacuum changes landing for 9.2.4. Can someone please describe what those changes were and how they could affect usage? Those landed in 9.2.3, see release notes for that version: Fix performance problems with autovacuum truncation in busy workloads (Jan Wieck) Fix error in vacuum_freeze_table_age implementation (Andres Freund) There should be no change in usage, unless you were taking some heroic methods to overcome the problems and can now discontinue them. That is what is confusing me, I could be cracked but messages like these: automatic vacuum of table pg_catalog.pg_attribute: could not (re)acquire exclusive lock for truncate scan Seem to be new? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] don't own lock of type?
On 08/05/2013 11:38 AM, Robert Haas wrote: On Mon, Aug 5, 2013 at 2:32 PM, Joshua D. Drake j...@commandprompt.com wrote: Hello, What exactly causes this? WARNING: you don't own a lock of type ExclusiveLock Does this mean the user calling the lock doesn't own the object? It means there's a bug. Either in PostgreSQL, or some loadable module you're using. I am getting this rather frequently, I will check modules but I don't believe we are running any. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] don't own lock of type?
On 08/05/2013 02:58 PM, Robert Haas wrote: On Mon, Aug 5, 2013 at 3:37 PM, Joshua D. Drake j...@commandprompt.com wrote: It means there's a bug. Either in PostgreSQL, or some loadable module you're using. I am getting this rather frequently, I will check modules but I don't believe we are running any. We fixed a bug in this area a while back. Are you running the latest point release? 9.2.4 so yep. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Kudos for Reviewers -- wrapping it up
On 07/12/2013 10:49 AM, Andrew Dunstan wrote: On 07/12/2013 01:28 PM, Alvaro Herrera wrote: Josh Berkus wrote: -- a couple of compromise proposals were made: a) that reviewers who do actual code modification of the patch get credited on the feature, and those who just review it get credited at the bottom of the release notes, or I'd probably say substantial or non-trivial, but otherwise +1 Right cause if a reviewer ends up writing (or cleaning up) all the docs, I would say they deserve very close to equal credit. As an example. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] --with-libedit-preferred is bad design
On 7/12/2013 7:10 PM, Josh Berkus wrote: That would hardly be only true of libedit, on Apple. It's also broken on some Red Hat versions, last I checked. Last I heard, libedit was completely borked. Here is a report (two years old) of still broken libedit in Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608442 IMO, we either need to take up the mantle of libedit maintenance or we need to remove it from a configure option, it doesn't work anyway. JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Millisecond-precision connect_timeout for libpq
On 7/5/2013 1:01 PM, Josh Berkus wrote: If you are issuing a fresh connection for each sub-100ms query, you're doing it wrong anyway ... It's fairly common with certain kinds of apps, including Rails and PHP. This is one of the reasons why we've discussed having a kind of stripped-down version of pgbouncer built into Postgres as a connection manager. If it weren't valuable to be able to relocate pgbouncer to other hosts, I'd still say that was a good idea. No kidding. I think a lot of -hackers forget that the web rules here and the web is stateless, which means a huge performance loss for postgresql unless we add yet another piece of software. Pre-forking here would really help us. JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Improvement of checkpoint IO scheduler for stable transaction responses
On 07/04/2013 06:05 AM, Andres Freund wrote: Presumably the smaller segsize is better because we don't completely stall the system by submitting up to 1GB of io at once. So, if we were to do it in 32MB chunks and then do a final fsync() afterwards we might get most of the benefits. Yes, I try to test this setting './configure --with-segsize=0.03125' tonight. I will send you this test result tomorrow. I did testing on this a few years ago, I tried with 2MB segments over 16MB thinking similarly to you. It failed miserably, performance completely tanked. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Documentation/help for materialized and recursive views
On 07/01/2013 07:20 AM, David Fetter wrote: On Mon, Jul 01, 2013 at 10:05:24AM -0400, Peter Eisentraut wrote: On 6/28/13 2:27 PM, David Fetter wrote: You can run \! man from within psql, And if you're on Windows, you're Sadly Out of Luck with that. Is there an equivalent we could #ifdef in for that platform? If you are using psql on Windows extensively, you probably have one of mingw, cygwin, or pgadmin handy, all of which can get you to the documentation. I don't think it's worth devising a mechanism for those not covered by this. With deepest respect, failing to provide documentation to users on our widest-deployed platform seems pretty hostile to me. There was an earlier suggestion that we provide URLs, which seems like a decent way forward as those environments so locked down as to disallow outbound HTTP are pretty rare, and non-networked computers are even more rare. Although I agree with the sentiment the idea that postgres more widely deployed on windows than other platforms is rather laughable. The only metrics we have are downloads which doesn't count cause linux ships with postgres with a simple yum or apt-get. Whatever solution we decide, we should not push this responsibility off on pgadmin as pgadmin is not part of PostgreSQL but a third party tool. The standard postgresql client is psql (for good or bad) and we should support psql fully on all platforms. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] vacuumlo - use a cursor
On 06/29/2013 08:35 AM, Bruce Momjian wrote: On Sat, Jun 29, 2013 at 11:33:54AM -0400, Andrew Dunstan wrote: Nobody seemed interested. But I do think it's a good idea still. Well, if no one replied, and you thought it was a good idea, then it was a good idea. ;-) I think it is a good idea just of limited use. I only have one customer that still uses large objects. Which is a shame really as they are more efficient that bytea. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C++ compiler
On 06/24/2013 09:16 PM, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: Right. I don't think there are any C features we want to avoid; are there any? We're avoiding C99-and-later features that are not in C89, such as // for comments, as well as more useful things. It might be time to reconsider whether we should move the baseline portability requirement up to C99. The problem here is we lose the MS compilers which are not being updated for C99. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Kudos for Reviewers -- straw poll
On 06/25/2013 10:17 AM, Josh Berkus wrote: Hackers, I'd like to take a straw poll here on how we should acknowledge reviewers. Please answer the below with your thoughts, either on-list or via private email. How should reviewers get credited in the release notes? a) not at all b) in a single block titled Reviewers for this version at the bottom. c) on the patch they reviewed, for each patch C. The idea that reviewers are somehow less than authors is rather disheartening. Should there be a criteria for a creditable review? a) no, all reviews are worthwhile b) yes, they have to do more than it compiles c) yes, only code reviews should count B. I think it compiles, and I tested it via X should be the minimum. Here is a case. I was considering taking a review of the new Gin Cache patch. I can't really do a code review but I can certainly run benchmarking tests before/after and apply the patch etc. Should reviewers for 9.4 get a prize, such as a t-shirt, as a promotion to increase the number of non-submitter reviewers? a) yes b) no c) yes, but submitters and committers should get it too Thanks for your feedback! B. We already give them a multi-million dollar piece of software for free. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Kudos for Reviewers -- straw poll
On 06/25/2013 11:26 AM, Andres Freund wrote: On 2013-06-25 11:04:38 -0700, Joshua D. Drake wrote: a) not at all b) in a single block titled Reviewers for this version at the bottom. c) on the patch they reviewed, for each patch C. The idea that reviewers are somehow less than authors is rather disheartening. It's not about the reviewers being less. It's a comparison of effort. The effort for a casual review simply isn't comparable with the effort spent on developing a nontrivial patch. I think this is a backwards way to look at it. The effort may not be comparable but the drudgery certainly is. Reviewing patches sucks. Writing patches (for the most part) is fun. Should the patch submitter get first billing? Yes. Should the reviewer that makes sure to a reasonable level that the patch is sane also get billing? Absolutely. Should the reviewer get billing that is about the patch they reviewed. Yes. As I mentioned before in the release notes something like: Author: Tom Lane Reviewer(s): Greg Stark, Andrew Dunstan I think that is perfectly reasonable. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List
On 06/24/2013 08:40 AM, Maciej Gajewski wrote: Maybe this policy should be mentioned on the Wiki, so newbies like myself (who wouldn't even dare reviewing patches submitted be seasoned hackers) are not surprised by seeing own name on a shame wall? It is mentioned. Of course now I can't find it but it is there. However, I believe you are taking the wrong perspective on this. This is not a shame wall. It is a transparent reminder of the policy and those who have not assisted in reviewing a patch but have submitted a patch themselves. In short, leave the ego at the door. Sincerely, JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List
On 06/24/2013 10:10 AM, Josh Berkus wrote: On 06/24/2013 10:02 AM, Dimitri Fontaine wrote: Josh Berkus j...@agliodbs.com writes: patch. The vast majority chose not to respond to my email to them at all. When private email fails, the next step is public email. The only problem I have here is that I don't remember about deciding to publish a list of failures by public email at all. I hope it's not my memory failing me here, because then I would have to remember why I didn't speak up against that idea at the time. You didn't decide anything. As the CFM, I did. My job for this month is to make sure that 100% of patches in that queue get reviewed and either committed or bounced by July 15th. I'm doing my job. I will be more than happy to resign as CFM and turn it over to someone else if people have a problem with it. Let's not be hasty :) I think JoshB is spot on in this. He sent previous private emails, and a week later opened up the transparency so that everyone understood what was going on. What I find unfortunate is people are spending a bunch of time on this argument which has been clearl thought out by Josh instead of reviewing patches. I repeat: Leave your ego at the door. Josh is doing what could be considered one of the most thankless (public) jobs in this project. How about we support him in getting these patches taken care of instead of whining about the fact that he called us out for not doing our jobs (reviewing patches) in the first place. Sincerely, Joshua D. Drkae -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List
On 06/24/2013 10:22 AM, Josh Berkus wrote: Mind you, we wouldn't be able to reward a few reviewers, because they live in countries to which it's impossible to ship from abroad. I have previously proposed that all of the reviewers of a given PostgreSQL release be honored in the release notes as a positive incentive, and was denied on this from doing so. Not coincidentally, we don't seem to have any reviewers-at-large anymore. I don't like idea of sending gifts. I do like the idea of public thanks. We should put full recognition in the release notes for someone who reviews a patch. If they didn't review the patch, the person that wrote the patch would not have gotten the patch committed anyway. Writing the patch is only have the battle. Heck, think about the FKLocks patch, Alvaro wrote that patch but I know that Noah (as well as others) put a herculean effort into helping get it committed. Reviewer recognition should be on the same level as the submitter. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List
On 06/24/2013 10:48 AM, Claudio Freire wrote: Reviewer recognition should be on the same level as the submitter. The problem with that is that that HUGELY depends on the patch and the review. There are patches where reviewers do a good percentage of the work and others where they mostly tell that compiles runs. Well, you can't so arbitrarily pick who you're recognizing as contributor and who you aren't. So why not mention them all? They did work for it, some more than others, but they all worked. And since whoever submitted a patch (and got it committed) must have reviewed something as well, they'd be recognized for both reviewing and submitting. Exactly. Just make it a simple policy: Submitters and Reviewers are listed in that order: Submitter, reviewer, reviewer That way submitter gets first bill, satisfying the ego (as well as professional consideration) but reviewers are also fully recognized. Sincerely, Joshua D. Drkae -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List
On 06/24/2013 10:59 AM, Andres Freund wrote: On 2013-06-24 10:50:42 -0700, Josh Berkus wrote: The problem with that is that that HUGELY depends on the patch and the review. There are patches where reviewers do a good percentage of the work and others where they mostly tell that compiles runs. This project is enormously stingy with giving credit to people. It's not like it costs us money, you know. Listing a reviewer that didn't do all that much at the same level as the author or an somebody having done an extensive review will cost you contributors in the long run. I am all for introducing a Contributed by reviewing: ... section in the release notes. It should be listed with the specific feature. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C++ compiler
On 06/24/2013 04:59 PM, Bruce Momjian wrote: On Thu, Jun 20, 2013 at 12:45:48PM +0800, Craig Ringer wrote: I see value in making the codebase compileable with g++... and down the track I can see being able to use basic class features as quite useful given Pg's fairly OO internal design. Inline template functions instead of macros would be nice. I've worked with C++ enough not to be overly excited by the idea of going down the rabbit hole of Boost, complex template arrangements, etc, and I don't love the STL ... but any sensible project adopting C++ carefully picks and chooses what features it uses and how. I think the big question is whether you can _control_ what C++ features are used, or whether you are perpetually instructing users what C++ features not to use. How is that different than us having to do the same with C? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] C++ compiler
On 06/24/2013 05:37 PM, Bruce Momjian wrote: On Mon, Jun 24, 2013 at 09:21:26PM -0300, Claudio Freire wrote: On Mon, Jun 24, 2013 at 9:19 PM, Joshua D. Drake j...@commandprompt.com wrote: I think the big question is whether you can _control_ what C++ features are used, or whether you are perpetually instructing users what C++ features not to use. How is that different than us having to do the same with C? Perhaps the size of C++ ? Right. I don't think there are any C features we want to avoid; are there any? Anything supported by C99 and not other versions I would say. However, my point is if done correctly we would state which features ahead of time we are willing to use and make them part of the developer faq? Sincerely, JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hardware donation
On 06/21/2013 09:48 AM, Jim Nasby wrote: We've got some recently decommissioned servers and Enova is willing to donate 2 of them to the community. There's nothing terribly spectacular about the servers except for memory. We have one 512G server available and the other would be either 192G or 96G. I know that folks already have access to machines with a lot of cores, but I haven't seen reports of large memory machines. CPU details vary but we're only looking at 20ish cores (though AFAIK they're all 4 socket servers if that matters). Local drives are nothing fancy (though some might possibly be SSD). A couple 192G machines to put in the performance lab would be nice, especially if they have SSD. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bad error message on valuntil
On 06/19/2013 08:24 AM, Peter Eisentraut wrote: I think it's intentional that we don't tell the *client* that level of detail. I could see emitting a log message about it, but it's not clear whether that will help an unsophisticated user. Usually, when I log in somewhere and the password is expired, it tells me that the password is expired. I don't think we gain anything by hiding that from the user. FTR: there is an actual patch for this sitting over at the, Change authentication error message thread. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Change authentication error message (patch)
On 06/18/2013 02:25 AM, Markus Wanner wrote: On 06/16/2013 06:02 PM, Joshua D. Drake wrote: Instead of pushing extra info to the logs I decided that we could without giving away extra details per policy. I wrote the error message in a way that tells the most obvious problems, without admitting to any of them. Please see attached: +1 for solving this with a bit of word-smithing. However, the proposed wording doesn't sound like a full sentence to my ears, because a password or username cannot fail per-se. I believe it actually can. The error message that is returned for a bad password, bad user or expired password is all the same. Which is why I put the username in there. How about: password authentication failed or account expired for user \%s\ It's a bit longer, but sounds more like a full sentence, no? Yes but I don't think it is accurate, what about: Authentication failed or password has expired for user \%s\ Authentication failed covers any combination of a username/password being wrong and obviously password expired covers the other. Sincerely, Joshua D. Drake Regards Markus Wanner -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Change authentication error message (patch)
On 06/19/2013 01:18 PM, Markus Wanner wrote: Authentication failed or password has expired for user \%s\ Authentication failed covers any combination of a username/password being wrong and obviously password expired covers the other. Works for me. Considering the password to be the thing that expires (rather than the account) is probably more accurate as well. It is also how it is worded in the docs (which is why I used it). Patch below. JD diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c index 415b614..f129fe1 100644 --- a/src/backend/libpq/auth.c +++ b/src/backend/libpq/auth.c @@ -270,7 +270,7 @@ auth_failed(Port *port, int status) break; case uaPassword: case uaMD5: - errstr = gettext_noop(password authentication failed for user \%s\); + errstr = gettext_noop(Authentication failed or password has expired for user \%s\); /* We use it to indicate if a .pgpass password failed. */ errcode_return = ERRCODE_INVALID_PASSWORD; break; -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Change authentication error message (patch)
Hello, Instead of pushing extra info to the logs I decided that we could without giving away extra details per policy. I wrote the error message in a way that tells the most obvious problems, without admitting to any of them. Please see attached: diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c index 415b614..a775534 100644 --- a/src/backend/libpq/auth.c +++ b/src/backend/libpq/auth.c @@ -270,7 +270,7 @@ auth_failed(Port *port, int status) break; case uaPassword: case uaMD5: - errstr = gettext_noop(password authentication failed for user \%s\); + errstr = gettext_noop(password, username or password expiry failed for user \%s\); /* We use it to indicate if a .pgpass password failed. */ errcode_return = ERRCODE_INVALID_PASSWORD; break; -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hard to Use WAS: Hard limit on WAL space
On 06/14/2013 11:18 PM, Craig Ringer wrote: On 06/15/2013 02:08 PM, Brendan Jurd wrote: On 15 June 2013 14:43, Craig Ringer cr...@2ndquadrant.com wrote: The #1 question I see on Stack Overflow has to be confusion about pg_hba.conf, mostly from people who have no idea it exists, don't understand how to configure it, etc. The totally non-obvious name of the file probably has something to do with that. It should be called 'auth.conf'. Not convinced; since it only controls one facet of auth - it doesn't define users, passwords, grants, etc - that'd probably be just as confusing. Yeah this one is not making the grade. pg_hba is just that host based auth but I think we are bikeshedding now. JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hard to Use WAS: Hard limit on WAL space
On 06/14/2013 11:44 PM, Brendan Jurd wrote: If they see something called 'pg_hba.conf', they may very reasonably assume that it is some internal/advanced stuff that they don't need to worry about just yet, because what the heck is a 'pg_hba'? The 'pg' Only the uneducated. Look, I am not trying to be an ass but seriously. Read the docs. I will argue vigorously against the idea of us designing a system that has people NOT reading the docs. JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] request a new feature in fuzzystrmatch
On 06/14/2013 10:11 AM, David Fetter wrote: ok, thanks, I will wait. Hi Joe, Do you have some time in the weekend to help me submit the patch? Thanks, Liming Liming, Is your git skill good enough to create a patch vs. PostgreSQL's git master? If so, send that and once it's hit the mailing list, record same on commitfest.postgresql.org in the current open commitfest. If not, let us know where in that process you got stuck. Cheers, David. This sounds like a wiki page FAQ in the making. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] another error perhaps to be enhanced
ERROR: index foo_idx We should probably add the schema. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] another error perhaps to be enhanced
On 06/14/2013 10:47 AM, Peter Geoghegan wrote: I think you'll need to better describe what you mean here. postgres=# create schema foo; CREATE SCHEMA postgres=# create schema bar; CREATE SCHEMA postgres=# create table foo.foo(id serial); NOTICE: CREATE TABLE will create implicit sequence foo_id_seq for serial column foo.id CREATE TABLE postgres=# create table bar.bar(id serial); NOTICE: CREATE TABLE will create implicit sequence bar_id_seq for serial column bar.id CREATE TABLE postgres=# create index one_idx on foo.foo(id); CREATE INDEX postgres=# create index one_idx on bar.bar(id); CREATE INDEX postgres=# Now, with the error previously shown, which one_idx needs to be reindexed? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] another error perhaps to be enhanced
On 06/14/2013 11:01 AM, Peter Geoghegan wrote: On Fri, Jun 14, 2013 at 10:54 AM, Joshua D. Drake j...@commandprompt.com wrote: Now, with the error previously shown, which one_idx needs to be reindexed? Well, you didn't show an actual error message. ERROR: index foo_idx Is not an error message? Granted I didn't show the whole error message but my point is, it should ALWAYS be fully qualified. But if you \set VERBOSITY verbose within psql while connected to a 9.3 server, you'll get fully qualified details of the constraint blamed for the error, if any. Example: postgres=# insert into a(a, b) values (3, 'test'); ERROR: 23505: duplicate key value violates unique constraint a_pkey DETAIL: Key (a)=(3) already exists. SCHEMA NAME: public TABLE NAME: a CONSTRAINT NAME: a_pkey LOCATION: _bt_check_unique, nbtinsert.c:398 I was looking in the logs. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hard to Use WAS: Hard limit on WAL space
On 06/14/2013 11:16 AM, Josh Berkus wrote: On 06/12/2013 02:03 PM, Joshua D. Drake wrote: What concerns me is we seem to be trying to make this easy. It isn't supposed to be easy. This is hard stuff. Smart people built it and it takes a smart person to run it. When did it become a bad thing to be something that smart people need to run? 1997, last I checked. Our unofficial motto: PostgreSQL: making very hard things possible, and simple things hard. It *is* hard. But that's because we've *made* it hard to understand and manage, not because the problem is inherently hard. For example: can you explain to me in 10 words or less how to monitor to see if archiving is falling behind? I'll bet you can't, and that's because we've provided no reliable way to do so. Hey, I never said we shouldn't have a complete feature set. I agree with you. IMO it should not have even been committed without the ability to actually know what is going on and we have had it since (in theory) 8.1? My primary concern is: Don't make it stupid. I liked Claudio's comment, More than easy, it should be obvious.. It should be obvious from a review of the documentation how to manage this stuff. It isn't, and worse even if we wrote the documentation it still isn't because the feature is not complete. With great power comes great responsibility :P JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pluggable compression support
On 06/14/2013 06:56 PM, Robert Haas wrote: On Fri, Jun 14, 2013 at 8:45 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-14 17:35:02 -0700, Josh Berkus wrote: No. I think as long as we only have pglz and one new algorithm (even if that is lz4 instead of the current snappy) we should just always use the new algorithm. Unless I missed it nobody seemed to have voiced a contrary position? For testing/evaluation the guc seems to be sufficient. Then it's not pluggable, is it? It's upgradable compression support, if anything. Which is fine, but let's not confuse people. The point is that it's pluggable on the storage level in the sense of that several different algorithms can coexist and new ones can relatively easily added. That part is what seems to have blocked progress for quite a while now. So fixing that seems to be the interesting thing. I am happy enough to do the work of making it configurable if we want it to be... But I have zap interest of doing it and throw it away in the end because we decide we don't need it. I don't think we need it. I think what we need is to decide is which algorithm is legally OK to use. And then put it in. In the past, we've had a great deal of speculation about that legal question from people who are not lawyers. Maybe it would be valuable to get some opinions from people who ARE lawyers. Tom and Heikki both work for real big companies which, I'm guessing, have substantial legal departments; perhaps they could pursue getting the algorithms of possible interest vetted. Or, I could try to find out whether it's possible do something similar through EnterpriseDB. We have IP legal representation through Software in the Public interest who pretty much specializes in this type of thing. Should I follow up? If so, I need a summary of the exact question including licenses etc. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)
On 06/12/2013 08:49 AM, Robert Haas wrote: Sure, remote archiving is great, and I'm glad you've been working on it. In general, I think that's a cleaner approach, but there are still enough people using archive_command that we can't throw them under the bus. Correct. I guess archiving to a nfs mount or so isn't too bad, but archiving and using a cronjob to get the files off is typically a great way to loose data, and we really shouldn't encourage that by default, Imo. We certainly not by default but it is also something that can be easy to set up reliably if you know what you are doing. Well, I think what we're encouraging right now is for people to do it wrong. The proliferation of complex tools to manage this process suggests that it is not easy to manage without a complex tool. No. It suggests that people have more than one requirement that the project WILL NEVER be able to solve. Granted we have solved some of them, for example pg_basebackup. However, pg_basebackup isn't really useful for a large database. Multithreaded rsync is much more efficient. That's a problem. And we regularly have users who discover, under a variety of circumstances, that they've been doing it wrong. If there's a better solution than hard-wiring some smarts about local directories, I'm all ears - but making the simple case just work would still be better than doing nothing. Agreed. Right now you have to be a rocket scientist no matter what configuration you're running. This is quite a bit overblown. Assuming your needs are simple. Archiving is at it is now, a relatively simple process to set up, even without something like PITRTools. Where we run into trouble is when they aren't and that is ok because we can't solve every problem. We can only provide tools for others to solve their particular issue. What concerns me is we seem to be trying to make this easy. It isn't supposed to be easy. This is hard stuff. Smart people built it and it takes a smart person to run it. When did it become a bad thing to be something that smart people need to run? Yes, we need to make it reliable. We don't need to be the Nanny database. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)
On 06/10/2013 04:42 PM, Josh Berkus wrote: Actually we describe what archive_command needs to fulfill, and tell them to use something that accomplishes that. The example with cp is explicitly given as an example, not a recommendation. If we offer cp as an example, we *are* recommending it. If we don't recommend it, we shouldn't have it as an example. In fact, if we don't recommend cp, then PostgreSQL should ship with some example shell scripts for archive commands, just as we do for init scripts. Not a bad idea. One that supports rsync and another that supports robocopy. That should cover every platform we support. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] small patch to crypt.c
On 06/09/2013 09:28 AM, Tom Lane wrote: Even aside from that, the proposed change seems like a bad idea because it introduces an unnecessary call of GetCurrentTimestamp() in the common case where there's no valuntil limit. On some platforms that call is pretty slow. And that would explain why we don't do something like this: index f01d904..4935c9f 100644 --- a/src/backend/libpq/crypt.c +++ b/src/backend/libpq/crypt.c @@ -145,12 +145,10 @@ md5_crypt_verify(const Port *port, const char *role, char *client_pass) /* * Password OK, now check to be sure we are not past rolvaliduntil */ - if (isnull) + if (isnull || vuntil GetCurrentTimestamp()) retval = STATUS_OK; - else if (vuntil GetCurrentTimestamp()) - retval = STATUS_ERROR; else - retval = STATUS_OK; + retval = STATUS_ERROR; } Right. Ty for the feedback, I know it was just a little bit of code but it just looked off and I appreciate the explanation. JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)
On 06/07/2013 12:14 PM, Josh Berkus wrote: Right now, what we're telling users is You can have continuous backup with Postgres, but you'd better hire and expensive consultant to set it up for you, or use this external tool of dubious provenance which there's no packages for, or you might accidentally cause your database to shut down in the middle of the night. This is an outright falsehood. We are telling them, You better know what you are doing or You should call a consultant. This is no different than, You better know what you are doing or You should take driving lessons. At which point most sensible users say no thanks, I'll use something else. Josh I have always admired your flair for dramatics, it almost rivals mine. Users are free to use what they want, some will chose lesser databases. I am ok with that because eventually if PostgreSQL is the right tool, they will come back to us, and PgExperts or CMD or OmniTI or they will know what they are doing and thus don't need us. JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)
On 06/08/2013 07:36 AM, MauMau wrote: 1. If the machine or postgres crashes while archive_command is copying a WAL file, later archive recovery fails. This is because cp leaves a file of less than 16MB in archive area, and postgres refuses to start when it finds such a small archive WAL file. The solution, which IIRC Tomas san told me here, is to do like cp %p /archive/dir/%f.tmp mv /archive/dir/%f.tmp /archive/dir/%f. Well it seems to me that one of the problems here is we tell people to use copy. We should be telling people to use a command (or supply a command) that is smarter than that. 3. You cannot know the reason of archive_command failure (e.g. archive area full) if you don't use PostgreSQL's server logging. This is because archive_command failure is not logged in syslog/eventlog. Wait, what? Is this true (someone else?) JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)
On 06/06/2013 07:52 AM, Heikki Linnakangas wrote: I think it can be made fairly robust otherwise, and the performance impact should be pretty easy to measure with e.g pgbench. Once upon a time in a land far, far away, we expected users to manage their own systems. We had things like soft and hard quotas on disks and last log to find out who was logging into the system. Alas, as far as I know soft and hard quotas are kind of a thing of the past but that doesn't mean that their usefulness has ended. The idea that we PANIC is not just awful, it is stupid. I don't think anyone is going to disagree with that. However, there is a question of what to do instead. I think the idea of sprinkling checks into the higher level code before specific operations is not invalid but I also don't think it is necessary. To me, a more pragmatic approach makes sense. Obviously having some kind of code that checks the space makes sense but I don't know that it needs to be around any operation other than we are creating a segment. What do we care why the segment is being created? If we don't have enough room to create the segment, the transaction rollsback with some OBVIOUS not OBTUSE error. Obviously this could cause a ton of transactions to roll back but I think keeping the database consistent and rolling back a transaction in case of error is exactly what we are supposed to do. Sincerely, Joshua D. Drake - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bad error message on valuntil
On 06/07/2013 12:31 PM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail. Why? That seems rather silly. The general policy on authentication failure reports is that we don't tell the client anything it doesn't know already about what the auth method is. We can log additional info into the postmaster log if it I was looking at the code and I saw this catchall: default: errstr = gettext_noop(authentication failed for user \%s\: invalid authentication method); break; I think we could make the argument that if valuntil is expired that the authentication method is invalid. Thoughts? Else I am trying to come up with some decent wording... something like: Authentication failed: not all authentication tokens were met ? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)
On 06/08/2013 11:27 AM, Andres Freund wrote: On 2013-06-08 11:15:40 -0700, Joshua D. Drake wrote: To me, a more pragmatic approach makes sense. Obviously having some kind of code that checks the space makes sense but I don't know that it needs to be around any operation other than we are creating a segment. What do we care why the segment is being created? If we don't have enough room to create the segment, the transaction rollsback with some OBVIOUS not OBTUSE error. Obviously this could cause a ton of transactions to roll back but I think keeping the database consistent and rolling back a transaction in case of error is exactly what we are supposed to do. You know, the PANIC isn't there just because we like to piss of users. There's actual technical reasons that don't just go away by judging the PANIC as stupid. Yes I know we aren't trying to piss off users. What I am saying is that it is stupid to the user that it PANICS. I apologize if that came out wrong. At the points where the XLogInsert()s happens we're in critical sections out of which we *cannot* ERROR out because we already may have made modifications that cannot be allowed to be performed partially/unlogged. That's why we're throwing a PANIC which will force a cluster wide restart including *NOT* writing any further buffers from s_b out. Does this preclude (sorry I don't know this part of the code very well) my suggestion of on log create? JD Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] small patch to crypt.c
Hello, In my quest to understand how all the logging etc works with authentication I came across the area of crypt.c that checks for valid_until but it seems like it has an extraneous check. If I am wrong I apologize for the noise but wouldn't mind an explanation. index f01d904..8d809b2 100644 --- a/src/backend/libpq/crypt.c +++ b/src/backend/libpq/crypt.c @@ -145,9 +145,7 @@ md5_crypt_verify(const Port *port, const char *role, char *client_pass) /* * Password OK, now check to be sure we are not past rolvaliduntil */ - if (isnull) - retval = STATUS_OK; - else if (vuntil GetCurrentTimestamp()) + if (vuntil GetCurrentTimestamp()) retval = STATUS_ERROR; else retval = STATUS_OK; -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] small patch to crypt.c
On 06/08/2013 08:47 PM, Stephen Frost wrote: JD, * Joshua D. Drake (j...@commandprompt.com) wrote: In my quest to understand how all the logging etc works with authentication I came across the area of crypt.c that checks for valid_until but it seems like it has an extraneous check. If I am wrong I apologize for the noise but wouldn't mind an explanation. Alright, there probably aren't too many people out there running with their clock set to pre-2000, but wouldn't this end up giving the wrong result in those cases, as GetCurrentTimestamp() would end up returning a negative value, which would make it less than vuntil's default of zero? Perhaps we could change what vuntil is set to by default, but I think it's probably better to keep things as-is; we should really be checking for null cases explicitly in general. Well I was more referring to the default is: check if null, if true return ok check if valuntil today, if true return error else return ok To me we don't need the null check. However, when I tested it, without the null check you can't login. So now I am curious about what is going on. JD Thanks, Stephen -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Bad error message on valuntil
Hello, I had a customer pulling their hair out today because they couldn't login to their system. The error was consistently: 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password authentication failed for user user However the problem had nothing to do with password authentication. It was because the valuntil on the user had been set till a date in the past. Now technically if we just removed the word password from the error it would be accurate but it seems it would be better to say, FATAL: the user user has expired. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bad error message on valuntil
On 06/07/2013 11:57 AM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: I had a customer pulling their hair out today because they couldn't login to their system. The error was consistently: 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password authentication failed for user user However the problem had nothing to do with password authentication. It was because the valuntil on the user had been set till a date in the past. Now technically if we just removed the word password from the error it would be accurate but it seems it would be better to say, FATAL: the user user has expired. I think it's intentional that we don't tell the *client* that level of detail. Why? That seems rather silly. I could see emitting a log message about it, but it's not clear whether that will help an unsophisticated user. This is not an unsophisticated user. They tried resetting the password, even changing the username to lowercase in case it was some weird folding issue. Granted they didn't check pg_user but then again, I didn't at first either because, well the error was rather obvious until it wasn't. Sincerely, JD regards, tom lane -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bad error message on valuntil
On 06/07/2013 12:31 PM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail. Why? That seems rather silly. The general policy on authentication failure reports is that we don't tell the client anything it doesn't know already about what the auth method is. We can log additional info into the postmaster log if it seems useful to do so, but the more you tell a client, the more you risk undesirable info leakage to a bad guy. As an example here, reporting the valuntil condition would be acking to an attacker that he had the right password. So security by obscurity? Alright, without getting into that argument how about we change the error message to: FATAL: Authentication failed: Check server log for specifics And then we make sure we log proper info? Sincerely, Joshua D. Drake regards, tom lane -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bad error message on valuntil
On 06/07/2013 01:41 PM, David Johnston wrote: Please check server log for specifics is not a good message for something sent to a client that in many normal situation would have no access to said logs. I don't agree. The user doesn't need access to the logs. If they get that error they report it to their support staff. We don't need to tell them any more than that. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Redesigning checkpoint_segments
On 6/5/2013 10:54 PM, Peter Geoghegan wrote: On Wed, Jun 5, 2013 at 10:27 PM, Joshua D. Drake j...@commandprompt.com wrote: I just wonder if we are looking in the right place (outside of some obvious badness like the PANIC running out of disk space). So you don't think we should PANIC on running out of disk space? If you don't think we should do that, and you don't think that WAL writing should be throttled, what's the alternative? As I mentioned in my previous email: Instead of running out of disk space PANIC we should just write to an emergency location within PGDATA and log very loudly that the SA isn't paying attention. Perhaps if that area starts to get to an unhappy place we immediately bounce into read-only mode and log even more loudly that the SA should be fired. I would think read-only mode is safer and more polite than an PANIC crash. I do not think we should worry about filling up the hard disk except to protect against data loss in the event. It is not user unfriendly to assume that a user will pay attention to disk space. Really? JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Redesigning checkpoint_segments
On 6/5/2013 11:09 PM, Daniel Farina wrote: Instead of running out of disk space PANIC we should just write to an emergency location within PGDATA and log very loudly that the SA isn't paying attention. Perhaps if that area starts to get to an unhappy place we immediately bounce into read-only mode and log even more loudly that the SA should be fired. I would think read-only mode is safer and more polite than an PANIC crash. I do not think we should worry about filling up the hard disk except to protect against data loss in the event. It is not user unfriendly to assume that a user will pay attention to disk space. Really? Okay, then I will say it's user unfriendly, especially for a transient use of space, and particularly if there's no knob for said SA to attenuate what's going on. You appear to assume the SA can lean on the application to knock off whatever is going on or provision more disk in time, or that disk is reliable enough to meet one's goals. In my case, none of these precepts are true or desirable. I have zero doubt that in your case it is true and desirable. I just don't know that it is a positive solution to the problem as a whole. Your case is rather limited to your environment, which is rather limited to the type of user that your environment has. Which lends itself to the idea that this should be a Heroku Postgres thing, not a .Org wide thing. Sincerely, JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Redesigning checkpoint_segments
On 6/5/2013 11:25 PM, Harold Giménez wrote: Instead of running out of disk space PANIC we should just write to an emergency location within PGDATA This merely buys you some time, but with aggressive and sustained write throughput you are left on the same spot. Practically speaking it's the same situation as increasing the pg_xlog disk space. Except that you likely can't increase pg_xlog space (easily). The point here is to have overflow, think swap space. I agree it is better than PANIC, but read-only mode is definitely also a form of throttling; a much more abrupt and unfriendly one if I may add. I would think read only is less unfriendly than an all out failure. Consider if done correctly, the database would move back into read-write mode once the problem was resolved. JD
Re: [HACKERS] Redesigning checkpoint_segments
On 6/5/2013 11:31 PM, Peter Geoghegan wrote: On Wed, Jun 5, 2013 at 11:28 PM, Joshua D. Drake j...@commandprompt.com wrote: I have zero doubt that in your case it is true and desirable. I just don't know that it is a positive solution to the problem as a whole. Your case is rather limited to your environment, which is rather limited to the type of user that your environment has. Which lends itself to the idea that this should be a Heroku Postgres thing, not a .Org wide thing. If you look through the -general archives, or on stack overflow you'll find ample evidence that it is a problem that lots of people have. Not to be unkind but the problems of the uniformed certainly are not the problems of the informed. Or perhaps they are certainly the problems of the informed :P. I do read -general and I don't see it much honestly. I don't watch stackoverflow that much but I am sure it probably does come up here, sometimes but I bet I can point once again to a lack of provisioning on their part. This reminds me of the time that someone from Heroku said at PgEast, with a show of hands how many people here don't backup there database to S3. Almost everyone in the audience raised their hands. Again, I don't question your need but just because it is hot and now doesn't mean it is healthy. I honestly do no see the requirement you are trying to represent as a need for the wider, production community. (in short, not a single one of my customers would benefit from it, and 90% of them are running databases Heroku can't.) That is not a slight, honestly. I think your service is cool. I am just being honest. Sincerely, JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Redesigning checkpoint_segments
On 6/6/2013 1:11 AM, Heikki Linnakangas wrote: (I'm sure you know this, but:) If you perform a checkpoint as fast and short as possible, the sudden burst of writes and fsyncs will overwhelm the I/O subsystem, and slow down queries. That's what we saw before spread checkpoints: when a checkpoint happens, the response times of queries jumped up. That isn't quite right. Previously we had lock issues as well and checkpoints would take considerable time to complete. What I am talking about is that the background writer (and wal writer where applicable) have done all the work before a checkpoint is even called. Consider that everyone of my clients that I am active with sets the checkpoint_completion_target to 0.9. With a proper bgwriter config this works. 4. Bgwriter. We should be adjusting bgwriter so that it is writing everything in a manner that allows any checkpoint to be in the range of never noticed. Oh, I see where you're going. O.k. good. I am not nuts :D Yeah, that would be one way to do it. However, spread checkpoints has pretty much the same effect. Imagine that you tune your system like this: disable bgwriter altogether, and set checkpoint_completion_target=0.9. With that, there will be a checkpoint in progress most of the time, because by the time one checkpoint completes, it's almost time to begin the next one already. In that case, the checkpointer will be slowly performing the writes, all the time, in the background, without affecting queries. The effect is the same as what you described above, except that it's the checkpointer doing the writing, not bgwriter. O.k. if that is true, then we have redundant systems and we need to remove one of them. Yeah, wal_keep_segments is a hack. We should replace it with something else, like having a registry of standbys in the master, and how far they've streamed. That way the master could keep around the amount of WAL actually needed by them, not more not less. But that's a different story. Other oddities: Yes checkpoint_segments is awkward. We shouldn't have to set it at all. It should be gone. The point of having checkpoint_segments or max_wal_size is to put a limit (albeit a soft one) on the amount of disk space used. If you don't care about that, I guess we could allow max_wal_size=-1 to mean infinite, and checkpoints would be driven off purely based on time, not WAL consumption. I would not only agree with that, I would argue that max_wal_size doesn't need to be there at least as a default. Perhaps as an advanced configuration option that only those in the know see. Basically we start with X amount perhaps to be set at initdb time. That X amount changes dynamically based on the amount of data being written. In order to not suffer from recycling and creation penalties we always keep X+N where N is enough to keep up with new data. To clarify, here you're referring to controlling the number of WAL segments preallocated/recycled, rather than how often checkpoints are triggered. Currently, both are derived from checkpoint_segments, but I proposed to separate them. The above is exactly what I proposed to do for the preallocation/recycling, it would be tuned automatically, but you still need something like max_wal_size for the other thing, to trigger a checkpoint if too much WAL is being consumed. You think so? I agree with 90% of this paragraph but it seems to me that we can find an algortihm that manages this without the idea of max_wal_size (at least as a user settable). Along with the above, I don't see any reason for checkpoint_timeout. Because of bgwriter we should be able to rather indefinitely not worry about checkpoints (with a few exceptions such as pg_start_backup()). Perhaps a setting that causes a checkpoint to happen based on some non-artificial threshold (timeout) such as amount of data currently in need of a checkpoint? Either I'm not understanding what you said, or you're confused. The point of checkpoint_timeout is put a limit on the time it will take to recover in case of crash. The relation between the two, checkpoint_timeout and how long it will take to recover after a crash, it not straightforward, but that's the best we have. I may be confused but it is my understanding that bgwriter writes out the data from the shared buffer cache that is dirty based on an interval and a max pages written. If we are writing data continuously, we don't need checkpoints except for special cases (like pg_start_backup())? Bgwriter does not worry about checkpoints. By amount of data currently in need of a checkpoint, do you mean the number of dirty buffers in shared_buffers, or something else? I don't see how or why that should affect when you perform a checkpoint. Heikki said, I propose that we do something similar, but not exactly the same. Let's have a setting, max_wal_size, to control the max. disk space reserved for WAL. Once that's reached (or you get close
Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)
On 06/06/2013 09:30 PM, Jeff Janes wrote: Archiving - In some ways, this is the simplest case. Really, we just need a way to know when the available WAL space has become 90% full, and abort archiving at that stage. Once we stop attempting to archive, we can clean up the unneeded log segments. I would oppose that as the solution, either an unconditional one, or configurable with is it as the default. Those segments are not unneeded. I need them. That is why I set up archiving in the first place. If you need to shut down the database rather than violate my established retention policy, then shut down the database. Agreed and I would oppose it even as configurable. We set up the archiving for a reason. I do think it might be useful to be able to store archiving logs as well as wal_keep_segments logs in a different location than pg_xlog. What we need is a better way for the DBA to find out that archiving is falling behind when it first starts to fall behind. Tailing the log and examining the rather cryptic error messages we give out isn't very effective. The archive command can be made a shell script (or that matter a compiled program) which can do anything it wants upon failure, including emailing people. Yep, that is what PITRTools does. You can make it do whatever you want. JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Redesigning checkpoint_segments
On 06/05/2013 05:37 PM, Robert Haas wrote: On Wed, Jun 5, 2013 at 3:24 PM, Fujii Masao masao.fu...@gmail.com wrote: OTOH, if we use max_wal_size as a hard limit, we can avoid such PANIC error and long down time. Of course, in this case, once max_wal_size is reached, we cannot complete any query writing WAL until the checkpoint has completed and removed old WAL files. During that time, the database service looks like down from a client, but its down time is shorter than the PANIC error case. So I'm thinking that some users might want the hard limit of pg_xlog size. I wonder if we could tie this in with the recent proposal from the Heroku guys to have a way to slow down WAL writing. Maybe we have several limits: I didn't see that proposal, link? Because the idea of slowing down wal-writing sounds insane. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Redesigning checkpoint_segments
On 06/05/2013 05:37 PM, Robert Haas wrote: - If it looks like we're going to exceed limit #3 before the checkpoint completes, we start exerting back-pressure on writers by making them wait every time they write WAL, probably in proportion to the number of bytes written. We keep ratcheting up the wait until we've slowed down writers enough that will finish within limit #3. As we reach limit #3, the wait goes to infinity; only read-only operations can proceed until the checkpoint finishes. Alright, perhaps I am dense. I have read both this thread and the other one on better handling of archive command (http://www.postgresql.org/message-id/cam3swzqcynxvpaskr-pxm8deqh7_qevw7uqbhpcsg1fpsxk...@mail.gmail.com). I recognize there are brighter minds than mine on this thread but I just honestly don't get it. 1. WAL writes are already fast. They are the fastest write we have because it is sequential. 2. We don't want them to be slow. We want data written to disk as quickly as possible without adversely affecting production. That's the point. 3. The spread checkpoints have always confused me. If anything we want a checkpoint to be fast and short because: 4. Bgwriter. We should be adjusting bgwriter so that it is writing everything in a manner that allows any checkpoint to be in the range of never noticed. Now perhaps my customers workloads are different but for us: 1. Checkpoint timeout is set as high as reasonable, usually 30 minutes to an hour. I wish I could set them even further out. 2. Bgwriter is set to be aggressive but not obtrusive. Usually adjusting based on an actual amount of IO bandwidth it may take per second based on their IO constraints. (Note I know that wal_writer comes into play here but I honestly don't remember where and am reading up on it to refresh my memory). 3. The biggest issue we see with checkpoint segments is not running out of space because really 10GB is how many checkpoint segments? It is with wal_keep_segments. If we don't want to fill up the pg_xlog directory, put the wal logs that are for keep_segments elsewhere. Other oddities: Yes checkpoint_segments is awkward. We shouldn't have to set it at all. It should be gone. Basically we start with X amount perhaps to be set at initdb time. That X amount changes dynamically based on the amount of data being written. In order to not suffer from recycling and creation penalties we always keep X+N where N is enough to keep up with new data. Along with the above, I don't see any reason for checkpoint_timeout. Because of bgwriter we should be able to rather indefinitely not worry about checkpoints (with a few exceptions such as pg_start_backup()). Perhaps a setting that causes a checkpoint to happen based on some non-artificial threshold (timeout) such as amount of data currently in need of a checkpoint? Heikki said, I propose that we do something similar, but not exactly the same. Let's have a setting, max_wal_size, to control the max. disk space reserved for WAL. Once that's reached (or you get close enough, so that there are still some segments left to consume while the checkpoint runs), a checkpoint is triggered. In this proposal, the number of segments preallocated is controlled separately from max_wal_size, so that you can set max_wal_size high, without actually consuming that much space in normal operation. It's just a backstop, to avoid completely filling the disk, if there's a sudden burst of activity. The number of segments preallocated is auto-tuned, based on the number of segments used in previous checkpoint cycles. This makes sense except I don't see a need for the parameter. Why not just specify how the algorithm works and adhere to that without the need for another GUC? Perhaps at any given point we save 10% of available space (within a 16MB calculation) for pg_xlog, you hit it, we checkpoint and LOG EXACTLY WHY. Instead of running out of disk space PANIC we should just write to an emergency location within PGDATA and log very loudly that the SA isn't paying attention. Perhaps if that area starts to get to an unhappy place we immediately bounce into read-only mode and log even more loudly that the SA should be fired. I would think read-only mode is safer and more polite than an PANIC crash. I do not think we should worry about filling up the hard disk except to protect against data loss in the event. It is not user unfriendly to assume that a user will pay attention to disk space. Really? Open to people telling me I am off in left field. Sorry if it is noise. Sincerely, JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
Re: [HACKERS] Redesigning checkpoint_segments
On 06/05/2013 06:23 PM, Daniel Farina wrote: On Wed, Jun 5, 2013 at 6:00 PM, Joshua D. Drake j...@commandprompt.com wrote: I didn't see that proposal, link? Because the idea of slowing down wal-writing sounds insane. It's not as insane as introducing an archiving gap, PANICing and crashing, or running this hunk o junk I wrote http://github.com/fdr/ratchet Well certainly we shouldn't PANIC and crash but that is a simple fix. You have a backup write location and start logging really loudly that you are using it. Sincerely, JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Redesigning checkpoint_segments
On 6/5/2013 10:07 PM, Daniel Farina wrote: If I told you there were some of us who would prefer to attenuate the rate that things get written rather than cancel or delay archiving for a long period of time, would that explain the framing of the problem? I understand that based on what you said above. Or, is it that you understand that's what I want, but find the notion of such a operation hard to relate to? I think this is where I am at. To me, you don't attenuate the rate that things get written, you fix the problem in needing to do so. The problem is one of provisioning. Please note that I am not suggesting there aren't improvements to be made, there absolutely are. I just wonder if we are looking in the right place (outside of some obvious badness like the PANIC running out of disk space). Or, am I misunderstanding your confusion? To be honest part of my confusion was just trying to parse all the bits that people were talking about into a cohesive, this is the actual problem. Sincerely, JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_rewind, a tool for resynchronizing an old master after failover
On 06/04/2013 01:55 PM, Josh Berkus wrote: That seems rather like a catch-22 Bruce. If they don't check with the legal department, it's dangerous, but if they do check, it's dangerous? Presumably if they checked with the legal department, it's cleared. We should be wary of stuff contributed by company employees who *didn't* check. This particular tool seems highly unlikely to be legitimately patentable, anyway. There's too much prior art. legitimately patentable is a rather ethereal phrase in our industry right now don't you think? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] units in postgresql.conf comments
On 05/30/2013 12:01 AM, Heikki Linnakangas wrote: We could make it mandatory to specify the unit in the value. Ie. throw an error on wal_sender_timeout = 50: ERROR: unit required for option wal_sender_timeout HINT: Valid units for this parameter are ms, s, min, h, and d. Then you wouldn't need a comment to explain what the unit of a naked value is. The only problem I see with that is backwards-compatibility. Old postgresql.conf files containing naked values would no longer work. But all you'd need to do is to add in the units, which would work on older versions too, and would be good for readability anyway. I like this idea with one addition. We should have a default unit for each. For wal_sender_timeout seconds makes sense, but for checkpoint_timeout minutes makes sense (for example). JD - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] units in postgresql.conf comments
On 05/30/2013 12:55 AM, Magnus Hagander wrote: I like this idea with one addition. We should have a default unit for each. For wal_sender_timeout seconds makes sense, but for checkpoint_timeout minutes makes sense (for example). This sounds like a good way to make things even more confusing. Right now the confusion is only in the comments - this would make it confusing in the actual values. Requiring a unit seems like a much better idea. That way, there is no way for confusion. I can buy into that. JD -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] units in postgresql.conf comments
On 05/30/2013 01:14 AM, Heikki Linnakangas wrote: On 30.05.2013 10:52, Joshua D. Drake wrote: On 05/30/2013 12:01 AM, Heikki Linnakangas wrote: We could make it mandatory to specify the unit in the value. Ie. throw an error on wal_sender_timeout = 50: ERROR: unit required for option wal_sender_timeout HINT: Valid units for this parameter are ms, s, min, h, and d. Then you wouldn't need a comment to explain what the unit of a naked value is. The only problem I see with that is backwards-compatibility. Old postgresql.conf files containing naked values would no longer work. But all you'd need to do is to add in the units, which would work on older versions too, and would be good for readability anyway. I like this idea with one addition. We should have a default unit for each. For wal_sender_timeout seconds makes sense, but for checkpoint_timeout minutes makes sense (for example). Uh, if specifying the unit is mandatory, what exactly would the default unit mean? Yeah, see my other email. I missed that part. It is late for me. Sorry for the noise. JD - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 08:36 AM, Hannu Krosing wrote: The conversation does not change. Further, we are not Firefox. We are not user software. We are developer software. At least some of the real-world problems with PostgreSQL comes from We are developer software mentality. Yes, We are developer software, but we are also a DBA/maintainer/infrastructure manager I would not hire any of those three that weren't smart enough to understand our versioning scheme or had the wits to open a web browser and google: PostgreSQL versioning The answer is link #1 on Google. That said, I won't raise a stink. I am not really of a strong opinion either way except to say we are not solving a problem. We are just tickling each other's fancies. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 02:18 PM, Bruce Momjian wrote: I would like to see the ability to define if a query is read only at the protocol level, so that load balances that speak libpq can know what to do with the query without parsing it. Sounds nice, but how would we do that? That would require libpq to know it, right? Do we pass anything back after parsing but before execution? Could it be optional? What about functions that modify the database --- isn't that only known at execution time? I can't speak to the actual C code that would be required but from a user space, I could see something like this: con = psycopg2.connect(database='testdb', user='test', transaction-type='r') Thus when the connection is made, before anything else is done, we know it is a read only connection and therefore any load balancer speaking libpq would also know it is a read only. The default of course would be r/w and you would use a different connection handler for r/w or w queries. The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 03:36 PM, Bruce Momjian wrote: The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Well, you could do SET TRANSACTION READ ONLY, and that would prevent any write transactions. You could assume it is a read query, and get the error and resubmit on the master if that happens, but that sounds inefficient. I thought you were asking for something where you could submit a query and it would report back as read/write or read-only. No I am suggesting something that before anything happens with the parser, the protocol knows what is up. So things like pgpool-ii don't even need a parser, it just knows it is a read only query because the protocol says so. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 04:05 PM, Bruce Momjian wrote: On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote: On 05/28/2013 03:36 PM, Bruce Momjian wrote: The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Well, you could do SET TRANSACTION READ ONLY, and that would prevent any write transactions. You could assume it is a read query, and get the error and resubmit on the master if that happens, but that sounds inefficient. I thought you were asking for something where you could submit a query and it would report back as read/write or read-only. No I am suggesting something that before anything happens with the parser, the protocol knows what is up. So things like pgpool-ii don't even need a parser, it just knows it is a read only query because the protocol says so. Oh, that is an interesting idea. The application is indicating it is read-only via the protocol, and poolers can optimize that. Don't we have the ability to pass arbitrary GUC values back through the protocol, e.g. transaction_read_only? If not, that might be a way to do this cleanly. I don't know but I don't think so. Anything that is calling SET is going to run through the parser. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] pg_upgrade -u
On 05/28/2013 07:55 PM, Bruce Momjian wrote: Perhaps just documenting the behavior is all that is needed, but -U is everywhere and I think that's a good thing. [ moved to hacker ] Wow, I never realized other tools used -U for user, instead of -u. Should I change pg_upgrade to use -U for 9.4? I can keep supporting -u as an undocumented option. Yes, -U makes the most sense as that is what is used with the other tools. I think you should just drop -u, this isn't something people are doing everyday (like psql). The backward compatibility argument is pretty slim. JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/27/2013 04:58 PM, Craig Ringer wrote: On 05/28/2013 12:41 AM, Simon Riggs wrote: I'm happy with that. I was also thinking about collecting changes not related just to disk format, if any exist. Any wire protocol or syntax changes? I can't seem to find a things we want to do in wire protocol v4 doc in the wiki but I know I've seen occasional discussion of things that can't be done without protocol changes. Anyone with a better memory than me able to pitch in? What'd be required to support in-band query cancellation? Sending per-statement GUCs (to allow true statement timeout)? I would like to see the ability to define if a query is read only at the protocol level, so that load balances that speak libpq can know what to do with the query without parsing it. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/27/2013 06:53 PM, Craig Ringer wrote: On 05/28/2013 09:39 AM, Gavin Flower wrote: Yes, I hate the Firefox style number inflation. I was arguing *for* it ;-) I don't like it much either, but (a) we do about one release a year, not one every few weeks and (b) it's very clear from a quick look at Stack Overflow or first-posts to pgsql-general how confusing two-part major versions are to users. If it's a bit less aesthetically pleasing I'm OK with that. This argument comes up every couple of years and the people that are trying to solve the problem by changing the versioning are ignoring the fact that there is no problem to solve. Consider the following exchange: Client: I have X problem with PostgreSQL CMD: What version? Client: 9 CMD: Which version of 9? Client: 9.0.2 CMD: You should be running 9.2.4 or at least 9.0.13 Now, if we change the version numbers: Client: I have X problem with PostgreSQL CMD: What version? Client: 9 CMD: Which version of 9? Client: 9.0.2 CMD: You should be running 10.0.5 or at least 9.0.13 The conversation does not change. Further, we are not Firefox. We are not user software. We are developer software. Sincerely, Joshua D. Drake -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Remaining beta blockers
On 05/06/2013 08:17 AM, Tom Lane wrote: Per my other mail, I think adding an AMV option at this time is inadvisable. I could go either way on removing or keeping the is_scannable function --- anybody else have an opinion on that point? Which of us is going to commit this? We're running low on time ... As a my two cents, I have been watching this thread and the concern on timeline is bothering me. I fully understand our want to get into Beta and I know we don't want to slip schedule too much but quality is important. It is what makes our project what it is more than any other value we hold. I also know we already slipped the beta once but we are not a corporation, we do not have shareholders and nobody can fire us. If we need to push it again for quality, shouldn't we? Sincerely, JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Documentation epub format
On 05/01/2013 09:27 AM, Fabien COELHO wrote: Hello devs, I've given a try to the PostgreSQL documentation in epub format. I must admit that there is a bit of a disappointement as far as the user experience is concerned: the generated file is barely usable on an iPad2 with the default iBooks reader, which was clearly not designed for handling a 4592 pages book (from its point of view). The table of contents too much detailed, so it is long and slow to scan, and there is no clear shortcut. Flipping pages in the documentation takes ages (well, close to one second or more if I flip a few pages). Do not try search. This seems to suggest that instead of generating one large ebook, the build should generate a set of ebooks, say one for each part. At the minimum, a less detailed toc could be more usable and help navigate the huge contents. Once upon a time we had multiple books as documentation, then at some point we merged them. It was quite a few years ago. I would agree at this point that we need to consider breaking them up again. The documentation is unwieldy. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Documentation epub format
On 05/01/2013 10:52 AM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: Once upon a time we had multiple books as documentation, then at some point we merged them. It was quite a few years ago. I would agree at this point that we need to consider breaking them up again. The documentation is unwieldy. The reason we merged them was to allow hyperlink cross-references between different parts of the docs. I would be sad to lose that.\ Defintely. Is there no way to cross reference multiple documents? Peter? JD regards, tom lane -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Documentation epub format
On 05/01/2013 10:56 AM, Andrew Satori wrote: I would second Tom on this, and if ePub is really a longer term goal of the documentation, the various eBook formats have differing levels of support for hyperlinking that would merit retaining everything in a single book that can be linked from direct references. I don't think ePub is a problem here, we will have the same problem with PDF. The issue is the sheer size of the manual. If we can solve the cross referencing issue, breaking them up makes sense I would think. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal to add --single-row to psql
On 04/25/2013 07:42 AM, Tom Lane wrote: Christopher Manning c...@christophermanning.org writes: Fabr�zio and Tom, I know that you can use --variable=FETCH_COUNT=1 from the psql command line, but internally that uses a CURSOR to batch the rows and [Redshift doesn't support CURSOR]( https://forums.aws.amazon.com/thread.jspa?threadID=122664tstart=0) so it's not an option when using psql to download data from Redshift. I don't know what redshift is, It is a PostgreSQL fork based on ancient source code. From Amazon: Amazon Redshift is based on PostgreSQL 8.0.2. Amazon Redshift and PostgreSQL have a number of very important differences that you must be aware of as you design and develop your data warehouse applications. but I have very little patience with the idea of burdening psql with yet another random feature in order to work around deficiencies in somebody else's software. Considering that the earliest any such thing could reach the field would be 9.4, it seems not unlikely that the need for it would be gone by next year anyway. +1 this is really an amazon problem not a postgresql problem. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.3 release notes suggestions
On 04/25/2013 04:48 PM, Daniel Farina wrote: On Wed, Apr 24, 2013 at 6:30 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Apr 23, 2013 at 11:41 PM, Bruce Momjian br...@momjian.us wrote: Thanks for the many suggestions on improving the 9.3 release notes. There were many ideas I would have never thought of. Please keep the suggestions coming. Bruce, Thanks for writing them! Consider the sentiment duplicated. Thank you, Bruce. Isn't that a primary key violation? /me runs Sorry it has been a really long two days. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] page 1 of relation global/11787 was uninitialized
On 04/09/2013 09:21 AM, Stephen R. van den Berg wrote: - Looking at global/11787, doesn't reveal any obvious corruption. The server was running with: synchronous_commit = off full_page_writes = off full_page_writes = off is the problem. From the docs: Turning this parameter off speeds normal operation, but might lead to either unrecoverable data corruption, or silent data corruption, after a system failure. The risks are similar to turning off fsync, though smaller, and it should be turned off only based on the same circumstances recommended for that parameter. http://www.postgresql.org/docs/9.0/static/runtime-config-wal.html#GUC-FULL-PAGE-WRITES Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump/restore syntax checking bug?
On 03/22/2013 10:13 PM, Josh Kupershmidt wrote: On Fri, Mar 22, 2013 at 9:35 PM, Joshua D. Drake j...@commandprompt.com wrote: postgres@jd-laptop:~$ pg_restore -d test -P 'by(),hello()' foo.sqlc Note, the pg_restore doc makes no mention of trying to squeeze multiple function prototypes in a single argument you've done here, or of using multiple -P flags. It appears we need better syntax checking. Can't really argue with this. But if you think these pg_restore examples are bad, try this gem: reindexdb --table='foo; ALTER ROLE limited WITH superuser' That is HORRIBLE! Looks like our base utilities need some attention. jD Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_dump/restore syntax checking bug?
Hello, In testing some pg_restore functionality I found the following: postgres@jd-laptop:~$ pg_dump -U postgres -Fc -s --file=foo.sqlc postgres@jd-laptop:~$ dropdb test; postgres@jd-laptop:~$ createdb test; postgres@jd-laptop:~$ pg_restore -d test -P 'by()' foo.sqlc postgres@jd-laptop:~$ psql -U postgres test psql (9.1.8) Type help for help. test=# select by(); by by (1 row) test=# select hello(); ERROR: function hello() does not exist LINE 1: select hello(); ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts. The above is as expected. test=# \q postgres@jd-laptop:~$ pg_restore -d test -P 'by(),hello()' foo.sqlc postgres@jd-laptop:~$ psql -U postgres test; psql (9.1.8) Type help for help. test=# select hello(); ERROR: function hello() does not exist LINE 1: select hello(); This is where I am confused. It didn't restore hello() and it also didn't error that the syntax of the restore command was invalid. ^ test=# drop function by(); DROP FUNCTION test=# test=# q test-# \q postgres@jd-laptop:~$ pg_restore -d test -P 'by(),hello()' foo.sqlc postgres@jd-laptop:~$ psql -U postgres test; psql (9.1.8) Type help for help. test=# drop function by(); ERROR: function by() does not exist by() not restored by above command test=# \q postgres@jd-laptop:~$ pg_restore -d test -P 'by()','hello()' foo.sqlc postgres@jd-laptop:~$ psql -U postgres test; psql (9.1.8) Type help for help. test=# drop function by(); ERROR: function by() does not exist test=# select hello(); ERROR: function hello() does not exist LINE 1: select hello(); ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts. by() and hello() also not restored test=# \q postgres@jd-laptop:~$ pg_restore -d test -P 'by()' -P'hello()' foo.sqlc postgres@jd-laptop:~$ psql -U postgres test; psql (9.1.8) Type help for help. test=# select hello(); hello --- hello (1 row) test=# select by(); ERROR: function by() does not exist LINE 1: select by(); hello() restored but by() was not. It appears we need better syntax checking. Sincerely, JD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] REFRESH MATERIALIZED VIEW locklevel
On 03/08/2013 10:09 AM, Merlin Moncure wrote: On Fri, Mar 8, 2013 at 11:59 AM, Josh Berkus j...@agliodbs.com wrote: Andres, Further, we get pretty much one and only one chance to promote a new major feature, which is when that feature is first introduced. Improving the feature in the next version of Postgres is not news, so we can't successfully promote it. If we soft-pedal MVs in the 9.3 announcement, we will not be able to get people excited about them in 9.4; they will be yesterday's news. +1 on this. they are useful to me as immediately and I work in busy environments. the formal matview feature is a drop in replace for my ad hoc implementation of 'drop cache table, replace from view'. I already have to work around the locking issue anyways -- sure, it would be great if I didn't have to do that either but I'll take the huge syntactical convenience alone. Just to throw my +1 into the ring. Well written Josh. JD merlin -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Enabling Checksums
On 03/06/2013 03:06 PM, Robert Haas wrote: On Wed, Mar 6, 2013 at 6:00 PM, Josh Berkus j...@agliodbs.com wrote: We've had a few EnterpriseDB customers who have had fantastically painful experiences with PostgreSQL + ZFS. Supposedly, aligning the ZFS block size to the PostgreSQL block size is supposed to make these problems go away, but in my experience it does not have that effect. So I think telling people who want checksums go use ZFS is a lot like telling them oh, I see you have a hangnail, we recommend that you solve that by cutting your arm off with a rusty saw. Wow, what platform are you using ZFS on? (we have a half-dozen clients on ZFS ...) Not us, customers. But as to platform, I have yet to run across anyone running ZFS on anything but Solaris. I'd be interested to hear your experiences. Mine rhyme with sun a play dreaming. I would guess he meant on X86_64 or Sparc. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] transforms
On 03/05/2013 02:52 PM, Josh Berkus wrote: plperlh=# \c postgres You are now connected to database postgres as user josh. postgres=# create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: PL_thr_key STATEMENT: create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so: undefined symbol: PL_thr_key What happens if you set your LD_LIBRARY_PATH to reflect each installation before you start the database? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fractal tree indexing
On 02/13/2013 09:54 AM, Simon Riggs wrote: I'd call it out as a marketing name. I guess it's fractal in the sense that all levels of the tree can hold leaf tuples in the buffers; the structure looks the same no matter how deep you zoom, like a fractal.. But Buffered would be more appropriate IMO. I hope for their sake there is more to it than that. It's hard to see how buffering can be patented. Talk to Apple about that. It only needs to be worded correctly. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Too frequent checkpoints ?
On 02/08/2013 02:37 AM, Pavan Deolasee wrote: I wonder if this is all expected. The database is getting ZERO activity. There are no connections open at this time. The checkpoints are happening at every 30 seconds and new WAL files are being created, AFAIK because the old ones are getting archived. Can't we be smart about not archiving new files if we did not generate any new WAL since the last archive ? It is because you have it set up so that the longest time you can go without checkpoints is 30 seconds. So no matter what, if you have 1000 checkpoint segments you are still going to checkpoint ever 30 seconds. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Considering Gerrit for CFs
On 02/06/2013 01:53 PM, Tom Lane wrote: ... if it's going to try to coerce us out of our email-centric habits, then I for one am very much against it. To me, the problems with the existing CF app are precisely that it's not well enough integrated with the email discussions. The way to fix that is not to abandon email (or at least, it's not a way I wish to pursue). The email centric habits are by far the biggest limiting factor we have. Email was never designed for integral collaboration. That said, I see no way around it. I have brought up this idea before but, Redmine has by far served the purposes (with a little effort) of CMD and it also integrates with GIT. It also does not break the email work flow. It does not currently allow commands via email but that could be easily (when I say easily, I mean it) added. Just another thought. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 01/23/2013 09:51 AM, Josh Berkus wrote: The only way to fix increasing bug counts is through more-comprehensive regular testing. Currently we have regression/unit tests which cover maybe 30% of our code. Performance testing is largely ad-hoc. We don't require comprehensive acceptance testing for new patches. And we have 1m lines of code. Of course our bug count is increasing. And... slow down the release cycle or slow down the number of features that are accepted. Don't get me wrong I love everything we have and are adding every cycle but there does seem to be a definite weight difference between # of features added and time spent allowing those features to settle. Sincerely, JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] logical changeset generation v4 - Heikki's thoughts about the patch state
On 01/23/2013 05:17 PM, Robert Haas wrote: Of course, I have no evidence that that will happen. But it is a really big piece of code, and therefore unless you are superman, it's probably got a really large number of bugs. The scary thing is that it is not as if we can say, well, this is a big hunk of code, but it doesn't really touch the core of the system, so if it's broken, it'll be broken itself, but it won't break anything else. Rather, this code is deeply in bed with WAL, with MVCC, and with the on-disk format of tuples, and makes fundamental changes to the first two of those. You agreed with Tom that 9.2 is the buggiest release in recent memory, but I think logical replication could easily be an order of magnitude worse. Command Prompt worked for YEARS to get logical replication right and we never got it to the point where I would have been happy submitting it to -core. It behooves .Org to be extremely conservative about this feature. Granted, it is a feature we should have had years ago but still. It is not a simple thing, it is not an easy thing. It is complicated and complex to get correcft. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] LLVM / CLang / PostgreSQL
Hello, Has anyone played with this? Seen any results? It looks like most testing is being done on Mac OSX (via buildfarm). JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamic SQL - possible performance regression in 9.2
On 01/04/2013 12:05 PM, Josh Berkus wrote: Next question is what people think about back-patching into 9.2 so as to eliminate the performance regression vs 9.1. I believe this would be safe (although some care would have to be taken to put the added boolean fields into places where they'd not result in an ABI break). However it may not be worth the risk. The 40% slowdown seen with Pavel's example seems to me to be an extreme corner case --- Dong's result of 8% slowdown is probably more realistic for normal uses of SPI_execute. Might be better to just live with it in 9.2. Thoughts? 8% is a pretty serious regression, for those of us with applications which do a lot of dynamic SQL. As a reminder, many people do dynamic SQL even in repetitive, performance-sensitive functions in order to avoid plan caching. Also partition-handlers often use dynamic SQL, and a 10% drop in loading rows/second would be a big deal. Let's put it this way: if the community doesn't backport it, we'll end up doing so ad-hoc for some of our customers. Exactly. This is a significant reduction in the production quality of PostgreSQL as it pertains to dynamic SQL. To put it more bluntly, we will have people not upgrade to 9.2 specifically because of this problem. Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
