Re: [HACKERS] Changing pg_dump default file format

2013-11-07 Thread Joshua D. Drake
On 11/07/2013 10:00 AM, Josh Berkus wrote: On 11/07/2013 08:26 AM, Tom Lane wrote: Magnus Hagander mag...@hagander.net writes: On Thu, Nov 7, 2013 at 5:02 PM, Joshua D. Drake j...@commandprompt.com wrote: I humbly request on behalf of those who manage production postgresql instances that we

Re: [HACKERS] Changing pg_dump default file format

2013-11-07 Thread Joshua D. Drake
On 11/07/2013 11:01 AM, Alvaro Herrera wrote: Maybe we could provide a master controller program (pg_backup or whatever name we agree on), which could receive commands much like pg_ctl. $ pg_backup --help pg_backup is a backup handler program for PostgreSQL. Usage: pg_backup backup [-p

Re: [HACKERS] Changing pg_dump default file format

2013-11-07 Thread Joshua D. Drake
On 11/07/2013 10:54 AM, Kevin Grittner wrote: That is who I am thinking of. A DBA team may have hundreds of databases to manage, each with many scripts which have been running nicely for years. A change like this is bound to break some of those crontab scripts they may not even remember they

Re: [HACKERS] Changing pg_dump default file format

2013-11-07 Thread Joshua D. Drake
On 11/07/2013 12:42 PM, Josh Berkus wrote: All, I'm amused at how this has become a let's pile on everything which has ever been missing in pg_dump into one thread. Well it has been broken longer than most of our utilities. Sorry... not broken but certainly not complete. It is to be

Re: [HACKERS] Commitfest II CLosed

2013-10-22 Thread Joshua D. Drake
(myself included) can do a cursory review (patch applies, docs are good, indentation is appropriate, works as advertised). The commitfest app would have to be modified for this but what do people think? Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579

Re: [HACKERS] Auto-tuning work_mem and maintenance_work_mem

2013-10-17 Thread Joshua D. Drake
initdb detect how much memory is available on the machine in TOTAL and pick the most appropriate. Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion

Re: [HACKERS] Auto-tuning work_mem and maintenance_work_mem

2013-10-17 Thread Joshua D. Drake
On 10/17/2013 09:49 AM, Robert Haas wrote: A lot. A whole lot, more than what most people have in production with more than that. You are forgetting a very large segment of the population who run... VMs. That's true, but are you actually arguing for keeping work_mem at 1MB? Even on a VM

Re: [HACKERS] Auto-tuning work_mem and maintenance_work_mem

2013-10-17 Thread Joshua D. Drake
a user to likely but possibly not worry about changing the conf. Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams

Re: [HACKERS] Auto-tuning work_mem and maintenance_work_mem

2013-10-09 Thread Joshua D. Drake
tells them (I am not being rude here). We could argue all day what the best equation is for this, the key is to pick something reasonable, not perfect. Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services

Re: [HACKERS] Could ANALYZE estimate bloat?

2013-09-20 Thread Joshua D. Drake
On 09/20/2013 11:59 AM, Josh Berkus wrote: Hackers, I've been tinkering with a number of table bloat checks, and it occurred to me that the problm is that these are all approximations based on overall gross statistics, and as such highly inaccurate. It seems like would could have ANALYZE,

Re: [HACKERS] get rid of SQL_ASCII?

2013-09-05 Thread Joshua D. Drake
On 09/05/2013 09:42 AM, Josh Berkus wrote: Peter, Other ideas? Are there legitimate uses for SQL_ASCII? Migrating from MySQL. We've had some projects where we couldn't fix MySQL's non-enforcement text garbage, and had to use SQL_ASCII on the receiving side. If it hadn't been available,

Re: [HACKERS] Kudos for Reviewers -- wrapping it up

2013-08-07 Thread Joshua D. Drake
of the dome that is -hackers provide those incentives. Give reviewers the just recognition they deserve and I believe we will see more reviewing effort. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services

[HACKERS] Autovacuum different in 9.2.4?

2013-08-05 Thread Joshua D. Drake
Hello, I seem to recall autovacuum changes landing for 9.2.4. Can someone please describe what those changes were and how they could affect usage? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High

[HACKERS] don't own lock of type?

2013-08-05 Thread Joshua D. Drake
Hello, What exactly causes this? WARNING: you don't own a lock of type ExclusiveLock Does this mean the user calling the lock doesn't own the object? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development

Re: [HACKERS] Autovacuum different in 9.2.4?

2013-08-05 Thread Joshua D. Drake
On 08/05/2013 12:13 PM, Jeff Janes wrote: On Mon, Aug 5, 2013 at 11:16 AM, Joshua D. Drake j...@commandprompt.com wrote: Hello, I seem to recall autovacuum changes landing for 9.2.4. Can someone please describe what those changes were and how they could affect usage? Those landed in 9.2.3

Re: [HACKERS] don't own lock of type?

2013-08-05 Thread Joshua D. Drake
On 08/05/2013 11:38 AM, Robert Haas wrote: On Mon, Aug 5, 2013 at 2:32 PM, Joshua D. Drake j...@commandprompt.com wrote: Hello, What exactly causes this? WARNING: you don't own a lock of type ExclusiveLock Does this mean the user calling the lock doesn't own the object? It means there's

Re: [HACKERS] don't own lock of type?

2013-08-05 Thread Joshua D. Drake
On 08/05/2013 02:58 PM, Robert Haas wrote: On Mon, Aug 5, 2013 at 3:37 PM, Joshua D. Drake j...@commandprompt.com wrote: It means there's a bug. Either in PostgreSQL, or some loadable module you're using. I am getting this rather frequently, I will check modules but I don't believe we

Re: [HACKERS] Kudos for Reviewers -- wrapping it up

2013-07-12 Thread Joshua D. Drake
On 07/12/2013 10:49 AM, Andrew Dunstan wrote: On 07/12/2013 01:28 PM, Alvaro Herrera wrote: Josh Berkus wrote: -- a couple of compromise proposals were made: a) that reviewers who do actual code modification of the patch get credited on the feature, and those who just review it get

Re: [HACKERS] --with-libedit-preferred is bad design

2013-07-12 Thread Joshua D. Drake
On 7/12/2013 7:10 PM, Josh Berkus wrote: That would hardly be only true of libedit, on Apple. It's also broken on some Red Hat versions, last I checked. Last I heard, libedit was completely borked. Here is a report (two years old) of still broken libedit in Debian:

Re: [HACKERS] Millisecond-precision connect_timeout for libpq

2013-07-05 Thread Joshua D. Drake
On 7/5/2013 1:01 PM, Josh Berkus wrote: If you are issuing a fresh connection for each sub-100ms query, you're doing it wrong anyway ... It's fairly common with certain kinds of apps, including Rails and PHP. This is one of the reasons why we've discussed having a kind of stripped-down

Re: [HACKERS] Improvement of checkpoint IO scheduler for stable transaction responses

2013-07-04 Thread Joshua D. Drake
On 07/04/2013 06:05 AM, Andres Freund wrote: Presumably the smaller segsize is better because we don't completely stall the system by submitting up to 1GB of io at once. So, if we were to do it in 32MB chunks and then do a final fsync() afterwards we might get most of the benefits. Yes, I try

Re: [HACKERS] Documentation/help for materialized and recursive views

2013-07-01 Thread Joshua D. Drake
should not push this responsibility off on pgadmin as pgadmin is not part of PostgreSQL but a third party tool. The standard postgresql client is psql (for good or bad) and we should support psql fully on all platforms. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http

Re: [HACKERS] vacuumlo - use a cursor

2013-06-29 Thread Joshua D. Drake
On 06/29/2013 08:35 AM, Bruce Momjian wrote: On Sat, Jun 29, 2013 at 11:33:54AM -0400, Andrew Dunstan wrote: Nobody seemed interested. But I do think it's a good idea still. Well, if no one replied, and you thought it was a good idea, then it was a good idea. ;-) I think it is a good

Re: [HACKERS] C++ compiler

2013-06-25 Thread Joshua D. Drake
On 06/24/2013 09:16 PM, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: Right. I don't think there are any C features we want to avoid; are there any? We're avoiding C99-and-later features that are not in C89, such as // for comments, as well as more useful things. It might be time

Re: [HACKERS] Kudos for Reviewers -- straw poll

2013-06-25 Thread Joshua D. Drake
On 06/25/2013 10:17 AM, Josh Berkus wrote: Hackers, I'd like to take a straw poll here on how we should acknowledge reviewers. Please answer the below with your thoughts, either on-list or via private email. How should reviewers get credited in the release notes? a) not at all b) in a

Re: [HACKERS] Kudos for Reviewers -- straw poll

2013-06-25 Thread Joshua D. Drake
On 06/25/2013 11:26 AM, Andres Freund wrote: On 2013-06-25 11:04:38 -0700, Joshua D. Drake wrote: a) not at all b) in a single block titled Reviewers for this version at the bottom. c) on the patch they reviewed, for each patch C. The idea that reviewers are somehow less than authors

Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List

2013-06-24 Thread Joshua D. Drake
On 06/24/2013 08:40 AM, Maciej Gajewski wrote: Maybe this policy should be mentioned on the Wiki, so newbies like myself (who wouldn't even dare reviewing patches submitted be seasoned hackers) are not surprised by seeing own name on a shame wall? It is mentioned. Of course now I can't find

Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List

2013-06-24 Thread Joshua D. Drake
On 06/24/2013 10:10 AM, Josh Berkus wrote: On 06/24/2013 10:02 AM, Dimitri Fontaine wrote: Josh Berkus j...@agliodbs.com writes: patch. The vast majority chose not to respond to my email to them at all. When private email fails, the next step is public email. The only problem I have here

Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List

2013-06-24 Thread Joshua D. Drake
On 06/24/2013 10:22 AM, Josh Berkus wrote: Mind you, we wouldn't be able to reward a few reviewers, because they live in countries to which it's impossible to ship from abroad. I have previously proposed that all of the reviewers of a given PostgreSQL release be honored in the release notes

Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List

2013-06-24 Thread Joshua D. Drake
On 06/24/2013 10:48 AM, Claudio Freire wrote: Reviewer recognition should be on the same level as the submitter. The problem with that is that that HUGELY depends on the patch and the review. There are patches where reviewers do a good percentage of the work and others where they mostly tell

Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List

2013-06-24 Thread Joshua D. Drake
On 06/24/2013 10:59 AM, Andres Freund wrote: On 2013-06-24 10:50:42 -0700, Josh Berkus wrote: The problem with that is that that HUGELY depends on the patch and the review. There are patches where reviewers do a good percentage of the work and others where they mostly tell that compiles

Re: [HACKERS] C++ compiler

2013-06-24 Thread Joshua D. Drake
On 06/24/2013 04:59 PM, Bruce Momjian wrote: On Thu, Jun 20, 2013 at 12:45:48PM +0800, Craig Ringer wrote: I see value in making the codebase compileable with g++... and down the track I can see being able to use basic class features as quite useful given Pg's fairly OO internal design.

Re: [HACKERS] C++ compiler

2013-06-24 Thread Joshua D. Drake
On 06/24/2013 05:37 PM, Bruce Momjian wrote: On Mon, Jun 24, 2013 at 09:21:26PM -0300, Claudio Freire wrote: On Mon, Jun 24, 2013 at 9:19 PM, Joshua D. Drake j...@commandprompt.com wrote: I think the big question is whether you can _control_ what C++ features are used, or whether you

Re: [HACKERS] Hardware donation

2013-06-21 Thread Joshua D. Drake
On 06/21/2013 09:48 AM, Jim Nasby wrote: We've got some recently decommissioned servers and Enova is willing to donate 2 of them to the community. There's nothing terribly spectacular about the servers except for memory. We have one 512G server available and the other would be either 192G or

Re: [HACKERS] Bad error message on valuntil

2013-06-19 Thread Joshua D. Drake
On 06/19/2013 08:24 AM, Peter Eisentraut wrote: I think it's intentional that we don't tell the *client* that level of detail. I could see emitting a log message about it, but it's not clear whether that will help an unsophisticated user. Usually, when I log in somewhere and the password is

Re: [HACKERS] Change authentication error message (patch)

2013-06-19 Thread Joshua D. Drake
On 06/18/2013 02:25 AM, Markus Wanner wrote: On 06/16/2013 06:02 PM, Joshua D. Drake wrote: Instead of pushing extra info to the logs I decided that we could without giving away extra details per policy. I wrote the error message in a way that tells the most obvious problems, without

Re: [HACKERS] Change authentication error message (patch)

2013-06-19 Thread Joshua D. Drake
On 06/19/2013 01:18 PM, Markus Wanner wrote: Authentication failed or password has expired for user \%s\ Authentication failed covers any combination of a username/password being wrong and obviously password expired covers the other. Works for me. Considering the password to be the thing

[HACKERS] Change authentication error message (patch)

2013-06-16 Thread Joshua D. Drake
Hello, Instead of pushing extra info to the logs I decided that we could without giving away extra details per policy. I wrote the error message in a way that tells the most obvious problems, without admitting to any of them. Please see attached: diff --git a/src/backend/libpq/auth.c

Re: [HACKERS] Hard to Use WAS: Hard limit on WAL space

2013-06-15 Thread Joshua D. Drake
On 06/14/2013 11:18 PM, Craig Ringer wrote: On 06/15/2013 02:08 PM, Brendan Jurd wrote: On 15 June 2013 14:43, Craig Ringer cr...@2ndquadrant.com wrote: The #1 question I see on Stack Overflow has to be confusion about pg_hba.conf, mostly from people who have no idea it exists, don't

Re: [HACKERS] Hard to Use WAS: Hard limit on WAL space

2013-06-15 Thread Joshua D. Drake
On 06/14/2013 11:44 PM, Brendan Jurd wrote: If they see something called 'pg_hba.conf', they may very reasonably assume that it is some internal/advanced stuff that they don't need to worry about just yet, because what the heck is a 'pg_hba'? The 'pg' Only the uneducated. Look, I am not

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-06-14 Thread Joshua D. Drake
On 06/14/2013 10:11 AM, David Fetter wrote: ok, thanks, I will wait. Hi Joe, Do you have some time in the weekend to help me submit the patch? Thanks, Liming Liming, Is your git skill good enough to create a patch vs. PostgreSQL's git master? If so, send that and once it's hit the

[HACKERS] another error perhaps to be enhanced

2013-06-14 Thread Joshua D. Drake
ERROR: index foo_idx We should probably add the schema. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image

Re: [HACKERS] another error perhaps to be enhanced

2013-06-14 Thread Joshua D. Drake
On 06/14/2013 10:47 AM, Peter Geoghegan wrote: I think you'll need to better describe what you mean here. postgres=# create schema foo; CREATE SCHEMA postgres=# create schema bar; CREATE SCHEMA postgres=# create table foo.foo(id serial); NOTICE: CREATE TABLE will create implicit sequence

Re: [HACKERS] another error perhaps to be enhanced

2013-06-14 Thread Joshua D. Drake
On 06/14/2013 11:01 AM, Peter Geoghegan wrote: On Fri, Jun 14, 2013 at 10:54 AM, Joshua D. Drake j...@commandprompt.com wrote: Now, with the error previously shown, which one_idx needs to be reindexed? Well, you didn't show an actual error message. ERROR: index foo_idx Is not an error

Re: [HACKERS] Hard to Use WAS: Hard limit on WAL space

2013-06-14 Thread Joshua D. Drake
On 06/14/2013 11:16 AM, Josh Berkus wrote: On 06/12/2013 02:03 PM, Joshua D. Drake wrote: What concerns me is we seem to be trying to make this easy. It isn't supposed to be easy. This is hard stuff. Smart people built it and it takes a smart person to run it. When did it become a bad thing

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Joshua D. Drake
On 06/14/2013 06:56 PM, Robert Haas wrote: On Fri, Jun 14, 2013 at 8:45 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-14 17:35:02 -0700, Josh Berkus wrote: No. I think as long as we only have pglz and one new algorithm (even if that is lz4 instead of the current snappy) we

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-12 Thread Joshua D. Drake
On 06/12/2013 08:49 AM, Robert Haas wrote: Sure, remote archiving is great, and I'm glad you've been working on it. In general, I think that's a cleaner approach, but there are still enough people using archive_command that we can't throw them under the bus. Correct. I guess archiving

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-10 Thread Joshua D. Drake
On 06/10/2013 04:42 PM, Josh Berkus wrote: Actually we describe what archive_command needs to fulfill, and tell them to use something that accomplishes that. The example with cp is explicitly given as an example, not a recommendation. If we offer cp as an example, we *are* recommending it.

Re: [HACKERS] small patch to crypt.c

2013-06-09 Thread Joshua D. Drake
On 06/09/2013 09:28 AM, Tom Lane wrote: Even aside from that, the proposed change seems like a bad idea because it introduces an unnecessary call of GetCurrentTimestamp() in the common case where there's no valuntil limit. On some platforms that call is pretty slow. And that would explain

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Joshua D. Drake
On 06/07/2013 12:14 PM, Josh Berkus wrote: Right now, what we're telling users is You can have continuous backup with Postgres, but you'd better hire and expensive consultant to set it up for you, or use this external tool of dubious provenance which there's no packages for, or you might

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Joshua D. Drake
On 06/08/2013 07:36 AM, MauMau wrote: 1. If the machine or postgres crashes while archive_command is copying a WAL file, later archive recovery fails. This is because cp leaves a file of less than 16MB in archive area, and postgres refuses to start when it finds such a small archive WAL file.

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Joshua D. Drake
not OBTUSE error. Obviously this could cause a ton of transactions to roll back but I think keeping the database consistent and rolling back a transaction in case of error is exactly what we are supposed to do. Sincerely, Joshua D. Drake - Heikki -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Bad error message on valuntil

2013-06-08 Thread Joshua D. Drake
On 06/07/2013 12:31 PM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail. Why? That seems rather silly. The general policy on authentication failure reports

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Joshua D. Drake
On 06/08/2013 11:27 AM, Andres Freund wrote: On 2013-06-08 11:15:40 -0700, Joshua D. Drake wrote: To me, a more pragmatic approach makes sense. Obviously having some kind of code that checks the space makes sense but I don't know that it needs to be around any operation other than we

[HACKERS] small patch to crypt.c

2013-06-08 Thread Joshua D. Drake
Hello, In my quest to understand how all the logging etc works with authentication I came across the area of crypt.c that checks for valid_until but it seems like it has an extraneous check. If I am wrong I apologize for the noise but wouldn't mind an explanation. index f01d904..8d809b2

Re: [HACKERS] small patch to crypt.c

2013-06-08 Thread Joshua D. Drake
On 06/08/2013 08:47 PM, Stephen Frost wrote: JD, * Joshua D. Drake (j...@commandprompt.com) wrote: In my quest to understand how all the logging etc works with authentication I came across the area of crypt.c that checks for valid_until but it seems like it has an extraneous check. If I am

[HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
authentication. It was because the valuntil on the user had been set till a date in the past. Now technically if we just removed the word password from the error it would be accurate but it seems it would be better to say, FATAL: the user user has expired. Sincerely, Joshua D. Drake -- Command Prompt

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
On 06/07/2013 11:57 AM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: I had a customer pulling their hair out today because they couldn't login to their system. The error was consistently: 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password authentication

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
On 06/07/2013 12:31 PM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail. Why? That seems rather silly. The general policy on authentication failure reports

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
On 06/07/2013 01:41 PM, David Johnston wrote: Please check server log for specifics is not a good message for something sent to a client that in many normal situation would have no access to said logs. I don't agree. The user doesn't need access to the logs. If they get that error they

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-06 Thread Joshua D. Drake
On 6/5/2013 10:54 PM, Peter Geoghegan wrote: On Wed, Jun 5, 2013 at 10:27 PM, Joshua D. Drake j...@commandprompt.com wrote: I just wonder if we are looking in the right place (outside of some obvious badness like the PANIC running out of disk space). So you don't think we should PANIC

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-06 Thread Joshua D. Drake
On 6/5/2013 11:09 PM, Daniel Farina wrote: Instead of running out of disk space PANIC we should just write to an emergency location within PGDATA and log very loudly that the SA isn't paying attention. Perhaps if that area starts to get to an unhappy place we immediately bounce into read-only

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-06 Thread Joshua D. Drake
On 6/5/2013 11:25 PM, Harold Giménez wrote: Instead of running out of disk space PANIC we should just write to an emergency location within PGDATA This merely buys you some time, but with aggressive and sustained write throughput you are left on the same spot. Practically speaking

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-06 Thread Joshua D. Drake
On 6/5/2013 11:31 PM, Peter Geoghegan wrote: On Wed, Jun 5, 2013 at 11:28 PM, Joshua D. Drake j...@commandprompt.com wrote: I have zero doubt that in your case it is true and desirable. I just don't know that it is a positive solution to the problem as a whole. Your case is rather limited

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-06 Thread Joshua D. Drake
On 6/6/2013 1:11 AM, Heikki Linnakangas wrote: (I'm sure you know this, but:) If you perform a checkpoint as fast and short as possible, the sudden burst of writes and fsyncs will overwhelm the I/O subsystem, and slow down queries. That's what we saw before spread checkpoints: when a

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-06 Thread Joshua D. Drake
On 06/06/2013 09:30 PM, Jeff Janes wrote: Archiving - In some ways, this is the simplest case. Really, we just need a way to know when the available WAL space has become 90% full, and abort archiving at that stage. Once we stop attempting to archive, we can

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-05 Thread Joshua D. Drake
On 06/05/2013 05:37 PM, Robert Haas wrote: On Wed, Jun 5, 2013 at 3:24 PM, Fujii Masao masao.fu...@gmail.com wrote: OTOH, if we use max_wal_size as a hard limit, we can avoid such PANIC error and long down time. Of course, in this case, once max_wal_size is reached, we cannot complete any

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-05 Thread Joshua D. Drake
On 06/05/2013 05:37 PM, Robert Haas wrote: - If it looks like we're going to exceed limit #3 before the checkpoint completes, we start exerting back-pressure on writers by making them wait every time they write WAL, probably in proportion to the number of bytes written. We keep ratcheting up

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-05 Thread Joshua D. Drake
On 06/05/2013 06:23 PM, Daniel Farina wrote: On Wed, Jun 5, 2013 at 6:00 PM, Joshua D. Drake j...@commandprompt.com wrote: I didn't see that proposal, link? Because the idea of slowing down wal-writing sounds insane. It's not as insane as introducing an archiving gap, PANICing and crashing

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-05 Thread Joshua D. Drake
On 6/5/2013 10:07 PM, Daniel Farina wrote: If I told you there were some of us who would prefer to attenuate the rate that things get written rather than cancel or delay archiving for a long period of time, would that explain the framing of the problem? I understand that based on what you

Re: [HACKERS] pg_rewind, a tool for resynchronizing an old master after failover

2013-06-04 Thread Joshua D. Drake
On 06/04/2013 01:55 PM, Josh Berkus wrote: That seems rather like a catch-22 Bruce. If they don't check with the legal department, it's dangerous, but if they do check, it's dangerous? Presumably if they checked with the legal department, it's cleared. We should be wary of stuff contributed

Re: [HACKERS] units in postgresql.conf comments

2013-05-30 Thread Joshua D. Drake
On 05/30/2013 12:01 AM, Heikki Linnakangas wrote: We could make it mandatory to specify the unit in the value. Ie. throw an error on wal_sender_timeout = 50: ERROR: unit required for option wal_sender_timeout HINT: Valid units for this parameter are ms, s, min, h, and d. Then you wouldn't

Re: [HACKERS] units in postgresql.conf comments

2013-05-30 Thread Joshua D. Drake
On 05/30/2013 12:55 AM, Magnus Hagander wrote: I like this idea with one addition. We should have a default unit for each. For wal_sender_timeout seconds makes sense, but for checkpoint_timeout minutes makes sense (for example). This sounds like a good way to make things even more confusing.

Re: [HACKERS] units in postgresql.conf comments

2013-05-30 Thread Joshua D. Drake
On 05/30/2013 01:14 AM, Heikki Linnakangas wrote: On 30.05.2013 10:52, Joshua D. Drake wrote: On 05/30/2013 12:01 AM, Heikki Linnakangas wrote: We could make it mandatory to specify the unit in the value. Ie. throw an error on wal_sender_timeout = 50: ERROR: unit required for option

Re: [HACKERS] Planning incompatibilities for Postgres 10.0

2013-05-28 Thread Joshua D. Drake
On 05/28/2013 08:36 AM, Hannu Krosing wrote: The conversation does not change. Further, we are not Firefox. We are not user software. We are developer software. At least some of the real-world problems with PostgreSQL comes from We are developer software mentality. Yes, We are developer

Re: [HACKERS] Planning incompatibilities for Postgres 10.0

2013-05-28 Thread Joshua D. Drake
, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart

Re: [HACKERS] Planning incompatibilities for Postgres 10.0

2013-05-28 Thread Joshua D. Drake
On 05/28/2013 03:36 PM, Bruce Momjian wrote: The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Well, you could do SET TRANSACTION READ ONLY, and that

Re: [HACKERS] Planning incompatibilities for Postgres 10.0

2013-05-28 Thread Joshua D. Drake
On 05/28/2013 04:05 PM, Bruce Momjian wrote: On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote: On 05/28/2013 03:36 PM, Bruce Momjian wrote: The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time

Re: [HACKERS] [GENERAL] pg_upgrade -u

2013-05-28 Thread Joshua D. Drake
On 05/28/2013 07:55 PM, Bruce Momjian wrote: Perhaps just documenting the behavior is all that is needed, but -U is everywhere and I think that's a good thing. [ moved to hacker ] Wow, I never realized other tools used -U for user, instead of -u. Should I change pg_upgrade to use -U for

Re: [HACKERS] Planning incompatibilities for Postgres 10.0

2013-05-27 Thread Joshua D. Drake
On 05/27/2013 04:58 PM, Craig Ringer wrote: On 05/28/2013 12:41 AM, Simon Riggs wrote: I'm happy with that. I was also thinking about collecting changes not related just to disk format, if any exist. Any wire protocol or syntax changes? I can't seem to find a things we want to do in wire

Re: [HACKERS] Planning incompatibilities for Postgres 10.0

2013-05-27 Thread Joshua D. Drake
: I have X problem with PostgreSQL CMD: What version? Client: 9 CMD: Which version of 9? Client: 9.0.2 CMD: You should be running 10.0.5 or at least 9.0.13 The conversation does not change. Further, we are not Firefox. We are not user software. We are developer software. Sincerely, Joshua D

Re: [HACKERS] Remaining beta blockers

2013-05-06 Thread Joshua D. Drake
On 05/06/2013 08:17 AM, Tom Lane wrote: Per my other mail, I think adding an AMV option at this time is inadvisable. I could go either way on removing or keeping the is_scannable function --- anybody else have an opinion on that point? Which of us is going to commit this? We're running low

Re: [HACKERS] Documentation epub format

2013-05-01 Thread Joshua D. Drake
On 05/01/2013 09:27 AM, Fabien COELHO wrote: Hello devs, I've given a try to the PostgreSQL documentation in epub format. I must admit that there is a bit of a disappointement as far as the user experience is concerned: the generated file is barely usable on an iPad2 with the default iBooks

Re: [HACKERS] Documentation epub format

2013-05-01 Thread Joshua D. Drake
On 05/01/2013 10:52 AM, Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: Once upon a time we had multiple books as documentation, then at some point we merged them. It was quite a few years ago. I would agree at this point that we need to consider breaking them up again

Re: [HACKERS] Documentation epub format

2013-05-01 Thread Joshua D. Drake
On 05/01/2013 10:56 AM, Andrew Satori wrote: I would second Tom on this, and if ePub is really a longer term goal of the documentation, the various eBook formats have differing levels of support for hyperlinking that would merit retaining everything in a single book that can be linked from

Re: [HACKERS] Proposal to add --single-row to psql

2013-04-25 Thread Joshua D. Drake
software. Considering that the earliest any such thing could reach the field would be 9.4, it seems not unlikely that the need for it would be gone by next year anyway. +1 this is really an amazon problem not a postgresql problem. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http

Re: [HACKERS] 9.3 release notes suggestions

2013-04-25 Thread Joshua D. Drake
On 04/25/2013 04:48 PM, Daniel Farina wrote: On Wed, Apr 24, 2013 at 6:30 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Apr 23, 2013 at 11:41 PM, Bruce Momjian br...@momjian.us wrote: Thanks for the many suggestions on improving the 9.3 release notes. There were many ideas I would

Re: [HACKERS] page 1 of relation global/11787 was uninitialized

2013-04-09 Thread Joshua D. Drake
for that parameter. http://www.postgresql.org/docs/9.0/static/runtime-config-wal.html#GUC-FULL-PAGE-WRITES Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion

Re: [HACKERS] pg_dump/restore syntax checking bug?

2013-03-23 Thread Joshua D. Drake
On 03/22/2013 10:13 PM, Josh Kupershmidt wrote: On Fri, Mar 22, 2013 at 9:35 PM, Joshua D. Drake j...@commandprompt.com wrote: postgres@jd-laptop:~$ pg_restore -d test -P 'by(),hello()' foo.sqlc Note, the pg_restore doc makes no mention of trying to squeeze multiple function prototypes

[HACKERS] pg_dump/restore syntax checking bug?

2013-03-22 Thread Joshua D. Drake
Hello, In testing some pg_restore functionality I found the following: postgres@jd-laptop:~$ pg_dump -U postgres -Fc -s --file=foo.sqlc postgres@jd-laptop:~$ dropdb test; postgres@jd-laptop:~$ createdb test; postgres@jd-laptop:~$ pg_restore -d test -P 'by()' foo.sqlc postgres@jd-laptop:~$ psql

Re: [HACKERS] REFRESH MATERIALIZED VIEW locklevel

2013-03-08 Thread Joshua D. Drake
On 03/08/2013 10:09 AM, Merlin Moncure wrote: On Fri, Mar 8, 2013 at 11:59 AM, Josh Berkus j...@agliodbs.com wrote: Andres, Further, we get pretty much one and only one chance to promote a new major feature, which is when that feature is first introduced. Improving the feature in the next

Re: [HACKERS] Enabling Checksums

2013-03-06 Thread Joshua D. Drake
On 03/06/2013 03:06 PM, Robert Haas wrote: On Wed, Mar 6, 2013 at 6:00 PM, Josh Berkus j...@agliodbs.com wrote: We've had a few EnterpriseDB customers who have had fantastically painful experiences with PostgreSQL + ZFS. Supposedly, aligning the ZFS block size to the PostgreSQL block size is

Re: [HACKERS] transforms

2013-03-05 Thread Joshua D. Drake
On 03/05/2013 02:52 PM, Josh Berkus wrote: plperlh=# \c postgres You are now connected to database postgres as user josh. postgres=# create extension hstore_plperl; ERROR: could not load library /home/josh/pg93/lib/postgresql/hstore_plperl.so: /home/josh/pg93/lib/postgresql/hstore_plperl.so:

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Joshua D. Drake
On 02/13/2013 09:54 AM, Simon Riggs wrote: I'd call it out as a marketing name. I guess it's fractal in the sense that all levels of the tree can hold leaf tuples in the buffers; the structure looks the same no matter how deep you zoom, like a fractal.. But Buffered would be more appropriate

Re: [HACKERS] Too frequent checkpoints ?

2013-02-08 Thread Joshua D. Drake
On 02/08/2013 02:37 AM, Pavan Deolasee wrote: I wonder if this is all expected. The database is getting ZERO activity. There are no connections open at this time. The checkpoints are happening at every 30 seconds and new WAL files are being created, AFAIK because the old ones are getting

Re: [HACKERS] Considering Gerrit for CFs

2013-02-06 Thread Joshua D. Drake
On 02/06/2013 01:53 PM, Tom Lane wrote: ... if it's going to try to coerce us out of our email-centric habits, then I for one am very much against it. To me, the problems with the existing CF app are precisely that it's not well enough integrated with the email discussions. The way to fix

Re: [HACKERS] CF3+4 (was Re: Parallel query execution)

2013-01-23 Thread Joshua D. Drake
On 01/23/2013 09:51 AM, Josh Berkus wrote: The only way to fix increasing bug counts is through more-comprehensive regular testing. Currently we have regression/unit tests which cover maybe 30% of our code. Performance testing is largely ad-hoc. We don't require comprehensive acceptance

Re: [HACKERS] logical changeset generation v4 - Heikki's thoughts about the patch state

2013-01-23 Thread Joshua D. Drake
On 01/23/2013 05:17 PM, Robert Haas wrote: Of course, I have no evidence that that will happen. But it is a really big piece of code, and therefore unless you are superman, it's probably got a really large number of bugs. The scary thing is that it is not as if we can say, well, this is a

[HACKERS] LLVM / CLang / PostgreSQL

2013-01-11 Thread Joshua D. Drake
Hello, Has anyone played with this? Seen any results? It looks like most testing is being done on Mac OSX (via buildfarm). JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion,

Re: [HACKERS] dynamic SQL - possible performance regression in 9.2

2013-01-04 Thread Joshua D. Drake
will have people not upgrade to 9.2 specifically because of this problem. Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579

<    1   2   3   4   5   6   7   8   9   10   >