Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-23 Thread Fujii Masao
On Sun, Jan 23, 2011 at 8:33 PM, Magnus Hagander wrote: > I've committed this with the current name, pg_basebackup Great! But, per subsequent commit logs, I should have reviewed more about portability issues :( Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-23 Thread Magnus Hagander
On Thu, Jan 20, 2011 at 17:17, Magnus Hagander wrote: > On Thu, Jan 20, 2011 at 16:45, Bruce Momjian wrote: >> Do we envision pg_basebackup as something we will enahance, and if so, >> should we consider a generic name? > > Well, it's certainly going to be enhanced. I think there are two main > u

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-21 Thread Tom Lane
Fujii Masao writes: > Probably true. But we cannot create a tablespace for root-owned directory. > The directory must be owned by the PostgreSQL system user. So ISTM that > you says that creating a tablespace on a mount point itself is a security > hole. Generally, the root user would have to mo

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-21 Thread Magnus Hagander
On Fri, Jan 21, 2011 at 07:02, Fujii Masao wrote: > On Fri, Jan 21, 2011 at 1:00 AM, Tom Lane wrote: >> Fujii Masao writes: >>> On Thu, Jan 20, 2011 at 10:53 AM, Tom Lane wrote: >> In the case you sketch, there would be nothing to stop the (non root) >> postgres user from renaming $PGDATA/mnt t

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-21 Thread Fujii Masao
On Wed, Jan 19, 2011 at 1:12 PM, Fujii Masao wrote: >>> +               r = PQgetCopyData(conn, ©buf, 0); >>> +               if (r == -1) >>> >>> Since -1 of PQgetCopyData might indicate an error, in this case, >>> we would need to call PQgetResult?. >> >> Uh, -1 means end of data, no? -2 means e

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Fujii Masao
On Fri, Jan 21, 2011 at 1:00 AM, Tom Lane wrote: > Fujii Masao writes: >> On Thu, Jan 20, 2011 at 10:53 AM, Tom Lane wrote: I'm not sure why that's the right solution. Why do you think that we should not create the tablespace under the $PGDATA directory? I'm not surprised that peo

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Tom Lane
Robert Haas writes: > On Thu, Jan 20, 2011 at 2:10 PM, Dimitri Fontaine > wrote: >> Robert Haas writes: >>> I think that the basic problem with wal_level is that to increase it >>> you need to somehow ensure that all the backends have the new setting, >>> and then checkpoint. >> >> Well, you ju

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Heikki Linnakangas
On 20.01.2011 22:15, Dimitri Fontaine wrote: Robert Haas writes: Sending them a signal seems like a promising approach, but the trick is guaranteeing that they've actually acted on it before you start the checkpoint. How much using a latch here would help? Or be overkill? A latch doesn't g

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Dimitri Fontaine
Robert Haas writes: > Sending them a signal seems like a promising approach, but the trick > is guaranteeing that they've actually acted on it before you start the > checkpoint. How much using a latch here would help? Or be overkill? -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : E

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Robert Haas
On Thu, Jan 20, 2011 at 2:10 PM, Dimitri Fontaine wrote: > Robert Haas writes: >> I think that the basic problem with wal_level is that to increase it >> you need to somehow ensure that all the backends have the new setting, >> and then checkpoint.  Right now, the backends get the value through >

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Dimitri Fontaine
Robert Haas writes: > I think that the basic problem with wal_level is that to increase it > you need to somehow ensure that all the backends have the new setting, > and then checkpoint. Right now, the backends get the value through > the GUC machinery, and so there's no particular bound on how l

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Robert Haas
On Thu, Jan 20, 2011 at 11:59 AM, Dimitri Fontaine wrote: > Robert Haas writes: >> Also, it won't actually work unless the server has replication >> configured (wal_level!=minimal, max_wal_senders>0, and possibly some >> setting for wal_keep_segments), which has been the main point of the >> nami

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Dimitri Fontaine
Robert Haas writes: > Also, it won't actually work unless the server has replication > configured (wal_level!=minimal, max_wal_senders>0, and possibly some > setting for wal_keep_segments), which has been the main point of the > naming discussion thus far. Now, you know what would be REALLY cool?

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Magnus Hagander
On Thu, Jan 20, 2011 at 16:45, Bruce Momjian wrote: > Robert Haas wrote: >> > I read the description as: >> > >> > + ? ?You can also use the tool to take >> > + ? ?the backup, instead of manually copying the files. This tool will take >> > + ? ?care of the pg_start_backup(), copy and >> > + ? ?pg

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Tom Lane
Fujii Masao writes: > On Thu, Jan 20, 2011 at 10:53 AM, Tom Lane wrote: >>> I'm not sure why that's the right solution. Why do you think that we should >>> not create the tablespace under the $PGDATA directory? I'm not surprised >>> that people mounts the filesystem on $PGDATA/mnt and creates the

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Bruce Momjian
Robert Haas wrote: > > I read the description as: > > > > + ? ?You can also use the tool to take > > + ? ?the backup, instead of manually copying the files. This tool will take > > + ? ?care of the pg_start_backup(), copy and > > + ? ?pg_stop_backup() steps automatically, and transfers the > > + ?

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Robert Haas
On Thu, Jan 20, 2011 at 10:15 AM, Bruce Momjian wrote: > Robert Haas wrote: >> On Thu, Jan 20, 2011 at 10:01 AM, Bruce Momjian wrote: >> > Magnus Hagander wrote: >> >> On Mon, Jan 17, 2011 at 16:27, Simon Riggs wrote: >> >> > On Mon, 2011-01-17 at 16:20 +0100, Magnus Hagander wrote: >> >> >> On

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Bruce Momjian
Robert Haas wrote: > On Thu, Jan 20, 2011 at 10:01 AM, Bruce Momjian wrote: > > Magnus Hagander wrote: > >> On Mon, Jan 17, 2011 at 16:27, Simon Riggs wrote: > >> > On Mon, 2011-01-17 at 16:20 +0100, Magnus Hagander wrote: > >> >> On Mon, Jan 17, 2011 at 16:18, Robert Haas > >> >> wrote: > >> >

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Robert Haas
On Thu, Jan 20, 2011 at 10:01 AM, Bruce Momjian wrote: > Magnus Hagander wrote: >> On Mon, Jan 17, 2011 at 16:27, Simon Riggs wrote: >> > On Mon, 2011-01-17 at 16:20 +0100, Magnus Hagander wrote: >> >> On Mon, Jan 17, 2011 at 16:18, Robert Haas wrote: >> >> > On Mon, Jan 17, 2011 at 8:55 AM, Mag

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Bruce Momjian
Magnus Hagander wrote: > On Mon, Jan 17, 2011 at 16:27, Simon Riggs wrote: > > On Mon, 2011-01-17 at 16:20 +0100, Magnus Hagander wrote: > >> On Mon, Jan 17, 2011 at 16:18, Robert Haas wrote: > >> > On Mon, Jan 17, 2011 at 8:55 AM, Magnus Hagander > >> > wrote: > >> >> Hmm. I don't like those n

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Magnus Hagander
On Thu, Jan 20, 2011 at 12:42, Magnus Hagander wrote: > On Thu, Jan 20, 2011 at 05:23, Fujii Masao wrote: >> It's helpful to document what to set to allow pg_basebackup connection. >> That is not only the REPLICATION privilege but also max_wal_senders and >> pg_hba.conf. > > Hmm. Yeha, i guess th

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-20 Thread Magnus Hagander
On Thu, Jan 20, 2011 at 05:23, Fujii Masao wrote: > On Wed, Jan 19, 2011 at 9:37 PM, Magnus Hagander wrote: >>> Great. Thanks for the quick update! >>> >>> Here are another comments: > > Here are comments against the documents. The other code looks good. Thanks! > It's helpful to document what

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-19 Thread Fujii Masao
On Wed, Jan 19, 2011 at 9:37 PM, Magnus Hagander wrote: >> Great. Thanks for the quick update! >> >> Here are another comments: Here are comments against the documents. The other code looks good. It's helpful to document what to set to allow pg_basebackup connection. That is not only the REPLIC

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-19 Thread Fujii Masao
On Thu, Jan 20, 2011 at 10:53 AM, Tom Lane wrote: >> I'm not sure why that's the right solution. Why do you think that we should >> not create the tablespace under the $PGDATA directory? I'm not surprised >> that people mounts the filesystem on $PGDATA/mnt and creates the >> tablespace on it. > >

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-19 Thread Tom Lane
Fujii Masao writes: > On Thu, Jan 20, 2011 at 2:21 AM, Tom Lane wrote: >> Fujii Masao writes: >>> What I'm worried about is the case where a tablespace is created >>> under the $PGDATA directory. >> What would be the sense of that?  If you're concerned about whether the >> code handles it corre

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-19 Thread Fujii Masao
On Thu, Jan 20, 2011 at 2:21 AM, Tom Lane wrote: > Fujii Masao writes: >> What I'm worried about is the case where a tablespace is created >> under the $PGDATA directory. > > What would be the sense of that?  If you're concerned about whether the > code handles it correctly, maybe the right solut

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-19 Thread Fujii Masao
On Wed, Jan 19, 2011 at 9:37 PM, Magnus Hagander wrote: >> The "fast or slow" seems to lead users to always choose "fast". Instead, >> what about "fast or smooth", "fast or spread" or "immediate or delayed"? > > Hmm. "fast or spread" seems reasonable to me. And I want to use "fast" > for the fast

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-19 Thread Tom Lane
Fujii Masao writes: > What I'm worried about is the case where a tablespace is created > under the $PGDATA directory. What would be the sense of that? If you're concerned about whether the code handles it correctly, maybe the right solution is to add code to CREATE TABLESPACE to disallow it.

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-19 Thread Magnus Hagander
On Wed, Jan 19, 2011 at 06:14, Fujii Masao wrote: > On Wed, Jan 19, 2011 at 4:12 AM, Magnus Hagander wrote: >> Ah, ok. I've added the errorcode now, PFA. I also fixed an error in >> the change for result codes I broke in the last patch. github branch >> updated as usual. > > Great. Thanks for the

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Fujii Masao
On Wed, Jan 19, 2011 at 4:12 AM, Magnus Hagander wrote: > Ah, ok. I've added the errorcode now, PFA. I also fixed an error in > the change for result codes I broke in the last patch. github branch > updated as usual. Great. Thanks for the quick update! Here are another comments: + * IDENTIFICAT

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Fujii Masao
On Tue, Jan 18, 2011 at 8:40 PM, Magnus Hagander wrote: >> When I untar the tar file taken by pg_basebackup, I got the following >> messages: >> >>    $ tar xf base.tar >>    tar: Skipping to next header >>    tar: Archive contains obsolescent base-64 headers >>    tar: Error exit delayed from pre

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Simon Riggs
On Tue, 2011-01-18 at 18:03 +0100, Magnus Hagander wrote: > So it'd be pg_receive_wal and pg_receive_base_backup then? OK for me. Maybe even pg_receive_wal_stream Don't see any reason why command names can't be long. We have many function names already that long. -- Simon Riggs htt

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Alvaro Herrera
Excerpts from Magnus Hagander's message of mar ene 18 11:53:55 -0300 2011: > On Tue, Jan 18, 2011 at 15:49, Alvaro Herrera > wrote: > > Excerpts from Magnus Hagander's message of mar ene 18 10:47:03 -0300 2011: > > > >> Ok, thanks for clarifying. I've updated to use strerror(). Guess it's > >> tim

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Robert Haas
On Tue, Jan 18, 2011 at 12:03 PM, Magnus Hagander wrote: > So it'd be pg_receive_wal and pg_receive_base_backup then? Votes from > others? (it's easy to rename so far, so I'll keep plugging away under > the name pg_basebackup based on Fujii-sans comments until such a time > as we have a reasonable

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Cédric Villemain
2011/1/18 Magnus Hagander : > On Tue, Jan 18, 2011 at 17:31, Tom Lane wrote: >> Magnus Hagander writes: > Actually, after some IM chats, I think pg_streamrecv should be > renamed, probably to pg_walstream (or pg_logstream, but pg_walstream > is a lot more specific than that) >> p

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Magnus Hagander
On Tue, Jan 18, 2011 at 17:31, Tom Lane wrote: > Magnus Hagander writes: Actually, after some IM chats, I think pg_streamrecv should be renamed, probably to pg_walstream (or pg_logstream, but pg_walstream is a lot more specific than that) > >>> pg_stream_log >>> pg_stream_backup >

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Tom Lane
Magnus Hagander writes: >>> Actually, after some IM chats, I think pg_streamrecv should be >>> renamed, probably to pg_walstream (or pg_logstream, but pg_walstream >>> is a lot more specific than that) >> pg_stream_log >> pg_stream_backup > Those seem better. > Tom, would those solve your conce

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Magnus Hagander
On Tue, Jan 18, 2011 at 15:49, Alvaro Herrera wrote: > Excerpts from Magnus Hagander's message of mar ene 18 10:47:03 -0300 2011: > >> Ok, thanks for clarifying. I've updated to use strerror(). Guess it's >> time for another patch, PFA :-) > > Thanks ...  Message nitpick: > +   if (compresslevel >

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Alvaro Herrera
Excerpts from Magnus Hagander's message of mar ene 18 10:47:03 -0300 2011: > Ok, thanks for clarifying. I've updated to use strerror(). Guess it's > time for another patch, PFA :-) Thanks ... Message nitpick: + if (compresslevel > 0) + { + fprintf(stderr, + _("%s: this bu

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Alvaro Herrera
Excerpts from Magnus Hagander's message of mar ene 18 08:40:50 -0300 2011: > On Tue, Jan 18, 2011 at 10:49, Fujii Masao wrote: > > +                                       fprintf(stderr, _("%s: could not > > write to file '%s': %m\n"), > > > > %m in fprintf is portable? > > Hmm. I just assumed

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Magnus Hagander
On Tue, Jan 18, 2011 at 12:40, Magnus Hagander wrote: > On Tue, Jan 18, 2011 at 10:49, Fujii Masao wrote: > Yeah that sounds like a good idea. Shouldn't be too hard to do (will > reuqire a backend patch as well, of course). Should we use "-f" for > fast? Though that may be an unfortunate overload

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Magnus Hagander
On Mon, Jan 17, 2011 at 16:27, Simon Riggs wrote: > On Mon, 2011-01-17 at 16:20 +0100, Magnus Hagander wrote: >> On Mon, Jan 17, 2011 at 16:18, Robert Haas wrote: >> > On Mon, Jan 17, 2011 at 8:55 AM, Magnus Hagander >> > wrote: >> >> Hmm. I don't like those names at all :( >> > >> > I agree.  

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Magnus Hagander
On Tue, Jan 18, 2011 at 10:49, Fujii Masao wrote: > On Mon, Jan 17, 2011 at 10:30 PM, Fujii Masao wrote: >> Though I haven't seen the core part of the patch (i.e., >> ReceiveTarFile, etc..) yet, >> here is the comments against others. > > Here are another comments: Thanks! These are all good and

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Magnus Hagander
On Tue, Jan 18, 2011 at 03:14, Fujii Masao wrote: > On Mon, Jan 17, 2011 at 10:50 PM, Magnus Hagander wrote: >>> +       printf(_("  -D, --pgdata=directory   receive base backup into >>> directory\n")); >>> +       printf(_("  -T, --tardir=directory    receive base backup into tar >>> files\n"

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Cédric Villemain
2011/1/18 Fujii Masao : > On Mon, Jan 17, 2011 at 10:30 PM, Fujii Masao wrote: >> Though I haven't seen the core part of the patch (i.e., >> ReceiveTarFile, etc..) yet, >> here is the comments against others. > > Here are another comments: > > > When I untar the tar file taken by pg_basebackup, I

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-18 Thread Fujii Masao
On Mon, Jan 17, 2011 at 10:30 PM, Fujii Masao wrote: > Though I haven't seen the core part of the patch (i.e., > ReceiveTarFile, etc..) yet, > here is the comments against others. Here are another comments: When I untar the tar file taken by pg_basebackup, I got the following messages: $ t

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Fujii Masao
On Mon, Jan 17, 2011 at 10:50 PM, Magnus Hagander wrote: >> +       printf(_("  -D, --pgdata=directory   receive base backup into >> directory\n")); >> +       printf(_("  -T, --tardir=directory    receive base backup into tar >> files\n" >> +                        "                            

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2011 at 20:31, Tom Lane wrote: > Magnus Hagander writes: >> Actually, after some IM chats, I think pg_streamrecv should be >> renamed, probably to pg_walstream (or pg_logstream, but pg_walstream >> is a lot more specific than that) > > What I like about streamrecv is it's fairly c

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Tom Lane
Magnus Hagander writes: > Actually, after some IM chats, I think pg_streamrecv should be > renamed, probably to pg_walstream (or pg_logstream, but pg_walstream > is a lot more specific than that) What I like about streamrecv is it's fairly clear which end of the connection it's supposed to be use

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Dimitri Fontaine
Magnus Hagander writes: > The walsender can't read pg_class for example, so it can't generate > that mapping file. I don't see any way out here. So let's call .tar good enough for now… Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Simon Riggs
On Mon, 2011-01-17 at 16:20 +0100, Magnus Hagander wrote: > On Mon, Jan 17, 2011 at 16:18, Robert Haas wrote: > > On Mon, Jan 17, 2011 at 8:55 AM, Magnus Hagander > > wrote: > >> Hmm. I don't like those names at all :( > > > > I agree. I don't think your original names are bad, as long as > > t

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2011 at 16:18, Robert Haas wrote: > On Mon, Jan 17, 2011 at 8:55 AM, Magnus Hagander wrote: >> Hmm. I don't like those names at all :( > > I agree.  I don't think your original names are bad, as long as > they're well-documented.  I sympathize with Simon's desire to make it > clea

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Robert Haas
On Mon, Jan 17, 2011 at 8:55 AM, Magnus Hagander wrote: > Hmm. I don't like those names at all :( I agree. I don't think your original names are bad, as long as they're well-documented. I sympathize with Simon's desire to make it clear that these use the replication framework, but I really don'

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Simon Riggs
On Mon, 2011-01-17 at 14:55 +0100, Magnus Hagander wrote: > >> > >> It uses the replication features for it. You also have to set > >> max_walsenders > 0, which is in the replication section of the > >> postgresql.conf file. > >> > >> The point I wanted to make downthread was that it's useful witho

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2011 at 14:49, Simon Riggs wrote: > On Mon, 2011-01-17 at 13:43 +0100, Magnus Hagander wrote: >> Downthread you say that this tool is also useful for making base >> backups >> > independent of replication functionality.  Sounds good.  But then >> the >> > documentation says that th

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2011 at 14:43, Fujii Masao wrote: > On Mon, Jan 17, 2011 at 9:43 PM, Magnus Hagander wrote: >>> Probably yes, for consistency.  I have been thinking for a while, >>> however, that it would be very good if the tools also supported a >>> conninfo string, so you don't have to invent

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2011 at 14:30, Fujii Masao wrote: > On Mon, Jan 17, 2011 at 7:14 PM, Fujii Masao wrote: >>> Oh, sorry about that. There is only one that contains postgresql though :P >>> >>> http://github.com/mhagander/postgres, branch streaming_base. >> >> Thanks! > > Though I haven't seen the c

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Simon Riggs
On Mon, 2011-01-17 at 13:43 +0100, Magnus Hagander wrote: > Downthread you say that this tool is also useful for making base > backups > > independent of replication functionality. Sounds good. But then > the > > documentation says that the connection must be with a user that has > the > > replic

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Fujii Masao
On Mon, Jan 17, 2011 at 9:43 PM, Magnus Hagander wrote: >> Probably yes, for consistency.  I have been thinking for a while, >> however, that it would be very good if the tools also supported a >> conninfo string, so you don't have to invent a new option for every new >> connection option.  psql a

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Fujii Masao
On Mon, Jan 17, 2011 at 7:14 PM, Fujii Masao wrote: >> Oh, sorry about that. There is only one that contains postgresql though :P >> >> http://github.com/mhagander/postgres, branch streaming_base. > > Thanks! Though I haven't seen the core part of the patch (i.e., ReceiveTarFile, etc..) yet, here

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2011 at 13:38, Peter Eisentraut wrote: > On lör, 2011-01-15 at 19:10 +0100, Magnus Hagander wrote: >> This patch creates pg_basebackup in bin/, being a client program for >> the streaming base backup feature. >> >> I think it's more or less done now. I've again split it out of >> p

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Peter Eisentraut
On lör, 2011-01-15 at 19:10 +0100, Magnus Hagander wrote: > This patch creates pg_basebackup in bin/, being a client program for > the streaming base backup feature. > > I think it's more or less done now. I've again split it out of > pg_streamrecv, because it had very little shared code with that

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Fujii Masao
On Mon, Jan 17, 2011 at 6:53 PM, Magnus Hagander wrote: > On Mon, Jan 17, 2011 at 09:50, Fujii Masao wrote: >> On Mon, Jan 17, 2011 at 5:44 PM, Magnus Hagander wrote: >>> Weird, no idea. Will have to look into that later - meanwhile you can grab >>> the branch tip from my github repo if you want

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2011 at 09:50, Fujii Masao wrote: > On Mon, Jan 17, 2011 at 5:44 PM, Magnus Hagander wrote: >> Weird, no idea. Will have to look into that later - meanwhile you can grab >> the branch tip from my github repo if you want to review it. > > Which repo should I grab? You seem to have

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Fujii Masao
On Mon, Jan 17, 2011 at 5:44 PM, Magnus Hagander wrote: > Weird, no idea. Will have to look into that later - meanwhile you can grab > the branch tip from my github repo if you want to review it. Which repo should I grab? You seem to have many repos :) http://git.postgresql.org/gitweb Regards,

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Magnus Hagander
On Jan 17, 2011 9:16 AM, "Fujii Masao" wrote: > > On Sun, Jan 16, 2011 at 11:31 PM, Magnus Hagander wrote: > > Ok. Updated patch that includes this change attached. > > I could not apply the patch cleanly against the git master. > Do you know what the cause is? > > $ patch -p1 -d. < /hoge/pg_base

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-17 Thread Fujii Masao
On Sun, Jan 16, 2011 at 11:31 PM, Magnus Hagander wrote: > Ok. Updated patch that includes this change attached. I could not apply the patch cleanly against the git master. Do you know what the cause is? $ patch -p1 -d. < /hoge/pg_basebackup.patch patching file doc/src/sgml/backup.sgml patching

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-16 Thread Magnus Hagander
On Sun, Jan 16, 2011 at 19:21, Dimitri Fontaine wrote: > Magnus Hagander writes: >> If you're doing a regular base backup, that's *not* for replication, >> you might want them in files. > > +1 > > So, is that pg_restore -l idea feasible with your current tar format?  I > guess that would translat

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-16 Thread Dimitri Fontaine
Magnus Hagander writes: > If you're doing a regular base backup, that's *not* for replication, > you might want them in files. +1 So, is that pg_restore -l idea feasible with your current tar format? I guess that would translate to pg_basebackup -l |.tar. Regards, -- Dimitri Fontaine http://2

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-16 Thread Magnus Hagander
On Sun, Jan 16, 2011 at 19:03, Tom Lane wrote: > Magnus Hagander writes: >> On Sun, Jan 16, 2011 at 18:59, Tom Lane wrote: >>> Just stick with the OID.  There's no reason that I can see to have >>> "friendly" names for these tarfiles --- in most cases, the DBA will >>> never even deal with them,

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-16 Thread Tom Lane
Magnus Hagander writes: > On Sun, Jan 16, 2011 at 18:59, Tom Lane wrote: >> Just stick with the OID.  There's no reason that I can see to have >> "friendly" names for these tarfiles --- in most cases, the DBA will >> never even deal with them, no? > No, this is the output mode where the DBA choo

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-16 Thread Magnus Hagander
On Sun, Jan 16, 2011 at 18:59, Tom Lane wrote: > Magnus Hagander writes: >> Well, we'd try to name the file for that "-/foo/bar.tar", which I >> guess would break badly, yes. > >> I guess we could normalize the tablespace name into [a-zA-Z0-9] or so, >> which would still be useful for the majorit

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-16 Thread Tom Lane
Magnus Hagander writes: > Well, we'd try to name the file for that "-/foo/bar.tar", which I > guess would break badly, yes. > I guess we could normalize the tablespace name into [a-zA-Z0-9] or so, > which would still be useful for the majority of cases, I think? Just stick with the OID. There's

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-16 Thread Dimitri Fontaine
Magnus Hagander writes: > On Sun, Jan 16, 2011 at 18:18, Tom Lane wrote: >> No.  Don't even think of going there --- we got rid of user-accessible >> names in the filesystem years ago and we're not going back.  Consider >>        CREATE TABLESPACE "/foo/bar" LOCATION '/foo/bar'; > > Well, we'd tr

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-16 Thread Magnus Hagander
On Sun, Jan 16, 2011 at 18:18, Tom Lane wrote: > Magnus Hagander writes: >>> + * The file will be named base.tar[.gz] if it's for the main data directory >>> + * or .tar[.gz] if it's for another tablespace. >>> >>> Well we have UNIQUE, btree (spcname), so maybe we can use that here? > >> We could

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-16 Thread Tom Lane
Magnus Hagander writes: >> + * The file will be named base.tar[.gz] if it's for the main data directory >> + * or .tar[.gz] if it's for another tablespace. >> >> Well we have UNIQUE, btree (spcname), so maybe we can use that here? > We could, but that would make it more likely to run into encodi

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-16 Thread Magnus Hagander
On Sat, Jan 15, 2011 at 23:10, Dimitri Fontaine wrote: >>> That should be -D --pgdata, for consistency with pg_dump. >> >> pg_dump doesn't have a -D. I assume you mean pg_ctl / initdb? > > Yes, sorry, been too fast. Ok. Updated patch that includes this change attached. I also changed the tar dire

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-15 Thread Dimitri Fontaine
Magnus Hagander writes: > Not sure pg_ctl clone would be the proper name, since it's not > actually a clone at this point (it might be with the second patch I > ust posted that includes the WAL files) Let's keep the clone name for the client that makes it all then :) >> That should be -D --pgdat

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-15 Thread Euler Taveira de Oliveira
Em 15-01-2011 15:10, Magnus Hagander escreveu: One thing I'm thinking about - right now the tool just takes -c to connect to the database. Should it instead be taught to take the connection parameters that for example pg_dump does - one for each of host, port, user, password? (shouldn't be hard

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-15 Thread Magnus Hagander
On Sat, Jan 15, 2011 at 21:16, Dimitri Fontaine wrote: > Hi, > > I have an unexpected 5 mins window to do a first reading of the patch, > so here goes the quick doc and comments proof reading of it. :) :-) > Magnus Hagander writes: >> This patch creates pg_basebackup in bin/, being a client pr

Re: [HACKERS] pg_basebackup for streaming base backups

2011-01-15 Thread Dimitri Fontaine
Hi, I have an unexpected 5 mins window to do a first reading of the patch, so here goes the quick doc and comments proof reading of it. :) Magnus Hagander writes: > This patch creates pg_basebackup in bin/, being a client program for > the streaming base backup feature. Great! We have pg_ctl i

[HACKERS] pg_basebackup for streaming base backups

2011-01-15 Thread Magnus Hagander
This patch creates pg_basebackup in bin/, being a client program for the streaming base backup feature. I think it's more or less done now. I've again split it out of pg_streamrecv, because it had very little shared code with that (basically just the PQconnectdb() wrapper). One thing I'm thinking