Re: [ADMIN] rsync and streaming replication

2011-11-17 Thread Jean-Armel Luce
2011/11/16 Scott Ribe scott_r...@elevated-dev.com: Does the network connection have high latency? When I ping the server,time=15ms. Also, I believe you said -z seemed to slow it down? I confirm I'm certainly glad that you've got a 3x speed increase-- Me too :-) . Thanks for your

Re: [ADMIN] rsync and streaming replication

2011-11-17 Thread Cédric Villemain
2011/11/17 Jean-Armel Luce jaluc...@gmail.com: 2011/11/16 Scott Ribe scott_r...@elevated-dev.com: Does the network connection have high latency?  When I ping the server,time=15ms. Also, I believe you said -z seemed to slow it down? I confirm I'm certainly glad that you've got a 3x

Re: [ADMIN] rsync and streaming replication

2011-11-17 Thread Scott Ribe
On Nov 17, 2011, at 10:11 AM, Jean-Armel Luce wrote: Also, I believe you said -z seemed to slow it down? I confirm Is there a lot of your data that is already compressed? Either something like images which you are storing as bytea or large objects? Or long text values that pg will itself

Re: [ADMIN] rsync and streaming replication

2011-11-16 Thread Jean-Armel Luce
Hi, Today I tried to promote a slave as master using rsync --checksum (without doing vacuum freeze) instead of rsync --all. It takes only 30 minutes with rsync --checksum. Only a few tables are rsynced. Most of the time is consumed by checksum. With rsync --all, it takes 1h40 min. So, rsync

Re: [ADMIN] rsync and streaming replication

2011-11-16 Thread Kevin Grittner
Jean-Armel Luce jaluc...@gmail.com wrote: So, rsync --checksum looks better than rsync --all I've never heard of an --all option for rsync. What does that do? -Kevin -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription:

Re: [ADMIN] rsync and streaming replication

2011-11-16 Thread Scott Ribe
On Nov 16, 2011, at 9:37 AM, Jean-Armel Luce wrote: So, rsync --checksum looks better than rsync --all --all??? What the heck is that and why were you using it? -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice -- Sent via pgsql-admin mailing

Re: [ADMIN] rsync and streaming replication

2011-11-16 Thread Cédric Villemain
2011/11/16 Jean-Armel Luce jaluc...@gmail.com: Hi, Today I tried to promote a slave as master using rsync --checksum (without doing vacuum freeze) instead of rsync --all. It takes only 30 minutes with rsync --checksum.  Only a few tables are rsynced. Most of the time is consumed by

Re: [ADMIN] rsync and streaming replication

2011-11-16 Thread Jean-Armel Luce
You are right. I used -a, and I was wanting to be more meaningful so I wrote --all in my post. Please read --archive insterad of --all I kept --recursive. I didn't use --owner , --group, --perms (permissions, group and owner are the same on each side). I rsynced all the directory (without /*) as

Re: [ADMIN] rsync and streaming replication

2011-11-16 Thread Scott Ribe
On Nov 16, 2011, at 10:11 AM, Jean-Armel Luce wrote: You are right. I used -a, and I was wanting to be more meaningful so I wrote --all in my post. Please read --archive insterad of --all Oh, OK. Still seems odd that it took so much longer. Granted, for the files with different timestamps

Re: [ADMIN] rsync and streaming replication

2011-11-16 Thread Cédric Villemain
2011/11/16 Scott Ribe scott_r...@elevated-dev.com: On Nov 16, 2011, at 10:11 AM, Jean-Armel Luce wrote: You are right. I used -a, and I was wanting to be more meaningful so I wrote --all in my post. Please read --archive insterad of --all Oh, OK. Still seems odd that it took so much

Re: [ADMIN] rsync and streaming replication

2011-11-15 Thread Cédric Villemain
2011/11/14 Scott Ribe scott_r...@elevated-dev.com: On Nov 14, 2011, at 10:59 AM, Jean-Armel Luce wrote: just for the value : rsync --checksum is the option to use to prevent copying of identical files No, that's not what it's used for. It already avoids sending identical blocks by using

Re: [ADMIN] rsync and streaming replication

2011-11-15 Thread Scott Ribe
On Nov 15, 2011, at 3:02 AM, Cédric Villemain wrote: no, you are wrong. -c, --checksum This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a quick check that (by default) checks if each file's size and time of last

Re: [ADMIN] rsync and streaming replication

2011-11-15 Thread Cédric Villemain
2011/11/15 Scott Ribe scott_r...@elevated-dev.com: On Nov 15, 2011, at 3:02 AM, Cédric Villemain wrote: no, you are wrong. -c, --checksum This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a quick check that (by

Re: [ADMIN] rsync and streaming replication

2011-11-15 Thread Scott Ribe
On Nov 15, 2011, at 8:30 AM, Cédric Villemain wrote: Seriously, I did. Is my post just for the value : rsync --checksum is the option to use to prevent copying of **identical files** incorrect ? It's at least incomplete and somewhat misleading. But I guess you could say the same about my

Re: [ADMIN] rsync and streaming replication

2011-11-15 Thread Jean-Armel Luce
Sorry to be so long to answer :-( 2011/11/14 Scott Ribe scott_r...@elevated-dev.com - How large is the db? By which I mean how much disk space does the data directory occupy? 20 GB - What's the bandwidth of the network connection to the distant slave? during rsync, I see narly

Re: [ADMIN] rsync and streaming replication

2011-11-15 Thread Jean-Armel Luce
Humm sorry, I did a mistake 2011/11/15 Jean-Armel Luce jaluc...@gmail.com Sorry to be so long to answer :-( - How large is the db? By which I mean how much disk space does the data directory occupy? 100 GB 2011/11/14 Scott Ribe scott_r...@elevated-dev.com - How large is the db?

Re: [ADMIN] rsync and streaming replication

2011-11-15 Thread Cédric Villemain
This afternoon, I have again sent some updates requests, which were replicated to the sslaves.  : - I am looking modification of modification dates and checksums of 2 tables among my 6000 tables : For each file, the checksum is the same on all the slaves, but different from the checksum of

Re: [ADMIN] rsync and streaming replication

2011-11-14 Thread Cédric Villemain
2011/11/13 Jean-Armel Luce jaluc...@gmail.com: Hi Jerry and Kevin, Thanks for your answers. Jerry, I tried as you said with the parameter recovery_target_timeline = 'latest' and it works. I tried on a smaller test database (only 15MB) with PG9.1.1 and only 1 slave. My switchover

Re: [ADMIN] rsync and streaming replication

2011-11-14 Thread Jean-Armel Luce
Hi, I tried many times with different options of rsync : with rsync -a as explained in the wiki ( http://www.postgresql.org/docs/9.0/interactive/warm-standby.html#STANDBY-SERVER-SETUP), it takes 1h40 for each distant slave and quite all data files are transferred with -c (checksum) or -z

Re: [ADMIN] rsync and streaming replication

2011-11-14 Thread Scott Ribe
On Nov 14, 2011, at 10:59 AM, Jean-Armel Luce wrote: just for the value : rsync --checksum is the option to use to prevent copying of identical files No, that's not what it's used for. It already avoids sending identical blocks by using checksums. --checksum forces a checksum on files that

Re: [ADMIN] rsync and streaming replication

2011-11-14 Thread Scott Ribe
I think there have been two similar threads recently, and I want to be sure I'm not confusing them. So: - How large is the db? By which I mean how much disk space does the data directory occupy? - What's the bandwidth of the network connection to the distant slave? - What's the CPU disk on

Re: [ADMIN] rsync and streaming replication

2011-11-13 Thread Jean-Armel Luce
Hi Jerry and Kevin, Thanks for your answers. Jerry, I tried as you said with the parameter recovery_target_timeline = 'latest' and it works. I tried on a smaller test database (only 15MB) with PG9.1.1 and only 1 slave. My switchover procedure was : Step 1 : stop the old master

[ADMIN] rsync and streaming replication

2011-11-12 Thread Jean-Armel Luce
Hi all, I am using Postgres 9.0.3 in production with Slony 2.0.6 for replication. I am currently testing the streaming replication with 9.0.3. My database contains 100 GBytes of data (6000 tables). I have 1 master node and 3 slave nodes. The 1st slave node is in the same datacenter as the

Re: [ADMIN] rsync and streaming replication

2011-11-12 Thread Kevin Grittner
Jean-Armel Luce wrote: Please, could you help me to understand why it is so long to rsync the data from the new master to the other slaves ? Your post is a little light on details. I think the most useful information would be the output of: rysnc --version and the exact rsync command

Re: [ADMIN] rsync and streaming replication

2011-11-12 Thread Jerry Sievers
Jean-Armel Luce jaluc...@gmail.com writes: Hi all, I am using Postgres 9.0.3 in production with Slony 2.0.6 for replication. I am currently testing the streaming replication with 9.0.3.? My database contains 100 GBytes of data (6000 tables). I have 1 master node and 3 slave nodes. The 1st