RE: rsync backup
I've setup SSH for auto login. It seems I can just do rsync -e ssh -aupg 10.10.10.24:/home/MYDOMAIN /home fine Do I still need /etc/rsyncd.conf on the server? Nope. I do the same thing, and I've never needed it. As far as I know you should only have the rsync binary installed. When you look at your processes by `top` you should see rsync claim CPU time. So, rysnc should be installed on the server. That's all. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Upload only
On Wed, Jan 14, 2004 at 09:32:37AM +, a a wrote: I know there is a read only option that can be specified in rsyncd.conf - is there any way to make it upload only? You can exclude everything from the download, like this: exclude = * That would prevent rsync of allowing any file to be downloaded. Files can still be sent to the server without restriction. The bad thing is that if you're worried about other people reading files they shouldn't, you should be worried about other people writing files they shouldn't. I'd recommend you switch the backup server over to a pull model. Have it grab files from each machine, either using ssh (with an authorized key to allow it to login without a password) or by running an rsync daemon on each machine that you want to backup and restricting who can connect to it just the backup server's IP. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: -vn doesn't list empty directories, -v does
On Wed, Jan 14, 2004 at 03:57:21PM -0500, Joe Harrington wrote: The -vn option lists all files to be copied or deleted, including symlinks, but it does not appear to list empty directories to be copied, even tough it copies them. Unfortunately, this is a known bug that we haven't taken the time to fix just yet. Another bug in unmentioned files (that isn't related to -n) is that rsync doesn't mention a file if we just twiddle its mode, time, user, or group -- we only mention it if we started a transfer for it. I think that both of these things should be fixed; hopefully soon. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
compilation problem
Hello, I have a RedHat 7.3 on which I deinstall the openssl-0.9.6b-35.7 and the openssl-devel-0.9.6b-35.7 packages and compiledinstall openssl-0.9.7c-1. After I downloaded, compiled install rsync-2.6.0 successfully exept that when I try to use I get the error: OpenSSL version mismatch. Built against 90602f, you have 90703f rsync: connection unexpectedly closed (0 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(189) I tryed to specify the ssl libraries in the rsync's config but I didn't find any posibility for doing that. And also it's wierd form me how the compilation uses the old ssl libraries even I desinstall openssl-devel-0.9.6b-35.7 and there is no ssl libraries in /usr/lib exept the sim link I created : libssl.so.2 - /usr/local/lib/libssl.so.0.9.7 Can some one give me a hint what I should do to make rsyn working? Thanks Vlad -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: compilation problem
On Fri, Jan 16, 2004 at 12:23:56PM -0500, Vlady wrote: OpenSSL version mismatch. Built against 90602f, you have 90703f Rsync doesn't do anything directly with ssl, it runs the ssh program (by default in 2.6.0). You should ensure that you can ssh successfully into the system you want to rsync to/from, and then rsync should be fine. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Any (known) scaling issues?
I'm considering using rsync in our data center but I'm worried about whether it will scale to the numbers and sizes we deal with. We would be moving up to a terabyte in a typical sync, consisting of about a million files. Our data mover machines are RedHat Linux Advanced Server 2.1 and all the sources and destinations are NFS mounts. The data is stored on big NFS file servers. The destination will typically be empty and rsync will have to copy everything. However, the copy operation takes many hours and often gets interrupted by an outage. In that case, the operator should be able to restart the process and it resumes where it left off. The current, less than desirable, method uses tar. In the event of an outage, everything needs to be copied again. I'm hoping rsync could avoid this and pick up where it left off. There are really two scaling problems here: 1) Number and size of files - What are the theoretical limits in rsycn? What are the demonstrated maxima? 2) Performance - The current tar-based method breaks the mount points down into (a few dozen) subdirectories and runs multiple tar processes. This does a much better job of keeping the GigE pipes full than a single process and allows the load to be spread over the 4 CPUs in the Linux box. Is there a better way to do this with rsync or would we do the same thing, generate one rsync call for each subdirectory? A major drawback of the subdirectory approach is that tuning to find the optimum number of copy processes is almost impossible. Is anyone looking at multithreading rsync to copy many files at once and get more CPU utilization from a multi-CPU machine? We're moving about 10 terabytes a week (and rising) so whatever we use has to keep those GigE pipes full. Thanks, Bret -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Any (known) scaling issues?
On Fri, Jan 16, 2004 at 03:55:42PM -0800, Bret Foreman wrote: I'm considering using rsync in our data center but I'm worried about whether it will scale to the numbers and sizes we deal with. We would be moving up to a terabyte in a typical sync, consisting of about a million files. Our data mover machines are RedHat Linux Advanced Server 2.1 and all the sources and destinations are NFS mounts. The data is stored on big NFS file servers. The destination will typically be empty and rsync will have to copy everything. However, the copy operation takes many hours and often gets interrupted by an outage. In that case, the operator should be able to restart the process and it resumes where it left off. The current, less than desirable, method uses tar. In the event of an outage, everything needs to be copied again. I'm hoping rsync could avoid this and pick up where it left off. There are really two scaling problems here: 1) Number and size of files - What are the theoretical limits in rsycn? What are the demonstrated maxima? 2) Performance - The current tar-based method breaks the mount points down into (a few dozen) subdirectories and runs multiple tar processes. This does a much better job of keeping the GigE pipes full than a single process and allows the load to be spread over the 4 CPUs in the Linux box. Is there a better way to do this with rsync or would we do the same thing, generate one rsync call for each subdirectory? A major drawback of the subdirectory approach is that tuning to find the optimum number of copy processes is almost impossible. Is anyone looking at multithreading rsync to copy many files at once and get more CPU utilization from a multi-CPU machine? We're moving about 10 terabytes a week (and rising) so whatever we use has to keep those GigE pipes full. The numbers you site should be no problem for rsync. However, the scenario is one that rsync has no real advantage and several disadvantages. You are copying, not syncing so rsync will be slower. Your network is faster than the disks and rsync is designed for disks several times faster than the network. Rsync is even worse over NFS and you are doing NFS to NFS copies. All in all, i wouldn't use rsync. My inclination would be to use cpio -p with no -u. The one thing rsync gets you is checksumming and NFS over udp has a measurable data corruption rate but caches are likely to defeat rsync's checksums so a seperate checksum cycle would still be wanted. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
compatibility of snprintf() return value?
It is my belief that there is not a compatibility problem with the return value of snprintf() like there is with sprintf() (i.e. on some systems, like SunOS, sprintf() returns a char* instead of an int, but I don't believe that such systems even have snprintf()). Does anyone know of a system where snprintf() doesn't return an int? (We don't care about systems that don't have it, because we supply a compatibility function for them.) It would just be nice to be able to use the return value in the rsync code if it is returned consistently. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--delete-sent-files (AKA --move-files)
Yes, it's time once again to return to the subject of moving files.
With the recent changes to the communications code between the receiver
and the generator, there is now a non-clogging channel that we can use
to signal the sender when a file has been successfully transferred,
which allows us delete the original for all transferred files. I have
in the past waffled on whether this feature needs to be in rsync. I'm
currently back on the side that it's a nice thing to support. YMMV.
Here's a new implementation of the feature that adds a generic message
(MSG_SUCCESS) for the receiver to send back to the sender (through the
generator). I made the generator simply forward this message on to the
sender, but to do that it means that the generator must be in multi-
plexed IO mode, which it used to be only when it was on the server side.
My patch adds a new internal flag that lets the code request that the
generator have messaging capability even when it is on the client side
(the non-delete-specific variable makes the code more generic). The one
not-so-nice thing about this setup is that the sender process gets the
MSG_SUCCESS for a particular index when it is down in the I/O code, and
that code doesn't know about the file list. I decided to make this code
call a new function, successful_send(), which is back in sender.c. This
function is the one that handles translating the index into a file name
and deleting the source file (assuming that the delete_sent_files flag
is on, which is currently the only time that MSG_SUCCESS gets sent). I
also added a run-time flag to mark the items we sent off to the
receiver, just to make sure that nothing funny is going on in the
sequence of events (aside: the sender side has no copy-on-write issues
to make us avoid tweaking the flags).
So, feel free to take a look and see if you like what I've done.
..wayne..
Index: flist.c
--- flist.c 17 Jan 2004 01:16:49 - 1.165
+++ flist.c 17 Jan 2004 05:04:54 -
@@ -602,7 +602,7 @@ void receive_file_entry(struct file_stru
if (!file-basename)
out_of_memory(receive_file_entry 1);
- file-flags = flags;
+ file-flags = flags LIVE_FLAGS;
file-length = read_longint(f);
if (!(flags SAME_TIME))
modtime = (time_t)read_int(f);
Index: io.c
--- io.c16 Jan 2004 16:31:47 - 1.119
+++ io.c17 Jan 2004 05:04:54 -
@@ -222,6 +222,14 @@ static void read_msg_fd(void)
read_loop(fd, buf, 4);
redo_list_add(IVAL(buf,0));
break;
+ case MSG_SUCCESS:
+ if (len != 4) {
+ rprintf(FERROR, invalid message %d:%d\n, tag, len);
+ exit_cleanup(RERR_STREAMIO);
+ }
+ read_loop(fd, buf, 4);
+ io_multiplex_write(MSG_SUCCESS, buf, 4);
+ break;
case MSG_INFO:
case MSG_ERROR:
case MSG_LOG:
@@ -637,6 +645,16 @@ static int read_unbuffered(int fd, char
}
read_loop(fd, buffer, remaining);
bufferIdx = 0;
+ break;
+ case MSG_SUCCESS:
+ if (remaining != 4) {
+ rprintf(FERROR, invalid multi-message %d:%ld\n,
+ tag, (long)remaining);
+ exit_cleanup(RERR_STREAMIO);
+ }
+ read_loop(fd, line, 4);
+ successful_send(IVAL(line, 0));
+ remaining = 0;
break;
case MSG_INFO:
case MSG_ERROR:
Index: main.c
--- main.c 17 Jan 2004 05:04:04 - 1.181
+++ main.c 17 Jan 2004 05:04:54 -
@@ -41,6 +41,7 @@ extern int list_only;
extern int local_server;
extern int log_got_error;
extern int module_id;
+extern int need_messages_from_generator;
extern int orig_umask;
extern int preserve_hard_links;
extern int protocol_version;
@@ -558,6 +559,8 @@ void start_server(int f_in, int f_out, i
io_start_multiplex_out(f_out);
if (am_sender) {
+ if (need_messages_from_generator)
+ io_start_multiplex_in(f_in);
if (!read_batch) {
recv_exclude_list(f_in);
if (cvs_exclude)
@@ -623,6 +626,9 @@ int client_run(int f_in, int f_out, pid_
io_flush(FULL_FLUSH);
exit_cleanup(status);
}
+
+ if (need_messages_from_generator)
+ io_start_multiplex_out(f_out);
if (argc == 0) {
list_only = 1;
Index: options.c
--- options.c 15 Jan 2004 17:43:34 - 1.124
+++ options.c 17 Jan 2004 05:04:55 -
@@ -81,12 +81,14 @@ int copy_unsafe_links=0;
int size_only=0;
int bwlimit=0;
int delete_after=0;
+int delete_sent_files
Re: --delete-sent-files (AKA --move-files)
On Fri, Jan 16, 2004 at 09:30:57PM -0800, Wayne Davison wrote: Yes, it's time once again to return to the subject of moving files. With the recent changes to the communications code between the receiver and the generator, there is now a non-clogging channel that we can use to signal the sender when a file has been successfully transferred, which allows us delete the original for all transferred files. I have in the past waffled on whether this feature needs to be in rsync. I'm currently back on the side that it's a nice thing to support. YMMV. Here's a new implementation of the feature that adds a generic message (MSG_SUCCESS) for the receiver to send back to the sender (through the generator). I made the generator simply forward this message on to the sender, but to do that it means that the generator must be in multi- plexed IO mode, which it used to be only when it was on the server side. My patch adds a new internal flag that lets the code request that the generator have messaging capability even when it is on the client side (the non-delete-specific variable makes the code more generic). The one not-so-nice thing about this setup is that the sender process gets the MSG_SUCCESS for a particular index when it is down in the I/O code, and that code doesn't know about the file list. I decided to make this code call a new function, successful_send(), which is back in sender.c. This function is the one that handles translating the index into a file name and deleting the source file (assuming that the delete_sent_files flag is on, which is currently the only time that MSG_SUCCESS gets sent). I also added a run-time flag to mark the items we sent off to the receiver, just to make sure that nothing funny is going on in the sequence of events (aside: the sender side has no copy-on-write issues to make us avoid tweaking the flags). So, feel free to take a look and see if you like what I've done. I'll leave aside the issue of whether we want a delete-after. What you have to underpin the delete-after is almost a per-file status being communicated. Long term this would give us the ability to log at the right point, after it is actually been completed but done on the local end. Right now a push logs the completion when the sender has finished with the file even if the last bit fails. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Sat Jan 17 01:16:49 2004 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/home/wayned/src/rsync Modified Files: flist.c Log Message: Items read from a per-directory .cvsignore file should be added to the local_exclude_list, not the exclude_list. Revisions: flist.c 1.164 = 1.165 http://www.samba.org/cgi-bin/cvsweb/rsync/flist.c.diff?r1=1.164r2=1.165 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Sat Jan 17 01:23:41 2004 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv18002 Modified Files: NEWS Log Message: A --cvs-ignore fix. Revisions: NEWS1.144 = 1.145 http://www.samba.org/cgi-bin/cvsweb/rsync/NEWS.diff?r1=1.144r2=1.145 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Sat Jan 17 04:58:24 2004 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/home/wayned/src/rsync Modified Files: main.c Log Message: Unified the externs. Revisions: main.c 1.179 = 1.180 http://www.samba.org/cgi-bin/cvsweb/rsync/main.c.diff?r1=1.179r2=1.180 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
