Rsync Problem mith UNC paths
We're using rsync 2.5.6 version 26. Want to synch two W2K directories which are not on my machine (machine1 synchs machine2 - machine3) I would like to synch the 2 direcories with the following syntax: rsync.exe -rva //remote_server1/share$/dir1/default.asp remote_server2::dir2/ When I try this it seems to work: building file list ... 1 file to consider send_files failed to open //remote_server1/share$/dir1/default.asp: No such file or directory No files have been synched When I connect the remote drive to my machine with Net use x: \\remote_server1\share$ \\remote_server1\share$ Then use the cygdrive syntax (as follows) rsync.exe -rva /cygdrive/x/dir1/default.asp remote_server::dir2/ Everything works fine Because I run this synch as a batch job I would like to connect directly with the UNC path. Does anyone know how to do that? Thanks Thomas Müller-Lynch -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trouble with password (daemon mode)
Dennis Chelukanov writes: I running rsync in daemon mode (rsync --daemon) Everything seems to work well until I try to protect item with password. here is my /etc/rsyncd.conf : use chroot = yes max connections = 10 syslog facility = local5 [ftp] path = /var/ftp comment = ftp secrets file = /etc/rsyncd.secrets auth users = gate1 here is my /etc/rsyncd.secrets gate1:abcdefg Here is the result of execution : /etc rsync rsync://[EMAIL PROTECTED]/ftp/ Password: @ERROR: auth failed on module ftp rsync: connection unexpectedly closed (87 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) /etc What could it be ??? Either the permissions on /etc/rsyncd.secrets are too generous or you are missing a newline at the end of /etc/rsyncd.secrets. Craig -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Is anyone replicating .5TB or higher?
Greetings! I've heard about using rsync to replicate data across the WAN, but need to know if anyone is using it on a large scale. I have a client who is contemplating consolidating Windows file/print servers into a Linux partition on an iSeries. The show stopper is whether rsync (or any replication product) can and will replicate a) at the file level, and b)a database approaching .6TB in size. Automation and accuracy are key. Thanks in advance for any information! Laura __ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Ihre Ausschreibungen
Wir sind es noch einmal! ;-) Sehr geehrte Damen und Herren, aller Anfang ist schwer Vor einigen Tagen haben wir Sie auf den neuen AUSSCHREIBUNGS-MONITOR hingewiesen. Über die riesige Resonanz und Ihre zahlreichen Anregungen haben wir uns sehr gefreut. Leider ist uns ein bedauerlicher Fehler unterlaufen. Aus technischen Gründen wurde nur ein kleiner Teil der deutschen Ausschreibungen gezeigt. So musste der Eindruck entstehen, dass der AUSSCHREIBUNGS-MONITOR zwar eine ausgezeichnete Idee ist, Ihnen jedoch keinen konkreten Nutzen bietet. Inzwischen haben wir diesen Fehler behoben. Selbstverständlich bietet der AUSSCHREIBUNGS-MONITOR alle zwei Wochen eine sehr große Anzahl deutscher Ausschreibungen. Bitte überzeugen Sie sich selbst: Unter www.ausschreibungs-monitor.de finden Sie unsere vollständig überarbeitete Übersicht aller aktuellen Ausschreibungen dieser Woche. Noch eine guten Nachricht: Bis zum 31. Oktober 2003 erhalten Sie den AUSSCHREIBUNGS-MONITOR zu einem nochmals reduzierten Jahresbezugspreis von nur 118,00 Euro (statt zuletzt 198,00 Euro). Bitte geben Sie dafür bei Ihrer Online-Bestellung den nachstehenden Rabatt-Code ein: SPNARBTS-081003 Diese Chance sollten Sie sich nicht entgehen lassen! Für Ihre Rückfragen stehe ich Ihnen natürlich jederzeit gerne zur Verfügung. Mit freundlichen Grüßen Ihre Verlagsleitung P.S: Es ist viel einfacher, als Sie glauben, an öffentliche Aufträge zu gelangen - mehr darüber in unserem Buch Ausschreibungen - Gewusst wie. Sie erhalten diesen Leitfaden als Geschenk, wenn Sie bis zum 31. Oktober 2003 unter www.ausschreibungs-monitor.de bestellen. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is anyone replicating .5TB or higher?
On Tue, Oct 21, 2003 at 06:56:20AM -0700, Laura Ausel wrote: I've heard about using rsync to replicate data across the WAN, but need to know if anyone is using it on a large scale. Many are using it on large scales. There are scaling issues and mitigation depends on the definition of large. I have a client who is contemplating consolidating Windows file/print servers into a Linux partition on an iSeries. The show stopper is whether rsync (or any replication product) can and will replicate a) at the file level, and b)a database approaching .6TB in size. Automation and accuracy are key. Is that a single file nearly 600GB or multiple files? Rsync's memory footprint is dependant on two things: the file list and the number of checksum blocks of the largest file. 2.5.6 and earlier releases also have a performance issue with large files. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is anyone replicating .5TB or higher?
jw schultz wrote On Tue, Oct 21, 2003 at 06:56:20AM -0700, Laura Ausel wrote: I've heard about using rsync to replicate data across the WAN, but need to know if anyone is using it on a large scale. Many are using it on large scales. There are scaling issues and mitigation depends on the definition of large. Whether rsync is useable for large datasets also depends on what your requirements for timeliness are. Running rsync against 600GB of data will take quite a long time, so you can only be guaranteed to be 'in sync' during relatively short time windows. -justinb -- Justin Banks Constant Data, Inc. http://www.constantdata.com -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Feature Request - Recursive Rsync Parameter - Example Script
I wanted to flag a problem and offer a possible solution.
The problem:
large rsync operation fails on machines with modest amounts of memory.
Proposal:
Add a parameter to rsync recursive to specify a recursion level (See
example bash wrapper below). (works with recursive file system rsyncs
only, i.e. -a or -r) The logic goes:
if recursion switch true and recursion_level 0
-rsync this directory only
-call rsync for each subdirectory with a decremented recursion_level and
pass the same switches along
else (recursion_level really is 0)
-perform the full rsync (from this level)
What this does is help break up the job into smaller pieces. Otherwise
rsync can consume hundreds of megabyte of memory attempting to perform a
single operation. In this scenario you'll see one rsync process for each
level of recursion.
Here's and example bash script that is an attempt at this idea: (it
supports the -n options so you can see the calls it makes)
My bash scripting skills need some work, but, you get the idea. If
someone wants to further develop this script, feel free.
Cheers.
#!/bin/bash
# copyright 2003 Matt Olson, Kavi Corporation ([EMAIL PROTECTED])
# Licence: General Public License
# set our environment
IFS=$'\n' # This keeps bash from breaking up file names with space in them.
if [ ! -n $1 ]
then
echo Usage: `basename $0` recursion_level \quoted_rsync_parameters\ source
path destination_path
echo Note: argument parsing is order dependent.
exit 1
fi
# Some debugging help, fifth parm will echo args
if [ -n $5 -a $5 = args ]
then
for arg in $@
do
echo Arg #$index = $arg
done
fi
# Assign parameters to some variables.
r_level=$1
rsync_options=$2
source_path_parm=$3
dest_path_parm=$4
rsync_no_r_options=`echo $rsync_options | sed -e s/r// | sed -e s/a/lptgoD/`
# Let's support the rsync test mode.
test_run=`echo $rsync_options | grep n`
# We need to decide if the source is a remote host
# Parse out the source path and if it is remote, capture the hostname
if [ `echo $source_path_parm | grep :` ]
then
remote_source_host=${source_path_parm%:*}
remote_source_path=${source_path_parm#*:}
fi
# We need to also decide if the destination is a remote host.
# Parse out the source path and if it is remote, capture the hostname
if [ `echo $dest_path_parm | grep :` ]
then
remote_dest_host=${dest_path_parm%:*}
remote_dest_path=${dest_path_parm#*:}
fi
# At this point we need to see if there are additional directories to
# call with lrsync, as long as our recursion level is 0.
# To build a list of targets, we need to determine if the host is remote.
if [ $remote_source_host ]
then
# If host is remote, get a file list via rsh
directory_object=`rsh $remote_source_host ls -1p $remote_source_path | grep /`
else
# If host is local, get a file list
directory_object=`ls -1p $source_path_parm | grep /`
fi
if [ $test_run ]
then
echo lrsync: directory_object: $directory_object
fi
# With these results walk through list returned and call rsync/lrsync
# Testing the recursion level.
# At this point if we are at recursion level 0 then do some rsyncs.
# If not at r_level 0 then call lrsync with a decremented r_level.
# If no additional directory objects to recurse, do the rsync.
if [ $1 = 0 ] || [ -z $directory_object ]
then
# Do rsync(s)
# If this is a test run, echo some extra info.
if [ $test_run ]
then
echo lrsync: rsync $rsync_options $source_path_parm $dest_path_parm
rsync $rsync_options $source_path_parm $dest_path_parm
else
rsync
Re: Feature Request - Recursive Rsync Parameter - Example Script
On Tue, Oct 21, 2003 at 06:25:51PM -0700, Matt Olson wrote: I wanted to flag a problem and offer a possible solution. The problem: large rsync operation fails on machines with modest amounts of memory. Proposal: Add a parameter to rsync recursive to specify a recursion level (See example bash wrapper below). (works with recursive file system rsyncs only, i.e. -a or -r) The logic goes: Limiting the depth of recursion is already supported just not intuitive. rsync -r --exclude='/*/*/*/' Your idea for a shell script to automate picking up the lower levels is good and could compose the --exclude pattern. The next step would be to set the job partition level based on path count as in find $subtree -print|wc -l. if recursion switch true and recursion_level 0 -rsync this directory only -call rsync for each subdirectory with a decremented recursion_level and pass the same switches along else (recursion_level really is 0) -perform the full rsync (from this level) What this does is help break up the job into smaller pieces. Otherwise rsync can consume hundreds of megabyte of memory attempting to perform a single operation. In this scenario you'll see one rsync process for each level of recursion. Here's and example bash script that is an attempt at this idea: (it supports the -n options so you can see the calls it makes) My bash scripting skills need some work, but, you get the idea. If someone wants to further develop this script, feel free. Cheers. #!/bin/bash [snip] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync on OpenVMS
jw schultz wrote: Sounds promising. The pitfall you with rsync in threads is that rsync forks with a COW expectation using a great deal of data set prior to the fork. Some of that data is altered. In particular a slew of global variables that must become thread unique when modified or things will break in subtle ways. Yes, I can see a few cases of this. There are approximately 235 global and static variables in rsync, and of those 105 are obviously never modified after the fork() takes place. That may be the case for several of the others, like the 10 in batch.c As the use of the remaining ones are identified, that issue is easily fixed. What I need to do is map the routines called by the receive process/thread to see what static/global variables that they modify. All the more reason to get more people working on this that want it to work on the OpenVMS operating system. Several people have contacted me that they are willing to assist. -John [EMAIL PROTECTED] Personal Opinion Only -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync on OpenVMS
On Tue, Oct 21, 2003 at 11:28:11PM -0400, John E. Malmberg wrote: jw schultz wrote: Sounds promising. The pitfall you with rsync in threads is that rsync forks with a COW expectation using a great deal of data set prior to the fork. Some of that data is altered. In particular a slew of global variables that must become thread unique when modified or things will break in subtle ways. Yes, I can see a few cases of this. There are approximately 235 global and static variables in rsync, and of those 105 are obviously never modified after the fork() takes place. That may be the case for several of the others, like the 10 in batch.c As the use of the remaining ones are identified, that issue is easily fixed. What I need to do is map the routines called by the receive process/thread to see what static/global variables that they modify. All the more reason to get more people working on this that want it to work on the OpenVMS operating system. Several people have contacted me that they are willing to assist. If you put the ones that get modified post-fork into a thread-local structure there is a reasonable chance of that getting into mainline. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
