[Bug 2790] Add support for converting filenames into different encodings
https://bugzilla.samba.org/show_bug.cgi?id=2790 [EMAIL PROTECTED] changed: What|Removed |Added Severity|major |enhancement Status|NEW |ASSIGNED Summary|mkstemp fails for paths |Add support for converting |which include extended (8- |filenames into different |bit) characters |encodings --- Additional Comments From [EMAIL PROTECTED] 2005-06-11 23:40 --- Rsync does not change the filenames it transfers in any way, so if the OS refuses to store a certain sequence of characters, that is currently out of rsync's hands. (The backslashes you see is just rsync's way of outputting high-bit characters is a visible manner.) It would be nice if rsync supported some kind of filename transformation support so that conversion to and from UTF-8 (or whatever) would be possible. OS X is known to reject certain multi-byte high-bit characters that aren't compatible with its own high-bit character encoding. Your current choices are to (1) change the character encoding on the source FS to match the encoding of the destination FS, making the names compatible; (2) not use high-bit character sequences that conflict between OSes; (3) pre-process the files to convert high-bit characters into sequences that won't fail; (4) use the fname-conv.diff patch in the patches dir to enhance rsync with some basic name-conversion support; (5) help to create a better filename-conversion solution. I didn't really like the solution in the fname-conv.diff because it typically results in a huge number of forked command calls, one for each filename processed. It is a very versatile solution, but is probably overkill for what rsync really needs: the optional(!) ability to use iconv() on the filenames it sends (transferring names in UTF-8 and converting the names via library calls to the local encoding needed). If someone would like to work on a solution for this, please let me know. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.6.5 segfault using --fuzzy & --link-dest
On Sat, Jun 11, 2005 at 02:05:48PM -0400, Erik Jan Tromp wrote:
> #0 0x08060566 in flist_find ()
> #1 0x0804c6cd in recv_generator ()
OK, the crash turned out to be caused by an empty file-list not getting
its "high" value set correctly. If such an empty list gets passed to
flist_find(), it would crash. This is not something that normally
happens, but in the case where an empty destination directory is matched
up with a list-dest directory that has a file that is present but not
up-to-date, rsync triggers the bug. Attached is a patch to fix this.
Thanks for your help!
..wayne..
--- flist.c 27 May 2005 18:15:18 - 1.297
+++ flist.c 12 Jun 2005 06:04:10 -
@@ -1471,8 +1471,12 @@ static void clean_flist(struct file_list
{
int i, prev_i = 0;
- if (!flist || flist->count == 0)
+ if (!flist)
return;
+ if (flist->count == 0) {
+ flist->high = -1;
+ return;
+ }
sorting_flist = flist;
qsort(flist->files, flist->count,
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Feature request: rsync of device content
On 6/11/05, Martin Scharrer wrote: > It would be nice if rsync would also be able to synchronise the content of a > device, i.e. > handle the device like a regular file. Of course only if this is requested > with a special command line option. The idea of a block-device-rsync comes up from time to time. April was the last time I saw it pop up and I gave a summary of what others had said: http://lists.samba.org/archive/rsync/2005-April/012196.html The consensus seems to be that this would be a great idea for a librsync-based standalone tool rather than rsync itself. However, I have still seen no signs that anyone (including myself) has had the time to build such a tool. -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: source side
John Van Essen wrote: > On Sat, 11 Jun 2005, Juergen Busam <[EMAIL PROTECTED]> wrote: > >>Yes, the source side is a CIFS share on a NetApp filer and YES, they >>change everytime I run rsync and they go back another hour. >>no, they do NOT restore to their "normal" unchanged values, after >>unmounting and mounting again... > > > OK. Thanks for verifying that. > > >>they shouldn't change at all, because rsync shouldn't do anything on the >>source side, the CIFS share is mounted readonly AND the Windows Domain >>User has only read rights in this share > > > According to your mount command that I quoted below, it is not mounted > read-only on the *nix side (there's no "ro" option). sorry about that, but after my problems I didn't trust the Domain policies anymore and changed my mount command into: mount -t smbfs -o username=user,password=pwd,ro //server/share /backup/sync sorry, that I didn't mention it explicit before. I thought it is enough to say I mount it read-only... sorry about this... > > I don't use windows shares, but I have a theory about what's going wrong. > > Since the mount is not read-only, RHEL will be trying to update the > last-access times for directories and files that rsync reads. AFAIK, > last-accesss and last-modified timestamp updates are always done in > the same operation, so it's also setting the last-modified time. If > the smbfs/NetApp routine that handles that incorrectly processes a DST > flag, that can explain the hour shift. If the windows side accepts > a timestamp change operation even though its a read-only share (which > might apply only to opens and not to meta-data access), then this > combination explains your problem. To re-iteerate, this is not an > rsync problem - it's just rsync triggering some other bug. > > Add an "ro" to your mount options and see if that makes the problem > go away. Other than that, I have no further help for you. Sorry. > Maybe someone with smbfs/NetApp experience can offer more advice. I did this, but the problem is still the same! thanks a lot for your help and I hope that NetApp AND/OR RedHat have further ideas to solve this BUG. > > John > > > > >>John Van Essen wrote: >> >>>In your original "timestamps" thread back on May 25: >>> >>> http://www.mail-archive.com/rsync@lists.samba.org/msg13496.html >>> >>>you said the source is a "windows share from a NetApp filer" that >>>is mounted on a RHEL3 box via: >>> >>> mount -t smbfs -o username=user,password=pwd //server/share /backup/sync >>> >>>Your timestamp example below seems to indicate a loss of the daylight >>>savings time flag, producing times that are an hour earlier. I think >>>that this is some kind of timestamp _reporting_ problem rather than a >>>timestamp _changing_ problem, and is related to the kernel routine >>>that translates smbfs data to unix fs data. Rsync's heavy disk usage >>>have exposed software shortcomings in the past and this may be yet >>>another example. >>> >>>Have you looked at the timestamps via the windows system to see if they >>>have actually changed when this "problem" happens? >>> >>>Do they change every time you run rsync and go back another hour each >>>time? Or do you unmount the windows share and remount it and the >>>timestamps are restored to their "normal" unchanged values? >>> >>>John >>> >>> >>>On Sat, 11 Jun 2005, Juergen Busam <[EMAIL PROTECTED]> wrote: >>> >>> I've ran a "ls -altr" on the source before syncing and after it and it definitely shows that the timestamps of the source side changed after rsync has finished. The destination side gets the timestamps of the source side before the sync. Example: bevor sync: -rwxr-xr-x1 root root 572928 Feb 23 22:26 Accept_PatXfer.dot after sync: -rwxr-xr-x1 root root 572928 Feb 23 21:26 Accept_PatXfer.dot destination shows -rwxr-xr-x1 root root 572928 Feb 23 22:26 Accept_PatXfer.dot after the rsync run. The "problem hereby is, that the sourceside is mounted readonly! Nevertheless the timestamp gets changed... Wayne Davison wrote: >On Fri, Jun 10, 2005 at 09:27:42AM +1000, Juergen Busam wrote: > > > >>I still have my formerly mentioned timestamp problem. IS rsync changing >>the timestamps on the sourceside? > > >No, rsync doesn't do anything on the source-side except read things >(unless --remove-sent-files is specified, in which case it also removes >things). If your timestamps on the source are changing, something else >is changing them. (You may wish to verify that the timestamps on the >source are the ones that are changing and not the destination -- rsync >updates the files when the timestamps are different, but you don't know >for sure which side has the changed timestamps unless you investigate.) > >..wayne.. > > > -- To unsubscri
Re: source side
On Sat, 11 Jun 2005, Juergen Busam <[EMAIL PROTECTED]> wrote: > Yes, the source side is a CIFS share on a NetApp filer and YES, they > change everytime I run rsync and they go back another hour. > no, they do NOT restore to their "normal" unchanged values, after > unmounting and mounting again... OK. Thanks for verifying that. > they shouldn't change at all, because rsync shouldn't do anything on the > source side, the CIFS share is mounted readonly AND the Windows Domain > User has only read rights in this share According to your mount command that I quoted below, it is not mounted read-only on the *nix side (there's no "ro" option). I don't use windows shares, but I have a theory about what's going wrong. Since the mount is not read-only, RHEL will be trying to update the last-access times for directories and files that rsync reads. AFAIK, last-accesss and last-modified timestamp updates are always done in the same operation, so it's also setting the last-modified time. If the smbfs/NetApp routine that handles that incorrectly processes a DST flag, that can explain the hour shift. If the windows side accepts a timestamp change operation even though its a read-only share (which might apply only to opens and not to meta-data access), then this combination explains your problem. To re-iteerate, this is not an rsync problem - it's just rsync triggering some other bug. Add an "ro" to your mount options and see if that makes the problem go away. Other than that, I have no further help for you. Sorry. Maybe someone with smbfs/NetApp experience can offer more advice. John > John Van Essen wrote: >> In your original "timestamps" thread back on May 25: >> >> http://www.mail-archive.com/rsync@lists.samba.org/msg13496.html >> >> you said the source is a "windows share from a NetApp filer" that >> is mounted on a RHEL3 box via: >> >> mount -t smbfs -o username=user,password=pwd //server/share /backup/sync >> >> Your timestamp example below seems to indicate a loss of the daylight >> savings time flag, producing times that are an hour earlier. I think >> that this is some kind of timestamp _reporting_ problem rather than a >> timestamp _changing_ problem, and is related to the kernel routine >> that translates smbfs data to unix fs data. Rsync's heavy disk usage >> have exposed software shortcomings in the past and this may be yet >> another example. >> >> Have you looked at the timestamps via the windows system to see if they >> have actually changed when this "problem" happens? >> >> Do they change every time you run rsync and go back another hour each >> time? Or do you unmount the windows share and remount it and the >> timestamps are restored to their "normal" unchanged values? >> >> John >> >> >> On Sat, 11 Jun 2005, Juergen Busam <[EMAIL PROTECTED]> wrote: >> >>>I've ran a "ls -altr" on the source before syncing and after it and it >>>definitely shows that the timestamps of the source side changed after >>>rsync has finished. The destination side gets the timestamps of the >>>source side before the sync. >>> >>>Example: >>> >>>bevor sync: >>>-rwxr-xr-x1 root root 572928 Feb 23 22:26 Accept_PatXfer.dot >>> >>>after sync: >>>-rwxr-xr-x1 root root 572928 Feb 23 21:26 Accept_PatXfer.dot >>> >>>destination shows >>>-rwxr-xr-x1 root root 572928 Feb 23 22:26 Accept_PatXfer.dot >>> >>>after the rsync run. >>> >>> >>>The "problem hereby is, that the sourceside is mounted readonly! >>>Nevertheless the timestamp gets changed... >>> >>> >>>Wayne Davison wrote: >>> On Fri, Jun 10, 2005 at 09:27:42AM +1000, Juergen Busam wrote: >I still have my formerly mentioned timestamp problem. IS rsync changing >the timestamps on the sourceside? No, rsync doesn't do anything on the source-side except read things (unless --remove-sent-files is specified, in which case it also removes things). If your timestamps on the source are changing, something else is changing them. (You may wish to verify that the timestamps on the source are the ones that are changing and not the destination -- rsync updates the files when the timestamps are different, but you don't know for sure which side has the changed timestamps unless you investigate.) ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Feature request: rsync of device content
Hi list, I often save the content of a hardware device (a harddisk partition with a not-so-stable non-unix OS on it) on a backup disk which is connected over the network. Until now I used 'dd' to make a clone file of the device (dd if=/dev/hda2 of=disk.raw bs=2M) and then I copied or rsynced this file with the network copy. It would be nice if rsync would also be able to synchronise the content of a device, i.e. handle the device like a regular file. Of course only if this is requested with a special command line option. I had a short look to the rsync source code and I think it could be done by adding some code at the same positions like for the -D/--devices option. I tried a little hack by my own and got the client to read the device file but the server aborted with a error message (apologies for not providing a message). To get this running someone needs more knowledge about rsync and about handling device files in C. I don't have much knowledge about either of this two, but maybe it is just a short thing for someone with more knowledge in this. Best regards, Martin Scharrer -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2790] New: mkstemp fails for paths which include extended (8-bit) characters
https://bugzilla.samba.org/show_bug.cgi?id=2790 Summary: mkstemp fails for paths which include extended (8-bit) characters Product: rsync Version: 2.6.5 Platform: All OS/Version: Mac OS X Status: NEW Severity: major Priority: P3 Component: core AssignedTo: [EMAIL PROTECTED] ReportedBy: [EMAIL PROTECTED] QAContact: [EMAIL PROTECTED] When mirroring a directory tree with "rsync -aq --delete-after", there are many error messages reported similar to: rsync: mkstemp "/backup/home/kuwe/.kdb/.\376\254\026T.zTE32v" failed: Invalid argument (22) It seems that extended characters are converted to octal codes by the protocol but have not been converted back to 8-bit codes prior to passing to mkstemp, so mkstemp fails under Mac OS X. The problem does not occur under FreeBSD or Solaris. Apparently Mac OS X rejects the backslashes in file names while FreeBSD and Solaris accept them. The same problem is observed with the rsync that came with Mac OS X (2.6.2). -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.6.5 segfault using --fuzzy & --link-dest
On Sat, 11 Jun 2005 09:03:07 -0700 Wayne Davison <[EMAIL PROTECTED]> wrote: > On Fri, Jun 10, 2005 at 05:14:57AM -0400, Erik Jan Tromp wrote: > > if I remove every possible option except --fuzzy & --link-dest, > > segfault every time. > > I haven't seen that in my testing. One easy thing to do is to make sure > that core dumping is enabled and look at a backtrace: > > ulimit -c unlimited > /path/to/non-stripped/rsync ... > gdb /path/to/non-stripped/rsync /path/to/core > bt Concise directions.. perfect. > The backtract (bt) command will tell us where the program is crashing, > and should help me to find the bug. Note that a non-stripped rsync can > be found in the build dir, and that the core file will probably be in > the destination directory of your pull transfer. Source tree was wiped after I packaged, so did a quick rebuild & tossed the non-stripped bin in ~/bin/. No dev tools on oxygen (backup server), so into ~/bin/ for gdb as well. Following is a complete dump. -- 8< -- [EMAIL PROTECTED] ~]# ulimit -c unlimited [EMAIL PROTECTED] ~]# ~/bin/rsync --archive --delete-during --fuzzy --hard-links --nu meric-ids --quiet --sparse --temp-dir /backup/helium/ --link-dest /backup/hydrog en/saturday/ --password-file /backup/helium/.password rsync://[EMAIL PROTECTED]/back up/ /backup/helium/saturday/ Segmentation fault (core dumped) [EMAIL PROTECTED] ~]# ~/bin/gdb ~/bin/rsync /backup/helium/saturday/core GNU gdb 6.3 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i486-slackware-linux"...Using host libthread_db libr ary "/lib/libthread_db.so.1". Core was generated by `/root/bin/rsync --archive --delete-during --fuzzy --hard- links --numeric-ids --'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/lib/libpopt.so.0...done. Loaded symbols for /usr/lib/libpopt.so.0 Reading symbols from /lib/libresolv.so.2...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libnss_files.so.2...done. Loaded symbols for /lib/libnss_files.so.2 #0 0x08060566 in flist_find () (gdb) bt #0 0x08060566 in flist_find () #1 0x0804c6cd in recv_generator () #2 0x0804db9a in generate_files () #3 0x08056390 in do_recv () #4 0x08056942 in client_run () #5 0x0806ab2b in start_socket_client () #6 0x08056e9f in start_client () #7 0x080573fd in main () (gdb) quit -- 8< -- Erik -- "I really want a license to do just two things: make the code available to others, and make sure that improvements stay that way. That's really it. Nothing more, nothing less. Everything else is fluff." -- Linus Torvalds -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.6.5 segfault using --fuzzy & --link-dest
On Fri, Jun 10, 2005 at 05:14:57AM -0400, Erik Jan Tromp wrote: > if I remove every possible option except --fuzzy & --link-dest, > segfault every time. I haven't seen that in my testing. One easy thing to do is to make sure that core dumping is enabled and look at a backtrace: ulimit -c unlimited /path/to/non-stripped/rsync ... gdb /path/to/non-stripped/rsync /path/to/core bt The backtract (bt) command will tell us where the program is crashing, and should help me to find the bug. Note that a non-stripped rsync can be found in the build dir, and that the core file will probably be in the destination directory of your pull transfer. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: source side
Yes, the source side is a CIFS share on a NetApp filer and YES, they change everytime I run rsync and they go back another hour. no, they do NOT restore to their "normal" unchanged values, after unmounting and mounting again... they shouldn't change at all, because rsync shouldn't do anything on the source side, the CIFS share is mounted readonly AND the Windows Domain User has only read rights in this share John Van Essen wrote: > In your original "timestamps" thread back on May 25: > > http://www.mail-archive.com/rsync@lists.samba.org/msg13496.html > > you said the source is a "windows share from a NetApp filer" that > is mounted on a RHEL3 box via: > > mount -t smbfs -o username=user,password=pwd //server/share /backup/sync > > Your timestamp example below seems to indicate a loss of the daylight > savings time flag, producing times that are an hour earlier. I think > that this is some kind of timestamp _reporting_ problem rather than a > timestamp _changing_ problem, and is related to the kernel routine > that translates smbfs data to unix fs data. Rsync's heavy disk usage > have exposed software shortcomings in the past and this may be yet > another example. > > Have you looked at the timestamps via the windows system to see if they > have actually changed when this "problem" happens? > > Do they change every time you run rsync and go back another hour each > time? Or do you unmount the windows share and remount it and the > timestamps are restored to their "normal" unchanged values? > > John > > > On Sat, 11 Jun 2005, Juergen Busam <[EMAIL PROTECTED]> wrote: > >>I've ran a "ls -altr" on the source before syncing and after it and it >>definitely shows that the timestamps of the source side changed after >>rsync has finished. The destination side gets the timestamps of the >>source side before the sync. >> >>Example: >> >>bevor sync: >>-rwxr-xr-x1 root root 572928 Feb 23 22:26 Accept_PatXfer.dot >> >>after sync: >>-rwxr-xr-x1 root root 572928 Feb 23 21:26 Accept_PatXfer.dot >> >>destination shows >>-rwxr-xr-x1 root root 572928 Feb 23 22:26 Accept_PatXfer.dot >> >>after the rsync run. >> >> >>The "problem hereby is, that the sourceside is mounted readonly! >>Nevertheless the timestamp gets changed... >> >> >>Wayne Davison wrote: >> >>>On Fri, Jun 10, 2005 at 09:27:42AM +1000, Juergen Busam wrote: >>> >>> I still have my formerly mentioned timestamp problem. IS rsync changing the timestamps on the sourceside? >>> >>> >>>No, rsync doesn't do anything on the source-side except read things >>>(unless --remove-sent-files is specified, in which case it also removes >>>things). If your timestamps on the source are changing, something else >>>is changing them. (You may wish to verify that the timestamps on the >>>source are the ones that are changing and not the destination -- rsync >>>updates the files when the timestamps are different, but you don't know >>>for sure which side has the changed timestamps unless you investigate.) >>> >>>..wayne.. > > > -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: source side
In your original "timestamps" thread back on May 25: http://www.mail-archive.com/rsync@lists.samba.org/msg13496.html you said the source is a "windows share from a NetApp filer" that is mounted on a RHEL3 box via: mount -t smbfs -o username=user,password=pwd //server/share /backup/sync Your timestamp example below seems to indicate a loss of the daylight savings time flag, producing times that are an hour earlier. I think that this is some kind of timestamp _reporting_ problem rather than a timestamp _changing_ problem, and is related to the kernel routine that translates smbfs data to unix fs data. Rsync's heavy disk usage have exposed software shortcomings in the past and this may be yet another example. Have you looked at the timestamps via the windows system to see if they have actually changed when this "problem" happens? Do they change every time you run rsync and go back another hour each time? Or do you unmount the windows share and remount it and the timestamps are restored to their "normal" unchanged values? John On Sat, 11 Jun 2005, Juergen Busam <[EMAIL PROTECTED]> wrote: > > I've ran a "ls -altr" on the source before syncing and after it and it > definitely shows that the timestamps of the source side changed after > rsync has finished. The destination side gets the timestamps of the > source side before the sync. > > Example: > > bevor sync: > -rwxr-xr-x1 root root 572928 Feb 23 22:26 Accept_PatXfer.dot > > after sync: > -rwxr-xr-x1 root root 572928 Feb 23 21:26 Accept_PatXfer.dot > > destination shows > -rwxr-xr-x1 root root 572928 Feb 23 22:26 Accept_PatXfer.dot > > after the rsync run. > > > The "problem hereby is, that the sourceside is mounted readonly! > Nevertheless the timestamp gets changed... > > > Wayne Davison wrote: >> On Fri, Jun 10, 2005 at 09:27:42AM +1000, Juergen Busam wrote: >> >>>I still have my formerly mentioned timestamp problem. IS rsync changing >>>the timestamps on the sourceside? >> >> >> No, rsync doesn't do anything on the source-side except read things >> (unless --remove-sent-files is specified, in which case it also removes >> things). If your timestamps on the source are changing, something else >> is changing them. (You may wish to verify that the timestamps on the >> source are the ones that are changing and not the destination -- rsync >> updates the files when the timestamps are different, but you don't know >> for sure which side has the changed timestamps unless you investigate.) >> >> ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Has anyone seen this?
http://use.perl.org/~Matts/journal/25138 Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
