RFE blues (Re: [Bug 8188] Mechanism for taking an rsync server down for maintenance)
On Wed, 2011-06-01 at 17:08 -0400, Brian K. White wrote: > On 6/1/2011 3:26 PM, Matt McCutchen wrote: > > On Wed, 2011-06-01 at 14:57 -0400, Brian K. White wrote: > >> I like the built-in idea as I don't happen to use rsync via inetd/xinetd > >> or any other on-demand starter. > >> > >> It's not an actual problem for me, today, but that's no excuse to avoid > >> doing the right thing once you recognize it. > > > > And "the right thing" for upstream rsync is in Wayne's sole discretion. > > In my view, the perl script is a fine solution and what convenience > > there may be in a built-in option does not merit the effort of > > maintaining the feature as part of rsync. > > > > That is a work-around, not a solution. The difference between the two is a matter of opinion. I don't share yours. > The work-around requires using (x)inetd This is not true of the program in comment 1. > If the service aims to be usable stand-alone, which rsync clearly does, > then imo this is just part of that. A nicity perhaps, not critical, but > trivial to implement and maintain in comparison with things like > child/session management which are definitely in there solely to support > standalone daemon operation. Right. Feel free to maintain an enhanced version of rsync with every proposed feature that is straightforward to maintain in isolation and potentially helpful to rsync's mission, and if you do a good enough job the distros will surely adopt it. Wayne will maintain what he judges is worthwhile; you should not expect him to do otherwise, unless you hire him. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Selective propagation of deletions
On Wed, 2011-06-01 at 17:08 -0400, Brian K. White wrote: > There are other problems too, relating to the difference between "This > directory was deleted from the source and so you DO want to delete it in > your mirror.", vs "This directory was removed only because the entire > tree was removed because we are no longer supporting this old version of > the distribution, you probably do NOT want to delete it from your > private copy just because we removed it from our public site." You could have just asked this on the list (or even read the docs and found the answer yourself). Just write protect filters for the directories that should be treated that way. If appropriate, the source could even provide filter files in-tree for you to use. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync is coring
Greetings! We are running an rsync process every hour and it is producing a core file. We thought initially there was a corrupt file but that is gone and the core file is still being produced. Would appreciate any help in analysing the core file. Thanks, Vincent Soosai Java Developer Phone: 503 745 2615 If you are not the intended addressee, please inform us immediately that you have received this e-mail in error, and delete it. We thank you for your cooperation. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync is coring
On Thu, 2011-06-02 at 09:27 -0700, vincent.soo...@daimler.com wrote: > We are running an rsync process every hour and it is producing a core > file. We thought initially there was a corrupt file but that is gone > and the core file is still being produced. Would appreciate any help > in analysing the core file. Two things you can do: - Open the core file in a debugger and get a stack trace. - Try to reproduce the problem on a smaller set of files or with simpler options, etc., until you find a minimal case that fails. This may shed light on what is causing the problem. If the same minimal case fails on another machine, that is probably grounds to file a bug. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Why is IO_BUFFER_SIZE set to 4092?
I'm curious as to why IO_BUFFER_SIZE is only set to 4092 in rsync.h? This seems very small. Is there any harm in increasing it? If not, what is the proper way to determine what is should be set to? Greg -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Selective propagation of deletions
On 6/2/2011 11:28 AM, Matt McCutchen wrote: On Wed, 2011-06-01 at 17:08 -0400, Brian K. White wrote: There are other problems too, relating to the difference between "This directory was deleted from the source and so you DO want to delete it in your mirror.", vs "This directory was removed only because the entire tree was removed because we are no longer supporting this old version of the distribution, you probably do NOT want to delete it from your private copy just because we removed it from our public site." You could have just asked this on the list (or even read the docs and found the answer yourself). Just write protect filters for the directories that should be treated that way. If appropriate, the source could even provide filter files in-tree for you to use. What do you mean "directories that should be treated this way"? That question, plus the inability to distinguish an admittedly tidy work-around from a solution makes me think I am wasting keystrokes here but let's answer the charges just for the record if nothing else. The very same directories will have files being deleted for a while, and during that while, the deleted files from the source should be deleted from the destination. Then at some point they will be either still present but changed and mostly empty, or present but completely empty, or not-present. The disappearance of a file because they were all deleted at eol is indistinguishable from the disappearance of a file because it was replaced with a newer one. Perhaps the server side could specify all the deleted files and directories as being excluded before they are actually removed from the server. Then the client could use --delete but not --delete-excluded and get essentially the optimal behavior. That may or may not be practical because it depends on just how the distribution source happens to want to organize files and manage ongoing turnover. But really as I already said this kind of thing is probably outside of rsync's scope. The reason I said this could really be easily outside the expected scope of rsync to ever provide an answer for is that this is really pretty arbitrary high level application specific oddball behavior. rsync for all it power and flexibility is still essentially a low level and agnostic utility like cp or tar. It's some other higher level script or application's job to decide what to copy and where and when and why. It's only cp's job to copy a file as directed. But it's not black & white either. rsync is in fact specifically intended to be a whole lot more than cp. It's by definition and by design a rather more high level, feature rich, powerful, flexible tool, and aims, and succeeds, in providing many and various ways to express even very complex rules to govern file transfers. So it's entirely possible to someday invent some generic feature that might be used in any number of scenarios, and which this one might be just one example it serves. Almost all of rsync's features could be argued either way, that they are doable from without, and therefore should be done from without instead of from within. But if that was really the only guiding principle then rsync would not be very useful today, like a special cp with one twist, and we'd all be using something else since most people have more complex needs than that. I don't know what your argument is really. Do you suppose I am attacking rsync or Wayne or anyone else such that they need defending? -- bkw -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 8188] Mechanism for taking an rsync server down for maintenance
https://bugzilla.samba.org/show_bug.cgi?id=8188 --- Comment #4 from Brian K. White 2011-06-02 21:47:32 UTC --- (In reply to comment #3) > (In reply to comment #1) > > That's very cute, but not quite what I had in mind. > > I just attached a proof of concept patch that I surprised myself by being able > to hack together the other night. > > The real thing probably needs to be configurable, and it'd probably be better > to get the message from the contents of the file (if there was any). Someone else emailed me an untested idea to just use exclude file and place/remove/edit the exclude file to toggle the access. That doesn't provide the truly graceful/informative response code or message, but: * toggles the access * avoids downing the service even momentarily * potentially avoids requiring a host admin to even do anything, since a user may have permission to place/remove the exclude file * it's at least distinct from failed to connect * uses already existing features * does not require any, let alone two different, wrapper layers in the form of the perl (or other) wrapper script and (x)inetd to run it. I haven't finished playing around with that to see just how close that can come to the desired fully graceful process but I felt it should be mentioned as another workaround option. -- bkw -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Selective propagation of deletions, + rsync scope
On Thu, 2011-06-02 at 17:43 -0400, Brian K. White wrote: > Perhaps the server side could specify all the deleted files and > directories as being excluded before they are actually removed from the > server. Then the client could use --delete but not --delete-excluded and > get essentially the optimal behavior. Yes, that's what I meant. > But really as I already said this kind of thing is probably outside of > rsync's scope. The reason I said this could really be easily outside the > expected scope of rsync to ever provide an answer for is that this is > really pretty arbitrary high level application specific oddball > behavior. rsync for all it power and flexibility is still essentially a > low level and agnostic utility like cp or tar. It's some other higher > level script or application's job to decide what to copy and where and > when and why. It's only cp's job to copy a file as directed. > > But it's not black & white either. rsync is in fact specifically > intended to be a whole lot more than cp. It's by definition and by > design a rather more high level, feature rich, powerful, flexible tool, > and aims, and succeeds, in providing many and various ways to express > even very complex rules to govern file transfers. So it's entirely > possible to someday invent some generic feature that might be used in > any number of scenarios, and which this one might be just one example it > serves. > > Almost all of rsync's features could be argued either way, that they are > doable from without, and therefore should be done from without instead > of from within. But if that was really the only guiding principle then > rsync would not be very useful today, like a special cp with one twist, > and we'd all be using something else since most people have more complex > needs than that. Indeed. Of all the software packages I'm familiar with, rsync seems to attract this kind of debate the most often, for precisely the reasons you mention. > I don't know what your argument is really. Do you suppose I am attacking > rsync or Wayne or anyone else such that they need defending? In the first place, I'm telling you not to expect Wayne to change his decision. I'm not sure I would defend it per se, but I have gained a great deal of respect for his approach of being selective about features and working to make the ones that are there as robust as possible. If you think a different maintenance approach would give a product that is better for some set of purposes, it is your prerogative to try it, in collaboration with anyone you think might have similar goals. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Selective propagation of deletions, + rsync scope
On 6/2/2011 6:03 PM, Matt McCutchen wrote: On Thu, 2011-06-02 at 17:43 -0400, Brian K. White wrote: Perhaps the server side could specify all the deleted files and directories as being excluded before they are actually removed from the server. Then the client could use --delete but not --delete-excluded and get essentially the optimal behavior. Yes, that's what I meant. Almost all of rsync's features could be argued either way, Indeed. Of all the software packages I'm familiar with, rsync seems to attract this kind of debate the most often, for precisely the reasons you mention. Heh. Well bash and ksh just built-in a good chunk of an OS. And zsh and emacs built-in just as much PLUS a plugin architecture that adds everything else imaginable. In the first place, I'm telling you not to expect Wayne to change his decision. I'm not sure I would defend it per se, but I have gained a great deal of respect for his approach of being selective about features and working to make the ones that are there as robust as possible. Fair enough. Similarly, I don't necessarily need the OP's suspend-gracefully-with-informative-server-response feature at the moment. I'm only saying, that I think it does align with existing design and development thus far. Many existing features clearly and explicitly exist to allow rsync to be used as a standalone daemon, and in public settings, and clearly always strives to be as robust as possible both in itself as well as provide for as robust as possible usage by the user/script/app. And, unless someone discovers a trick I haven't spotted yet, as far as I can tell the lack of this feature does prevent or at least make less robust that basic intended usage in some situations. I haven't finished playing with the exclude file idea so it may be that the difference in behavior of an excluded directory vs a missing directory vs down service may be detectable from the client and serve as a good enough replacement for an explicit status code response from the server. -- bkw -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: IO error encountered, skipping file deletion - WTF?
On Sun, May 29, 2011 at 11:04:25PM -0700, travis+ml-rs...@subspacefield.org wrote: > Could a dangling symlink cause this problem? To answer my own question, yes, a dangling symlink on the source was the cause of the problem, and I had --copy-unsafe-symlinks is on. Thanks for the help people, that one was baffling me. -- http://www.subspacefield.org/~travis/ He who lives by the computer, dies by the computer. If you are a spammer, please email j...@subspacefield.org to get blacklisted. pgpfiQHMa9lhH.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 8188] Mechanism for taking an rsync server down for maintenance
https://bugzilla.samba.org/show_bug.cgi?id=8188 --- Comment #5 from Andrew Pollock 2011-06-03 03:56:14 UTC --- (In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #1) > > > > That's very cute, but not quite what I had in mind. > > > > I just attached a proof of concept patch that I surprised myself by being > > able > > to hack together the other night. > > > > The real thing probably needs to be configurable, and it'd probably be > > better > > to get the message from the contents of the file (if there was any). > > Someone else emailed me an untested idea to just use exclude file and > place/remove/edit the exclude file to toggle the access. > > That doesn't provide the truly graceful/informative response code or message, > but: > * toggles the access > * avoids downing the service even momentarily > * potentially avoids requiring a host admin to even do anything, since a user > may have permission to place/remove the exclude file > * it's at least distinct from failed to connect > * uses already existing features > * does not require any, let alone two different, wrapper layers in the form of > the perl (or other) wrapper script and (x)inetd to run it. > > I haven't finished playing around with that to see just how close that can > come > to the desired fully graceful process but I felt it should be mentioned as > another workaround option. The same could be said for my "patch". If the file that is checked is configurable, and the contents of the file are used to obtain the message, then it also * toggles the access * avoids downing the service even momentarily * can potentially avoid the host admin having to do anything if the file is in a user-writable location (I personally don't consider this to be a desirable feature) * doesn't require wrappers I'm prepared to continue futzing around with my patch if it has merit, I'm just not a very competent C programmer. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
