Robin Lee Powell wrote at about 16:28:43 -0800 on Monday, December 14, 2009: > On Mon, Dec 14, 2009 at 02:17:01PM -0500, Jeffrey J. Kosowsky wrote: > > Robin Lee Powell wrote at about 10:12:28 -0800 on Monday, December 14, > > 2009: > > > On Mon, Dec 14, 2009 at 07:57:10AM -0600, Les Mikesell wrote: > > > > > > > > You can, however, explicitly break the runs at top-level > > > > directory boundaries and mount points if you have a problem > > > > with the size. > > > > > > That doesn't always work; it certainly doesn't work in my case. > > > Millions of files scattered unevenly around a single file > > > system; I don't even know where the concentrations are because > > > it takes so long to run du/find on this filesystem, and it > > > degrades performance in a way that makes the client upset. > > > > > > > I wonder how common your use case is where the files are scattered > > so unevenly, so unpredictably, and in such a dynamically changing > > manner that you can't make a dent in the complexity by subdividing > > the share into smaller pieces. If the system is so dynamic and > > unpredictable, then perhaps the more robust solution is to see > > whether the data storage can be organized better... > > We're a hosting company; these are backups for clients. We can't > enforce that sort of shift. > > Since I've got two clients with exactly the same issue (file trees > in the millions of files, that take ten hours or more just to run a > "find" on, let alone du), though, I'm inclined to think that it's > not *that* uncommon. > > I find it more than a little odd that you are telling me "OMG > HARDLINKS!" on the one hand, and "subdivide the share" on the other, > since by definition subdivided shares break hard links. What's up > with that? (not intended confrontationally, just confused)
No confrontation taken ;) But if you follow the thread, I wasn't the one initially suggesting subdividing the share nor am I pushing that solution. However, I think that if you really can't do it all in one session, then subdividing is a more recommended solution than suggesting a change to the backuppc source (unless you just do it on your own). Perhaps your setup is pushing the envelope more than others, but I haven't seen many people with issues like yours. It would be interesting to hear whether others are experiencing the same problem. Also, before you go off rewriting backuppc, it might be productive to investigate whether there are any correctable software/hardware issues in your hosting setup that might be contributing. You postulate several potential causes for your SIGPIPE errors, but it would be good to identify the actual source(s) and trigger(s). ssh and rsync are pretty mature and robust software and they shouldn't just "flake" out just from being active for 15 hours or from being pounded hard with lots of files. While my setup is much much smaller than yours, I keep ssh connections open for months using very off-the-shelf consumer grade hardware. And if there is something flakey in your hardware/software, you probably want to know that if you are a hosting company... ------------------------------------------------------------------------------ Return on Information: Google Enterprise Search pays you back Get the facts. http://p.sf.net/sfu/google-dev2dev _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/