Prepared meta directory for one to many sync
Hi to all, sorry, but I am not subscribed to this list. I have a question, because I did not find any answer for my problem. If you have so, please let me know! We use rsync to synchronise "one-to-many" workstations. We use our obnovang software (http://obnova.bp.opf.slu.cz/). You can read more info on this page, but in general, we are doing backups and refresh of many workstations (aprox. 200) which are in our PC labs. So we prepare one workstation, rsync data from local filesystem to rsync server and next, all stations will connect to rsync server and refresh their local filesystems to be copy of server. Everything works great, thank you for great rsync code :) but we have some suggestions how to improve speed of our process. - We cannot enable compression because file is compressed just when it is sended. If 100PCs are connected to rsync server and we will turn on compression, server will die. - We would want to use compression because some links are slow and compression would increase speed. - Same to partial file transfer. We use --whole-file , because if server would compute hashes each time when client connect, server will die.. I know our scenario is similar to some rsync download mirror and maybe it is solved somewhere. But I did not find any solution yet. Maybe it would be great if rsync have some "meta" directory, where it would prepare all infos about stored files. Like hashes, precompressed files, etc. This "meta" directory could be created or refreshed by some options. After creating and pointing to that directory, entire mirroring process could be much more faster. Maybe: $ rsync -r --prepare-meta /tmp/meta ./ $ rsync -r --meta-dir /tmp/meta/ ./ rsync://somewhere/ Please is it theoreticaly possible ? Or is there some problem to precompress and prehash files? Thank you for any answer, Lukas Macura -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Prepared meta directory for one to many sync
On Wed, 2009-05-06 at 09:20 +0200, Lukas Macura wrote: > Hi to all, > > sorry, but I am not subscribed to this list. I have a question, because > I did not find any answer for my problem. If you have so, please let me > know! > > We use rsync to synchronise "one-to-many" workstations. We use our > obnovang software (http://obnova.bp.opf.slu.cz/). You can read more info > on this page, but in general, we are doing backups and refresh of many > workstations (aprox. 200) which are in our PC labs. So we prepare one > workstation, rsync data from local filesystem to rsync server and next, > all stations will connect to rsync server and refresh their local > filesystems to be copy of server. > > Everything works great, thank you for great rsync code :) but we have > some suggestions how to improve speed of our process. > > - We cannot enable compression because file is compressed just when it > is sended. If 100PCs are connected to rsync server and we will turn on > compression, server will die. > - We would want to use compression because some links are slow and > compression would increase speed. > - Same to partial file transfer. We use --whole-file , because if server > would compute hashes each time when client connect, server will die.. Why not try to use multi-server. for example, this is a central server. central server ---> backup to three mirror server,named with #1, #2, #3 #1 mirror server ---> 50 PC #2 mirror server ---> 50 PC #3 mirror server ---> 50 PC #4 mirror server ---> 50 PC It might reduce the work load of central server. What do u think? > > I know our scenario is similar to some rsync download mirror and maybe > it is solved somewhere. But I did not find any solution yet. > > Maybe it would be great if rsync have some "meta" directory, where it > would prepare all infos about stored files. Like hashes, precompressed > files, etc. This "meta" directory could be created or refreshed by some > options. After creating and pointing to that directory, entire mirroring > process could be much more faster. > > Maybe: > $ rsync -r --prepare-meta /tmp/meta ./ > $ rsync -r --meta-dir /tmp/meta/ ./ rsync://somewhere/ > > Please is it theoreticaly possible ? Or is there some problem to > precompress and prehash files? > > Thank you for any answer, > Lukas Macura > > > -- USI/SH RD Dept. Universal Scientific Industrial (Shanghai) Co., Ltd Fax: +86-21-58967931Tel: +86-21-58966996 ext. 1400 42th building, 1387 Zhang Dong road, Pudong New Area, Shanghai, P.R. China 201203 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Prepared meta directory for one to many sync
Hi Luke, 2009/5/6 Lukas Macura : > I know our scenario is similar to some rsync download mirror and maybe > it is solved somewhere. But I did not find any solution yet. > > Maybe it would be great if rsync have some "meta" directory, where it > would prepare all infos about stored files. Like hashes, precompressed > files, etc. This "meta" directory could be created or refreshed by some > options. After creating and pointing to that directory, entire mirroring > process could be much more faster. > > Maybe: > $ rsync -r --prepare-meta /tmp/meta ./ > $ rsync -r --meta-dir /tmp/meta/ ./ rsync://somewhere/ > > Please is it theoreticaly possible ? Or is there some problem to > precompress and prehash files? Have a look at --write-batch and --read-batch. David -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
