[SLUG] Bi-directional rsync?
Learned ones! To overcome some crappy web design (two servers, load balanced, no shared storage), I need to implement rsync to synchronise a directory between two servers. Trouble is, it's got to be a two way sync. I.E. File uploaded to web server 1 (don't ask - I told you it was crap web design) needs to be available to download from web server 2 if necessary for future sessions - I.E. I've got to copy the file from server 1 to server 2 in short order (sub 5 minutes). I've ALSO got to do the same from server 2 back to server 1 - so if the incoming upload goes to server 2, it's got to be copied back to server 1. Currently, I've got rsync running on both servers every 5 minutes and synchronising files from 1 to 2, and from 2 to 1. I'd like to just do this in once process - run rsync on server 1, have it connect to server 2 and pull/push all files until the directories match. I don't know if this is even possible with rsync - and if so, I don't know the options required to make it work. The current command being used is rsync -v -rlt directories Can anyone suggest a better option set which could make this a two way sync by just running on the one server? Thanks. DaZZa -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Bi-directional rsync?
rsync -v -rlt directories Can anyone suggest a better option set which could make this a two way sync by just running on the one server? You can run rsync twice on the one server rsync -options server1:/dir1 /dir1 rsync -options /dir1 server1:/dir1 with the rsync options, you probably want -u, --updateskip files that are newer on the receiver you could also play with git-annex for a smarter sync. In a nutshell the file metadata is kept under version control, not the files. There's also ceph, there was a talk on ceph at slug, i think, a few months ago. -- http://linuxsupportsydney.com.au -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Bi-directional rsync?
On 14 May 2013 09:34, Michael Chesterton che...@chesterton.id.au wrote: rsync -v -rlt directories Can anyone suggest a better option set which could make this a two way sync by just running on the one server? You can run rsync twice on the one server rsync -options server1:/dir1 /dir1 rsync -options /dir1 server1:/dir1 You know, I never thought of that. Guess this shows how often I use rsync, huh? I didn't realise I could pull and push from the same machine. :-) Thanks - will investigate that too. DaZZa -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Bi-directional rsync?
If you can time stamp when you do it. Update server 1 with files timestamp from server 2 Record new timestamp. Update server 2 from server 1 with delete. (i.e. fetch new files from server 2 onto 1, then sync server 2 which will remove dead files that you have deleted. Server 1 is the baseline server). -Original Message- From: DaZZa Sent: Tuesday, May 14, 2013 9:16 AM To: SLUG Subject: [SLUG] Bi-directional rsync? Learned ones! To overcome some crappy web design (two servers, load balanced, no shared storage), I need to implement rsync to synchronise a directory between two servers. Trouble is, it's got to be a two way sync. I.E. File uploaded to web server 1 (don't ask - I told you it was crap web design) needs to be available to download from web server 2 if necessary for future sessions - I.E. I've got to copy the file from server 1 to server 2 in short order (sub 5 minutes). I've ALSO got to do the same from server 2 back to server 1 - so if the incoming upload goes to server 2, it's got to be copied back to server 1. Currently, I've got rsync running on both servers every 5 minutes and synchronising files from 1 to 2, and from 2 to 1. I'd like to just do this in once process - run rsync on server 1, have it connect to server 2 and pull/push all files until the directories match. I don't know if this is even possible with rsync - and if so, I don't know the options required to make it work. The current command being used is rsync -v -rlt directories Can anyone suggest a better option set which could make this a two way sync by just running on the one server? Thanks. DaZZa -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html - No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.3336 / Virus Database: 3162/6320 - Release Date: 05/13/13 - No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.3336 / Virus Database: 3162/6320 - Release Date: 05/13/13 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Bi-directional rsync?
Are you forced to use rsync? If not you can use csync2 which I think would do the trick for you. On 14 May 2013 09:16, DaZZa dazzagi...@gmail.com wrote: Learned ones! To overcome some crappy web design (two servers, load balanced, no shared storage), I need to implement rsync to synchronise a directory between two servers. Trouble is, it's got to be a two way sync. I.E. File uploaded to web server 1 (don't ask - I told you it was crap web design) needs to be available to download from web server 2 if necessary for future sessions - I.E. I've got to copy the file from server 1 to server 2 in short order (sub 5 minutes). I've ALSO got to do the same from server 2 back to server 1 - so if the incoming upload goes to server 2, it's got to be copied back to server 1. Currently, I've got rsync running on both servers every 5 minutes and synchronising files from 1 to 2, and from 2 to 1. I'd like to just do this in once process - run rsync on server 1, have it connect to server 2 and pull/push all files until the directories match. I don't know if this is even possible with rsync - and if so, I don't know the options required to make it work. The current command being used is rsync -v -rlt directories Can anyone suggest a better option set which could make this a two way sync by just running on the one server? Thanks. DaZZa -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Bi-directional rsync?
On 14/05/13 09:16, DaZZa wrote: The current command being used is rsync -v -rlt directories As Michael wrote, you probably want the -u option, otherwise it's even chance whether a new file will be copied or overwritten. Jiri -- Jiří Baum j...@baum.com.au http://www.baum.com.au/sabik -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Bi-directional rsync?
Why not consider unison? On 14 May 2013 10:23, Jiří Baum j...@baum.com.au wrote: On 14/05/13 09:16, DaZZa wrote: The current command being used is rsync -v -rlt directories As Michael wrote, you probably want the -u option, otherwise it's even chance whether a new file will be copied or overwritten. Jiri -- Jiří Baum j...@baum.com.au http://www.baum.com.au/sabik -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- Christopher Vance -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Bi-directional rsync?
glusterfs http://www.gluster.org/community/documentation/index.php/Getting_started_rrqsg http://www.gluster.org/ :) -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Bi-directional rsync?
If it were me I'd look at making something based on ionotify Then scp the files between the servers. (note you will need to check the remote side to see if the file exists so you don't make a loop) Upside is there's no 5 minute window. I'd keep the rsync as well but run it much less often, mainly as a belt and suspenders option to keep the files in sync. Otherwise cheat. If the load isn't high (and the use of rsync implies it isnt) just mount from one to the other over something like sshfs(bad but easy) or nfs (better). Perhaps put an rsync push from the file server to the backup system for backup purposes (or again ionotify). On 14/05/13 09:16, DaZZa wrote: Learned ones! To overcome some crappy web design (two servers, load balanced, no shared storage), I need to implement rsync to synchronise a directory between two servers. Trouble is, it's got to be a two way sync. I.E. File uploaded to web server 1 (don't ask - I told you it was crap web design) needs to be available to download from web server 2 if necessary for future sessions - I.E. I've got to copy the file from server 1 to server 2 in short order (sub 5 minutes). I've ALSO got to do the same from server 2 back to server 1 - so if the incoming upload goes to server 2, it's got to be copied back to server 1. Currently, I've got rsync running on both servers every 5 minutes and synchronising files from 1 to 2, and from 2 to 1. I'd like to just do this in once process - run rsync on server 1, have it connect to server 2 and pull/push all files until the directories match. I don't know if this is even possible with rsync - and if so, I don't know the options required to make it work. The current command being used is rsync -v -rlt directories Can anyone suggest a better option set which could make this a two way sync by just running on the one server? Thanks. DaZZa -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Bi-directional rsync?
Btsync might also be an option? I haven't tested this myself but will be doing so this week. On 14/05/2013 12:12 PM, Jake Anderson ya...@vapourforge.com wrote: If it were me I'd look at making something based on ionotify Then scp the files between the servers. (note you will need to check the remote side to see if the file exists so you don't make a loop) Upside is there's no 5 minute window. I'd keep the rsync as well but run it much less often, mainly as a belt and suspenders option to keep the files in sync. Otherwise cheat. If the load isn't high (and the use of rsync implies it isnt) just mount from one to the other over something like sshfs(bad but easy) or nfs (better). Perhaps put an rsync push from the file server to the backup system for backup purposes (or again ionotify). On 14/05/13 09:16, DaZZa wrote: Learned ones! To overcome some crappy web design (two servers, load balanced, no shared storage), I need to implement rsync to synchronise a directory between two servers. Trouble is, it's got to be a two way sync. I.E. File uploaded to web server 1 (don't ask - I told you it was crap web design) needs to be available to download from web server 2 if necessary for future sessions - I.E. I've got to copy the file from server 1 to server 2 in short order (sub 5 minutes). I've ALSO got to do the same from server 2 back to server 1 - so if the incoming upload goes to server 2, it's got to be copied back to server 1. Currently, I've got rsync running on both servers every 5 minutes and synchronising files from 1 to 2, and from 2 to 1. I'd like to just do this in once process - run rsync on server 1, have it connect to server 2 and pull/push all files until the directories match. I don't know if this is even possible with rsync - and if so, I don't know the options required to make it work. The current command being used is rsync -v -rlt directories Can anyone suggest a better option set which could make this a two way sync by just running on the one server? Thanks. DaZZa -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/**mailinglists.htmlhttp://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
