Thanks for the explanation Vincent. I had a similar experience when I did the full backup because my BACKUPDIR was in the fossy user directory. My full backup (60GB) also took about 6 hours with a single rsync thread.
I will look at the instructions and add something specific about excluding BACKUPDIR using the include a&b file. Mary ________________________________ From: fossology-boun...@fossology.org [mailto:fossology-boun...@fossology.org] On Behalf Of Ma, Dong (vinc...@gdcc-bj-most) Sent: Thursday, January 28, 2010 8:34 PM To: Furosh One Cc: fossology@fossology.org Subject: Re: [FOSSology] Backups take a very, very long time... Possible culprit? Hi Gibran, I think your point will be right, if BACKUPDIR is located in an area that is being backed up, there may be causing a loop. Why you mounted BACKUPDIR directory share with other servers in the repository, I didn't suggest you do this. Anyway, if you want to exclude BACKUPDIR use include a&b files, just add like this: + /repository/ + /db/ - /* + /repository/clear[1-3]/ - /repository/* -/repository/clear1/backup/ The BACKUPDIR will not be backup. Also I should share my experience about backup whole repository time, there will be a long time to cost, in my test my repository upload a Fedora11 source ISO, the gold files is about 4.2G, and unupacked files cost more than 55G storage, I backup this more than 60G repository use 2 rsync thread, it cost about 4 hours to finished. This test can give you for a reference. If cross NFS, I think will be slower than this. So we also provide a only backup gold files solution here: http://fossology.org/task:fossology_data_backup_and_restore_instruction_-_gold_files Hope helps for you. Thanks, Vincent From: Furosh One [mailto:fur...@gmail.com] Sent: Friday, January 29, 2010 7:19 AM To: Ma, Dong (vinc...@gdcc-bj-most) Subject: Fwd: Backups take a very, very long time... Possible culprit? Hello Ma Dong, I know you gave me some good help last time with the include/exclude files in the backup-scripts for fossology. Just wanted to forward this to you directly in case you might not see the mailing list ones for a while. Any help is appreciated. -Regards, Gibran ---------- Forwarded message ---------- From: Furosh One <fur...@gmail.com<mailto:fur...@gmail.com>> Date: Thu, Jan 28, 2010 at 1:30 PM Subject: Backups take a very, very long time... Possible culprit? To: fossology@fossology.org<mailto:fossology@fossology.org> I've played with the backups of our multi-setup repositories and had a couple of questions. Although the script seems to work that I downloaded from : http://fossology.svn.sourceforge.net/viewvc/fossology/trunk/fossology/utils/backup/ Our repos are fairly big in size and we are directly connected to a giga-switch, I don't think the bottleneck is the network here. It is using network because all hosts are mounted via NFS and the BACKUP_LOCATION is on one of those mounted shares (the one with the most space avail). But all 6 machines are on the same subnet, same switch, and shouldn't really need to go out to the router and then back again. I would think it would be quick as far as network. However, its possible that the BACKUPDIR is located in an area that is also being backed up, which may be causing a "loop". BACKUPDIR=/srv/fossology/repository/clear1/BACKUPS There's a small debate here at the office whether this is a network issue or a backup issue, or simply just a lot of data and processing (compression) of the entire backup-script that is taking such a long time. I just caught the fact that include has +/repository/ (which may also include our BACKUPDIR). To give you an idea of how long it was taking I started the script at 9 AM and by 3 PM it still hadn't finished. It was probably just 50% completed. We needed to do a scan so we had to kill the backup job. Exisintg include a&b files: =================== gmarq...@san-clear5:~/Backup_Scripts/fossy_BackupScripts$ cat backup_include_a.mod + /repository/ + /db/ - /* + /repository/clear[1-3]/ - /repository/* gmarq...@san-clear5:~/Backup_Scripts/fossy_BackupScripts$ cat backup_include_b.mod + /repository/ - /* + /repository/clear[4-6]/ - /repository/* =================== Do I need to add my BACKUPDIR to "-" exclude? How would I write that to be able to still "include" the repo in clear1? -/repository/clear1/BACKUPS/* <-- Would this be the way to do it? And does the order in the file matter? Thanks! -Regards, Gibran
_______________________________________________ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
