Re: [BackupPC-users] Different Blackout Periods
Hi, I used one file per message. The amount of messages are about 100 per hour. In my setup now, I can see that the number of files are increasing every incremental backup (the incr backup self takes 0.5mins). IncrLevels is set to 1. If read the doc about this value but I think I didn't understand this right. In my case, were I takes (7*24) around 168 incr backups between two full backups, which value were a good choice for IncrLevels? Is it 1,2,3,4,5...,167? I get a little bit confused because my english is not very well. Thank you for helping! Greetings Alex -Ursprüngliche Nachricht- Von: Les Mikesell [mailto:lesmikes...@gmail.com] Gesendet: Freitag, 3. April 2015 18:16 An: General list for user discussion, questions and support Betreff: Re: [BackupPC-users] Different Blackout Periods On Thu, Apr 2, 2015 at 4:25 AM, Alexander Rehbein alexander.rehb...@fmex.de wrote: I've looked in the archive. Sometimes the incr backup is only 20mb, sometimes 200mb. Perhaps it is an better idea to only backup /var/vmail directory hourly instead of the full root path. First, make sure you understand how backuppc works - and the type of your mail files. If you use the old mbox mail format where messages are appended to a single growing mailbox file, and use rsync, only the difference will be transfered, but there is some CPU work involved to compute the changes, and on the server side a complete new file copy will be constructed and stored separately. Xfer methods other than rsync will send the whole changed file.If your mail storage uses one message per file like maildir format, rsync will only send the new files (whether doing a full or incremental) and the ones where the content matches the previous copy will all be pooled without taking additional space. Non-rsync xfers will send only new files on incrementals and everything on fulls, but existing content is still found and pooled in storage. Note that 'new' means newer than the previous full unless you have configured incremental levels. So, there won't be much difference in the storage used regardless of your mix of incremental/full runs. The main difference will be in the time they take to complete and possibly the performance impact on the target system. An incremental rsync run quickly skips fields where the length/timestamp match the previous copy where the fulls do a full read of the target content to do a block-checksum comparison, Also, the fulls will rebuild the archive directory tree which may be time-consuming with many small files, but that becomes the new base for incrementals, making the next ones more efficient. -- Les Mikesell lesmikes...@gmail.com -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Different Blackout Periods
Hello Falko, thank you. The bbc Option is a very good idea for archive the mails. I will have a look. Until now I setup two hosts in backuppc. One which only backup mails hourly and one which backup the whole server daily without maildir. Greetings Alex -Ursprüngliche Nachricht- Von: Falko Trojahn [mailto:n...@trojahn.de] Gesendet: Freitag, 3. April 2015 15:56 An: General list for user discussion, questions and support Betreff: Re: [BackupPC-users] Different Blackout Periods Hello Alexander, there's a slightly off-topic suggestion for you. Alexander Rehbein wrote on 02.04.2015 11:25: You told something about an delta. Are diff backups better for me? I want to do the following: I have an Mailserver which should do a full backup every week. It could happen that I delete an email faulty, so I decided to backup the mails every hour. This I want do with incr backups. If a mail get lost I can restore this Mail from the backup archive. I've looked in the archive. Sometimes the incr backup is only 20mb, sometimes 200mb. Perhaps it is an better idea to only backup /var/vmail directory hourly instead of the full root path. According to your needs, may be there is another possibility apart from stressing your mail server with hourly backups. And, your statement If a mail get lost I can restore this Mail from the backup archive. is only partly true. If both the mail arrives and is deleted between two incremental backups, it will get lost, too. So what can you do is: use the always_bcc feature of your MTA - e.g. if you use dovecot with postfix, have a look at http://www.postfix.org/ADDRESS_REWRITING_README.html#auto_bcc and put it in master.cf in the cleanup process. I assume your backuppc server is physically on another host. There are several possibilities: - send all incoming and outgoing mail of the server to one or several bcc accounts, using fetchmail on the backup host to fetch the mail and deliver it * to local account * to a file, e.g. using procmail - send all incoming and outgoing mail per domain or per user directly to an archive host, which must not be the same as the backuppc host, but can = backup the archive host using backuppc as usual Find attached an example .procmailrc which we use together with one incoming and one outgoing bcc account per server to split important mail from not so relevant ones. If there are several mail processes on the server (content/antivirus filters etc.) make sure you generate the bcc mails only once. The generated files can be gzipped at end of month, and even archived (to fullfill the archiving duty in germany: http://de.wikipedia.org/wiki/E-Mail-Archivierung ) HTH Falko # === ~/.procmailrc === PATH=$HOME/bin:/usr/bin:/bin:/usr/local/bin MAILDIR=$HOME/Mail # Youd better make sure it exists YEAR=`date +%Y` MONTH=`date +%m` DEFAULT=$MAILDIR/IN-$YEAR-$MONTH # for outgoing control user: # DEFAULT=$MAILDIR/OUT-$YEAR-$MONTH LOGFILE=$MAILDIR/_maillog-$YEAR-$MONTH :0: * ^X-Spam-Status: Yes,.*$ SPAM-$YEAR-$MONTH :0: * ^X-Quarantine-ID: .*$ BADHEADER-$YEAR-$MONTH :0: * ^From: logcheck@ SYSTEM-$YEAR-$MONTH :0: * ^From: root@ SYSTEM-$YEAR-$MONTH :0: * ^List-Id: .*$ MAILINGLIST-$YEAR-$MONTH -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Different Blackout Periods
One more idea. Instead of saving so much incr backups, perhaps it would be better to keep for example only 3 and delete the other, because, as you told me, the incr backups hold every changed file since the full backup. -Ursprüngliche Nachricht- Von: Les Mikesell [mailto:lesmikes...@gmail.com] Gesendet: Freitag, 3. April 2015 18:16 An: General list for user discussion, questions and support Betreff: Re: [BackupPC-users] Different Blackout Periods On Thu, Apr 2, 2015 at 4:25 AM, Alexander Rehbein alexander.rehb...@fmex.de wrote: I've looked in the archive. Sometimes the incr backup is only 20mb, sometimes 200mb. Perhaps it is an better idea to only backup /var/vmail directory hourly instead of the full root path. First, make sure you understand how backuppc works - and the type of your mail files. If you use the old mbox mail format where messages are appended to a single growing mailbox file, and use rsync, only the difference will be transfered, but there is some CPU work involved to compute the changes, and on the server side a complete new file copy will be constructed and stored separately. Xfer methods other than rsync will send the whole changed file.If your mail storage uses one message per file like maildir format, rsync will only send the new files (whether doing a full or incremental) and the ones where the content matches the previous copy will all be pooled without taking additional space. Non-rsync xfers will send only new files on incrementals and everything on fulls, but existing content is still found and pooled in storage. Note that 'new' means newer than the previous full unless you have configured incremental levels. So, there won't be much difference in the storage used regardless of your mix of incremental/full runs. The main difference will be in the time they take to complete and possibly the performance impact on the target system. An incremental rsync run quickly skips fields where the length/timestamp match the previous copy where the fulls do a full read of the target content to do a block-checksum comparison, Also, the fulls will rebuild the archive directory tree which may be time-consuming with many small files, but that becomes the new base for incrementals, making the next ones more efficient. -- Les Mikesell lesmikes...@gmail.com -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Different Blackout Periods
On Thu, Apr 2, 2015 at 4:25 AM, Alexander Rehbein alexander.rehb...@fmex.de wrote: I've looked in the archive. Sometimes the incr backup is only 20mb, sometimes 200mb. Perhaps it is an better idea to only backup /var/vmail directory hourly instead of the full root path. First, make sure you understand how backuppc works - and the type of your mail files. If you use the old mbox mail format where messages are appended to a single growing mailbox file, and use rsync, only the difference will be transfered, but there is some CPU work involved to compute the changes, and on the server side a complete new file copy will be constructed and stored separately. Xfer methods other than rsync will send the whole changed file.If your mail storage uses one message per file like maildir format, rsync will only send the new files (whether doing a full or incremental) and the ones where the content matches the previous copy will all be pooled without taking additional space. Non-rsync xfers will send only new files on incrementals and everything on fulls, but existing content is still found and pooled in storage. Note that 'new' means newer than the previous full unless you have configured incremental levels. So, there won't be much difference in the storage used regardless of your mix of incremental/full runs. The main difference will be in the time they take to complete and possibly the performance impact on the target system. An incremental rsync run quickly skips fields where the length/timestamp match the previous copy where the fulls do a full read of the target content to do a block-checksum comparison, Also, the fulls will rebuild the archive directory tree which may be time-consuming with many small files, but that becomes the new base for incrementals, making the next ones more efficient. -- Les Mikesell lesmikes...@gmail.com -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Backups Schedule
Hello, How to configure backuppc Schedule for this: 1-day - full 2-day - empty 3-day - empty 4-day - incremental 5-day - empty 6-day - empty 7-day - incremental . 14-day (or 15-day) - clear 1 day full and create new full Need ONE full on 2 week and time to time incremental I resetup FullPeriod - 13.97 FullKeepCnt - 1 FullKeepCntMin - 1 IncrPeriod - 2.97 IncrKeepCnt - 12 But I see backupams: Backup# Type Filled Level Start Date Duration/mins Age/days Server Backup Path 3 http://77.241.206.72:69/backuppc?action=browsehost=mc2.minehost.ltnum=3 full yes 0 3/7 10:46 1010.1 27.2 /var/lib/BackupPC//pc/www/3 4 http://77.241.206.72:69/backuppc?action=browsehost=mc2.minehost.ltnum=4 incr no 1 3/10 12:00 545.8 24.2 /var/lib/BackupPC//pc/www/4 5 http://77.241.206.72:69/backuppc?action=browsehost=mc2.minehost.ltnum=5 incr no 1 3/13 13:00 497.3 21.1 /var/lib/BackupPC//pc/www/5 6 http://77.241.206.72:69/backuppc?action=browsehost=mc2.minehost.ltnum=6 full yes 0 3/16 23:19 1114.2 17.7 /var/lib/BackupPC//pc/www/6 7 http://77.241.206.72:69/backuppc?action=browsehost=mc2.minehost.ltnum=7 incr no 1 3/20 11:24 164.9 14.2 /var/lib/BackupPC//pc/www/7 8 http://77.241.206.72:69/backuppc?action=browsehost=mc2.minehost.ltnum=8 incr no 1 3/24 06:04 275.8 10.4 /var/lib/BackupPC//pc/www/8 9 http://77.241.206.72:69/backuppc?action=browsehost=mc2.minehost.ltnum=9 incr no 1 3/27 09:36 204.3 7.3 /var/lib/BackupPC//pc/www/9 10 http://77.241.206.72:69/backuppc?action=browsehost=mc2.minehost.ltnum=10 full yes 0 4/2 13:55 511.9 1.1 /var/lib/BackupPC//pc/www/10 (sorry FullPeriod - change from 7) But why one FULL and two incremental then full, why not full - 10x incremental... -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Different Blackout Periods
Hello Alexander, there's a slightly off-topic suggestion for you. Alexander Rehbein wrote on 02.04.2015 11:25: You told something about an delta. Are diff backups better for me? I want to do the following: I have an Mailserver which should do a full backup every week. It could happen that I delete an email faulty, so I decided to backup the mails every hour. This I want do with incr backups. If a mail get lost I can restore this Mail from the backup archive. I've looked in the archive. Sometimes the incr backup is only 20mb, sometimes 200mb. Perhaps it is an better idea to only backup /var/vmail directory hourly instead of the full root path. According to your needs, may be there is another possibility apart from stressing your mail server with hourly backups. And, your statement If a mail get lost I can restore this Mail from the backup archive. is only partly true. If both the mail arrives and is deleted between two incremental backups, it will get lost, too. So what can you do is: use the always_bcc feature of your MTA - e.g. if you use dovecot with postfix, have a look at http://www.postfix.org/ADDRESS_REWRITING_README.html#auto_bcc and put it in master.cf in the cleanup process. I assume your backuppc server is physically on another host. There are several possibilities: - send all incoming and outgoing mail of the server to one or several bcc accounts, using fetchmail on the backup host to fetch the mail and deliver it * to local account * to a file, e.g. using procmail - send all incoming and outgoing mail per domain or per user directly to an archive host, which must not be the same as the backuppc host, but can = backup the archive host using backuppc as usual Find attached an example .procmailrc which we use together with one incoming and one outgoing bcc account per server to split important mail from not so relevant ones. If there are several mail processes on the server (content/antivirus filters etc.) make sure you generate the bcc mails only once. The generated files can be gzipped at end of month, and even archived (to fullfill the archiving duty in germany: http://de.wikipedia.org/wiki/E-Mail-Archivierung ) HTH Falko # === ~/.procmailrc === PATH=$HOME/bin:/usr/bin:/bin:/usr/local/bin MAILDIR=$HOME/Mail # Youd better make sure it exists YEAR=`date +%Y` MONTH=`date +%m` DEFAULT=$MAILDIR/IN-$YEAR-$MONTH # for outgoing control user: # DEFAULT=$MAILDIR/OUT-$YEAR-$MONTH LOGFILE=$MAILDIR/_maillog-$YEAR-$MONTH :0: * ^X-Spam-Status: Yes,.*$ SPAM-$YEAR-$MONTH :0: * ^X-Quarantine-ID: .*$ BADHEADER-$YEAR-$MONTH :0: * ^From: logcheck@ SYSTEM-$YEAR-$MONTH :0: * ^From: root@ SYSTEM-$YEAR-$MONTH :0: * ^List-Id: .*$ MAILINGLIST-$YEAR-$MONTH -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
