Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Well, I'm having concurrent backups, but they use different TCP ports, thus I can --sport 8873 and --sport 8874 and so on for my clients. If you get this working, let us know how far off it is from the values shown for duration and transfer rate (and a 10-25% allowance for ssh compression). Will let know, gotta try it out. F. -- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian Group, R/GA, Big Spaceship. http://www.creativitycat.com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Depending on whether you want to entertain bored users, bill your clients for bandwidth, or conduct scientific measurements, the answers are likely to be vastly different. I need to give my customers statistics about how much data has been transferred each month, which means summing up day-by-day the transferred amount of data. For the HDD space usage, I simply rely on what my Host Summary page is telling me (Full Size). [cut] Do you already have some sort of practical suggestion? iptables -I INPUT -s client_addr -d backuppc_server_addr -p tcp --sport 22 (and delete it with the same rule with -D instead of -I). Supposing you only have one concurrent backup to one host and no other ssh usage. You might prefer to count outgoing traffic (well, no, but maybe incoming + outgoing). Note that the rule has no target - it's only for accounting. Well, I'm having concurrent backups, but they use different TCP ports, thus I can --sport 8873 and --sport 8874 and so on for my clients. If you talk about the same things I am clueless about (skip, create, pool, same, and so on), then I'd be *too* interested in some precise definitions. I can give you some imprecise ones. same - rsync determined that file matches and does not need to be transferred pool - transferred file matched one already in the pool create - transferred file did not match any existing pool file What I interpreted was that same and skip have the same meaning: file is not getting transferred. Why then using *two* words to define a seamingly identical behaviour? The create was clear to me, but the pool one not so clear: I had in Incr Backup 16 a create statement for a big file (2.7GB). The subsequent backup (Incr Backup 17) the same file (which has *not* been changed in any way, because it's a static ZIP file that was added before backup 16) was indicated as pool: would this be meaning that the file had been transferred again and only after being transferred, BackupPC recognized that it was already in the pool (and matching)? This is the part I don't understand actually... To be honest, I think they're all rather self-explaining. I didn't do any research on the answers, it's just what figures from observation. In particular, on my import backup from local source quest, I was really worried when I got lots of pool lines on the first remote backup where it should have read same. That explained rather well why the backup was taking ages. So your case was the same as mine above? Are you saying that the backup was taking ages because it was re-transferring your data? Thanks for helping, F. -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Hi, Boniforti Flavio wrote on 2009-05-19 08:53:31 +0200 [RE: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format]: Depending on whether you want to entertain bored users, bill your clients for bandwidth, or conduct scientific measurements, the answers are likely to be vastly different. I need to give my customers statistics about how much data has been transferred each month, which means summing up day-by-day the transferred amount of data. so it's the entertain bored users case :-). Your definition leaves room for interpretation. For instance, if that number is more readily available, you could sum up the uncompressed data streams. On the other hand, you could leave iptables accounting rules in place all the time and just read out the counters (and zero them) once a month (assuming your BackupPC server doesn't reboot). Aside from that, you will probably be counting in GB, not in bytes. Do you already have some sort of practical suggestion? iptables -I INPUT -s client_addr -d backuppc_server_addr -p tcp --sport 22 [...] Well, I'm having concurrent backups, but they use different TCP ports, thus I can --sport 8873 and --sport 8874 and so on for my clients. Even better. Those ports will not be used for maintainance, I suppose? Even if so, I guess counting that traffic wouldn't strictly be wrong ... What I interpreted was that same and skip have the same meaning: file is not getting transferred. Why then using *two* words to define a seamingly identical behaviour? I can't actually find skip in my XferLOGs. Probably because it only appears with logLevel = 2 (at least for rsync). Strange. You always seem to find issues which, when looking at the code, disappear. There is one issue though: your logLevel is set too high (unless you are actually tracking a problem, which, in my experience, is not the case). I bet an 'ls -l' of your pc/ directories doesn't show as nicely which backups are full and which are incremental. With mine it's really obvious from the XferLOG files. skip - unchanged file skipped in incremental same - file that would normally have been transferred (full backup or attribute change) turns out to be identical to reference file, no transfer needed You obviously won't get same for tar/smb backups (nor will you get skip, because the files are skipped by the sender without notice to BackupPC). The create was clear to me, but the pool one not so clear: I had in Incr Backup 16 a create statement for a big file (2.7GB). The subsequent backup (Incr Backup 17) the same file (which has *not* been changed in any way, because it's a static ZIP file that was added before backup 16) was indicated as pool: would this be meaning that the file had been transferred again and only after being transferred, BackupPC recognized that it was already in the pool (and matching)? This is the part I don't understand actually... By default, that is normal. Read about how incremental backups work, in particular, which backups they are based on. There are, of course, cases where this would not be normal, but I've run out of motivation for going into details about hypothetical problems. [...] In particular, on my import backup from local source quest, I was really worried when I got lots of pool lines on the first remote backup where it should have read same. That explained rather well why the backup was taking ages. So your case was the same as mine above? No. Are you saying that the backup was taking ages because it was re-transferring your data? Yes. Regards, Holger -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Hallo Holger, I need to give my customers statistics about how much data has been transferred each month, which means summing up day-by-day the transferred amount of data. so it's the entertain bored users case :-). Your definition leaves room for interpretation. For instance, if that number is more readily available, you could sum up the uncompressed data streams. On the other hand, you could leave iptables accounting rules in place all the time and just read out the counters (and zero them) once a month (assuming your BackupPC server doesn't reboot). I *really* think I'll be collecting iptables data... Well, I'm having concurrent backups, but they use different TCP ports, thus I can --sport 8873 and --sport 8874 and so on for my clients. Even better. Those ports will not be used for maintainance, I suppose? Even if so, I guess counting that traffic wouldn't strictly be wrong ... Nothing else than ssh-tunnelled BackupPC dumps. What I interpreted was that same and skip have the same meaning: file is not getting transferred. Why then using *two* words to define a seamingly identical behaviour? I can't actually find skip in my XferLOGs. Probably because it only appears with logLevel = 2 (at least for rsync). Indeed I'm using XferLogLevel = 2 Skip actually only appears in Incr XFerLogs, which it seems then to be substituted by same in Full backups... Strange. You always seem to find issues which, when looking at the code, disappear. There is one issue though: your logLevel is set too high (unless you are actually tracking a problem, which, in my experience, is not the case). I bet an 'ls -l' of your pc/ directories doesn't show as nicely which backups are full and which are incremental. With mine it's really obvious from the XferLOG files. I actually put XferLogLevel back to 1, let's see what will happen... By default, that is normal. Read about how incremental backups work, in particular, which backups they are based on. Will read about it in the BackupPC Documentation Are you saying that the backup was taking ages because it was re-transferring your data? Yes. OK, still a bit confused... Will eventually come back on this issue later on... Regards and thanks for your time. F. -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Boniforti Flavio wrote: Hallo Holger, I need to give my customers statistics about how much data has been transferred each month, which means summing up day-by-day the transferred amount of data. so it's the entertain bored users case :-). Your definition leaves room for interpretation. For instance, if that number is more readily available, you could sum up the uncompressed data streams. On the other hand, you could leave iptables accounting rules in place all the time and just read out the counters (and zero them) once a month (assuming your BackupPC server doesn't reboot). I *really* think I'll be collecting iptables data... Well, I'm having concurrent backups, but they use different TCP ports, thus I can --sport 8873 and --sport 8874 and so on for my clients. Even better. Those ports will not be used for maintainance, I suppose? Even if so, I guess counting that traffic wouldn't strictly be wrong ... Nothing else than ssh-tunnelled BackupPC dumps. What I interpreted was that same and skip have the same meaning: file is not getting transferred. Why then using *two* words to define a seamingly identical behaviour? I can't actually find skip in my XferLOGs. Probably because it only appears with logLevel = 2 (at least for rsync). Are you saying that the backup was taking ages because it was re-transferring your data? Yes. OK, still a bit confused... Will eventually come back on this issue later on... - From my reading of the mailing list, and usage of backuppc, I think it works something like this: A full backup (level 0) will backup all files, using rsync, only modified portions of existing files, and new files are transferred. An incremental backup (level 1 - 9) will backup all new files, and modified files since the last backup of a lower level. In older versions, (I think 2.x) there was only one level of incremental backups, hence all incremental backups would re-transfer all modified/new data compared to the last full backup. Even if the file has not changed since the last incremental backup. This would explain what you saw, the file was transferred again for each incremental backup (slow backup times) but after transfer, backuppc decided a matching file already existed in the pool, and so it was discarded and linked to the pool file. In newer versions, it is possible to configure (optional) different levels of backup (1 - 9). Thus, if you set IncrLevels = [1,2,3,4] and you have four incrementals between each full backup, then you will never re-transfer a file which rsync could know has not changed since the most recent backup unless the filename/path has changed. However, this increases the cpu of the backuppc because it needs to merge the full backup plus up to 3 incremental backups to complete the fourth incremental. I think you should find more definitive (and correct) documentation in the changelog or documentation if you search for IncrLevel/IncrLevels for where this feature was introduced. I also recall (but could be wrong) that the next version of backuppc would also (possibly) apply the same logic for a second backup of the SAME level. In effect, IncrLevels = [1,2,3,4] would become the same as IncrLevels = [1,1,1,1] (which is the same as IncrLevels = [1]). To me, especially with rsync/rsyncd backups, this seemed to make a lot of sense, though I can't comment on the effects this might have if using smb/tar/ftp/etc... So, in short, this is why I currently use IncrLevels = [1,2,3] and do full backups after every 3rd incremental If anyone thinks the above is grossly incorrect, please feel free to correct me before someone follows my totally wrong random ramblings... If anyone wants to implement or follow the above, I strongly suggest you find corroborating evidence elsewhere first. Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkoS0bwACgkQGyoxogrTyiXhsACguJxxmVd7pkX2zlHWmxgUfgA8 OjAAnAhNxHMZx6fOG8R2kaNpZmTixPmR =vkZT -END PGP SIGNATURE- -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Exactly, perhaps the a better (but more work involved) solution would be to create a new page which shows a nice pretty graph of the various numbers instead of stacks of numbers in great big tables... Well, if somebody is willing to cooperate with me to do it, I'll be willing to learn and contribute. A *very nice* add-on would be having graphs for each host, not only for the whole pool. I'd really love to see ho my single hosts evolve in relation to GBs occupied. Also, I'd like to see how transfer rates may be varying... Regards, F. -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
I am not used to consider minutes in decimal format (like 36.8 minutes). I don't think you are supposed to. The point of the web page, as I understand it, is to give you a rough idea of what is going on. Seeing a list of figures 36.8, 37.1, 35.9, 36.4, 242.8, 37.3 ... makes the full backup (or problem, or whatever) stand out much more than if you obfuscate it into 4h2m48s. A list like 36.1, 36.1, 36.2, 36.3, 36.7, 36.8 ... gives you much more of an impression of how the times are developping than 36m6s, 36m12s, ... would. It's simply about human format: humans do not speak about 36.8 minutes. But as you also pointed out: if I'd cut the seconds, nothing would change in such an important way: 36.8 or 36 is the same thing because I'm not interested into seconds... The main goal I'm seeking is to get *real* values of time and transferred amount of data (which I adressed in my other thread). Regards, F. -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
If you want an alternative more lay person format, I'm sure you could rather easily develop another web page that suits your purposes and contribute it back to the project. In fact, if you start by thinking of your audience and purpose, the ideal summary page for the lay person is likely to have many other changes beyond just time and storage formats. Of course would I be doing this, once I have the technical skills to achieve my goals. At this point in time, where I'm not enough skilled for creating a customized web page for my view, I'm just asking how to make it... Cheers, F. -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Boniforti Flavio wrote: Exactly, perhaps the a better (but more work involved) solution would be to create a new page which shows a nice pretty graph of the various numbers instead of stacks of numbers in great big tables... Well, if somebody is willing to cooperate with me to do it, I'll be willing to learn and contribute. A *very nice* add-on would be having graphs for each host, not only for the whole pool. I'd really love to see ho my single hosts evolve in relation to GBs occupied. Also, I'd like to see how transfer rates may be varying... I guess you could track the transfer times and sizes for each host/share, but there is a philosophical/practical issue in tracking the storage space since it is pooled and there is no handy way to tell which, if any, other hosts have links to a common file. In terms of real space consumed, all of your targets can have multiple copies of some large file and it will barely take any more room than one single copy on one host. -- Les Mikesell lesmikes...@gmail.com -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Boniforti Flavio wrote: I am not used to consider minutes in decimal format (like 36.8 minutes). I don't think you are supposed to. The point of the web page, as I understand it, is to give you a rough idea of what is going on. Seeing a list of figures 36.8, 37.1, 35.9, 36.4, 242.8, 37.3 ... makes the full backup (or problem, or whatever) stand out much more than if you obfuscate it into 4h2m48s. A list like 36.1, 36.1, 36.2, 36.3, 36.7, 36.8 ... gives you much more of an impression of how the times are developping than 36m6s, 36m12s, ... would. It's simply about human format: humans do not speak about 36.8 minutes. _Except_ when they are trying to compare different amounts of time, when the discrete jumps in days/hours/minutes/seconds become inconvenient to normal math operations. -- Les Mikesell lesmikes...@gmail.com -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
I guess you could track the transfer times and sizes for each host/share, but there is a philosophical/practical issue in tracking the storage space since it is pooled and there is no handy way to tell which, if any, other hosts have links to a common file. In terms of real space consumed, all of your targets can have multiple copies of some large file and it will barely take any more room than one single copy on one host. Yes, I was indeed thinking about the same thing too: because of pooling it's somehow unprecise defining the effective space used. Therefore I will be assuming that the data showing in the Host Summary is the space consumed by each single host. But is there any way to know how much data has been transferred from the start of the backup process until its end? Thanks, F. -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Les Mikesell wrote: I guess you could track the transfer times and sizes for each host/share, but there is a philosophical/practical issue in tracking the storage space since it is pooled and there is no handy way to tell which, if any, other hosts have links to a common file. In terms of real space consumed, all of your targets can have multiple copies of some large file and it will barely take any more room than one single copy on one host. Consider that from a cost perspective, it is imprecise to charge internet usage (ie downloads) which includes data downloaded from the proxy server, since that data may (or may not) have been downloaded by multiple people, and therefore has minimal real cost for the 'copies'. However, we still *do* charge based on the traffic on the interface to the users connection... Regardless of what pooling, or optimisation might be done in the network, cloud, backuppc system, there is still an 'argument' that says the user is responsible for the total amount of space consumed by a single host regardless of whether any other host also shares the same space In any case, I think what our colleague is asking for is how much transit bandwidth did a host consume during the backup process, and this has nothing at all to do with pooling. The bad news is, (AFAIK) that this data is not collected within backuppc, and would need a different implementation for each transfer method. The best suggestion I could make would be to measure this at the network interface of your backuppc host. ie, the simplest method to track bandwidth consumption for rsyncd transfers is to add an iptables allow rule for traffic to your client host on port 873 (or whatever the correct ip + port is)... Though, it would be nice if these stats could be collected by backuppc, and stored in some clearly defined file, whereby some other tool could easily collect the data and present it in whatever format is desired... PS, for clarity, I would also like to see the bandwidth consumption, and a statistic for the amount of disk space a backup consumes without regard to any shared files with other hosts... Though I think this last one can be done with du -sm /var/lib/backuppc/pc/host/124... Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkoRbvcACgkQGyoxogrTyiWXZQCgqQq3sZ8oIC44QtGYc6XXP3Z/ SVEAn220PHng6Uvb9oj3NFg6BhUmsD0W =ewyG -END PGP SIGNATURE- -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
In any case, I think what our colleague is asking for is how much transit bandwidth did a host consume during the backup process, and this has nothing at all to do with pooling. The Indeed, that's what I want to know: the really transferred bytes for that host-to-host connection (from the remote host to my backuppc server). Nothing concerning the backuppc pool... bad news is, (AFAIK) that this data is not collected within backuppc, and would need a different implementation for each transfer method. The best suggestion I could make would be to measure this at the network interface of your backuppc host. ie, the simplest method to track bandwidth consumption for rsyncd transfers is to add an iptables allow rule for traffic to your client host on port 873 (or whatever the correct ip + port is)... Well, if I'd achieve something like this, I guess I would be collecting *every single bit* (also commands issued and stuff like that). For my purpose, it would be sufficient to know that I transferred a total amount of 14 files, which sum up to 1034896 bytes. I tried to look at the log files, but I don't understand how to distinguish between the words like pool, same, skip, create... If I could assume that all the create ones are really transferred bytes, I'd be summing them up with a simple bash script. But what about pool ones (I just imagine that skip and same are really NO TRANSFERS)? Though, it would be nice if these stats could be collected by backuppc, and stored in some clearly defined file, whereby some other tool could easily collect the data and present it in whatever format is desired... Regards, F. -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Boniforti Flavio wrote: bad news is, (AFAIK) that this data is not collected within backuppc, and would need a different implementation for each transfer method. The best suggestion I could make would be to measure this at the network interface of your backuppc host. ie, the simplest method to track bandwidth consumption for rsyncd transfers is to add an iptables allow rule for traffic to your client host on port 873 (or whatever the correct ip + port is)... Well, if I'd achieve something like this, I guess I would be collecting *every single bit* (also commands issued and stuff like that). For my purpose, it would be sufficient to know that I transferred a total amount of 14 files, which sum up to 1034896 bytes. I tried to look at the log files, but I don't understand how to distinguish between the words like pool, same, skip, create... If I could assume that all the create ones are really transferred bytes, I'd be summing them up with a simple bash script. But what about pool ones (I just imagine that skip and same are really NO TRANSFERS)? Also, when using rsyncd, although the file is different, the entire file may not have been transferred. ie, a 100MB file might only transfer 5MB which was modified from yesterday's backup In addition, when using rsync+ssh, if you are using compression, then again, the actual data over the network will be less than the size of the files transferred (probably also correct for tar+ssh with compression, etc). So all of these are estimations, and whether backuppc can internally even know what the correct settings are/should be, is possibly (probably as I think about it) impossible. A simply pre/post script which sets up the iptables entry, and then records the result and deletes the entry would probably be the quickest solution for a couple of simple bash scripts. Still, this doesn't address the meanings of the various bits of data which *are* kept by backuppc. Did someone manage to find the documentation on the meaning of the data recorded in the backups files? Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkoRgAoACgkQGyoxogrTyiXCKwCcD9eR1JNCzOvnQE9JVdxYNcyZ FXQAn14DU0ediRfcfAXd6Ka7xoTb+8ze =QOCY -END PGP SIGNATURE- -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Boniforti Flavio wrote: I guess you could track the transfer times and sizes for each host/share, but there is a philosophical/practical issue in tracking the storage space since it is pooled and there is no handy way to tell which, if any, other hosts have links to a common file. In terms of real space consumed, all of your targets can have multiple copies of some large file and it will barely take any more room than one single copy on one host. Yes, I was indeed thinking about the same thing too: because of pooling it's somehow unprecise defining the effective space used. Therefore I will be assuming that the data showing in the Host Summary is the space consumed by each single host. But is there any way to know how much data has been transferred from the start of the backup process until its end? I don't know about the accuracy of the numbers, but I always assumed that the duration (*60)times the MB/sec for the same backup run would give me the bytes transferred. But if you are looking for bottlenecks you might need to note that rsync incrementals take quite a bit of wall-clock time even though they may transfer a tiny amount of data. For non-rsync xfers, the transfer size should be the same as the file size, with everything except the 'new' files discarded after the transfer and replace will pool links. In all cases, this would be skewed if you add the '-C' (compression) option to the ssh command since that would happen before backuppc sees the data. -- Les Mikesell lesmikes...@gmail.com -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Boniforti Flavio wrote: Il 18.05.09 18:14, Les Mikesell lesmikes...@gmail.com ha scritto: In all cases, this would be skewed if you add the '-C' (compression) option to the ssh command since that would happen before backuppc sees the data. And that's my case :-/ Do you also think that the best approach is the iptables one? I think the best approach is to have enough bandwidth that you don't need to worry about it - or at least can get by with tracking it at the interface or switch port level with an snmp tool like cacti or opennms. If you need to count bytes between machines, ntop might work but it is a fairly big load on the server. It is a handy thing to have around when you suspect some machines have viruses or just want to see the bandwidth hogs - http://www.ntop.org. -- Les Mikesell lesmikes...@gmail.com -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Hi, Boniforti Flavio wrote on 2009-05-18 22:14:50 +0200 [Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format]: Il 18.05.09 17:34, Adam Goryachev mailingli...@websitemanagers.com.au ha scritto: [...] the question I believe should have been asked long ago is: What problem are you trying to solve? Is there even one, or are you just trying to make use/sense of numbers that are displayed somewhere? Depending on whether you want to entertain bored users, bill your clients for bandwidth, or conduct scientific measurements, the answers are likely to be vastly different. You are quite right... I didn't want to put that stuff in our thread, but it *has* to be mentioned that rsync + ssh (and YES I am using -C compression!) does meanfully change the amount of data transferred. Depending on what you want, that may or may not be relevant. You can just as well bill your clients for your disk wear (and CPU usage), regardless of the bandwidth savings ssh compression and rsync algorithm gain you. In fact, bandwidth savings will probably not mean that you can do more concurrent backups. A simply pre/post script which sets up the iptables entry, and then records the result and deletes the entry would probably be the quickest solution for a couple of simple bash scripts. Do you already have some sort of practical suggestion? iptables -I INPUT -s client_addr -d backuppc_server_addr -p tcp --sport 22 (and delete it with the same rule with -D instead of -I). Supposing you only have one concurrent backup to one host and no other ssh usage. You might prefer to count outgoing traffic (well, no, but maybe incoming + outgoing). Note that the rule has no target - it's only for accounting. Still, this doesn't address the meanings of the various bits of data which *are* kept by backuppc. Did someone manage to find the documentation on the meaning of the data recorded in the backups files? Use the force, read the source :) If you talk about the same things I am clueless about (skip, create, pool, same, and so on), then I'd be *too* interested in some precise definitions. I can give you some imprecise ones. same - rsync determined that file matches and does not need to be transferred pool - transferred file matched one already in the pool create - transferred file did not match any existing pool file To be honest, I think they're all rather self-explaining. I didn't do any research on the answers, it's just what figures from observation. In particular, on my import backup from local source quest, I was really worried when I got lots of pool lines on the first remote backup where it should have read same. That explained rather well why the backup was taking ages. But, again, this is a *completely* different topic from counting bytes with iptables. You should first find out what you want, then ask for help finding a solution. Discussing all possible problems might be something people enjoy that have an abundance of free time. These people should spend more time on the BackupPC wiki ;-). Regards, Holger -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
and 31.21 GB instead of 31214312331231 bytes!! Ability to email a list of files backed up per host wood me kool too. I know, I know.. On Fri, May 15, 2009 at 7:39 AM, Boniforti Flavio fla...@piramide.chwrote: Hello people, I hope developers will read this and take it into account if possible: I am not used to consider minutes in decimal format (like 36.8 minutes). Would it be possible to convert that data into time format (like 36m48sec) and extend the same thing to hours (not anymore 242.8minutes, but instead 4hours 2mins 48sec)? Thanks, Flavio Boniforti PIRAMIDE INFORMATICA SAGL Via Ballerini 21 6600 Locarno Switzerland Phone: +41 91 751 68 81 Fax: +41 91 751 69 14 URL: http://www.piramide.ch E-mail: fla...@piramide.ch -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Hi, Bharat Mistry wrote on 2009-05-15 15:57:57 +0100 [Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format]: and 31.21 GB instead of 31214312331231 bytes!! (. instead of !!, too? :) Ability to email a list of files backed up per host wood me kool too. thank you for making this point (though I don't suppose you *wanted* to make it). On Fri, May 15, 2009 at 7:39 AM, Boniforti Flavio fla...@piramide.chwrote: I am not used to consider minutes in decimal format (like 36.8 minutes). I don't think you are supposed to. The point of the web page, as I understand it, is to give you a rough idea of what is going on. Seeing a list of figures 36.8, 37.1, 35.9, 36.4, 242.8, 37.3 ... makes the full backup (or problem, or whatever) stand out much more than if you obfuscate it into 4h2m48s. A list like 36.1, 36.1, 36.2, 36.3, 36.7, 36.8 ... gives you much more of an impression of how the times are developping than 36m6s, 36m12s, ... would. If you want more than a rough idea (or rather, if you have better use of your time than staring at the statistics), you'll try to automatically process the numbers, and parsing 4h2m48s into something you can calculate with will be nothing less than a nuisance (and, yes, if you're backuppc on the BackupPC server, you can parse the backups files, but if you only have HTTP access to one client's host page, you can't). Similarly, you won't enjoy the reduced precision of 31.21 GB. If you are generating an email summary, for instance, you can always convert the numbers to whatever format you want, and it's still easier to convert 242.8 minutes than to translate 4hours 2mins 48secs to a different language or shift the whitespace around to match your taste. Would it be possible to convert that data into time format (like 36m48sec) and extend the same thing to hours (not anymore 242.8minutes, but instead 4hours 2mins 48sec)? Yes, but who's the target audience? Are you saying you *need* to know more than my full backups take somewhere between 4 and 5 hours? If it's less than 4h17m25s it's ok, but if it's more, you'll need to speed it up somehow? As for the seconds, I'd argue to rather drop them(*). They're almost certainly below the exactness of the measurement (well, yes, the backup *did* take 36 minutes and 13.7 seconds, but that the next backup took 36 minutes and 50 seconds probably tells you more about the state of the machines and the link at that time than about the backup itself in relation to the other one). Measure with a micrometer, mark with chalk, cut with an axe. Regards, Holger (*) Well, no, keep them. They don't hurt as long as they're just decimal minutes :). -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
Actually, comment from a couple of USERS... you know the people who all this stuff is for [?] One of my users asked for some help with setup - He wanted to be familiar with the system for day-to-day operation. His comment was :- Great but why does everything to do with Linux seem to be Technical - why can't we just have Hours and Minutes, Gb rather than 1000Mb I did explain this was OpenSource and done on a volunatry basis. Pointlessly nurdy he replied - no wonder people don't take it (Linux) seriously You have to admit, he that has a very valid point [?] On Fri, May 15, 2009 at 8:18 PM, Holger Parplies wb...@parplies.de wrote: Hi, Bharat Mistry wrote on 2009-05-15 15:57:57 +0100 [Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format]: and 31.21 GB instead of 31214312331231 bytes!! (. instead of !!, too? :) Ability to email a list of files backed up per host wood me kool too. thank you for making this point (though I don't suppose you *wanted* to make it). On Fri, May 15, 2009 at 7:39 AM, Boniforti Flavio fla...@piramide.ch wrote: I am not used to consider minutes in decimal format (like 36.8 minutes). I don't think you are supposed to. The point of the web page, as I understand it, is to give you a rough idea of what is going on. Seeing a list of figures 36.8, 37.1, 35.9, 36.4, 242.8, 37.3 ... makes the full backup (or problem, or whatever) stand out much more than if you obfuscate it into 4h2m48s. A list like 36.1, 36.1, 36.2, 36.3, 36.7, 36.8 ... gives you much more of an impression of how the times are developping than 36m6s, 36m12s, ... would. If you want more than a rough idea (or rather, if you have better use of your time than staring at the statistics), you'll try to automatically process the numbers, and parsing 4h2m48s into something you can calculate with will be nothing less than a nuisance (and, yes, if you're backuppc on the BackupPC server, you can parse the backups files, but if you only have HTTP access to one client's host page, you can't). Similarly, you won't enjoy the reduced precision of 31.21 GB. If you are generating an email summary, for instance, you can always convert the numbers to whatever format you want, and it's still easier to convert 242.8 minutes than to translate 4hours 2mins 48secs to a different language or shift the whitespace around to match your taste. Would it be possible to convert that data into time format (like 36m48sec) and extend the same thing to hours (not anymore 242.8minutes, but instead 4hours 2mins 48sec)? Yes, but who's the target audience? Are you saying you *need* to know more than my full backups take somewhere between 4 and 5 hours? If it's less than 4h17m25s it's ok, but if it's more, you'll need to speed it up somehow? As for the seconds, I'd argue to rather drop them(*). They're almost certainly below the exactness of the measurement (well, yes, the backup *did* take 36 minutes and 13.7 seconds, but that the next backup took 36 minutes and 50 seconds probably tells you more about the state of the machines and the link at that time than about the backup itself in relation to the other one). Measure with a micrometer, mark with chalk, cut with an axe. Regards, Holger (*) Well, no, keep them. They don't hurt as long as they're just decimal minutes :). 343.gif360.gif-- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
I agree totally with Holger. It is much easier to convert raw data to less precise and/or more humanly readable formats than vica-versa. Plus, the data is usually something you only look at when troubleshooting or analyzing performance, not something that is of interest to the casual user. If you want an alternative more lay person format, I'm sure you could rather easily develop another web page that suits your purposes and contribute it back to the project. In fact, if you start by thinking of your audience and purpose, the ideal summary page for the lay person is likely to have many other changes beyond just time and storage formats. Holger Parplies wrote at about 21:18:54 +0200 on Friday, May 15, 2009: Hi, Bharat Mistry wrote on 2009-05-15 15:57:57 +0100 [Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format]: and 31.21 GB instead of 31214312331231 bytes!! (. instead of !!, too? :) Ability to email a list of files backed up per host wood me kool too. thank you for making this point (though I don't suppose you *wanted* to make it). On Fri, May 15, 2009 at 7:39 AM, Boniforti Flavio fla...@piramide.chwrote: I am not used to consider minutes in decimal format (like 36.8 minutes). I don't think you are supposed to. The point of the web page, as I understand it, is to give you a rough idea of what is going on. Seeing a list of figures 36.8, 37.1, 35.9, 36.4, 242.8, 37.3 ... makes the full backup (or problem, or whatever) stand out much more than if you obfuscate it into 4h2m48s. A list like 36.1, 36.1, 36.2, 36.3, 36.7, 36.8 ... gives you much more of an impression of how the times are developping than 36m6s, 36m12s, ... would. If you want more than a rough idea (or rather, if you have better use of your time than staring at the statistics), you'll try to automatically process the numbers, and parsing 4h2m48s into something you can calculate with will be nothing less than a nuisance (and, yes, if you're backuppc on the BackupPC server, you can parse the backups files, but if you only have HTTP access to one client's host page, you can't). Similarly, you won't enjoy the reduced precision of 31.21 GB. If you are generating an email summary, for instance, you can always convert the numbers to whatever format you want, and it's still easier to convert 242.8 minutes than to translate 4hours 2mins 48secs to a different language or shift the whitespace around to match your taste. Would it be possible to convert that data into time format (like 36m48sec) and extend the same thing to hours (not anymore 242.8minutes, but instead 4hours 2mins 48sec)? Yes, but who's the target audience? Are you saying you *need* to know more than my full backups take somewhere between 4 and 5 hours? If it's less than 4h17m25s it's ok, but if it's more, you'll need to speed it up somehow? As for the seconds, I'd argue to rather drop them(*). They're almost certainly below the exactness of the measurement (well, yes, the backup *did* take 36 minutes and 13.7 seconds, but that the next backup took 36 minutes and 50 seconds probably tells you more about the state of the machines and the link at that time than about the backup itself in relation to the other one). Measure with a micrometer, mark with chalk, cut with an axe. Regards, Holger (*) Well, no, keep them. They don't hurt as long as they're just decimal minutes :). -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] [SUGGESTION] Duration/mins not in decimal format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeffrey J. Kosowsky wrote: I agree totally with Holger. It is much easier to convert raw data to less precise and/or more humanly readable formats than vica-versa. Plus, the data is usually something you only look at when troubleshooting or analyzing performance, not something that is of interest to the casual user. If you want an alternative more lay person format, I'm sure you could rather easily develop another web page that suits your purposes and contribute it back to the project. In fact, if you start by thinking of your audience and purpose, the ideal summary page for the lay person is likely to have many other changes beyond just time and storage formats. Exactly, perhaps the a better (but more work involved) solution would be to create a new page which shows a nice pretty graph of the various numbers instead of stacks of numbers in great big tables... In fact, I've got one installation og backuppc which now has over 800 backups of each host, and it can take quite a while to view the host pages. As time goes on, and the number of backups increases, this will just get worse, again, it would be nice to simply have a few graphs (or a limit to the number of backups displayed (maybe last 20 backups etc). However, while we can all make suggestions of improvements, if we cared enough we would provide a patch, and if other people cared enough, it would be included into backuppc. I don't have the time, and I don't care enough about the above, and so I haven't *done* anything about it. So thanks for a great product, it is good enough for me to not need to put loads of work into it, and I can just use it to do useful things. Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkoOA5gACgkQGyoxogrTyiV2jgCeOCgRWvzTWWBgp730qBc7oY1K TKIAoJpLuHbMZbOe9Ymw6mFUh/n+JR2c =nOiV -END PGP SIGNATURE- -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
