[BackupPC-users] rsyncd full backup

[Sharuzzaman Ahmat Raslan]

Hi,

I have implemented BackupPC for my customer.

Initially, the backup transport is SMB, but recently, I noticed a lot of
machine backup (full and incremental) is not able to complete in 8 hours,
due to large number of file, and big file size.

Last week, I installed DeltaCopy (rsycnd server for Windows) on one
machine, and change the backup transport to rysncd. The backup runs well.

But today, I noticed, when BackupPC is running a full backup on the machine
that have rsyncd, it still takes 8 hours to do full backup.

Which is I considered weird, because rsync suppose to compare that full
backup, with the previous full backup (or previous full + incremental), so
that only modified file is transferred.

That is my expectation when I plan to use rsyncd.

Any explanation why BackupPC is not running in this way? Any configuration
that I can changed to make it work like what I expect?

Thanks.

-- 
Sharuzzaman Ahmat Raslan
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Les Mikesell]

On Mon, Oct 28, 2013 at 10:08 PM, Sharuzzaman Ahmat Raslan
 wrote:
> Hi,
>
> I have implemented BackupPC for my customer.
>
> Initially, the backup transport is SMB, but recently, I noticed a lot of
> machine backup (full and incremental) is not able to complete in 8 hours,
> due to large number of file, and big file size.
>
> Last week, I installed DeltaCopy (rsycnd server for Windows) on one machine,
> and change the backup transport to rysncd. The backup runs well.
>
> But today, I noticed, when BackupPC is running a full backup on the machine
> that have rsyncd, it still takes 8 hours to do full backup.
>
> Which is I considered weird, because rsync suppose to compare that full
> backup, with the previous full backup (or previous full + incremental), so
> that only modified file is transferred.
>
> That is my expectation when I plan to use rsyncd.
>
> Any explanation why BackupPC is not running in this way? Any configuration
> that I can changed to make it work like what I expect?
>

Rsync will only transfer the changed data, but in full runs the
contents of the files are read at both ends and compared with block
checksums, so it takes some time.  Incrementals runs will quickly skip
files where the file timestamps and lengths are identical.   See the
section on 'Rsync checksum caching' in
http://backuppc.sourceforge.net/faq/BackupPC.html regarding a way to
avoid having to read/uncompress on the server side after 2 fulls have
completed, but the data is always read on the target side.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Sharuzzaman Ahmat Raslan]

Hi Les,

In essence, if I enable

--checksum-seed=32761

then the rsync full backup will be faster?

Thanks.



On Tue, Oct 29, 2013 at 11:33 AM, Les Mikesell wrote:

> On Mon, Oct 28, 2013 at 10:08 PM, Sharuzzaman Ahmat Raslan
>  wrote:
> > Hi,
> >
> > I have implemented BackupPC for my customer.
> >
> > Initially, the backup transport is SMB, but recently, I noticed a lot of
> > machine backup (full and incremental) is not able to complete in 8 hours,
> > due to large number of file, and big file size.
> >
> > Last week, I installed DeltaCopy (rsycnd server for Windows) on one
> machine,
> > and change the backup transport to rysncd. The backup runs well.
> >
> > But today, I noticed, when BackupPC is running a full backup on the
> machine
> > that have rsyncd, it still takes 8 hours to do full backup.
> >
> > Which is I considered weird, because rsync suppose to compare that full
> > backup, with the previous full backup (or previous full + incremental),
> so
> > that only modified file is transferred.
> >
> > That is my expectation when I plan to use rsyncd.
> >
> > Any explanation why BackupPC is not running in this way? Any
> configuration
> > that I can changed to make it work like what I expect?
> >
>
> Rsync will only transfer the changed data, but in full runs the
> contents of the files are read at both ends and compared with block
> checksums, so it takes some time.  Incrementals runs will quickly skip
> files where the file timestamps and lengths are identical.   See the
> section on 'Rsync checksum caching' in
> http://backuppc.sourceforge.net/faq/BackupPC.html regarding a way to
> avoid having to read/uncompress on the server side after 2 fulls have
> completed, but the data is always read on the target side.
>
> --
>Les Mikesell
>  lesmikes...@gmail.com
>
>
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>



-- 
Sharuzzaman Ahmat Raslan
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Adam Goryachev]

On 29/10/13 15:14, Sharuzzaman Ahmat Raslan wrote:

Hi Les,


Thanks.|
|


On Tue, Oct 29, 2013 at 11:33 AM, Les Mikesell > wrote:


On Mon, Oct 28, 2013 at 10:08 PM, Sharuzzaman Ahmat Raslan
mailto:sharuzza...@gmail.com>> wrote:
> Hi,
>
> I have implemented BackupPC for my customer.
>
> Initially, the backup transport is SMB, but recently, I noticed
a lot of
> machine backup (full and incremental) is not able to complete in
8 hours,
> due to large number of file, and big file size.
>
> Last week, I installed DeltaCopy (rsycnd server for Windows) on
one machine,
> and change the backup transport to rysncd. The backup runs well.
>
> But today, I noticed, when BackupPC is running a full backup on
the machine
> that have rsyncd, it still takes 8 hours to do full backup.
>
> Which is I considered weird, because rsync suppose to compare
that full
> backup, with the previous full backup (or previous full +
incremental), so
> that only modified file is transferred.
>
> That is my expectation when I plan to use rsyncd.
>
> Any explanation why BackupPC is not running in this way? Any
configuration
> that I can changed to make it work like what I expect?
>

Rsync will only transfer the changed data, but in full runs the
contents of the files are read at both ends and compared with block
checksums, so it takes some time.  Incrementals runs will quickly skip
files where the file timestamps and lengths are identical. See the
section on 'Rsync checksum caching' in
http://backuppc.sourceforge.net/faq/BackupPC.html regarding a way to
avoid having to read/uncompress on the server side after 2 fulls have
completed, but the data is always read on the target side.

In essence, if I enable
|--checksum-seed=32761

|
then the rsync full backup will be faster?


Yes, the third full backup after you enable that option will be faster 
*IF* the slow speed is due to the backup server needing to decompress 
the file and check the content.


In the case that your backup client has really slow disk, then there is 
nothing you can do, except maybe modify backuppc for full backups to not 
send the ignore-times option to rsync (ie, every backup is an 
incremental). Or, of course, upgrade the client to improve performance.


Regards,
Adam



--
Adam Goryachev Website Managers www.websitemanagers.com.au
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Holger Parplies]

Hi,

Adam Goryachev wrote on 2013-10-29 15:29:42 +1100 [Re: [BackupPC-users] rsyncd 
full backup]:
> On 29/10/13 15:14, Sharuzzaman Ahmat Raslan wrote:
> > [...]
> >On Tue, Oct 29, 2013 at 11:33 AM, Les Mikesell  ><mailto:lesmikes...@gmail.com>> wrote:
> >On Mon, Oct 28, 2013 at 10:08 PM, Sharuzzaman Ahmat Raslan
> >mailto:sharuzza...@gmail.com>> wrote:
> >> [...]
> >> Initially, the backup transport is SMB, but recently, I noticed
> >> a lot of machine backup (full and incremental) is not able to
> >> complete in 8 hours, due to large number of file, and big file size.
> >>
> >> Last week, I installed DeltaCopy (rsycnd server for Windows) on
> >> one machine, and change the backup transport to rysncd. The backup
> >> runs well.
> >>
> >> But today, I noticed, when BackupPC is running a full backup on
> >> the machine that have rsyncd, it still takes 8 hours to do full
> >> backup. [...]
> >Rsync will only transfer the changed data, but in full runs the
> >contents of the files are read at both ends and compared with block
> >checksums, so it takes some time. [...]
> >
> >In essence, if I enable
> >|--checksum-seed=32761
> >
> >|
> >then the rsync full backup will be faster?
> 
> Yes, the third full backup after you enable that option will be faster 
> *IF* the slow speed is due to the backup server needing to decompress 
> the file and check the content.

let me stress that again: don't expect a speedup on the *first* full backup
after you enable that option. In my limited opinion (I haven't compared speeds
because I don't have any issues with slow backups), the *second* full backup
should be faster, as you have pre-existing full backups, i.e. the next full
can add the checksums. In any case, the *third* full backup should hopefully
be faster :-).

> In the case that your backup client has really slow disk, then there is 
> nothing you can do, except maybe modify backuppc for full backups to not 
> send the ignore-times option to rsync (ie, every backup is an 
> incremental). Or, of course, upgrade the client to improve performance.

Actually, it is worth noting that aside from a possible speed improvement the
switch from smb to rsync(d) gives you far more precise *incremental* backups,
so it might be an option to increase FullPeriod. This may transfer more data
(because the delta is always relative to the reference backup - normally the
previous full backup - and not to the previous incremental backup), but you
can always explore the IncrLevels setting. So, while you might not speed up
the full runs, you might get away with doing them less often. I would not
recommend patching the ignore-times option away altogether.

But Adams point is correct: you need to find out where the problem is, before
you can fix it. While you might be able to find the problem by trying out
fixes, that might not be the most efficient way :-).

Regards,
Holger

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Sharuzzaman Ahmat Raslan]

Hi Holger,

Based on short session of troubleshooting, I believe the machine actually
suffer from low I/O speed to the disk. Average read is about 3 MB/s, which
I considered slow for a SATA disk in IDE emulation.

I'm planning to suggest to the customer to have a RAID 1 setup to increase
the I/O speed. I'm looking at possibilities to speed things up by not
having to change the overall setup.

Thank you for providing new insights to me regarding rsync. Glad to learn
new things :)

Thanks.


On Thu, Oct 31, 2013 at 5:15 AM, Holger Parplies  wrote:

> Hi,
>
> Adam Goryachev wrote on 2013-10-29 15:29:42 +1100 [Re: [BackupPC-users]
> rsyncd full backup]:
> > On 29/10/13 15:14, Sharuzzaman Ahmat Raslan wrote:
> > > [...]
> > >On Tue, Oct 29, 2013 at 11:33 AM, Les Mikesell  > ><mailto:lesmikes...@gmail.com>> wrote:
> > >On Mon, Oct 28, 2013 at 10:08 PM, Sharuzzaman Ahmat Raslan
> > >mailto:sharuzza...@gmail.com>> wrote:
> > >> [...]
> > >> Initially, the backup transport is SMB, but recently, I noticed
> > >> a lot of machine backup (full and incremental) is not able to
> > >> complete in 8 hours, due to large number of file, and big file
> size.
> > >>
> > >> Last week, I installed DeltaCopy (rsycnd server for Windows) on
> > >> one machine, and change the backup transport to rysncd. The backup
> > >> runs well.
> > >>
> > >> But today, I noticed, when BackupPC is running a full backup on
> > >> the machine that have rsyncd, it still takes 8 hours to do full
> > >> backup. [...]
> > >Rsync will only transfer the changed data, but in full runs the
> > >contents of the files are read at both ends and compared with block
> > >checksums, so it takes some time. [...]
> > >
> > >In essence, if I enable
> > >|--checksum-seed=32761
> > >
> > >|
> > >then the rsync full backup will be faster?
> >
> > Yes, the third full backup after you enable that option will be faster
> > *IF* the slow speed is due to the backup server needing to decompress
> > the file and check the content.
>
> let me stress that again: don't expect a speedup on the *first* full backup
> after you enable that option. In my limited opinion (I haven't compared
> speeds
> because I don't have any issues with slow backups), the *second* full
> backup
> should be faster, as you have pre-existing full backups, i.e. the next full
> can add the checksums. In any case, the *third* full backup should
> hopefully
> be faster :-).
>
> > In the case that your backup client has really slow disk, then there is
> > nothing you can do, except maybe modify backuppc for full backups to not
> > send the ignore-times option to rsync (ie, every backup is an
> > incremental). Or, of course, upgrade the client to improve performance.
>
> Actually, it is worth noting that aside from a possible speed improvement
> the
> switch from smb to rsync(d) gives you far more precise *incremental*
> backups,
> so it might be an option to increase FullPeriod. This may transfer more
> data
> (because the delta is always relative to the reference backup - normally
> the
> previous full backup - and not to the previous incremental backup), but you
> can always explore the IncrLevels setting. So, while you might not speed up
> the full runs, you might get away with doing them less often. I would not
> recommend patching the ignore-times option away altogether.
>
> But Adams point is correct: you need to find out where the problem is,
> before
> you can fix it. While you might be able to find the problem by trying out
> fixes, that might not be the most efficient way :-).
>
> Regards,
> Holger
>
>
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>



-- 
Sharuzzaman Ahmat Raslan

Re: [BackupPC-users] rsyncd full backup

[Adam Goryachev]

On 31/10/13 13:06, Sharuzzaman Ahmat Raslan wrote:
> Hi Holger,
>
> Based on short session of troubleshooting, I believe the machine
> actually suffer from low I/O speed to the disk. Average read is about
> 3 MB/s, which I considered slow for a SATA disk in IDE emulation.

Is that under load, or while idle? If it is under load, then it might be 
expected, remember throughput is very bad for HD when you have random 
load due to seek times.

If it is idle and has that performance level, then there is something 
wrong. Even old IDE disks could do at least 30 to 50MB/s for large 
contiguous reads.

> I'm planning to suggest to the customer to have a RAID 1 setup to
> increase the I/O speed. I'm looking at possibilities to speed things
> up by not having to change the overall setup.
While RAID1 will assist in reliability and is one strategy to reduce 
downtime/data loss (but it isn't a backup), it also is not going to 
improve performance. With RAID1 you still need to write to both disks, 
and while it is theoretically possible to balance reads across both 
disks, it likely won't do that well without a proper hardware raid 
controller.

Personally, my suggestion would be to consider using a SSD, since you 
are using such an old drive, probably you don't need a lot of space, so 
a 120GB SSD might be suitable. An SSD will handle random IO 
significantly better than any one or two drive system, with much higher 
transfer rates as well (there is no penalty for seek times with SSD).

Again, personally, I've used a couple of systems with 5 x 480GB Intel 
520s SSD in RAID5, and they have been working really well (except they 
were difficult to actually get stock of them most of this year, and I 
hear they are now replaced by a new model).

Regards,
Adam

-- 
Adam Goryachev
Website Managers
P: +61 2 8304 a...@websitemanagers.com.au
F: +61 2 8304 0001 www.websitemanagers.com.au


-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Sharuzzaman Ahmat Raslan]

Hi Adam,

The low I/O is when the machine is under load.

Thank you for suggesting to use SSD. I have been thinking about that as
well, but currently, the storage of BackupPC is using a 1TB disk, with
about 80% utilization.

Changing to 1TB SSD might be a little bit overkill on the customer's budget
:)

Maybe I should look at bcache for Linux :)

https://lwn.net/Articles/497024/
http://bcache.evilpiepirate.org/





On Thu, Oct 31, 2013 at 10:34 AM, Adam Goryachev <
mailingli...@websitemanagers.com.au> wrote:

> On 31/10/13 13:06, Sharuzzaman Ahmat Raslan wrote:
> > Hi Holger,
> >
> > Based on short session of troubleshooting, I believe the machine
> > actually suffer from low I/O speed to the disk. Average read is about
> > 3 MB/s, which I considered slow for a SATA disk in IDE emulation.
>
> Is that under load, or while idle? If it is under load, then it might be
> expected, remember throughput is very bad for HD when you have random
> load due to seek times.
>
> If it is idle and has that performance level, then there is something
> wrong. Even old IDE disks could do at least 30 to 50MB/s for large
> contiguous reads.
>
> > I'm planning to suggest to the customer to have a RAID 1 setup to
> > increase the I/O speed. I'm looking at possibilities to speed things
> > up by not having to change the overall setup.
> While RAID1 will assist in reliability and is one strategy to reduce
> downtime/data loss (but it isn't a backup), it also is not going to
> improve performance. With RAID1 you still need to write to both disks,
> and while it is theoretically possible to balance reads across both
> disks, it likely won't do that well without a proper hardware raid
> controller.
>
> Personally, my suggestion would be to consider using a SSD, since you
> are using such an old drive, probably you don't need a lot of space, so
> a 120GB SSD might be suitable. An SSD will handle random IO
> significantly better than any one or two drive system, with much higher
> transfer rates as well (there is no penalty for seek times with SSD).
>
> Again, personally, I've used a couple of systems with 5 x 480GB Intel
> 520s SSD in RAID5, and they have been working really well (except they
> were difficult to actually get stock of them most of this year, and I
> hear they are now replaced by a new model).
>
> Regards,
> Adam
>
> --
> Adam Goryachev
> Website Managers
> P: +61 2 8304 a...@websitemanagers.com.au
> F: +61 2 8304 0001 www.websitemanagers.com.au
>
>
> --
> Adam Goryachev Website Managers www.websitemanagers.com.au
>
>
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>



-- 
Sharuzzaman Ahmat Raslan
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Adam Goryachev]

On 31/10/13 13:56, Sharuzzaman Ahmat Raslan wrote:
> Hi Adam,
>
> The low I/O is when the machine is under load.
>
> Thank you for suggesting to use SSD. I have been thinking about that
> as well, but currently, the storage of BackupPC is using a 1TB disk,
> with about 80% utilization.
>
> Changing to 1TB SSD might be a little bit overkill on the customer's
> budget :)
>
Sure, 2 x 480GB SSD in linear RAID is still relatively expensive :) 
though it certainly is a huge performance improvement. BTW, FYI, I get 
2.5GB/s read and 1.5GB/s write performance from my RAID5...

> Maybe I should look at bcache for Linux :)
>
> https://lwn.net/Articles/497024/
> http://bcache.evilpiepirate.org/
>
I've seen that also, but I'm not sure it is a good (stable) solution for 
real use (at least, I'm not prepared to use that for a server yet, your 
tolerance might be different). In addition, it probably won't help the 
backup work load, since you need to read the entire disk, and the entire 
disk won't fit into the cache

Regards,
Adam

-- 
Adam Goryachev
Website Managers
P: +61 2 8304 a...@websitemanagers.com.au
F: +61 2 8304 0001 www.websitemanagers.com.au


-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Micha Kersloot]

Hi,

You could also go from raid1 to raid10.

Met vriendelijke groet,

Micha Kersloot

Blijf op de hoogte en ontvang de laatste tips over Zimbra/KovoKs Contact:
http://twitter.com/kovoks

KovoKs B.V. is ingeschreven onder KvK nummer: 1104

- Oorspronkelijk bericht -
> Van: "Adam Goryachev" 
> Aan: "General list for user discussion, questions and support" 
> 
> Verzonden: Donderdag 31 oktober 2013 04:13:42
> Onderwerp: Re: [BackupPC-users] rsyncd full backup
> 
> On 31/10/13 13:56, Sharuzzaman Ahmat Raslan wrote:
> > Hi Adam,
> >
> > The low I/O is when the machine is under load.
> >
> > Thank you for suggesting to use SSD. I have been thinking about that
> > as well, but currently, the storage of BackupPC is using a 1TB disk,
> > with about 80% utilization.
> >
> > Changing to 1TB SSD might be a little bit overkill on the customer's
> > budget :)
> >
> Sure, 2 x 480GB SSD in linear RAID is still relatively expensive :)
> though it certainly is a huge performance improvement. BTW, FYI, I get
> 2.5GB/s read and 1.5GB/s write performance from my RAID5...
> 
> > Maybe I should look at bcache for Linux :)
> >
> > https://lwn.net/Articles/497024/
> > http://bcache.evilpiepirate.org/
> >
> I've seen that also, but I'm not sure it is a good (stable) solution for
> real use (at least, I'm not prepared to use that for a server yet, your
> tolerance might be different). In addition, it probably won't help the
> backup work load, since you need to read the entire disk, and the entire
> disk won't fit into the cache
> 
> Regards,
> Adam
> 
> --
> Adam Goryachev
> Website Managers
> P: +61 2 8304 a...@websitemanagers.com.au
> F: +61 2 8304 0001 www.websitemanagers.com.au
> 
> 
> --
> Adam Goryachev Website Managers www.websitemanagers.com.au
> 
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
> 

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Timothy J Massey]

Sharuzzaman Ahmat Raslan  wrote on 10/30/2013 
10:06:18 PM:

> Hi Holger,

> Based on short session of troubleshooting, I believe the machine 
> actually suffer from low I/O speed to the disk. Average read is 
> about 3 MB/s, which I considered slow for a SATA disk in IDE emulation.

*REAL* slow:  I consider anything under 20MB/s slow.

But where did that number come from?  The pattern of reads will make a 
*huge* difference...

> I'm planning to suggest to the customer to have a RAID 1 setup to 
> increase the I/O speed. I'm looking at possibilities to speed things
> up by not having to change the overall setup.

I think you might want to have a better idea of what is going on first 
before you just start throwing hardware at it.  If your numbers were 
correct but still too slow I'd say sure.  But your numbers are *broken* 
wrong.  You *might* fix your problem (by accident!) by throwing away some 
pieces and adding others, but you might not, too.  Then you've got a 
client that just spent a bunch of money for nothing...

Tim Massey


 
Out of the Box Solutions, Inc. 
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com 
 
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796 
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Sharuzzaman Ahmat Raslan]

Hi Timothy,

I got the number by observing the output of iotop while file transfer is
running. Also, on BackupPC host summary page, average transfer rate for
full backup is also around 3MB/s

It could be a network bottleneck also, as the customer is using 100Mbps
switch with around 80 PC, not including network printer and servers.
Inclusive should be around 100 network devices.

Any idea how to properly troubleshoot network bottleneck? My skill is a
little bit lacking on that area.

Thanks.



On Fri, Nov 1, 2013 at 2:12 AM, Timothy J Massey wrote:

> Sharuzzaman Ahmat Raslan  wrote on 10/30/2013
> 10:06:18 PM:
>
> > Hi Holger,
>
> > Based on short session of troubleshooting, I believe the machine
> > actually suffer from low I/O speed to the disk. Average read is
> > about 3 MB/s, which I considered slow for a SATA disk in IDE emulation.
>
> *REAL* slow:  I consider anything under 20MB/s slow.
>
> But where did that number come from?  The pattern of reads will make a
> *huge* difference...
>
> > I'm planning to suggest to the customer to have a RAID 1 setup to
> > increase the I/O speed. I'm looking at possibilities to speed things
> > up by not having to change the overall setup.
>
> I think you might want to have a better idea of what is going on first
> before you just start throwing hardware at it.  If your numbers were
> correct but still too slow I'd say sure.  But your numbers are *broken*
> wrong.  You *might* fix your problem (by accident!) by throwing away some
> pieces and adding others, but you might not, too.  Then you've got a client
> that just spent a bunch of money for nothing...
>
> Tim Massey
>
>*Out of the Box Solutions, Inc.* *
> Creative IT Solutions Made Simple!**
> **http://www.OutOfTheBoxSolutions.com*
> *
> **tmas...@obscorp.com*22108 Harper Ave.
> St. Clair Shores, MI 48080
> Office: (800)750-4OBS (4627)
> Cell: (586)945-8796
>
>
> --
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
>


-- 
Sharuzzaman Ahmat Raslan
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Les Mikesell]

On Wed, Oct 30, 2013 at 9:06 PM, Sharuzzaman Ahmat Raslan
 wrote:
> Hi Holger,
>
> Based on short session of troubleshooting, I believe the machine actually
> suffer from low I/O speed to the disk. Average read is about 3 MB/s, which I
> considered slow for a SATA disk in IDE emulation.

Where are you getting that number?   hdparm -tT  device_partition_name
should show about 30MB/s for the lower number even for old IDEs and at
least 2 or 3x that for SATA with a SATA controller - even old ones.

> I'm planning to suggest to the customer to have a RAID 1 setup to increase
> the I/O speed. I'm looking at possibilities to speed things up by not having
> to change the overall setup.

RAID1 is a good idea to protect against a single drive failure, but it
won't make a lot of difference in speed.  Writes go to both, reads can
overlap if the software is smart.   But, if you are currently running
RAID5, using bigger disks in RAID1 would help.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] rsyncd full backup

[Timothy J Massey]

Sharuzzaman Ahmat Raslan  wrote on 10/31/2013 
02:38:01 PM:

> Hi Timothy,
>
> I got the number by observing the output of iotop while file 
> transfer is running. Also, on BackupPC host summary page, average 
> transfer rate for full backup is also around 3MB/s

> It could be a network bottleneck also, as the customer is using 
> 100Mbps switch with around 80 PC, not including network printer and 
> servers. Inclusive should be around 100 network devices.

For file transfers, 100Mb/s is good for 7MB/s transfer rate.  Assuming a 
good quality switch (which is a *big* assumption), the number of computers 
shouldn't matter.

But I would think strongly about buying a good quality Gigabit switch (I 
recommend the HP V1910-24G) as your "backbone":  Plug all of your servers 
(including the BackupPC server) into it, as well as each of your 100Mb/s 
switches (even better if they have Gb uplink ports!).  That would 
eliminate the network as a bottleneck and only costs $300.  And improve 
network performance across the board, though your users may not notice it 
if they only work with small files.

> Any idea how to properly troubleshoot network bottleneck? My skill 
> is a little bit lacking on that area.

Sure:  Time the copying of files from one machine to another.  Assuming 
the source and destination hard drives are faster than 7MB/s (and they 
very well *better* be!), then you'll saturate a 100Mb network no problem.

For a more scientific approach, check out iperf.

I'd be *much* more worried about checking out your *disk* performance. You 
can do tests in exactly the same way:  copy files to and from the disk and 
see what happens.  Here are some very simple examples:

sync; time dd if=/dev/zero of=test.fil bs=1M count=1024; sync; sync; sync;
sync; time dd if=test.fil of=/dev/null bs=1M

The first line times the writing of a 1GB file named test.fil.  The second 
one times the reading of the same 1GB file.  Divide 1024 by the number of 
seconds it takes and that will give you the MB/s that you transferred. 
(The sync command is needed for accurate timing;  the three sync commands 
is kind of an old UNIX graybeard joke:  
http://unix.stackexchange.com/questions/5260/is-there-truth-to-the-philosophy-that-you-should-sync-sync-sync-sync
 
)

If you want more scientific disk performance information, check out iozone 
or iometer.

Remember:  always profile before you optimize.  ( 
http://www.phatcode.net/res/224/files/html/ch37/37-02.html )

Tim Massey


 
Out of the Box Solutions, Inc. 
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com 
 
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796 
--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/