[BackupPC-users] out of memory, and slow backups after moving to 3.0

[Nicholas Mistry]

I am seeing a very strange problem after transitioning to backuppc  
3.0.   Backups are taking considerably longer, and i am frequently  
getting out of memory kernel panics messages dumped to the syslog.


My previous config was a P3-450 w/ 256MB ram, 10GB Internal drive,  
and a 700GB external RAID-5 array.   This machine ran debian 3.0 and  
Backuppc 2.x ( i frequently updated it).


The new machine is a P3-800 w/ 256MB ram, 10GB Internal drive, and a  
2.0TB external RAID-6 array.This machine is a fresh install of  
debian 4.x and runs Backuppc 3.0


Both machines had aproximately 800MB of swap, and were configured to  
do backups via RSYNC/RSYNCD or SMB.


I know that the first machine was slow, it took a whole day to do a  
100GB backup over a 100Mbit link.   But the second machine is taking  
many days to complete the same job.  I started a full backup on the  
3rd of october and it is still running today.


I have already run extensive diagnostics on all the hardware, so i  
know that the machine is fine.


Before i start racking my brain, i wanted to find out if anyone else  
had experienced this type of problem after moving to backuppc 3?  
Has anyone seen memory leaks with backuppc running on debian 4?



Thanks

-N

--
Nicholas Mistry
[EMAIL PROTECTED]



-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Re: [BackupPC-users] out of memory, and slow backups after moving to 3.0

[Nicholas Mistry]

Nils,

When i built my first backuppc box, i only had 256 MB of ram on  
hand.   I wanted 512+ but sometimes you have to do with what you  
got.It was always my intention to buy more memory, but as time  
passed I was quite impressed with the performance on such a low power  
box.  All my backups ran fine and always completed between the hours  
of 6pm and 6am.   Even as the number of clients grew, and the size of  
the backups grew, this machine was keeping up and doing well. 
Finally i came to the realization that i really did not need to buy  
the extra memory.


Dont get me wrong, i did see a some amount of swap activity on the  
older box with 256MB, but it is NOTHING compared to what i am seeing  
now.


Both boxes are custom configured to run a stripped down version of  
debian.   Trimming out all the fat possible.   Only services i run on  
the box are SSH, Munin, and Backupc.


Another thing to note, on the 2.x config i was able to run 2+  
simultaneous backup commands.   On the new box i can only run a  
single backuppc job, otherwise it crashes backuppc w/ an out of  
memory kernel panic.


What i would really like to know from people running 3.x, what is  
your base ram, and what is your swap configured at. ?  Are there any  
users running 3.x on a machine w/ less than 512?


-N

Nicholas Mistry
[EMAIL PROTECTED]



On Oct 8, 2007, at 1:14 PM, Nils Breunese (Lemonbit) wrote:


Nicholas Mistry wrote:

I am seeing a very strange problem after transitioning to backuppc  
3.0.   Backups are taking considerably longer, and i am frequently  
getting out of memory kernel panics messages dumped to the syslog.


My previous config was a P3-450 w/ 256MB ram, 10GB Internal drive,  
and a 700GB external RAID-5 array.   This machine ran debian 3.0  
and Backuppc 2.x ( i frequently updated it).


The new machine is a P3-800 w/ 256MB ram, 10GB Internal drive, and  
a 2.0TB external RAID-6 array.This machine is a fresh install  
of debian 4.x and runs Backuppc 3.0


Both machines had aproximately 800MB of swap, and were configured  
to do backups via RSYNC/RSYNCD or SMB.


I know that the first machine was slow, it took a whole day to do  
a 100GB backup over a 100Mbit link.   But the second machine is  
taking many days to complete the same job.  I started a full  
backup on the 3rd of october and it is still running today.


I have already run extensive diagnostics on all the hardware, so i  
know that the machine is fine.


Before i start racking my brain, i wanted to find out if anyone  
else had experienced this type of problem after moving to backuppc  
3? Has anyone seen memory leaks with backuppc running on  
debian 4?


No, but 256 MB seems like not a lot of memory if you're using rsync  
and arrays that large. Swapping will kill performance. Are you  
running your first full backup? I have no idea why it would be  
slower now than before.


Nils Breunese.
-- 
---

This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a  
browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/ 
___

BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/


-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Re: [BackupPC-users] out of memory, and slow backups after moving to 3.0

[Nicholas Mistry]

Thanks for the info.  I did try and search the archives before  
posting, but the interface through sourceforge was not giving me  
results that were pertinent to my problem.   I will do some more  
digging and see what i can find.


-N

--
Nicholas Mistry
[EMAIL PROTECTED]



On Oct 10, 2007, at 11:10 AM, Ambrose Li wrote:


On 08/10/2007, Nicholas Mistry <[EMAIL PROTECTED]> wrote:


I am seeing a very strange problem after transitioning to backuppc  
3.0.
Backups are taking considerably longer, and i am frequently  
getting out of

memory kernel panics messages dumped to the syslog.


This has been reported back when 3.0 was first released; it has to do
with the way the nightly job is scheduled, so you certainly are not  
seeing
something not seen before. I don't have the info offhand, though;  
but you

should be able to find some info about it in the list archives.

--
cheers,
-ambrose

Yahoo and Gmail must die. Yes, I use them, but they still must die.
PS: Don't trust everything you read in Wikipedia. (Very Important)


-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Re: [BackupPC-users] out of memory, and slow backups after moving to 3.0

[Nicholas Mistry]

Searching the archives for "Update to 3.0" came up with the following  
thread:


<http://sourceforge.net/mailarchive/message.php? 
msg_id=0F231DDFB7C8D040BE8B29848740FC4365502A%40exchange.matthey.cz>


These are exactly the problems that i am experiencing.As some of  
the responses have suggested, i had already tried to reduce  
MaxBackups (currently set to 1), but it has only allowed me to  
prevent some of the out of memory errors.   Unfortunately, from what  
i am seeing on other threads around that time, there was no real  
resolution to the problem.


I just cant believe that a P3-450 w/ 256MB, and software RAID-5  
running Backuppc 2.x outperformed a  P3-850 w/ Hardware RAID 10  
running backuppc 3.x.


I may have to roll back to the latest 2.x release and see if that  
makes any difference.  And if it does, find out why!


-N
--
Nicholas Mistry
[EMAIL PROTECTED]



On Oct 11, 2007, at 1:39 AM, Ambrose Li wrote:


On 11/10/2007, Craig Barratt <[EMAIL PROTECTED]> wrote:

Ambrose writes:

This has been reported back when 3.0 was first released; it has  
to do
with the way the nightly job is scheduled, so you certainly are  
not seeing
something not seen before. I don't have the info offhand, though;  
but you

should be able to find some info about it in the list archives.


I don't recall seeing that.  Can you point me at a reference?


I dug around my old mails a bit and found a thread "Update to 3.0"  
dated
January 29. I haven't dug further though so I am not sure if there  
are more

threads, but I remember that this problem has been reported more than
once.

--
cheers,
-ambrose

Yahoo and Gmail must die. Yes, I use them, but they still must die.
PS: Don't trust everything you read in Wikipedia. (Very Important)


-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

[BackupPC-users] Building an inexpensive backup pc.

[Nicholas Mistry]

Yesterday I was walking the streets of costco (a us bulk shopping club) and 
noticed that WD has put out a new 1TB mybook w/ usb/firewire 400/eSATA.   Since 
we dont use our backuppc for our mission critical backups, I thought why not 
build a new box with these drives.   Paired with software raid and LVM you 
could in essence have a very expandable solution for home/soho users.

Has anyone tried building a backuppc using usb drives (without a usb hub)?
 
Esata would be the better choice, but would require adding a card in most pcs.  -
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[BackupPC-users] My thoughts on building an inexpensive backuppc server.

[Nicholas Mistry]

For the past couple years I have been constantly researching for a way  
to create an inexpensive backup server that gives at least 1TB in full  
RAID 1 redundancy while keeping $/GB as low as possible.  I am  
targeting this deployment for something that i would use in a home or  
small business where the users could help themselves to restoring  
files (which backuppc does well) and could assist with offsite  
rotations by using additional external USB drives.I have tried  
this w/ other low power small form factor machines but in the past i  
found that they did not have enough driver support or bus speed to  
keep up at an acceptable rate.


Recently, i came across an article on the web where someone created a  
$200 pc... Using this article as a starting point I think that i am  
getting closer to what i want to do:



Starting with the hardware he suggests in the article, I plan to  
substitute his drives for larger drives in a software RAID1  
configuration we have:


Intel BLKD201GLY2 Motherboard + CPU - $70
1GB Kingston Value Ram - @20
APEX MI-100 Case w/ power supply - $56 (or any case that holds 2  
drives).

Two (2) WD 1TB drives.. WD10EACS -- $240each, $480.
Install your favorite flavor of linux with backuppc (CentOS, Fedora,  
Ubuntu, Debian) but install a stripped down version w/o the gui and  
the like.


Adding up the $ before tax and shipping yeilds: $626  or roughly  
$0.62 / GB in RAID 1 using linux software raid or LVM.


Some other thoughts:
I am aware that this is not going to be the fastest box out there, but  
you already know that is not what i am going for.

Possibly add an inexpensive PCI Gigabit Card for faster transfer speeds.
or add a Raid Card to support 5+ drives in a RAID 6 setup to increase  
storage to 4TB+ (w/ a bigger case)
Im not a fan of WD hard drives as i have had many fail on me... But  
these happened to be the best bang for the buck at the time of writing  
this email.  I would probably spend a bit more on drives I know that  
would last longer.


I am curious if anyone had some ideas , feedback, or has tried to  
create something similar. If i try this i will post some info on  
my findings in terms of performance and my total experience with  
setting this type of system up.


Thanks in advance.

-N

ps:  My benchmark is a P2-450 w/ 512MB RAM and 3x500GB SATA drives in  
a software RAID 5 config running under debian.  Backuppc runs quite  
well on this machine and backs up 4 linux servers  and 2 windows  
servers in less than 8 hours at night.  There is probably 3TB of data  
total across all the machines, and after compression and pooling it  
comes to around 5-600GB).  This system has been rock solid and has  
been in production for quite some time now.   I run rsyncd on the  
linux boxes and use SMB for the windows servers.This machine is  
only part of my total backup plan.  It exists to provide users the  
ability to restore files on their own, or if they need to pull an  
older revision. 
  -
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] My thoughts on building an inexpensive backuppc server.

[Nicholas Mistry]

Justin,

To be honest, i am open to any type of solution for the hardware  
platform, but i do not want to venture into recycling older  
hardware.If the system was only intended for my personal use, i  
definitely would have used one of my old Dell servers w/ an ESATA card  
with a bunch of external drives in a frankenstein configuration that  
was hidden in my basement.   Unfortunately my friends / family /  
clients generally cringe at the thought of relying on something that  
is "old".I find that people are much more comfortable to purchase  
the required hardware and have me come over and assemble it, as long  
as i stay within the budget.


Its a good suggestion, but just not exactly what i am looking to do..

thanks for your feedback..

-N

On Feb 11, 2008, at 9:51 PM, Justin Best wrote:

Well, my first concern would be that the board you selected doesn't  
seem to have great Linux support. See one of the reviews at http://www.newegg.com/Product/Product.aspx?Item=N82E16813121326


Why not just buy a used machine, or better still, re-use one you've  
already got? All my BackupPC servers are old, re-used hardware, and  
they work great. And, if you get a machine that is well known  
(example: PowerEdge 400SC) you'll be able to google for help if you  
have weird issues with the OS.


On Feb 11, 2008, at 7:18 PM, Nicholas Mistry wrote:

For the past couple years I have been constantly researching for a  
way to create an inexpensive backup server that gives at least 1TB  
in full RAID 1 redundancy while keeping $/GB as low as possible.  I  
am targeting this deployment for something that i would use in a  
home or small business where the users could help themselves to  
restoring files (which backuppc does well) and could assist with  
offsite rotations by using additional external USB drives.I  
have tried this w/ other low power small form factor machines but  
in the past i found that they did not have enough driver support or  
bus speed to keep up at an acceptable rate.


Recently, i came across an article on the web where someone created  
a $200 pc... Using this article as a starting point I think that i  
am getting closer to what i want to do:

<http://paulstamatiou.com/2008/01/25/diy-200-dollar-pc>

Starting with the hardware he suggests in the article, I plan to  
substitute his drives for larger drives in a software RAID1  
configuration we have:


Intel BLKD201GLY2 Motherboard + CPU - $70
1GB Kingston Value Ram - @20
APEX MI-100 Case w/ power supply - $56 (or any case that holds 2  
drives).

Two (2) WD 1TB drives.. WD10EACS -- $240each, $480.
Install your favorite flavor of linux with backuppc (CentOS,  
Fedora, Ubuntu, Debian) but install a stripped down version w/o the  
gui and the like.


Adding up the $ before tax and shipping yeilds: $626  or roughly  
$0.62 / GB in RAID 1 using linux software raid or LVM.


Some other thoughts:
I am aware that this is not going to be the fastest box out there,  
but you already know that is not what i am going for.
Possibly add an inexpensive PCI Gigabit Card for faster transfer  
speeds.
or add a Raid Card to support 5+ drives in a RAID 6 setup to  
increase storage to 4TB+ (w/ a bigger case)
Im not a fan of WD hard drives as i have had many fail on me... But  
these happened to be the best bang for the buck at the time of  
writing this email.  I would probably spend a bit more on drives I  
know that would last longer.


I am curious if anyone had some ideas , feedback, or has tried to  
create something similar. If i try this i will post some info  
on my findings in terms of performance and my total experience with  
setting this type of system up.


Thanks in advance.

-N

ps:  My benchmark is a P2-450 w/ 512MB RAM and 3x500GB SATA drives  
in a software RAID 5 config running under debian.  Backuppc runs  
quite well on this machine and backs up 4 linux servers  and 2  
windows servers in less than 8 hours at night.  There is probably  
3TB of data total across all the machines, and after compression  
and pooling it comes to around 5-600GB).  This system has been rock  
solid and has been in production for quite some time now.   I run  
rsyncd on the linux boxes and use SMB for the windows servers. 
This machine is only part of my total backup plan.  It exists to  
provide users the ability to restore files on their own, or if they  
need to pull an older revision.

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: 

Re: [BackupPC-users] My thoughts on building an inexpensive backuppc server.

[Nicholas Mistry]

On Feb 12, 2008, at 10:29 AM, dan wrote:


for instance $200
http://configure.us.dell.com/dellstore/config.aspx?c=us&cs=04&kc=6W300&l=en&oc=bsdwd2a&s=bsd&dgc=BF&cid=7420&lid=197374



Dan, the link did not resolve for me. Can you let me know what config  
you used?



make a shell script to add that mirror to your RAID1 and let it  
sync, then fail the drive and unmount it and notify you via email or  
something.  put that script into the hotplug actions so that when  
the USB drive with the externals ID is hotplugged, run the script.   
then you just plug the drive in and let it run and wait for the  
email that says its done.


Great suggestion... Have you tried this personally?   Do you have a  
script that accomplishes this?


-N-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] My thoughts on building an inexpensive backuppc server.

[Nicholas Mistry]

On Feb 14, 2008, at 3:12 PM, David Morton wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
>
> On Feb 11, 2008, at 9:18 PM, Nicholas Mistry wrote:
>
>> For the past couple years I have been constantly researching for a  
>> way to create an inexpensive backup server that gives at least 1TB  
>> in full RAID 1 redundancy while keeping $/GB as low as possible.  I  
>> am targeting this deployment for something that i would use in a  
>> home or small business where the users could help themselves to  
>> restoring files (which backuppc does well) and could assist with  
>> offsite rotations by using additional external USB drives.I  
>> have tried this w/ other low power small form factor machines but  
>> in the past i found that they did not have enough driver support or  
>> bus speed to keep up at an acceptable rate.
>
> I just recently did this, for my own home network and then for a  
> couple of others.  I came out just a little under $700 for 1.5 TB of  
> RAID 5 storage.  I wrote up a little bit on slashdot:  
> http://linux.slashdot.org/comments.pl?sid=367261&cid=21435931


David, I took a look at your post on slashdot, but did not see what  
motherboard and case you used.  Would you mind sharing these details..

Thanks

-N


-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] My thoughts on building an inexpensive backuppc server.

[Nicholas Mistry]

On Feb 15, 2008, at 9:47 AM, Mirco Piccin wrote:

> Hi.
>
>>> For the past couple years I have been constantly researching for a
>>> way to create an inexpensive backup server that gives at least 1TB
>>> in full RAID 1 redundancy while keeping $/GB as low as possible.
>
> Also i've done something that.
> I use a :
> - jetway motherboard J7F4K1G5D (1500 mhz, dual gigabit lan, dual sata
> channel, low consumption);
> - ram 1gb;
> - disk on module 1gb (for the OS - debian etch + backuppc);
> - controller sata with 4 additional sata port;
> - 4 hd barracuda 500gb sata2 (raid1, for backup data);
>
> It Works like a charm.
> I choose to use also a low noise psu + thermaltake case, but those are
> clearly not important.
> Without case and psu, the total price was very low (the 500 gb sata hd
> were in special sale :-D).


Mirco, Thanks for the suggestion of the jetway motherboard.  This  
pretty close to what i am looking for.   The addition of the DOM (Disk  
on Module) really makes this an ideal solution.

How is the performance of the Via C7 chip and the dual Gigabit?How  
have you configured the two gigabit ports?  Did you pair them?

Could you also share how you setup the DOM.  I am curious to find out  
what you did for a swap partition (if you even used one at all).

Lastly what case /  power supply  did you end up sticking all this  
stuff in?
thanks
-N

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/