Re: [Bacula-users] Why it takes so much to Building directory tree
With that result, I would not increase the innodb buffer setting. Your system is already swapping (660 MB of 2 GB swap space in use). There's nothing that impairs overall system performance as much as that. -Original Message- From: f-otake [mailto:f-ot...@kinryokai.net] Sent: 05 July 2015 16:31 To: bacula-users Subject: Re: [Bacula-users] Why it takes so much to Building directory tree Thanks for Radosław Korzeniewski and Luc Van der Veken I measure the time of Building directory tree on my previous case and it took only 12seconds. also check free command when running. total used free sharedbuffers cached Mem: 1020300 932360 87940220 46276 341924 -/+ buffers/cache: 544160 476140 Swap: 2097148 6601681436980 It maybe assign more to innodb_buffer_pool_size, but keep same setting. -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Why it takes so much to Building directory tree
You should look out when copying an innodb_buffer_pool_size from a webpage or so, the best value for that setting depends too much on local conditions, esp. available RAM, database size, and what else is running (and needing memory) on the same server. In general, the higher the setting, the better, but you have to take care not to take so much that other components suffer from it. And of course, setting it higher than your database size is useless, but I don't think that will be an issue with Bacula (my bacula db is 11 GB). I think it's a good approach to log on to the server while backups are running and examine the output of the 'free' command a number of times, if possible under different circumstances (while clients are being backed up, while the director is being backed up, while the MySQL database itself is being dumped, while a restore is running, etc.). If you see that 'free' reports a lot of free memory all of the time, allocate part of that for database use by adding it to innodb_buffer_pool_size. If you experience problems or slow behavior later, check it again. If it is reporting low memory then (or especially swap space use of more than a few MB), decrease innodb_buffer_pool_size to free some up. Even better is setting up Munin (or similar) to monitor your Bacula machine and examine the memory use graphs. -Original Message- From: f-ot...@kinryokai.net [mailto:f-ot...@kinryokai.net] Sent: 05 July 2015 1:17 To: Bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Why it takes so much to Building directory tree Many thanks for Heitor Faria(Mr./Ms.) and Randoslaw Koreniewski(Mr./Ms) Mysql DB was MyISAM therefore changed to InnoDB. and also inserted few lines as suggested on http://bacula.us/tuning/ as follows sort_buffer_size = 2MB innodb_buffer_pool_size = 128MB innodb_flush_log_at_trx_commit = 0 innodb_flush_method = O_DIRECT then Bulding directory tree finish almost immidiatly. FYI following case take only less than 30seconds +---+---+--+---+-++ | JobId | Level | JobFiles | JobBytes | StartTime | VolumeName | +---+---+--+---+-++ | 4 | F | 258,441 | 2,262,481,683 | 2015-07-02 14:32:50 | Vol-0004 | |14 | I | 158 |79,623,411 | 2015-07-03 04:32:49 | Vol-0014 | +---+---+--+---+-++ Thanks again for the help. F.Otake -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Why it takes so much to Building directory tree
Hello, 2015-07-05 1:17 GMT+02:00 f-ot...@kinryokai.net: Many thanks for Heitor Faria(Mr./Ms.) and Randoslaw Koreniewski(Mr./Ms) Mysql DB was MyISAM therefore changed to InnoDB. and also inserted few lines as suggested on http://bacula.us/tuning/ as follows sort_buffer_size = 2MB innodb_buffer_pool_size = 128MB innodb_flush_log_at_trx_commit = 0 innodb_flush_method = O_DIRECT then Bulding directory tree finish almost immidiatly. FYI following case take only less than 30seconds I can guess there are a plenty space for tunning. # time echo restore client=radekk fileset=\MacOSX Server Full Set\ select current done|bconsole Using Catalog BackupCatalog ++---+---+---+-++ | jobid | level | jobfiles | jobbytes | starttime | volumename | ++---+---+---+-++ | 34,154 | F | 2,258,200 | 749,116,220 | 2015-05-09 20:10:38 | 0029C | | 34,161 | I |26,263 | 561,273,244 | 2015-05-12 20:14:14 | 006D9 | | 34,182 | I | 5,690 | 434,817,665 | 2015-05-13 20:13:12 | 001B9 | | 34,201 | I | 3,813 | 253,870,636 | 2015-05-14 20:16:03 | 001A9 | | 34,220 | I | 6,890 | 303,529,259 | 2015-05-15 20:14:16 | 00013 | | 34,257 | I | 2,466 |90,700,158 | 2015-05-17 20:14:16 | 00550 | | 34,288 | I |17,622 | 542,569,957 | 2015-05-18 20:25:34 | 0005B | | 34,306 | I | 9,959 | 177,332,163 | 2015-05-19 20:20:44 | 00149 | | 34,324 | I | 4,495 | 352,862,913 | 2015-05-20 20:20:45 | 0029E | | 34,342 | I |11,945 | 544,958,861 | 2015-05-21 20:21:43 | 006C7 | | 34,386 | I |83,163 | 4,344,990,138 | 2015-06-28 21:44:03 | 0038C | | 34,410 | I | 2,679 | 2,420,709,204 | 2015-06-29 20:00:00 | 00357 | | 34,431 | I | 8,877 | 307,858,068 | 2015-06-30 20:00:01 | 000B4 | | 34,456 | I | 2,391 | 286,562,783 | 2015-07-01 20:00:00 | 0017F | | 34,501 | I | 8,843 | 680,933,784 | 2015-07-03 20:00:01 | 001EC | | 34,522 | I | 2,008 | 164,777,395 | 2015-07-04 20:00:00 | 001D2 | ++---+---+---+-++ You have selected the following JobIds: 34154,34161,34182,34201,34220,34257,34288,34306,34324,34342,34386,34410,34431,34456,34501,34522 Building directory tree for JobId(s) 34154,34161,34182,34201,34220,34257,34288,34306,34324,34342,34386,34410,34431,34456,34501,34522 ... ++ 1,906,548 files inserted into the tree. No files selected to be restored. You have messages. real 0m24.743s user 0m0.028s sys 0m0.000s So it took 24 seconds to handle almost 2M files restore. It is a 5Y old small server with a postgresql database. best regards -- Radosław Korzeniewski rados...@korzeniewski.net -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Why it takes so much to Building directory tree
Thanks for Radosław Korzeniewski and Luc Van der Veken I measure the time of Building directory tree on my previous case and it took only 12seconds. also check free command when running. total used free sharedbuffers cached Mem: 1020300 932360 87940220 46276 341924 -/+ buffers/cache: 544160 476140 Swap: 2097148 6601681436980 It maybe assign more to innodb_buffer_pool_size, but keep same setting. -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Why it takes so much to Building directory tree
Many thanks for Heitor Faria(Mr./Ms.) and Randoslaw Koreniewski(Mr./Ms) Mysql DB was MyISAM therefore changed to InnoDB. and also inserted few lines as suggested on http://bacula.us/tuning/ as follows sort_buffer_size = 2MB innodb_buffer_pool_size = 128MB innodb_flush_log_at_trx_commit = 0 innodb_flush_method = O_DIRECT then Bulding directory tree finish almost immidiatly. FYI following case take only less than 30seconds +---+---+--+---+-++ | JobId | Level | JobFiles | JobBytes | StartTime | VolumeName | +---+---+--+---+-++ | 4 | F | 258,441 | 2,262,481,683 | 2015-07-02 14:32:50 | Vol-0004 | |14 | I | 158 |79,623,411 | 2015-07-03 04:32:49 | Vol-0014 | +---+---+--+---+-++ Thanks again for the help. F.Otake -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Why it takes so much to Building directory tree
I am bigginer of bacula, and I had tried restore on following condition. +---+---+--+---+-++ | JobId | Level | JobFiles | JobBytes | StartTime | VolumeName| +---+---+--+---+-++ | 4 | F | 258,441 | 2,262,481,683 | 2015-07-02 14:32:50 | Vol-0004 | |14 | I | 158 |79,623,411 | 2015-07-03 04:32:49 | Vol-0014 | +---+---+--+---+-++ You have selected the following JobIds: 4,14 Building directory tree for JobId(s) 4,14 ... It takes more then 5hours 20min for just to builde directory tree. Is it normal to take such long time or my setting may be wrong? Since bacula-dir.conf is very big thus may not good to write here. For your information OS is CentOS Ver 6.6 on KVM and cpu assigned only one to this host. CPU is AMD Athlon 5350。4core、TDP２５W、２GHｚ(DB is mysql ver 5.1.73) Appliciate your kind help. Hello f-otake: please try the mysql settings suggested here = http://bacula.us/tuning/ Regards, === Heitor Medrado de Faria - LPIC-III | ITIL-F | Bacula Systems Certified Administrator II Do you need Bacula training? https://www.udemy.com/bacula-backup-software/?couponCode=bacula-list +55 61 8268-4220 Site: http://bacula.us FB: heitor.faria === -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Why it takes so much to Building directory tree
Hello, 2015-07-03 7:11 GMT+02:00 f-otake f-ot...@kinryokai.net: I am bigginer of bacula, and I had tried restore on following condition. +---+---+--+---+-++ | JobId | Level | JobFiles | JobBytes | StartTime | VolumeName| +---+---+--+---+-++ | 4 | F | 258,441 | 2,262,481,683 | 2015-07-02 14:32:50 | Vol-0004 | |14 | I | 158 |79,623,411 | 2015-07-03 04:32:49 | Vol-0014 | +---+---+--+---+-++ You have selected the following JobIds: 4,14 Building directory tree for JobId(s) 4,14 ... It takes more then 5hours 20min for just to builde directory tree. Is it normal to take such long time or my setting may be wrong? Your catalog database has a terribly bad performance. You should tune its configuration or upgrade your hardware. BTW, I had a better results on i486/300MHz/128MB Ram and single 120GB HDD. Since bacula-dir.conf is very big thus may not good to write here. It is not required. For your information OS is CentOS Ver 6.6 on KVM and cpu assigned only one to this host. CPU is AMD Athlon 5350。4core、TDP２５W、２GHｚ(DB is mysql ver 5.1.73) Appliciate your kind help. Check your utilization first - cpu, i/o, memory. Next do the db tuning. best regards -- Radosław Korzeniewski rados...@korzeniewski.net -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Why it takes so much to Building directory tree
I am bigginer of bacula, and I had tried restore on following condition. |+---+---+--+---+-++ | JobId | Level | JobFiles | JobBytes | StartTime | VolumeName| +---+---+--+---+-++ | 4 | F | 258,441 | 2,262,481,683 | 2015-07-02 14:32:50 | Vol-0004 | |14 | I | 158 |79,623,411 | 2015-07-03 04:32:49 | Vol-0014 | +---+---+--+---+-++ You have selected the following JobIds: 4,14 Building directory tree for JobId(s) 4,14 ... | It takes more then 5hours 20min for just to builde directory tree. Is it normal to take such long time or my setting may be wrong? Since bacula-dir.conf is very big thus may not good to write here. For your information OS is CentOS Ver 6.6 on KVM and cpu assigned only one to this host. CPU is AMD Athlon 5350。4core、TDP２５W、２GHｚ(DB is mysql ver 5.1.73) Appliciate your kind help. -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
