[Bacula-users] bacula cloud backup
I have recently upgraded my bacula server to version 9.4.4 and wanted to test the cloud backup feature. Currently we are backing up to LTO7 tapes. Is it possible to set-up bacula to backup both to the tape and to the cloud simultaneously? If so I would appreciate some help with that. Thank you. UR. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-28 11:06, Thomas Lohman wrote: >> Hello >> >> I am used to this principle with Linux but I don't understand why it just >> takes it when Bacula is working and it slows down the server so much that I >> can no longer access it in ssh. > > How is your storage allocated on the server? i.e. how are things > partitioned with regard to your backup disks and your database? If your > DB is located on the same physical disks as your OS and/or your actual > backup data then you could see such "freeze ups" while Bacula is running > due to I/O limitations. I find it helps to separate the OS, DB data and > any Bacula storage volumes so they are all on separate disk devices if > possible - separate controllers even better. In particular it is a VERY BAD IDEA to have your catalog DB and your Bacula backup storage on the same physical devices. Both of those are high-I/O operations. You can EASILY saturate your I/O with a configuration like this, especially if you're writing to RAID5 where every write requires a parity calculation. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
Hello I am used to this principle with Linux but I don't understand why it just takes it when Bacula is working and it slows down the server so much that I can no longer access it in ssh. How is your storage allocated on the server? i.e. how are things partitioned with regard to your backup disks and your database? If your DB is located on the same physical disks as your OS and/or your actual backup data then you could see such "freeze ups" while Bacula is running due to I/O limitations. I find it helps to separate the OS, DB data and any Bacula storage volumes so they are all on separate disk devices if possible - separate controllers even better. --tom ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 1/28/2020 9:45 AM, Jean Mark Orfali wrote:
Hello
I am used to this principle with Linux but I don't understand why it just takes
it when Bacula is working and it slows down the server so much that I can no
longer access it in ssh.
Typically,
- failing drive,
- saturated network,
- broken name resolution.
If it only happens when bacula is running, I'd first run `iostat -dmx 5
5` and see if the drive where "File1" is located is clocking high wait
time and near-100% utilization.
If yes, throw it away and replace with a TLER ("NAS") HDD.
Dima
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-28 10:45, Jean Mark Orfali wrote: > Hello > > I am used to this principle with Linux but I don't understand why it just > takes it when Bacula is working and it slows down the server so much that I > can no longer access it in ssh. 1 - Presumably only Bacula is doing enough file access on this server that it needs to cache that many files. 2 - A slowdown such as you describe has nothing to do with file caching in memory. Memory is FAST. That's the POINT of caching recently opened files in memory. Have you looked at your storage I/O on this server? What is its physical and logical storage configuration? Do you know how to use iostat? Have you ruled out the case that your server is simply I/O bound - which is to say, you are saturating the I/O capacity of your physical storage? -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-28 10:08, Jean Mark Orfali wrote: > Hello Phil and Thomas, > > Here is the result of the bconsol. > > > Select daemon type for status (1-5): 5 > bacula-dir Version: 7.0.5 (28 July 2014) x86_64-redhat-linux-gnu redhat > Enterprise release > Daemon started 27-Jan-20 14:53. Jobs: run=0, running=5 mode=0,0 > Heap: heap=270,336 smbytes=652,990 max_bytes=737,558 bufs=557 max_bufs=563 > > Scheduled Jobs: > Level Type Pri Scheduled Job Name Volume > === > IncrementalBackup10 28-Jan-20 23:00BackOVH*unknown* > IncrementalBackup10 28-Jan-20 23:00BackphppROD*unknown* > IncrementalBackup10 28-Jan-20 23:00BackServeurFichier *unknown* > IncrementalBackup10 28-Jan-20 23:00BackBlanchard *unknown* > > > Running Jobs: > Console connected at 27-Jan-20 15:36 > Console connected at 28-Jan-20 10:01 > JobId Type Level Files Bytes Name Status > == > 75 Back Full 0 0 BackphppROD is running > 76 Back Incr 0 0 BackOVH is waiting on > Storage "File1" > 77 Back Full 0 0 BackphppROD is waiting on max > Client jobs > 78 Back Full 0 0 BackServeurFichier is waiting on > Storage "File1" > 79 Back Full 0 0 BackBlanchard is waiting on > Storage "File1" > OK, this looks pretty clear that something is not working as intended in your Storage configuration. You have a full backup running on BackphppROD, and your Storage is configured such that nothing else can write while that job is running. What's more, you have a second job waiting on BackphppROD that can't run. Now we have some idea where to look. For one thing, you should probably be using the following directives in your Director configuration: Allow Duplicate Jobs = no Cancel Queued Duplicates = yes What this does is tell the Director not to queue or run any new job on a client while another copy of that job, at the same or different level, is already running. This will prevent the situation above where you have a Full backup job BackphppROD running, and another Full backup copy (probably promoted from Incremental) queued to run as soon as it finishes. Now on to your Storage configuration. Let's see... What it looks like to me is that your fundamental problem here is that you have each client assigned its own POOL, with its own set of Volumes, but you only have one Storage device, which can only have one Volume open at a time. This means that EVERY job will block on ANY other running job because your Storage can only service one client at a time since your Clients each have their own unique Pools. (I see you've set up the virtual autochanger, but I don't use that and have no experience with it, so I can't speak about it.) Do you have an OPERATIONAL NEED to keep the backup data from different Clients separated on different Volumes? -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
Hello I am used to this principle with Linux but I don't understand why it just takes it when Bacula is working and it slows down the server so much that I can no longer access it in ssh. Jean Mark Orfali • Sys admin T. 1-877-258-4542 P. 227 bravad.ca/Fabrique Avis de confidentialité Le contenu de ce message ainsi que du ou des fichiers qui y sont joints est strictement confidentiel et destiné exclusivement à son ou sa destinataire. Si vous n’êtes pas cette personne, nous attirons votre attention sur le fait qu’il est strictement interdit de copier, de faire suivre ou d’utiliser les informations contenues dans ce courriel. Si vous l’avez reçu par erreur, nous vous remercions de nous le faire savoir. -Message d'origine- De : Phil Stracchino Envoyé : Tuesday, 28 January, 2020 10:36 AM À : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Bacula and memory usage On 2020-01-28 09:49, Thomas Lohman wrote: > > >> %Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 52.9 id, 46.5 wa, 0.0 hi, 0.2 si, 0.0 >> st >> KiB Mem : 29987532 total, 220092 free, 697356 used, 29070084 buff/cache >> KiB Swap: 15138812 total, 15138812 free,0 used. 28880936 avail Mem > > It looks like your memory is being used by the Linux file cache. This > is typical and if the system needs the memory for something else, it > will use it. Yup, agreed. This is something that often looks alarming to people coming from the Windows world, but in thew Unix world it is *perfectly normal*. Memory that you're not using is memory that isn't doing anything for you, and if you have free memory Unix will put it to use for something useful — caching files — until it needs it for something else. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-28 09:49, Thomas Lohman wrote: > > >> %Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 52.9 id, 46.5 wa, 0.0 hi, 0.2 si, 0.0 >> st >> KiB Mem : 29987532 total, 220092 free, 697356 used, 29070084 buff/cache >> KiB Swap: 15138812 total, 15138812 free,0 used. 28880936 avail Mem > > It looks like your memory is being used by the Linux file cache. This is > typical and if the system needs the memory for something else, it will > use it. Yup, agreed. This is something that often looks alarming to people coming from the Windows world, but in thew Unix world it is *perfectly normal*. Memory that you're not using is memory that isn't doing anything for you, and if you have free memory Unix will put it to use for something useful — caching files — until it needs it for something else. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
Hello Phil and Thomas, Here is the result of the bconsol. Select daemon type for status (1-5): 5 bacula-dir Version: 7.0.5 (28 July 2014) x86_64-redhat-linux-gnu redhat Enterprise release Daemon started 27-Jan-20 14:53. Jobs: run=0, running=5 mode=0,0 Heap: heap=270,336 smbytes=652,990 max_bytes=737,558 bufs=557 max_bufs=563 Scheduled Jobs: Level Type Pri Scheduled Job Name Volume === IncrementalBackup10 28-Jan-20 23:00BackOVH*unknown* IncrementalBackup10 28-Jan-20 23:00BackphppROD*unknown* IncrementalBackup10 28-Jan-20 23:00BackServeurFichier *unknown* IncrementalBackup10 28-Jan-20 23:00BackBlanchard *unknown* Running Jobs: Console connected at 27-Jan-20 15:36 Console connected at 28-Jan-20 10:01 JobId Type Level Files Bytes Name Status == 75 Back Full 0 0 BackphppROD is running 76 Back Incr 0 0 BackOVH is waiting on Storage "File1" 77 Back Full 0 0 BackphppROD is waiting on max Client jobs 78 Back Full 0 0 BackServeurFichier is waiting on Storage "File1" 79 Back Full 0 0 BackBlanchard is waiting on Storage "File1" Terminated Jobs: JobId LevelFiles Bytes Status FinishedName 37 Full 2 0 OK 22-Jan-20 11:55 BackOVH 382 0 OK 22-Jan-20 12:26 RestoreOVH 40 Full 2 0 OK 22-Jan-20 13:52 BackphppROD 412 0 OK 22-Jan-20 13:55 RestoreOVH 42 Full 2 0 OK 22-Jan-20 14:42 BackServeurFichier 432 0 OK 22-Jan-20 14:43 RestoreOVH 44 Full 2 0 OK 22-Jan-20 15:08 BackBlanchard 452 0 OK 22-Jan-20 15:09 RestoreOVH 54 Full 0 0 Error23-Jan-20 11:02 BackphppROD 56 Full173382.6 G OK 25-Jan-20 07:07 BackOVH Connecting to Storage daemon File1 at 51.79.119.27:9103 bacula-fd Version: 7.0.5 (28 July 2014) x86_64-redhat-linux-gnu redhat Enterprise release Daemon started 27-Jan-20 14:53. Jobs: run=0, running=3. Heap: heap=135,168 smbytes=985,447 max_bytes=1,070,207 bufs=277 max_bufs=285 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 mode=0,0 Running Jobs: Writing: Full Backup job BackphppROD JobId=75 Volume="PHPProd-0007" pool="PHPProd" device="FileStorage" (/home/bacula/backup) spooling=0 despooling=0 despool_wait=0 Files=3,613,617 Bytes=141,000,253,000 AveBytes/sec=2,125,098 LastBytes/sec=2,125,098 FDReadSeqNo=32,312,962 in_msg=22605665 out_msg=5 fd=5 Writing: Incremental Backup job BackOVH JobId=76 Volume="PHPProd-0007" pool="Default" device="FileStorage" (/home/bacula/backup) spooling=0 despooling=0 despool_wait=0 Files=0 Bytes=0 AveBytes/sec=0 LastBytes/sec=0 FDSocket closed Writing: Full Backup job BackServeurFichier JobId=78 Volume="PHPProd-0007" pool="Default" device="FileStorage" (/home/bacula/backup) spooling=0 despooling=0 despool_wait=0 Files=0 Bytes=0 AveBytes/sec=0 LastBytes/sec=0 FDSocket closed Writing: Full Backup job BackBlanchard JobId=79 Volume="PHPProd-0007" pool="Default" device="FileStorage" (/home/bacula/backup) spooling=0 despooling=0 despool_wait=0 Files=0 Bytes=0 AveBytes/sec=0 LastBytes/sec=0 FDSocket closed Jobs waiting to reserve a drive: 3608 JobId=76 wants Pool="Default" but have Pool="PHPProd" nreserve=0 on file device "FileStorage" (/home/bacula/backup). 3608 JobId=78 wants Pool="Default" but have Pool="PHPProd" nreserve=0 on file device "FileStorage" (/home/bacula/backup). 3608 JobId=79 wants Pool="Default" but have Pool="PHPProd" nreserve=0 on file device "FileStorage" (/home/bacula/backup). Terminated Jobs: JobId LevelFiles Bytes Status FinishedName === 37 Full 2 169 OK 22-Jan-20 11:55 BackOVH 382 169 OK 22-Jan-20 12:26 RestoreOVH 40 Full 2 176 OK 22-Jan-20 13:52 BackphppROD 412 176 OK 22-Jan-20 13:55 RestoreOVH 42 Full 2 184 OK 22-Jan-20 14:42 BackServeurFichier 432 184 OK 22-Jan-20 14:43 RestoreOVH 44 Full 2 185 OK 22-Jan-20 15:08 BackBlanchard 452 185 OK 22-Jan-20 15:09 RestoreOVH 54 Full 45,6234.994 G Cancel 23-Jan-20
Re: [Bacula-users] Bacula and memory usage
%Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 52.9 id, 46.5 wa, 0.0 hi, 0.2 si, 0.0 st KiB Mem : 29987532 total, 220092 free, 697356 used, 29070084 buff/cache KiB Swap: 15138812 total, 15138812 free,0 used. 28880936 avail Mem It looks like your memory is being used by the Linux file cache. This is typical and if the system needs the memory for something else, it will use it. As mentioned in my previous e-mail, can you run status within the director (bconsole) and see what the clients are doing when the backups are running? Is bacula actually backing anything up? The first thing to determine is if there is a problem/malfunction or if possibly your backups are simply taking too long to run (due to data total, # of files, etc.). --tom ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-28 08:30, Jean Mark Orfali wrote: > Hello Phil, > > I restarted the backups last night and automatically the memory increased to > 96% of use. 28.33GIB. Here are the results for mysqltuner and top. > Performance Metrics > --- > [--] Up for: 17h 30m 18s (109K q [1.735 qps], 53 conn, TX: 19M, RX: 639M) > [--] Reads / Writes: 5% / 95% > [--] Binary logging is disabled > [--] Physical Memory : 28.6G > [--] Max MySQL memory: 987.0M OK, so MariaDB is still using inder 1GB of RAM. So that's not the problem. > > top - 08:26:22 up 17:32, 2 users, load average: 5.43, 5.28, 5.53 > Tasks: 219 total, 1 running, 209 sleeping, 0 stopped, 9 zombie > %Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 52.9 id, 46.5 wa, 0.0 hi, 0.2 si, 0.0 > st > KiB Mem : 29987532 total, 220092 free, 697356 used, 29070084 buff/cache > KiB Swap: 15138812 total, 15138812 free,0 used. 28880936 avail Mem > > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 2093 bacula20 0 763968 6688 3100 S 15.6 0.0 23:33.51 bacula-sd > 6 root 20 0 0 0 0 S 2.3 0.0 3:10.41 > ksoftirqd/0 >93 root 20 0 0 0 0 S 0.3 0.0 0:16.80 kswapd0 > 2088 bacula20 0 1070748 9480 3820 S 0.3 0.0 2:39.34 bacula-dir > And according to top, bacula is using almost no memory either. In fact according to top, nothing is using any significiant amount of CPU *OR* memory. Upon what evidence are you basing your belief that you have a memory usage problem? Because top says you don't. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
Hello Phil, I restarted the backups last night and automatically the memory increased to 96% of use. 28.33GIB. Here are the results for mysqltuner and top. [!!] Your MySQL version 5.5.64-MariaDB is EOL software! Upgrade soon! [OK] Operating on 64-bit architecture Log file Recommendations -- [OK] Log file /var/log/mariadb/mariadb.log exists [--] Log file: /var/log/mariadb/mariadb.log(26K) [OK] Log file /var/log/mariadb/mariadb.log is readable. [OK] Log file /var/log/mariadb/mariadb.log is not empty [OK] Log file /var/log/mariadb/mariadb.log is smaller than 32 Mb [!!] /var/log/mariadb/mariadb.log contains 1 warning(s). [OK] /var/log/mariadb/mariadb.log doesn't contain any error. [--] 18 start(s) detected in /var/log/mariadb/mariadb.log [--] 1) 200127 14:53:46 [Note] /usr/libexec/mysqld: ready for connections. [--] 2) 200127 14:31:06 [Note] /usr/libexec/mysqld: ready for connections. [--] 3) 200127 13:42:01 [Note] /usr/libexec/mysqld: ready for connections. [--] 4) 200123 10:30:09 [Note] /usr/libexec/mysqld: ready for connections. [--] 5) 200123 9:30:38 [Note] /usr/libexec/mysqld: ready for connections. [--] 6) 200122 13:40:50 [Note] /usr/libexec/mysqld: ready for connections. [--] 7) 200122 11:25:59 [Note] /usr/libexec/mysqld: ready for connections. [--] 8) 200121 20:22:27 [Note] /usr/libexec/mysqld: ready for connections. [--] 9) 200121 18:11:25 [Note] /usr/libexec/mysqld: ready for connections. [--] 10) 200121 8:59:06 [Note] /usr/libexec/mysqld: ready for connections. [--] 13 shutdown(s) detected in /var/log/mariadb/mariadb.log [--] 1) 200127 14:53:14 [Note] /usr/libexec/mysqld: Shutdown complete [--] 2) 200127 14:30:34 [Note] /usr/libexec/mysqld: Shutdown complete [--] 3) 200122 11:25:23 [Note] /usr/libexec/mysqld: Shutdown complete [--] 4) 200121 20:21:55 [Note] /usr/libexec/mysqld: Shutdown complete [--] 5) 200121 18:10:56 [Note] /usr/libexec/mysqld: Shutdown complete [--] 6) 200121 8:58:36 [Note] /usr/libexec/mysqld: Shutdown complete [--] 7) 200117 15:17:29 [Note] /usr/libexec/mysqld: Shutdown complete [--] 8) 200117 13:32:59 [Note] /usr/libexec/mysqld: Shutdown complete [--] 9) 200117 11:08:05 [Note] /usr/libexec/mysqld: Shutdown complete [--] 10) 200116 14:37:52 [Note] /usr/libexec/mysqld: Shutdown complete Storage Engine Statistics - [--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA [--] Data in MyISAM tables: 1.0K (Tables: 1) [--] Data in InnoDB tables: 1.7M (Tables: 41) [OK] Total fragmented tables: 0 Analysis Performance Metrics -- [--] innodb_stats_on_metadata: OFF [OK] No stat updates during querying INFORMATION_SCHEMA. Security Recommendations -- [OK] There are no anonymous accounts for any database users [OK] All database users have passwords assigned [!!] There is no basic password file list! CVE Security Recommendations -- [--] Skipped due to --cvefile option undefined Performance Metrics --- [--] Up for: 17h 30m 18s (109K q [1.735 qps], 53 conn, TX: 19M, RX: 639M) [--] Reads / Writes: 5% / 95% [--] Binary logging is disabled [--] Physical Memory : 28.6G [--] Max MySQL memory: 987.0M [--] Other process memory: 0B [--] Total buffers: 416.0M global + 3.8M per thread (151 max threads) [--] P_S Max memory usage: 0B [--] Galera GCache Max memory usage: 0B [OK] Maximum reached memory usage: 438.7M (1.50% of installed RAM) [OK] Maximum possible memory usage: 987.0M (3.37% of installed RAM) [OK] Overall possible memory usage with other process is compatible with memory available [OK] Slow queries: 0% (1K/109K) [OK] Highest usage of available connections: 3% (6/151) [!!] Aborted connections: 5.66% (3/53) [!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance [!!] Query cache may be disabled by default due to mutex contention. [!!] Query cache efficiency: 0.0% (0 cached / 6K selects) [OK] Query cache prunes per day: 0 [OK] Sorts requiring temporary tables: 0% (0 temp sorts / 32 sorts) [OK] No joins without indexes [OK] Temporary tables created on disk: 9% (3 on disk / 31 total) [!!] Thread cache is disabled [OK] Table cache hit rate: 143% (83 open / 58 opened) [OK] table_definition_cache(400) is upper than number of tables(145) [OK] Open file limit used: 5% (53/962) [OK] Table locks acquired immediately: 100% (109K immediate / 109K locks) Performance schema [--] Performance schema is disabled. [--] Memory used by P_S: 0B [--] Sys schema
