Hi all,
Kern Sibbald wrote:
> To answer your question, you will need to do some timing. See below.
>
> On Monday 08 August 2005 09:37, Reinier Haasjes wrote:
>>Hi, sorry for the late reply, weekend.
>>I did the test again like you did it (with the time command) and the
>>result is as follow:
>>
>>-bash-2.05b$ time echo "restore jobid=1553,1561,1576,1598,1607,1617"
>>
>>|bconsole
>>
>>Connecting to Director tapeserver:9101
>>1000 OK: tapeserver-dir Version: 1.37.30 (14 July 2005)
>>Enter a period to cancel a command.
>>restore jobid=1553,1561,1576,1598,1607,1617
>>Using default Catalog name=MyCatalog DB=bacula
>>You have selected the following JobIds: 1553,1561,1576,1598,1607,1617
>
> The time between printing the above line and printing the following
> line from your email is
> pure database time. After that, it is a bit of both, with probably
> 10%
> database and 90% putting the records in memory.
>
> So a good cut of DB vs Bacula memory time would be messured here.
Here are some timings (i hope you ment this). Restarted bacula and mysql
so all the cpu-time taken are for the bacula build directory tree.
Before:
PID, TIME, COMMAND
20457 00:00:00 /bin/sh /usr/bin/mysqld_safe
20493 00:00:00 /usr/sbin/mysqld
20579 00:00:00 /opt/bacula-1.37.30/sbin/bacula-sd
20583 00:00:00 /opt/bacula-1.37.30/sbin/bacula-fd
20588 00:00:00 /opt/bacula-1.37.30/sbin/bacula-dir
After:
20457 00:00:00 /bin/sh /usr/bin/mysqld_safe
20493 00:00:08 /usr/sbin/mysqld
20579 00:00:00 /opt/bacula-1.37.30/sbin/bacula-sd
20583 00:00:00 /opt/bacula-1.37.30/sbin/bacula-fd
20588 00:42:48 /opt/bacula-1.37.30/sbin/bacula-dir
As you can see almost all the cputime is taken by bacula and the DB only
takes 8 seconds of cpu-time. So I think the problem lies in putting the
records into memory.
> You might check that you *really* have the indexes that are defined
> in
> the 1.37 src/cats/create_xx_databases files. Perhaps you are missing
> an
> index or two because you upgraded from an older version, or if you
> fiddled with your database, the old indexes could have been dropped.
>
> See your vendor's manual for how to see which indexes exist.
checked all the indexes and they *all* exist.
This part of the restore takes (almost) no time. The next line "Building
directory tree for JobId 1553 " takes the most time (+- 50 minutes),
'writing' all the +-signs takes this time.
>>Building directory tree for JobId 1553 ...
>>+++++++++++++++++++++++++++++++++++++++++++++++++
the next 5 lines takes about 5 minutes total.
Thank Reinier
>>Building directory tree for JobId 1561 ...
>>Building directory tree for JobId 1576 ...
>>Building directory tree for JobId 1598 ... +
>>Building directory tree for JobId 1607 ...
>>Building directory tree for JobId 1617 ...
>>6 Jobs, 429,836 files inserted into the tree.
>>
>>You are now entering file selection mode where you add (mark) and
>>remove (unmark) files to be restored. No files are initially added,
>> unless you used the "all" keyword on the command line.
>>Enter "done" to leave this mode.
>>
>>cwd is: /
>>$
>>real 54m48.701s
>>user 0m0.031s
>>sys 0m0.030s
>>
>>
>>54 minutus on the bacula server (Pentium III 800Mhz, 512Mb RAM) with a
>>local database (on a RAID0 vinum (FreeBSD) disc)
>>
>>the same test on my laptop (Pentium 4, 1,8Ghz, 1Gb RAM):
>>[EMAIL PROTECTED]:/opt/bacula/etc # time echo "restore
>>jobid=1553,1561,1576,1598,1607,1617" |bconsole
>>Connecting to Director penta:9101
>>1000 OK: penta-dir Version: 1.37.30 (14 July 2005)
>>Enter a period to cancel a command.
>>restore jobid=1553,1561,1576,1598,1607,1617
>>Using default Catalog name=MyCatalog DB=bacula
>>You have selected the following JobIds: 1553,1561,1576,1598,1607,1617
>>
>>Building directory tree for JobId 1553 ...
>>+++++++++++++++++++++++++++++++++++++++++++++++++
>>Building directory tree for JobId 1561 ...
>>Building directory tree for JobId 1576 ...
>>Building directory tree for JobId 1598 ...
>>Building directory tree for JobId 1607 ...
>>Building directory tree for JobId 1617 ...
>>6 Jobs, 424,612 files inserted into the tree.
>>
>>You are now entering file selection mode where you add (mark) and
>>remove (unmark) files to be restored. No files are initially added,
>> unless you used the "all" keyword on the command line.
>>Enter "done" to leave this mode.
>>
>>cwd is: /
>>$
>>real 43m32.431s
>>user 0m0.006s
>>sys 0m0.010s
>>
>>So yes it's a little bit faster but not as fast as you (a few minutes).
>>
>>My question is what is the biggest problem the 'slow' database or the
>>slow processor/memory combination?
>>
>>Thanks,
>>
>>Reinier
>>
>>Thomas Simmons wrote:
>>>That seems pretty slow to me. I just did a test and it took 10 seconds
>>>to build the tree for ~400,000 files. Like you, I too have an opteron
>>>system, a dual 246 w 1/GB ram, however I keep the database on a set of
>>>mirrored sata disks on the local server. Have you tried installing the
>>>database on the same server?
>>>
>>>sioux:~# time echo "restore jobid=1,2,3,4" |bconsole
>>>Connecting to Director sioux:9101
>>>1000 OK: sioux-dir Version: 1.37.30 (14 July 2005)
>>>Enter a period to cancel a command.
>>>restore jobid=1,2,3,4
>>>Using default Catalog name=MyCatalog DB=bacula
>>>You have selected the following JobIds: 1,2,3,4
>>>
>>>Building directory tree for JobId 1 ...
>>>+++++++++++++++++++++++++++++++++++++++++++++++++
>>>Building directory tree for JobId 2 ... ++++++++
>>>Building directory tree for JobId 3 ... ++
>>>Building directory tree for JobId 4 ... ++
>>>4 Jobs, 397,356 files inserted into the tree.
>>>
>>>You are now entering file selection mode where you add (mark) and
>>>remove (unmark) files to be restored. No files are initially added,
>>>unless you used the "all" keyword on the command line.
>>>Enter "done" to leave this mode.
>>>
>>>cwd is: /
>>>$
>>>real 0m10.371s
>>>user 0m0.001s
>>>sys 0m0.006s
>>>
>>>Thanks,
>>>Thomas
>>>
>>>[EMAIL PROTECTED] wrote:
>>>>I was just thinking. With my setup. The new bacula
>>>>server is an Opteron 246 server with 4GB of memory and
>>>>the database is running on an Athlon 2400 with only
>>>>256 MB of memory and it takes (1 to 3) minutes to get
>>>>the file list for around 10,000 files with version
>>>>1.36.3 and a postgresql database. The hard drive light
>>>>on machine with the database is solid for the whole
>>>>time. If I ran 400,000 files which is 40 times as many
>>>>files it could easily take an hour. I'm thinking its
>>>>time to update my database server...
>>>>
>>>>John
>>>>
>>>>--- Reinier Haasjes <[EMAIL PROTECTED]> wrote:
>>>>>[EMAIL PROTECTED] wrote:
>>>>>>>Hi,
>>>>>>>
>>>>>>>I'm using bacula 1.37.30 for a few days now and I
>>>>>>>decided to test a full server recovery.
>>>>>>>I discovered that the building the directory tree
>>>>>>>takes a very long time (almost an hour) for
>>>>>
>>>>>429,836
>>>>>
>>>>>>>files.
>>>>>>>I started the building of the tree at 11:20 and at
>>>>>>>12:15 I got a prompt.
>>>>>>>The machine is a dedicated bacula machine and was
>>>>>>>doing nothing else than building the tree, it'a a
>>>>>>>Penium III at 800Mhz with 512MB memory.
>>>>>>>
>>>>>>>My question is if this is normal for builing the
>>>>>>>tree? Because if one of the server dies we want to
>>>>>>>recover as soon as possible.
>>>>>>>
>>>>>>>Thanks,
>>>>>>>
>>>>>>>Reinier
>>>>>>>
>>>>>>>output bconsole:
>>>>>>>----
>>>>
>>>>+-------+-------+----------+----------------+---------------------+----
>>>>- -------+-----------+
>>>>
>>>>>>>| JobId | Level | JobFiles | JobBytes |
>>>>>>>
>>>>>>>StartTime | VolumeName | StartFile |
>>>>
>>>>+-------+-------+----------+----------------+---------------------+----
>>>>- -------+-----------+
>>>>
>>>>>>>| 1,553 | F | 426,357 | 13,464,836,018 |
>>>>>>>
>>>>>>>2005-07-30 15:40:00 | 000017L1 | 100 |
>>>>>>>
>>>>>>>| 1,561 | I | 41 | 41,277,841 |
>>>>>>>
>>>>>>>2005-07-31 01:32:52 | 000034L1 | 106 |
>>>>>>>
>>>>>>>| 1,576 | I | 137 | 101,733,681 |
>>>>>>>
>>>>>>>2005-08-01 01:31:02 | 000034L1 | 116 |
>>>>>>>
>>>>>>>| 1,598 | I | 2,315 | 101,235,058 |
>>>>>>>
>>>>>>>2005-08-03 02:09:03 | 000034L1 | 146 |
>>>>>>>
>>>>>>>| 1,607 | I | 1,641 | 220,426,382 |
>>>>>>>
>>>>>>>2005-08-04 02:03:39 | 000034L1 | 191 |
>>>>>>>
>>>>>>>| 1,617 | I | 1,345 | 100,591,549 |
>>>>>>>
>>>>>>>2005-08-05 01:51:58 | 000035L1 | 28 |
>>>>
>>>>+-------+-------+----------+----------------+---------------------+----
>>>>- -------+-----------+
>>>>
>>>>>>>You have selected the following JobIds:
>>>>>>>1553,1561,1576,1598,1607,1617
>>>>>>>
>>>>>>>Building directory tree for JobId 1553 ...
>>>>>>>+++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>Building directory tree for JobId 1561 ... Building directory
>>>>>>>tree
>>>>>>>for JobId 1576 ... Building directory tree for JobId 1598 ... +
>>>>>>>Building directory tree for JobId 1607 ... Building directory
>>>>>>>tree
>>>>>>>for JobId 1617 ... 6 Jobs, 429,836 files inserted into the tree.
>>>>>>>----
>>>>>>>
>>>>>>>
>>>>>>>Output top (during building tree):
>>>>>>>----
>>>>>>>32 processes: 2 running, 30 sleeping
>>>>>>>CPU states: 98.1% user, 0.0% nice, 0.4% system, 1.6% interrupt,
>>>>>>>0.0% idle
>>>>>>>Mem: 131M Active, 36M Inact, 89M Wired, 60M Buf,
>>>>>>>242M Free
>>>>>>>Swap: 1020M Total, 120K Used, 1020M Free
>>>>>>>
>>>>>>>PID USERNAME PRI NICE SIZE RES STATE TIME
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>WCPU CPU COMMAND
>>>>>>>91877 root 64 0 88948K 87612K RUN 269:36
>>>>>>>98.39% 98.39% bacula-dir
>>>>>>>93618 root 29 0 1908K 1072K RUN 0:00
>>>>>>>
>>>>>>>
>>>>>>>2.96% 0.54% top
>>>>>>>----
>>>>
>>>>-------------------------------------------------------
>>>>
>>>>>>>SF.Net email is Sponsored by the Better Software
>>>>>>>Conference & EXPO
>>>>>>>September 19-22, 2005 * San Francisco, CA *
>>>>>>>Development Lifecycle Practices
>>>>>>>Agile & Plan-Driven Development * Managing
>>>>>
>>>>>Projects
>>>>>
>>>>>>>& Teams * Testing & QA
>>>>>>>Security * Process Improvement & Measurement *
>>>>>>>http://www.sqe.com/bsce5sf
>>>>>>>_______________________________________________
>>>>>>>Bacula-users mailing list
>>>>>>>Bacula-users@lists.sourceforge.net
>>>>
>>>>https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>>
>>>>>>You are restoring 14GB of data right? For most
>>>>>
>>>>>tape
>>>>>
>>>>>>drives this is not a long time. For me a 40GB
>>>>>
>>>>>DLT-IV
>>>>>
>>>>>>(native) tape takes 3 to 4 hours to restore. The
>>>>>
>>>>>drive
>>>>>
>>>>>>has a 3MB/s data rate which is about 11GB / hour
>>>>>>(native). And that is as fast as it will go. If
>>>>>
>>>>>you
>>>>>
>>>>>>manage ot get a fileset that is highly
>>>>>
>>>>>compressible
>>>>>
>>>>>>you can get better times. But most of my data I
>>>>>
>>>>>get no
>>>>>
>>>>>>where near a compression rate of 2.0.
>>>>>>John
>>>>>
>>>>>I'm not talking about the actual recovery itself
>>>>>(data transfer) but
>>>>>about building the directory tree (before the 'mark
>>>>>*' command).
>>>>>
>>>>>The actual restore takes 'only' 2:35 hours.
>>>>>
>>>>>Reinier
>>>>
>>>>-------------------------------------------------------
>>>>
>>>>>SF.Net email is Sponsored by the Better Software
>>>>>Conference & EXPO
>>>>>September 19-22, 2005 * San Francisco, CA *
>>>>>Development Lifecycle Practices
>>>>>Agile & Plan-Driven Development * Managing Projects
>>>>>& Teams * Testing & QA
>>>>>Security * Process Improvement & Measurement *
>>>>>http://www.sqe.com/bsce5sf
>>>>>_______________________________________________
>>>>>Bacula-users mailing list
>>>>>Bacula-users@lists.sourceforge.net
>>>>
>>>>https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>>
>>>>
>>>>
>>>>
>>>>-------------------------------------------------------
>>>>SF.Net email is Sponsored by the Better Software Conference & EXPO
>>>>September 19-22, 2005 * San Francisco, CA * Development Lifecycle
>>>>Practices
>>>>Agile & Plan-Driven Development * Managing Projects & Teams * Testing
>>>>& QA
>>>>Security * Process Improvement & Measurement *
>>>>http://www.sqe.com/bsce5sf
>>>>_______________________________________________
>>>>Bacula-users mailing list
>>>>Bacula-users@lists.sourceforge.net
>>>>https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>>-------------------------------------------------------
>>SF.Net email is Sponsored by the Better Software Conference & EXPO
>>September 19-22, 2005 * San Francisco, CA * Development Lifecycle
>> Practices Agile & Plan-Driven Development * Managing Projects & Teams
>> *
>> Testing & QA Security * Process Improvement & Measurement *
>> http://www.sqe.com/bsce5sf
>> _______________________________________________
>>Bacula-users mailing list
>>Bacula-users@lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/bacula-users
