[Bacula-users] Fwd: Freebsd snapshots and complaints "Will not descend"

[Troy Kocher]

> Listers, 
> 
> I'm having intermittent issues with a backup job that completes "OK" but 
> complains saying "Will not descend" into various user folders within a jail.  
> Over the weekend this job ran twice and on the first occasion it complained 
> about a larger set than on the second occasion.  Although many of the same 
> complaints were generated by both jobs.  The file daemon is running on 
> FreeBSD 7.3-RELEASE-p4, bacula-client-5.0.2, and takes snapshots of the jails 
> and mounts them for the director.  The director daemon is running FreeBSD 
> 8.2-RELEASE-p3, bacula-server-5.0.3, and has the following config for this 
> server:
> 
> FileSet{
>   Name = "foobar"
>   Include {
>  Options {
>Compression = GZIP
>signature = SHA1
>Sparse = yes
>Hardlinks = no
>}
>@/usr/local/etc/backuplist/foobar_backuplist
> }}
> 
> Pool
> {
>   Name  =  foobarF
>   Pool Type = Backup
>   Storage = kfoobarb
>   Maximum Volume Jobs = 4
>   Maximum Volume Bytes = 270g 
>   Volume Use Duration =  2d
>   Volume Retention = 10 days
>   Catalog Files = yes
>   Label Format = foobarF-
>   Recycle = yes
>   AutoPrune = yes
>   }
> 
> Job {
>   Name = Foobar_Backup
>   Enabled  = Yes#[Yes|NO]
>   Type = Backup  #[Backup,Restore,Verify,Admin]
>   Level= Full
>   Accurate= yes #[Yes|No]
>   Client  = tao-fd
>   Fileset = "foobar"
>   Schedule = WeeklyCycle
>   Storage = kfoobarb
>   Messages = Standard
>   Pool = foobarF
>   Write Bootstrap   = "/data/working/foobar.bsr"
>   Priority = 40
> }
> 
> #cat /usr/local/etc/foobar_backuplist
> File = /mnt/foobar/usr
> File = /mnt/foobar/var
> File = /mnt/foobar/etc
> File = /mnt/foobar/root
> 

Truncated job log:
08-Oct 09:11 kfoobarb-dir JobId 2858: Start Backup JobId 2858, 
Job=foobar_Backup.2011-10-08_08.00.01_01 

08-Oct 09:11 kfoobarb-dir JobId 2858: Using Device "kfoobarbpool" 

08-Oct 09:10 jailhost-fd JobId 2858: DIR and FD clocks differ by -29 seconds, 
FD automatically compensating. 

08-Oct 09:11 kfoobarb-dir JobId 2858: Sending Accurate information. 

08-Oct 09:11 kfoobarb-sd JobId 2858: Volume "foobarF-0011" previously written, 
moving to end of data. 

08-Oct 09:11 kfoobarb-sd JobId 2858: Ready to append to end of Volume 
"foobarF-0011" size=1031166 
08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/ltartmp is a 
different filesystem. Will not descend from /mnt/foobar/usr into 
/mnt/foobar/usr/home/ltartmp 

08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/ltlclerk is a 
different filesystem. Will not descend from /mnt/foobar/usr into 
/mnt/foobar/usr/home/ltlclerk 

08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/tester is a 
different filesystem. Will not descend from /mnt/foobar/usr into 
/mnt/foobar/usr/home/tester 

08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/purchasing is a 
different filesystem. Will not descend from /mnt/foobar/usr into 
/mnt/foobar/usr/home/purchasing 

08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/whsebol is a 
different filesystem. Will not descend from /mnt/foobar/usr into 
/mnt/foobar/usr/home/whsebol 

08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/sysview is a 
different filesystem. Will not descend from /mnt/foobar/usr into 
/mnt/foobar/usr/home/sysview 

08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/rtessier is a 
different filesystem. Will not descend from /mnt/foobar/usr into 
/mnt/foobar/usr/home/rtessier 

08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home2 is a different 
filesystem. Will not descend from /mnt/foobar/usr into /mnt/foobar/usr/home2 

08-Oct 23:57 kfoobarb-sd JobId 2858: Job write elapsed time = 14:45:49, 
Transfer rate = 2.702 M Bytes/second 

08-Oct 23:57 kfoobarb-dir JobId 2858: Bacula kfoobarb-dir 5.0.3 (04Aug10): 
08-Oct-2011 23:57:42 
Build OS: amd64-portbld-freebsd8.2 freebsd 8.2-RELEASE 

JobId: 2858 
Job: foobar_Backup.2011-10-08_08.00.01_01 

Backup Level: Differential, since=2011-09-24 23:52:49 
Client: "jailhost-fd" 
5.0.2 (28Apr10) amd64-portbld-freebsd7.2,freebsd,7.2-RELEASE 

FileSet: "foobar" 2010-08-21 02:05:01 

Pool: "foobarF" (From Job resource) 

Catalog: "MyCatalog" (From Client resource) 

Storage: "kfoobarb" (From Pool resource) 

Scheduled time: 08-Oct-2011 08:00:01 

Start time: 08-Oct-2011 09:11:24 

End time: 08-Oct-2011 23:57:42 

Elapsed time: 14 hours 46 mins 18 secs 

Priority: 40 

FD Files Written: 222,266 foobar
SD 
Files Written: 222,266 

FD Bytes Written: 143,571,912,970 (143.5 GB) 

SD Bytes Written: 143,609,540,425 (143.6 GB) 

Rate: 2699.8 KB/s 

Software Compression: 44.5 % 

VSS: no 

Encryption: no 

Accurate: yes 

Volume name(s): foobarF-0011 

Volume Session Id: 59 

Volume Session Time: 1317927904 

Last Volume Bytes: 154,053,751,603 (154.0 GB) 
Non-fatal 
FD errors: 0 

SD Errors: 0 

FD termination status: OK 

SD termination status: OK 

Termination: Backup OK


> Given the intermittent natur

Re: [Bacula-users] Fwd: Freebsd snapshots and complaints "Will not descend"

[Troy Kocher]

On 10,Oct 2011, at 12:02 PM, David Romerstein wrote:

> On Mon, Oct 10, 2011 at 12:51 PM, Troy Kocher
>  wrote:
>> 08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/ltartmp is a 
>> different filesystem. Will not descend from /mnt/foobar/usr into 
>> /mnt/foobar/usr/home/ltartmp
> 
> From :
> 
> Bacula's default for processing directories is to recursively descend
> in the directory saving all files and subdirectories. Bacula will not
> by default cross filesystems (or mount points in Unix parlance). This
> means that if you specify the root partition (e.g. /), Bacula will
> save only the root partition and not any of the other mounted
> filesystems. Similarly on Windows systems, you must explicitly specify
> each of the drives you want saved (e.g. c:/ and d:/ ...). In addition,
> at least for Windows systems, you will most likely want to enclose
> each specification within double quotes particularly if the directory
> (or file) name contains spaces. The df command on Unix systems will
> show you which mount points you must specify to save everything.
> 
> -- D
> 
> 

Thanks for the response, I'm aware of how recursively descending works.  I use 
it on many other snapshot/jails and haven't had any issues getting other deeply 
buried folders I need backed up.  It works just as the manual describes.  There 
seems to be something unique about how this particular job/snapshot combination 
behaves, and that is what I'm trying to troubleshoot.  

Troy 


--
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Fwd: Freebsd snapshots and complaints "Will not descend"

[Martin Simmons]

> On Mon, 10 Oct 2011 11:51:14 -0500, Troy Kocher said:
> 
> Truncated job log:
> 08-Oct 09:11 kfoobarb-dir JobId 2858: Start Backup JobId 2858, 
> Job=foobar_Backup.2011-10-08_08.00.01_01 

> 08-Oct 09:11 kfoobarb-dir JobId 2858: Using Device "kfoobarbpool" 
> 
08-Oct 09:10 jailhost-fd JobId 2858: DIR and FD clocks differ by -29 
> seconds, FD automatically compensating. 
> 
08-Oct 09:11 kfoobarb-dir JobId 2858: Sending Accurate information. 
> 
08-Oct 09:11 kfoobarb-sd JobId 2858: Volume "foobarF-0011" previously 
> written, moving to end of data. 

> 08-Oct 09:11 kfoobarb-sd JobId 2858: Ready to append to end of Volume 
> "foobarF-0011" size=1031166 
> 08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/ltartmp is a 
> different filesystem. Will not descend from /mnt/foobar/usr into 
> /mnt/foobar/usr/home/ltartmp 
> 
08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/ltlclerk is a 
> different filesystem. Will not descend from /mnt/foobar/usr into 
> /mnt/foobar/usr/home/ltlclerk 
> 
08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/tester is a 
> different filesystem. Will not descend from /mnt/foobar/usr into 
> /mnt/foobar/usr/home/tester 

> 08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/purchasing is a 
> different filesystem. Will not descend from /mnt/foobar/usr into 
> /mnt/foobar/usr/home/purchasing 
> 
08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/whsebol is a 
> different filesystem. Will not descend from /mnt/foobar/usr into 
> /mnt/foobar/usr/home/whsebol 

> 08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/sysview is a 
> different filesystem. Will not descend from /mnt/foobar/usr into 
> /mnt/foobar/usr/home/sysview 
> 
08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home/rtessier is a 
> different filesystem. Will not descend from /mnt/foobar/usr into 
> /mnt/foobar/usr/home/rtessier 
> 
08-Oct 23:48 jailhost-fd JobId 2858: /mnt/foobar/usr/home2 is a different 
> filesystem. Will not descend from /mnt/foobar/usr into /mnt/foobar/usr/home2 

> 08-Oct 23:57 kfoobarb-sd JobId 2858: Job write elapsed time = 14:45:49, 
> Transfer rate = 2.702 M Bytes/second 

Are you running an automounter for home directories?  That could explain both
the "Will not descend" messages and also why the warnings vary over time.

__Martin

--
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Fwd: Freebsd snapshots and complaints "Will not descend"

[Troy Kocher]

On 10,Oct 2011, at 1:12 PM, Martin Simmons wrote:

>> On Mon, 10 Oct 2011 11:51:14 -0500, Troy Kocher said:
>> 
> 
>> 08-Oct 23:57 kfoobarb-sd JobId 2858: Job write elapsed time = 14:45:49, 
>> Transfer rate = 2.702 M Bytes/second 
> 
> Are you running an automounter for home directories?  That could explain both
> the "Will not descend" messages and also why the warnings vary over time.
> 
> __Martin
> 
> 

I'm not running an automounter.  And as I mentioned this error is intermittent. 
 I run this job incremental daily without complaint, I get this issue on the 
differential weekly run.  Regarding the time warning, I corrected this once by 
forcing an ntp on the fd client.  I think my ntp must not be running properly 
over there.

Beginning to feel like it's something with the snapshot (/mnt/foobar) not 
responding as a normal file system under load, and telling bacula-fd access is 
delayed/denied/?, then bacula understands the delay as device unreachable?

Troy


--
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Fwd: Freebsd snapshots and complaints "Will not descend"

[Christian Manal]

Am 10.10.2011 21:11, schrieb Troy Kocher:
> 
> On 10,Oct 2011, at 1:12 PM, Martin Simmons wrote:
> 
>>> On Mon, 10 Oct 2011 11:51:14 -0500, Troy Kocher said:
>>>
>>
>>> 08-Oct 23:57 kfoobarb-sd JobId 2858: Job write elapsed time = 14:45:49, 
>>> Transfer rate = 2.702 M Bytes/second 
>>
>> Are you running an automounter for home directories?  That could explain both
>> the "Will not descend" messages and also why the warnings vary over time.
>>
>> __Martin
>>
>>
> 
> I'm not running an automounter.  And as I mentioned this error is 
> intermittent.  I run this job incremental daily without complaint, I get this 
> issue on the differential weekly run.  Regarding the time warning, I 
> corrected this once by forcing an ntp on the fd client.  I think my ntp must 
> not be running properly over there.
> 
> Beginning to feel like it's something with the snapshot (/mnt/foobar) not 
> responding as a normal file system under load, and telling bacula-fd access 
> is delayed/denied/?, then bacula understands the delay as device unreachable?
> 
> Troy


Hi,

bacula won't recurse filesystems if you don't explicitly tell it to.
Look at the "onefs" option for the fileset resource:

http://bacula.org/5.0.x-manuals/en/main/main/Configuring_Director.html#8566


Regards,
Christian Manal

--
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users