Re: [Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs

2019-01-26 Thread Kern Sibbald 

  
  
Hello Radek,

The code has been there since 2011.  I now have a patch that makes
the error message
very clear.

Yes, I am aware how bvfs works and why the cache is there.  I was
just surprised that
the existence of the cache would disable a dbcheck function.

As far as I can see, this is not a temporary situation.

Note, in the community version some of the bvfs functions may not be
implemented or may only
be implemented simply because the catalog database schemas between
the Enterprise version and
the community version are different simply because I did not have
the time to backport those changes
because the backport would be very tricky.  I will remove these
database differences in the next major 
release, and then the Community will have access to all these bvfs
functions.   

I haven't confirmed it, but from what I can see, the error message
is to avoid having a search
cache that is out of date with the actual state of the path table --
so the solution is to clear the
path cache table, and the new message explains exactly how to do it.

Best regards,
Kern

On 1/26/19 8:01 AM, Radosław
  Korzeniewski wrote:


  
  

  

  Hello Kern,
  
  
pt., 25 sty 2019 o
  17:06 Kern Sibbald 
  napisał(a):


   Hello,

I find the error message very cryptic, and I really
don't like just giving up, so I have asked the
author
of that particular code to supply a fix.
  



From what I understand here bvfs is using a special
  cache tables (pathvisibility, pathhierarchy) for fast
  job files browsing. These cache tables have a
  references to 'path' and job tables but these
  references aren't guarded by a database constraints,
  so I call it a soft references (I assume it is for
  faster update and search). 



  bacula=# \d
  pathvisibility
  Table
  "public.pathvisibility"
   Column | 
  Type   | Modifiers
  +-+---
   pathid |
  integer | not null
   jobid  |
  integer | not null
   size   |
  bigint  | default 0
   files  |
  integer | default 0
  Indexes:
     
  "pathvisibility_pkey" PRIMARY KEY, btree (jobid,
  pathid)
     
  "pathvisibility_jobid" btree (jobid)
  

  bacula=# \d
  pathhierarchy
   Table
  "public.pathhierarchy"
   Column  | 
  Type   | Modifiers
  -+-+---
   pathid  |
  integer | not null
   ppathid |
  integer | not null
  Indexes:
     
  "pathhierarchy_pkey" PRIMARY KEY, btree (pathid)
     
  "pathhierarchy_ppathid" btree (ppathid)



The dbcheck utility has no code to deal with bvfs
  cache tables, so when it finds these soft references
  it actually raise an error. I can imagine that this is
  a temporary solution.


best regards
 

   
Best regards,
Kern

On
  1/23/19 11:03 AM, Radosław Korzeniewski wrote:


  
Hello,


  wt.,
22 sty 2019 o 14:39 Stefan Muenkner 
napisał(a):
  
  (...)
Searching for hascache in the rest of the
source I found that hascache in
the job table seems to be only reset in

Re: [Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs

2019-01-25 Thread Radosław Korzeniewski 
Hello Kern,

pt., 25 sty 2019 o 17:06 Kern Sibbald  napisał(a):

> Hello,
>
> I find the error message very cryptic, and I really don't like just giving
> up, so I have asked the author
> of that particular code to supply a fix.
>

>From what I understand here bvfs is using a special cache tables
(pathvisibility, pathhierarchy) for fast job files browsing. These cache
tables have a references to 'path' and job tables but these references
aren't guarded by a database constraints, so I call it a soft references (I
assume it is for faster update and search).

bacula=# \d pathvisibility
Table "public.pathvisibility"
 Column |  Type   | Modifiers
+-+---
 pathid | integer | not null
 jobid  | integer | not null
 size   | bigint  | default 0
 files  | integer | default 0
Indexes:
"pathvisibility_pkey" PRIMARY KEY, btree (jobid, pathid)
"pathvisibility_jobid" btree (jobid)

bacula=# \d pathhierarchy
 Table "public.pathhierarchy"
 Column  |  Type   | Modifiers
-+-+---
 pathid  | integer | not null
 ppathid | integer | not null
Indexes:
"pathhierarchy_pkey" PRIMARY KEY, btree (pathid)
"pathhierarchy_ppathid" btree (ppathid)

The dbcheck utility has no code to deal with bvfs cache tables, so when it
finds these soft references it actually raise an error. I can imagine that
this is a temporary solution.

best regards


>
> Best regards,
> Kern
>
> On 1/23/19 11:03 AM, Radosław Korzeniewski wrote:
>
> Hello,
>
> wt., 22 sty 2019 o 14:39 Stefan Muenkner 
> napisał(a):
>
>> (...)
>> Searching for hascache in the rest of the source I found that hascache in
>> the job table seems to be only reset in ".bvfs_clear_cache".
>>
>
> Great!
>
>
>> >> 
>> >>   9) Check for orphaned Path records
>> >> 
>> >> Select function number: 9
>> >> Pruning orphaned Path entries isn't possible when using BVFS.
>> >> 
>>
>
> So, it seems the message above should be more descriptive and user
> friendly and points to bvfs_clear_cache command.
>
> best regards
> --
> Radosław Korzeniewski
> rados...@korzeniewski.net
>
>
>

-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs

2019-01-25 Thread Kern Sibbald 

  
  
Hello,

I find the error message very cryptic, and I really don't like just
giving up, so I have asked the author
of that particular code to supply a fix.

Best regards,
Kern

On 1/23/19 11:03 AM, Radosław
  Korzeniewski wrote:


  
  
Hello,


  wt., 22 sty 2019 o
14:39 Stefan Muenkner 
napisał(a):
  
  (...)
Searching for hascache in the rest of the source I found
that hascache in
the job table seems to be only reset in ".bvfs_clear_cache".
  
  
  
  Great!
  
  
  
>> 
>>   9) Check for orphaned Path records
>> 
>> Select function number: 9
>> Pruning orphaned Path entries isn't possible when
using BVFS.
>> 
  
  
  
  So, it seems the message above should be more descriptive
and user friendly and points to bvfs_clear_cache command. 
  
  

best regards
-- 
Radosław Korzeniewski
  rados...@korzeniewski.net
  


  


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs

2019-01-23 Thread Stefan Muenkner 
Am 23.01.19 um 11:03 schrieb Radosław Korzeniewski:
> Hello,

Hi,

> wt., 22 sty 2019 o 14:39 Stefan Muenkner  > napisał(a):
> 
> (...)
> Searching for hascache in the rest of the source I found that hascache in
> the job table seems to be only reset in ".bvfs_clear_cache".
> 
> 
> Great!
> 
> 
> >> 
> >>   9) Check for orphaned Path records
> >> 
> >> Select function number: 9
> >> Pruning orphaned Path entries isn't possible when using BVFS.
> >> 
> 
> 
> So, it seems the message above should be more descriptive and user
> friendly and points to bvfs_clear_cache command. 

No I don't think so. As far as I remember the .bvfs_xxx commands are
actually only there to help GUIs interact with the server - I think they
are not meant to be widely used.

It is actually the case that whenever bat (at least my version 7.4.1 here)
is called and you select any job in brestore, it sets hascache=1 for the
respective job (and seemingly some more). Originally I thought the
occasionally crashing bat left those behind, but even if bat is closed
regularly it never clears up the hascache, which I think it should.

I have no idea what consequences it would have to ignore hascache for the
pruning of orphaned paths, but I suspect, that behaviour is there for a
reason. I only made sure, that bat was not running when I used
.bvfs_clear_cache.

> 
> best regards
> -- 
> Radosław Korzeniewski
> rados...@korzeniewski.net 

Best regards,

  Stefan Münkner
-- 
this is an empty signature :-)


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs

2019-01-23 Thread Radosław Korzeniewski 
Hello,

wt., 22 sty 2019 o 14:39 Stefan Muenkner 
napisał(a):

> (...)
> Searching for hascache in the rest of the source I found that hascache in
> the job table seems to be only reset in ".bvfs_clear_cache".
>

Great!


> >> 
> >>   9) Check for orphaned Path records
> >> 
> >> Select function number: 9
> >> Pruning orphaned Path entries isn't possible when using BVFS.
> >> 
>

So, it seems the message above should be more descriptive and user friendly
and points to bvfs_clear_cache command.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs

2019-01-22 Thread Stefan Muenkner 
Hello Kern,

thanks for the answer. I'll do.  But for the time being I found a
workaround. Digging in the source, I found, that dbcheck only tests
whether any entry in the job table has "hascache" <>0. Even if there is
just one entry, dbcheck declines to celar up the orphaned path entries.
Searching for hascache in the rest of the source I found that hascache in
the job table seems to be only reset in ".bvfs_clear_cache".

I suspect that the occasionally crashing bat session left those lying
around. I manually called ".bvfs_clear_cache yes" in bconsole without bat
running and then dbcheck could clear up the orphaned path entries.

Best regards,

  Stefan

Am 20.01.19 um 13:51 schrieb Kern Sibbald:
> Hello Stefan,
> 
> Please submit a bug report on this.  When running dbcheck, it is my opinion
> that we should not be relying on .bvfs.  If this is true (as it seems from
> your
> output) then I must see why and make sure it is justified.  dbcheck should
> always
> be able to prune.
> 
> Best regards,
> Kern
> 
> On 1/17/19 11:13 AM, Stefan Muenkner wrote:
>> Hi,
>>
>> I have an almost 9 years running Bacula installation that accumulated more
>> than 800,000 orphaned paths entries in the database (around 5% of all
>> entries in the path table). dbcheck claims it cannot prune those when BVFS
>> is used.
>>
>> 
>>   9) Check for orphaned Path records
>> 
>> Select function number: 9
>> Pruning orphaned Path entries isn't possible when using BVFS.
>> 
>>
>> Is there anything I can do about it?
>>
>> Best regards
>>    Stefan
>>

-- 
this is an empty signature :-)


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs

2019-01-20 Thread Kern Sibbald 

Hello Stefan,

Please submit a bug report on this.  When running dbcheck, it is my opinion
that we should not be relying on .bvfs.  If this is true (as it seems 
from your
output) then I must see why and make sure it is justified.  dbcheck 
should always

be able to prune.

Best regards,
Kern

On 1/17/19 11:13 AM, Stefan Muenkner wrote:

Hi,

I have an almost 9 years running Bacula installation that accumulated more
than 800,000 orphaned paths entries in the database (around 5% of all
entries in the path table). dbcheck claims it cannot prune those when BVFS
is used.


  9) Check for orphaned Path records

Select function number: 9
Pruning orphaned Path entries isn't possible when using BVFS.


Is there anything I can do about it?

Best regards
   Stefan





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs

2019-01-17 Thread Stefan Muenkner 
Hi,

I have an almost 9 years running Bacula installation that accumulated more
than 800,000 orphaned paths entries in the database (around 5% of all
entries in the path table). dbcheck claims it cannot prune those when BVFS
is used.


 9) Check for orphaned Path records

Select function number: 9
Pruning orphaned Path entries isn't possible when using BVFS.


Is there anything I can do about it?

Best regards
  Stefan

-- 
this is an empty signature :-)


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users