Re: [Bacula-users] [Bacula-devel] IBAdmin

[Marcin Haba]

Hello Radoslaw,

On Thu, 24 Jan 2019 at 20:17, Radosław Korzeniewski
 wrote:
>
> Hello Dan,
>
> czw., 24 sty 2019 o 14:28 Dan Langille  napisał(a):
>>
>> On Jan 24, 2019, at 5:30 AM, Radosław Korzeniewski 
>>  wrote:
>>
>> Hello Dan,
>>
>> śr., 23 sty 2019 o 23:59 Dan Langille  napisał(a):
>>>
>>> On Jan 23, 2019, at 4:56 PM, Radosław Korzeniewski 
>>>  wrote:
>>>

 Does IBAdmin work with a readonly connection to the Catalog database?

>>>
>>> No.
>>>
>>>
>>> What features will be available with a read-only connection?
>>>
>>
>> Why do you need a read-only database?
>>
>>
>> Isolation.
>
>
> What do you want to isolate and from what?
>
>>
>> Bacula manipulates the Catalog. IBAdmin reports upon it.
>
>
> I do not understand it.
>
>>
>>  Why would it need to modify my Catalog?
>
>
> Could you be more descriptive what a Catalog is for you? I assume the Catalog 
> is the database structure used by Bacula Director to perform its operations. 
> In this case the only catalog manipulation (update) the IBAdmin is doing is 
> job and volume comment field update. All other write operations are performed 
> using IBAdmin specific objects structure which is not a part of the Catalog.
>
>>
>>
>> All management is based on database objects. When you want to create or 
>> change any configuration it will change/update database objects for you. 
>> When you log in your session is stored in a database. These are the most 
>> basic operations which require read-write access to database objects.
>>
>>
>> OK, but all of this must be in the Bacula Catalog?
>
>
> I assume the Bacula Catalog are all the objects used by Bacula Director to 
> perform its operations, so, no. IBAdmin does not store its configuration data 
> in Bacula Catalog but stores it in the same Postgresql database.
>
>>
>> If so, why? Why not put IBAdmin-specific data in a separate database?
>
>
> What is funny, when I designed IBAdmin I had a lot of opposite questions. :) 
> Why do you need a separate database for IBAdmin data when you already have 
> one? You will have to connect to it anyway. :) Do you think an average Bacula 
> user or even a newbie who have a serious problems to setup a basic Bacula 
> installation (just check bacula-users mailing list) will be happy when he has 
> to setup an another database just to use a simple IBAdmin when all other 
> Bacula GUIs do not require it?
> I think it is impossible to make everybody happy. :)
>
>>
>>
>> I assume you want to test it, right?
>>
>>
>> Yes, I do, I am interested, but not if I must give it write access to my 
>> Bacula database.
>
>
> Do you use any Bacula GUI except BAT? Do you know that all of them require  
> the same database write access as IBAdmin does.

For me explaining one incorrect thing by other incorrect things
doesn't lead to anything good. Something like "They do wrong so I can
do wrong as well".

Second thing, you wrote that all Bacula GUIs require database write
access. Later you wrote that almost all Bacula GUIs require database
write access. Could you explain me the difference in your sentences?

As Baculum and BWeb Enterprise developer I would like to say about the
write access to the Catalog in these web interfaces:

- Baculum - the one place where Baculum API requires do any
modification in the Catalog is dropping temporary Bvfs table that was
created by Bvfs during restore prepare. It is because I missed
.bvfs_cleanup command. Nevertheless this discussion encuouraged me to
fix it quickly.

- BWeb Enterprise - Bacula Systems provides BWeb as one from the
Bacula Enterprise Edition's products. It means that Bacula Systems is
responsible the same for Bacula Enterprise as for BWeb as for other
products. If they want to add (or modify) new SQL tables for some of
these components they can just do that.

Last thing, you wrote that in your opinion the only GUI which comes
from Bacula is BWeb Enterprise. You have right to believe what you
want however in this case it isn't "opinion" matter but facts. And the
fact is that Baculum comes from Bacula as well.

Best regards.
Marcin Haba (gani)

> From technical point of view it is possible to redesign IBAdmin to use 
> multiple databases, but it requires some development which (in my opinion) 
> has no real value. If you think this kind of separation has any real value 
> other then "do separation - it is good" I'll be happy if you will share it 
> with me.
>
> best regards
> --
> Radosław Korzeniewski
> rados...@korzeniewski.net
> ___
> Bacula-devel mailing list
> bacula-de...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-devel



-- 
"Greater love hath no man than this, that a man lay down his life for
his friends." Jesus Christ

"Większej miłości nikt nie ma nad tę, jak gdy kto życie swoje kładzie
za przyjaciół swoich." Jezus Chrystus


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://

Re: [Bacula-users] Lack Persistent Device Naming to use with Alert Command

[Nate K]

There are symlinks for the archive devices but not the tape drive control
devices which is what the tape alert commands need. I did already have it
set to use the symlink for the autochanger at least so that won't jump
around.

*> ls -l /dev/tape/by-id*











*total 0lrwxrwxrwx 1 root root  9 Jan 26 23:35 scsi-1ADIC_A0C0036905_LLA ->
../../sg8lrwxrwxrwx 1 root root  9 Jan 26 23:35 scsi-3500308c099903000 ->
../../st4lrwxrwxrwx 1 root root 10 Jan 26 23:35 scsi-3500308c099903000-nst
-> ../../nst4lrwxrwxrwx 1 root root  9 Jan 26 23:35 scsi-3500308c099903004
-> ../../st1lrwxrwxrwx 1 root root 10 Jan 26 23:35
scsi-3500308c099903004-nst -> ../../nst1lrwxrwxrwx 1 root root  9 Jan 26
23:35 scsi-3500308c099903094 -> ../../st3lrwxrwxrwx 1 root root 10 Jan 26
23:35 scsi-3500308c099903094-nst -> ../../nst3lrwxrwxrwx 1 root root  9 Jan
26 23:35 scsi-3500308c099903098 -> ../../st0lrwxrwxrwx 1 root root 10 Jan
26 23:35 scsi-3500308c099903098-nst -> ../../nst0lrwxrwxrwx 1 root root  9
Jan 26 23:35 scsi-3500308c09990309c -> ../../st2*
*lrwxrwxrwx 1 root root 10 Jan 26 23:35 scsi-3500308c09990309c-nst ->
../../nst2*

My only other idea is that maybe you could create a script that rewrites
the bacula-sd.conf every boot by parsing the output of this command as it
shows which sg* devices correspond to the st* ones:











*> lsscsi -g[0:0:0:0]cd/dvd  HL-DT-ST DVD-ROM GDR-T10N 1.02  /dev/sr0
/dev/sg0[2:0:0:0]tapeIBM  ULTRIUM-TD3  93GM  /dev/st0
/dev/sg1[4:0:0:0]tapeIBM  ULTRIUM-TD3  93GM  /dev/st1
/dev/sg2[5:0:0:0]tapeIBM  ULTRIUM-TD3  93GM  /dev/st4
/dev/sg6[5:0:0:1]mediumx ADIC Scalar i500  630G  /dev/sch0
/dev/sg8[6:0:0:0]tapeIBM  ULTRIUM-TD3  93GM  /dev/st2
/dev/sg3[7:0:0:0]tapeIBM  ULTRIUM-TD3  93GM  /dev/st3
/dev/sg4[9:0:0:0]diskATA  ST1000VN002-2EY1 SC60  /dev/sda
/dev/sg5[9:0:1:0]diskATA  ST1000VN002-2EY1 SC60  /dev/sdb
/dev/sg7*

Thanks,
Nate

On Sun, Jan 27, 2019 at 12:42 AM Adam Nielsen  wrote:

> > I'm using Ubuntu 18.04.1 with Bacula 9.0.6 from the main repo.  I'd like
> to
> > enable the above alerts from the sample bacula-sd.conf but can't because
> my
> > system is assigning random device names after each reboot so that
> > /dev/nst0, /dev/nst1, etc. can be any /dev/sg* number.  I looked into
> > creating a udev rule for the drives but there's no output from the
> control
> > device that can be uniquely identified (DEVPATH can't be used in a udev
> > rule I've found out):
>
> On my Arch Linux system, udev already has persistent rules for tapes
> in /dev/tape:
>
> $ ls /dev/tape/by-id/
> total 0
> drwxr-xr-x 2 root root 100 Dec 18 20:45 .
> drwxr-xr-x 4 root root  80 Dec 18 18:22 ..
> lrwxrwxrwx 1 root root   9 Dec 19 23:21 scsi-350014380125815e0 -> ../../sg6
> lrwxrwxrwx 1 root root   9 Dec 19 23:21 scsi-350014380125815e1 -> ../../st0
> lrwxrwxrwx 1 root root  10 Dec 19 23:21 scsi-350014380125815e1-nst ->
> ../../nst0
>
> sg6 is the autochanger on my system.
>
> If your system doesn't have these, and no update to udev is available,
> maybe you can copy the rules across from a newer version of udev[1]?
>
> > Is there another way do make this work?  There's a more recent firmware
> for
> > my tape library, a Dell ML6000, that I've held off installing.  Do you
> > think a new firmware might allow the sg driver to provide more output?
>
> Nope, this is all part of the udev rule system.
>
> Cheers,
> Adam.
>
> [1]:
> https://github.com/systemd/systemd/blob/master/rules/60-persistent-storage-tape.rules
>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Lack Persistent Device Naming to use with Alert Command

[Adam Nielsen]

> I'm using Ubuntu 18.04.1 with Bacula 9.0.6 from the main repo.  I'd like to
> enable the above alerts from the sample bacula-sd.conf but can't because my
> system is assigning random device names after each reboot so that
> /dev/nst0, /dev/nst1, etc. can be any /dev/sg* number.  I looked into
> creating a udev rule for the drives but there's no output from the control
> device that can be uniquely identified (DEVPATH can't be used in a udev
> rule I've found out):

On my Arch Linux system, udev already has persistent rules for tapes
in /dev/tape:

$ ls /dev/tape/by-id/
total 0
drwxr-xr-x 2 root root 100 Dec 18 20:45 .
drwxr-xr-x 4 root root  80 Dec 18 18:22 ..
lrwxrwxrwx 1 root root   9 Dec 19 23:21 scsi-350014380125815e0 -> ../../sg6
lrwxrwxrwx 1 root root   9 Dec 19 23:21 scsi-350014380125815e1 -> ../../st0
lrwxrwxrwx 1 root root  10 Dec 19 23:21 scsi-350014380125815e1-nst -> ../../nst0

sg6 is the autochanger on my system.

If your system doesn't have these, and no update to udev is available,
maybe you can copy the rules across from a newer version of udev[1]?

> Is there another way do make this work?  There's a more recent firmware for
> my tape library, a Dell ML6000, that I've held off installing.  Do you
> think a new firmware might allow the sg driver to provide more output?

Nope, this is all part of the udev rule system.

Cheers,
Adam.

[1]: 
https://github.com/systemd/systemd/blob/master/rules/60-persistent-storage-tape.rules


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Lack Persistent Device Naming to use with Alert Command

[Nate K]

*#  ##  # New alert command in Bacula 9.0.0#  #  Note: you must have the
sg3_utils (rpms) or the#  #sg3-utils (deb) installed on your
system.#  #and you must set the correct control device that#
#corresponds to the Archive Device#  Control Device = /dev/sg?? #
must be SCSI ctl for /dev/nst0#  Alert Command =
"/etc/bacula/scripts/tapealert %l"##  ##  # Enable the Alert command only
if you have the mtx package loaded#  # Note, apparently on some systems,
tapeinfo resets the SCSI controller#  #  thus if you turn this on, make
sure it does not reset your SCSI#  #  controller.  I have never had any
problems, and smartctl does#  #  not seem to cause such problems.#  ##
Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"#  If you have
smartctl, enable this, it has more info than tapeinfo#  Alert Command = "sh
-c 'smartctl -H -l error %c'"*
*#}*

I'm using Ubuntu 18.04.1 with Bacula 9.0.6 from the main repo.  I'd like to
enable the above alerts from the sample bacula-sd.conf but can't because my
system is assigning random device names after each reboot so that
/dev/nst0, /dev/nst1, etc. can be any /dev/sg* number.  I looked into
creating a udev rule for the drives but there's no output from the control
device that can be uniquely identified (DEVPATH can't be used in a udev
rule I've found out):









*> udevadm info -n /dev/sg1 --query=allP:
/devices/pci:00/:00:02.0/:05:00.0/:06:01.0/:09:00.0/:0a:01.0/:0b:00.0/host2/rport-2:0-0/target2:0:0/2:0:0:0/scsi_generic/sg1N:
sg1E: DEVNAME=/dev/sg1E:
DEVPATH=/devices/pci:00/:00:02.0/:05:00.0/:06:01.0/:09:00.0/:0a:01.0/:0b:00.0/host2/rport-2:0-0/target2:0:0/2:0:0:0/scsi_generic/sg1E:
MAJOR=21E: MINOR=1E: SUBSYSTEM=scsi_generic*

Is there another way do make this work?  There's a more recent firmware for
my tape library, a Dell ML6000, that I've held off installing.  Do you
think a new firmware might allow the sg driver to provide more output?
Should I quit worrying about getting alerts working since the tape library
itself has a similar capability or is it important for Bacula to function
correctly?

Thanks,
Nate
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [External] Re: [Bacula-devel] IBAdmin

[Dmitri Maziuk via Bacula-users]

On Sat, 26 Jan 2019 14:27:10 +0100
Kern Sibbald  wrote:

> On 1/25/19 7:06 PM, Josip Deanovic wrote:

> > API would be really cool thing to have.
> 
> Yes, I will check if we have some documentation on it.

FWIW I don't think a) having strictly read-only access to the catalog is
a bad thing and b) you can stop someone with a little knowledge of psql
from getting in. I would start with a read-only role created by install
script (easy in postgres 9+) and detailed documentation of the database
schema.

It's placing your own objects in the catalog schema that I have a
serious problem with.

APIs are work and IME everyone wants something just different enough
that you have to add more API calls, and more, until you end up looking
like matlab.

-- 
Dmitri Maziuk 


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [External] Re: [Bacula-devel] IBAdmin

[Heitor Faria]

Hello Kern, All,

>> API would be really cool thing to have.
> 
> Yes, I will check if we have some documentation on it.

Let me diverge the thread a little.
Now it was mentioned, I think it would be really good too have an API command 
that allows exporting terminated Copy Jobs metadata from one Dir+Catalog to a 
remote Dir+Catalog. Or something similar.
Or the hard way: a Destination Catalog Copy Jobs directive.
This is a common request from our customers and prospects depending on the 
proposed architectures.

Regards,
-- 
MSc Heitor Faria 
CEO Bacula LATAM 
mobile1: + 1 909 655-8971 
mobile2: + 55 61 98268-4220 

Brasilia - Miami - New York 

www.bacula.com.br


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs

[Kern Sibbald]

  
  
Hello Radek,

The code has been there since 2011.  I now have a patch that makes
the error message
very clear.

Yes, I am aware how bvfs works and why the cache is there.  I was
just surprised that
the existence of the cache would disable a dbcheck function.

As far as I can see, this is not a temporary situation.

Note, in the community version some of the bvfs functions may not be
implemented or may only
be implemented simply because the catalog database schemas between
the Enterprise version and
the community version are different simply because I did not have
the time to backport those changes
because the backport would be very tricky.  I will remove these
database differences in the next major 
release, and then the Community will have access to all these bvfs
functions.   

I haven't confirmed it, but from what I can see, the error message
is to avoid having a search
cache that is out of date with the actual state of the path table --
so the solution is to clear the
path cache table, and the new message explains exactly how to do it.

Best regards,
Kern

On 1/26/19 8:01 AM, Radosław
  Korzeniewski wrote:


  
  

  

  Hello Kern,
  
  
pt., 25 sty 2019 o
  17:06 Kern Sibbald 
  napisał(a):


   Hello,

I find the error message very cryptic, and I really
don't like just giving up, so I have asked the
author
of that particular code to supply a fix.
  



From what I understand here bvfs is using a special
  cache tables (pathvisibility, pathhierarchy) for fast
  job files browsing. These cache tables have a
  references to 'path' and job tables but these
  references aren't guarded by a database constraints,
  so I call it a soft references (I assume it is for
  faster update and search). 



  bacula=# \d
  pathvisibility
  Table
  "public.pathvisibility"
   Column | 
  Type   | Modifiers
  +-+---
   pathid |
  integer | not null
   jobid  |
  integer | not null
   size   |
  bigint  | default 0
   files  |
  integer | default 0
  Indexes:
     
  "pathvisibility_pkey" PRIMARY KEY, btree (jobid,
  pathid)
     
  "pathvisibility_jobid" btree (jobid)
  

  bacula=# \d
  pathhierarchy
   Table
  "public.pathhierarchy"
   Column  | 
  Type   | Modifiers
  -+-+---
   pathid  |
  integer | not null
   ppathid |
  integer | not null
  Indexes:
     
  "pathhierarchy_pkey" PRIMARY KEY, btree (pathid)
     
  "pathhierarchy_ppathid" btree (ppathid)



The dbcheck utility has no code to deal with bvfs
  cache tables, so when it finds these soft references
  it actually raise an error. I can imagine that this is
  a temporary solution.


best regards
 

   
Best regards,
Kern

On
  1/23/19 11:03 AM, Radosław Korzeniewski wrote:


  
Hello,


  wt.,
22 sty 2019 o 14:39 Stefan Muenkner 
napisał(a):
  
  (...)
Searching for hascache in the rest of the
source I found that hascache in
the job table seems to be only reset in
".bvfs_clear_c

Re: [Bacula-users] [External] Re: [Bacula-devel] IBAdmin

[Kern Sibbald]

On 1/25/19 7:06 PM, Josip Deanovic wrote:

On Friday 2019-01-25 16:47:55 Kern Sibbald wrote:

Hello guys,

Interesting conversation.  I thought I would throw in some general
comments of my own.

[...]

API would be really cool thing to have.


Yes, I will check if we have some documentation on it.



Don't forget to support some statistical and health information
that could be used by different graphing and monitoring systems.


This (statistics info) exists in the Enterprise version, and will also 
be back ported to the

community version at future time (hopefully by mid-year).

Regards,
Kern




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users