Re: [Bacula-users] bacula - optimize storage for cloud sync
Hello Kern,
Thank you for all info. I needed to install cloud plugin (yum install
bacula-cloud-storage) in order to use this configuration. Relevant
part of bacula-sd.conf is pasted bellow. Files did get uploaded to
Oracle object storage.
I intend to use lifecycle policy on the cloud bucket. It will archive
60days old files to a 10times cheaper storage. For bacula that means
that some fileparts will not be available until I manually restore
them (after 60 days status will change from available to archived).
Do you expect that this will represent any kind of problem for backup
job? Perhaps fileparts of the last full backup need to be available at
local folder in order to run incremental backup?
What about restore procedure (eg. from 90 days old incremental
backup)? Will bacula notify me which fileparts are missing at local
folder in order to complete the restore?
I will continue with my tests and hopefully move to production in a
month or two. I intend to publish on the bacula-users list, if
everything worked for me. If I forget, do not hesitate to ask me about
that.
Kind regards,
Ziga
Relevant part of bacula-sd.conf:
Device {
Name = FSOciCloudStandard
Device type = Cloud
Cloud = OracleViaStorageGateway
# Maximum Part Size = 100 MB
Media Type = File1
Archive Device = /mnt/backup_bacula/backup
LabelMedia = yes; # lets Bacula label unlabeled media
Random Access = Yes;
AutomaticMount = yes; # when device opened, read it
RemovableMedia = no;
AlwaysOpen = no;
}
Cloud {
Name = OracleViaStorageGateway
Driver = "File"
HostName = "/mnt/baculatest_standard/backup"
BucketName = "DummyBucket"
AccessKey = "DummyAccessKey"
SecretKey = "DummySecretKey"
Protocol = HTTPS
UriStyle = VirtualHost
}
On 08/07/2020 16:32, Kern Sibbald wrote:
Hello Ziga,
Yes, you might be able to do what you want using a "debug" feature of
the Bacula Cloud driver. It is not very well documented, but there
is one section "3 File Driver for the Cloud" in the "Bacula Cloud
Backup" that mentions it.
Basically instead of using the "S3" driver in the Cloud resource of
your Storage Daemon, you use "File" and the HostName becomes the path
where the Cloud volumes (directories + parts) will be written. For
example, I use the following for writing to disk instead of an S3 cloud.
Cloud {
Name = DummyCloud
Driver = "File"
HostName = "/home/kern/bacula/k/regress/tmp/cloud"
BucketName = "DummyBucket"
AccessKey = "DummyAccessKey"
SecretKey = "DummySecretKey"
Protocol = HTTPS
UriStyle = VirtualHost
}
The Device resource looks like:
Device {
Name = FileStorage1
Media Type = File1
Archive Device = /home/kern/bacula/k/regress/tmp
LabelMedia = yes; # lets Bacula label unlabelled
media
Random Access = Yes;
AutomaticMount = yes; # when device opened, read it
RemovableMedia = no;
AlwaysOpen = no;
Device Type = Cloud
Cloud = DummyCloud
}
I know the code runs and produces correct output, but I am not sure
how it will work in your environment. If it works, great. If it
doesn't work, for the near future, unfortunately I cannot provide
support, but at some point (probably 3-6 months) the project may
support this feature.
Note: the next version of Bacula coming in a few months will have a
good number of new features and improvements to the Cloud driver
(more bconsole commands for example).
Good luck, and best regards,
Kern
PS: If it does work for you, I would appreciate it if you would
document it and publish it on the bacula-users list so others can use
the Oracle cloud.
On 7/8/20 3:41 PM, Žiga Žvan wrote:
Hi Mr. Kern,
My question was a bit different. I have noticed that Oracle S3 is
not compatible, therefore I have implemented Oracle Storage gateway
(a docker image that uses local filesystem as a cache and moves the
data automatically to oracle cloud). I have this filesystem mounted
(nfsv4) on bacula server and I am able to backup data to this
storage (and hence in cloud).
I have around 1 TB data daily and I'm a bit concerned about the
bandwidth. It will take app. 4 hours to sync to the cloud and I need
to count in the future growth. As long as bacula writes data to one
file/volume, where it stores full and incremental backups, this is
not optimal for the cloud (the file will change and all the data
will upload each day). I have noticed that bacula stores data
differently in the cloud configuration. Volume is not a file, but a
folder with fileparts. This would be better for me, because only
some fileparts would change and move to the cloud via Storage
gateway. So the question is:
Can I configure bacula-sd to store data in fileparts, without actual
cloud sync? Is this possible? I have tried several configurations of
a bacula-sd device with no luck. Should I configure some dummy
cloud resource?
Kind regards,
Ziga Zvan
On 07/07/2020 14:40,
Re: [Bacula-users] bacula - optimize storage for cloud sync
Hello Ziga,
Yes, you might be able to do what you want using a "debug"
feature of the Bacula Cloud driver. It is not very well
documented, but there is one section "3 File Driver for the Cloud"
in the "Bacula Cloud Backup" that mentions it.
Basically instead of using the "S3" driver in the Cloud resource
of your Storage Daemon, you use "File" and the HostName becomes
the path where the Cloud volumes (directories + parts) will be
written. For example, I use the following for writing to disk
instead of an S3 cloud.
Cloud {
Name = DummyCloud
Driver = "File"
HostName = "/home/kern/bacula/k/regress/tmp/cloud"
BucketName = "DummyBucket"
AccessKey = "DummyAccessKey"
SecretKey = "DummySecretKey"
Protocol = HTTPS
UriStyle = VirtualHost
}
The Device resource looks like:
Device {
Name = FileStorage1
Media Type = File1
Archive Device = /home/kern/bacula/k/regress/tmp
LabelMedia = yes; # lets Bacula label
unlabelled media
Random Access = Yes;
AutomaticMount = yes; # when device opened, read
it
RemovableMedia = no;
AlwaysOpen = no;
Device Type = Cloud
Cloud = DummyCloud
}
I know the code runs and produces correct output, but I am not
sure how it will work in your environment. If it works, great.
If it doesn't work, for the near future, unfortunately I cannot
provide support, but at some point (probably 3-6 months) the
project may support this feature.
Note: the next version of Bacula coming in a few months will have
a good number of new features and improvements to the Cloud driver
(more bconsole commands for example).
Good luck, and best regards,
Kern
PS: If it does work for you, I would appreciate it if you would
document it and publish it on the bacula-users list so others can
use the Oracle cloud.
On 7/8/20 3:41 PM, Žiga Žvan wrote:
Hi Mr. Kern,
My question was a bit different. I have noticed that Oracle S3
is not compatible, therefore I have implemented Oracle Storage
gateway (a docker image that uses local filesystem as a cache
and moves the data automatically to oracle cloud). I have this
filesystem mounted (nfsv4) on bacula server and I am able to
backup data to this storage (and hence in cloud).
I have around 1 TB data daily and I'm a bit concerned about the
bandwidth. It will take app. 4 hours to sync to the cloud and I
need to count in the future growth. As long as bacula writes
data to one file/volume, where it stores full and incremental
backups, this is not optimal for the cloud (the file will change
and all the data will upload each day). I have noticed that
bacula stores data differently in the cloud configuration.
Volume is not a file, but a folder with fileparts. This would be
better for me, because only some fileparts would change and move
to the cloud via Storage gateway. So the question is:
Can I configure bacula-sd to store data in fileparts, without
actual cloud sync? Is this possible? I have tried several
configurations of a bacula-sd device with no luck. Should I
configure some dummy cloud resource?
Kind regards,
Ziga Zvan
On 07/07/2020 14:40, Kern Sibbald
wrote:
Hello,
Oracle S3 is not compatible with Amazon S3 or at least with
the libs3 that we use to interface to AWS and other compatible
S3 cloud offerings.
Yes, Bacula Enterprise has a separate Oracle cloud driver
that they wrote. There are no plans at the moment to backport
it to the community version.
Best regards,
Kern
On 7/7/20 8:43 AM, Žiga Žvan wrote:
Dear all,
I'm testing communty version of bacula in order to
change backup sw for app. 100 virtual and physical
hosts. I would like to move all the data to local
storage and then move them to public cloud (Oracle
Object storage).
I believe that community version of the software suites
our needs. I have installed:
-version 9.6.5 of bacula on centos 7 computer
-oracle storage gateway (similar to aws SG - it moves
data to object storage and exposes it localy as nfsv4;
for bacula this is backup destination).
I have read this two documents regarding bacula and
Re: [Bacula-users] bacula - optimize storage for cloud sync
Hi Mr. Kern,
My question was a bit different. I have noticed that Oracle S3 is not
compatible, therefore I have implemented Oracle Storage gateway (a
docker image that uses local filesystem as a cache and moves the data
automatically to oracle cloud). I have this filesystem mounted (nfsv4)
on bacula server and I am able to backup data to this storage (and hence
in cloud).
I have around 1 TB data daily and I'm a bit concerned about the
bandwidth. It will take app. 4 hours to sync to the cloud and I need to
count in the future growth. As long as bacula writes data to one
file/volume, where it stores full and incremental backups, this is not
optimal for the cloud (the file will change and all the data will upload
each day). I have noticed that bacula stores data differently in the
cloud configuration. Volume is not a file, but a folder with fileparts.
This would be better for me, because only some fileparts would change
and move to the cloud via Storage gateway. So the question is:
Can I configure bacula-sd to store data in fileparts, without actual
cloud sync? Is this possible? I have tried several configurations of a
bacula-sd device with no luck. Should I configure some dummy cloud
resource?
Kind regards,
Ziga Zvan
On 07/07/2020 14:40, Kern Sibbald wrote:
Hello,
Oracle S3 is not compatible with Amazon S3 or at least with the libs3
that we use to interface to AWS and other compatible S3 cloud offerings.
Yes, Bacula Enterprise has a separate Oracle cloud driver that they
wrote. There are no plans at the moment to backport it to the
community version.
Best regards,
Kern
On 7/7/20 8:43 AM, Žiga Žvan wrote:
Dear all,
I'm testing communty version of bacula in order to change backup sw
for app. 100 virtual and physical hosts. I would like to move all
the data to local storage and then move them to public cloud (Oracle
Object storage).
I believe that community version of the software suites our needs. I
have installed:
-version 9.6.5 of bacula on centos 7 computer
-oracle storage gateway (similar to aws SG - it moves data to object
storage and exposes it localy as nfsv4; for bacula this is backup
destination).
I have read this two documents regarding bacula and cloud
https://blog.bacula.org/whitepapers/CloudBackup.pdf
https://blog.bacula.org/whitepapers/ObjectStorage.pdf
It is mentioned it the document above, that Oracle Object storage is
not supported at the moment.
Is it possible to *configure* bacula Storage device in a way that
uses *Cloud media format* (directory with file parts as a volume,
instead of a single file as a volume) *without actual cloud sync*
(Storage Gateway does this in my case)? I am experimenting with
variations of the definition bellow, but I am unable to solve this
issue for now (it tries to initialize cloud plugin or it writes to a
file, instead of a directory).
Device {
Name = FSOciCloudStandard
# Device type = Cloud
Device type = File
# Cloud = OracleViaStorageGateway
Maximum Part Size = 100 MB
# Media Type = File
Media Type = CloudType
Archive Device = /mnt/baculatest_standard/backup
LabelMedia = yes; # lets Bacula label unlabeled
media
Random Access = Yes;
AutomaticMount = yes; # when device opened, read it
RemovableMedia = no;
AlwaysOpen = no;
}
Is there any plan to support oracle object storage in near future? It
has S3 compatible API and bacula enterprise supports it...
Kind regards,
Ziga Zvan
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula - optimize storage for cloud sync
Hello,
Oracle S3 is not compatible with Amazon S3 or at least with the
libs3 that we use to interface to AWS and other compatible S3
cloud offerings.
Yes, Bacula Enterprise has a separate Oracle cloud driver that
they wrote. There are no plans at the moment to backport it to
the community version.
Best regards,
Kern
On 7/7/20 8:43 AM, Žiga Žvan wrote:
Dear all,
I'm testing communty version of bacula in order to change
backup sw for app. 100 virtual and physical hosts. I would
like to move all the data to local storage and then move
them to public cloud (Oracle Object storage).
I believe that community version of the software suites our
needs. I have installed:
-version 9.6.5 of bacula on centos 7 computer
-oracle storage gateway (similar to aws SG - it moves data
to object storage and exposes it localy as nfsv4; for bacula
this is backup destination).
I have read this two documents regarding bacula and cloud
https://blog.bacula.org/whitepapers/CloudBackup.pdf
https://blog.bacula.org/whitepapers/ObjectStorage.pdf
It is mentioned it the document above, that Oracle Object
storage is not supported at the moment.
Is it possible to configure bacula Storage device in
a way that uses Cloud media format (directory with
file parts as a volume, instead of a single file as a
volume) without actual cloud sync (Storage Gateway
does this in my case)? I am experimenting with variations of
the definition bellow, but I am unable to solve this issue
for now (it tries to initialize cloud plugin or it writes to
a file, instead of a directory).
Device {
Name = FSOciCloudStandard
# Device type = Cloud
Device type = File
# Cloud = OracleViaStorageGateway
Maximum Part Size = 100 MB
# Media Type = File
Media Type = CloudType
Archive Device = /mnt/baculatest_standard/backup
LabelMedia = yes; # lets Bacula label
unlabeled media
Random Access = Yes;
AutomaticMount = yes; # when device opened,
read it
RemovableMedia = no;
AlwaysOpen = no;
}
Is there any plan to support oracle object storage in near
future? It has S3 compatible API and bacula enterprise supports
it...
Kind regards,
Ziga Zvan
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] bacula - optimize storage for cloud sync
Dear all,
I'm testing communty version of bacula in order to change backup sw
for app. 100 virtual and physical hosts. I would like to move all the
data to local storage and then move them to public cloud (Oracle
Object storage).
I believe that community version of the software suites our needs. I
have installed:
-version 9.6.5 of bacula on centos 7 computer
-oracle storage gateway (similar to aws SG - it moves data to object
storage and exposes it localy as nfsv4; for bacula this is backup
destination).
I have read this two documents regarding bacula and cloud
https://blog.bacula.org/whitepapers/CloudBackup.pdf
https://blog.bacula.org/whitepapers/ObjectStorage.pdf
It is mentioned it the document above, that Oracle Object storage is
not supported at the moment.
Is it possible to *configure* bacula Storage device in a way that uses
*Cloud media format* (directory with file parts as a volume, instead
of a single file as a volume) *without actual cloud sync* (Storage
Gateway does this in my case)? I am experimenting with variations of
the definition bellow, but I am unable to solve this issue for now (it
tries to initialize cloud plugin or it writes to a file, instead of a
directory).
Device {
Name = FSOciCloudStandard
# Device type = Cloud
Device type = File
# Cloud = OracleViaStorageGateway
Maximum Part Size = 100 MB
# Media Type = File
Media Type = CloudType
Archive Device = /mnt/baculatest_standard/backup
LabelMedia = yes; # lets Bacula label unlabeled media
Random Access = Yes;
AutomaticMount = yes; # when device opened, read it
RemovableMedia = no;
AlwaysOpen = no;
}
Is there any plan to support oracle object storage in near future? It
has S3 compatible API and bacula enterprise supports it...
Kind regards,
Ziga Zvan
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
