Hello,
A word about 1.38.0:
Release 1.38.0 is evolving much as prior releases, that is to say, once people
started to use it, we received a number of bug reports, some rather
important. All the critical, serious, and important bugs have been fixed and
if you want a copy, you can pull it from the Bacula CVS using branch
Branch-1_38_0. If all continues normally, I'll release a version 1.38.1
this weekend which will contain fixes for all the currently known bugs.
Concerning future development of Bacula:
As I noted in the 1.38 ReleaseNotes, version 1.38 was different from prior
versions because it had a lot more contributions. I expect that this trend
will continue. As a consequence, I am going to modify how I normally do
development, and instead of making a list of all the features that I will
implement in the next version, I will personally sign up for one (maybe two)
projects at a time, and when they are complete, I will release a new version.
The difference is that I will have more time to review the new code that is
being contributed, and will be able to devote more time to a smaller number
of projects (1.38 had too many new features for me to handle correctly).
I expect that future release schedules will be much the same, and the number
of new features will also be much the same providing that the contributions
continue to come -- and they show no signs of let up :-)
Feature requests -- RFCs:
In addition, I would like to "formalize" the feature requests (RFC) a bit.
Instead of me maintaining an informal list of everything I run into
(kernstodo), I would like to maintain a "formal" list of projects. This
means that all new feature requests, including those recently discussed on
the email lists, must be formally submitted and approved.
Formal submission of feature requests will take two forms: 1. non-mandatory,
but highly recommended is to discuss proposed new features on the mailing
list. 2. Formal submission of an RFC in a special format. I'll give an
example of this below, but you can also find it on the web site under
"Support -> Feature Requests". Since it takes a bit of time to properly fill
out a RFC form, you probably should check on the email list first.
Once I receive the RFC, I will either accept it, send it back asking for
clarification, send it to the email list asking for opinions, or reject it.
If it is accepted, it will go in the "projects" file (a simple ASCII file)
maintained in the main Bacula source directory.
Implementation of RFCs:
Any qualified developer can sign up for a project. The project must have an
entry in the projects file, and the developer's name will appear in the
Status field.
How RFCs are accepted:
Acceptance of RFCs depends on several things: 1. feedback from users. If it is
negative, the RFC will probably not be accepted. 2. the difficulty of the
project. A project that is so difficult that I cannot imagine finding someone
to implement probably won't be accepted. 3. whether or not the RFC fits
within the current stategy of Bacula (for example an RFC that requests
changing the tape to tar format would not be accepted, ...)
How RFCs are prioritized:
Once an RFC is accepted, it needs to be implemented. If you can find a
developer for it, or one signs up for implementing it, then the RFC becomes
top priority (at least for that developer).
Between releases of Bacula, I will generally solicit RFC input for the next
version, and by way of this email, I suggest that you send discuss and send
in your RFCs for the next release. Please verify that the RFC is not in the
current list (attached to this email).
Once users have had several weeks to submit RFCs, I will organize them, and
request users to vote on them. This will allow fixing prioritizing the RFCs.
Having a priority is one thing, but getting it implement is another thing --
I am hoping that the Bacula community will take more responsibility for
assuring the implementation of accepted RFCs.
RFC format:
============= Empty RFC form ===========
Item n: One line summary ...
Date: Date submitted
Origin: Name and email of originator.
Status:
What: More detailed explanation ...
Why: Why it is important ...
Notes: Additional notes or features (omit if not used)
============== End RFC form ==============
============= Example Completed RFC form ===========
Item 1: Implement a Migration job type that will move the job
data from one device to another.
Origin: Sponsored by Riege Sofware International GmbH. Contact:
Daniel Holtkamp <holtkamp at riege dot com>
Date: 28 October 2005
Status: Partially coded in 1.37 -- much more to do. Assigned to
Kern.
What: The ability to copy, move, or archive data that is on a
device to another device is very important.
Why: An ISP might want to backup to disk, but after 30 days
migrate the data to tape backup and delete it from
disk. Bacula should be able to handle this
automatically. It needs to know what was put where,
and when, and what to migrate -- it is a bit like
retention periods. Doing so would allow space to be
freed up for current backups while maintaining older
data on tape drives.
Notes: Migration could be triggered by:
Number of Jobs
Number of Volumes
Age of Jobs
Highwater size (keep total size)
Lowwater mark
=================================================
By the end of November, I would like to have all the RFCs for the projects
file, and have requested users to vote on them. As a consequence, please send
your RFCs before 21 November, after which we will vote and prioritize
them ...
This email is already way too long, so I will explain in a subsequent email
(tomorrow) why I have signed up for project Item 1 -- see the attached
projects file.
Your comments on this would be welcome -- as well as your RFCs.
--
Best regards,
Kern
(">
/\
V_V
Projects:
Bacula Projects Roadmap
08 November 2005
Below, you will find more information on future projects:
Item 1: Implement a Migration job type that will move the job
data from one device to another.
Origin: Sponsored by Riege Sofware International GmbH. Contact:
Daniel Holtkamp <holtkamp at riege dot com>
Date: 28 October 2005
Status: Partially coded in 1.37 -- much more to do. Assigned to
Kern.
What: The ability to copy, move, or archive data that is on a
device to another device is very important.
Why: An ISP might want to backup to disk, but after 30 days
migrate the data to tape backup and delete it from
disk. Bacula should be able to handle this
automatically. It needs to know what was put where,
and when, and what to migrate -- it is a bit like
retention periods. Doing so would allow space to be
freed up for current backups while maintaining older
data on tape drives.
Notes: Migration could be triggered by:
Number of Jobs
Number of Volumes
Age of Jobs
Highwater size (keep total size)
Lowwater mark
Item 2: Implement extraction of Win32 BackupWrite data.
Origin: Thorsten Engel <thorsten.engel at matrix-computer dot com>
Date: 28 October 2005
Status: Assigned to Thorsten. Implemented in current CVS
What: This provides the Bacula File daemon with code that
can pick apart the stream output that Microsoft writes
for BackupWrite data, and thus the data can be read
and restored on non-Win32 machines.
Why: BackupWrite data is the portable=no option in Win32
FileSets, and in previous Baculas, this data could
only be extracted using a Win32 FD. With this new code,
the Windows data can be extracted and restored on
any OS.
Item 3: Implement a Bacula GUI/management tool using Python
and Qt.
Origin: Kern
Date: 28 October 2005
Status:
What: Implement a Bacula console, and management tools
using Python and Qt.
Why: Don't we already have a wxWidgets GUI? Yes, but
it is written in C++ and changes to the user interface
must be hand tailored using C++ code. By developing
the user interface using Qt designer, the interface
can be very easily updated and most of the new Python
code will be automatically created. The user interface
changes become very simple, and only the new features
must be implement. In addition, the code will be in
Python, which will give many more users easy (or easier)
access to making additions or modifications.
Item 4: Implement a Python interface to the Bacula catalog.
Date: 28 October 2005
Origin: Kern
Status:
What: Implement an interface for Python scripts to access
the catalog through Bacula.
Why: This will permit users to customize Bacula through
Python scripts.
Item 5: Implement more Python events in Bacula.
Date: 28 October 2005
Origin:
Status:
What: Allow Python scripts to be called at more places
within Bacula and provide additional access to Bacula
internal variables.
Why: This will permit users to customize Bacula through
Python scripts.
Notes: Recycle event
Scratch pool event
NeedVolume event
Item 6: Implement Base jobs.
Date: 28 October 2005
Origin: Kern
Status:
What: A base job is sort of like a Full save except that you
will want the FileSet to contain only files that are
unlikely to change in the future (i.e. a snapshot of
most of your system after installing it). After the
base job has been run, when you are doing a Full save,
you specify one or more Base jobs to be used. All
files that have been backed up in the Base job/jobs but
not modified will then be excluded from the backup.
During a restore, the Base jobs will be automatically
pulled in where necessary.
Why: This is something none of the competition does, as far as
we know (except perhpas BackupPC, which is a Perl program that
saves to disk only). It is big win for the user, it
makes Bacula stand out as offering a unique
optimization that immediately saves time and money.
Basically, imagine that you have 100 nearly identical
Windows or Linux machine containing the OS and user
files. Now for the OS part, a Base job will be backed
up once, and rather than making 100 copies of the OS,
there will be only one. If one or more of the systems
have some files updated, no problem, they will be
automatically restored.
Notes: Huge savings in tape usage even for a single machine.
Will require more resources because the DIR must send
FD a list of files/attribs, and the FD must search the
list and compare it for each file to be saved.
Item 7: Add Plug-ins to the FileSet Include statements.
Date: 28 October 2005
Origin:
Status: Partially coded in 1.37 -- much more to do.
What: Allow users to specify wild-card and/or regular
expressions to be matched in both the Include and
Exclude directives in a FileSet. At the same time,
allow users to define plug-ins to be called (based on
regular expression/wild-card matching).
Why: This would give the users the ultimate ability to control
how files are backed up/restored. A user could write a
plug-in knows how to backup his Oracle database without
stopping/starting it, for example.
Item 8: Implement huge exclude list support using hashing.
Date: 28 October 2005
Origin: Kern
Status:
What: Allow users to specify very large exclude list (currently
more than about 1000 files is too many).
Why: This would give the users the ability to exclude all
files that are loaded with the OS (e.g. using rpms
or debs). If the user can restore the base OS from
CDs, there is no need to backup all those files. A
complete restore would be to restore the base OS, then
do a Bacula restore. By excluding the base OS files, the
backup set will be *much* smaller.
Item 9: Implement data encryption (as opposed to communications
encryption)
Date: 28 October 2005
Origin: Sponsored by Landon and 13 contributors to EFF.
Status: Landon Fuller is currently implementing this.
What: Currently the data that is stored on the Volume is not
encrypted. For confidentiality, encryption of data at
the File daemon level is essential.
Data encryption encrypts the data in the File daemon and
decrypts the data in the File daemon during a restore.
Why: Large sites require this.
Item 10: Permit multiple Media Types in an Autochanger
Origin:
Status:
What: Modify the Storage daemon so that multiple Media Types
can be specified in an autochanger. This would be somewhat
of a simplistic implementation in that each drive would
still be allowed to have only one Media Type. However,
the Storage daemon will ensure that only a drive with
the Media Type that matches what the Director specifies
is chosen.
Why: This will permit user with several different drive types
to make full use of their autochangers.
Item 11: Allow two different autochanger definitions that refer
to the same autochanger.
Date: 28 October 2005
Origin: Kern
Status:
What: Currently, the autochanger script is locked based on
the autochanger. That is, if multiple drives are being
simultaneously used, the Storage daemon ensures that only
one drive at a time can access the mtx-changer script.
This change would base the locking on the control device,
rather than the autochanger. It would then permit two autochanger
definitions for the same autochanger, but with different
drives. Logically, the autochanger could then be "partitioned"
for different jobs, clients, or class of jobs, and if the locking
is based on the control device (e.g. /dev/sg0) the mtx-changer
script will be locked appropriately.
Why: This will permit users to partition autochangers for specific
use. It would also permit implementation of multiple Media
Types with no changes to the Storage daemon.
Item 12: Implement red/black binary tree routines.
Date: 28 October 2005
Origin: Kern
Status:
What: Implement a red/black binary tree class. This could
then replace the current binary insert/search routines
used in the restore in memory tree. This could significantly
speed up the creation of the in memory restore tree.
Why: Performance enhancement.
============= Empty RFC form ===========
Item n: One line summary ...
Date: Date submitted
Origin: Name and email of originator.
Status:
What: More detailed explanation ...
Why: Why it is important ...
Notes: Additional notes or features (omit if not used)
============== End RFC form ==============
Items completed for release 1.38.0:
#4 Embedded Python Scripting (implemented in all Daemons)
#5 Events that call a Python program (Implemented in all
daemons, but more cleanup work to be done).
#6 Select one from among Multiple Storage Devices for Job.
This is already implemented in 1.37.
#7 Single Job Writing to Multiple Storage Devices. This is
currently implemented with a Clone feature.
#- Full multiple drive Autochanger support (done in 1.37)
#- Built in support for communications encryption (TLS)
done by Landon Fuller.
# Support for Unicode characters
(via UTF-8) on Win32 machines thanks to Thorsten Engel.
Item 8: Break the one-to-one Relationship between a Job and a
Specific Storage Device (or Devices if #10 is implemented).
Completed items from last year's list:
Item 1: Multiple simultaneous Jobs. (done)
Item 3: Write the bscan program -- also write a bcopy program (done).
Item 5: Implement Label templates (done).
Item 6: Write a regression script (done)
Item 9: Add SSL to daemon communications (done by Landon Fuller)
Item 10: Define definitive tape format (done)
Item 3: GUI for interactive restore. Partially Implemented in 1.34
Note, there is now a complete Webmin plugin, a partial
GNOME console, and an excellent wx-console GUI.
Item 4: GUI for interactive backup
Item 2: Job Data Spooling.
Done: Regular expression matching.
Item 10: New daemon communication protocol (this has been dropped).
