This case was approved at today's PSARC meeting. I marked the IAM file
as "closed approved fast-track 04/28/2010".
Brian
On 04/13/10 05:53 PM, Brian Cameron wrote:
Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI
This information is Copyright (c) 2010, Oracle and/or its affiliates. All
rights reserved.
1. Introduction
1.1. Project/Component Working Name:
Time Slider Phase 2 (external backup)
1.2. Name of Document Author/Supplier:
Author: Brian Cameron
1.3 Date of This Document:
13 April, 2010
4. Technical Description
1. Introduction
1.1. Project/Component Working Name:
Time Slider Phase 2 (external backup)
1.2. Name of Document Author/Supplier:
Author: Erwann Chenede, Niall Power
1.3 Date of This Document:
Apr 8, 2010
4. Technical Description
Time slider's primary purpose is to enable automatic ZFS snapshots to be
taken for the user.
This project's goal is to provide an external backup solution on top of
Time Slider's local ZFS snapshot feature.
Within the time slider setup dialog the user will only need to specify a
path to an external filesystem where ZFS filesystem snapshots will be
replicated using rsync and the filesystems the user wishes to replicate.
This project makes the following high level changes :
- auto-snapshot:
Functional components responsible for scheduling, creation, deletion
and backup of snapshots are to be replaced by a more modular
scheduler provided by time slider. All interfaces are backward
compatible or replaced. These interfaces are currently uncommitted.
What remains of auto-snapshot are its SMF service instances which
define automatic snapshot schedules.
- time-slider:
Replacement of the functional components removed from auto-snapshot
via a snapshot daemon managed by the time-slider SMF service and
introduction of a plugin mechanism to allow various types of
time based backup strategies to be implemented.
4.1 Details:
4.1.1 Auto Snapshot :
auto-snapshot as previously described in (LSARC 2008/571),
consists of several SMF service instances (one per schedule: hourly,
daily, weekly, monthly& frequent by default).
A SMF service method reads the configuration properties for a given
schedule and creates a cron job that matches the schedule parameters
for each instance.
The cron job command is the auto-snapshot service method script
invoked with arguments that cause it to execute in a snapshot
creation operational mode. Additionally, the auto-snapshot SMF
service provides a project private hook to allow execution of an
administrator specified command immediately after creation of a
scheduled snapshot set, which allows for a basic backup or
replication mechanism.
As an example for illustrative purposes only, an administrator could
set the value of this property to:
"ssh host zfs receive -d poolB/received"
When a snapshot set is created the auto-snapshot service invokes
"/usr/sbin/zfs send [-i<previous>]<snapshot>"
for each snapshot in the set and pipes the output to the
administrator defined backup command which will receive the
snapshot stream through standard input redirection.
If it is the only existing snapshot matching the specified
dataset and schedule, a full snapshot stream is created.
Otherwise, if previous snapshots matching the dataset and schedule
exist, an incremental difference stream between the previous and
new snapshot is generated using the "-i" argument.
In the above example, snapshot streams will be received and
replicated on "host" under the target filesystem:
"poolB/received"
This feature is rather limited and has a number of drawbacks that
make it unsuitable for further enhancement. The major issues with
it are:
- It can only send data in ZFS snapshot stream format and the
mechanism to determine whether to send incremental or full
snapshot streams is poor. It would incorrectly send a full
stream instead of an incremental stream if all previous
snapshots matching a given schedule and dataset were
destroyed for example. Or worse, if the backup command
feature was enabled some time after the containing
auto-snapshot service was enabled and at least one snapshot
event occurred, then it would incorrectly send an
incremental stream instead of a full stream. In short, it
has no way of knowing about relevant, previous events.
- It can't operate asynchronously, it blocks when called and
the auto-snapshot service waits until the backup command
exits
- If a backup can't be performed at the moment of the creation
of a snapshot set for whatever reason, it can't be deferred
until a more appropriate time in the future.
Time slider phase 2 will replace this feature with a more modular
and flexible plugin framework.
For phase 2 of time slider, auto-snapshot will be greatly
simplified and the SMF method script used to start and stop it's
service instances will no longer include this snapshot taking
functionality nor will it create cron jobs in the role user
zfssnap's cron tab to go with it.
What will remain is the SMF instance along with it's private,
properties, and a simpler SMF start/stop method script that will
be limited to performing validation and sanity checking of the
configuration of each auto-snapshot instance. The Time Slider
service will process the configuration of each auto-snapshot SMF
instance and schedule snapshotting actions accordingly.
4.1.2 Time Slider :
Time slider will enhance the existing time slider smf service:
"svc:/application/time-slider:default" by managing a new
project private daemon (/usr/lib/time-sliderd). The daemon
replaces a number of existing features and provides new
additional features.
Replacement features of the time slider service daemon include:
- creation and management of automatic snapshots (both default
and custom defined) as defined by all enabled SMF instances of
the auto-snapshot service. This is a direct replacement for the
snapshot creation/destruction and cron scheduling features of
the existing auto-snapshot mechanism.
- Integrates the functionality of /usr/lib/time-slider-cleanup:
The auto-snapshot service performs no monitoring of available
space on the ZFS Zpools upon which it creates automatic
snapshots. auto-snapshot has retention rules that govern how
many snapshots are kept according to each schedule (eg. 12
monthly snapshots, 24 hourly snapshots) but a ZFS Zpool fills
up due to these snapshots grabbing too much space on the Zpool
before they reach their "sell by date" and get removed by the
auto-snapshot service then it will go unnoticed and
auto-snapshot will not take any remedial action.
The existing solution to this is provided by:
svc:/application/time-slider:default, which scheduled cron jobs
alongside those created by the auto-snapshot SMF instances.
The additional cron job invoked /usr/lib/time-slider-cleanup
at regular intervals to monitor Zpool capacity and take remedial
action if necessary to remove sufficient snapshots to maintain
a Zpool capacity. /usr/lib/time-slider-cleanup is a project
private interface, and all of the functionality of
/usr/lib/time-slider-cleanup is integrated into the new time
slider daemon
- Adds a D-Bus based messaging system.
Previously, when /usr/lib/time-slider-cleanup had to
perform remedial action to recover space on a ZPool,
it notified desktop users by executing the project private
interface /usr/lib/time-slider-notify
Doing so involved some nasty hacks including identifying
users logged into the system on a desktop session, su'ing
to their user id and scanning their desktop environment in
order to launch /usr/lib/time-slider-notify on their desktop
with appropriate arguments and environment variables to pop up
the relevant notification on their desktop.
The new time-slider service daemon makes use of D-Bus
(LSARC/2006/368) to provide a centralized system message
passing service. The message passing service provides
information on the severity of the situation and the actions
take by time-slider to remedy the situation. The notification
applet /usr/lib/time-slider-notify is no longer executed by
/usr/lib/time-slider-cleanup (nor it's functional replacement
within the new daemon: /usr/lib/time-sliderd). Instead it is
launched automatically as part of the desktop user's session
and upon startup, registers with the message service provided
by time slider on the system D-Bus.
It receives asynchronous notification when remedial actions
occur. The communication between the time-slider service and
applets listening on the system D-Bus is one way only from
service to applet. There are no interfaces defined to allow
applets or other D-Bus clients to execute commands on the
time-slider service.
New, additional features of the time slider service:
- Supports ZFS snapshot holds and deferred deletion:
Time slider now uses deferred deletion as introduced by PSARC/2009/297
which will add compatibility with snapshots that have user holds
placed on them.
- A flexible plugin mechanism that facilitates various backup or
replication mechanisms.
A new time slider plugin service is provided via SMF:
svc:/application/time-slider/plugin
The plugin service defines no "default" plugin instance.
Plugins are represented as instances of the plugin SMF
service. For example, the rsync plugin instance is specified by
the following project private SMF FMRI:
svc:/application/time-slider/plugin:rsync
Additional plugins can in future be added by creating new
instances of the pluginSMF service eg. a new plugin called
"my-custom-backup: would have the following SMF FMRI:
svc:/application/time-slider/plugin:my-custom-backup
The plugin mechanism and it's SMF interfaces are project
private.
As per the standard property inheritance model of SMF, all
plugin instances inherit SMF configuration property values from
svc:/application/time-slider/plugin. Property values can
optionally be overrided, and new configuration property values
unique to a specific plugin instance can be defined. This forms
the basis for time slider's plugin framework. All plugin SMF
instances and properties defined within are project private.
svc:/application/time-slider/plugin defines the following
properties. The following format is used to describe the
properties:
<property-group/property-name> type=defaultvalue
<plugin/trigger_command> string=Not Set
A fully expanded command path including all required arguments
for execution of a trigger command. The purpose of the
trigger command is to receive notification from time slider
when a scheduled snapshot event occurs. The trigger command
can opt to take immediate action or defer further action until
some later time. To assist the trigger command to identify
exactly what snapshots were taken and what auto-snapshot
schedule they belong to time-slider provides additional
project private environmental variables to the trigger
command:
AUTOSNAP_LABEL: the snapshot name component after the "@"
corresponding to the snapshot set just
created.
eg. "rp...@zfs-auto-snap:daily-abcdef
rpool/r...@zfs-auto-snap:daily-abcdef
rpool/ot...@zfs-auto-snap:daily-abcdef
In this case AUTOSNAP_LABEL would be:
"zfs-auto-snap:daily-abcdef"
AUTOSNAP_FMRI: the FMRI of the autosnapshot schedule instance
corresponding to the just created snapshot
set. For the above AUTOSNAP_LABEL example,
the AUTOSNAP_FMRI value would be:
svc:/system/filesystem/zfs/auto-snapshot:daily
PLUGIN_FMRI: the FMRI of the plugin instance that time-slider
is invoking the trigger command on behalf of eg.
svc:/application/time-slider/plugin:rsync
<plugin/verbose> boolean=false
Produce extra verbose output. Useful for debugging purposes.
<plugin/trigger_on> string= hourly,daily,weekly,monthly
Plugins may not be interested in certain snapshot schedules
eg. a frequent snapshot that takes place several times an
hour and with a very short lifespan on the system might not
be of interest to a nightly backup.
This project private property allows customization of what
snapshot schedules a plugin should get notified about. A
wildcard value of "All" is supported which matches all
schedules. Otherwise a comma separated list of auto-snapshot
schedule instance names is excepted. Only the instance name
of the FMRI and not the full FMRI is required eg. "daily"
instead of "svc:/system/filesystem/zfs/auto-snapshot:daily"
With the above information, plugins can identify the
snapshots involved and the schedules that match them and further
details if necessary. Depending on the specific implementation,
the plugin can decide whether to take immediate action, no
action, or defer action until a more appropriate time later.
Options available to plugins include tagging snapshots to
identify them for later operations, or if necessary, placing
a hold using ZFS holds to ensure the snapshot does not get
destroyed before the plugin has had an opportunity to take
action against it such as archival or replication. The
time-slider service will co-operate with any holds placed on
snapshots by using deferred deletion to destroy them, but it
becomes the responsibility of the plugin to later release the
previous hold once it no further need of the snapshot.
Once time slider has invoked a plugin instance's trigger
command, it will not block on the plugin's trigger command, so
that snapshot scheduling and other operations are not
disrupted while waiting for the plugin trigger command to
exit. Time slider will not however, invoke a given plugin
instances' trigger command again while a previous invocation
of the same command is still running.
Enabling and disabling of plugins is achieved by using
svcadm (1M) to enable or disable a specified plugin instance
eg.(/usr/sbin/svcadm enable svc:/application/time-slider/plugin:rsync
/usr/sbin/svcadm disable svc:/application/time-slider/plugin:rsync)
Time slider keeps track of the enabled status of plugins and
ignores plugins that are disabled or in SMF maintenance mode.
Enabling and disabling of plugins is project private and is managed
by the setup GUI (/usr/bin/time-slider-setup) and occurs when an
authorized user changes configuration to either enable or disable the
rsync plugin.
Time Slider will provide 2 project private plugins:
- An rsync plugin that will archive snapshots of tagged
filesystems to the specified backup target (a locally visible
filesystem path).
- a zfs-send plugin that provides feature equivalence and
compatibility for the project private feature present in
existing auto-snapshot SMF service instances. The plugin
will supply ZFS snapshot streams in the same format as the
existing auto-snapshot feature does for compatibility with
the auto-snapshot project private feature.
Further project private plugins can be added by creation of
additional instances of the SMF service:
svc:/application/time-slider/plugin
4.1.3 Rsync Plugin Details:
The rsync plugin consists of two components:
- A trigger command invoked by time-sliderd :
When a scheduled snapshot event occurs that the rsync
plugin is interested in, this script identifies the set of
ZFS filesystem snapshots that were just previously created.
The rsync plugin looks for a project private ZFS user
property:
"org.opensolaris:time-slider-rsync" amongst the ZFS filesystems
to identify those whose snapshots should be archived.
This allows fine-grained control over what what filesystems get
backed up.
From this, the trigger command determines the subset of the
snapshot set that should be backed up by rsync. It tags the
selected snapshots using a project private ZFS user property:
"org.opensolaris:time-slider-plugin:rsync"
It sets a value of "pending" to indicate that the snapshot
should be backed up as soon as possible by rsync.
- A backup command that performs the actual rsync backup:
This command is executed as a cron job run every 30 minutes.
The command is project private. The cron job is setup in
root's crontab when the rsync plugin instance is enabled and
online, and removed when disabled or offline.
The rsync-backup script also supports a project private direct
form of invocation. This enables the rsync backup to be operated
in both scheduled and adhoc manners. To ensure only one instance
of the rsync backup command is executing at any given moment,
the first running instance creates and obtains an exclusive,
non-blocking lock file under "/tmp/.time-slide/rsync-backup.lock"
The lock file is a project private interface. Subsequent parallel
invocations of the rsync-backup script will immediately exit when
they cannot exclusively lock the lock file. The lock on the file
is automatically released when the process owning it exits either
normally or abnormally.
If the configured rsync backup target is not accessible the
command simply exits. Otherwise it performs cleanup operations
on the backup target then proceeds to work through the queue of
snapshots that were previously tagged earlier with the project
private ZFS dataset user property:
"org.opensolaris:time-slider-plugin:rsync"
To provide user notification, it registers with the system D-Bus
service and sends D-Bus messages corresponding to it's status
and current activities. The D-Bus interfaces it provides for
consumption by the notification applet are project private.
The communication path via the system D-Bus is one way only and
there are no interfaces available to clients to run queries or
execute commands on the backup service. It's purpose is for
notification to the desktop applet only.
4.1.4 GUI Changes:
The configuration GUI (time-slider-setup) has been extended to
support configuration of the rsync plugin as an option, including
individual filesystem selection for rsync backup.
Because the rsync plugin backs up snapshots of ZFS filesystems
rather than the live filesystem, a strict subset policy is
enforced in the selection ie. It is not possible to select a
filesystem for rsync archival without it first being selected
for automatic snapshotting.
Desktop Notification Applet
The notification applet as previously ARC'ed under LSARC 2008/571
is now automatically launched as part of a user's desktop session.
On startup it registers with the system D-Bus service, and listens to
project private D-Bus system events that inform it when the
rsync plugin is active and the activities that it is performing.
It uses these notifications to provide appropriate visual notification
on the user's desktop.
4.1.5 Security:
As SMF services, configuration of auto-snapshot, time-slider and
time slider plugin services instances is restricted to authorized
users and is enforced by the SMF administration commands such
as svcadm(1M) and svccfg(1M). Unauthorized and non-privileged
users can not modify, enable or disable these services.
The setup GUI requires the Primary Administrator profile to
launch to ensure the user is authorized to perform configuration
of all features of time slider.
Configuration relating to the datasets time-slider operates on
both in terms of snapshotting and archival using the
rsync plugin is based on ZFS dataset user properties. The
services will only snapshot datasets and backup snapshots on
appropriately tagged datasets. Tagging any type of ZFS dataset
is a restricted operation performed using the zfs(1M) command
which restrict these operations to authorized users only.
To reduce the risk of potentially sensitive data being backed up
to untrusted locations, the rsync plugin only permits backups to
locally attached storage such as external storage devices or
trusted network filesystems with local mount points (eg. NFS)
Backup to remote devices using network protocols is not
supported. Furthermore, the administrator has full control over
what ZFS filesystems get backed up using rsync via the time slider
setup GUI (/usr/bin/time-slider-setup), which allows individual
ZFS filesystem selection/deselection.
Please refer to :
http://jdswiki.ireland.sun.com/twiki/bin/view/JDS/ZFSBackupToRM
4.2 Bug/RFE Number(s):
6942008
4.5 Interfaces:
New Imported Interfaces:
--------------------
ZFS snapshot holds (PSARC/2009/297)
D-Bus Message Bus System (LSARC/2006/368)
Add rsync command to Solaris WOS (PSARC/2006/642)
New Exported Interfaces:
--------------------
svc:/application/time-slider/plugin:rsync Project Private
svc:/application/time-slider/plugin:zfs-send Project Private
/usr/lib/time-sliderd Project Private
/usr/lib/time-slider/plugins/rsync/rsync-trigger Project Private
/usr/lib/time-slider/plugins/rsync/rsync-backup Project Private
/usr/lib/time-slider/plugins/zfssend/zfssend Project Private
/etc/dbus-1/system.d/time-slider.conf Project Private
Removed Interfaces:
--------------------
/usr/lib/time-slider-cleanup Uncommitted
5. Documentation
snapshot manager :
http://jdswiki.ireland.sun.com/twiki/bin/view/JDS/ZFSBackupToRM
auto snapshot service :
http://blogs.sun.com/timf/
6. Resources and Schedule
6.1. Projected Availability
Solaris Next
6.4. Steering Committee requested information
6.4.1. Consolidation C-team Name:
Desktop C-Team
6.5. ARC review type: FastTrack
6.6. ARC Exposure: open
6. Resources and Schedule
6.4. Steering Committee requested information
6.4.1. Consolidation C-team Name:
Desktop
6.5. ARC review type: OnePager
6.6. ARC Exposure: open
_______________________________________________
opensolaris-arc mailing list
opensolaris-arc@opensolaris.org
_______________________________________________
opensolaris-arc mailing list
opensolaris-arc@opensolaris.org
