[
https://issues.apache.org/jira/browse/SLING-13396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Schneider updated SLING-13396:
----------------------------------------
Description:
The journal subscriber installs one package at a time. When the install step of
the package builder is slow, for example because it calls a remote system for
every package, a backlog of many small and independent packages drains slowly.
A typical case is a content change that is automatically propagated to many
locations, which produces one distribution package per path.
SLING-13021 proposes importing packages concurrently in the subscriber. That
drops the ordering between packages (e.g. a delete of /a and an update of /a/b
imported at the same time).
h3. Proposal
Keep the subscriber single-threaded, but let the package builder install
several consecutive packages in one call. The builder knows the paths and can
run independent packages concurrently while keeping the order of overlapping
ones.
* New exported SPI
{{org.apache.sling.distribution.journal.spi.BatchDistributionPackageBuilder}}
(extends {{DistributionPackageBuilder}}) with
{{installPackages(ResourceResolver, List<DistributionPackage>)}}. Packages are
passed in journal order; the result must be the same as installing them one by
one.
* New subscriber setting {{maxBatchSize}} (default 1 = current behaviour).
Applies to editable and non-editable agents when the package builder implements
the SPI and {{packageHandling}} is {{Off}}; otherwise a warning is logged and
packages are imported one by one.
* The poller hands packages to an importer thread through a bounded queue. The
importer takes the packages that are already queued (up to {{maxBatchSize}}),
so single packages get no extra latency.
* Consecutive ADD/DELETE packages accepted by the precondition and not cleared
are batched. Any other package ends the batch and is processed as today.
* The offset of the last package is stored in the same commit as the batch. If
the batch fails, no offset is stored and the packages are imported one by one
with the existing retry and error queue handling (at-least-once, installs must
be idempotent).
* New metrics {{imported_batch_size}} and {{imported_batch_duration}}.
h3. Editable agents
Builders that push content to a remote system typically run on editable agents
with a limited {{maxRetries}}, so that failed packages end up in the error
queue and items can be cleared. Batching must support them:
* The publisher only reacts to {{REMOVED_FAILED}} statuses (error queue), and
{{PackageStatusWatcher}} considers an offset {{IMPORTED}} once a status for a
higher offset has arrived. A successful batch therefore stores a single
{{IMPORTED}} status for its last offset, in the same commit as the offset.
* A stored status is sent before and after each batch, as in the single-package
path.
* Failed batches fall back to per-package import, which reports
{{REMOVED_FAILED}} per package as today. Cleared packages end the batch and are
removed as today.
h3. packageHandling
{{packageHandling}} controls what happens to FileVault content packages (.zip
under /etc/packages) contained in imported content: {{Off}} (default) only
stores them, {{Extract}} unpacks their content into the repository, {{Install}}
fully installs them including sub-packages and install hooks. Batching requires
{{Off}}, because extracting or installing runs in the importer session and any
package error would fail the whole batch. Builders that do not write content to
the repository are not affected by this setting.
h3. Notes for builder implementations
* A builder that groups its remote calls into phases (e.g. all updates first,
then all deletes concurrently) must start a new group when a path, or one of
its parents, appears again in the batch. Otherwise an ADD followed by a DELETE
of the same path can race.
* Side effects done per package outside the given resolver (e.g. updating
properties with a separate session) must run only after the batch succeeded and
must be idempotent, because a failed batch is replayed package by package.
* The expected gain comes from running independent remote calls concurrently
and from using bulk operations of the remote system for many paths, within its
rate limits.
was:
The journal subscriber installs one package at a time. When the install step of
the package builder is slow, for example because it calls a remote system for
every package, a backlog of many small and independent packages is drained
slowly. A typical case is a content change that is automatically propagated to
many locations, which produces one distribution package per path.
SLING-13021 proposes importing packages concurrently in the subscriber. That
drops the ordering between packages (e.g. a delete of /a and an update of /a/b
imported at the same time).
Proposal: keep the subscriber single-threaded, but let the package builder
install several consecutive packages in one call. The builder knows the paths
and can install only non-overlapping packages concurrently.
* New exported SPI
{{org.apache.sling.distribution.journal.spi.BatchDistributionPackageBuilder}}
(extends {{DistributionPackageBuilder}}) with
{{installPackages(ResourceResolver, List<DistributionPackage>)}}. Packages are
passed in journal order; the result must be the same as installing them one by
one.
* New subscriber setting {{maxBatchSize}} (default 1 = current behaviour). Only
applies to non-editable agents with {{packageHandling=Off}} and a package
builder implementing the SPI; otherwise a warning is logged.
* The poller hands packages to an importer thread through a bounded queue. The
importer takes the packages that are already queued (up to {{maxBatchSize}}),
so single packages get no extra latency.
* Consecutive ADD/DELETE packages accepted by the precondition are batched. Any
other package ends the batch and is processed as today.
* The offset of the last package is stored in the same commit as the batch. If
the batch fails, no offset is stored and the packages are imported one by one
with the existing retry and error queue handling (at-least-once, installs must
be idempotent).
* New metrics {{imported_batch_size}} and {{imported_batch_duration}}.
> Journal subscriber: batch install SPI for slow package builders
> ---------------------------------------------------------------
>
> Key: SLING-13396
> URL: https://issues.apache.org/jira/browse/SLING-13396
> Project: Sling
> Issue Type: Improvement
> Components: Content Distribution
> Reporter: Christian Schneider
> Priority: Major
>
> The journal subscriber installs one package at a time. When the install step
> of the package builder is slow, for example because it calls a remote system
> for every package, a backlog of many small and independent packages drains
> slowly. A typical case is a content change that is automatically propagated
> to many locations, which produces one distribution package per path.
> SLING-13021 proposes importing packages concurrently in the subscriber. That
> drops the ordering between packages (e.g. a delete of /a and an update of
> /a/b imported at the same time).
> h3. Proposal
> Keep the subscriber single-threaded, but let the package builder install
> several consecutive packages in one call. The builder knows the paths and can
> run independent packages concurrently while keeping the order of overlapping
> ones.
> * New exported SPI
> {{org.apache.sling.distribution.journal.spi.BatchDistributionPackageBuilder}}
> (extends {{DistributionPackageBuilder}}) with
> {{installPackages(ResourceResolver, List<DistributionPackage>)}}. Packages
> are passed in journal order; the result must be the same as installing them
> one by one.
> * New subscriber setting {{maxBatchSize}} (default 1 = current behaviour).
> Applies to editable and non-editable agents when the package builder
> implements the SPI and {{packageHandling}} is {{Off}}; otherwise a warning is
> logged and packages are imported one by one.
> * The poller hands packages to an importer thread through a bounded queue.
> The importer takes the packages that are already queued (up to
> {{maxBatchSize}}), so single packages get no extra latency.
> * Consecutive ADD/DELETE packages accepted by the precondition and not
> cleared are batched. Any other package ends the batch and is processed as
> today.
> * The offset of the last package is stored in the same commit as the batch.
> If the batch fails, no offset is stored and the packages are imported one by
> one with the existing retry and error queue handling (at-least-once, installs
> must be idempotent).
> * New metrics {{imported_batch_size}} and {{imported_batch_duration}}.
> h3. Editable agents
> Builders that push content to a remote system typically run on editable
> agents with a limited {{maxRetries}}, so that failed packages end up in the
> error queue and items can be cleared. Batching must support them:
> * The publisher only reacts to {{REMOVED_FAILED}} statuses (error queue), and
> {{PackageStatusWatcher}} considers an offset {{IMPORTED}} once a status for a
> higher offset has arrived. A successful batch therefore stores a single
> {{IMPORTED}} status for its last offset, in the same commit as the offset.
> * A stored status is sent before and after each batch, as in the
> single-package path.
> * Failed batches fall back to per-package import, which reports
> {{REMOVED_FAILED}} per package as today. Cleared packages end the batch and
> are removed as today.
> h3. packageHandling
> {{packageHandling}} controls what happens to FileVault content packages (.zip
> under /etc/packages) contained in imported content: {{Off}} (default) only
> stores them, {{Extract}} unpacks their content into the repository,
> {{Install}} fully installs them including sub-packages and install hooks.
> Batching requires {{Off}}, because extracting or installing runs in the
> importer session and any package error would fail the whole batch. Builders
> that do not write content to the repository are not affected by this setting.
> h3. Notes for builder implementations
> * A builder that groups its remote calls into phases (e.g. all updates first,
> then all deletes concurrently) must start a new group when a path, or one of
> its parents, appears again in the batch. Otherwise an ADD followed by a
> DELETE of the same path can race.
> * Side effects done per package outside the given resolver (e.g. updating
> properties with a separate session) must run only after the batch succeeded
> and must be idempotent, because a failed batch is replayed package by package.
> * The expected gain comes from running independent remote calls concurrently
> and from using bulk operations of the remote system for many paths, within
> its rate limits.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
