Christian Schneider created SLING-13396:
-------------------------------------------
Summary: Journal subscriber: batch install SPI for slow package
builders
Key: SLING-13396
URL: https://issues.apache.org/jira/browse/SLING-13396
Project: Sling
Issue Type: Improvement
Components: Content Distribution
Reporter: Christian Schneider
The journal subscriber installs one package at a time. When the install step of
the package builder is slow, for example because it calls a remote system for
every package, a backlog of many small and independent packages is drained
slowly. A typical case is a content change that is automatically propagated to
many locations, which produces one distribution package per path.
SLING-13021 proposes importing packages concurrently in the subscriber. That
drops the ordering between packages (e.g. a delete of /a and an update of /a/b
imported at the same time).
Proposal: keep the subscriber single-threaded, but let the package builder
install several consecutive packages in one call. The builder knows the paths
and can install only non-overlapping packages concurrently.
* New exported SPI
{{org.apache.sling.distribution.journal.spi.BatchDistributionPackageBuilder}}
(extends {{DistributionPackageBuilder}}) with
{{installPackages(ResourceResolver, List<DistributionPackage>)}}. Packages are
passed in journal order; the result must be the same as installing them one by
one.
* New subscriber setting {{maxBatchSize}} (default 1 = current behaviour). Only
applies to non-editable agents with {{packageHandling=Off}} and a package
builder implementing the SPI; otherwise a warning is logged.
* The poller hands packages to an importer thread through a bounded queue. The
importer takes the packages that are already queued (up to {{maxBatchSize}}),
so single packages get no extra latency.
* Consecutive ADD/DELETE packages accepted by the precondition are batched. Any
other package ends the batch and is processed as today.
* The offset of the last package is stored in the same commit as the batch. If
the batch fails, no offset is stored and the packages are imported one by one
with the existing retry and error queue handling (at-least-once, installs must
be idempotent).
* New metrics {{imported_batch_size}} and {{imported_batch_duration}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
