Joerg Hoh created SLING-13021:
---------------------------------
Summary: Option to import content concurrently
Key: SLING-13021
URL: https://issues.apache.org/jira/browse/SLING-13021
Project: Sling
Issue Type: New Feature
Components: Content Distribution
Reporter: Joerg Hoh
the import of distribution messages currently is a single-threaded serial
process, which of course limits the amount of messages to get imported per
hour. This also gives some strong guarantees, that if a package A is submitted
before a package B, package A is also imported before package B.
Now these strong guarantees are not always required, but instead they could be
dropped in favor of a higher throughput.
the constraints:
* the number of concurrent imports is configurable via OSGI, the default will
be 1 (serialized, as it is now)
* the persisted offset (which indicates the "last" message imported) is only
updated, when no message with a lower offset is still being imported. That
means if with a concurrency of 4 the messages with the offsets 1,2,3 and 4 are
being imported, and they are imported in the order "3,2,1,4" then only the
offsets 3 and 4 are persisted (3 is persisted when the message 1 was processed,
4 when the message 4 was persisted). With a concurrency level of 1 the stored
offsets are 1,2,3 and 4 (as of today).
* That also means that a message import is idempotent: Importing it twice must
not be a problem. (In the above example if the system crashes after having 2
imported, it will import 3 and 2 on the next attempt as well, even though they
have already been imported.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
