zan-mateusz opened a new pull request, #1439:
URL: https://github.com/apache/brooklyn-server/pull/1439
## Problem
`OpenSearchLogStore` sorts logbook queries on `timestamp` only. Timestamps
have millisecond
precision, and a burst of output (for example many lines of script stdout
over SSH) is often
logged with dozens of lines in the same millisecond. OpenSearch /
Elasticsearch return entries
with equal sort values in no guaranteed order.
It is most visible in "latest" mode: the query sorts descending and the
store then reverses the
result, so the entries in each millisecond are shown in reverse.
## Fix
Sort on `timestamp` and then on `_seq_no`, both in the same direction.
`_seq_no` is assigned by
OpenSearch / Elasticsearch in the order documents are written to a shard. A
shipper such as
Fluentd writes a log file's lines in order, so this is the order they were
logged.
- No configuration, log format or ingestion changes are needed, and it also
applies to entries
indexed before the upgrade.
- If the store rejects the sort (HTTP 400), it logs one warning and falls
back to sorting by
timestamp only, which is the previous behaviour.
Limitation: `_seq_no` is per shard. With several primary shards, entries
from different shards
logged in the same millisecond can still interleave. A single primary shard,
the default, is
fully ordered.
## Testing
- `OpenSearchLogStoreTest`: query JSON with the new sort in both directions,
the query without
it, and the fallback when the sort is rejected. The test class now uses
TestNG, like the rest
of `core`, so it runs in the build.
- Manual, with an AMP build, Fluentd and OpenSearch 2.11 (1 primary shard),
using bursts of
5,000 stdout lines (~50 per millisecond):
- before: "latest" view 4,999 of 5,000 lines out of order
- after: 0 out of order in both "oldest first" and "latest"
- Same check against Elasticsearch 9.4.3.
- The file log store is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
