This is an automated email from the ASF dual-hosted git repository.
daim pushed a change to branch DetailedGC/OAK-10199
in repository https://gitbox.apache.org/repos/asf/jackrabbit-oak.git
from 667e2d234b OAK-10328 fix scope of "jackrabbit-jcr-tests" (#1005)
add 5aa3c1b4dd oak-search-elastic: field values should be deduplicated
(#1006)
add e50de3a317 OAK-10226: fix lucene->elastic conversion for WordDelimiter
(#1003)
add e50f432887 OAK-10313: Identify revisions created by late-write scenario
add 41e78a3a77 OAK-10313: Identify revisions created by late-write scenario
add e2bbbebf49 Merge branch 'trunk' into OAK-10313
add c5fe2b73cd OAK-10313: Identify revisions created by late-write scenario
add 3669b31f39 Merge branch 'trunk' into OAK-10313
add 56366607a4 Merge pull request #995 from mreutegg/OAK-10313
add cbfacdb90f Initial commit of pipelined download strategy.
add c505124a35 Minor fixes
add 03e3be8992 WIP
add 58a36a1bd2 Merge remote-tracking branch 'upstream/trunk' into
GRANITE-45911
add f805fdab28 Add recovery from broken MongoDB connections to downloader.
add 9bfb9fc54a Merge remote-tracking branch 'upstream/trunk' into
GRANITE-45911
add 45d0eaf291 Shutdown gracefully if one of the dump stages fails with an
exception.
add 9d1382ccb0 Add documentation. Change the configuration for the retrial
mechanism for MongoDB connection failures: instead of number of retrials, use
the amount of time to keep trying before giving up.
add db824bd6dc Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add 832074fe64 Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add 7384304e73 Add support for auto-tuning working set memory based on
total available memory to the indexer. Switch to using system properties to
configure the pipelined strategy (instead of env variables).
add 981e673e6c Always use read preference secondaryPreferred for
downloading from Mongo.
add 0e10939319 Address reviews comments.
add 8633be3e60 Use 0 and Long.MAX_VALUE as boundaries for download range,
no need to query Mongo to determine the earliest and latest values of _modified.
add 5edda975aa Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add c0b80afe51 Add a new stage to merge sorted files. Log how much time
the download and transform threads spend waiting to enqueue their outputs in
the out queues (this indicates that the stages after them are too slow).
Collect and log metrics describing the download and transform stages, like
mongo documents downloaded, node state entries extracted, filtered, mongo
documents that do not match a node state (garbage) and a few others. Use only
FileUtils.byteCountToDisplaySize() to p [...]
add ea7ce8a70f Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add 2b84a30d44 Improve collection and logging of metrics.
add c68fad0dcc Add unit test for merge-sort task.
add 93f9eacd85 Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add 8c5dc547e8 Add more tests, including a first draft of an integration
test. Refactor code.
add 787a4d575c Fix
add 62697ca44e Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add 66d176bb90 Add an integration test
add 6387d4dc5b Add license header.
add a9dd47546f Add license header.
add ab27625a83 Rename test files.
add fabea754fd Do not filter on _modified != null when doing
non-recoverable download.
add 1cf44cf70c Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add a7f48a0370 Fixes based on review comments.
add 5590997ded Bound the size of the histograms.
add ab06697edb Add missing license files.
add 2839cc9d45 Improve logging and code clean of bounded histogram.
add ef2182e4aa Display human readable byte counts with 2 decimal places
instead of abbreviating to closest integer value.
add 27f9b710ef Improved logging.
add 4f92e897d9 Refactoring
add 4c33e932b4 Remove changes unrelated to PR.
add ec2b12fd8e Merge remote-tracking branch 'upstream/trunk' into OAK-10294
add a4b7bf1016 Add tests for error handling when configuration properties
have invalid values. Minor refactoring.
add 11f308c86f Add test for when the path predicate does not match any
existing path in the document store. Address other review comments.
add 81eaeead4a Update
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMergeSortTask.java
add 560dd56d5d Update
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMongoDownloadTask.java
add 00923a38c0 Merge remote-tracking branch 'origin/OAK-10294' into
OAK-10294
add 074e4af804 Add unit test to test recovery from broken mongo
connections. Fix: when a connection to Mongo is lost, the documents that were
collected in a block but not yet enqueued were being lost.
add 70e6b89a86 Add missing license.
add 25c01b8176 Merge pull request #979 from nfsantos/OAK-10294
add 8677db8d98 OAK-10331 add caller information to different warnings
(#1010)
add 965d6703b5 OAK-10337: mvn jetty:run fails to start oak-web (#1016)
add db3bf04066 OAK-10307: re-add guava to several subprojects to re-enable
isolated compilation (will be removed later on) - fix import range
add e1d3e3e8db OAK-10338 - Do not use platform-dependent newlines in for
files generated when creating a FlatFileStore (#1018)
add 6b1686326e OAK-10307: re-add guava to several subprojects to re-enable
isolated compilation (will be removed later on) - fix import range
add 288e81cacb OAK-10342: Update Apache Parent POM to version 30
add 1a6acc0a8b OAK-10338: Change test to tolerate different line endings
add 9ea124cfbb OAK-10327 Embedded dependencies should have "provided"
scope (#1004)
add 53bc40653e OAK-10326: remove baseline comparison version (back to
default)
add 4d7cf4b58d Revert "OAK-10342: Update Apache Parent POM to version 30"
add 03b703e91f OAK-10332 Allow reusing FilterProviderImpl outside OSGi
context (#1014)
No new revisions were added by this update.
Summary of changes:
.../principalbased/impl/FilterProviderImpl.java | 23 +-
.../impl/AbstractPrincipalBasedTest.java | 4 +-
oak-blob-cloud-azure/pom.xml | 4 +
.../org/apache/jackrabbit/oak/commons/IOUtils.java | 21 +
.../jackrabbit/oak/commons/sort/ExternalSort.java | 4 +-
.../org/apache/jackrabbit/oak/query/QueryImpl.java | 7 +-
oak-examples/webapp/pom.xml | 19 +-
oak-parent/pom.xml | 2 -
oak-run-commons/pom.xml | 7 +
.../indexer/document/DocumentStoreIndexerBase.java | 63 +-
.../indexer/document/NodeStateEntryTraverser.java | 7 +-
.../document/NodeStateEntryTraverserFactory.java | 5 +-
.../flatfile/FlatFileNodeStoreBuilder.java | 39 +-
.../document/flatfile/FlatFileStoreUtils.java | 8 +-
.../MultithreadedTraverseWithSortStrategy.java | 10 +-
.../document/flatfile/NodeStateEntryWriter.java | 26 +-
.../document/flatfile/StoreAndSortStrategy.java | 11 +-
.../document/flatfile/TraverseAndSortTask.java | 10 +-
.../flatfile/TraverseWithSortStrategy.java | 19 +-
.../flatfile/pipelined/BoundedHistogram.java | 89 +++
.../document/flatfile/pipelined/ConfigHelper.java} | 44 +-
.../document/flatfile/pipelined/DownloadRange.java | 74 +++
.../flatfile/pipelined/NodeStateEntryBatch.java | 96 +++
.../NodeStateHolder.java} | 39 +-
.../{ => pipelined}/PathElementComparator.java | 32 +-
.../flatfile/pipelined/PipelinedMergeSortTask.java | 140 ++++
.../pipelined/PipelinedMongoDownloadTask.java | 327 ++++++++++
.../flatfile/pipelined/PipelinedSortBatchTask.java | 155 +++++
.../flatfile/pipelined/PipelinedStrategy.java | 433 +++++++++++++
.../flatfile/pipelined/PipelinedTransformTask.java | 246 +++++++
.../document/flatfile/pipelined/SortKey.java | 89 +++
.../pipelined/TransformStageStatistics.java | 171 +++++
.../document/mongo/MongoDocumentStoreHelper.java | 9 +
.../document/mongo/MongoDocumentTraverser.java | 65 +-
.../plugins/document/mongo/TraversingRange.java | 81 +++
.../document/flatfile/FlatFileStoreTest.java | 8 +-
.../MultithreadedTraverseWithSortStrategyTest.java | 4 +-
.../document/flatfile/TraverseAndSortTaskTest.java | 2 +-
.../flatfile/pipelined/BoundedHistogramTest.java | 56 ++
.../pipelined/NodeStateEntryBatchTest.java | 88 +++
.../document/flatfile/pipelined/PipelinedIT.java | 214 +++++++
.../pipelined/PipelinedMergeSortTaskTest.java | 121 ++++
.../pipelined/PipelinedMongoDownloadTaskTest.java | 104 +++
.../pipelined/PipelinedSortBatchTaskTest.java | 186 ++++++
.../test/resources/pipelined/merge-expected.json | 6 +
.../test/resources/pipelined/merge-stage-1.json | 3 +
.../test/resources/pipelined/merge-stage-2.json | 3 +
...rphanedNodeCheck.java => ConsistencyCheck.java} | 80 +--
.../plugins/document/check/DocumentStoreCheck.java | 18 +-
.../oak/run/DocumentStoreCheckCommand.java | 9 +
.../document/mongo/DocumentTraverserTest.java | 2 +-
.../index/ElasticCustomAnalyzerMappings.java | 43 ++
.../index/elastic/index/ElasticDocument.java | 10 +-
.../index/elastic/ElasticAbstractQueryTest.java | 15 +
.../plugins/index/elastic/ElasticContentTest.java | 24 +
oak-search-mt/pom.xml | 1 +
.../plugins/index/FullTextAnalyzerCommonTest.java | 37 ++
oak-segment-azure/pom.xml | 1 +
oak-segment-tar/pom.xml | 14 +-
oak-store-document/pom.xml | 1 +
.../oak/plugins/document/Consistency.java | 138 ++++
.../oak/plugins/document/FailingDocumentStore.java | 37 +-
.../plugins/document/WriteAfterRecoveryTest.java | 707 +++++++++++++++++++++
63 files changed, 4002 insertions(+), 309 deletions(-)
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/BoundedHistogram.java
copy
oak-run-commons/src/{test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/CountingIterable.java
=>
main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/ConfigHelper.java}
(53%)
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/DownloadRange.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/NodeStateEntryBatch.java
copy
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/{SimpleNodeStateHolder.java
=> pipelined/NodeStateHolder.java} (55%)
copy
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/{
=> pipelined}/PathElementComparator.java (73%)
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMergeSortTask.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMongoDownloadTask.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedSortBatchTask.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedStrategy.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedTransformTask.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/SortKey.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/TransformStageStatistics.java
create mode 100644
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/plugins/document/mongo/TraversingRange.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/BoundedHistogramTest.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/NodeStateEntryBatchTest.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedIT.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMergeSortTaskTest.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMongoDownloadTaskTest.java
create mode 100644
oak-run-commons/src/test/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedSortBatchTaskTest.java
create mode 100644
oak-run-commons/src/test/resources/pipelined/merge-expected.json
create mode 100644
oak-run-commons/src/test/resources/pipelined/merge-stage-1.json
create mode 100644
oak-run-commons/src/test/resources/pipelined/merge-stage-2.json
copy
oak-run/src/main/java/org/apache/jackrabbit/oak/plugins/document/check/{OrphanedNodeCheck.java
=> ConsistencyCheck.java} (53%)
create mode 100644
oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/Consistency.java
create mode 100644
oak-store-document/src/test/java/org/apache/jackrabbit/oak/plugins/document/WriteAfterRecoveryTest.java
