Hi,
I am using the SearchElasticSearch (1.20.0) processor to retrieve all
documents (~20M) from an index, process and eventually return results to a
new index, although for this test I'm retrieving and processing then
discarding. I'm using opensearch.
My problem is that the process restarts after completion - I discovered
this, and docs confirm, after seeing warnings from my processing code
(which reformats json ready for other work) being repeated for the same
document ID.
How do I configure the processor to stop after the completing the first
query.
I've tried the following:
Query: {"query" : {"match_all" :{}}}
with pagination_type SCROLL
I haven't found a combination of the properties that doesn't lead to
repeated cycles through the index.
I've also tried {"query" : {"match_all" :{}}, "sort" : [{"Visit_DateTime" :
"asc"]}}
and SEARCH_AFTER pagination type, with the same problem.
What am I missing?
Thanks
