One further question - what is the recommended way of checking for updates in an index and fetching new records in a similar manner to GenerateTableFetch for an sql DB? Thanks
On Thu, Aug 17, 2023 at 7:21 AM Richard Beare <[email protected]> wrote: > Sounds perfect. Thanks > > On Thu, Aug 17, 2023 at 5:11 AM Chris Sampson <[email protected]> wrote: > >> What you describe sounds like the processor is working as designed & >> documented, i.e. it will restart the same query once it has reached the end >> of the paginated scroll (or search_after, or point-in-time) query. >> >> Instead, it sounds like you want to try using the >> PaginatedJsonQueryElasticsearch [1] processor instead. This will execute >> the query given to it, either as the query property or the body of an >> incoming FlowFile, output the results, and then stop. >> >> >> [1] >> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-elasticsearch-restapi-nar/1.23.0/org.apache.nifi.processors.elasticsearch.PaginatedJsonQueryElasticsearch/index.html >> >> On 2023/08/16 07:57:43 Richard Beare wrote: >> > Hi, >> > I am using the SearchElasticSearch (1.20.0) processor to retrieve all >> > documents (~20M) from an index, process and eventually return results >> to a >> > new index, although for this test I'm retrieving and processing then >> > discarding. I'm using opensearch. >> > >> > My problem is that the process restarts after completion - I discovered >> > this, and docs confirm, after seeing warnings from my processing code >> > (which reformats json ready for other work) being repeated for the same >> > document ID. >> > >> > How do I configure the processor to stop after the completing the first >> > query. >> > >> > I've tried the following: >> > >> > Query: {"query" : {"match_all" :{}}} >> > >> > with pagination_type SCROLL >> > >> > I haven't found a combination of the properties that doesn't lead to >> > repeated cycles through the index. >> > >> > I've also tried {"query" : {"match_all" :{}}, "sort" : >> [{"Visit_DateTime" : >> > "asc"]}} >> > >> > and SEARCH_AFTER pagination type, with the same problem. >> > >> > What am I missing? >> > Thanks >> > >> >
