Hi, I am building a service where I have to continously read data from a Solr collection and insert it into another database. Collection will receive daily updates. Initial size of collection is very large. After I have indexed whole data(through cursor mark), on daily basis I want to only do incremental inserts.
My documents don't have anything like timestamp which I can use to fetch "only newly added" documents after a certain point. Is there any internal field which I can use to create this checkpoint and then later use that to fetch "only incremental updates" from that point onwards? I initially tried to sort the document by ID and use last fetched cursor mark, but my unique-ID field is a string and there is NO guarantee that newly added document's ID will be in sorted order. Solr version is 8.2.0. Regards, Vidit
