Hi, IIUC the Hybrid indexing on the master operates in parallel with the master index writer, performing the same task but repeatedly throwing its work away when the master provides an update. IIUC, it effectively performs many soft commits to achieve NRT behaviour.
I wonder if there is an opportunity to use the Hybrid indexer on the master instance and every n seconds (or even minutes) perform a hard commit. That hard commit being the output of the master index writer, committed by Oak to the DS. This would avoid doing the work and follows the pattern used by Solr and ES, where an indexing update is written to a WAL, soft committed and periodically hard committed. The WAL comes free as part of Oak so if the soft commits are lost, the index and WAL starts from the last hard commit. To be clear. I am only talking about de-duplicating the effort performed on the master node by the hybrid indexer and the master index writer. I am not talking about anything performed on slave index reader instances which also have a hybrid indexer. Those hybrid indexers will still work as they do now. wdyt? Best Regards Ian
