> Can there be a situation where the index writer fails after the document
was added to tlog and a success is sent to the user? I think we want to
avoid such a situation, isn't it?
> I suppose failures would be returned to the client one the async response?
To make things more clear, the response for async update will be something
like this
{ "trackId" : "<update_version>" }
Then the user will call another endpoint for tracking the response like GET
status_updates?trackId=<update_version>, the response will tell that
whether the update is in_queue, processing, succeed or failed. Currently we
are also adding to tlog first then call writer.addDoc later.
Later we can convert current sync operations by waiting until the update
gets processed before return to users.>How would one keep the tlog from growing forever if the actual indexing took a long time? I think it won't be very different from what we are having now, since on commit (producer threads do the commit) we rotate to a new tlog. > I'd like to add another wrinkle to this. Which is to store the information about each batch as a record in the index. Each batch record would contain a fingerprint for the batch. This solves lots of problems, and allows us to confirm the integrity of the batch. It also means that we can compare indexes by comparing the batch fingerprints rather than building a fingerprint from the entire index. Thank you, it adds another pros to this model :P On Fri, Oct 9, 2020 at 2:10 AM Joel Bernstein <[email protected]> wrote: > I think this model has a lot of potential. > > I'd like to add another wrinkle to this. Which is to store the information > about each batch as a record in the index. Each batch record would contain > a fingerprint for the batch. This solves lots of problems, and allows us to > confirm the integrity of the batch. It also means that we can compare > indexes by comparing the batch fingerprints rather than building a > fingerprint from the entire index. > > > Joel Bernstein > http://joelsolr.blogspot.com/ > > > On Thu, Oct 8, 2020 at 11:31 AM Erick Erickson <[email protected]> > wrote: > >> I suppose failures would be returned to the client one the async response? >> >> How would one keep the tlog from growing forever if the actual indexing >> took a long time? >> >> I'm guessing that this would be optional.. >> >> On Thu, Oct 8, 2020, 11:14 Ishan Chattopadhyaya < >> [email protected]> wrote: >> >>> Can there be a situation where the index writer fails after the document >>> was added to tlog and a success is sent to the user? I think we want to >>> avoid such a situation, isn't it? >>> >>> On Thu, 8 Oct, 2020, 8:25 pm Cao Mạnh Đạt, <[email protected]> wrote: >>> >>>> > Can you explain a little more on how this would impact durability of >>>> updates? >>>> Since we persist updates into tlog, I do not think this will be an issue >>>> >>>> > What does a failure look like, and how does that information get >>>> propagated back to the client app? >>>> I did not be able to do much research but I think this is gonna be the >>>> same as the current way of our asyncId. In this case asyncId will be the >>>> version of an update (in case of distributed queue it will be offset) >>>> failures update will be put into a time-to-live map so users can query the >>>> failure, for success we can skip that by leverage the max succeeded version >>>> so far. >>>> >>>> On Thu, Oct 8, 2020 at 9:31 PM Mike Drob <[email protected]> wrote: >>>> >>>>> Interesting idea! Can you explain a little more on how this would >>>>> impact durability of updates? What does a failure look like, and how does >>>>> that information get propagated back to the client app? >>>>> >>>>> Mike >>>>> >>>>> On Thu, Oct 8, 2020 at 9:21 AM Cao Mạnh Đạt <[email protected]> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> First of all it seems that I used the term async a lot recently :D. >>>>>> Recently I have been thinking a lot about changing the current >>>>>> indexing model of Solr from sync way like currently (user submit an >>>>>> update >>>>>> request waiting for response). What about changing it to async model, >>>>>> where >>>>>> nodes will only persist the update into tlog then return immediately much >>>>>> like what tlog is doing now. Then we have a dedicated executor which >>>>>> reads >>>>>> from tlog to do indexing (producer consumer model with tlog acting like >>>>>> the >>>>>> queue). >>>>>> >>>>>> I do see several big benefits of this approach >>>>>> >>>>>> - We can batching updates in a single call, right now we do not >>>>>> use writer.add(documents) api from lucene, by batching updates this >>>>>> gonna >>>>>> boost the performance of indexing >>>>>> - One common problems with Solr now is we have lot of threads >>>>>> doing indexing so that can ends up with many small segments. Using >>>>>> this >>>>>> model we can have bigger segments so less merge cost >>>>>> - Another huge reason here is after switching to this model, we >>>>>> can remove tlog and use a distributed queue like Kafka, Pulsar. Since >>>>>> the >>>>>> purpose of leader in SolrCloud now is ordering updates, the >>>>>> distributed >>>>>> queue is already ordering updates for us, so no need to have a >>>>>> dedicated >>>>>> leader. That is just the beginning of things that we can do after >>>>>> using a >>>>>> distributed queue. >>>>>> >>>>>> What do your guys think about this? Just want to hear from your guys >>>>>> before going deep into this rabbit hole. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -- *Best regards,* *Cao Mạnh Đạt* *E-mail: [email protected] <[email protected]>*
