Re: A few questions about minibatch

Thanks, that all makes sense! On Wed, Jan 27, 2021 at 7:00 PM Jark Wu wrote: > Hi Rex, > > Could you share your query here? It would be helpful to identify the root > cause if we have the query. > > 1) watermark > The framework automatically adds a node (the MiniBatchAssigner) to > generate

Re: A few questions about minibatch

Hi Rex, Could you share your query here? It would be helpful to identify the root cause if we have the query. 1) watermark The framework automatically adds a node (the MiniBatchAssigner) to generate watermark events as the mini-batch id to broadcast and trigger mini-batch in the pipeline. 2)

Re: A few questions about minibatch

I am pulling Jark and Godfrey who are more familiar with the planner internals. Best, Dawid On 22/01/2021 20:11, Rex Fenley wrote: > Hello, > > Does anyone have any more information here? > > Thanks! > > On Wed, Jan 20, 2021 at 9:13 PM Rex Fenley > wrote: > > Hi,

Re: A few questions about minibatch

Hello, Does anyone have any more information here? Thanks! On Wed, Jan 20, 2021 at 9:13 PM Rex Fenley wrote: > Hi, > > Our job was experiencing high write amplification on aggregates so we > decided to give mini-batch a go. There's a few things I've noticed that are > different from our

A few questions about minibatch

Hi, Our job was experiencing high write amplification on aggregates so we decided to give mini-batch a go. There's a few things I've noticed that are different from our previous job and I would like some clarification. 1) Our operators now say they have Watermarks. We never explicitly added