When back pressure is enabled (default) the upstream operators are blocked till space is freed up by downstream operators consuming data.
Since bufferserver also provides fault recovery functionality it cannot immediately clear out the data when it is consumed by the downstream operators and needs to keep it around till next checkpoints thoughout the dag and the spillover to disk can come into play if the amount of data between checkpoints is greater than the in memory buffer capacity. Thanks On Wed, Jun 20, 2018 at 4:41 PM Mateusz Zakarczemny <m.zakarcze...@gmail.com> wrote: > HI, > I'm reading Apex documentation regarding buffer servers. I'm wondering > what will happen if buffers between operators became overflowed (lets > assume non partitioned operator)? > I read somewhere that data is split to disk. But what's next? What if disk > space will be exhausted? > > Regards, > Mateusz Zakarczemny >
