Look at the GroupIntoBatches[1] transform. It will buffer "batches" of size X for you.
1: https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/ On Thu, Oct 1, 2020 at 2:51 PM Vincent Marquez <vincent.marq...@gmail.com> wrote: > the downstream consumer has these requirements. > > *~Vincent* > > > On Thu, Oct 1, 2020 at 2:29 PM Luke Cwik <lc...@google.com> wrote: > >> Why do you want to only emit X? (e.g. running out of memory in the runner) >> >> On Thu, Oct 1, 2020 at 2:08 PM Vincent Marquez <vincent.marq...@gmail.com> >> wrote: >> >>> Hello all. If I want to 'throttle' the number of messages I pull off >>> say, Kafka or some other queue, in order to make sure I only emit X amount >>> per trigger, is there a way to do that and ensure that I get 'at least >>> once' delivery guarantees? If this isn't supported, would the better way >>> be to pull the limited amount opposed to doing it on the output side? >>> >>> >>> *~Vincent* >>> >>
