Hey Xinyu,
I am trying to implement "Batched RPC" as described in -
https://beam.apache.org/blog/2017/08/28/timely-processing.html
The documentation for GroupIntoBatches says "Batches will contain only elements
of a single key".
And my understanding is for "Batched RPC", I need a batch of keys. So, I am not
sure if I can use GroupIntoBatches.
On 1/18/19, 10:40 AM, "Xinyu Liu" <xinyuliu...@gmail.com> wrote:
This email is from an external sender.
sorry, the correct link to the first reference:
https://github.com/apache/beam/blob/master/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
.
Thanks,
Xinyu
On Fri, Jan 18, 2019 at 10:35 AM Xinyu Liu <xinyuliu...@gmail.com> wrote:
> Hi, Omkar,
>
> Your observation is correct. Currently bundle is implemented in a
> per-event basis (Code is DoFnOp.processElement,
>
https://docs.google.com/spreadsheets/d/1pIUQ8J658B7GPNDt5dwiJBRQyWep1n2rXlkONyA6ZMM/edit#gid=1709587251).
> We are working on supporting bundles in Samza right now so in Beam we can
> take advantage of it. Bundling is also critical to have better python
> performance so we are trying to get it out very soon (Feb-March).
>
> On the other hand, in java if you want to process in a batch fashion, you
> can use the Beam GroupIntoBatches api (
>
https://beam.apache.org/releases/javadoc/2.0.0/org/apache/beam/sdk/transforms/GroupIntoBatches.html).
> This will group the elements into batches and then deliver to your ParDo
> afterwards. Please let us know whether this works for you.
>
> Thanks,
> Xinyu
>
>
>
> On Fri, Jan 18, 2019 at 10:27 AM Daniel Chen <dch...@linkedin.com> wrote:
>
>> + Xinyu
>>
>> On 1/18/19, 10:05 AM, "Deshpande, Omkar" <omkar_deshpa...@intuit.com>
>> wrote:
>>
>> Hello,
>>
>> I am using Samza runner with Apache Beam. Is there any documentation
>> available on how bundles are implemented in the Samza runner?
>> I have observed every Kafka record getting processed in its own
>> bundle. How can I get larger bundles?
>>
>> <beam.version>2.9.0 <samza.version>0.14.1
>>
>> Thanks,
>> Omkar
>>
>>
>>
