Have you considered drafting in detail what you think this API might look
like?
If it's a radically different API, it might be more appropriate as an
alternative parallel Beam API rather than a replacement for the current API
(there is also one such fluent API in the works).
On Sat, Mar 10, 2018 at 7:23 AM Romain Manni-Bucau <rmannibu...@gmail.com>
wrote:
>
>
> 2018-03-10 16:19 GMT+01:00 Reuven Lax <re...@google.com>:
>
>> This is another version (maybe a better, Java 8 idiomatic one?) of what
>> Kenn suggested.
>>
>> Note that with NewDoFn this need not be incompatible (so might not
>> require waiting till Beam 3.0). We can recognize new parameters to
>> processElement and populate add needed.
>>
>
> This is right however in my head it was a single way movemenent to enforce
> the design to be reactive and not fake a reactive API with a sync and not
> reactive impl which is what would be done today with both support I fear.
>
>
>>
>> On Sat, Mar 10, 2018, 12:13 PM Romain Manni-Bucau <rmannibu...@gmail.com>
>> wrote:
>>
>>> Yes, for the dofn for instance, instead of having
>>> processcontext.element()=<T> you get a CompletionStage<T> and output gets
>>> it as well.
>>>
>>> This way you register an execution chain. Mixed with streams you get a
>>> big data java 8/9/10 API which enabkes any connectivity in a wel performing
>>> manner ;).
>>>
>>> Le 10 mars 2018 13:56, "Reuven Lax" <re...@google.com> a écrit :
>>>
>>>> So you mean the user should have a way of registering asynchronous
>>>> activity with a callback (the callback must be registered with Beam,
>>>> because Beam needs to know not to mark the element as done until all
>>>> associated callbacks have completed). I think that's basically what Kenn
>>>> was suggesting, unless I'm missing something.
>>>>
>>>>
>>>> On Fri, Mar 9, 2018 at 11:07 PM Romain Manni-Bucau <
>>>> rmannibu...@gmail.com> wrote:
>>>>
>>>>> Yes, callback based. Beam today is synchronous and until
>>>>> bundles+combines are reactive friendly, beam will be synchronous whatever
>>>>> other parts do. Becoming reactive will enable to manage the threading
>>>>> issues properly and to have better scalability on the overall execution
>>>>> when remote IO are involved.
>>>>>
>>>>> However it requires to break source, sdf design to use completionstage
>>>>> - or equivalent - to chain the processing properly and in an unified
>>>>> fashion.
>>>>>
>>>>> Le 9 mars 2018 23:48, "Reuven Lax" <re...@google.com> a écrit :
>>>>>
>>>>> If you're talking about reactive programming, at a certain level beam
>>>>> is already reactive. Are you referring to a specific way of writing the
>>>>> code?
>>>>>
>>>>>
>>>>> On Fri, Mar 9, 2018 at 1:59 PM Reuven Lax <re...@google.com> wrote:
>>>>>
>>>>>> What do you mean by reactive?
>>>>>>
>>>>>> On Fri, Mar 9, 2018, 6:58 PM Romain Manni-Bucau <
>>>>>> rmannibu...@gmail.com> wrote:
>>>>>>
>>>>>>> @Kenn: why not preferring to make beam reactive? Would alow to scale
>>>>>>> way more without having to hardly synchronize multithreading. Elegant
>>>>>>> and
>>>>>>> efficient :). Beam 3?
>>>>>>>
>>>>>>>
>>>>>>> Le 9 mars 2018 22:49, "Kenneth Knowles" <k...@google.com> a écrit :
>>>>>>>
>>>>>>>> I will start with the "exciting futuristic" answer, which is that
>>>>>>>> we envision the new DoFn to be able to provide an automatic
>>>>>>>> ExecutorService
>>>>>>>> parameters that you can use as you wish.
>>>>>>>>
>>>>>>>> new DoFn<>() {
>>>>>>>> @ProcessElement
>>>>>>>> public void process(ProcessContext ctx, ExecutorService
>>>>>>>> executorService) {
>>>>>>>> ... launch some futures, put them in instance vars ...
>>>>>>>> }
>>>>>>>>
>>>>>>>> @FinishBundle
>>>>>>>> public void finish(...) {
>>>>>>>> ... block on futures, output results if appropriate ...
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> This way, the Java SDK harness can use its overarching knowledge of
>>>>>>>> what is going on in a computation to, for example, share a thread pool
>>>>>>>> between different bits. This was one reason to delete
>>>>>>>> IntraBundleParallelization - it didn't allow the runner and user code
>>>>>>>> to
>>>>>>>> properly manage how many things were going on concurrently. And mostly
>>>>>>>> the
>>>>>>>> runner should own parallelizing to max out cores and what user code
>>>>>>>> needs
>>>>>>>> is asynchrony hooks that can interact with that. However, this feature
>>>>>>>> is
>>>>>>>> not thoroughly considered. TBD how much the harness itself manages
>>>>>>>> blocking
>>>>>>>> on outstanding requests versus it being your responsibility in
>>>>>>>> FinishBundle, etc.
>>>>>>>>
>>>>>>>> I haven't explored rolling your own here, if you are willing to do
>>>>>>>> the knob tuning to get the threading acceptable for your particular use
>>>>>>>> case. Perhaps someone else can weigh in.
>>>>>>>>
>>>>>>>> Kenn
>>>>>>>>
>>>>>>>> On Fri, Mar 9, 2018 at 1:38 PM Josh Ferge <
>>>>>>>> josh.fe...@bounceexchange.com> wrote:
>>>>>>>>
>>>>>>>>> Hello all:
>>>>>>>>>
>>>>>>>>> Our team has a pipeline that make external network calls. These
>>>>>>>>> pipelines are currently super slow, and the hypothesis is that they
>>>>>>>>> are
>>>>>>>>> slow because we are not threading for our network calls. The github
>>>>>>>>> issue
>>>>>>>>> below provides some discussion around this:
>>>>>>>>>
>>>>>>>>> https://github.com/apache/beam/pull/957
>>>>>>>>>
>>>>>>>>> In beam 1.0, there was IntraBundleParallelization, which helped
>>>>>>>>> with this. However, this was removed because it didn't comply with a
>>>>>>>>> few
>>>>>>>>> BEAM paradigms.
>>>>>>>>>
>>>>>>>>> Questions going forward:
>>>>>>>>>
>>>>>>>>> What is advised for jobs that make blocking network calls? It
>>>>>>>>> seems bundling the elements into groups of size X prior to passing to
>>>>>>>>> the
>>>>>>>>> DoFn, and managing the threading within the function might work.
>>>>>>>>> thoughts?
>>>>>>>>> Are these types of jobs even suitable for beam?
>>>>>>>>> Are there any plans to develop features that help with this?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>
>>>>>
>
