Hey Jorn,
Appreciate the prompt reply.
Yeah that would surely work, we have tried a similar approach. The only
concern here is that to make the solution low latency, we want to avoid
routing through a message broker.
Regards,
Sandeep.
On Tue, Sep 25, 2018 at 12:53 PM Jörn Franke wrote:
>
What is the ultimate goal of this algorithm? There could be already algorithms
that can do this within Spark. You could also put a message on Kafka (or
another broker) and have spark applications listen to them to trigger further
computation. This would be also more controlled and can be done
That’s a pretty major architectural change and would be extremely difficult
to do at this stage.
On Tue, Sep 25, 2018 at 9:31 AM sandeep mehandru
wrote:
> Hi Folks,
>
>There is a use-case , where we are doing large computation on two large
> vectors. It is basically a scenario, where we run
Hi Folks,
There is a use-case , where we are doing large computation on two large
vectors. It is basically a scenario, where we run a flatmap operation on the
Left vector and run co-relation logic by comparing it with all the rows of
the second vector. When this flatmap operation is running on
