Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1565#issuecomment-179225513
Yeah I thought so that your question on the mailing list was related to
this PR. It would be great to have a fully parallelized version of the
algorithm, because if it only runs on a single machine then you could
directly use sklearn or another ML-library to solve the problem.
You can also share the sketch of your algorithm right now, if you want to.
That way, others could directly chime in and maybe someone knows how to do
the alternating pair join operation.
Cheers,
Till
On Wed, Feb 3, 2016 at 12:32 PM, Fridtjof <[email protected]> wrote:
> Thanks for your Feedback!
>
> Yes, scalability is the main issue for us too. We are not aware of any
> other parallel implementation
> he main issue for us too. We also talked to the original author of Spark's
> IR implementation (which is equivalent too ours) about this with the same
> result. However, we think we have a theoretical approach to solving this,
> but it depends on the self join without duplicates. Remember our
discussion
> on the user-mailing list with subject join with no element appearing in
> multiple join-pairs? I need that for this.
>
> I will link a sketch to our algorithm design here in a few days, If we
> haven't found a way to solve this. I guess IR won't make it into Flink
> without a fully parallelized way?
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/flink/pull/1565#issuecomment-179177999>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
