Mikael Goldmann created CRUNCH-655:
--------------------------------------
Summary: DefaultJoinStrategy full outer join failing for spark
pipeline
Key: CRUNCH-655
URL: https://issues.apache.org/jira/browse/CRUNCH-655
Project: Crunch
Issue Type: Bug
Components: Spark
Environment: Mac OSX, crunch 0.13.0 and 0.15.0 (with reproduction
code), Ubuntu 14.04 (repro code not tried, but similar issue in production with
0.13.0)
Reporter: Mikael Goldmann
Attachments: OuterJoinTest.java
When the left and right table in the the join have entries with the same key,
they do not alway end up together. Cannot reproduce when running the join with
a single reducer, and happens more commonly if there are many reducers and very
few copies of each key to the left and right.
My guess is that it sometimes happens that the left value for key k ends up on
a different reducer from the right value with key k.
With my production issue, it went away if I either used a single reducer or
used cogroup instead.
I've attached a class to reproduce the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
