[
https://issues.apache.org/jira/browse/CRUNCH-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Wills resolved CRUNCH-670.
-------------------------------
Fix Version/s: 1.0.0
Resolution: Fixed
And merged!
> Make the AvroPathPerKeyTarget work with the SparkRuntime
> --------------------------------------------------------
>
> Key: CRUNCH-670
> URL: https://issues.apache.org/jira/browse/CRUNCH-670
> Project: Crunch
> Issue Type: Improvement
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Major
> Fix For: 1.0.0
>
> Attachments: CRUNCH-670-pt2.patch, CRUNCH-670.patch
>
>
> There is an issue where the AvroPathPerKeyTarget won't properly copy the
> output of a Spark pipeline from the temp directory to the target directory
> because it assumes it will always get a valid Crunch output index (0, 1, 2,
> ...) and the SparkRuntime passes -1 to the Target's output handler method (to
> signal that it's the only output for the job.) I _think_ the right move is to
> have AvroPathPerKeyTarget rewrite a -1 index to 0 so as not to break any
> other implementations that depend on the SparkRuntime's behavior.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
