[jira] [Updated] (PIG-4173) Move to Spark 1.x
[ https://issues.apache.org/jira/browse/PIG-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Praveen Rachabattuni updated PIG-4173: -- Fix Version/s: spark-branch > Move to Spark 1.x > - > > Key: PIG-4173 > URL: https://issues.apache.org/jira/browse/PIG-4173 > Project: Pig > Issue Type: Sub-task > Components: spark >Reporter: bc Wong >Assignee: Richard Ding > Fix For: spark-branch > > Attachments: PIG-4173.patch, PIG-4173_2.patch, PIG-4173_3.patch, > PIG-4174_4.patch, PIG-4174_5.patch, TEST-org.apache.pig.spark.TestSpark.txt > > > The Spark branch is using Spark 0.9: > https://github.com/apache/pig/blob/spark/ivy.xml#L438. We should probably > switch to Spark 1.x asap, due to Spark interface changes since 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PIG-4173) Move to Spark 1.x
[ https://issues.apache.org/jira/browse/PIG-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-4173: -- Attachment: PIG-4174_5.patch This patch fixed the cogroup issue for Spark 1.1.0. Spark version is updated to 1.1.0. > Move to Spark 1.x > - > > Key: PIG-4173 > URL: https://issues.apache.org/jira/browse/PIG-4173 > Project: Pig > Issue Type: Sub-task > Components: spark >Reporter: bc Wong >Assignee: Richard Ding > Attachments: PIG-4173.patch, PIG-4173_2.patch, PIG-4173_3.patch, > PIG-4174_4.patch, PIG-4174_5.patch, TEST-org.apache.pig.spark.TestSpark.txt > > > The Spark branch is using Spark 0.9: > https://github.com/apache/pig/blob/spark/ivy.xml#L438. We should probably > switch to Spark 1.x asap, due to Spark interface changes since 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PIG-4173) Move to Spark 1.x
[ https://issues.apache.org/jira/browse/PIG-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-4173: -- Attachment: PIG-4174_4.patch This patch fixed the unit tests. The version of Spark used is 1.0.2. In Spark 1.1.0, the CoGroupRDD is changed and breaks the cogroup runtime. I'm looking into this. > Move to Spark 1.x > - > > Key: PIG-4173 > URL: https://issues.apache.org/jira/browse/PIG-4173 > Project: Pig > Issue Type: Sub-task > Components: spark >Reporter: bc Wong >Assignee: Richard Ding > Attachments: PIG-4173.patch, PIG-4173_2.patch, PIG-4173_3.patch, > PIG-4174_4.patch, TEST-org.apache.pig.spark.TestSpark.txt > > > The Spark branch is using Spark 0.9: > https://github.com/apache/pig/blob/spark/ivy.xml#L438. We should probably > switch to Spark 1.x asap, due to Spark interface changes since 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PIG-4173) Move to Spark 1.x
[ https://issues.apache.org/jira/browse/PIG-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-4173: -- Attachment: PIG-4173_3.patch Thanks for the review. The new patch incorporate the changes in the comments. > Move to Spark 1.x > - > > Key: PIG-4173 > URL: https://issues.apache.org/jira/browse/PIG-4173 > Project: Pig > Issue Type: Sub-task > Components: spark >Reporter: bc Wong >Assignee: Richard Ding > Attachments: PIG-4173.patch, PIG-4173_2.patch, PIG-4173_3.patch, > TEST-org.apache.pig.spark.TestSpark.txt > > > The Spark branch is using Spark 0.9: > https://github.com/apache/pig/blob/spark/ivy.xml#L438. We should probably > switch to Spark 1.x asap, due to Spark interface changes since 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PIG-4173) Move to Spark 1.x
[ https://issues.apache.org/jira/browse/PIG-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Praveen Rachabattuni updated PIG-4173: -- Attachment: TEST-org.apache.pig.spark.TestSpark.txt Hi Richard, I just found some of unit tests from PIG-4168 doesn't pass after the Spark-1.x upgrade. Please check attached log. I believe It would good to have this unit tests passing before the merge. > Move to Spark 1.x > - > > Key: PIG-4173 > URL: https://issues.apache.org/jira/browse/PIG-4173 > Project: Pig > Issue Type: Sub-task > Components: spark >Reporter: bc Wong >Assignee: Richard Ding > Attachments: PIG-4173.patch, PIG-4173_2.patch, > TEST-org.apache.pig.spark.TestSpark.txt > > > The Spark branch is using Spark 0.9: > https://github.com/apache/pig/blob/spark/ivy.xml#L438. We should probably > switch to Spark 1.x asap, due to Spark interface changes since 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PIG-4173) Move to Spark 1.x
[ https://issues.apache.org/jira/browse/PIG-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-4173: -- Attachment: PIG-4173_2.patch Adding javax.servlet dependency > Move to Spark 1.x > - > > Key: PIG-4173 > URL: https://issues.apache.org/jira/browse/PIG-4173 > Project: Pig > Issue Type: Sub-task > Components: spark >Reporter: bc Wong >Assignee: Richard Ding > Attachments: PIG-4173.patch, PIG-4173_2.patch > > > The Spark branch is using Spark 0.9: > https://github.com/apache/pig/blob/spark/ivy.xml#L438. We should probably > switch to Spark 1.x asap, due to Spark interface changes since 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PIG-4173) Move to Spark 1.x
[ https://issues.apache.org/jira/browse/PIG-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-4173: -- Attachment: PIG-4173.patch Attaching the initial patch to upgrade Spark to 1.1.0. I made some local changes so that the patch now compiles with the latest Spark jar. I have a question though: why don't we use JavaRDD throughout the code? Is this due to performance concerns? > Move to Spark 1.x > - > > Key: PIG-4173 > URL: https://issues.apache.org/jira/browse/PIG-4173 > Project: Pig > Issue Type: Sub-task > Components: spark >Reporter: bc Wong >Assignee: Richard Ding > Attachments: PIG-4173.patch > > > The Spark branch is using Spark 0.9: > https://github.com/apache/pig/blob/spark/ivy.xml#L438. We should probably > switch to Spark 1.x asap, due to Spark interface changes since 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PIG-4173) Move to Spark 1.x
[ https://issues.apache.org/jira/browse/PIG-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bc Wong updated PIG-4173: - Component/s: spark > Move to Spark 1.x > - > > Key: PIG-4173 > URL: https://issues.apache.org/jira/browse/PIG-4173 > Project: Pig > Issue Type: Sub-task > Components: spark >Reporter: bc Wong > > The Spark branch is using Spark 0.9: > https://github.com/apache/pig/blob/spark/ivy.xml#L438. We should probably > switch to Spark 1.x asap, due to Spark interface changes since 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
