[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274758#comment-14274758 ] Brock Noland commented on HIVE-9257: Thank you cleaning up my mess! :) > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274490#comment-14274490 ] Szehon Ho commented on HIVE-9257: - [~gopalv] [~brocknoland] tests passed for HIVE-9340 and I submitted a new merge to trunk in HIVE-9352. Will merge as soon as possible. Sorry for inconvenience. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274399#comment-14274399 ] Brock Noland commented on HIVE-9257: Ahh yes, we discussed today that I forgot that in my patch. Nevermind me. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274390#comment-14274390 ] Szehon Ho commented on HIVE-9257: - Yea I was going to reply earlier to Brock. I submitted it for removal it now in HIVE-9340, it is one of the other cleanups, thats why I wanted to collect them and then merge. I might be able to post a patch this evening. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[
https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274386#comment-14274386
]
Gopal V commented on HIVE-9257:
---
Thanks, [~brocknoland].
I traced the ec2 fetching to the root pom.xml, which has
{code}
spark-snapshot
http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark_2.10-1.2-SNAPSHOT/
...
{code}
I don't see that removed with HIVE-7674 (or in the branch) -
https://github.com/apache/hive/blob/spark/pom.xml#L217
> Merge from spark to trunk January 2015
> --
>
> Key: HIVE-9257
> URL: https://issues.apache.org/jira/browse/HIVE-9257
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: 0.15.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Labels: TODOC15
> Fix For: 0.15.0
>
> Attachments: trunk-mr2-spark-merge.properties
>
>
> The hive on spark work has reached a point where we can merge it into the
> trunk branch. Note that spark execution engine is optional and no current
> users should be impacted.
> This JIRA will be used to track the merge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274314#comment-14274314 ] Brock Noland commented on HIVE-9257: [~szehon] - do you think we could merge just that one change? > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274284#comment-14274284 ] Szehon Ho commented on HIVE-9257: - Yea that will be removed in next merge. It's already checked in for HIVE-7674 and will be done in HIVE-9352 > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[
https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274279#comment-14274279
]
Gopal V commented on HIVE-9257:
---
Can you remove all the EC2 random references from the poms? My builds are
trying to pull hive JARs from some random EC2 hosts.
{code}
Downloading:
http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark_2.10-1.2-SNAPSHOT/org/apache/hive/hive-service/0.15.0-SNAPSHOT/maven-metadata.xml
{code}
> Merge from spark to trunk January 2015
> --
>
> Key: HIVE-9257
> URL: https://issues.apache.org/jira/browse/HIVE-9257
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: 0.15.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Labels: TODOC15
> Fix For: 0.15.0
>
> Attachments: trunk-mr2-spark-merge.properties
>
>
> The hive on spark work has reached a point where we can merge it into the
> trunk branch. Note that spark execution engine is optional and no current
> users should be impacted.
> This JIRA will be used to track the merge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274185#comment-14274185 ] Szehon Ho commented on HIVE-9257: - Thanks, +1 on moving the link to under AdminDocs. There might be some parts that should be in UserDocs, but at this point I am not sure which (if any) to put in there. Maybe some of the recommended configuration, as other configurations are under "UserDocs". > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274161#comment-14274161 ] Szehon Ho commented on HIVE-9257: - Yea sorry thats the way it went about the weekend. We made the JIRA on Monday for people to know the intention. The code cleanup took all week, longer than I anticipated due to state of spark-branch, and first patch was linked early on Friday during the workday. And as Brock said, it is really hard to merge such a big branch during the week when trunk is in constant motion (tons of conflicts), making the weekend the only viable choice to do it. Also its my fault about missing the snapshot as part of the patch, it is fixed already in HIVE 7674 last week and will be merged in next merge (I missed it not being in this merge). I could fix it immediately too, but was going to collect some more fixes and then merge by tomorrow/Wed. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274100#comment-14274100 ] Brock Noland commented on HIVE-9257: Also, obviously, if there is any additional feedback on the work, we'd be happy discuss. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274088#comment-14274088 ] Brock Noland commented on HIVE-9257: Hi Sergey, The change was created a many days ago. It's exceedingly hard to do a merge during the week due to commit volume. W.r.t Spark SNAPSHOT, we have depended on Calcite SNAPSHOT for a long time, even sadly in the 0.14.0 release. However, I agree a SNAPSHOT dep in trunk is not ideal and we actually have a change to resolve this committed to spark and we'll be merging that soon. I actually thought that was getting merged this time. Brock > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274083#comment-14274083 ] Sergey Shelukhin commented on HIVE-9257: It's not very nice to do such a large branch merge over the weekend with no advance warning. This appears to depend on spark build snapshot; this is a bad practice in general, and I don't think it would be acceptable to have a Hive release with such dependency. Is it possible to change to a released version of Spark? > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273295#comment-14273295 ] Szehon Ho commented on HIVE-9257: - Thanks Brock. FYI, I am planning another merge in next few days to incorporate the review fixes, and also to get rid of the spark-snapshot dependency. It will hopefully be a more managable patch that can uploaded this time and tested the normal way :) > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273258#comment-14273258 ] Brock Noland commented on HIVE-9257: I also created HIVE-9344 to fix that optimize_nullscan tests that are so flaky. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273256#comment-14273256 ] Brock Noland commented on HIVE-9257: The only test which failed and is not flaky is was the windowing test. I created HIVE-9343 to track the issue. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273223#comment-14273223 ] Brock Noland commented on HIVE-9257: FYI - I am running a build here: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-HADOOP-2/1/ to ensure all the tests are passing post merge. If there is any failing tests, due to the merge, we should address them ASAP tomorrow. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273209#comment-14273209 ] Szehon Ho commented on HIVE-9257: - Thanks I noticed, was collecting those as well as some other items I saw: HIVE-9340 > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273177#comment-14273177 ] Xuefu Zhang commented on HIVE-9257: --- Actually my comments on RB were not covered by HIVE-9335, which already has +1 pending. We may need a separate JIRA to cover them. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272823#comment-14272823 ] Szehon Ho commented on HIVE-9257: - Latest test results looks much better: [http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build-Upload/6/|http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build-Upload/6/] windowing.q has been failing for spark for some time, and is tracked to be fixed (HIVE-9104), should be soon. The other test failure does not look related. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: trunk-mr2-spark-merge.properties > > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272801#comment-14272801 ] Xuefu Zhang commented on HIVE-9257: --- I reviewed most of patches in Spark branch during the months, and also produced some. I reviewed the mega patch here, and left a couple of comments on RB. However, these can be addressed as followups. +1 pending on test. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272800#comment-14272800 ] Szehon Ho commented on HIVE-9257: - For reference, URL of new patch. Only difference is that the new spark.out files have updated file sizes: alter_merge_orc.q.out, alter_merge_stats_orc.q.out, and vectorized_ptf.q.out. [http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark-to-trunk-merge/HIVE-9257.3.patch|http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark-to-trunk-merge/HIVE-9257.3.patch] > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272789#comment-14272789 ] Szehon Ho commented on HIVE-9257: - Analyzing the test failures, the orc file sizes are reported different per OS, so regenerating some golden files on Ubuntu and trying again. This should probably be looked at. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272776#comment-14272776 ] Szehon Ho commented on HIVE-9257: - Yea to get [http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build-Upload/5/|http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build-Upload/5/] to run spark tests, I had to create a new build profile (trunk-mr2-spark-merge) with the spark test-properties. Its a chicken-egg problem (see HIVE-7801), as I tried changing the main build properties (trunk-mr2) but that screwed up all the other precommit builds Anyway the last step post merge will be putting the spark properties in trunk-mr2. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272774#comment-14272774 ] Brock Noland commented on HIVE-9257: I am very familiar with this work. We have some follow-on jiras like HIVE-9335 but that shouldn't hold the merge as we've done in the past. +1 pending tests to merge this to trunk > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272754#comment-14272754 ] Brock Noland commented on HIVE-9257: I created HIVE-9335 and put a patch up for the review items I saw. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272751#comment-14272751 ] Brock Noland commented on HIVE-9257: Nice, only seven failed tests: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build-Upload/5/ > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272350#comment-14272350 ] Szehon Ho commented on HIVE-9257: - For reference, patch is uploaded here: [http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark-to-trunk-merge/HIVE-9257.patch|http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark-to-trunk-merge/HIVE-9257.patch] to get around the JIRA 10MB restriction. Ran a separate build that uses this patch, and the existing tests are passing: [http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build-Upload/lastCompletedBuild/testReport/|http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build-Upload/lastCompletedBuild/testReport/]. But spark tests are not running as ptest-master is missing spark test properties. Going to reconfigure the build machine with sparkCliDriver properties and give it another spin. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272107#comment-14272107 ] Szehon Ho commented on HIVE-9257: - Adding review request: [https://reviews.apache.org/r/29787/|https://reviews.apache.org/r/29787/]. As the entire patch is too big, this shows the modified files. These have been cleanuped as part of HIVE-9319, HIVE-9306, HIVE-9305. The new files can be found here: [http://svn.apache.org/repos/asf/hive/branches/spark/|http://svn.apache.org/repos/asf/hive/branches/spark/] or [https://github.com/apache/hive/tree/spark|https://github.com/apache/hive/tree/spark] under: data/conf/spark/ itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithLocalClusterSpark.java itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestMultiSessionsHS2WithLocalClusterSpark.java itests/qtest-spark/ ql/src/java/org/apache/hadoop/hive/ql/exec/SparkHashTableSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ ql/src/java/org/apache/hadoop/hive/ql/lib/TypeRule.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/SparkMapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenSparkSkewJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkCrossProductCheck.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/ ql/src/java/org/apache/hadoop/hive/ql/parse/spark/ ql/src/java/org/apache/hadoop/hive/ql/plan/SparkBucketMapJoinContext.java ql/src/java/org/apache/hadoop/hive/ql/plan/SparkEdgeProperty.java ql/src/java/org/apache/hadoop/hive/ql/plan/SparkHashTableSinkDesc.java ql/src/java/org/apache/hadoop/hive/ql/plan/SparkWork.java ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorSpark.java ql/src/test/org/apache/hadoop/hive/ql/exec/spark/ ql/src/test/queries/clientpositive/auto_join_stats.q ql/src/test/queries/clientpositive/auto_join_stats2.q ql/src/test/queries/clientpositive/bucket_map_join_spark1.q ql/src/test/queries/clientpositive/bucket_map_join_spark2.q ql/src/test/queries/clientpositive/bucket_map_join_spark3.q ql/src/test/queries/clientpositive/bucket_map_join_spark4.q ql/src/test/queries/clientpositive/multi_insert_mixed.q ql/src/test/queries/clientpositive/multi_insert_union_src.q ql/src/test/queries/clientpositive/parallel_join0.q ql/src/test/queries/clientpositive/parallel_join1.q ql/src/test/queries/clientpositive/spark_test.q ql/src/test/queries/clientpositive/udf_example_add.q ql/src/test/results/clientpositive/auto_join_stats.q.out ql/src/test/results/clientpositive/auto_join_stats2.q.out ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out ql/src/test/results/clientpositive/multi_insert_mixed.q.out ql/src/test/results/clientpositive/multi_insert_union_src.q.out ql/src/test/results/clientpositive/parallel_join0.q.out ql/src/test/results/clientpositive/parallel_join1.q.out ql/src/test/results/clientpositive/spark/ ql/src/test/results/clientpositive/spark_test.q.out ql/src/test/results/clientpositive/udf_example_add.q.out spark-client/ Cleanup and review of those have been done as part of HIVE-9281 and HIVE-9288. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14271918#comment-14271918 ] Szehon Ho commented on HIVE-9257: - Actually this is not going to work, the other patch is still larger. Need to think of another way. > Merge from spark to trunk January 2015 > -- > > Key: HIVE-9257 > URL: https://issues.apache.org/jira/browse/HIVE-9257 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 0.15.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > The hive on spark work has reached a point where we can merge it into the > trunk branch. Note that spark execution engine is optional and no current > users should be impacted. > This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
