Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9480
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155537739
I think I can resolve the conflicts manually. Merging to master and 1.6.
Thanks!
---
If your project is set up for it, you can reply to this email and have yo
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155282055
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155282054
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155281866
**[Test build #45468 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45468/consoleFull)**
for PR 9480 at commit
[`6cf0186`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155259441
**[Test build #45468 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45468/consoleFull)**
for PR 9480 at commit
[`6cf0186`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155259185
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155259218
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155258413
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155250085
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155250029
**[Test build #45443 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45443/consoleFull)**
for PR 9480 at commit
[`6cf0186`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155236703
**[Test build #45443 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45443/consoleFull)**
for PR 9480 at commit
[`6cf0186`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155236517
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155236534
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155185293
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155185089
**[Test build #45384 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45384/consoleFull)**
for PR 9480 at commit
[`5dbc047`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155145415
**[Test build #45384 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45384/consoleFull)**
for PR 9480 at commit
[`5dbc047`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155143767
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155143818
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154599464
>
I'm still not sure how you are suggesting we execute the version you are
proposing. I'd guess you do that by adding an additional project that creates a
new
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44205755
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression ex
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154590883
I don't think the most recent failure is your fault.
@rxin, did you patch break the build?
```
File
"/home/jenkins/workspace/NewSparkPullRequestBui
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154590257
**[Test build #1999 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1999/consoleFull)**
for PR 9480 at commit
[`3a10f7d`](https://gi
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154586930
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154586827
**[Test build #45263 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45263/consoleFull)**
for PR 9480 at commit
[`3a10f7d`](https://git
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154585715
This is awesome. LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have th
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44202414
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -135,6 +147,7 @@ abstract class Expression extends T
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44202061
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala
---
@@ -275,8 +275,9 @@ object Genera
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44201792
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
---
@@ -0,0 +1,108 @@
+/*
+ * Licensed t
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44201615
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression extends
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44201576
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression extends
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44200916
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class AttributeReferenc
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154570265
**[Test build #1999 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1999/consoleFull)**
for PR 9480 at commit
[`3a10f7d`](https://git
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154570059
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled an
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44197514
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class AttributeReference(
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154567189
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154567121
**[Test build #45259 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45259/consoleFull)**
for PR 9480 at commit
[`e65def9`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154562314
**[Test build #45263 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45263/consoleFull)**
for PR 9480 at commit
[`3a10f7d`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154561767
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154561748
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44194583
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class AttributeReference(
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154554772
Thanks for the numbers, that's cool.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154554001
**[Test build #45259 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45259/consoleFull)**
for PR 9480 at commit
[`e65def9`](https://gith
Github user nongli commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154553190
I ran this benchmark:
```
val data = sqlContext.range(20 * 1024 *
1024).toDF("v").registerTempTable("t")
for (i <- 0 until 4) {
val t1 =
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44191412
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class AttributeReference(
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154551510
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154551545
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44190675
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class AttributeReference(
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44190096
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -341,7 +440,18 @@ abstract class CodeGenera
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154518695
This looks pretty good to me, just a few minor comments. Should we have a
benchmark to see is there any regression for tiny common subexpression? If not
much, then we don
Github user nongli commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154518082
> In general, I think we'd better add more IR (Intermediate Representation,
should be the Expression or Logical Plan node here) for common sub expression
cases, and we th
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44179462
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression extends Tr
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44178151
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -341,7 +440,18 @@ abstract class CodeGenera
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44177613
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44177472
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class AttributeReference(
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44108481
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression ex
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154316345
In general, I think we'd better add more IR (Intermediate Representation,
should be the Expression or Logical Plan node here) for common sub expression
cases, and
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44106783
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression ex
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44105110
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression extends Tr
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154288464
I took one pass. Overall pretty good! @davies Can you also take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on Git
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44099965
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression ex
Github user nongli commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154137436
@chenghao-intel I'm not sure what you are suggesting. How do you suggest we
execute the expressions? The transformation you have as an example makes sense
but how does it
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44043807
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression extends Tr
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153948155
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153948156
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153948080
**[Test build #45075 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45075/consoleFull)**
for PR 9480 at commit
[`2feafbc`](https://git
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153926546
My 2 cents, Common Expression Elimination is a very interesting/useful
optimization, and I did the same thing for Shark quite long time ago, for my
understanding,
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153925305
**[Test build #45075 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45075/consoleFull)**
for PR 9480 at commit
[`2feafbc`](https://gith
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r43966655
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression ex
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153924518
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153924543
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user nongli opened a pull request:
https://github.com/apache/spark/pull/9480
[SPARK-10371] [SQL] Implement subexpr elimination for UnsafeProjections
This patch adds the building blocks for codegening subexpr elimination and
implements
it end to end for UnsafeProjection. T
72 matches
Mail list logo