Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10266#discussion_r47428568
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -1029,24 +1029,27 @@ case class ScalaUDF(
/
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10266
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/10266#discussion_r47392510
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -1029,24 +1029,27 @@ case class ScalaUDF(
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-164022078
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-164022075
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-164021742
**[Test build #47585 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47585/consoleFull)**
for PR 10266 at commit
[`2125a1b`](https://g
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10266#discussion_r47391422
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -1029,24 +1029,27 @@ case class ScalaUDF(
// su
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-164020361
**[Test build #2205 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2205/consoleFull)**
for PR 10266 at commit
[`2125a1b`](https://
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10266#discussion_r47391269
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -1029,24 +1029,27 @@ case class ScalaUDF(
// su
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/10266#discussion_r47391109
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -1029,24 +1029,27 @@ case class ScalaUDF(
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/10266#discussion_r47389510
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -1029,24 +1029,27 @@ case class ScalaUDF(
// s
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/10266#discussion_r47388587
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -1029,24 +1029,27 @@ case class ScalaUDF(
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-164009410
**[Test build #2204 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2204/consoleFull)**
for PR 10266 at commit
[`2125a1b`](https://
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163998492
**[Test build #47585 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47585/consoleFull)**
for PR 10266 at commit
[`2125a1b`](https://gi
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163997902
**[Test build #2205 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2205/consoleFull)**
for PR 10266 at commit
[`2125a1b`](https://g
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163997363
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featur
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163996602
**[Test build #47584 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47584/consoleFull)**
for PR 10266 at commit
[`2125a1b`](https://g
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163996672
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163996670
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163986683
**[Test build #47584 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47584/consoleFull)**
for PR 10266 at commit
[`2125a1b`](https://gi
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163983570
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featur
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163984799
**[Test build #2204 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2204/consoleFull)**
for PR 10266 at commit
[`2125a1b`](https://g
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10266#discussion_r47374514
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -1029,24 +1029,27 @@ case class ScalaUDF(
// su
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163972430
Works for me. Thanks, guys!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does no
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10266#discussion_r47351058
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -1029,24 +1029,27 @@ case class ScalaUDF(
/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163903666
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163903664
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163903552
**[Test build #47578 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47578/consoleFull)**
for PR 10266 at commit
[`2125a1b`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163879143
**[Test build #47578 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47578/consoleFull)**
for PR 10266 at commit
[`2125a1b`](https://gi
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163876482
@cloud-fan @markhamstra They should be all fixed (handling null in
arguments and results).
---
If your project is set up for it, you can reply to this email and have yo
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163874386
hi @markhamstra , can you add some log in your UDF, to see if the NPE
occurred before run into your UDF code or after?
---
If your project is set up for it, you can
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163868758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163868667
**[Test build #47574 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47574/consoleFull)**
for PR 10266 at commit
[`c96b512`](https://g
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163868756
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163867183
**[Test build #2202 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2202/consoleFull)**
for PR 10266 at commit
[`c96b512`](https://
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163862856
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163862860
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163862544
**[Test build #47572 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47572/consoleFull)**
for PR 10266 at commit
[`c0f85bb`](https://g
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163861470
@davies The exact same UDF worked fine in 1.5.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If yo
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163860387
@markhamstra I think it's because of your UDF did not handle null correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163858860
Still doesn't work for me. Now it ends up in a different place, but a NPE:
```
...
2015-12-11 06:48:09,285 INFO
org.apache.spark.sql.catalyst.expression
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163856161
@davies No -- see the other PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163855980
**[Test build #47574 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47574/consoleFull)**
for PR 10266 at commit
[`c96b512`](https://gi
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163855851
LGTM pending tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163855762
@markhamstra Once it works, I will merge this to unblock RC2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as w
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163855637
No problem; I'll cherry-pick another.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163855563
@markhamstra Sorry, just pushed a commit to fix it now, added a regression
test.
---
If your project is set up for it, you can reply to this email and have your
reply a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163855486
**[Test build #2202 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2202/consoleFull)**
for PR 10266 at commit
[`c96b512`](https://g
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163855481
@davies This results in a slightly different failure from the one I
previously reported:
Everything looks the same as the prior post except now:
```
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163855323
It's not a Janino bug, `(Integer)-1` does not work in Java, faint :-(
---
If your project is set up for it, you can reply to this email and have your
reply appear on Git
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163852763
I changed `int` to `Integer` and tried again ,the result is the same. And I
also tried `Integer i = (Integer) -1;` which also failed to compile. I think
the problem i
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163851041
Wenchen, should the type of i be Integer?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163848082
Could you try to add `(Integer)` before `null`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your p
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163847683
**[Test build #47572 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47572/consoleFull)**
for PR 10266 at commit
[`c0f85bb`](https://gi
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/10266#issuecomment-163847721
I tried it locally, here is my findings:
* `int i = false ? null : (Integer) -1;` **doesn't compile**
* `int i = false ? null : (Integer) 1;` compiles
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163846453
@markhamstra Does https://github.com/apache/spark/pull/10266 fix it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitH
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/10266
[SPARK-12258] [SQL] passing null into ScalaUDF (follow-up)
This is a follow-up PR for #10259
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dav
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163844582
hi @markhamstra , can you share you test code? so that we can reproduce it,
thanks!
---
If your project is set up for it, you can reply to this email and have your
r
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163843237
Sorry, but after this I am now seeing codeGen errors. Like this:
```
testTimestampArithmeticUDFs(com.clearstorydata.dataengine.udfs.TestTimestampArithmeticU
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10259
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163802039
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163802040
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163801918
**[Test build #47552 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47552/consoleFull)**
for PR 10259 at commit
[`4636fe3`](https://g
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163801607
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabl
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163797546
LGTM pending jenkins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this f
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163796308
I tried use `java.lang` types and the result look good.
@davies how about we update our doc (the `Data Types` section) to explain
that those primitive types are n
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163794997
Ok, I see your point. This is a possible workaround. It works well when the
input values of primitive types are not null.
I am just afraid how users know th
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163792666
@gatorsmile Because your UDF is using primitive types, we have to chance to
pass `null` in. In order to got the behavior you expect, you should change the
UDF to use box
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163792580
Yeah. That is my understanding.
```
inputsNullCheck.map(If(_, Literal.create(null, udf.dataType),
udf)).getOrElse(udf)
```
---
If your projec
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163790754
the analyzer rule was introduced by
https://github.com/apache/spark/pull/9770 ?
---
If your project is set up for it, you can reply to this email and have your
reply app
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163788875
Yeah. Below is the result of
```scala
sql("SELECT * FROM test").show();
```
```
+++---+++++++++
| ti|
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163788347
Does `INSERT INTO TABLE test VALUES` actually work?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If you
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163787742
**[Test build #47552 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47552/consoleFull)**
for PR 10259 at commit
[`4636fe3`](https://gi
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163787044
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163787017
Thank you @davies !
I guess we might still have a bug in the code. As long as any input
variable is Null, the return result is `null`. Is that by design?
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163787042
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/10259#issuecomment-163786748
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featur
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/10259
[SPARK-12258] [SQL] passing null into ScalaUDF
Check nullability and passing them into ScalaUDF.
Closes #10249
You can merge this pull request into a Git repository by running:
$ git
78 matches
Mail list logo
