[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8492 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135894603 Merged into master and 1.5 branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135894175 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135887382 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41753/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135887262 [Test build #41753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41753/console) for PR 8492 at commit [`32ddc9f`](https://github.com/apache/spark/commit/32ddc9f5ee870a44b87ec1402c96ef404d2c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135887380 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135871271 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135871275 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41751/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135871119 [Test build #41751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41751/console) for PR 8492 at commit [`6887185`](https://github.com/apache/spark/commit/68871851a3ea7343b28cfaf9087964f3ef1c6a93). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135858114 [Test build #41753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41753/consoleFull) for PR 8492 at commit [`32ddc9f`](https://github.com/apache/spark/commit/32ddc9f5ee870a44b87ec1402c96ef404d2c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135856198 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135856179 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8492#discussion_r38229646
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -145,46 +143,42 @@ case class ArrayContains(left: Expression, right:
Expression)
}
}
- override def nullable: Boolean = false
+ override def nullable: Boolean = {
+left.nullable || right.nullable ||
left.dataType.asInstanceOf[ArrayType].containsNull
+ }
- override def eval(input: InternalRow): Boolean = {
-val arr = left.eval(input)
-if (arr == null) {
- false
-} else {
- val value = right.eval(input)
- if (value == null) {
-false
- } else {
-arr.asInstanceOf[ArrayData].foreach(right.dataType, (i, v) =>
- if (v == value) return true
-)
-false
+ override def nullSafeEval(arr: Any, value: Any): Any = {
+var hasNull = false
+arr.asInstanceOf[ArrayData].foreach(right.dataType, (i, v) =>
+ if (v == null) {
+hasNull = true
+ } else if (v == value) {
+return true
}
+)
+if (hasNull) {
+ null
+} else {
+ false
}
}
override def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode):
String = {
-val arrGen = left.gen(ctx)
-val elementGen = right.gen(ctx)
-val i = ctx.freshName("i")
-val getValue = ctx.getValue(arrGen.primitive, right.dataType, i)
-s"""
- ${arrGen.code}
- boolean ${ev.isNull} = false;
- boolean ${ev.primitive} = false;
- if (!${arrGen.isNull}) {
-${elementGen.code}
-if (!${elementGen.isNull}) {
- for (int $i = 0; $i < ${arrGen.primitive}.numElements(); $i ++) {
-if (${ctx.genEqual(right.dataType, elementGen.primitive,
getValue)}) {
- ${ev.primitive} = true;
- break;
-}
- }
+nullSafeCodeGen(ctx, ev, (arr, value) => {
--- End diff --
nullSafeCodeGen is similar to defineCodeGen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8492#discussion_r38229685
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ---
@@ -382,15 +378,13 @@ class DataFrameFunctionsSuite extends QueryTest with
SharedSQLContext {
df.selectExpr("array_contains(null, 1)")
}
-// In hive, if either argument has a matching type has a null value,
return false, even if
-// the first argument array contains a null and the second argument is
null
checkAnswer(
- df.selectExpr("array_contains(array(array(1), null)[1], 1)"),
- Seq(Row(false), Row(false))
+ df.selectExpr("array_contains(array(array(1), null)[0], 1)"),
+ Seq(Row(true), Row(true))
--- End diff --
Will add this to CollectionFunctionSuite
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135854641 +1 to follow postgres. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8492#discussion_r38229098
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ---
@@ -382,15 +378,13 @@ class DataFrameFunctionsSuite extends QueryTest with
SharedSQLContext {
df.selectExpr("array_contains(null, 1)")
}
-// In hive, if either argument has a matching type has a null value,
return false, even if
-// the first argument array contains a null and the second argument is
null
checkAnswer(
- df.selectExpr("array_contains(array(array(1), null)[1], 1)"),
- Seq(Row(false), Row(false))
+ df.selectExpr("array_contains(array(array(1), null)[0], 1)"),
+ Seq(Row(true), Row(true))
--- End diff --
Do you want to test something like `array_contains(null, 1)`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8492#discussion_r38227265
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
---
@@ -269,15 +269,5 @@ class ConstantFoldingSuite extends PlanTest {
testRelation
.select('a)
.where(In(Literal(1), Seq(Literal(1), 'a.attr)))
--- End diff --
Remove this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8492#discussion_r38225619
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -145,46 +143,42 @@ case class ArrayContains(left: Expression, right:
Expression)
}
}
- override def nullable: Boolean = false
+ override def nullable: Boolean = {
+left.nullable || right.nullable ||
left.dataType.asInstanceOf[ArrayType].containsNull
+ }
- override def eval(input: InternalRow): Boolean = {
-val arr = left.eval(input)
-if (arr == null) {
- false
-} else {
- val value = right.eval(input)
- if (value == null) {
-false
- } else {
-arr.asInstanceOf[ArrayData].foreach(right.dataType, (i, v) =>
- if (v == value) return true
-)
-false
+ override def nullSafeEval(arr: Any, value: Any): Any = {
+var hasNull = false
+arr.asInstanceOf[ArrayData].foreach(right.dataType, (i, v) =>
+ if (v == null) {
+hasNull = true
+ } else if (v == value) {
+return true
}
+)
+if (hasNull) {
+ null
+} else {
+ false
}
}
override def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode):
String = {
-val arrGen = left.gen(ctx)
-val elementGen = right.gen(ctx)
-val i = ctx.freshName("i")
-val getValue = ctx.getValue(arrGen.primitive, right.dataType, i)
-s"""
- ${arrGen.code}
- boolean ${ev.isNull} = false;
- boolean ${ev.primitive} = false;
- if (!${arrGen.isNull}) {
-${elementGen.code}
-if (!${elementGen.isNull}) {
- for (int $i = 0; $i < ${arrGen.primitive}.numElements(); $i ++) {
-if (${ctx.genEqual(right.dataType, elementGen.primitive,
getValue)}) {
- ${ev.primitive} = true;
- break;
-}
- }
+nullSafeCodeGen(ctx, ev, (arr, value) => {
--- End diff --
Do we need to call {defineCodeGen}?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135836177 [Test build #41751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41751/consoleFull) for PR 8492 at commit [`6887185`](https://github.com/apache/spark/commit/68871851a3ea7343b28cfaf9087964f3ef1c6a93). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135834195 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135834146 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8492#discussion_r38219598
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -145,46 +143,43 @@ case class ArrayContains(left: Expression, right:
Expression)
}
}
- override def nullable: Boolean = false
+ override def nullable: Boolean = {
+val dt = right.dataType
+left.nullable || dt.isInstanceOf[ArrayType] &&
dt.asInstanceOf[ArrayType].containsNull
+ }
--- End diff --
Looks like we need to swap `left` and `right`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135711833 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135711841 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41733/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135711152 [Test build #41733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41733/console) for PR 8492 at commit [`5ca7823`](https://github.com/apache/spark/commit/5ca7823dea93b2341fdb2af6bf548812471ba48b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135658488 [Test build #41727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41727/console) for PR 8492 at commit [`3be8d05`](https://github.com/apache/spark/commit/3be8d054245a953a3db5314ce5edaa9ce8e97852). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135658527 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41727/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135658526 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135657968 [Test build #41733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41733/consoleFull) for PR 8492 at commit [`5ca7823`](https://github.com/apache/spark/commit/5ca7823dea93b2341fdb2af6bf548812471ba48b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135657152 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135657175 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135654582 [Test build #1701 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1701/console) for PR 8492 at commit [`4328e46`](https://github.com/apache/spark/commit/4328e46d479c21fd616b101d29dd7653c64a3e21). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135640246 I'd follow postgres here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135637490 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41725/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135637488 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135636727 [Test build #41727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41727/consoleFull) for PR 8492 at commit [`3be8d05`](https://github.com/apache/spark/commit/3be8d054245a953a3db5314ce5edaa9ce8e97852). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135636351 [Test build #1701 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1701/consoleFull) for PR 8492 at commit [`4328e46`](https://github.com/apache/spark/commit/4328e46d479c21fd616b101d29dd7653c64a3e21). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135636231 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135636238 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135635306 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135635297 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135634760 From PostgresSQL: ``` If the array expression yields a null array, the result of ANY will be null. If the left-hand expression yields null, the result of ANY is ordinarily null (though a non-strict comparison operator could possibly yield a different result). Also, if the right-hand array contains any null elements and no true comparison result is obtained, the result of ANY will be null, not false (again, assuming a strict comparison operator). This is in accordance with SQL's normal rules for Boolean combinations of null values. ``` It's more consistent in PostgresSQL, I'd like to follow it. cc @rxin @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135623342 OK. I guess the main question at here is if we want to have a different semantic with hive on `array_contains` regarding `null`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/8492#issuecomment-135621549
postgresql's output regarding `in`...
```
yhuai=# select cast(null as char(10)) in ('1', cast(null as char(10)));
?column?
--
(1 row)
yhuai=# select cast(null as char(10)) in ('1', cast(null as char(10))) is
null;
?column?
--
t
(1 row)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/8492#issuecomment-135620750
Here is the output of some sample tests using hive 1.2.1
```
hive> select cast(null as string) in ('1', cast(null as string));
OK
NULL
Time taken: 0.042 seconds, Fetched: 1 row(s)
hive> select array_contains(array('1', cast(null as string)), cast(null as
string));
OK
false
Time taken: 0.042 seconds, Fetched: 1 row(s)
hive>
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135591739 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135591740 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41712/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135591715 [Test build #41712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41712/console) for PR 8492 at commit [`c3c65f8`](https://github.com/apache/spark/commit/c3c65f864d2c39dc9bebd652cc009cfe56790c90). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135589035 [Test build #41712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41712/consoleFull) for PR 8492 at commit [`c3c65f8`](https://github.com/apache/spark/commit/c3c65f864d2c39dc9bebd652cc009cfe56790c90). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135588705 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135588718 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
GitHub user davies opened a pull request: https://github.com/apache/spark/pull/8492 [SPARK-10323] [SQL] fix nullability of In/InSet/ArrayContain After this PR, In/InSet/ArrayContain will return null if value is null, instead of false. They also will return null even if there is a null in the set/array. You can merge this pull request into a Git repository by running: $ git pull https://github.com/davies/spark fix_in Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8492.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8492 commit c3c65f864d2c39dc9bebd652cc009cfe56790c90 Author: Davies Liu Date: 2015-08-28T00:02:16Z fix nullability of In/InSet/ArrayContain --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10323] [SQL] fix nullability of In/InSe...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/8492#issuecomment-135588372 cc @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
