[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2857/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20973: [SPARK-20114][ML] spark.ml parity for sequential pattern...
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/20973 Jenkins, test this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21226 **[Test build #90114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90114/testReport)** for PR 21226 at commit [`e1491f0`](https://github.com/apache/spark/commit/e1491f0aeb62d1eda8cd8c55f890c8f87eec5761). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21190: [SPARK-22938][SQL][followup] Assert that SQLConf....
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21190#discussion_r185750124 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -102,22 +105,32 @@ object TypeCoercion { case (_: TimestampType, _: DateType) | (_: DateType, _: TimestampType) => Some(TimestampType) -case (t1 @ StructType(fields1), t2 @ StructType(fields2)) if t1.sameType(t2) => - Some(StructType(fields1.zip(fields2).map { case (f1, f2) => -// Since `t1.sameType(t2)` is true, two StructTypes have the same DataType -// except `name` (in case of `spark.sql.caseSensitive=false`) and `nullable`. -// - Different names: use f1.name -// - Different nullabilities: `nullable` is true iff one of them is nullable. -val dataType = findTightestCommonType(f1.dataType, f2.dataType).get -StructField(f1.name, dataType, nullable = f1.nullable || f2.nullable) - })) +case (t1 @ StructType(fields1), t2 @ StructType(fields2)) => + val isSameType = if (caseSensitive) { +t1.sameType(t2) --- End diff -- Call `DataType.equalsIgnoreNullability` here for better show the difference between the call of `DataType.equalsIgnoreCaseAndNullability` below? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21226 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2856/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90113 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90113/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21050: [SPARK-23912][SQL]add array_distinct
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21050#discussion_r185731095 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1059,3 +1059,96 @@ case class Flatten(child: Expression) extends UnaryExpression { override def prettyName: String = "flatten" } + +/** + * Removes duplicate values from the array. + */ +@ExpressionDescription( + usage = "_FUNC_(array) - Removes duplicate values from the array.", + examples = """ +Examples: + > SELECT _FUNC_(array(1, 2, 3, null, 3)); + [1,2,3,null] + """, since = "2.4.0") +case class ArrayDistinct(child: Expression) + extends UnaryExpression with ExpectsInputTypes { + + override def inputTypes: Seq[AbstractDataType] = Seq(ArrayType) + + override def dataType: DataType = child.dataType + + override def nullSafeEval(array: Any): Any = { +val elementType = child.dataType.asInstanceOf[ArrayType].elementType +val data = array.asInstanceOf[ArrayData].toArray[AnyRef](elementType).distinct +new GenericArrayData(data.asInstanceOf[Array[Any]]) + } + + override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { +val elementType = dataType.asInstanceOf[ArrayType].elementType +nullSafeCodeGen(ctx, ev, (array) => { + val arrayClass = classOf[GenericArrayData].getName + val distinctArray = ctx.freshName("distinctArray") + val i = ctx.freshName("i") + val j = ctx.freshName("j") + val pos = ctx.freshName("arrayPosition") + val getValue1 = CodeGenerator.getValue(array, elementType, i) + val getValue2 = CodeGenerator.getValue(array, elementType, j) + s""" + |int $pos = 0; + |for (int $i = 0; $i < $array.numElements(); $i ++) { --- End diff -- I guess nested loop must be very slow. Can you use a hash table or something to check duplication? Maybe `OpenHashSet`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21050: [SPARK-23912][SQL]add array_distinct
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21050#discussion_r185739950 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1059,3 +1059,96 @@ case class Flatten(child: Expression) extends UnaryExpression { override def prettyName: String = "flatten" } + +/** + * Removes duplicate values from the array. + */ +@ExpressionDescription( + usage = "_FUNC_(array) - Removes duplicate values from the array.", + examples = """ +Examples: + > SELECT _FUNC_(array(1, 2, 3, null, 3)); + [1,2,3,null] + """, since = "2.4.0") +case class ArrayDistinct(child: Expression) + extends UnaryExpression with ExpectsInputTypes { + + override def inputTypes: Seq[AbstractDataType] = Seq(ArrayType) + + override def dataType: DataType = child.dataType + + override def nullSafeEval(array: Any): Any = { +val elementType = child.dataType.asInstanceOf[ArrayType].elementType +val data = array.asInstanceOf[ArrayData].toArray[AnyRef](elementType).distinct --- End diff -- Do we need to keep the element order? If so, does `distinct` keep the order? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21225 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90102/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21226 **[Test build #90102 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90102/testReport)** for PR 21226 at commit [`e1491f0`](https://github.com/apache/spark/commit/e1491f0aeb62d1eda8cd8c55f890c8f87eec5761). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90103/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90103 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90103/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90106/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90106/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21069#discussion_r185726735 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -691,6 +691,30 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSQLContext { } } + test("array remove") { +val df = Seq( + (Array[Int](2, 1, 2, 3), Array("a", "b", "c", "a"), Array("", "")), + (Array.empty[Int], Array.empty[String], Array.empty[String]), + (null, null, null) +).toDF("a", "b", "c") +checkAnswer( + df.select(array_remove(df("a"), 2), array_remove(df("b"), "a"), array_remove(df("c"), "")), --- End diff -- maybe `$"a"` form is preferred to `df("a")`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21069#discussion_r185723637 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -883,3 +883,70 @@ case class Concat(children: Seq[Expression]) extends Expression { override def sql: String = s"concat(${children.map(_.sql).mkString(", ")})" } + +/** + * Remove all elements that equal to element from the given array + */ +@ExpressionDescription( + usage = "_FUNC_(array, element) - Remove all elements that equal to element from array.", + examples = """ +Examples: + > SELECT _FUNC_(array(1, 2, 3, null, 3), 3); + [1,2,null] + """, since = "2.4.0") +case class ArrayRemove(left: Expression, right: Expression) + extends BinaryExpression with ImplicitCastInputTypes with CodegenFallback { + + override def dataType: DataType = left.dataType + + override def inputTypes: Seq[AbstractDataType] = +Seq(ArrayType, left.dataType.asInstanceOf[ArrayType].elementType) + + override def nullSafeEval(arr: Any, value: Any): Any = { +val elementType = left.dataType.asInstanceOf[ArrayType].elementType +val data = arr.asInstanceOf[ArrayData].toArray[AnyRef](elementType).filter(_ != value) +new GenericArrayData(data.asInstanceOf[Array[Any]]) + } + + override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { +val elementType = left.dataType.asInstanceOf[ArrayType].elementType +nullSafeCodeGen(ctx, ev, (arr, value) => { + val arrayClass = classOf[GenericArrayData].getName + val values = ctx.freshName("values") + val i = ctx.freshName("i") + val pos = ctx.freshName("arrayPosition") + val numsToRemove = ctx.freshName("newArrLen") + val getValue = CodeGenerator.getValue(arr, right.dataType, i) + s""" + |int $pos = 0; + |int $numsToRemove = 0; + |Object[] $values; + | + |for (int $i = 0; $i < $arr.numElements(); $i ++) { + | if (!$arr.isNullAt($i) && ${ctx.genEqual(right.dataType, value, getValue)}) { + |$numsToRemove = $numsToRemove + 1; + | } + |} + |$values = new Object[$arr.numElements() - $numsToRemove]; + |for (int $i = 0; $i < $arr.numElements(); $i ++) { + | if ($arr.isNullAt($i)) { + | $values[$pos] = null; + | $pos = $pos + 1; + | } + | else { + | if (${ctx.genEqual(right.dataType, value, getValue)}) { + | ; + | } + | else { + | $values[$pos] = ${CodeGenerator.getValue(arr, elementType, s"$i")}; --- End diff -- `$values[$pos] = $getValue;`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21069#discussion_r185721700 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala --- @@ -280,4 +280,35 @@ class CollectionExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper checkEvaluation(Concat(Seq(aa0, aa1)), Seq(Seq("a", "b"), Seq("c"), Seq("d"), Seq("e", "f"))) } + + test("Array remove") { +val a0 = Literal.create(Seq(1, 2, 3, 2, 2, 5), ArrayType(IntegerType)) +val a1 = Literal.create(Seq("b", "a", "a", "c", "b"), ArrayType(StringType)) +val a2 = Literal.create(Seq[String](null, "", null, ""), ArrayType(StringType)) +val a3 = Literal.create(Seq.empty[Integer], ArrayType(IntegerType)) +val a4 = Literal.create(null, ArrayType(StringType)) +val a5 = Literal.create(Seq(1, null, 8, 9, null), ArrayType(IntegerType)) +val a6 = Literal.create(Seq(true, false, false, true), ArrayType(BooleanType)) + +checkEvaluation(ArrayRemove(a0, Literal(0)), Seq(1, 2, 3, 2, 2, 5)) +checkEvaluation(ArrayRemove(a0, Literal(1)), Seq(2, 3, 2, 2, 5)) +checkEvaluation(ArrayRemove(a0, Literal(2)), Seq(1, 3, 5)) +checkEvaluation(ArrayRemove(a0, Literal(3)), Seq(1, 2, 2, 2, 5)) +checkEvaluation(ArrayRemove(a0, Literal(5)), Seq(1, 2, 3, 2, 2)) --- End diff -- Can you add a case for something like `ArrayRemove(a0, Literal(10))` to check no value is removed with not contained value? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21069#discussion_r185720852 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -883,3 +883,70 @@ case class Concat(children: Seq[Expression]) extends Expression { override def sql: String = s"concat(${children.map(_.sql).mkString(", ")})" } + +/** + * Remove all elements that equal to element from the given array + */ +@ExpressionDescription( + usage = "_FUNC_(array, element) - Remove all elements that equal to element from array.", + examples = """ +Examples: + > SELECT _FUNC_(array(1, 2, 3, null, 3), 3); + [1,2,null] + """, since = "2.4.0") +case class ArrayRemove(left: Expression, right: Expression) + extends BinaryExpression with ImplicitCastInputTypes with CodegenFallback { + + override def dataType: DataType = left.dataType + + override def inputTypes: Seq[AbstractDataType] = +Seq(ArrayType, left.dataType.asInstanceOf[ArrayType].elementType) + + override def nullSafeEval(arr: Any, value: Any): Any = { +val elementType = left.dataType.asInstanceOf[ArrayType].elementType +val data = arr.asInstanceOf[ArrayData].toArray[AnyRef](elementType).filter(_ != value) +new GenericArrayData(data.asInstanceOf[Array[Any]]) + } + + override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { +val elementType = left.dataType.asInstanceOf[ArrayType].elementType +nullSafeCodeGen(ctx, ev, (arr, value) => { + val arrayClass = classOf[GenericArrayData].getName + val values = ctx.freshName("values") + val i = ctx.freshName("i") + val pos = ctx.freshName("arrayPosition") + val numsToRemove = ctx.freshName("newArrLen") + val getValue = CodeGenerator.getValue(arr, right.dataType, i) + s""" + |int $pos = 0; + |int $numsToRemove = 0; + |Object[] $values; + | + |for (int $i = 0; $i < $arr.numElements(); $i ++) { + | if (!$arr.isNullAt($i) && ${ctx.genEqual(right.dataType, value, getValue)}) { + |$numsToRemove = $numsToRemove + 1; + | } + |} + |$values = new Object[$arr.numElements() - $numsToRemove]; --- End diff -- Can you specialize creating `ArrayData` for primitive types to avoid boxing? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21069#discussion_r185715072 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -287,3 +287,44 @@ case class ArrayContains(left: Expression, right: Expression) override def prettyName: String = "array_contains" } + +/** + * Remove all elements that equal to element from the given array + */ +@ExpressionDescription( + usage = "_FUNC_(array, element) - Remove all elements that equal to element from array.", + examples = """ +Examples: + > SELECT _FUNC_(array(1, 2, 3, null, 3), 3); + [1,2,null] + """, since = "2.4.0") +case class ArrayRemove(left: Expression, right: Expression) + extends BinaryExpression with ImplicitCastInputTypes with CodegenFallback { --- End diff -- Now that there is a codegen, we can remove `CodegenFallback`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21069#discussion_r185722053 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala --- @@ -280,4 +280,35 @@ class CollectionExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper checkEvaluation(Concat(Seq(aa0, aa1)), Seq(Seq("a", "b"), Seq("c"), Seq("d"), Seq("e", "f"))) } + + test("Array remove") { +val a0 = Literal.create(Seq(1, 2, 3, 2, 2, 5), ArrayType(IntegerType)) +val a1 = Literal.create(Seq("b", "a", "a", "c", "b"), ArrayType(StringType)) +val a2 = Literal.create(Seq[String](null, "", null, ""), ArrayType(StringType)) +val a3 = Literal.create(Seq.empty[Integer], ArrayType(IntegerType)) +val a4 = Literal.create(null, ArrayType(StringType)) +val a5 = Literal.create(Seq(1, null, 8, 9, null), ArrayType(IntegerType)) +val a6 = Literal.create(Seq(true, false, false, true), ArrayType(BooleanType)) + +checkEvaluation(ArrayRemove(a0, Literal(0)), Seq(1, 2, 3, 2, 2, 5)) +checkEvaluation(ArrayRemove(a0, Literal(1)), Seq(2, 3, 2, 2, 5)) +checkEvaluation(ArrayRemove(a0, Literal(2)), Seq(1, 3, 5)) +checkEvaluation(ArrayRemove(a0, Literal(3)), Seq(1, 2, 2, 2, 5)) +checkEvaluation(ArrayRemove(a0, Literal(5)), Seq(1, 2, 3, 2, 2)) + +checkEvaluation(ArrayRemove(a1, Literal("")), Seq("b", "a", "a", "c", "b")) +checkEvaluation(ArrayRemove(a1, Literal("a")), Seq("b", "c", "b")) +checkEvaluation(ArrayRemove(a1, Literal("b")), Seq("a", "a", "c")) +checkEvaluation(ArrayRemove(a1, Literal("c")), Seq("b", "a", "a", "b")) + +checkEvaluation(ArrayRemove(a2, Literal("")), Seq(null, null)) +checkEvaluation(ArrayRemove(a2, Literal.create(null, StringType)), null) + +checkEvaluation(ArrayRemove(a3, Literal(1)), Seq.empty[Integer]) + +checkEvaluation(ArrayRemove(a4, Literal("a")), null) + +checkEvaluation(ArrayRemove(a5, Literal(9)), Seq(1, null, 8, null)) +checkEvaluation(ArrayRemove(a6, Literal(false)), Seq(true, true)) --- End diff -- Can you add a case for something like `ArrayRemove(a0, Literal(null, IntegerType))`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21069#discussion_r185716705 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -883,3 +883,70 @@ case class Concat(children: Seq[Expression]) extends Expression { override def sql: String = s"concat(${children.map(_.sql).mkString(", ")})" } + +/** + * Remove all elements that equal to element from the given array + */ +@ExpressionDescription( + usage = "_FUNC_(array, element) - Remove all elements that equal to element from array.", + examples = """ +Examples: + > SELECT _FUNC_(array(1, 2, 3, null, 3), 3); + [1,2,null] + """, since = "2.4.0") +case class ArrayRemove(left: Expression, right: Expression) + extends BinaryExpression with ImplicitCastInputTypes with CodegenFallback { + + override def dataType: DataType = left.dataType + + override def inputTypes: Seq[AbstractDataType] = +Seq(ArrayType, left.dataType.asInstanceOf[ArrayType].elementType) + + override def nullSafeEval(arr: Any, value: Any): Any = { +val elementType = left.dataType.asInstanceOf[ArrayType].elementType +val data = arr.asInstanceOf[ArrayData].toArray[AnyRef](elementType).filter(_ != value) +new GenericArrayData(data.asInstanceOf[Array[Any]]) + } + + override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { +val elementType = left.dataType.asInstanceOf[ArrayType].elementType +nullSafeCodeGen(ctx, ev, (arr, value) => { + val arrayClass = classOf[GenericArrayData].getName + val values = ctx.freshName("values") + val i = ctx.freshName("i") + val pos = ctx.freshName("arrayPosition") + val numsToRemove = ctx.freshName("newArrLen") + val getValue = CodeGenerator.getValue(arr, right.dataType, i) --- End diff -- `CodeGenerator.getValue(arr, elementType, i)`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21069#discussion_r185716774 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -883,3 +883,70 @@ case class Concat(children: Seq[Expression]) extends Expression { override def sql: String = s"concat(${children.map(_.sql).mkString(", ")})" } + +/** + * Remove all elements that equal to element from the given array + */ +@ExpressionDescription( + usage = "_FUNC_(array, element) - Remove all elements that equal to element from array.", + examples = """ +Examples: + > SELECT _FUNC_(array(1, 2, 3, null, 3), 3); + [1,2,null] + """, since = "2.4.0") +case class ArrayRemove(left: Expression, right: Expression) + extends BinaryExpression with ImplicitCastInputTypes with CodegenFallback { + + override def dataType: DataType = left.dataType + + override def inputTypes: Seq[AbstractDataType] = +Seq(ArrayType, left.dataType.asInstanceOf[ArrayType].elementType) + + override def nullSafeEval(arr: Any, value: Any): Any = { +val elementType = left.dataType.asInstanceOf[ArrayType].elementType +val data = arr.asInstanceOf[ArrayData].toArray[AnyRef](elementType).filter(_ != value) +new GenericArrayData(data.asInstanceOf[Array[Any]]) + } + + override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { +val elementType = left.dataType.asInstanceOf[ArrayType].elementType +nullSafeCodeGen(ctx, ev, (arr, value) => { + val arrayClass = classOf[GenericArrayData].getName + val values = ctx.freshName("values") + val i = ctx.freshName("i") + val pos = ctx.freshName("arrayPosition") + val numsToRemove = ctx.freshName("newArrLen") + val getValue = CodeGenerator.getValue(arr, right.dataType, i) + s""" + |int $pos = 0; + |int $numsToRemove = 0; + |Object[] $values; + | + |for (int $i = 0; $i < $arr.numElements(); $i ++) { + | if (!$arr.isNullAt($i) && ${ctx.genEqual(right.dataType, value, getValue)}) { --- End diff -- `ctx.genEqual(elementType, value, getValue)`? Btw, this expression is used twice, so we should extract it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21069#discussion_r185716615 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -883,3 +883,70 @@ case class Concat(children: Seq[Expression]) extends Expression { override def sql: String = s"concat(${children.map(_.sql).mkString(", ")})" } + +/** + * Remove all elements that equal to element from the given array + */ +@ExpressionDescription( + usage = "_FUNC_(array, element) - Remove all elements that equal to element from array.", + examples = """ +Examples: + > SELECT _FUNC_(array(1, 2, 3, null, 3), 3); + [1,2,null] + """, since = "2.4.0") +case class ArrayRemove(left: Expression, right: Expression) + extends BinaryExpression with ImplicitCastInputTypes with CodegenFallback { + + override def dataType: DataType = left.dataType + + override def inputTypes: Seq[AbstractDataType] = +Seq(ArrayType, left.dataType.asInstanceOf[ArrayType].elementType) + + override def nullSafeEval(arr: Any, value: Any): Any = { +val elementType = left.dataType.asInstanceOf[ArrayType].elementType +val data = arr.asInstanceOf[ArrayData].toArray[AnyRef](elementType).filter(_ != value) +new GenericArrayData(data.asInstanceOf[Array[Any]]) + } + + override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { +val elementType = left.dataType.asInstanceOf[ArrayType].elementType +nullSafeCodeGen(ctx, ev, (arr, value) => { + val arrayClass = classOf[GenericArrayData].getName + val values = ctx.freshName("values") + val i = ctx.freshName("i") + val pos = ctx.freshName("arrayPosition") + val numsToRemove = ctx.freshName("newArrLen") --- End diff -- `ctx.freshName("numsToRemove")`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20761 Cool, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21210: [SPARK-23489][SQL][TEST] HiveExternalCatalogVersionsSuit...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21210 Thank you for merging, @cloud-fan. And thank you, all. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21219 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2855/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21219 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21219 **[Test build #90112 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90112/testReport)** for PR 21219 at commit [`41d06e1`](https://github.com/apache/spark/commit/41d06e13d0f95f1dd146b6b512a0becc88eb2caa). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/21219 jenkins retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Github user lokm01 commented on the issue: https://github.com/apache/spark/pull/21215 Hey @maropu, So we've encountered a number of issues with casting: 1. Casting an empty array to an array of primitive types caused an exception on 2.2.1, but works on 2.3.0+ so that's sorted 2. We're still facing an issue on 2.3.0 when we try to cast an empty array to an array of complex types. See the following example: ` case class Outer(a: List[Inner]) case class Inner(b: Int, c: String) object App4 extends App { val spark = SparkSession.builder().appName("").master("local[*]").getOrCreate() import spark.implicits._ import org.apache.spark.sql.functions._ val df = spark.createDataFrame(Seq[Outer]()) val r = spark.range(100).select(array().cast(df.schema("a").dataType)) r.printSchema() r.show } ` This code produces > Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve 'array()' due to data type mismatch: cannot cast array to array>;; --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21224: [SPARK-24167][SQL] ParquetFilters should not access SQLC...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21224 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21224: [SPARK-24167][SQL] ParquetFilters should not access SQLC...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21224 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2854/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21201 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21201 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2853/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21145: [SPARK-24073][SQL]: Rename DataReaderFactory to ReadTask...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21145 I agree this is not a real factory, but it's not a real task either. I feel something like `DataReaderHolder` or `ReadTaskDescriptor` should be better. also cc @rxin @marmbrus --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21169 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2852/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21169 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21224: [SPARK-24167][SQL] ParquetFilters should not access SQLC...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21224 **[Test build #90110 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90110/testReport)** for PR 21224 at commit [`d7dc8a8`](https://github.com/apache/spark/commit/d7dc8a85489122e5b91cf5bc7cc0190f4d474a2c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21222 **[Test build #90111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90111/testReport)** for PR 21222 at commit [`c1ad1c5`](https://github.com/apache/spark/commit/c1ad1c557e6165455457adb6f148d6d9616548a1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21223 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2851/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21223 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2850/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21210: [SPARK-23489][SQL][TEST] HiveExternalCatalogVersi...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21210 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21201 **[Test build #90109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90109/testReport)** for PR 21201 at commit [`42d0241`](https://github.com/apache/spark/commit/42d0241feef21c71be1a5223ff7c359b7ad4c27b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21222 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21210: [SPARK-23489][SQL][TEST] HiveExternalCatalogVersionsSuit...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21210 thanks, merging to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21201 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2849/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90106/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21223 **[Test build #90107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90107/testReport)** for PR 21223 at commit [`d900b4c`](https://github.com/apache/spark/commit/d900b4c04f5def9bac6cec33c1c7753761a19658). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21169 **[Test build #90108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90108/testReport)** for PR 21169 at commit [`b6d91db`](https://github.com/apache/spark/commit/b6d91db2fd71b50389cf3647a31eefc83d5dbc44). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2848/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2847/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90103/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user szyszy commented on the issue: https://github.com/apache/spark/pull/20761 Hey! Exactly, I almost addressed all of Marcelo's comments so please expect a PR update in the coming days. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21193 **[Test build #90104 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90104/testReport)** for PR 21193 at commit [`5fe425c`](https://github.com/apache/spark/commit/5fe425c2d2837f00bdfe9ba5e6f446829fba32c1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21226 **[Test build #90102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90102/testReport)** for PR 21226 at commit [`e1491f0`](https://github.com/apache/spark/commit/e1491f0aeb62d1eda8cd8c55f890c8f87eec5761). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21169 **[Test build #90105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90105/testReport)** for PR 21169 at commit [`b6d91db`](https://github.com/apache/spark/commit/b6d91db2fd71b50389cf3647a31eefc83d5dbc44). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21169 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21223 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21169 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21226 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21193 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90091/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90097/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21219 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90093/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21169 **[Test build #90099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90099/testReport)** for PR 21169 at commit [`b6d91db`](https://github.com/apache/spark/commit/b6d91db2fd71b50389cf3647a31eefc83d5dbc44). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21219 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90092/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21223 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90095/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #90091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90091/testReport)** for PR 21073 at commit [`77ae014`](https://github.com/apache/spark/commit/77ae014bea3d9fbc20fbd11d5508c3606e26343d). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90097 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90097/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21226 **[Test build #90098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90098/testReport)** for PR 21226 at commit [`e1491f0`](https://github.com/apache/spark/commit/e1491f0aeb62d1eda8cd8c55f890c8f87eec5761). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21219 **[Test build #90093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90093/testReport)** for PR 21219 at commit [`41d06e1`](https://github.com/apache/spark/commit/41d06e13d0f95f1dd146b6b512a0becc88eb2caa). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21201 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90101/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90098/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21169 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90099/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21169 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21223 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21201 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21193 **[Test build #90092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90092/testReport)** for PR 21193 at commit [`5fe425c`](https://github.com/apache/spark/commit/5fe425c2d2837f00bdfe9ba5e6f446829fba32c1). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ExprCode(var code: Block, var isNull: ExprValue, var value: ExprValue)` * `trait Block extends JavaCode ` * ` implicit class BlockHelper(val sc: StringContext) extends AnyVal ` * `case class CodeBlock(codeParts: Seq[String], exprValues: Seq[Any]) extends Block ` * `case class Blocks(blocks: Seq[Block]) extends Block ` * `trait ExprValue extends JavaCode ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21201 **[Test build #90101 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90101/testReport)** for PR 21201 at commit [`42d0241`](https://github.com/apache/spark/commit/42d0241feef21c71be1a5223ff7c359b7ad4c27b). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21223 **[Test build #90095 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90095/testReport)** for PR 21223 at commit [`d900b4c`](https://github.com/apache/spark/commit/d900b4c04f5def9bac6cec33c1c7753761a19658). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21187: [SPARK-24035][SQL] SQL syntax for Pivot
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21187 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90094/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21187: [SPARK-24035][SQL] SQL syntax for Pivot
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21187 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org