[GitHub] spark issue #22938: [SPARK-25935][SQL] Prevent null rows from JSON parser
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22938 @cloud-fan @HyukjinKwon Do you agree with the proposed changes, or there is anything which blocks the PR for now? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics table doesn...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23038 **[Test build #98856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98856/testReport)** for PR 23038 at commit [`c53ca48`](https://github.com/apache/spark/commit/c53ca48885d14dc6a70412fe27a00c8f71aff561). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23034: [WIP][SPARK-26035][PYTHON] Break large streaming/tests.p...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23034 **[Test build #98857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98857/testReport)** for PR 23034 at commit [`c542b8d`](https://github.com/apache/spark/commit/c542b8d1aed852978f72a8b01ec5c89cc04e682f). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23041: [SPARK-26069][TESTS]Fix flaky test: RpcIntegrationSuite....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23041 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98860/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23041: [SPARK-26069][TESTS]Fix flaky test: RpcIntegrationSuite....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23041 **[Test build #98860 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98860/testReport)** for PR 23041 at commit [`6bebcb5`](https://github.com/apache/spark/commit/6bebcb5e004ed4b434c550d26ed1a922d13e0446). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23031: [SPARK-26060][SQL] Track SparkConf entries and make SET ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23031 **[Test build #98858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98858/testReport)** for PR 23031 at commit [`37ebae4`](https://github.com/apache/spark/commit/37ebae4c08f3746d7ce739057e81a8cc9c83e106). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22974 **[Test build #98855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98855/testReport)** for PR 22974 at commit [`ec9b83f`](https://github.com/apache/spark/commit/ec9b83f5fa0f91a180ad6af36784f6404ddf2299). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23041: [SPARK-26069][TESTS]Fix flaky test: RpcIntegrationSuite....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23041 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23034: [WIP][SPARK-26035][PYTHON] Break large streaming/tests.p...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23034 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98857/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23034: [WIP][SPARK-26035][PYTHON] Break large streaming/tests.p...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23034 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23031: [SPARK-26060][SQL] Track SparkConf entries and make SET ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23031 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98858/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics table doesn...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23038 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98856/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23031: [SPARK-26060][SQL] Track SparkConf entries and make SET ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23031 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics table doesn...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23038 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22974 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98855/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22974 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23031: [SPARK-26060][SQL] Track SparkConf entries and make SET ...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/23031 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23031: [SPARK-26060][SQL] Track SparkConf entries and make SET ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23031 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5045/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23031: [SPARK-26060][SQL] Track SparkConf entries and make SET ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23031 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23031: [SPARK-26060][SQL] Track SparkConf entries and make SET ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23031 **[Test build #98862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98862/testReport)** for PR 23031 at commit [`37ebae4`](https://github.com/apache/spark/commit/37ebae4c08f3746d7ce739057e81a8cc9c83e106). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23035: [SPARK-26057][SQL] Transform also analyzed plans when de...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23035 **[Test build #98861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98861/testReport)** for PR 23035 at commit [`98d91a3`](https://github.com/apache/spark/commit/98d91a323d891b8995791fc991d80cee9c4f7f97). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23035: [SPARK-26057][SQL] Transform also analyzed plans ...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23035#discussion_r233744065
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
---
@@ -2554,4 +2554,34 @@ class DataFrameSuite extends QueryTest with
SharedSQLContext {
checkAnswer(swappedDf.filter($"key"($"map") > "a"), Row(2, Map(2 ->
"b")))
}
+
+ test("SPARK-26057: attribute deduplication on already analyzed plans") {
+withTempView("cc", "p", "c") {
+ val df1 = Seq(("1-1", "sp", 6)).toDF("id", "layout", "n")
+ df1.createOrReplaceTempView("cc")
+ val df2 = Seq(("sp", 1)).toDF("layout", "ts")
+ df2.createOrReplaceTempView("p")
+ val df3 = Seq(("1-1", "sp", 3)).toDF("id", "layout", "ts")
+ df3.createOrReplaceTempView("c")
+ spark.sql(
+"""
+ |SELECT cc.id, cc.layout, count(*) as m
+ |FROM cc
+ |JOIN p USING(layout)
+ |WHERE EXISTS(
+ | SELECT 1
+ | FROM c
+ | WHERE c.id = cc.id AND c.layout = cc.layout AND c.ts > p.ts)
+ |GROUP BY cc.id, cc.layout
+""".stripMargin).createOrReplaceTempView("pcc")
+ val res = spark.sql(
--- End diff --
yes, I simplified as much as I was able to. I hope now it is fine. Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics table doesn...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/23038 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23035: [SPARK-26057][SQL] Transform also analyzed plans when de...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23035 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23035: [SPARK-26057][SQL] Transform also analyzed plans when de...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23035 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5046/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics table doesn...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23038 **[Test build #98863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98863/testReport)** for PR 23038 at commit [`c53ca48`](https://github.com/apache/spark/commit/c53ca48885d14dc6a70412fe27a00c8f71aff561). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...
Github user zhengruifeng commented on the issue: https://github.com/apache/spark/pull/22974 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23042: [SPARK-26070] add rule for implicit type coercion...
GitHub user uzadude opened a pull request: https://github.com/apache/spark/pull/23042 [SPARK-26070] add rule for implicit type coercion for decimal(x,0) ## What changes were proposed in this pull request? Adding another excpetion rule for a popular case where ID columns are saved as decimal(x,0) and are compared with same ID columns in other tables which are saved as strings ## How was this patch tested? added a unit tests and made some manual tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/uzadude/spark SPARK-26070-decimal-coercion Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23042.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23042 commit 5bb97493c5a8d214762f36bbb824d34b6b44a5e2 Author: oraviv Date: 2018-11-15T08:13:07Z add another exception rule for a wide use case in which ID columns are saved as decimal(x, 0) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23042: [SPARK-26070] add rule for implicit type coercion for de...
Github user uzadude commented on the issue: https://github.com/apache/spark/pull/23042 @cloud-fan - could you please take a look? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23042: [SPARK-26070] add rule for implicit type coercion for de...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23042 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23042: [SPARK-26070] add rule for implicit type coercion for de...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23042 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22974 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23042: [SPARK-26070] add rule for implicit type coercion for de...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23042 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22974 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5047/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22974 **[Test build #98864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98864/testReport)** for PR 22974 at commit [`ec9b83f`](https://github.com/apache/spark/commit/ec9b83f5fa0f91a180ad6af36784f6404ddf2299). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23034: [WIP][SPARK-26035][PYTHON] Break large streaming/tests.p...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23034 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23034: [WIP][SPARK-26035][PYTHON] Break large streaming/tests.p...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23034 **[Test build #98865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98865/testReport)** for PR 23034 at commit [`c542b8d`](https://github.com/apache/spark/commit/c542b8d1aed852978f72a8b01ec5c89cc04e682f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23034: [WIP][SPARK-26035][PYTHON] Break large streaming/tests.p...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23034 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5048/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23034: [WIP][SPARK-26035][PYTHON] Break large streaming/tests.p...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23034 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23043: [SPARK-26021][SQL] replace minus zero with zero i...
GitHub user adoron opened a pull request: https://github.com/apache/spark/pull/23043 [SPARK-26021][SQL] replace minus zero with zero in UnsafeProjection GROUP BY treats -0.0 and 0.0 as different values which is unlike hive's behavior. In addition current behavior with codegen is unpredictable (see example in JIRA ticket). ## What changes were proposed in this pull request? In BoundReference class, in the generated code, replace -0.0 with 0.0 if the data type is double or float. ## How was this patch tested? Added tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/adoron/spark adoron-spark-26021-replace-minus-zero-with-zero Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23043.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23043 commit ee0ef91a7047d47328efac753e66ec97a91c0e37 Author: Alon Doron Date: 2018-11-14T16:18:30Z replace -0.0 with 0.0 in BoundAttribute added tests commit 63b7f59ad44d0876ea6dde02e4204fc0140d0df6 Author: Alon Doron Date: 2018-11-14T16:27:24Z minor remove var type --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Unsaf...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23043 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Unsaf...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23043 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Unsaf...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23043 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Unsaf...
Github user adoron commented on the issue: https://github.com/apache/spark/pull/23043 @srowen @gatorsmile @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22976: [SPARK-25974][SQL]Optimizes Generates bytecode for order...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/22976 gentle ping @rednaxelafx --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 Merged to master. Thanks @felixcheung. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22976: [SPARK-25974][SQL]Optimizes Generates bytecode fo...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22976#discussion_r233765313
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
---
@@ -68,62 +68,55 @@ object GenerateOrdering extends
CodeGenerator[Seq[SortOrder], Ordering[InternalR
genComparisons(ctx, ordering)
}
+ /**
+ * Creates the variables for ordering based on the given order.
+ */
+ private def createOrderKeys(
+ ctx: CodegenContext,
+ row: String,
+ ordering: Seq[SortOrder]): Seq[ExprCode] = {
+ctx.INPUT_ROW = row
+// to use INPUT_ROW we must make sure currentVars is null
+ctx.currentVars = null
--- End diff --
nit: we only need one `ctx.currentVars = null`. Maybe move it out of this
function and put it at previous position.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22976: [SPARK-25974][SQL]Optimizes Generates bytecode fo...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22976#discussion_r233764857
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
---
@@ -133,30 +126,26 @@ object GenerateOrdering extends
CodeGenerator[Seq[SortOrder], Ordering[InternalR
returnType = "int",
makeSplitFunction = { body =>
s"""
- InternalRow ${ctx.INPUT_ROW} = null; // Holds current row being
evaluated.
- $body
- return 0;
-"""
+ |$body
+ |return 0;
+""".stripMargin
},
foldFunctions = { funCalls =>
funCalls.zipWithIndex.map { case (funCall, i) =>
val comp = ctx.freshName("comp")
s"""
-int $comp = $funCall;
-if ($comp != 0) {
- return $comp;
-}
- """
+|int $comp = $funCall;
+|if ($comp != 0) {
+| return $comp;
+|}
+ """.stripMargin
}.mkString
})
ctx.currentVars = oldCurrentVars
ctx.INPUT_ROW = oldInputRow
// make sure INPUT_ROW is declared even if splitExpressions
// returns an inlined block
--- End diff --
This comment is not invalid and can be removed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23012 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22976: [SPARK-25974][SQL]Optimizes Generates bytecode for order...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22976 Two minor comments. LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22976: [SPARK-25974][SQL]Optimizes Generates bytecode for order...
Github user rednaxelafx commented on the issue: https://github.com/apache/spark/pull/22976 +1 LGTM, thanks for making this improvement! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22976: [SPARK-25974][SQL]Optimizes Generates bytecode fo...
Github user rednaxelafx commented on a diff in the pull request:
https://github.com/apache/spark/pull/22976#discussion_r233771269
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
---
@@ -68,62 +68,55 @@ object GenerateOrdering extends
CodeGenerator[Seq[SortOrder], Ordering[InternalR
genComparisons(ctx, ordering)
}
+ /**
+ * Creates the variables for ordering based on the given order.
+ */
+ private def createOrderKeys(
+ ctx: CodegenContext,
+ row: String,
+ ordering: Seq[SortOrder]): Seq[ExprCode] = {
+ctx.INPUT_ROW = row
+// to use INPUT_ROW we must make sure currentVars is null
+ctx.currentVars = null
--- End diff --
I thought about making the same suggestion, but on a second thought it's
fine to just leave it here as well. The overhead isn't gonna be high, but it's
nice to see how the `ctx.INPUT_ROW` and `ctx.currentVars` interact in adjacent
lines.
So I'm okay either way (as-is or with @viirya 's suggestion)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22976: [SPARK-25974][SQL]Optimizes Generates bytecode fo...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22976#discussion_r233771918
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
---
@@ -68,62 +68,55 @@ object GenerateOrdering extends
CodeGenerator[Seq[SortOrder], Ordering[InternalR
genComparisons(ctx, ordering)
}
+ /**
+ * Creates the variables for ordering based on the given order.
+ */
+ private def createOrderKeys(
+ ctx: CodegenContext,
+ row: String,
+ ordering: Seq[SortOrder]): Seq[ExprCode] = {
+ctx.INPUT_ROW = row
+// to use INPUT_ROW we must make sure currentVars is null
+ctx.currentVars = null
--- End diff --
Very agreed. :)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22976: [SPARK-25974][SQL]Optimizes Generates bytecode for order...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22976 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22976: [SPARK-25974][SQL]Optimizes Generates bytecode fo...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22976#discussion_r233787377
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
---
@@ -133,30 +126,26 @@ object GenerateOrdering extends
CodeGenerator[Seq[SortOrder], Ordering[InternalR
returnType = "int",
makeSplitFunction = { body =>
s"""
- InternalRow ${ctx.INPUT_ROW} = null; // Holds current row being
evaluated.
- $body
- return 0;
-"""
+ |$body
+ |return 0;
+""".stripMargin
},
foldFunctions = { funCalls =>
funCalls.zipWithIndex.map { case (funCall, i) =>
val comp = ctx.freshName("comp")
s"""
-int $comp = $funCall;
-if ($comp != 0) {
- return $comp;
-}
- """
+|int $comp = $funCall;
+|if ($comp != 0) {
+| return $comp;
+|}
+ """.stripMargin
}.mkString
})
ctx.currentVars = oldCurrentVars
ctx.INPUT_ROW = oldInputRow
// make sure INPUT_ROW is declared even if splitExpressions
// returns an inlined block
--- End diff --
sorry didn't see this comment when merging. feel free to send a follow-up.
thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22976: [SPARK-25974][SQL]Optimizes Generates bytecode fo...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22976 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23040: [SPARK-26068][Core]ChunkedByteBufferInputStream should h...
Github user LinhongLiu commented on the issue: https://github.com/apache/spark/pull/23040 Problem: ChunkedByteBuffer has signature `ChunkedByteBuffer(var chunks: Array[ByteBuffer])`. This means user is allowed to pass any kind of `chunks` to it. Then we will face the problem described in [JIRA](https://issues.apache.org/jira/browse/SPARK-26068). That's why I submit this PR. But on the other hand: This is an internal class and Spark safely uses `ChunkedByteBuffer` in 2 ways: 1. Use `ChunkedByteBuffer(byteBuffer: ByteBuffer)` to pass only one buffer. Even if it's an empty one, spark will handle this case correctly. 2. Use `ChunkedByteBufferOutputStream` to create `ChunkedByteBuffer` with multiple `chunks`. In this case, empty ByteBuffer will never happen. As a result, current spark code will never reach the problem as far as we won't use `ChunkedByteBuffer(var chunks: Array[ByteBuffer])` directly. So it's both OK either we fix this or not. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22976: [SPARK-25974][SQL]Optimizes Generates bytecode for order...
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/22976 thanks, All @cloud-fan, @rednaxelafx, @viirya, @kiszk --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comment as ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23044 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comment as ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23044 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comm...
GitHub user heary-cao opened a pull request: https://github.com/apache/spark/pull/23044 [SPARK-26073][SQL][FOLLOW-UP] remove invalid comment as we don't use it anymore ## What changes were proposed in this pull request? remove invalid comment as we don't use it anymore More details: #22976 (comment) ## How was this patch tested? N/A Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/heary-cao/spark followUpOrdering Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23044.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23044 commit 786cd4d3e4c06033203618129de080a8e7d72530 Author: caoxuewen Date: 2018-11-15T11:00:39Z remove invalid comment as we don't use it anymore --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comment as ...
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/23044 cc @cloud-fan, @viirya --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comment as ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23044 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23045: [SPARK-26071][SQL] disallow map as map key
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23045 cc @gatorsmile @dongjoon-hyun @viirya --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23045: [SPARK-26071][SQL] disallow map as map key
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/23045 [SPARK-26071][SQL] disallow map as map key ## What changes were proposed in this pull request? Due to implementation limitation, currently Spark can't compare or do equality check between map types. As a result, map values can't appear in EQUAL or comparison expressions, can't be grouping key, etc. The more important thing is, map loop up needs to do equality check of the map key, and thus can't support map as map key when looking up values from a map. Thus it's not useful to have map as map key. This PR proposes to stop users from creating maps using map type as key. The list of expressions that are updated: `CreateMap`, `MapFromArrays`, `MapFromEntries`, `MapConcat`, `TransformKeys`. I manually checked all the places that create `MapType`, and came up with this list. Note that, maps with map type key still exist, via reading from parquet files, converting from scala/java map, etc. This PR is not to completely forbid map as map key, but to avoid creating it by Spark itself. Motivation: when I was trying to fix the duplicate key problem, I found it's impossible to do it with map type map key. I think it's reasonable to avoid map type map key for builtin functions. ## How was this patch tested? updated test You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloud-fan/spark map-key Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23045.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23045 commit 3ff0cd592c52839d0aac739b44cee0cf02e951bc Author: Wenchen Fan Date: 2018-11-15T10:23:58Z disallow map as map key --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23045: [SPARK-26071][SQL] disallow map as map key
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23045 **[Test build #98866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98866/testReport)** for PR 23045 at commit [`3ff0cd5`](https://github.com/apache/spark/commit/3ff0cd592c52839d0aac739b44cee0cf02e951bc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23045: [SPARK-26071][SQL] disallow map as map key
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23045 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5049/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23045: [SPARK-26071][SQL] disallow map as map key
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23045 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.en...
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/23046 [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`. ## What changes were proposed in this pull request? This is a follow-up of #20393. We should read the conf `"spark.sql.sort.enableRadixSort"` from `SQLConf` instead of `SparkConf`, i.e., use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`, otherwise the config is never read. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ueshin/apache-spark issues/SPARK-23207/conf Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23046.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23046 commit d8db5f3759b998e3ef5720f56c1a048c1494a1c1 Author: Takuya UESHIN Date: 2018-11-15T11:30:13Z Use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRad...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/23046 cc @jiangxb1987 @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRad...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23046 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRad...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23046 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5050/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRad...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23046 **[Test build #98867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98867/testReport)** for PR 23046 at commit [`d8db5f3`](https://github.com/apache/spark/commit/d8db5f3759b998e3ef5720f56c1a048c1494a1c1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23035: [SPARK-26057][SQL] Transform also analyzed plans when de...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23035 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23035: [SPARK-26057][SQL] Transform also analyzed plans when de...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23035 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98861/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23031: [SPARK-26060][SQL] Track SparkConf entries and make SET ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23031 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98862/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23031: [SPARK-26060][SQL] Track SparkConf entries and make SET ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23031 **[Test build #98862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98862/testReport)** for PR 23031 at commit [`37ebae4`](https://github.com/apache/spark/commit/37ebae4c08f3746d7ce739057e81a8cc9c83e106). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23031: [SPARK-26060][SQL] Track SparkConf entries and make SET ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23031 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23042: [SPARK-26070][SQL] add rule for implicit type coe...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/23042#discussion_r233816966
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -138,6 +138,11 @@ object TypeCoercion {
case (DateType, TimestampType)
=> if (conf.compareDateTimestampInTimestamp) Some(TimestampType)
else Some(StringType)
+// to support a popular use case of tables using Decimal(X, 0) for
long IDs instead of strings
+// see SPARK-26070 for more details
+case (n: DecimalType, s: StringType) if n.scale == 0 =>
Some(DecimalType(n.precision, n.scale))
--- End diff --
what if the decimal is (1, 0) and the string is something like `.`?
The string can be anything: a very big integer, a fraction with many digits
after the dot, etc. I don't think there is a perfect solution, casting to
double is the best we can do here.
I'd suggest end users to manually do the cast which fits their data best.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23035: [SPARK-26057][SQL] Transform also analyzed plans when de...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23035 thanks, merging to master/2.4! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRad...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23046 good catch! LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comm...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/23044#discussion_r233819066
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
---
@@ -76,8 +76,6 @@ object GenerateOrdering extends
CodeGenerator[Seq[SortOrder], Ordering[InternalR
row: String,
ordering: Seq[SortOrder]): Seq[ExprCode] = {
ctx.INPUT_ROW = row
-// to use INPUT_ROW we must make sure currentVars is null
-ctx.currentVars = null
--- End diff --
I think the conclusion was to keep it?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23035: [SPARK-26057][SQL] Transform also analyzed plans ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23035 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comment as ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23044 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comment as ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23044 add to whitelist --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comment as ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23044 **[Test build #98868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98868/testReport)** for PR 23044 at commit [`786cd4d`](https://github.com/apache/spark/commit/786cd4d3e4c06033203618129de080a8e7d72530). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23040: [SPARK-26068][Core]ChunkedByteBufferInputStream should h...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23040 It's good to fix a potential bug, can you add a unit test? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Unsaf...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23043 This only works for attribute, not literal or intermedia result. Is there a better place to fix it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22989: [SPARK-25986][Build] Add rules to ban throw Error...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22989#discussion_r233821654
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/UnsafeAlignedOffset.java ---
@@ -39,7 +39,9 @@ public static int getSize(Object object, long offset) {
case 8:
return (int)Platform.getLong(object, offset);
default:
+// checkstyle.off: RegexpSinglelineJava
throw new AssertionError("Illegal UAO_SIZE");
--- End diff --
yea, that's exactly the use case of `IllegalStateException`, which can also
pass the style check here.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23029: [SPARK-26055][CORE] InterfaceStability annotations shoul...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23029 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23029: [SPARK-26055][CORE] InterfaceStability annotation...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23029 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22974 **[Test build #98864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98864/testReport)** for PR 22974 at commit [`ec9b83f`](https://github.com/apache/spark/commit/ec9b83f5fa0f91a180ad6af36784f6404ddf2299). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics table doesn...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23038 **[Test build #98863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98863/testReport)** for PR 23038 at commit [`c53ca48`](https://github.com/apache/spark/commit/c53ca48885d14dc6a70412fe27a00c8f71aff561). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics table doesn...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23038 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98863/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22974 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98864/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22974 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics table doesn...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23038 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23000: [SPARK-26002][SQL] Fix day of year calculation for Julia...
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/23000 ping @10110346 @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comm...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23044#discussion_r233827093
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
---
@@ -76,8 +76,6 @@ object GenerateOrdering extends
CodeGenerator[Seq[SortOrder], Ordering[InternalR
row: String,
ordering: Seq[SortOrder]): Seq[ExprCode] = {
ctx.INPUT_ROW = row
-// to use INPUT_ROW we must make sure currentVars is null
-ctx.currentVars = null
--- End diff --
I think so.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23034: [WIP][SPARK-26035][PYTHON] Break large streaming/tests.p...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23034 Will go and merge this tomorrow if there's no outstanding issues. cc @zsxwing and @tdas. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
