[GitHub] [spark] AmplabJenkins commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base
AmplabJenkins commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base URL: https://github.com/apache/spark/pull/25195#issuecomment-513882926 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base
SparkQA removed a comment on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base URL: https://github.com/apache/spark/pull/25195#issuecomment-513795422 **[Test build #108015 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108015/testReport)** for PR 25195 at commit [`d376e95`](https://github.com/apache/spark/commit/d376e95a7ac7b97347e9bc24a94198e774f514ec). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base
SparkQA commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base URL: https://github.com/apache/spark/pull/25195#issuecomment-513882238 **[Test build #108015 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108015/testReport)** for PR 25195 at commit [`d376e95`](https://github.com/apache/spark/commit/d376e95a7ac7b97347e9bc24a94198e774f514ec). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base
SparkQA commented on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25161#issuecomment-513881926 **[Test build #108021 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108021/testReport)** for PR 25161 at commit [`c4ad657`](https://github.com/apache/spark/commit/c4ad6575f12e838d80e1a6ddd6da6a2e2722b6ff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25208: [SPARK-28455][core] Avoid overflow when calculating executor timeout time.
SparkQA commented on issue #25208: [SPARK-28455][core] Avoid overflow when calculating executor timeout time. URL: https://github.com/apache/spark/pull/25208#issuecomment-513878907 **[Test build #108020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108020/testReport)** for PR 25208 at commit [`392e585`](https://github.com/apache/spark/commit/392e5858c9029dc2e4d81198e241b2fadc5335eb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25208: [SPARK-28455][core] Avoid overflow when calculating executor timeout time.
AmplabJenkins removed a comment on issue #25208: [SPARK-28455][core] Avoid overflow when calculating executor timeout time. URL: https://github.com/apache/spark/pull/25208#issuecomment-513878138 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25208: [SPARK-28455][core] Avoid overflow when calculating executor timeout time.
AmplabJenkins removed a comment on issue #25208: [SPARK-28455][core] Avoid overflow when calculating executor timeout time. URL: https://github.com/apache/spark/pull/25208#issuecomment-513878146 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13128/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25208: [SPARK-28455][core] Avoid overflow when calculating executor timeout time.
AmplabJenkins commented on issue #25208: [SPARK-28455][core] Avoid overflow when calculating executor timeout time. URL: https://github.com/apache/spark/pull/25208#issuecomment-513878138 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25208: [SPARK-28455][core] Avoid overflow when calculating executor timeout time.
AmplabJenkins commented on issue #25208: [SPARK-28455][core] Avoid overflow when calculating executor timeout time. URL: https://github.com/apache/spark/pull/25208#issuecomment-513878146 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13128/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] erikerlandson commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
erikerlandson commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-513877664 I was originally anticipating/hoping that spark's config for making use of volumes could also be managed w/out new code to detect configured volumes, e.g. with `--conf`. If there is a logistic or UX issue with that, then we should upgrade the code, but I'm trying to clarify if that is only possible approach. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] erikerlandson commented on a change in pull request #24879: [SPARK-28042][K8S] Support using volume mount as local storage
erikerlandson commented on a change in pull request #24879: [SPARK-28042][K8S]
Support using volume mount as local storage
URL: https://github.com/apache/spark/pull/24879#discussion_r305938310
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala
##
@@ -43,33 +44,52 @@ private[spark] class LocalDirsFeatureStep(
val localDirVolumes = resolvedLocalDirs
.zipWithIndex
.map { case (localDir, index) =>
-new VolumeBuilder()
- .withName(s"spark-local-dir-${index + 1}")
- .withNewEmptyDir()
-.withMedium(if (useLocalDirTmpFs) "Memory" else null)
- .endEmptyDir()
- .build()
+val name = s"spark-local-dir-${index + 1}"
+findVolume(pod, name) match {
+ case Some(volume) => volume
+ case None =>
+new VolumeBuilder()
+ .withName(s"spark-local-dir-${index + 1}")
Review comment:
What is the scope of visibility on volume names "spark-local-dir-1",
"spark-local-dir-2" ...
Can those names collide with volumes created by other running jobs?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] erikerlandson commented on issue #24962: [SPARK-28149][K8S] Added variable to disable negative DNS caching
erikerlandson commented on issue #24962: [SPARK-28149][K8S] Added variable to disable negative DNS caching URL: https://github.com/apache/spark/pull/24962#issuecomment-513871155 @jlpedrosa I'm guessing no specific doc was added because config-files were intended to be just one of an arbitrary number of things that could be managed via templates, starting with https://github.com/apache/spark/pull/22146/ If config-files are not working via pod template files, that needs to be reconciled somehow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
vanzin commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-513870028 I just checked the k8s API and it does not seem like the `VolumeMount` type has labels or anything much other than volume name / mount point. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] erikerlandson commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
erikerlandson commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-513867466 > "unfortunately the API doesn't have any way to do that explicitly" is "the API" the fabric8 api, or k8s itself? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
vanzin commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-513858728 > unfortunately the API doesn't have any way to do that explicitly Yeah that's the part I did not get around to actually testing. > we could do it by having a naming convention e.g. local-dir- That sounds good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error
AmplabJenkins removed a comment on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error URL: https://github.com/apache/spark/pull/25226#issuecomment-513852239 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108010/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error
AmplabJenkins removed a comment on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error URL: https://github.com/apache/spark/pull/25226#issuecomment-513852229 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23677: [SPARK-26755][SCHEDULER] : Optimize Spark Scheduler to dequeue speculative tasks…
SparkQA commented on issue #23677: [SPARK-26755][SCHEDULER] : Optimize Spark Scheduler to dequeue speculative tasks… URL: https://github.com/apache/spark/pull/23677#issuecomment-513852277 **[Test build #108019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108019/testReport)** for PR 23677 at commit [`685bbae`](https://github.com/apache/spark/commit/685bbae298eb84de4282b690fd410d8404e6c424). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error
AmplabJenkins commented on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error URL: https://github.com/apache/spark/pull/25226#issuecomment-513852229 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error
AmplabJenkins commented on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error URL: https://github.com/apache/spark/pull/25226#issuecomment-513852239 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108010/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error
SparkQA removed a comment on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error URL: https://github.com/apache/spark/pull/25226#issuecomment-513756280 **[Test build #108010 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108010/testReport)** for PR 25226 at commit [`e4ae98c`](https://github.com/apache/spark/commit/e4ae98cb8ea51815a9e9c0f2ee470ed2ebc55718). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error
SparkQA commented on issue #25226: [SPARK-28468][INFRA][2.4] Upgrade pip to fix `sphinx` install error URL: https://github.com/apache/spark/pull/25226#issuecomment-513851793 **[Test build #108010 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108010/testReport)** for PR 25226 at commit [`e4ae98c`](https://github.com/apache/spark/commit/e4ae98cb8ea51815a9e9c0f2ee470ed2ebc55718). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25202: [SPARK-28289][SQL][PYTHON][TESTS] Convert and port 'union.sql' into UDF test base
SparkQA removed a comment on issue #25202: [SPARK-28289][SQL][PYTHON][TESTS] Convert and port 'union.sql' into UDF test base URL: https://github.com/apache/spark/pull/25202#issuecomment-513760941 **[Test build #108011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108011/testReport)** for PR 25202 at commit [`760eb02`](https://github.com/apache/spark/commit/760eb02b067c83cf3ffcddd7d88e5a638ae35c57). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
SparkQA removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513839115 **[Test build #108018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108018/testReport)** for PR 25228 at commit [`3fa6e53`](https://github.com/apache/spark/commit/3fa6e53c6020f70e4469ed18fc5d4d8167c1c23f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513846150 **[Test build #108018 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108018/testReport)** for PR 25228 at commit [`3fa6e53`](https://github.com/apache/spark/commit/3fa6e53c6020f70e4469ed18fc5d4d8167c1c23f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25202: [SPARK-28289][SQL][PYTHON][TESTS] Convert and port 'union.sql' into UDF test base
SparkQA commented on issue #25202: [SPARK-28289][SQL][PYTHON][TESTS] Convert and port 'union.sql' into UDF test base URL: https://github.com/apache/spark/pull/25202#issuecomment-513845561 **[Test build #108011 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108011/testReport)** for PR 25202 at commit [`760eb02`](https://github.com/apache/spark/commit/760eb02b067c83cf3ffcddd7d88e5a638ae35c57). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
gengliangwang commented on a change in pull request #25172: [SPARK-28412][SQL]
ANSI SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r305909883
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
##
@@ -452,6 +452,45 @@ class StringExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
// scalastyle:on
}
+ test("overlay for byte array") {
+val input = Literal(Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9))
+checkEvaluation(new Overlay(input, Literal(Array[Byte](-1)),
+ Literal.create(6, IntegerType)), Array[Byte](1, 2, 3, 4, 5, -1, 7, 8, 9))
Review comment:
Nit: Also test corner case, e.g. position is negative, or position is larger
than 9
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #25227: [SPARK-28467][CORE][TEST] Increase timeout to up executors for tests
srowen commented on issue #25227: [SPARK-28467][CORE][TEST] Increase timeout to up executors for tests URL: https://github.com/apache/spark/pull/25227#issuecomment-513841354 Although this change isn't so bad, as it won't cause faster machines to test more slowly, I'm also hesitant as here just seems causes by using underpowered test machines. We wouldn't up this to help Spark tests on say a small 4 core machine. If a1.4xlarge or equivalent works then let's suggest that this is what the tests need. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on issue #25138: [SPARK-26175][PYSPARK] Closing stdin of the worker process right after fork
WeichenXu123 commented on issue #25138: [SPARK-26175][PYSPARK] Closing stdin of the worker process right after fork URL: https://github.com/apache/spark/pull/25138#issuecomment-513839860 Gently ping @HyukjinKwon @ueshin This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513839115 **[Test build #108018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108018/testReport)** for PR 25228 at commit [`3fa6e53`](https://github.com/apache/spark/commit/3fa6e53c6020f70e4469ed18fc5d4d8167c1c23f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513838300 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13127/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513838285 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513838285 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513838300 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13127/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25227: [SPARK-28467][CORE][TEST] Increase timeout to up executors for tests
dongjoon-hyun commented on issue #25227: [SPARK-28467][CORE][TEST] Increase timeout to up executors for tests URL: https://github.com/apache/spark/pull/25227#issuecomment-513827923 I'm here because I read that email. :) However, that cannot be a reason to accept this PR. Since this is not a general issue for `aarch64`, I'm reluctant for this kind of assumption. In general, EC2 is a de-facto standard infra which is more easily accessible to most of the users. If we needs `aarch64` support, I'd like to recommend our community to use `a1.4xlarge` as a standard instance for release testing and benchmarking. BTW, please don't forget my previous comments. You didn't update this PR according to my advices at all. PMC members can override my opinion. - https://github.com/apache/spark/pull/25227#pullrequestreview-264630874 - https://github.com/apache/spark/pull/25227#issuecomment-513698776 cc @srowen , @rxin This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README
AmplabJenkins commented on issue #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-513820735 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README
AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-513820471 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README
AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-513818331 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README
AmplabJenkins commented on issue #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-513820471 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README
AmplabJenkins commented on issue #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-513818331 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Mister-Meeseeks opened a new pull request #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README
Mister-Meeseeks opened a new pull request #25231: [SPARK-28473][DOCUMENTATION]Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231 ## What changes were proposed in this pull request? Change the format of the build command in the README to start with a `./` prefix ./build/mvn -DskipTests clean package This increases stylistic consistency across the README- all the other commands have a `./` prefix. Having a visible `./` prefix also makes it clear to the user that the shell command is dependent on the current working directory. ## How was this patch tested? README.md was reviewed both in raw markdown and in the Github rendered landing page for stylistic consistency. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25230: [SPARK-28471][SQL] Replace `yyyy` by `uuuu` in date-timestamp patterns without era
AmplabJenkins removed a comment on issue #25230: [SPARK-28471][SQL] Replace `` by `` in date-timestamp patterns without era URL: https://github.com/apache/spark/pull/25230#issuecomment-513813738 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25230: [SPARK-28471][SQL] Replace `yyyy` by `uuuu` in date-timestamp patterns without era
SparkQA commented on issue #25230: [SPARK-28471][SQL] Replace `` by `` in date-timestamp patterns without era URL: https://github.com/apache/spark/pull/25230#issuecomment-513815096 **[Test build #108017 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108017/testReport)** for PR 25230 at commit [`3e4426c`](https://github.com/apache/spark/commit/3e4426c17b035e6d39794c5d0bea5c3debaca8dc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25230: [SPARK-28471][SQL] Replace `yyyy` by `uuuu` in date-timestamp patterns without era
AmplabJenkins removed a comment on issue #25230: [SPARK-28471][SQL] Replace `` by `` in date-timestamp patterns without era URL: https://github.com/apache/spark/pull/25230#issuecomment-513814174 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25230: [SPARK-28471][SQL] Replace `yyyy` by `uuuu` in date-timestamp patterns without era
AmplabJenkins removed a comment on issue #25230: [SPARK-28471][SQL] Replace `` by `` in date-timestamp patterns without era URL: https://github.com/apache/spark/pull/25230#issuecomment-513814179 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13126/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25230: [SPARK-28471][SQL] Replace `yyyy` by `uuuu` in date-timestamp patterns without era
AmplabJenkins commented on issue #25230: [SPARK-28471][SQL] Replace `` by `` in date-timestamp patterns without era URL: https://github.com/apache/spark/pull/25230#issuecomment-513814179 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13126/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25230: [SPARK-28471][SQL] Replace `yyyy` by `uuuu` in date-timestamp patterns without era
AmplabJenkins commented on issue #25230: [SPARK-28471][SQL] Replace `` by `` in date-timestamp patterns without era URL: https://github.com/apache/spark/pull/25230#issuecomment-513814174 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25230: [SPARK-28471][SQL] Replace `yyyy` by `uuuu` in date-timestamp patterns without era
AmplabJenkins commented on issue #25230: [SPARK-28471][SQL] Replace `` by `` in date-timestamp patterns without era URL: https://github.com/apache/spark/pull/25230#issuecomment-513813738 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk opened a new pull request #25230: [SPARK-28471][SQL] Replace `yyyy` by `uuuu` in date-timestamp patterns without era
MaxGekk opened a new pull request #25230: [SPARK-28471][SQL] Replace `` by `` in date-timestamp patterns without era URL: https://github.com/apache/spark/pull/25230 ## What changes were proposed in this pull request? In the PR, I propose to use `` for years instead of `` in date/timestamp patterns without the era pattern `G` (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html). **Parsing/formatting of positive years (current era) will be the same.** The difference is in formatting negative years belong to previous era - BC (Before Christ). Before the changes, the year of common era `100` and the year of BC era `-99`, showed similarly as `100`. After the changes negative years will be formatted with the `-` sign. Before: ```Scala scala> Seq(java.time.LocalDate.of(-99, 1, 1)).toDF().show +--+ | value| +--+ |0100-01-01| +--+ ``` After: ```Scala scala> Seq(java.time.LocalDate.of(-99, 1, 1)).toDF().show +---+ | value| +---+ |-0099-01-01| +---+ ``` ## How was this patch tested? By existing test suites. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-513809967 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108008/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-513809967 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108008/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-513809958 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-513809958 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python]
Fix error when PythonUDF is used in correlated scalar subquery
URL: https://github.com/apache/spark/pull/25204#discussion_r305868232
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##
@@ -318,17 +318,31 @@ object RewriteCorrelatedScalarSubquery extends
Rule[LogicalPlan] {
/**
* Statically evaluate an expression containing zero or more placeholders,
given a set
- * of bindings for placeholder values.
+ * of bindings for placeholder values, if the expression is evaluable. If it
is not,
+ * bind statically evaluated expression results to an expression.
*/
- private def evalExpr(expr: Expression, bindings: Map[ExprId, Option[Any]]) :
Option[Any] = {
+ private def bindingExpr(
+ expr: Expression,
+ bindings: Map[ExprId, Option[Expression]]): Option[Expression] = {
val rewrittenExpr = expr transform {
case r: AttributeReference =>
bindings(r.exprId) match {
- case Some(v) => Literal.create(v, r.dataType)
+ case Some(v) => v
case None => Literal.default(NullType)
}
}
-Option(rewrittenExpr.eval())
+if (!rewrittenExpr.foldable) {
Review comment:
so it is not covered by added test. Let me add test for it...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
SparkQA removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-513719209 **[Test build #108008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108008/testReport)** for PR 25001 at commit [`3d01650`](https://github.com/apache/spark/commit/3d01650292794adc639f878cce6025155264144d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python]
Fix error when PythonUDF is used in correlated scalar subquery
URL: https://github.com/apache/spark/pull/25204#discussion_r305867691
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##
@@ -318,17 +318,31 @@ object RewriteCorrelatedScalarSubquery extends
Rule[LogicalPlan] {
/**
* Statically evaluate an expression containing zero or more placeholders,
given a set
- * of bindings for placeholder values.
+ * of bindings for placeholder values, if the expression is evaluable. If it
is not,
+ * bind statically evaluated expression results to an expression.
*/
- private def evalExpr(expr: Expression, bindings: Map[ExprId, Option[Any]]) :
Option[Any] = {
+ private def bindingExpr(
+ expr: Expression,
+ bindings: Map[ExprId, Option[Expression]]): Option[Expression] = {
val rewrittenExpr = expr transform {
case r: AttributeReference =>
bindings(r.exprId) match {
- case Some(v) => Literal.create(v, r.dataType)
+ case Some(v) => v
case None => Literal.default(NullType)
}
}
-Option(rewrittenExpr.eval())
+if (!rewrittenExpr.foldable) {
Review comment:
yes. this is not possible for PythonUDF, but it is potential for other not
foldable expression.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
SparkQA commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-513809094 **[Test build #108008 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108008/testReport)** for PR 25001 at commit [`3d01650`](https://github.com/apache/spark/commit/3d01650292794adc639f878cce6025155264144d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22502: [SPARK-25474][SQL]When the "fallBackToHdfsForStats= true", Size in bytes is coming as default size in bytes ( 8.0 EB)
SparkQA commented on issue #22502: [SPARK-25474][SQL]When the "fallBackToHdfsForStats= true", Size in bytes is coming as default size in bytes ( 8.0 EB) URL: https://github.com/apache/spark/pull/22502#issuecomment-513804905 **[Test build #108016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108016/testReport)** for PR 22502 at commit [`f2b69cf`](https://github.com/apache/spark/commit/f2b69cf239d08b8b4bb3c0858cebf03b41ae9068). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #25047: [WIP][SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
tgravescs commented on a change in pull request #25047:
[WIP][SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
URL: https://github.com/apache/spark/pull/25047#discussion_r305859452
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -35,6 +35,16 @@ package object config {
private[spark] val SPARK_EXECUTOR_PREFIX = "spark.executor"
private[spark] val SPARK_TASK_PREFIX = "spark.task"
+ private[spark] val SPARK_RESOURCES_DIR =
+ConfigBuilder("spark.resources.dir")
+ .doc("Directory used to create spark-resources directory under it. If
user does not" +
+"config it explicitly, SPARK_HOME will be used by default. And since
spark-resources" +
Review comment:
I was meaning having separate configs - spark.worker.resource.dir and
spark.driver.resource.dir. Its not required and I'm not an expert in running
standalone mode, its just an option and I'll default to people who run
standalone mode more then myself as to what the normal setup is. If people
generally use separate config files and setups for Workers then drivers then
its not an issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #25047: [WIP][SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
tgravescs commented on a change in pull request #25047:
[WIP][SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
URL: https://github.com/apache/spark/pull/25047#discussion_r305858178
##
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##
@@ -268,6 +268,26 @@ private[spark] object Utils extends Logging {
file.setExecutable(true, true)
}
+ /**
+ * Create a directory given the abstract pathname
+ * @return true, if the directory is successfully created; otherwise, return
false.
+ */
+ def createDirectory(dir: File): Boolean = {
+try {
+ // This sporadically fails - not sure why ... !dir.exists() &&
!dir.mkdirs()
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rvesse commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
rvesse commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-513798601 @vanzin Sorry if I misconstrued your position The idea of having a tag is a nice one, unfortunately the API doesn't have any way to do that explicitly. I think we could do it by having a naming convention e.g. `local-dir-` in lieu of that if that would be acceptable? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base
HyukjinKwon closed pull request #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base URL: https://github.com/apache/spark/pull/25098 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base
HyukjinKwon commented on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base URL: https://github.com/apache/spark/pull/25098#issuecomment-513795860 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base
SparkQA commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base URL: https://github.com/apache/spark/pull/25195#issuecomment-513795422 **[Test build #108015 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108015/testReport)** for PR 25195 at commit [`d376e95`](https://github.com/apache/spark/commit/d376e95a7ac7b97347e9bc24a94198e774f514ec). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base
AmplabJenkins removed a comment on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base URL: https://github.com/apache/spark/pull/25195#issuecomment-513794657 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13125/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base
AmplabJenkins removed a comment on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base URL: https://github.com/apache/spark/pull/25195#issuecomment-513794650 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] younggyuchun commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base
younggyuchun commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base URL: https://github.com/apache/spark/pull/25195#issuecomment-513794750 Done. Thank you @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base
AmplabJenkins commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base URL: https://github.com/apache/spark/pull/25195#issuecomment-513794650 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base
AmplabJenkins commented on issue #25195: [SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base URL: https://github.com/apache/spark/pull/25195#issuecomment-513794657 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13125/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base
AmplabJenkins removed a comment on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base URL: https://github.com/apache/spark/pull/25098#issuecomment-513793603 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base
SparkQA removed a comment on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base URL: https://github.com/apache/spark/pull/25098#issuecomment-513713465 **[Test build #108006 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108006/testReport)** for PR 25098 at commit [`9dc5aa1`](https://github.com/apache/spark/commit/9dc5aa12cfd0783fb75876611bad4bb054a6d819). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base
AmplabJenkins removed a comment on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base URL: https://github.com/apache/spark/pull/25098#issuecomment-513793614 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108006/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base
AmplabJenkins commented on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base URL: https://github.com/apache/spark/pull/25098#issuecomment-513793614 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108006/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base
AmplabJenkins commented on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base URL: https://github.com/apache/spark/pull/25098#issuecomment-513793603 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base
SparkQA commented on issue #25098: [SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base URL: https://github.com/apache/spark/pull/25098#issuecomment-513792867 **[Test build #108006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108006/testReport)** for PR 25098 at commit [`9dc5aa1`](https://github.com/apache/spark/commit/9dc5aa12cfd0783fb75876611bad4bb054a6d819). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function
AmplabJenkins removed a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513785859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108003/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function
AmplabJenkins removed a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513785851 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function
AmplabJenkins commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513785851 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function
AmplabJenkins commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513785859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108003/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function
SparkQA removed a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513704780 **[Test build #108003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108003/testReport)** for PR 25210 at commit [`15c64d2`](https://github.com/apache/spark/commit/15c64d2dfcb616334e1d054f9fdb485f1870ed81). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function
SparkQA commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513785183 **[Test build #108003 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108003/testReport)** for PR 25210 at commit [`15c64d2`](https://github.com/apache/spark/commit/15c64d2dfcb616334e1d054f9fdb485f1870ed81). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513783057 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108013/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513783046 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513783046 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513783057 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108013/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25229: [SPARK-27900][K8s] Add jvm oom flag
AmplabJenkins removed a comment on issue #25229: [SPARK-27900][K8s] Add jvm oom flag URL: https://github.com/apache/spark/pull/25229#issuecomment-513782148 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108014/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513782946 **[Test build #108013 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108013/testReport)** for PR 25228 at commit [`d1f4d26`](https://github.com/apache/spark/commit/d1f4d26967b7cead5a131ef94eec5181a1f5368c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions
SparkQA removed a comment on issue #25228: [SPARK-28472][SQL][TEST][test-hadoop3.2] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-513775543 **[Test build #108013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108013/testReport)** for PR 25228 at commit [`d1f4d26`](https://github.com/apache/spark/commit/d1f4d26967b7cead5a131ef94eec5181a1f5368c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25229: [SPARK-27900][K8s] Add jvm oom flag
SparkQA removed a comment on issue #25229: [SPARK-27900][K8s] Add jvm oom flag URL: https://github.com/apache/spark/pull/25229#issuecomment-513778152 **[Test build #108014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108014/testReport)** for PR 25229 at commit [`81ec5c1`](https://github.com/apache/spark/commit/81ec5c1accde23820c84a5267e517825e848935b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25229: [SPARK-27900][K8s] Add jvm oom flag
AmplabJenkins removed a comment on issue #25229: [SPARK-27900][K8s] Add jvm oom flag URL: https://github.com/apache/spark/pull/25229#issuecomment-513782143 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25229: [SPARK-27900][K8s] Add jvm oom flag
AmplabJenkins commented on issue #25229: [SPARK-27900][K8s] Add jvm oom flag URL: https://github.com/apache/spark/pull/25229#issuecomment-513782143 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-513781409 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108005/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25229: [SPARK-27900][K8s] Add jvm oom flag
AmplabJenkins commented on issue #25229: [SPARK-27900][K8s] Add jvm oom flag URL: https://github.com/apache/spark/pull/25229#issuecomment-513782148 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108014/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25229: [SPARK-27900][K8s] Add jvm oom flag
SparkQA commented on issue #25229: [SPARK-27900][K8s] Add jvm oom flag URL: https://github.com/apache/spark/pull/25229#issuecomment-513782028 **[Test build #108014 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108014/testReport)** for PR 25229 at commit [`81ec5c1`](https://github.com/apache/spark/commit/81ec5c1accde23820c84a5267e517825e848935b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-513781403 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-513781409 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108005/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-513781403 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
SparkQA removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-513713469 **[Test build #108005 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108005/testReport)** for PR 25204 at commit [`7972d7c`](https://github.com/apache/spark/commit/7972d7c6508ed3f37a61054e90403a850a13bc0d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
SparkQA commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-513781113 **[Test build #108005 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108005/testReport)** for PR 25204 at commit [`7972d7c`](https://github.com/apache/spark/commit/7972d7c6508ed3f37a61054e90403a850a13bc0d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25134: [SPARK-28366][CORE] Logging in driver when loading single large unsplittable file
cloud-fan commented on a change in pull request #25134: [SPARK-28366][CORE]
Logging in driver when loading single large unsplittable file
URL: https://github.com/apache/spark/pull/25134#discussion_r305828949
##
File path: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala
##
@@ -207,6 +210,15 @@ class HadoopRDD[K, V](
} else {
allInputSplits
}
+ if (inputSplits.length == 1 && inputSplits(0).isInstanceOf[FileSplit]) {
+val fileSplit = inputSplits(0).asInstanceOf[FileSplit]
+val path = fileSplit.getPath
+if (Utils.isFileSplittable(path, codecFactory)
Review comment:
SGTM, let's include the reason in the message.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
