[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513651629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13097/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513651622 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513651629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13097/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513651622 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513649115 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107987/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
SparkQA removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513648666 **[Test build #107987 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107987/testReport)** for PR 25172 at commit [`2013890`](https://github.com/apache/spark/commit/2013890a9ee4ae467cf84b3446e0bd4622ae1170). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513649104 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513649104 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513649115 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107987/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513649101 **[Test build #107987 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107987/testReport)** for PR 25172 at commit [`2013890`](https://github.com/apache/spark/commit/2013890a9ee4ae467cf84b3446e0bd4622ae1170). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on a change in pull request #25222: [SPARK-28465][K8s] Fix integration tests which fail due to missing ceph-nano image
felixcheung commented on a change in pull request #25222: [SPARK-28465][K8s]
Fix integration tests which fail due to missing ceph-nano image
URL: https://github.com/apache/spark/pull/25222#discussion_r305680509
##
File path:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala
##
@@ -63,7 +63,7 @@ private[spark] trait DepsTestsSuite { k8sSuite:
KubernetesSuite =>
).asJava
new ContainerBuilder()
- .withImage("ceph/daemon:v4.0.0-stable-4.0-master-centos-7-x86_64")
+ .withImage("ceph/daemon:latest")
Review comment:
this might break in a future (or future major) version of ceph/daemon? we
are sure this will be always compatible?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513648666 **[Test build #107987 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107987/testReport)** for PR 25172 at commit [`2013890`](https://github.com/apache/spark/commit/2013890a9ee4ae467cf84b3446e0bd4622ae1170). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513647990 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107984/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513648275 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513648269 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25189: [SPARK-28435][SQL] Support cast StringType to IntervalType for SQL interface
dongjoon-hyun commented on issue #25189: [SPARK-28435][SQL] Support cast StringType to IntervalType for SQL interface URL: https://github.com/apache/spark/pull/25189#issuecomment-513648331 Otherwise, you can update this PR title and description according to your contribution. If you suggest that, of course, it's possible and those PR look more clear as a single PR. For (1), the PR should describe the context in the PR description instead of pointing this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
SparkQA removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513636436 **[Test build #107984 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107984/testReport)** for PR 25212 at commit [`98893e7`](https://github.com/apache/spark/commit/98893e74e7acde53424003b488d9eee226a9887e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513648275 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-513648269 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513647983 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513647990 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107984/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513647983 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
SparkQA commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513647896 **[Test build #107984 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107984/testReport)** for PR 25212 at commit [`98893e7`](https://github.com/apache/spark/commit/98893e74e7acde53424003b488d9eee226a9887e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization …
dongjoon-hyun closed pull request #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization … URL: https://github.com/apache/spark/pull/25223 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization …
AmplabJenkins removed a comment on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization … URL: https://github.com/apache/spark/pull/25223#issuecomment-513642796 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization …
AmplabJenkins removed a comment on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization … URL: https://github.com/apache/spark/pull/25223#issuecomment-513642799 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107974/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization …
AmplabJenkins commented on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization … URL: https://github.com/apache/spark/pull/25223#issuecomment-513642796 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization …
AmplabJenkins commented on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization … URL: https://github.com/apache/spark/pull/25223#issuecomment-513642799 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107974/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization …
SparkQA removed a comment on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization … URL: https://github.com/apache/spark/pull/25223#issuecomment-513601469 **[Test build #107974 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107974/testReport)** for PR 25223 at commit [`d187ae3`](https://github.com/apache/spark/commit/d187ae3c90cc126b6da53e9a8b23f9ac52b09500). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization …
SparkQA commented on issue #25223: [SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization … URL: https://github.com/apache/spark/pull/25223#issuecomment-513642573 **[Test build #107974 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107974/testReport)** for PR 25223 at commit [`d187ae3`](https://github.com/apache/spark/commit/d187ae3c90cc126b6da53e9a8b23f9ac52b09500). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public final class UnsafeArrayData extends ArrayData implements Externalizable, KryoSerializable ` * `final class UnsafeDataUtils ` * `public final class UnsafeMapData extends MapData implements Externalizable, KryoSerializable ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513640812 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107985/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513640808 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
SparkQA removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513638908 **[Test build #107985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107985/testReport)** for PR 25224 at commit [`be8ae40`](https://github.com/apache/spark/commit/be8ae4071e7057ea189679f1ae9f8459ba3ad88c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513640808 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513640812 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107985/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
SparkQA commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513640748 **[Test build #107985 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107985/testReport)** for PR 25224 at commit [`be8ae40`](https://github.com/apache/spark/commit/be8ae4071e7057ea189679f1ae9f8459ba3ad88c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
cloud-fan commented on a change in pull request #25204:
[SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated
scalar subquery
URL: https://github.com/apache/spark/pull/25204#discussion_r305673679
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##
@@ -318,17 +318,30 @@ object RewriteCorrelatedScalarSubquery extends
Rule[LogicalPlan] {
/**
* Statically evaluate an expression containing zero or more placeholders,
given a set
- * of bindings for placeholder values.
+ * of bindings for placeholder values, if the expression is evaluable. If it
is not,
+ * bind statically evaluated expression results to an expression.
*/
- private def evalExpr(expr: Expression, bindings: Map[ExprId, Option[Any]]) :
Option[Any] = {
+ private def bindingExpr(
+ expr: Expression,
+ bindings: Map[ExprId, Option[Expression]]): Option[Expression] = {
val rewrittenExpr = expr transform {
case r: AttributeReference =>
bindings(r.exprId) match {
- case Some(v) => Literal.create(v, r.dataType)
+ case Some(v) => v
case None => Literal.default(NullType)
}
}
-Option(rewrittenExpr.eval())
+if (rewrittenExpr.find(_.isInstanceOf[PythonUDF]).isDefined) {
+ // SPARK-28441: `PythonUDF` can't be statically evaluated.
Review comment:
It seems necessary to me to check foldable before calling `.eval()`,
otherwise there is no guarantee that `.eval()` can success.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)
SparkQA commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-513640191 **[Test build #107986 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107986/testReport)** for PR 24946 at commit [`b103acc`](https://github.com/apache/spark/commit/b103acc6631c8f5fe9292f34c53f53d2d8edc4ab). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)
AmplabJenkins commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-513639876 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13095/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)
AmplabJenkins commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-513639874 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)
AmplabJenkins removed a comment on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-513639874 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)
AmplabJenkins removed a comment on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-513639876 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13095/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
SparkQA commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513638908 **[Test build #107985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107985/testReport)** for PR 25224 at commit [`be8ae40`](https://github.com/apache/spark/commit/be8ae4071e7057ea189679f1ae9f8459ba3ad88c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513638546 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13094/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513638538 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513638546 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13094/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513638538 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
HyukjinKwon commented on a change in pull request #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#discussion_r305671844 ## File path: docs/sql-migration-guide-upgrade.md ## @@ -151,7 +151,9 @@ license: | - Since Spark 3.0, substitution order of nested WITH clauses is changed and an inner CTE definition takes precedence over an outer. In version 2.4 and earlier, `WITH t AS (SELECT 1), t2 AS (WITH t AS (SELECT 2) SELECT * FROM t) SELECT * FROM t2` returns `1` while in version 3.0 it returns `2`. The previous behaviour can be restored by setting `spark.sql.legacy.ctePrecedence.enabled` to `true`. - - Since Spark 3.0, the `add_months` function does not adjust the resulting date to a last day of month if the original date is a last day of months. For example, `select add_months(DATE'2019-02-28', 1)` results `2019-03-28`. In Spark version 2.4 and earlier, the resulting date is adjusted when the original date is a last day of months. For example, adding a month to `2019-02-28` resultes in `2019-03-31`. + - Since Spark 3.0, the `add_months` function does not adjust the resulting date to a last day of month if the original date is a last day of months. For example, `select add_months(DATE'2019-02-28', 1)` results `2019-03-28`. In Spark version 2.4 and earlier, the resulting date is adjusted when the original date is a last day of months. For example, adding a month to `2019-02-28` results in `2019-03-31`. Review comment: Just a typo fix from `resultes` -> `results`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
HyukjinKwon removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513636088 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python]
Fix error when PythonUDF is used in correlated scalar subquery
URL: https://github.com/apache/spark/pull/25204#discussion_r305671382
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##
@@ -318,17 +318,30 @@ object RewriteCorrelatedScalarSubquery extends
Rule[LogicalPlan] {
/**
* Statically evaluate an expression containing zero or more placeholders,
given a set
- * of bindings for placeholder values.
+ * of bindings for placeholder values, if the expression is evaluable. If it
is not,
+ * bind statically evaluated expression results to an expression.
*/
- private def evalExpr(expr: Expression, bindings: Map[ExprId, Option[Any]]) :
Option[Any] = {
+ private def bindingExpr(
+ expr: Expression,
+ bindings: Map[ExprId, Option[Expression]]): Option[Expression] = {
val rewrittenExpr = expr transform {
case r: AttributeReference =>
bindings(r.exprId) match {
- case Some(v) => Literal.create(v, r.dataType)
+ case Some(v) => v
case None => Literal.default(NullType)
}
}
-Option(rewrittenExpr.eval())
+if (rewrittenExpr.find(_.isInstanceOf[PythonUDF]).isDefined) {
+ // SPARK-28441: `PythonUDF` can't be statically evaluated.
Review comment:
For 1, AttributeReference was replaced with pre-evaluated value, if it comes
from aggregate function. It uses default value. It fakes empty input case. Or
null, if it is not.
For 2, I think it is potential issue.
Yeah, here the hack looks like foldable expression. It simulates empty input.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python]
Fix error when PythonUDF is used in correlated scalar subquery
URL: https://github.com/apache/spark/pull/25204#discussion_r305671382
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##
@@ -318,17 +318,30 @@ object RewriteCorrelatedScalarSubquery extends
Rule[LogicalPlan] {
/**
* Statically evaluate an expression containing zero or more placeholders,
given a set
- * of bindings for placeholder values.
+ * of bindings for placeholder values, if the expression is evaluable. If it
is not,
+ * bind statically evaluated expression results to an expression.
*/
- private def evalExpr(expr: Expression, bindings: Map[ExprId, Option[Any]]) :
Option[Any] = {
+ private def bindingExpr(
+ expr: Expression,
+ bindings: Map[ExprId, Option[Expression]]): Option[Expression] = {
val rewrittenExpr = expr transform {
case r: AttributeReference =>
bindings(r.exprId) match {
- case Some(v) => Literal.create(v, r.dataType)
+ case Some(v) => v
case None => Literal.default(NullType)
}
}
-Option(rewrittenExpr.eval())
+if (rewrittenExpr.find(_.isInstanceOf[PythonUDF]).isDefined) {
+ // SPARK-28441: `PythonUDF` can't be statically evaluated.
Review comment:
For 1, AttributeReference was replaced with pre-evaluated value, if it comes
from aggregate function. It uses default value. It fakes empty input case.
For 2, I think it is potential issue.
Yeah, here the hack looks like foldable expression. It simulates empty input.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513637323 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513637325 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13093/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513637205 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513637208 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107983/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513637323 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513637325 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13093/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513637208 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107983/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513637205 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
SparkQA removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513635225 **[Test build #107983 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107983/testReport)** for PR 25224 at commit [`3aaec73`](https://github.com/apache/spark/commit/3aaec73da4c6a42141bee1d4c4a9a7fadee71c2c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
SparkQA commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513637137 **[Test build #107983 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107983/testReport)** for PR 25224 at commit [`3aaec73`](https://github.com/apache/spark/commit/3aaec73da4c6a42141bee1d4c4a9a7fadee71c2c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #25209: [SPARK-28456][SQL] Add a public API `Encoder.makeCopy` to allow creating Encoder without touching Scala Reflection
cloud-fan closed pull request #25209: [SPARK-28456][SQL] Add a public API `Encoder.makeCopy` to allow creating Encoder without touching Scala Reflection URL: https://github.com/apache/spark/pull/25209 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
HyukjinKwon edited a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513634394 cc @gatorsmile and @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
SparkQA commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513636436 **[Test build #107984 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107984/testReport)** for PR 25212 at commit [`98893e7`](https://github.com/apache/spark/commit/98893e74e7acde53424003b488d9eee226a9887e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
HyukjinKwon commented on a change in pull request #25184: [SPARK-28431][SQL]
Set maximum error message length in CSV datasource's parsing and writing
URL: https://github.com/apache/spark/pull/25184#discussion_r305670440
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
##
@@ -2085,4 +2087,28 @@ class CSVSuite extends QueryTest with SharedSQLContext
with SQLTestUtils with Te
}
}
}
+
+ test("SPARK-28431: prevent CSV datasource throw TextParsingException with
large size message") {
+withTempDir { dir =>
Review comment:
I think it doesn't remove parent dir. Can we do
```diff
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
index f3ab1b812e4..abbca28434c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
@@ -2089,13 +2089,12 @@ class CSVSuite extends QueryTest with
SharedSQLContext with SQLTestUtils with Te
}
test("SPARK-28431: prevent CSV datasource throw TextParsingException with
large size message") {
-withTempDir { dir =>
+withTempPath { path =>
val maxCharsPerCol = 1
val str = "a" * (maxCharsPerCol + 1)
- val csvFile = new File(dir, "data.csv")
Files.write(
-csvFile.toPath,
+path.toPath,
Seq(str).asJava,
StandardOpenOption.CREATE, StandardOpenOption.WRITE
)
@@ -2103,7 +2102,7 @@ class CSVSuite extends QueryTest with SharedSQLContext
with SQLTestUtils with Te
val errMsg = intercept[TextParsingException] {
spark.read
.option("maxCharsPerColumn", maxCharsPerCol)
- .csv(csvFile.getAbsolutePath)
+ .csv(path.getAbsolutePath)
.count()
}.getMessage
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
HyukjinKwon removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513617002 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25204: [SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated scalar subquery
cloud-fan commented on a change in pull request #25204:
[SPARK-28441][SQL][Python] Fix error when PythonUDF is used in correlated
scalar subquery
URL: https://github.com/apache/spark/pull/25204#discussion_r305670236
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##
@@ -318,17 +318,30 @@ object RewriteCorrelatedScalarSubquery extends
Rule[LogicalPlan] {
/**
* Statically evaluate an expression containing zero or more placeholders,
given a set
- * of bindings for placeholder values.
+ * of bindings for placeholder values, if the expression is evaluable. If it
is not,
+ * bind statically evaluated expression results to an expression.
*/
- private def evalExpr(expr: Expression, bindings: Map[ExprId, Option[Any]]) :
Option[Any] = {
+ private def bindingExpr(
+ expr: Expression,
+ bindings: Map[ExprId, Option[Expression]]): Option[Expression] = {
val rewrittenExpr = expr transform {
case r: AttributeReference =>
bindings(r.exprId) match {
- case Some(v) => Literal.create(v, r.dataType)
+ case Some(v) => v
case None => Literal.default(NullType)
}
}
-Option(rewrittenExpr.eval())
+if (rewrittenExpr.find(_.isInstanceOf[PythonUDF]).isDefined) {
+ // SPARK-28441: `PythonUDF` can't be statically evaluated.
Review comment:
We can't call `Expression.eval(null)` if it's not foldable, otherwise
exception may be thrown:
1. `AttributeReference.eval(null)` fails with NPE
2. `Nondeterministic.eval(null)` fails because it needs to be initialized
first
Whatever hack we use, I'd expect it makes the expression foldable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25209: [SPARK-28456][SQL] Add a public API `Encoder.makeCopy` to allow creating Encoder without touching Scala Reflection
cloud-fan commented on issue #25209: [SPARK-28456][SQL] Add a public API `Encoder.makeCopy` to allow creating Encoder without touching Scala Reflection URL: https://github.com/apache/spark/pull/25209#issuecomment-513636234 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
HyukjinKwon commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513636088 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25206: [SPARK-28265][SQL] Add renameTable to TableCatalog API
cloud-fan commented on issue #25206: [SPARK-28265][SQL] Add renameTable to TableCatalog API URL: https://github.com/apache/spark/pull/25206#issuecomment-513635512 Let's enforce that to be consistent with the existing behaviors. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513634915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13092/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins removed a comment on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513634909 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
SparkQA commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513635225 **[Test build #107983 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107983/testReport)** for PR 25224 at commit [`3aaec73`](https://github.com/apache/spark/commit/3aaec73da4c6a42141bee1d4c4a9a7fadee71c2c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513634909 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
AmplabJenkins commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513634915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13092/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
HyukjinKwon commented on issue #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224#issuecomment-513634394 cc @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513634405 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107975/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change
HyukjinKwon opened a new pull request #25224: [SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change URL: https://github.com/apache/spark/pull/25224 ## What changes were proposed in this pull request? This PR proposes to add a note in the migration guide. See https://github.com/apache/spark/pull/25108#issuecomment-513526585 ## How was this patch tested? N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513634405 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107975/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513634403 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
AmplabJenkins commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513634403 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
SparkQA removed a comment on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513618310 **[Test build #107975 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107975/testReport)** for PR 25212 at commit [`1ddc3b6`](https://github.com/apache/spark/commit/1ddc3b6b91d772e2d8680d1313bc810f9c89e098). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835
SparkQA commented on issue #25212: [SPARK-28460][SQL][TEST] Port test from HIVE-11835 URL: https://github.com/apache/spark/pull/25212#issuecomment-513634211 **[Test build #107975 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107975/testReport)** for PR 25212 at commit [`1ddc3b6`](https://github.com/apache/spark/commit/1ddc3b6b91d772e2d8680d1313bc810f9c89e098). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] edgarRd commented on issue #25206: [SPARK-28265][SQL] Add renameTable to TableCatalog API
edgarRd commented on issue #25206: [SPARK-28265][SQL] Add renameTable to TableCatalog API URL: https://github.com/apache/spark/pull/25206#issuecomment-513634138 @cloud-fan @rdblue Originally I had committed a version enforcing the same namespace for the table identifiers since the spec does not define the behavior in that case, so I tried to be consistent with the existing behavior in Spark, the enforcement is already there in https://github.com/apache/spark/blob/24e1e41648de58d3437e008b187b84828830e238/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala#L641 Also, I think there could be cases when the catalog could rename a table across databases and not requiring any move (of data). However, if there has already been agreement that the `renameTable` operation only works within the same DB then I can revert the change I had enforcing the same namespace. If we want to add the enforcement at the parser level, maybe we can do it in another PR. Please let me know if there are any more comments on this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
HyukjinKwon commented on a change in pull request #25184: [SPARK-28431][SQL]
Set maximum error message length in CSV datasource's parsing and writing
URL: https://github.com/apache/spark/pull/25184#discussion_r305668525
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
##
@@ -2085,4 +2087,28 @@ class CSVSuite extends QueryTest with SharedSQLContext
with SQLTestUtils with Te
}
}
}
+
+ test("SPARK-28431: prevent CSV datasource throw TextParsingException with
large size message") {
+withTempDir { dir =>
+ val maxCharsPerCol = 1
+ val str = "a" * (maxCharsPerCol + 1)
+
+ val csvFile = new File(dir, "data.csv")
+ Files.write(
+csvFile.toPath,
+Seq(str).asJava,
Review comment:
and .. CSV datasource are pretty sensitive about encodings - I and @MaxGekk
had some extreme discussions about that before ... It might be better to write
out UTF-8 just to be more explicit :-) .
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513633765 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13091/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513633762 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513633762 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513633765 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13091/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
HyukjinKwon commented on a change in pull request #25184: [SPARK-28431][SQL]
Set maximum error message length in CSV datasource's parsing and writing
URL: https://github.com/apache/spark/pull/25184#discussion_r305668314
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
##
@@ -2085,4 +2087,28 @@ class CSVSuite extends QueryTest with SharedSQLContext
with SQLTestUtils with Te
}
}
}
+
+ test("SPARK-28431: prevent CSV datasource throw TextParsingException with
large size message") {
+withTempDir { dir =>
+ val maxCharsPerCol = 1
+ val str = "a" * (maxCharsPerCol + 1)
+
+ val csvFile = new File(dir, "data.csv")
+ Files.write(
+csvFile.toPath,
+Seq(str).asJava,
Review comment:
I think @MaxGekk just meant a nit: write a file with explicitly calling
`getBytes()`.
I think it's better to call `getBytes(StandardCharset.UTF-8)` actually.
Otherwise, the codes look like it's going to write out unicodes as are.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2
cloud-fan closed pull request #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2 URL: https://github.com/apache/spark/pull/24798 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
SparkQA commented on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513632985 **[Test build #107982 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107982/testReport)** for PR 25184 at commit [`f8c3f7f`](https://github.com/apache/spark/commit/f8c3f7fa5f2bb64ec57fc2d33aae77fd0165d26b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2
cloud-fan commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2 URL: https://github.com/apache/spark/pull/24798#issuecomment-513632963 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
HyukjinKwon commented on a change in pull request #25184: [SPARK-28431][SQL]
Set maximum error message length in CSV datasource's parsing and writing
URL: https://github.com/apache/spark/pull/25184#discussion_r305667546
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
##
@@ -2085,4 +2087,28 @@ class CSVSuite extends QueryTest with SharedSQLContext
with SQLTestUtils with Te
}
}
}
+
+ test("SPARK-28431: prevent CSV datasource throw TextParsingException with
large size message") {
+withTempDir { dir =>
+ val maxCharsPerCol = 1
+ val str = "a" * (maxCharsPerCol + 1)
Review comment:
Ah, yes. I thought the limit was not inclusive..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513632651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13090/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513632645 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513632651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13090/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513632645 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on a change in pull request #25184: [SPARK-28431][SQL] Set maximum error message length in CSV datasource's parsing and writing
WeichenXu123 commented on a change in pull request #25184: [SPARK-28431][SQL]
Set maximum error message length in CSV datasource's parsing and writing
URL: https://github.com/apache/spark/pull/25184#discussion_r305666988
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
##
@@ -2085,4 +2087,28 @@ class CSVSuite extends QueryTest with SharedSQLContext
with SQLTestUtils with Te
}
}
}
+
+ test("SPARK-28431: prevent CSV datasource throw TextParsingException with
large size message") {
+withTempDir { dir =>
+ val maxCharsPerCol = 1
+ val str = "a" * (maxCharsPerCol + 1)
Review comment:
It do need "+1". Without "+1" won't throw exception. Because it do not
exceed "max" value configured.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] edgarRd commented on a change in pull request #25206: [SPARK-28265][SQL] Add renameTable to TableCatalog API
edgarRd commented on a change in pull request #25206: [SPARK-28265][SQL] Add
renameTable to TableCatalog API
URL: https://github.com/apache/spark/pull/25206#discussion_r305666975
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala
##
@@ -153,6 +152,17 @@ class V2SessionCatalog(sessionState: SessionState)
extends TableCatalog {
}
}
+ override def renameTable(oldIdent: Identifier, newIdent: Identifier): Unit =
{
+if (Option(loadTable(newIdent)).isEmpty) throw new
TableAlreadyExistsException(newIdent)
+
+Option(loadTable(oldIdent)) match {
Review comment:
I got thrown off by
https://github.com/apache/spark/pull/25206/files#diff-662d75c51e01d8e70f7c1384cea5b4c3L141
that checks for null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
