[GitHub] [spark] cloud-fan commented on a change in pull request #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource

2020-03-31 Thread GitBox
cloud-fan commented on a change in pull request #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource URL: https://github.com/apache/spark/pull/28076#discussion_r401393426 ## File path: sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt ###

[GitHub] [spark] AmplabJenkins removed a comment on issue #28088: [SPARK-31320] Fix release script for 3.0.0

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28088: [SPARK-31320] Fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607070057 Merged build finished. Test PASSed. This is an automated messa

[GitHub] [spark] AmplabJenkins removed a comment on issue #28088: [SPARK-31320] Fix release script for 3.0.0

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28088: [SPARK-31320] Fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607070069 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jen

[GitHub] [spark] AmplabJenkins commented on issue #28088: [SPARK-31320] Fix release script for 3.0.0

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28088: [SPARK-31320] Fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607070069 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//jo

[GitHub] [spark] cloud-fan commented on a change in pull request #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource

2020-03-31 Thread GitBox
cloud-fan commented on a change in pull request #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource URL: https://github.com/apache/spark/pull/28076#discussion_r401393426 ## File path: sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt ###

[GitHub] [spark] AmplabJenkins commented on issue #28088: [SPARK-31320] Fix release script for 3.0.0

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28088: [SPARK-31320] Fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607070057 Merged build finished. Test PASSed. This is an automated message from

[GitHub] [spark] SparkQA commented on issue #28088: [SPARK-31320] Fix release script for 3.0.0

2020-03-31 Thread GitBox
SparkQA commented on issue #28088: [SPARK-31320] Fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607068987 **[Test build #120663 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120663/testReport)** for PR 28088 at c

[GitHub] [spark] SparkQA removed a comment on issue #28088: [SPARK-31320] Fix release script for 3.0.0

2020-03-31 Thread GitBox
SparkQA removed a comment on issue #28088: [SPARK-31320] Fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607017081 **[Test build #120663 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120663/testReport)** for PR 28

[GitHub] [spark] Ngone51 commented on issue #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling

2020-03-31 Thread GitBox
Ngone51 commented on issue #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling URL: https://github.com/apache/spark/pull/28053#issuecomment-607061160 LGTM, except one minor comment. ---

[GitHub] [spark] Ngone51 commented on a change in pull request #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling

2020-03-31 Thread GitBox
Ngone51 commented on a change in pull request #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling URL: https://github.com/apache/spark/pull/28053#discussion_r401379471 ## File path: core/src/main/scala/org/apache/spark/schedul

[GitHub] [spark] Ngone51 commented on a change in pull request #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling

2020-03-31 Thread GitBox
Ngone51 commented on a change in pull request #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling URL: https://github.com/apache/spark/pull/28053#discussion_r401376968 ## File path: core/src/main/scala/org/apache/spark/interna

[GitHub] [spark] AmplabJenkins removed a comment on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML URL: https://github.com/apache/spark/pull/28078#issuecomment-607053876 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.

[GitHub] [spark] AmplabJenkins removed a comment on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML URL: https://github.com/apache/spark/pull/28078#issuecomment-607053871 Merged build finished. Test PASSed. This is

[GitHub] [spark] AmplabJenkins commented on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML URL: https://github.com/apache/spark/pull/28078#issuecomment-607053871 Merged build finished. Test PASSed. This is an autom

[GitHub] [spark] AmplabJenkins commented on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML URL: https://github.com/apache/spark/pull/28078#issuecomment-607053876 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berke

[GitHub] [spark] SparkQA commented on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML

2020-03-31 Thread GitBox
SparkQA commented on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML URL: https://github.com/apache/spark/pull/28078#issuecomment-607053466 **[Test build #120666 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120666/testReport)**

[GitHub] [spark] zhengruifeng opened a new pull request #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML

2020-03-31 Thread GitBox
zhengruifeng opened a new pull request #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML URL: https://github.com/apache/spark/pull/28078 ### What changes were proposed in this pull request? 1, Move the impl of ChiSq from .mllib to the .ml side; 2, in `.mllib.ChiSqTe

[GitHub] [spark] zhengruifeng commented on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML

2020-03-31 Thread GitBox
zhengruifeng commented on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML URL: https://github.com/apache/spark/pull/28078#issuecomment-607051386 OK, I did not get the point. I think it worthwhile to move some impls to the .ml side, since in .ml we can use

[GitHub] [spark] huaxingao commented on a change in pull request #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
huaxingao commented on a change in pull request #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#discussion_r401372386 ## File path: docs/sql-ref-functions-udf-scalar.md ## @@ -1,22 +1,125 @@ --- layout

[GitHub] [spark] AmplabJenkins removed a comment on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource URL: https://github.com/apache/spark/pull/28076#issuecomment-607049060 Merged build finished. Test PASSed. -

[GitHub] [spark] Ngone51 commented on a change in pull request #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling

2020-03-31 Thread GitBox
Ngone51 commented on a change in pull request #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling URL: https://github.com/apache/spark/pull/28053#discussion_r401370604 ## File path: core/src/main/scala/org/apache/spark/schedul

[GitHub] [spark] AmplabJenkins removed a comment on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource URL: https://github.com/apache/spark/pull/28076#issuecomment-607049069 Test PASSed. Refer to this link for build results (access rights to CI server needed): htt

[GitHub] [spark] AmplabJenkins commented on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource URL: https://github.com/apache/spark/pull/28076#issuecomment-607049069 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amp

[GitHub] [spark] AmplabJenkins commented on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource URL: https://github.com/apache/spark/pull/28076#issuecomment-607049060 Merged build finished. Test PASSed. This

[GitHub] [spark] AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607048551 Merged build finished. Test PASSed. This is

[GitHub] [spark] AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607048556 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.

[GitHub] [spark] SparkQA commented on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource

2020-03-31 Thread GitBox
SparkQA commented on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource URL: https://github.com/apache/spark/pull/28076#issuecomment-607048593 **[Test build #120665 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120665/te

[GitHub] [spark] AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607048556 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berke

[GitHub] [spark] SparkQA removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
SparkQA removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607043432 **[Test build #120664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120664/testR

[GitHub] [spark] SparkQA commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
SparkQA commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607048390 **[Test build #120664 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120664/testReport)*

[GitHub] [spark] AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607048551 Merged build finished. Test PASSed. This is an auto

[GitHub] [spark] AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607047767 Merged build finished. Test PASSed. -

[GitHub] [spark] Ngone51 commented on issue #28049: [SPARK-31285][CORE] uppercase schedule mode string at config

2020-03-31 Thread GitBox
Ngone51 commented on issue #28049: [SPARK-31285][CORE] uppercase schedule mode string at config URL: https://github.com/apache/spark/pull/28049#issuecomment-607048295 LGTM, cc @dongjoon-hyun This is an automated message from

[GitHub] [spark] AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607047773 Test PASSed. Refer to this link for build results (access rights to CI serv

[GitHub] [spark] AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607047767 Merged build finished. Test PASSed. -

[GitHub] [spark] AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607047773 Test PASSed. Refer to this link for build results (access rights to CI server neede

[GitHub] [spark] maropu commented on a change in pull request #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
maropu commented on a change in pull request #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#discussion_r401369262 ## File path: docs/sql-ref-functions-udf-scalar.md ## @@ -1,22 +1,125 @@ --- layout: g

[GitHub] [spark] SparkQA commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window

2020-03-31 Thread GitBox
SparkQA commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607046965 **[Test build #120661 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullReq

[GitHub] [spark] SparkQA removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window

2020-03-31 Thread GitBox
SparkQA removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-606998340 **[Test build #120661 has started](https://amplab.cs.berkeley.edu/jenkins/job/Spark

[GitHub] [spark] wang-zhun commented on issue #28069: [SPARK-31265][YARN] Add -XX:MaxDirectMemorySize jvm options in yarn mode

2020-03-31 Thread GitBox
wang-zhun commented on issue #28069: [SPARK-31265][YARN] Add -XX:MaxDirectMemorySize jvm options in yarn mode URL: https://github.com/apache/spark/pull/28069#issuecomment-607045826 @dongjoon-hyun thanks for the reply. I think the `amMemoryOverhead` represents the usable off-heap memory.

[GitHub] [spark] AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607043968 Merged build finished. Test PASSed. This is an auto

[GitHub] [spark] AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607043976 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berk

[GitHub] [spark] cloud-fan commented on a change in pull request #28088: [SPARK-31320] Fix release script for 3.0.0

2020-03-31 Thread GitBox
cloud-fan commented on a change in pull request #28088: [SPARK-31320] Fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#discussion_r401366461 ## File path: dev/create-release/spark-rm/Dockerfile ## @@ -76,10 +75,8 @@ RUN apt-get clean && apt-get

[GitHub] [spark] AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607043968 Merged build finished. Test PASSed. This is

[GitHub] [spark] AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607043976 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab

[GitHub] [spark] cloud-fan commented on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource

2020-03-31 Thread GitBox
cloud-fan commented on issue #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource URL: https://github.com/apache/spark/pull/28076#issuecomment-607043797 can you fix the conflicts? This is an automa

[GitHub] [spark] SparkQA commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
SparkQA commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607043432 **[Test build #120664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120664/testReport)**

[GitHub] [spark] huaxingao commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
huaxingao commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-607041687 > Do we have a page for introducing the APIs for Scala, Java, Python UDFs/UDAFs/UDTF? If not, could we introduce it?

[GitHub] [spark] dongjoon-hyun commented on issue #28088: [SPARK-31320] Fix release script for 3.0.0

2020-03-31 Thread GitBox
dongjoon-hyun commented on issue #28088: [SPARK-31320] Fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607041154 cc @rxin and @gatorsmile This is an automated message from the Apache

[GitHub] [spark] huaxingao commented on a change in pull request #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
huaxingao commented on a change in pull request #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#discussion_r401363958 ## File path: docs/sql-ref-functions-udf.md ## @@ -1,25 +1,25 @@ --- layout: global

[GitHub] [spark] huaxingao commented on a change in pull request #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
huaxingao commented on a change in pull request #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#discussion_r401363852 ## File path: docs/sql-ref-functions-udf-scalar.md ## @@ -1,22 +1,125 @@ --- layout

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28088: [SPARK-31320] Fix release script for 3.0.0

2020-03-31 Thread GitBox
dongjoon-hyun commented on a change in pull request #28088: [SPARK-31320] Fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#discussion_r401363806 ## File path: dev/create-release/spark-rm/Dockerfile ## @@ -76,10 +75,8 @@ RUN apt-get clean && apt-

[GitHub] [spark] huaxingao commented on a change in pull request #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference

2020-03-31 Thread GitBox
huaxingao commented on a change in pull request #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#discussion_r401363788 ## File path: docs/sql-ref-functions-udf-aggregate.md ## @@ -1,22 +1,65 @@ --- layo

[GitHub] [spark] srowen commented on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML

2020-03-31 Thread GitBox
srowen commented on issue #28078: [SPARK-31309][ML] Migrate the ChiSquareTest from MLlib to ML URL: https://github.com/apache/spark/pull/28078#issuecomment-607039548 I don't think that means we can't refactor and improve the relationship between .ml and .mllib; I'm just saying I don't thin

[GitHub] [spark] AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-607037366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job

[GitHub] [spark] AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-607037361 Merged build finished. Test PASSed. This is an automated message from t

[GitHub] [spark] AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-607037366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenk

[GitHub] [spark] AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-607037361 Merged build finished. Test PASSed. This is an automated messag

[GitHub] [spark] SparkQA removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command

2020-03-31 Thread GitBox
SparkQA removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-606960582 **[Test build #120657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120657/testReport)** for PR 278

[GitHub] [spark] SparkQA commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command

2020-03-31 Thread GitBox
SparkQA commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-607036772 **[Test build #120657 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120657/testReport)** for PR 27897 at co

[GitHub] [spark] dongjoon-hyun commented on issue #28069: [SPARK-31265][YARN] Add -XX:MaxDirectMemorySize jvm options in yarn mode

2020-03-31 Thread GitBox
dongjoon-hyun commented on issue #28069: [SPARK-31265][YARN] Add -XX:MaxDirectMemorySize jvm options in yarn mode URL: https://github.com/apache/spark/pull/28069#issuecomment-607036183 Netty library itself also use some off-heap memory. -

[GitHub] [spark] dongjoon-hyun commented on issue #28080: [SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x

2020-03-31 Thread GitBox
dongjoon-hyun commented on issue #28080: [SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x URL: https://github.com/apache/spark/pull/28080#issuecomment-607034445 Thank you, @dbtsai ! This is an automated

[GitHub] [spark] HeartSaVioR commented on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP)

2020-03-31 Thread GitBox
HeartSaVioR commented on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP) URL: https://github.com/apache/spark/pull/28026#issuecomment-607032955 Maybe we also need to change two "create table" pages and migration guide page as well regardless of the approach, since we are

[GitHub] [spark] cloud-fan closed pull request #28082: [SPARK-31318][SQL] Split Parquet/Avro configs for rebasing dates/timestamps in read and in write

2020-03-31 Thread GitBox
cloud-fan closed pull request #28082: [SPARK-31318][SQL] Split Parquet/Avro configs for rebasing dates/timestamps in read and in write URL: https://github.com/apache/spark/pull/28082 This is an automated message from the Apa

[GitHub] [spark] cloud-fan commented on issue #28082: [SPARK-31318][SQL] Split Parquet/Avro configs for rebasing dates/timestamps in read and in write

2020-03-31 Thread GitBox
cloud-fan commented on issue #28082: [SPARK-31318][SQL] Split Parquet/Avro configs for rebasing dates/timestamps in read and in write URL: https://github.com/apache/spark/pull/28082#issuecomment-607032296 thanks, merging to master/3.0! --

[GitHub] [spark] yaooqinn commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only

2020-03-31 Thread GitBox
yaooqinn commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only URL: https://github.com/apache/spark/pull/28003#issuecomment-607031617 > @yaooqinn Could you backport this to 2.4? and add it to the migration guide since it could impact the query resul

[GitHub] [spark] AmplabJenkins removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-607029132 Test FAILed. Refer to this link for build results (access rights to CI server needed): htt

[GitHub] [spark] AmplabJenkins removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-607029129 Merged build finished. Test FAILed.

[GitHub] [spark] AmplabJenkins commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-607029132 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amp

[GitHub] [spark] AmplabJenkins commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-607029129 Merged build finished. Test FAILed. Thi

[GitHub] [spark] cloud-fan commented on issue #28041: [SPARK-30564][SQL] Improved extra new line and comment remove

2020-03-31 Thread GitBox
cloud-fan commented on issue #28041: [SPARK-30564][SQL] Improved extra new line and comment remove URL: https://github.com/apache/spark/pull/28041#issuecomment-607029102 Agree with @maropu . We can slowly update all comments with `CodegenContext.registerComment` --

[GitHub] [spark] SparkQA removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests

2020-03-31 Thread GitBox
SparkQA removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-606901165 **[Test build #120653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/

[GitHub] [spark] SparkQA commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests

2020-03-31 Thread GitBox
SparkQA commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-607028697 **[Test build #120653 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120653/

[GitHub] [spark] dbtsai commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet

2020-03-31 Thread GitBox
dbtsai commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-607021780 For the worst case, we don't see extra overhead from parquet side. I

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies

2020-03-31 Thread GitBox
HyukjinKwon commented on a change in pull request #27928: [SPARK-31167][BUILD] Refactor how we track Python test/build dependencies URL: https://github.com/apache/spark/pull/27928#discussion_r401347559 ## File path: docs/README.md ## @@ -88,7 +88,7 @@ Note: Other versions

[GitHub] [spark] HyukjinKwon edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

2020-03-31 Thread GitBox
HyukjinKwon edited a comment on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs URL: https://github.com/apache/spark/pull/27534#issuecomment-607021040 We can pin the version for release related ones; however, I doubt if we should do that for others e.g., CI, documentati

[GitHub] [spark] HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

2020-03-31 Thread GitBox
HyukjinKwon commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs URL: https://github.com/apache/spark/pull/27534#issuecomment-607021040 We can pin the version for release related ones; however, I doubt if we should do that for others e.g., CI, documentation. It'

[GitHub] [spark] HyukjinKwon edited a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet

2020-03-31 Thread GitBox
HyukjinKwon edited a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-607020100 It's true that it can depends on the nature of data or w

[GitHub] [spark] HyukjinKwon commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet

2020-03-31 Thread GitBox
HyukjinKwon commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-607020100 It's true that it can depends on the nature of data or workload

[GitHub] [spark] gatorsmile commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet

2020-03-31 Thread GitBox
gatorsmile commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-607018589 Do you know the perf number for the worst case (e.g., no row can

[GitHub] [spark] AmplabJenkins removed a comment on issue #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607017528 Merged build finished. Test PASSed. This is an automated messa

[GitHub] [spark] AmplabJenkins removed a comment on issue #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
AmplabJenkins removed a comment on issue #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607017538 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/je

[GitHub] [spark] AmplabJenkins commented on issue #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607017528 Merged build finished. Test PASSed. This is an automated message from

[GitHub] [spark] AmplabJenkins commented on issue #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
AmplabJenkins commented on issue #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607017538 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//j

[GitHub] [spark] SparkQA commented on issue #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
SparkQA commented on issue #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607017081 **[Test build #120663 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120663/testReport)** for PR 28088 at co

[GitHub] [spark] cloud-fan commented on issue #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
cloud-fan commented on issue #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607016636 Note that we should forward port this to the master branch. It was done at `branch-3.0` as the policy is to use the script of the correspond

[GitHub] [spark] cloud-fan commented on issue #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
cloud-fan commented on issue #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#issuecomment-607016103 cc @dongjoon-hyun @HyukjinKwon @nchammas This is an automated message from

[GitHub] [spark] cloud-fan commented on a change in pull request #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
cloud-fan commented on a change in pull request #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#discussion_r401342910 ## File path: dev/create-release/spark-rm/Dockerfile ## @@ -33,8 +33,8 @@ ENV DEBCONF_NONINTERACTIVE_SEE

[GitHub] [spark] cloud-fan commented on a change in pull request #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
cloud-fan commented on a change in pull request #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#discussion_r401342694 ## File path: dev/create-release/release-util.sh ## @@ -159,10 +159,14 @@ function get_release_info {

[GitHub] [spark] cloud-fan commented on a change in pull request #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
cloud-fan commented on a change in pull request #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088#discussion_r401342577 ## File path: dev/create-release/do-release-docker.sh ## @@ -93,7 +93,7 @@ done GPG_KEY_FILE="$WORKDI

[GitHub] [spark] cloud-fan opened a new pull request #28088: [SPARK-31320] fix release script for 3.0.0

2020-03-31 Thread GitBox
cloud-fan opened a new pull request #28088: [SPARK-31320] fix release script for 3.0.0 URL: https://github.com/apache/spark/pull/28088 ### What changes were proposed in this pull request? The release script stops working after https://github.com/apache/spark/commit/d5865493a

[GitHub] [spark] HeartSaVioR commented on issue #28086: [SPARK-31312][SQL][2.4] Cache Class instance for the UDF instance in HiveFunctionWrapper

2020-03-31 Thread GitBox
HeartSaVioR commented on issue #28086: [SPARK-31312][SQL][2.4] Cache Class instance for the UDF instance in HiveFunctionWrapper URL: https://github.com/apache/spark/pull/28086#issuecomment-607013918 Thanks for the quick review and merge!

[GitHub] [spark] dbtsai commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet

2020-03-31 Thread GitBox
dbtsai commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-60700 It really depends on the data and if we can skip most of the row gro

[GitHub] [spark] dbtsai edited a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet

2020-03-31 Thread GitBox
dbtsai edited a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-60700 It really depends on the data and if we can skip most of the

[GitHub] [spark] cloud-fan commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs

2020-03-31 Thread GitBox
cloud-fan commented on issue #27534: [SPARK-30879][DOCS] Refine workflow for building docs URL: https://github.com/apache/spark/pull/27534#issuecomment-607011028 This PR does have a good point to fix the dependency versions so that the script is more robust. I'm happy to see a working vers

[GitHub] [spark] dbtsai commented on issue #28080: [SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x

2020-03-31 Thread GitBox
dbtsai commented on issue #28080: [SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x URL: https://github.com/apache/spark/pull/28080#issuecomment-607009930 LGTM. Merged into master. This is an automated me

[GitHub] [spark] dbtsai closed pull request #28080: [SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x

2020-03-31 Thread GitBox
dbtsai closed pull request #28080: [SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x URL: https://github.com/apache/spark/pull/28080 This is an automated message from the Apache Git Service. To respond t

[GitHub] [spark] cloud-fan closed pull request #28086: [SPARK-31312][SQL][2.4] Cache Class instance for the UDF instance in HiveFunctionWrapper

2020-03-31 Thread GitBox
cloud-fan closed pull request #28086: [SPARK-31312][SQL][2.4] Cache Class instance for the UDF instance in HiveFunctionWrapper URL: https://github.com/apache/spark/pull/28086 This is an automated message from the Apache Git

[GitHub] [spark] cloud-fan commented on issue #28086: [SPARK-31312][SQL][2.4] Cache Class instance for the UDF instance in HiveFunctionWrapper

2020-03-31 Thread GitBox
cloud-fan commented on issue #28086: [SPARK-31312][SQL][2.4] Cache Class instance for the UDF instance in HiveFunctionWrapper URL: https://github.com/apache/spark/pull/28086#issuecomment-607009137 thanks, merging to 2.4! This

[GitHub] [spark] cloud-fan commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command

2020-03-31 Thread GitBox
cloud-fan commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#discussion_r401337418 ## File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala ## @@ -9

  1   2   3   4   5   6   7   >