[GitHub] [spark] dongjoon-hyun commented on issue #26173: [SPARK-29503][SQL] Remove conversion CreateNamedStruct to CreateNamedStructUnsafe
dongjoon-hyun commented on issue #26173: [SPARK-29503][SQL] Remove conversion CreateNamedStruct to CreateNamedStructUnsafe URL: https://github.com/apache/spark/pull/26173#issuecomment-578391512 Hi, All. This seems to exist since 2.1.1, do you think we can have a backport of this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25268: [SPARK-28375][SQL] Make pullupCorrelatedPredicate idempotent
dongjoon-hyun commented on issue #25268: [SPARK-28375][SQL] Make pullupCorrelatedPredicate idempotent URL: https://github.com/apache/spark/pull/25268#issuecomment-578391698 Hi, All. When I saw this PR at the first, it was an improvement PR. However, the JIRA is marked as `correctness`. Do you think we need to backport this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578391845 **[Test build #117395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117395/testReport)** for PR 27359 at commit [`c4cd053`](https://github.com/apache/spark/commit/c4cd05343523556676dfea1747a942b7bd733c98). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
SparkQA commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578391844 **[Test build #117396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117396/testReport)** for PR 27278 at commit [`83db814`](https://github.com/apache/spark/commit/83db814b384cc097d4f14e400a81caa49f4edede). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24519: [SPARK-27612][PYTHON] Use Python's default protocol instead of highest protocol
dongjoon-hyun commented on issue #24519: [SPARK-27612][PYTHON] Use Python's default protocol instead of highest protocol URL: https://github.com/apache/spark/pull/24519#issuecomment-578392025 Hi, @HyukjinKwon . This issue is marked as a correctness issue. Do we need to backport this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578392074 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22155/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578392070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22156/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578392070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22156/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578392074 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22155/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578392067 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578392072 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578392072 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578392067 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #24519: [SPARK-27612][PYTHON] Use Python's default protocol instead of highest protocol
dongjoon-hyun edited a comment on issue #24519: [SPARK-27612][PYTHON] Use Python's default protocol instead of highest protocol URL: https://github.com/apache/spark/pull/24519#issuecomment-578392025 Hi, @HyukjinKwon . ~This issue is marked as a correctness issue. Do we need to backport this?~ Never mind. I confirm that this doesn't happen in 2.4.4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578393232 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117395/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578391845 **[Test build #117395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117395/testReport)** for PR 27359 at commit [`c4cd053`](https://github.com/apache/spark/commit/c4cd05343523556676dfea1747a942b7bd733c98). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578393214 **[Test build #117395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117395/testReport)** for PR 27359 at commit [`c4cd053`](https://github.com/apache/spark/commit/c4cd05343523556676dfea1747a942b7bd733c98). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578393228 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578393228 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578393232 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117395/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #23811: [SPARK-26902][SQL] Support java.time.Instant as an external type of TimestampType
MaxGekk commented on issue #23811: [SPARK-26902][SQL] Support java.time.Instant as an external type of TimestampType URL: https://github.com/apache/spark/pull/23811#issuecomment-578394207 > How useful is this change? Please, take a look at the motivation points in the PR description. > Wouldn't it break a lot of user code that use Timestmap type when they upgrade to 3.0? No, it will not break because Spark still returns `java.sql.Timestamp` by default. > It seems like we wouldn't be able to ever remove the config flag. The flag has been removed already, and replaced by `spark.sql.datetime.java8API.enabled` (see https://github.com/apache/spark/blob/ed44926117d81aa5fa8bd823d401efd235260872/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2033-L2039) which is disabled by default. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun removed a comment on issue #27357: [SPARK-30639][BUILD] Upgrade Jersey to 2.30
dongjoon-hyun removed a comment on issue #27357: [SPARK-30639][BUILD] Upgrade Jersey to 2.30 URL: https://github.com/apache/spark/pull/27357#issuecomment-578330469 It seems that Microsoft repository is broken again. Please ignore `linter-r` installation step failure. ``` E: Failed to fetch https://packages.microsoft.com/repos/microsoft-ubuntu-bionic-prod/dists/bionic/main/binary-amd64/Packages.bz2 File has unexpected size (89974 != 89668). Mirror sync in progress? [IP: 40.76.35.62 443] 71 Hashes of expected file: 72 - Filesize:89668 [weak] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
SparkQA commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578400491 **[Test build #117396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117396/testReport)** for PR 27278 at commit [`83db814`](https://github.com/apache/spark/commit/83db814b384cc097d4f14e400a81caa49f4edede). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
SparkQA removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578391844 **[Test build #117396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117396/testReport)** for PR 27278 at commit [`83db814`](https://github.com/apache/spark/commit/83db814b384cc097d4f14e400a81caa49f4edede). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578400814 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117396/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578400814 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117396/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578400808 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-578400808 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578405187 **[Test build #117397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117397/testReport)** for PR 27359 at commit [`cb5bf59`](https://github.com/apache/spark/commit/cb5bf5930ae22822357ea176497dac58d4bd1a9e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578405392 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22157/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578405388 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578405388 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578405392 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22157/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #27337: [SPARK-30545][ML][PYSPARK] Impl Extremely Randomized Trees
zhengruifeng commented on issue #27337: [SPARK-30545][ML][PYSPARK] Impl Extremely Randomized Trees URL: https://github.com/apache/spark/pull/27337#issuecomment-578405903 @huaxingao Good point, you are right. Scikit-Learn does this exactly: ``` # Draw a random threshold current.threshold = rand_uniform(min_feature_value, max_feature_value, random_state) ``` In Scikit-Learn, both RF and ET use a greedy method to find an exact threshold. However, in MLLIB, all tree models are built on binned datasets (`treePoints`), to keep in line with other tree models and to minimize the change, I perfer to randomly draw a split from splits built at the beginning of training. Otherwise, I need to completely impl a new exact threshold finding method. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578406736 **[Test build #117397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117397/testReport)** for PR 27359 at commit [`cb5bf59`](https://github.com/apache/spark/commit/cb5bf5930ae22822357ea176497dac58d4bd1a9e). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578406743 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578406743 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578406747 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117397/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578405187 **[Test build #117397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117397/testReport)** for PR 27359 at commit [`cb5bf59`](https://github.com/apache/spark/commit/cb5bf5930ae22822357ea176497dac58d4bd1a9e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578406747 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117397/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng opened a new pull request #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
zhengruifeng opened a new pull request #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360 ### What changes were proposed in this pull request? 1, stack input vectors to blocks (like ALS/MLP); 2, add new param `blockSize` 3, standardize the input outside of optimization procedure; ### Why are the changes needed? 1, reduce RAM to persist traing dataset; (save ~40% in test) 2, use Level-2 BLAS routines; (12% ~ 28% faster, without native BLAS) ### Does this PR introduce any user-facing change? a new param `blockSize` ### How was this patch tested? existing and updated testsuites This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
zhengruifeng commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC
blockify input vectors
URL: https://github.com/apache/spark/pull/27360#issuecomment-578407327
testCode:
```scala
import org.apache.spark.ml.classification._
import org.apache.spark.storage.StorageLevel
var df =
spark.read.format("libsvm").load("/data1/Datasets/a9a/a9a").withColumn("label",
(col("label")+1)/2)
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count
(0 until 8).foreach{ _ => df = df.union(df) }
df.count
new LinearSVC().setMaxIter(10).fit(df) // warm up
val svc = new LinearSVC().setMaxIter(100)
val start = System.currentTimeMillis; val model = svc.fit(df); val end =
System.currentTimeMillis; end - start
val svc = new LinearSVC().setMaxIter(100).setFitIntercept(false)
val start = System.currentTimeMillis; val model = svc.fit(df); val end =
System.currentTimeMillis; end - start
```
result:
this PR:
RAM:1418.9M
Duration: 396524(fitIntercept=true), 324944(fitIntercept=false)
MASTER:
RAM: 2.3G
Duration: 446354(fitIntercept=true), 441961(fitIntercept=false)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
SparkQA commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578407641 **[Test build #117398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117398/testReport)** for PR 27360 at commit [`6dcb9eb`](https://github.com/apache/spark/commit/6dcb9eb77dafc258a9d3f61494e0a56f1b331966). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578407653 **[Test build #117399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117399/testReport)** for PR 27359 at commit [`05c3f8d`](https://github.com/apache/spark/commit/05c3f8d56aff8fb01c7175c2abaab16cfa96a75b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
zhengruifeng commented on a change in pull request #27360:
[SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
URL: https://github.com/apache/spark/pull/27360#discussion_r370934439
##
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
##
@@ -220,8 +227,22 @@ class LinearSVC @Since("2.2.0") (
None
}
- val getAggregatorFunc = new HingeAggregator(bcFeaturesStd,
$(fitIntercept))(_)
- val costFun = new RDDLossFunction(instances, getAggregatorFunc,
regularization,
+ val standardized = instances.map {
Review comment:
Move the standardization outside of `HingeAggregator`, so that no longer
need to standardize input in each iter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578407833 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578407838 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578407835 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22158/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578407833 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578407840 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22159/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578407840 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22159/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578407838 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578407835 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22158/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng edited a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
zhengruifeng edited a comment on issue #27360: [SPARK-30642][ML][PYSPARK]
LinearSVC blockify input vectors
URL: https://github.com/apache/spark/pull/27360#issuecomment-578407327
testCode:
```scala
import org.apache.spark.ml.classification._
import org.apache.spark.storage.StorageLevel
var df =
spark.read.format("libsvm").load("/data1/Datasets/a9a/a9a").withColumn("label",
(col("label")+1)/2)
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count
(0 until 8).foreach{ _ => df = df.union(df) }
df.count
new LinearSVC().setMaxIter(10).fit(df) // warm up
val svc = new LinearSVC().setMaxIter(100)
val start = System.currentTimeMillis; val model = svc.fit(df); val end =
System.currentTimeMillis; end - start
val svc = new LinearSVC().setMaxIter(100).setFitIntercept(false)
val start = System.currentTimeMillis; val model = svc.fit(df); val end =
System.currentTimeMillis; end - start
```
result:
this PR:
RAM:1418.9M
Duration: 396524(fitIntercept=true), 324944(fitIntercept=false)
MASTER:
RAM: 2.3G
Duration: 446354(fitIntercept=true), 441961(fitIntercept=false)
Native-BLAS is NOT used in above tests, maybe future performance gain can be
obtained by setting appropriate Native BLAS.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
SparkQA commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578408796 **[Test build #117400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117400/testReport)** for PR 27360 at commit [`c51c37b`](https://github.com/apache/spark/commit/c51c37b6db1479b2a41982e86fb24ac2ea945d82). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578410205 **[Test build #117399 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117399/testReport)** for PR 27359 at commit [`05c3f8d`](https://github.com/apache/spark/commit/05c3f8d56aff8fb01c7175c2abaab16cfa96a75b). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578407653 **[Test build #117399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117399/testReport)** for PR 27359 at commit [`05c3f8d`](https://github.com/apache/spark/commit/05c3f8d56aff8fb01c7175c2abaab16cfa96a75b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578410284 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117399/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578410281 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578410281 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578410284 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117399/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #27337: [SPARK-30545][ML][PYSPARK] Impl Extremely Randomized Trees
srowen commented on issue #27337: [SPARK-30545][ML][PYSPARK] Impl Extremely Randomized Trees URL: https://github.com/apache/spark/pull/27337#issuecomment-578410967 This is a lot of new code for a small variation on random forests. I'm not sure it's worth it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r370936472
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5",
"fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+val path1 = Files.createTempDirectory("db_")
+val path2 = Files.createTempDirectory(path1, "tbl_")
+val uri1 = preparePath(path1)
+val uri2 = preparePath(path2)
+
+// CREATE
+val dbDefinition = createDbDefinition(uri1)
+
+val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+val partition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" ->
"2"),
+ storage = CatalogStorageFormat.empty)
+
+catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") ::
Nil)
+
+catalog.createTable(tableDefinition, ignoreIfExists = false)
+checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5",
"tbl1") :: Nil)
+
+catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists =
false)
+checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists =
true)
+checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
Review comment:
I agree with you, it looks incorrect, but we can not check if the partition
has been created or not created in case of using `ignoreIfExists=true`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #23389: [2.4] revert [SPARK-26021][SQL] replace minus zero with zero in Platform.putDouble/Float
srowen commented on issue #23389: [2.4] revert [SPARK-26021][SQL] replace minus zero with zero in Platform.putDouble/Float URL: https://github.com/apache/spark/pull/23389#issuecomment-578411086 On the general logic: if this is a correctness issue, then to fix it, you have to change incorrect behavior. Why would it be better to leave the correctness issue in 2.4? this is, further, a second behavior change then. Is there more context? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ayudovin removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
ayudovin removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods URL: https://github.com/apache/spark/pull/27034#issuecomment-577879919 > Hi, @ayudovin . > > 1. In general, new feature should not create a breaking change. For example, this PR enforce service account configuration additionally. If executor service account is not given, we had better fallback to use driver service account. > 2. Do you think we can have a test case for this? Test case is important to protect your contribution from the future changes. @dongjoon-hyun , Unfortunately, I can not find a way to add a unit test for it. Maybe, you can give advice or tell where I can find an example of tests for these properties. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578412857 **[Test build #117401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117401/testReport)** for PR 27030 at commit [`30686b4`](https://github.com/apache/spark/commit/30686b4a19edf7ceb13cd1d86838d4a1dd5c861b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578413180 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578413185 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22160/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578413185 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22160/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578413180 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Victsm commented on issue #22173: [SPARK-24355] Spark external shuffle server improvement to better handle block fetch requests.
Victsm commented on issue #22173: [SPARK-24355] Spark external shuffle server improvement to better handle block fetch requests. URL: https://github.com/apache/spark/pull/22173#issuecomment-578414568 @cloud-fan What do you think of SPARK-30602 in the context of this perf regression you see? We have also been operating our Spark infrastructure with this change for quite some time, and we do not in general notice performance regressions. When doing shuffle in a large-scale multi-tenancy cluster, the issues we mentioned in SPARK-30602's SPIP doc becomes much more dominant. Without the change in SPARK-24355, before saturating the underlying network, the disk is first saturated due to the small random reads, which will then further propagate its impact to start timing out control plane RPCs. SPARK-24355 is basically an attempt to stop the small random reads impacting control plane RPCs to improve reliability of shuffle service. On top of these, SPARK-30602 will significantly improve the overall throughput and efficiency of Spark shuffle. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
SparkQA commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578418286 **[Test build #117398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117398/testReport)** for PR 27360 at commit [`6dcb9eb`](https://github.com/apache/spark/commit/6dcb9eb77dafc258a9d3f61494e0a56f1b331966). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait HasBlockSize extends Params ` * `class HasBlockSize(Params):` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
SparkQA removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578407641 **[Test build #117398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117398/testReport)** for PR 27360 at commit [`6dcb9eb`](https://github.com/apache/spark/commit/6dcb9eb77dafc258a9d3f61494e0a56f1b331966). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578418593 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117398/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578418593 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117398/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578418591 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578418591 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
SparkQA commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578420676 **[Test build #117400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117400/testReport)** for PR 27360 at commit [`c51c37b`](https://github.com/apache/spark/commit/c51c37b6db1479b2a41982e86fb24ac2ea945d82). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
SparkQA removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578408796 **[Test build #117400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117400/testReport)** for PR 27360 at commit [`c51c37b`](https://github.com/apache/spark/commit/c51c37b6db1479b2a41982e86fb24ac2ea945d82). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578420970 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578420971 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117400/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins removed a comment on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578420971 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117400/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
AmplabJenkins commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578420970 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #27354: [SPARK-30633][SQL] Append L to seed when type is LongType
srowen commented on a change in pull request #27354: [SPARK-30633][SQL] Append
L to seed when type is LongType
URL: https://github.com/apache/spark/pull/27354#discussion_r370944352
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
##
@@ -282,6 +282,7 @@ abstract class HashExpression[E] extends Expression {
}
val hashResultType = CodeGenerator.javaType(dataType)
+val typedSeed = if (dataType.sameType(LongType)) s"${seed}L" else s"$seed"
Review comment:
Would it change the result? it would just let it not fail.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors
srowen commented on issue #27360: [SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors URL: https://github.com/apache/spark/pull/27360#issuecomment-578429359 OK, I would generally support it if it's not a huge change, optimizes performance, and doesn't change behavior / correctness. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578432613 **[Test build #117401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117401/testReport)** for PR 27030 at commit [`30686b4`](https://github.com/apache/spark/commit/30686b4a19edf7ceb13cd1d86838d4a1dd5c861b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578412857 **[Test build #117401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117401/testReport)** for PR 27030 at commit [`30686b4`](https://github.com/apache/spark/commit/30686b4a19edf7ceb13cd1d86838d4a1dd5c861b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578432911 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578432913 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117401/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578432911 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-578432913 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117401/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578434410 **[Test build #117402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117402/testReport)** for PR 27359 at commit [`3247cd4`](https://github.com/apache/spark/commit/3247cd418baca9cebd579726e11c1e0a379441d8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578434664 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578434667 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22161/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578434667 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22161/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578434664 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA removed a comment on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578434410 **[Test build #117402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117402/testReport)** for PR 27359 at commit [`3247cd4`](https://github.com/apache/spark/commit/3247cd418baca9cebd579726e11c1e0a379441d8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
SparkQA commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578437271 **[Test build #117402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117402/testReport)** for PR 27359 at commit [`3247cd4`](https://github.com/apache/spark/commit/3247cd418baca9cebd579726e11c1e0a379441d8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins commented on issue #27359: [WIP][SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578437329 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117402/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27359: [SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0
AmplabJenkins removed a comment on issue #27359: [SPARK-23435][SPARKR][TESTS] Update testthat to >= 2.0.0 URL: https://github.com/apache/spark/pull/27359#issuecomment-578437323 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
