[GitHub] [spark] wangyum commented on a change in pull request #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
wangyum commented on a change in pull request #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#discussion_r357000600 ## File path: dev/create-release/spark-rm/Dockerfile ## @@ -62,14 +62,18 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \ curl -sL https://deb.nodesource.com/setup_11.x | bash && \ $APT_INSTALL nodejs && \ # Install needed python packages. Use pip for installing packages (for consistency). - $APT_INSTALL libpython2.7-dev libpython3-dev python-pip python3-pip && \ - pip install $BASE_PIP_PKGS && \ - pip install $PIP_PKGS && \ + $APT_INSTALL libpython3-dev python3-pip && \ + # Change default python version to python3. + update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && \ + update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2 && \ + update-alternatives --set python /usr/bin/python3.6 && \ + pip3 install $BASE_PIP_PKGS && \ + pip3 install $PIP_PKGS && \ cd && \ - virtualenv -p python3 /opt/p35 && \ + python /opt/p35 && \ Review comment: I reverted commit 1f94bf4 and d6be46e, I'd like to to confirm is it correct??  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26858: [SPARK-30120][ML] Use BoundedPriorityQueue for small dataset in LSH approxNearestNeighbors
SparkQA commented on issue #26858: [SPARK-30120][ML] Use BoundedPriorityQueue for small dataset in LSH approxNearestNeighbors URL: https://github.com/apache/spark/pull/26858#issuecomment-564892760 **[Test build #115214 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115214/testReport)** for PR 26858 at commit [`48a91ee`](https://github.com/apache/spark/commit/48a91eefad0901881c2b3abd44e6fb49d024ca8a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26861: [SPARK-30231][SQL][PYTHON] Support explain mode in PySpark df.explain
maropu commented on issue #26861: [SPARK-30231][SQL][PYTHON] Support explain mode in PySpark df.explain URL: https://github.com/apache/spark/pull/26861#issuecomment-564889766 I think the explain modes look useful for debugging, but I'm not sure that this fix to add an optional param in explain is a right approach. Could you check this, @HyukjinKwon @viirya ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu opened a new pull request #26861: [SPARK-30231][SQL][PYTHON] Support explain mode in PySpark df.explain
maropu opened a new pull request #26861: [SPARK-30231][SQL][PYTHON] Support explain mode in PySpark df.explain URL: https://github.com/apache/spark/pull/26861 ### What changes were proposed in this pull request? This pr intends to support explain modes implemented in #26829 for PySpark. ### Why are the changes needed? For better debugging info. in PySpark dataframes. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Added UTs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26828: [SPARK-30198][Core] BytesToBytesMap does not grow internal long array as expected
cloud-fan commented on issue #26828: [SPARK-30198][Core] BytesToBytesMap does not grow internal long array as expected URL: https://github.com/apache/spark/pull/26828#issuecomment-564888682 Can we at least provide a manual regression test in the PR description? so that people can try and evaluate the risk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-564881726 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115208/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-564881708 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-564881708 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-564881726 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115208/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions
AmplabJenkins commented on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions URL: https://github.com/apache/spark/pull/26808#issuecomment-564880957 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] fuwhu commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex
fuwhu commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex URL: https://github.com/apache/spark/pull/26850#issuecomment-564880985 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions
AmplabJenkins commented on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions URL: https://github.com/apache/spark/pull/26808#issuecomment-564880974 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115207/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions
AmplabJenkins removed a comment on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions URL: https://github.com/apache/spark/pull/26808#issuecomment-564880957 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions
AmplabJenkins removed a comment on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions URL: https://github.com/apache/spark/pull/26808#issuecomment-564880974 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115207/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
SparkQA removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-564829724 **[Test build #115208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115208/testReport)** for PR 26656 at commit [`94a4a06`](https://github.com/apache/spark/commit/94a4a0650e473d32eb1a8b6a7f9299a2a4e134fa). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
SparkQA commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-564880276 **[Test build #115208 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115208/testReport)** for PR 26656 at commit [`94a4a06`](https://github.com/apache/spark/commit/94a4a0650e473d32eb1a8b6a7f9299a2a4e134fa). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions
SparkQA commented on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions URL: https://github.com/apache/spark/pull/26808#issuecomment-564879606 **[Test build #115207 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115207/testReport)** for PR 26808 at commit [`d780dfc`](https://github.com/apache/spark/commit/d780dfc8660c5f78b683b4baf810908ec2a143ca). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions
SparkQA removed a comment on issue #26808: [SPARK-30184][SQL] Implement a helper method for aliasing functions URL: https://github.com/apache/spark/pull/26808#issuecomment-564829559 **[Test build #115207 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115207/testReport)** for PR 26808 at commit [`d780dfc`](https://github.com/apache/spark/commit/d780dfc8660c5f78b683b4baf810908ec2a143ca). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
beliefer commented on a change in pull request #26656: [SPARK-27986][SQL]
Support ANSI SQL filter clause for aggregate expression
URL: https://github.com/apache/spark/pull/26656#discussion_r356986082
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala
##
@@ -151,7 +244,7 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan]
{
}
// Setup unique distinct aggregate children.
- val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq.distinct
+ val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq
Review comment:
Sorry!, I fogot to restore it. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] edrevo commented on issue #26678: [SPARK-30226][SQL] Remove withXXX functions in WriteBuilder
edrevo commented on issue #26678: [SPARK-30226][SQL] Remove withXXX functions in WriteBuilder URL: https://github.com/apache/spark/pull/26678#issuecomment-564879129 @cloud-fan , I have addressed your PR feedback. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26859: [SPARK-29455][WEBUI] Improve tooltip information for Stages
AmplabJenkins commented on issue #26859: [SPARK-29455][WEBUI] Improve tooltip information for Stages URL: https://github.com/apache/spark/pull/26859#issuecomment-564878364 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26860: [SPARK-30230][SQL] Like ESCAPE syntax can not use '_' and '%'
AmplabJenkins commented on issue #26860: [SPARK-30230][SQL] Like ESCAPE syntax can not use '_' and '%' URL: https://github.com/apache/spark/pull/26860#issuecomment-564878353 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation
dongjoon-hyun commented on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation URL: https://github.com/apache/spark/pull/26857#issuecomment-564878048 Thank you for pinging me. I'll take a look now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
dongjoon-hyun commented on a change in pull request #26656: [SPARK-27986][SQL]
Support ANSI SQL filter clause for aggregate expression
URL: https://github.com/apache/spark/pull/26656#discussion_r356984048
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala
##
@@ -151,7 +244,7 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan]
{
}
// Setup unique distinct aggregate children.
- val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq.distinct
+ val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq
Review comment:
Do we need to remove `.distinct` here? At the first glance, the previous one
looks more robust. Do we invoke `.distinct` in all below cases?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
dongjoon-hyun commented on a change in pull request #26656: [SPARK-27986][SQL]
Support ANSI SQL filter clause for aggregate expression
URL: https://github.com/apache/spark/pull/26656#discussion_r356984048
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala
##
@@ -151,7 +244,7 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan]
{
}
// Setup unique distinct aggregate children.
- val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq.distinct
+ val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq
Review comment:
Do we need to remove `.distinct` here? At the first glance, the previous one
looks more robust.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
dongjoon-hyun commented on a change in pull request #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#discussion_r356983421 ## File path: sql/core/src/test/resources/sql-tests/inputs/group-by-filter.sql ## @@ -0,0 +1,150 @@ +-- Test filter clause for aggregate expression. + +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW testData AS SELECT * FROM VALUES +(1, 1), (1, 2), (2, 1), (2, 2), (3, 1), (3, 2), (null, 1), (3, null), (null, null) Review comment: My idea was to have more values in `EMP` table like `testData` table. For example, one null per column? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
dongjoon-hyun commented on a change in pull request #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#discussion_r356983170 ## File path: sql/core/src/test/resources/sql-tests/inputs/postgreSQL/aggregates_part3.sql ## @@ -230,16 +230,16 @@ select max(min(unique1)) from tenk1; -- [SPARK-27986] Support Aggregate Expressions with filter -- FILTER tests --- select min(unique1) filter (where unique1 > 100) from tenk1; +select min(unique1) filter (where unique1 > 100) from tenk1; --- select sum(1/ten) filter (where ten > 0) from tenk1; +select sum(1/ten) filter (where ten > 0) from tenk1; -- select ten, sum(distinct four) filter (where four::text ~ '123') from onek a -- group by ten; --- select ten, sum(distinct four) filter (where four > 10) from onek a --- group by ten --- having exists (select 1 from onek b where sum(distinct a.four) = b.four); +select ten, sum(distinct four) filter (where four > 10) from onek a +group by ten +having exists (select 1 from onek b where sum(distinct a.four) = b.four); Review comment: Thank you for the filing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on issue #26860: [SPARK-30230][SQL] Like ESCAPE syntax can not use '_' and '%'
ulysses-you commented on issue #26860: [SPARK-30230][SQL] Like ESCAPE syntax can not use '_' and '%' URL: https://github.com/apache/spark/pull/26860#issuecomment-564876203 cc @beliefer @cloud-fan @maropu @gengliangwang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface
dongjoon-hyun commented on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface URL: https://github.com/apache/spark/pull/26855#issuecomment-564876235 Yes. @HeartSaVioR . Jenkins has been very slow today. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you opened a new pull request #26860: [SPARK-30230][SQL] Like ESCAPE syntax can not use '_' and '%'
ulysses-you opened a new pull request #26860: [SPARK-30230][SQL] Like ESCAPE syntax can not use '_' and '%' URL: https://github.com/apache/spark/pull/26860 ### What changes were proposed in this pull request? Since [25001](https://github.com/apache/spark/pull/25001), spark support like escape syntax. But '%' and '_' is the reserve char in `Like` expression. We can not use them as escape char. ### Why are the changes needed? Avoid some unexpect problem when using like escape syntax. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Add UT. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sharangk opened a new pull request #26859: [SPARK-29455][WEBUI] Improve tooltip information for Stages
sharangk opened a new pull request #26859: [SPARK-29455][WEBUI] Improve tooltip information for Stages URL: https://github.com/apache/spark/pull/26859 ### What changes were proposed in this pull request? Adding tooltip to SQL tab for better usability. ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] prakharjain09 commented on a change in pull request #26569: [SPARK-29938] [SQL] Add batching support in Alter table add partition flow
prakharjain09 commented on a change in pull request #26569: [SPARK-29938] [SQL]
Add batching support in Alter table add partition flow
URL: https://github.com/apache/spark/pull/26569#discussion_r356981977
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##
@@ -470,14 +470,26 @@ case class AlterTableAddPartitionCommand(
CatalogTablePartition(normalizedSpec, table.storage.copy(
locationUri = location.map(CatalogUtils.stringToURI)))
}
-catalog.createPartitions(table.identifier, parts, ignoreIfExists =
ifNotExists)
+
+// Hive metastore may not have enough memory to handle millions of
partitions in single RPC.
+// Also the request to metastore times out when adding lot of partitions
in one shot.
+// we should split them into smaller batches
+val batchSize = sparkSession.sessionState.conf.partitionBatchSize
+parts.toIterator.grouped(batchSize).foreach { batch =>
+ catalog.createPartitions(table.identifier, batch, ignoreIfExists =
ifNotExists)
+}
if (table.stats.nonEmpty) {
if (sparkSession.sessionState.conf.autoSizeUpdateEnabled) {
-val addedSize = parts.map { part =>
- CommandUtils.calculateLocationSize(sparkSession.sessionState,
table.identifier,
-part.storage.locationUri)
-}.sum
+def calculatePartSize(part: CatalogTablePartition) =
CommandUtils.calculateLocationSize(
+ sparkSession.sessionState, table.identifier,
part.storage.locationUri)
+val threshold =
sparkSession.sparkContext.conf.get(RDD_PARALLEL_LISTING_THRESHOLD)
+val partSizes = if (parts.length > threshold) {
Review comment:
@srowen Yes this is not exactly related to batching. This is to parallelize
the file listing and improve overall performance. A similar change is done in
AlterTableRecoverPartition in the past -
https://github.com/apache/spark/blob/v2.4.4/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala#L621
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #26853: [SPARK-30104][SQL][FOLLOWUP] V2 catalog named 'global_temp' should always be masked
cloud-fan closed pull request #26853: [SPARK-30104][SQL][FOLLOWUP] V2 catalog named 'global_temp' should always be masked URL: https://github.com/apache/spark/pull/26853 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec
AmplabJenkins removed a comment on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec URL: https://github.com/apache/spark/pull/26846#issuecomment-564873536 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115205/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec
AmplabJenkins removed a comment on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec URL: https://github.com/apache/spark/pull/26846#issuecomment-564873527 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec
AmplabJenkins commented on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec URL: https://github.com/apache/spark/pull/26846#issuecomment-564873536 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115205/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec
AmplabJenkins commented on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec URL: https://github.com/apache/spark/pull/26846#issuecomment-564873527 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec
SparkQA removed a comment on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec URL: https://github.com/apache/spark/pull/26846#issuecomment-564824982 **[Test build #115205 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115205/testReport)** for PR 26846 at commit [`ce80397`](https://github.com/apache/spark/commit/ce80397e072a37170aaec0180076c777a9176b6a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec
SparkQA commented on issue #26846: [SPARK-30213][SQL] Remove the mutable status in ShuffleQueryStageExec URL: https://github.com/apache/spark/pull/26846#issuecomment-564871665 **[Test build #115205 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115205/testReport)** for PR 26846 at commit [`ce80397`](https://github.com/apache/spark/commit/ce80397e072a37170aaec0180076c777a9176b6a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26854: [SPARK-30040][SQL] DROP FUNCTION should do multi-catalog resolution
AmplabJenkins removed a comment on issue #26854: [SPARK-30040][SQL] DROP FUNCTION should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26854#issuecomment-564868394 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26854: [SPARK-30040][SQL] DROP FUNCTION should do multi-catalog resolution
AmplabJenkins removed a comment on issue #26854: [SPARK-30040][SQL] DROP FUNCTION should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26854#issuecomment-564868400 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20023/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
AmplabJenkins removed a comment on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564868351 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20024/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
AmplabJenkins removed a comment on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564868346 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
AmplabJenkins commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564868346 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
AmplabJenkins commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564868351 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20024/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26854: [SPARK-30040][SQL] DROP FUNCTION should do multi-catalog resolution
AmplabJenkins commented on issue #26854: [SPARK-30040][SQL] DROP FUNCTION should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26854#issuecomment-564868394 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26854: [SPARK-30040][SQL] DROP FUNCTION should do multi-catalog resolution
AmplabJenkins commented on issue #26854: [SPARK-30040][SQL] DROP FUNCTION should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26854#issuecomment-564868400 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20023/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #26858: [SPARK-30120][ML] Use BoundedPriorityQueue for small dataset in LSH approxNearestNeighbors
huaxingao commented on issue #26858: [SPARK-30120][ML] Use BoundedPriorityQueue for small dataset in LSH approxNearestNeighbors URL: https://github.com/apache/spark/pull/26858#issuecomment-564865584 cc @zhengruifeng @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #26858: [SPARK-30120][ML] Use BoundedPriorityQueue for small dataset in LSH approxNearestNeighbors
huaxingao commented on a change in pull request #26858: [SPARK-30120][ML] Use
BoundedPriorityQueue for small dataset in LSH approxNearestNeighbors
URL: https://github.com/apache/spark/pull/26858#discussion_r356972577
##
File path: mllib/src/main/scala/org/apache/spark/ml/feature/LSH.scala
##
@@ -138,21 +139,31 @@ private[ml] abstract class LSHModel[T <: LSHModel[T]]
// Limit the use of hashDist since it's controversial
val hashDistUDF = udf((x: Seq[Vector]) => hashDistance(x, keyHash),
DataTypes.DoubleType)
val hashDistCol = hashDistUDF(col($(outputCol)))
-
- // Compute threshold to get around k elements.
- // To guarantee to have enough neighbors in one pass, we need (p - err)
* N >= M
- // so we pick quantile p = M / N + err
- // M: the number of nearest neighbors; N: the number of elements in
dataset
- val relativeError = 0.05
- val approxQuantile = numNearestNeighbors.toDouble / count + relativeError
val modelDatasetWithDist = modelDataset.withColumn(distCol, hashDistCol)
- if (approxQuantile >= 1) {
-modelDatasetWithDist
+ // for a small dataset, use BoundedPriorityQueue
+ if (count < 1000) {
Review comment:
what is a good number to use here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao opened a new pull request #26858: [SPARK-30120][ML] Use BoundedPriorityQueue for small dataset in LSH approxNearestNeighbors
huaxingao opened a new pull request #26858: [SPARK-30120][ML] Use BoundedPriorityQueue for small dataset in LSH approxNearestNeighbors URL: https://github.com/apache/spark/pull/26858 ### What changes were proposed in this pull request? Use BoundedPriorityQueue for small datasets in ```LSH.approxNearestNeighbors``` ### Why are the changes needed? For small datasets, we can get exact result instead of using ```approxQuantile``` ### Does this PR introduce any user-facing change? no ### How was this patch tested? Use existing unit tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface
AmplabJenkins commented on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface URL: https://github.com/apache/spark/pull/26855#issuecomment-564864712 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface
AmplabJenkins removed a comment on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface URL: https://github.com/apache/spark/pull/26855#issuecomment-564864720 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20022/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface
AmplabJenkins commented on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface URL: https://github.com/apache/spark/pull/26855#issuecomment-564864720 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20022/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface
AmplabJenkins removed a comment on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface URL: https://github.com/apache/spark/pull/26855#issuecomment-564864712 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #26838: [SPARK-30144][ML][PySpark] Make MultilayerPerceptronClassificationModel extend MultilayerPerceptronParams
huaxingao commented on issue #26838: [SPARK-30144][ML][PySpark] Make MultilayerPerceptronClassificationModel extend MultilayerPerceptronParams URL: https://github.com/apache/spark/pull/26838#issuecomment-564863249 cc @zhengruifeng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #26838: [SPARK-30144][ML][PySpark] Make MultilayerPerceptronClassificationModel extend MultilayerPerceptronParams
huaxingao commented on issue #26838: [SPARK-30144][ML][PySpark] Make MultilayerPerceptronClassificationModel extend MultilayerPerceptronParams URL: https://github.com/apache/spark/pull/26838#issuecomment-564863152 @dongjoon-hyun @srowen @viirya Thanks for the review. Since ```MultilayerPerceptronParams``` has ```layers```, after ```MultilayerPerceptronClassificationModel``` extends ```MultilayerPerceptronParams```, I have to rename ```layers```. It's not good to rename ```layers``` in ```MultilayerPerceptronParams``` because the getter/setter are public APIs. ```MultilayerPerceptronClassificationModel``` is the only one that doesn't have the training params. All the other XXXModel extend the corresponding XXXParams. In addition, as what is said in the description of the jira https://issues.apache.org/jira/browse/SPARK-30144, user wants to have a way to track what parameters are best during a crossvalidation, so I think it makes sense to expose ```MultilayerPerceptronParams``` to ```MultilayerPerceptronClassificationModel``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
AmplabJenkins removed a comment on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564862412 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115210/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564862119 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115209/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
AmplabJenkins commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564862412 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115210/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
AmplabJenkins removed a comment on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564862403 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
AmplabJenkins commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564862403 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564862111 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564862111 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564862119 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115209/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
SparkQA removed a comment on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564842536 **[Test build #115210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115210/testReport)** for PR 26773 at commit [`68c496c`](https://github.com/apache/spark/commit/68c496c3e969692b7e2a01146691b980466ac09d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
SparkQA commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564861648 **[Test build #115210 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115210/testReport)** for PR 26773 at commit [`68c496c`](https://github.com/apache/spark/commit/68c496c3e969692b7e2a01146691b980466ac09d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function.
SparkQA commented on issue #26773: [SPARK-30126][CORE]support space in file path and name for addFile and addJar function. URL: https://github.com/apache/spark/pull/26773#issuecomment-564861157 **[Test build #115213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115213/testReport)** for PR 26773 at commit [`df3c1e3`](https://github.com/apache/spark/commit/df3c1e3f0095c13e254ac555f94ca08be30e7d4b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface
SparkQA commented on issue #26855: [SPARK-30227][SQL] Add close() on DataWriter interface URL: https://github.com/apache/spark/pull/26855#issuecomment-564861158 **[Test build #115212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115212/testReport)** for PR 26855 at commit [`8058dbf`](https://github.com/apache/spark/commit/8058dbf928686242b31133a57daa1bc13b22ce84). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
HyukjinKwon commented on a change in pull request #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#discussion_r356968880 ## File path: dev/create-release/spark-rm/Dockerfile ## @@ -62,14 +62,18 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \ curl -sL https://deb.nodesource.com/setup_11.x | bash && \ $APT_INSTALL nodejs && \ # Install needed python packages. Use pip for installing packages (for consistency). - $APT_INSTALL libpython2.7-dev libpython3-dev python-pip python3-pip && \ - pip install $BASE_PIP_PKGS && \ - pip install $PIP_PKGS && \ + $APT_INSTALL libpython3-dev python3-pip && \ + # Change default python version to python3. + update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && \ + update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2 && \ + update-alternatives --set python /usr/bin/python3.6 && \ + pip3 install $BASE_PIP_PKGS && \ + pip3 install $PIP_PKGS && \ cd && \ - virtualenv -p python3 /opt/p35 && \ + python /opt/p35 && \ Review comment: > `make: /opt/p35/bin/python: Command not found` Can we try to set `SPHINXPYTHON=python` and see if it works? Seems it's just because `python` was not found. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
HyukjinKwon commented on a change in pull request #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#discussion_r356968880 ## File path: dev/create-release/spark-rm/Dockerfile ## @@ -62,14 +62,18 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \ curl -sL https://deb.nodesource.com/setup_11.x | bash && \ $APT_INSTALL nodejs && \ # Install needed python packages. Use pip for installing packages (for consistency). - $APT_INSTALL libpython2.7-dev libpython3-dev python-pip python3-pip && \ - pip install $BASE_PIP_PKGS && \ - pip install $PIP_PKGS && \ + $APT_INSTALL libpython3-dev python3-pip && \ + # Change default python version to python3. + update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && \ + update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2 && \ + update-alternatives --set python /usr/bin/python3.6 && \ + pip3 install $BASE_PIP_PKGS && \ + pip3 install $PIP_PKGS && \ cd && \ - virtualenv -p python3 /opt/p35 && \ + python /opt/p35 && \ Review comment: > `make: /opt/p35/bin/python: Command not found` Can we try to set `SPHINXPYTHON=python` and see if it works? Seems it's just because `python` was not found. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
SparkQA removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564835385 **[Test build #115209 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115209/testReport)** for PR 26848 at commit [`1f2ede8`](https://github.com/apache/spark/commit/1f2ede8d7de242ed38007e302dcca3ed97593160). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
SparkQA commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564860841 **[Test build #115209 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115209/testReport)** for PR 26848 at commit [`1f2ede8`](https://github.com/apache/spark/commit/1f2ede8d7de242ed38007e302dcca3ed97593160). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564860528 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115204/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564860522 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564860522 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564860528 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115204/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on issue #26831: [SPARK-30201][SQL] HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT
ulysses-you commented on issue #26831: [SPARK-30201][SQL] HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT URL: https://github.com/apache/spark/pull/26831#issuecomment-564860219 Yes. It is that case. Change to `DEFAULT` is also more suitable with `HiveInspectors` than `JAVA`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564858964 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115206/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564858964 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115206/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564858757 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
AmplabJenkins removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564858757 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
SparkQA removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564823944 **[Test build #115204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115204/testReport)** for PR 26848 at commit [`98cd80a`](https://github.com/apache/spark/commit/98cd80aac0708094a1a66a4e9c629f24f0fd762a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3
dongjoon-hyun commented on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3 URL: https://github.com/apache/spark/pull/26856#issuecomment-564858309 Thank you, @HyukjinKwon and @maropu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
SparkQA commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564858180 **[Test build #115204 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115204/testReport)** for PR 26848 at commit [`98cd80a`](https://github.com/apache/spark/commit/98cd80aac0708094a1a66a4e9c629f24f0fd762a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
SparkQA commented on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564856963 **[Test build #115206 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115206/testReport)** for PR 26848 at commit [`02896e3`](https://github.com/apache/spark/commit/02896e38fcf42da5cd05c350cb2622ccd80fc061). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
SparkQA removed a comment on issue #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#issuecomment-564829513 **[Test build #115206 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115206/testReport)** for PR 26848 at commit [`02896e3`](https://github.com/apache/spark/commit/02896e38fcf42da5cd05c350cb2622ccd80fc061). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #26848: [SPARK-30216][INFRA] Use python3 in Docker release image
wangyum commented on a change in pull request #26848: [SPARK-30216][INFRA] Use python3 in Docker release image URL: https://github.com/apache/spark/pull/26848#discussion_r356963835 ## File path: dev/create-release/spark-rm/Dockerfile ## @@ -62,14 +62,18 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \ curl -sL https://deb.nodesource.com/setup_11.x | bash && \ $APT_INSTALL nodejs && \ # Install needed python packages. Use pip for installing packages (for consistency). - $APT_INSTALL libpython2.7-dev libpython3-dev python-pip python3-pip && \ - pip install $BASE_PIP_PKGS && \ - pip install $PIP_PKGS && \ + $APT_INSTALL libpython3-dev python3-pip && \ + # Change default python version to python3. + update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && \ + update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2 && \ + update-alternatives --set python /usr/bin/python3.6 && \ + pip3 install $BASE_PIP_PKGS && \ + pip3 install $PIP_PKGS && \ cd && \ - virtualenv -p python3 /opt/p35 && \ + python /opt/p35 && \ Review comment: Thank you @HyukjinKwon . We need to do it: ``` = Building documentation... Command: /opt/spark-rm/release-build.sh docs Log file: docs.log Command FAILED. Check full logs for details. Copying jquery.min.js from Scala API to Java API for page post-processing of badges Copying api_javadocs.js to Java API for page post-processing of badges Appending content of api-javadocs.css to JavaDoc stylesheet.css for badge styles Moving to python/docs directory and building sphinx. make: /opt/p35/bin/python: Command not found /opt/p35/bin/python -msphinx -b html -d _build/doctrees . _build/html make: /opt/p35/bin/python: Command not found Makefile:80: recipe for target 'html' failed make: *** [html] Error 127 jekyll 3.8.6 | Error: Python doc generation failed ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] XuQianJin-Stars commented on a change in pull request #26852: [SPARK-30221] Enhanced implementation of PrometheusPushGateWaySink
XuQianJin-Stars commented on a change in pull request #26852: [SPARK-30221]
Enhanced implementation of PrometheusPushGateWaySink
URL: https://github.com/apache/spark/pull/26852#discussion_r356963214
##
File path:
core/src/main/scala/org/apache/spark/metrics/sink/PrometheusPushGateWay.scala
##
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.metrics.sink
+
+import java.util
+import java.util.Properties
+
+import com.codahale.metrics.MetricRegistry
+import io.prometheus.client.{CollectorRegistry, Gauge}
+import io.prometheus.client.exporter.PushGateway
+import org.apache.commons.lang3.StringUtils
+
+import org.apache.spark.SecurityManager
+
+/**
+ * PrometheusPushGateWay that exports Metric Metrics via Prometheus
PushGateway.
+ */
+private[spark] class PrometheusPushGateWay(
Review comment:
> Could you explain the difference from #19775?
hi @dongjoon-hyun Thank you very much, There is no difference in
functionality. I don't see it in the spark source code, so I implemented it
myself. #19775 is better written. I will close this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26853: [SPARK-30104][SQL][FOLLOWUP] V2 catalog named 'global_temp' should always be masked
cloud-fan commented on issue #26853: [SPARK-30104][SQL][FOLLOWUP] V2 catalog named 'global_temp' should always be masked URL: https://github.com/apache/spark/pull/26853#issuecomment-564854026 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3
HyukjinKwon closed pull request #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3 URL: https://github.com/apache/spark/pull/26856 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3
HyukjinKwon commented on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3 URL: https://github.com/apache/spark/pull/26856#issuecomment-564853262 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation
AmplabJenkins removed a comment on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation URL: https://github.com/apache/spark/pull/26857#issuecomment-564852179 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation
AmplabJenkins removed a comment on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation URL: https://github.com/apache/spark/pull/26857#issuecomment-564852184 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20021/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation
AmplabJenkins commented on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation URL: https://github.com/apache/spark/pull/26857#issuecomment-564852184 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20021/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation
AmplabJenkins commented on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation URL: https://github.com/apache/spark/pull/26857#issuecomment-564852179 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3
AmplabJenkins removed a comment on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3 URL: https://github.com/apache/spark/pull/26856#issuecomment-564851013 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3
AmplabJenkins removed a comment on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3 URL: https://github.com/apache/spark/pull/26856#issuecomment-564851019 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115202/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3
AmplabJenkins commented on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3 URL: https://github.com/apache/spark/pull/26856#issuecomment-564851019 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115202/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3
AmplabJenkins commented on issue #26856: [SPARK-30228][BUILD] Update zstd-jni to 1.4.4-3 URL: https://github.com/apache/spark/pull/26856#issuecomment-564851013 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation
SparkQA commented on issue #26857: [SPARK-30162][SQL] Add PushedFilters to metadata in Parquet DSv2 implementation URL: https://github.com/apache/spark/pull/26857#issuecomment-564850717 **[Test build #115211 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115211/testReport)** for PR 26857 at commit [`72fd2bf`](https://github.com/apache/spark/commit/72fd2bf89054df554cadb8a7366b013112ad88bb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
