Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21320#discussion_r204288547
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala
---
@@ -0,0 +1,156 @@
+/*
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21320#discussion_r204288381
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/planning/SelectedFieldSuite.scala
---
@@ -0,0 +1,388 @@
+/*
+ * Licensed
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21320#discussion_r204288233
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala
---
@@ -0,0 +1,153 @@
+/*
+ *
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21775
**[Test build #93431 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93431/testReport)**
for PR 21775 at commit
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21775#discussion_r204287713
--- Diff: docs/sql-programming-guide.md ---
@@ -1843,6 +1843,7 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21839
@xuanyuanking Could you add an end-to-end test into
ScriptTransformationSuite to verify the results?
---
-
To unsubscribe,
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21839#discussion_r204286954
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -450,13 +450,16 @@ object ColumnPruning extends
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21845
**[Test build #93430 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93430/testReport)**
for PR 21845 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21845
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21845
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21123
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93422/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21123
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21123
**[Test build #93422 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93422/testReport)**
for PR 21123 at commit
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21775#discussion_r204286227
--- Diff: docs/sql-programming-guide.md ---
@@ -1843,6 +1843,7 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21845
**[Test build #93429 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93429/testReport)**
for PR 21845 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21845
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21845
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21775#discussion_r204285948
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
---
@@ -114,7 +114,10 @@ case class CatalogTablePartition(
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/21845
[SPARK-24886][INFRA] Fix the testing script to increase timeout for Jenkins
build (from 300m to 350m)
## What changes were proposed in this pull request?
Currently, looks we hit the
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21775#discussion_r204285853
--- Diff: docs/sql-programming-guide.md ---
@@ -1843,6 +1843,7 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21845
cc @rxin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21775
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21775
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93421/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21775
**[Test build #93421 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93421/testReport)**
for PR 21775 at commit
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21764
LGTM, too
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21764
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21764
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93419/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21764
**[Test build #93419 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93419/testReport)**
for PR 21764 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21764
LGTM pending Jenkins
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21764
**[Test build #93428 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93428/testReport)**
for PR 21764 at commit
Github user hthuynh2 commented on the issue:
https://github.com/apache/spark/pull/21653
@tgravescs Can you please run the test again, thank you.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21764
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user maryannxue commented on a diff in the pull request:
https://github.com/apache/spark/pull/21764#discussion_r204279843
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -175,6 +191,41 @@ abstract class
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21764
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21653
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21653
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93418/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21653
**[Test build #93418 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93418/testReport)**
for PR 21653 at commit
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21764#discussion_r204278968
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -175,6 +191,41 @@ abstract class
Github user httfighter commented on the issue:
https://github.com/apache/spark/pull/21826
I have a suggestion that I don't know if it is reasonable.
In our spark, since we already support â||â as a string concatenation
function, I don't know if we can make such an improvement.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21830#discussion_r204277225
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -1244,46 +1244,50 @@ case class
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21772
@liutang123 Thanks for this work. I'm curious that if this is an actual
problem you hit in real application, or you just think it is problematic?
---
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21772#discussion_r204276973
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
---
@@ -278,6 +278,39 @@ class HashedRelationSuite extends
Github user httfighter commented on the issue:
https://github.com/apache/spark/pull/21826
I did the following tests in mysql.
mysql> select "abc" || "def";
++
| "abc" || "def" |
++
| 0
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21772#discussion_r204276886
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
---
@@ -770,8 +771,10 @@ private[execution] final class
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21772#discussion_r204276788
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
---
@@ -726,9 +726,10 @@ private[execution] final class
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21836
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21802#discussion_r204276502
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
---
@@ -1419,4 +1421,71 @@ class
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21836
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93427/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21836
**[Test build #93427 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93427/testReport)**
for PR 21836 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21836
**[Test build #93427 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93427/testReport)**
for PR 21836 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21836
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21836
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21440
cc @squito @mridulm @markhamstra @jerryshao @vanzin @JoshRosen @rxin
@zsxwing
---
-
To unsubscribe, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21440
Although the code quality is pretty good, I am still afraid it could
introduce some unexpected issues. Is that possible we can introduce a conf to
fall back to the previous implementation? We
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21830#discussion_r204275800
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -1244,46 +1244,50 @@ case class
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21802#discussion_r204275498
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
---
@@ -1419,4 +1421,71 @@ class
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21764
Also, can you update the title? You need to at least add `[SQL]` in the
title: https://github.com/apache/spark/pull/21764#issuecomment-405832763
---
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21764#discussion_r204275071
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -175,6 +191,41 @@ abstract class
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21784
Logging is already configurable; a switch is overkill. This seems fine.
---
-
To unsubscribe, e-mail:
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21439#discussion_r204274790
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
---
@@ -101,6 +102,17 @@ class JacksonParser(
}
Github user liutang123 commented on the issue:
https://github.com/apache/spark/pull/21772
@viirya Hi, Could you have more time to review this PR?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user httfighter commented on a diff in the pull request:
https://github.com/apache/spark/pull/21826#discussion_r204274497
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
---
@@ -442,8 +442,6 @@ case class Or(left:
Github user httfighter commented on a diff in the pull request:
https://github.com/apache/spark/pull/21826#discussion_r204274481
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
---
@@ -442,8 +442,6 @@ case class Or(left:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21784
Adding a configuration to control some logs sounds an overkill. I wouldn't
go for this way.
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20699
**[Test build #93426 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93426/testReport)**
for PR 20699 at commit
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/21784
We need to listen to @vanzin opinion.
Because the relevant code is what he wrote.
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21828
@woodthom2, if you have some plans to update this PR quite soon, please see
https://spark.apache.org/contributing.html and proceed. Otherwise, I would
suggest to leave this closed so that
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21844
I don't think we should as discussed in
https://github.com/apache/spark/pull/21784. I think this PR would better be
left closed and go discuss there.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21844
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21844
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21844
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user hejiefang opened a pull request:
https://github.com/apache/spark/pull/21844
Spark 24873
[https://issues.apache.org/jira/browse/SPARK-24873](url)
I think it is better to add a switch to increase a switch to shielding
frequent interaction reports with yarn.
You can
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21784
It has never been printed before, right? I think we can consider to turn it
on specifically for spark-submit applications for some spark-submit
applications although I am not fully sure if it's
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/21784
But for some spark-submit applications, I want these Application report for
information.
What should I do?
---
-
To
Github user cuijunyao closed the pull request at:
https://github.com/apache/spark/pull/21843
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21843
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21843
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21843
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user cuijunyao opened a pull request:
https://github.com/apache/spark/pull/21843
for test
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20861#discussion_r204272567
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1994,6 +1996,20 @@ class Analyzer(
}
}
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21830
**[Test build #93425 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93425/testReport)**
for PR 21830 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21118
so where are we on this? looks like we have 2 LGTM?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21830
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21830
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21839
LGTM cc: @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21516
**[Test build #93424 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93424/testReport)**
for PR 21516 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21075
@tnachen @susanxhuynh @mgummelt @skonto
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21516
@tnachen @susanxhuynh @mgummelt @skonto
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21516
Jenkins, ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21754
@markhamstra @cloud-fan How about the current fix in this pr? Or, another
approach idea to fix is:
https://github.com/apache/spark/compare/master...maropu:SPARK-24705-3
In the latter idea, I
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21784
Maybe we could consider avoiding this logs in shell specifically. Adding a
switch for disable/enable logs sounds an overkill.
---
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21784
? I think we don't need a switch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/21827
Please add a switch. And represented by a constant. This configuration is
added to the running-on-yarn.md document. @hejiefang
---
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21584
I see. you are referring to
https://github.com/apache/spark/pull/21584#issuecomment-406091994
got it.
---
-
To
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21517
I think we can close this
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21802#discussion_r204270437
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -1184,6 +1186,137 @@ case class
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/21784
what? I think we need to add a switch.
https://github.com/apache/spark/pull/21827
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20838#discussion_r204269682
--- Diff: python/pyspark/sql/conf.py ---
@@ -59,7 +62,7 @@ def unset(self, key):
def _checkType(self, obj, identifier):
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21841
@gengliangwang, looks JIRA link is wrong. you mean SPARK-24883?
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21474
**[Test build #93423 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93423/testReport)**
for PR 21474 at commit
1 - 100 of 290 matches
Mail list logo
