Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
@sujith71955 Thanks. I see. The case is somehow different with the problem
this PR wants to solve. But I think it is a reasonable use case. May you want
to create a ticket for us to track it?
---
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22687
@viirya @HyukjinKwon I did the code changes and then I found the condition
is not reachable, as I have stated in PR description.
Just feel that it won't hurt to have such handling in
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22688
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22688
cc @cloud-fan and @rdblue, this is more conservative but I would prefer to
revert it at https://github.com/apache/spark/pull/22686 rather then exposing
append mode in 2.4. I don't think it's a
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22688
**[Test build #97196 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97196/testReport)**
for PR 22688 at commit
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22688
[SPARK-25700][SQL] Creates ReadSupport in only Append Mode in Data Source
V2 write path
## What changes were proposed in this pull request?
Alternative for
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r224023061
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -169,15 +174,16 @@ object
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22687#discussion_r224021798
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
---
@@ -534,6 +534,13 @@ private[parquet]
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22685
I think you're mostly trying to target changes styles. If there is not
actual benefit rather then just styles, I wouldn't do this and just help review
other PRs.
---
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/16677
Mainly i think we are trying to interpolate the number of partitions
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22686
**[Test build #97195 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97195/testReport)**
for PR 22686 at commit
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/16677
@viirya I am having a usecase where a normal query is taking around 5
seconds where same query with limit 5000 is taking around 17 sec. when i was
checking i could find bottleneck in the
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22686
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22686
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22687
Won't such predicates be simplified at `BooleanSimplification` rule?
---
-
To unsubscribe, e-mail:
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
@sujith71955 For `executeTake`, to optimize it we need to collect
statistics of RDD. `executeTake` incrementally scans partitions. Ideally, it
should just scan few partitions to return `n` rows, and
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22685
**[Test build #97194 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97194/testReport)**
for PR 22685 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22687
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22687
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r224013755
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -3356,21 +3356,11 @@ class Dataset[T] private[sql](
* user-registered
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22687
**[Test build #97193 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97193/testReport)**
for PR 22687 at commit
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/22687
[SPARK-25702][SQL] Push down filters with `Not` operator in Parquet
## What changes were proposed in this pull request?
Currently, in ParquetFilters, predicates inside `Not` operator
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22686
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22686
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r224012183
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala
---
@@ -75,95 +76,74 @@ trait QueryExecutionListener {
*/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22686
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97191/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22686
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22686
**[Test build #97191 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97191/testReport)**
for PR 22686 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22686
**[Test build #97192 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97192/testReport)**
for PR 22686 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22686
**[Test build #97191 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97191/testReport)**
for PR 22686 at commit
Github user sadhen commented on the issue:
https://github.com/apache/spark/pull/22685
@HyukjinKwon OK, most of changes are related to parens. I will consider to
reset it back for backporting friendliness.
---
-
To
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22686
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22686
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r224008348
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala ---
@@ -71,14 +72,35 @@ object SQLExecution {
val callSite
Github user shivusondur commented on a diff in the pull request:
https://github.com/apache/spark/pull/22677#discussion_r224007729
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala ---
@@ -159,6 +159,11 @@ private class AsyncEventQueue(
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r224007732
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -3356,21 +3356,11 @@ class Dataset[T] private[sql](
* user-registered
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22686
[SPARK-25700][SQL] Partially revert append mode support in Data Source V2
## What changes were proposed in this pull request?
This PR partially revert
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r224006828
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala
---
@@ -75,95 +76,74 @@ trait QueryExecutionListener {
*/
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22684#discussion_r224005596
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala
---
@@ -382,5 +382,40 @@ class OrcFilterSuite extends
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r224004348
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -169,15 +174,16 @@ object
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/16677
@viirya Are we also looking to optimize CollectLimitExec part? I saw in
SparkPlan we have an executeTake() method which basically interpolate the
number of partitions and processes the limit
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r224000145
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -3356,21 +3356,11 @@ class Dataset[T] private[sql](
* user-registered
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r224000809
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala ---
@@ -71,14 +72,35 @@ object SQLExecution {
val callSite =
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22682
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22682
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97190/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22682
**[Test build #97190 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97190/testReport)**
for PR 22682 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22685
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22685
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97189/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22685
**[Test build #97189 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97189/testReport)**
for PR 22685 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22685
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22682
**[Test build #97190 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97190/testReport)**
for PR 22682 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22685
**[Test build #97189 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97189/testReport)**
for PR 22685 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22682
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22682
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22685
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user sadhen opened a pull request:
https://github.com/apache/spark/pull/22685
[SQL][MINOR][Refactor] Refactor sql/core
## What changes were proposed in this pull request?
Only minor changes on Scala syntax.
## How was this patch tested?
Existing Tests
You
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22010
If this is not yet in 2.4 it shouldnât be merged now.
On Wed, Oct 10, 2018 at 10:57 AM Holden Karau
wrote:
> Open question: is this suitable for branch-2.4 since it predates the
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
Open question: is this suitable for branch-2.4 since it predates the branch
cut or not? (I know we've gone back and forth on how we do that).
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22598
**[Test build #97188 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97188/testReport)**
for PR 22598 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22684
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22684
**[Test build #97187 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97187/testReport)**
for PR 22684 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22684
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r223988838
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -169,15 +174,16 @@ object
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22684
@dbtsai @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/22598#discussion_r223987644
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/TokenUtilSuite.scala
---
@@ -0,0 +1,114 @@
+/*
+ * Licensed
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/22598#discussion_r223987456
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSecurityHelper.scala
---
@@ -0,0 +1,91 @@
+/*
+ *
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/22684
[SPARK-25699][SQL] Partially push down conjunctive predicated in Orc
## What changes were proposed in this pull request?
Inspired by https://github.com/apache/spark/pull/22574 .
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223983702
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ---
@@ -189,23 +192,34 @@ class QueryExecution(val sparkSession:
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223983537
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala ---
@@ -167,6 +172,58 @@ package object util {
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r223982672
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -169,15 +174,16 @@ object
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223982046
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala ---
@@ -167,6 +172,58 @@ package object util {
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223980665
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
---
@@ -455,21 +457,37 @@ abstract class TreeNode[BaseType <:
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223979931
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/CatalystDataToAvro.scala
---
@@ -52,7 +52,7 @@ case class CatalystDataToAvro(child:
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223979392
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -633,4 +633,14 @@ package object config {
.stringConf
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22429
@boy-uber the thing you are suggesting is a pretty big undertaking and
beyond the scope of this PR.
If you are going to add structured plans to the explain output, you
probably also
Github user LantaoJin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22678#discussion_r223966833
--- Diff: dev/run-tests-jenkins.py ---
@@ -176,7 +177,8 @@ def main():
build_display_name = os.environ["BUILD_DISPLAY_NAME"]
build_url =
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22682#discussion_r223966211
--- Diff: docs/sql-programming-guide.md ---
@@ -1890,6 +1890,10 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
#
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22682
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22682
**[Test build #97186 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97186/testReport)**
for PR 22682 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22682
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97186/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22683
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22683
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/22575
Is this still a WIP?
Using isStreaming tag in DDL to mark if a table is streaming or not is
brilliant. It keeps compatible with batch queries sql.
If possible, I think not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22683
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user httfighter opened a pull request:
https://github.com/apache/spark/pull/22683
[SPARK-25696] The storage memory displayed on spark Application UI isâ¦
⦠incorrect.
## What changes were proposed in this pull request?
Change the cardinality of the unit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21669
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97183/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21669
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21669
**[Test build #97183 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97183/testReport)**
for PR 21669 at commit
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22545#discussion_r223953269
--- Diff: python/pyspark/sql/session.py ---
@@ -156,7 +156,7 @@ def getOrCreate(self):
default.
>>> s1 =
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22682
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22682
**[Test build #97186 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97186/testReport)**
for PR 22682 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22682
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/22682
[SPARK-20946][SPARK-25525][SQL][FOLLOW-UP] Update the migration guide.
## What changes were proposed in this pull request?
This is a follow-up pr of #18536 and #22545 to update the
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/22682
cc @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/22575
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22466
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22466
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97182/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22466
**[Test build #97182 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97182/testReport)**
for PR 22466 at commit
401 - 499 of 499 matches
Mail list logo
