Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184645573
--- Diff:
core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala ---
@@ -63,10 +63,12 @@ private[spark] class ChunkedByteBuffer(var chunks:
Ar
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184644678
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala ---
@@ -56,6 +56,13 @@ class ChunkedByteBufferSuite extends SparkFunSuite {
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184644454
--- Diff:
core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala ---
@@ -63,10 +63,12 @@ private[spark] class ChunkedByteBuffer(var chunks:
Ar
Github user sadhen closed the pull request at:
https://github.com/apache/spark/pull/21166
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user sadhen commented on the issue:
https://github.com/apache/spark/pull/21166
I have checked the latest master code.
``` scala
// If the executor is no longer running any scheduled tasks, mark
it as idle
if (executorIdToTaskIds.contains(executor
GitHub user sadhen reopened a pull request:
https://github.com/apache/spark/pull/21166
[SPARK-11334][CORE] clear idle executors in executorIdToTaskIds keySet
## What changes were proposed in this pull request?
quote from #11205
> Executors may never be idle. Curren
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21153#discussion_r184632888
--- Diff: python/pyspark/ml/util.py ---
@@ -523,11 +534,29 @@ def getAndSetParams(instance, metadata):
"""
Extract Params from metad
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21021#discussion_r184632402
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -185,6 +183,8 @@ trait ArraySortUtil extend
Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21153#discussion_r184626842
--- Diff: python/pyspark/ml/util.py ---
@@ -523,11 +534,29 @@ def getAndSetParams(instance, metadata):
"""
Extract Params from
Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21153#discussion_r184620777
--- Diff: python/pyspark/ml/util.py ---
@@ -417,15 +419,24 @@ def _get_metadata_to_save(instance, sc,
extraMetadata=None, paramMap=None):
Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21153#discussion_r184620855
--- Diff: python/pyspark/ml/util.py ---
@@ -417,15 +419,24 @@ def _get_metadata_to_save(instance, sc,
extraMetadata=None, paramMap=None):
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r184625530
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -362,4 +371,34 @@ public static void verifyProxyAcce
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r184626752
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -362,4 +371,34 @@ public static void verifyProxyAcce
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r184625003
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -107,9 +109,16 @@ public HiveAuthFactory(HiveConf co
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r184626560
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -362,4 +371,34 @@ public static void verifyProxyAcce
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r184625977
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -362,4 +371,34 @@ public static void verifyProxyAcce
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21049
@henryr Since SubqueryAlias is used as a correlation name and used mostly
for resolving attributes, in my understanding its not safe to apply this
optimization. I will borrow @gatorsmile 's exa
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21179
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21179
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user ddna1021 opened a pull request:
https://github.com/apache/spark/pull/21179
Update StreamingKMeans.scala
## What changes were proposed in this pull request?
I think the ân_t+tâ in the following code may be wrong, it shoud be
ân_t+1â that means is the num
Github user liutang123 commented on the issue:
https://github.com/apache/spark/pull/21164
@cloud-fan hi, fan, do you have time to see this pr? I think this is a
critical bug.
---
-
To unsubscribe, e-mail: reviews-un
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21175
cc @kiszk @maropu @cloud-fan @jiangxb1987
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comm
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21143
@rdblue After some more thoughts, I don't think Spark can support per-split
physical plan changing in the near future. My best suggestion for mixed format
data source is to have different implemen
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21178
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21178
**[Test build #89917 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89917/testReport)**
for PR 21178 at commit
[`fd48235`](https://github.com/apache/spark/commit/f
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21178
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89917/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2713/
Tes
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21143
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21143
**[Test build #89919 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89919/testReport)**
for PR 21143 at commit
[`172dca0`](https://github.com/apache/spark/commit/17
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21174
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2712/
Tes
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21174
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21174
**[Test build #89918 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89918/testReport)**
for PR 21174 at commit
[`e6e9397`](https://github.com/apache/spark/commit/e6
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21070
ok, I'll work on it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21178
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21174
@gatorsmile Thanks a lot. Addressed the comments.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For addi
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21178
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2711/
Tes
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21178
**[Test build #89917 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89917/testReport)**
for PR 21178 at commit
[`fd48235`](https://github.com/apache/spark/commit/fd
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21178
Ping @mridulm , please help to review, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additiona
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21178
[SPARK-24110][Thrift-Server] Avoid UGI.loginUserFromKeytab in STS
## What changes were proposed in this pull request?
Spark ThriftServer will call UGI.loginUserFromKeytab twice in
initia
Github user manbuyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184612119
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala ---
@@ -56,6 +56,12 @@ class ChunkedByteBufferSuite extends SparkFunSuite {
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21168
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20280
Probably that'd work but also it'd be trickier to add / remove that
configuration. Another similar option maybe just close this for now and target
this for 3.0.0 since we already started to talk
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21070
@maropu Thank you for works! Could you also run/improve the
micro-benchmark?
https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21177
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21141
I am okay either way but I thought @bersprockets agreed up on doing this
separately here? Doctests stuff need more looks and I think this one alone can
be merged separately.
@bersprocke
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21177
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2710/
Tes
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21174#discussion_r184609062
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -55,7 +55,9 @@ private[sql] object
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21174#discussion_r184609073
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
---
@@ -955,4 +955,28 @@ class SubquerySuite extends QueryTest with
SharedSQ
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21174#discussion_r184609040
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -76,7 +76,10 @@ object FileSourceStrategy
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21177
**[Test build #89916 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89916/testReport)**
for PR 21177 at commit
[`0ef1b28`](https://github.com/apache/spark/commit/0e
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21169#discussion_r184608524
--- Diff: docs/sql-programming-guide.md ---
@@ -1805,12 +1805,13 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21177
@gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spar
GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/21177
[SPARK-24111][SQL] Add the TPCDS v2.7 (latest) queries in
TPCDSQueryBenchmark
## What changes were proposed in this pull request?
This pr added the TPCDS v2.7 (latest) queries in `TPCDSQueryBen
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21174
This fix is safe for us to backport to the previous versions. To achieve a
better performance, we can start a separate job to execute these subqueries. :
)
---
---
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184607965
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala ---
@@ -56,6 +56,12 @@ class ChunkedByteBufferSuite extends SparkFunSuite {
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/21121
I'm not sure we really need this function.
If the purpose is only for `transform` function you mentionsed at
https://github.com/apache/spark/pull/21121#issuecomment-383319312, how about
adding a
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21169
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89915/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21169
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89914/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21169
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21174#discussion_r184607398
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -76,7 +76,10 @@ object FileSourceStrategy e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21169
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21169
**[Test build #89915 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89915/testReport)**
for PR 21169 at commit
[`d248d4c`](https://github.com/apache/spark/commit/d
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21157
agree we should avoid removing test code
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional co
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21169
**[Test build #89914 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89914/testReport)**
for PR 21169 at commit
[`cbe37f2`](https://github.com/apache/spark/commit/c
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21141#discussion_r184607210
--- Diff: python/pyspark/sql/tests.py ---
@@ -3021,6 +3021,17 @@ def test_sort_with_nulls_order(self):
class HiveSparkSubmitTests(SparkSubmitT
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21174#discussion_r184607164
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -55,7 +55,9 @@ private[sql] object P
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21174#discussion_r184606904
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
---
@@ -955,4 +955,28 @@ class SubquerySuite extends QueryTest with
SharedSQL
301 - 367 of 367 matches
Mail list logo
