Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
I did a quick micro-benchmark on this and got:
> scala> :paste
> // Entering paste mode (ctrl-D to finish)
>
> import scala.collection.{mutable, Map}
> def
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22209
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95305/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22209
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22211
Thanks! Merged to 2.1
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22209
**[Test build #95305 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95305/testReport)**
for PR 22209 at commit
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22209#discussion_r213143932
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala ---
@@ -77,7 +77,14 @@ class UISeleniumSuite
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22209#discussion_r213143804
--- Diff:
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -350,11 +350,22 @@ private[spark] class AppStatusListener(
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22042
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22042
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95319/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22042
**[Test build #95319 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95319/testReport)**
for PR 22042 at commit
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22192#discussion_r213142394
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -130,6 +130,16 @@ private[spark] class Executor(
private val
Github user bersprockets commented on the issue:
https://github.com/apache/spark/pull/22188
@gatorsmile Thanks much!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user NiharS commented on a diff in the pull request:
https://github.com/apache/spark/pull/22192#discussion_r213140764
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -130,6 +130,16 @@ private[spark] class Executor(
private val
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22247
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95303/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22247
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22247
**[Test build #95303 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95303/testReport)**
for PR 22247 at commit
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22233#discussion_r213138024
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -671,7 +674,7 @@ case class AlterTableRecoverPartitionsCommand(
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22188
Normally, we do not backport such improvement PRs. However, the risk of
this PR is pretty small. I think it is fine. Let me do this.
---
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/22233#discussion_r213137139
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -671,7 +674,7 @@ case class AlterTableRecoverPartitionsCommand(
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22042
**[Test build #95319 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95319/testReport)**
for PR 22042 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22246
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22246
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95304/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22042
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22042
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user bersprockets commented on the issue:
https://github.com/apache/spark/pull/22188
@gatorsmile
>Why 2.2 only?
Only that I forgot that master is already on 2.4. We should do 2.3 as well,
but I haven't tested it yet.
Do I need to do anything on my end to
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22246
**[Test build #95304 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95304/testReport)**
for PR 22246 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22183
For Hive tables, column resolution is always case insensitive. However,
When `spark.sql.hive.convertMetastoreParquet` is true, users might face
inconsistent behaviors when they use native
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22184#discussion_r213135626
--- Diff: docs/sql-programming-guide.md ---
@@ -1895,6 +1895,10 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
-
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17280
**[Test build #95318 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95318/testReport)**
for PR 17280 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17280
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17280
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22188
@bersprockets The risk is pretty small I think. I am fine to backport it to
the previous versions. Why 2.2 only?
---
-
To
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/22193
@HyukjinKwon, those changes probably don't need to be in this PR, but this
is just a demonstration that we can remove `SaveMode` without changing test
cases. The larger issue is that this doesn't
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22104
**[Test build #95317 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95317/testReport)**
for PR 22104 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22104
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22104
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17280
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95316/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17280
**[Test build #95316 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95316/testReport)**
for PR 17280 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17280
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17280
**[Test build #95316 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95316/testReport)**
for PR 17280 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17280
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17280
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22208
**[Test build #95315 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95315/testReport)**
for PR 22208 at commit
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/22238#discussion_r213129120
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -2812,6 +2812,12 @@ See [Input Sources](#input-sources) and [Output
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22208
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22208
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user erikerlandson commented on the issue:
https://github.com/apache/spark/pull/22212
I agree there's an argument for keeping this, but an alternative would be
to leave the original for backward compatability, deprecate it, and recommend
people make use of custom pod templates
Github user erikerlandson commented on a diff in the pull request:
https://github.com/apache/spark/pull/22212#discussion_r213127037
--- Diff: docs/running-on-kubernetes.md ---
@@ -663,11 +663,21 @@ specific to Spark on Kubernetes.
-
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/2
@xuanyuanking, while this does remove the hack, it doesn't address the
underlying problem. The problem is that there is a single RDD, which may
contain InternalRow or may contain ColumnarBatch.
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213126158
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr,
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22205#discussion_r213124828
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1349,6 +1353,12 @@ object ConvertToLocalRelation
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22236
**[Test build #95314 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95314/testReport)**
for PR 22236 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22236
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22236
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22205
Yes. Disable this rule for testing only.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/22238#discussion_r213123711
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -2812,6 +2812,12 @@ See [Input Sources](#input-sources) and [Output
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22205
@gatorsmile what are you afraid of exactly? We could check which tests are
affected. Also do you want to disable this for testing only?
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21977
**[Test build #95313 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95313/testReport)**
for PR 21977 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21977
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21977
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/21977#discussion_r213122284
--- Diff: docs/configuration.md ---
@@ -179,6 +179,15 @@ of the most common options to set are:
(e.g. 2g, 8g).
+
+
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213121794
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr,
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/21977#discussion_r213121178
--- Diff: docs/configuration.md ---
@@ -179,6 +179,15 @@ of the most common options to set are:
(e.g. 2g, 8g).
+
+
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22104
**[Test build #95312 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95312/testReport)**
for PR 22104 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22104
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user rdblue closed the pull request at:
https://github.com/apache/spark/pull/22206
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213120096
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr,
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21546
Hey @HyukjinKwon , after going through the previous benchmarks, it seems
out-of-order batches had more of an effect on performance that I thought with
`toPandas`. The current revision of this
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/22206
@HyukjinKwon and @viirya, thank you for looking at this commit, but I like
@cloud-fan's approach to fixing this in #22244 better than this work-around.
I'm going to close this in favor of that
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22104
Build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22236
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22236
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95294/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22236
**[Test build #95294 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95294/testReport)**
for PR 22236 at commit
Github user bersprockets commented on the issue:
https://github.com/apache/spark/pull/22188
@cloud-fan @gatorsmile Should we merge this also onto 2.2? It was a clean
cherry-pick for me (from master to branch-2.2), and I ran the top and bottom
tests (6000 columns, 1 million rows, 67
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95295/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21638
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/7
**[Test build #95311 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95311/testReport)**
for PR 7 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21638
**[Test build #95295 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95295/testReport)**
for PR 21638 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22205
It would be safer to turn off this rule, since it will skip the actual
query execution. Normally, the tests are introduced for testing end-to-end
scenarios instead of applying this rule.
---
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22205#discussion_r213113632
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1349,6 +1353,12 @@ object ConvertToLocalRelation
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21546
**[Test build #95310 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95310/testReport)**
for PR 21546 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21546
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21546
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/4
Yeah, was just seeing if there were more comments. Merged to master
---
-
To unsubscribe, e-mail:
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213112726
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr,
Github user rekhajoshm commented on the issue:
https://github.com/apache/spark/pull/22231
thanks for the fyi @HyukjinKwon , ack.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21546
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/22210
cc @brkyvz
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213110408
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr,
Github user bogdanrdc commented on the issue:
https://github.com/apache/spark/pull/22205
Turning the whole rule off will disable also the existing
Project(LocalRelation). By invalid you mean they will not test what they are
supposed to test anymore?
---
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213109101
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr,
Github user bogdanrdc commented on a diff in the pull request:
https://github.com/apache/spark/pull/22205#discussion_r213108632
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1349,6 +1353,12 @@ object ConvertToLocalRelation
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22249
I remember that @rxin was against adding much of these functions here
through the various programmatic APIs: only the most used ones should have been
exposed and I am not sure this is a frequently
Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/22246#discussion_r213107503
--- Diff: repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala ---
@@ -124,6 +141,26 @@ class SparkILoop(in0: Option[BufferedReader], out:
Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/22246#discussion_r213107072
--- Diff: repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala ---
@@ -148,9 +148,13 @@ class SparkILoop(in0: Option[BufferedReader], out:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22205
Many test cases will be invalid after this rule is applied, since they are
built on LocalRelation. Thus, how about turning off the rule
`ConvertToLocalRelation` by using the conf
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22104
Build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22104
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95309/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22104
**[Test build #95309 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95309/testReport)**
for PR 22104 at commit
101 - 200 of 537 matches
Mail list logo
