Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21216
I'm OK with the current fix, just some minor style comments.
@vanzin would you please take another look? Thanks
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21216#discussion_r189240503
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
---
@@ -200,7 +200,31 @@ object
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21216#discussion_r189240461
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
---
@@ -200,7 +200,31 @@ object
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21290
branch 2.3 is not auto mergeable and the related code is changed, will not
backport to 2.3
---
-
To unsubscribe, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21290
LGTM, merging to master and branch 2.3.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21290#discussion_r187852287
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
@@ -180,6 +180,25 @@ class SparkSubmitSuite
appArgs.toString
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21290#discussion_r187852160
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -76,6 +75,7 @@ private[deploy] class SparkSubmitArguments(args
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21199
I was thinking if it is too overkill to receive data in the driver side and
publish them to the executors via RPC? This might give user a wrong impression
that data should be received
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21290
LGTM, just some minor comments.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21290#discussion_r187847736
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -76,6 +75,7 @@ private[deploy] class SparkSubmitArguments(args
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21290#discussion_r187847656
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
@@ -180,6 +180,25 @@ class SparkSubmitSuite
appArgs.toString
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21243
Merging to master branch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21243
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21279
@foxish would you please help to review this, thanks a lot!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21279
jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21279
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21268
I'm still thinking this change requires so many updates across the whole
project, and we may miss it in future if someone add new codes. Though I'm not
familiar with knox, my question
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21279
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21279
[SPARK-24219][k8s] Improve the docker building script to avoid copying
everything under examples to docker image
## What changes were proposed in this pull request?
Current docker build
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21243#discussion_r186918972
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -346,7 +346,7 @@ private[spark] class
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21267
Does it only happen in yarn client PySpark shell? I would suggest to fix
this in the SparkSubmit side, to treat this as a special case and set the
proper config
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21268
The changes here seems affect so many places, I'm wondering if there's any
other way to minimize the changes
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21243#discussion_r186639699
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -346,7 +346,7 @@ private[spark] class
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21243#discussion_r186634159
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
---
@@ -1073,14 +1074,14 @@ private[spark] class Client
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21243#discussion_r186633839
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -389,37 +389,40 @@ private[spark] class
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21243
What kind of exceptions will client AM meet usually? I think the logic is
quite simple for client AM, just wondering what kind of issue will it meet
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21245
LGTM, merging to master and branch 2.3.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21216#discussion_r186059087
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
---
@@ -196,11 +196,17 @@ object
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21216#discussion_r186015828
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
---
@@ -196,11 +196,17 @@ object
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21207
LGTM. Merging to master branch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20761
Cool, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20761
Hi @szyszy are you still going to work on this PR?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21216
I'm not so familiar with federated HDFS, but is it transparent to the
downside applications like Spark, or Spark should know all the configured NNs?
If it is transparent, then I think the token
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21178
Thanks @mridulm for your review, really appreciated!
Merging to master branch.
---
-
To unsubscribe, e-mail: reviews
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r185432085
--- Diff:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIService.scala
---
@@ -52,8 +52,22 @@ private[hive
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r185427751
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -92,7 +95,26 @@ public String getAuthName
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r185407534
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -92,7 +95,26 @@ public String getAuthName
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21178
@mridulm , can you review again? Thanks a lot.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21188
Isn't this a flat ramp-up smoothly increasing the rows per second? Your
proposal is another solution, but just two options
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r184844613
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -362,4 +371,34 @@ public static void
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21188
[SPARK-24046][SS] Fix rate source rowsPerSecond <= rampUpTime corner case
## What changes were proposed in this pull request?
Current Rate source has some issues when calculat
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21152
@HeartSaVioR what is your JIRA id?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21152
LGTM. Merging to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21166
1. We improve the DAGScheduler to always send TaskEnd message. So the issue
I found before may not be valid.
2. We refactored the LiveListenerQueue to make it more robust for internal
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r184833705
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -362,4 +371,34 @@ public static void
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r184833443
--- Diff:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIService.scala
---
@@ -52,8 +52,22 @@ private[hive
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r184833381
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -18,14 +18,11 @@
package
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21178#discussion_r184672313
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
---
@@ -362,4 +371,34 @@ public static void
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21178
Ping @mridulm , please help to review, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21178
[SPARK-24110][Thrift-Server] Avoid UGI.loginUserFromKeytab in STS
## What changes were proposed in this pull request?
Spark ThriftServer will call UGI.loginUserFromKeytab twice
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21168
The change is fail to build, please fix it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21166
Can you please check again with latest master code, I doubt the issue is
not valid any more in the latest code
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21152#discussion_r184316769
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
---
@@ -66,157 +66,115 @@ class ContinuousSuite
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21152#discussion_r184315510
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
---
@@ -66,157 +66,115 @@ class ContinuousSuite
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21138
Merging to master and branch 2.3.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21138
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21138
Thanks for the review @mridulm @vanzin . Let me test again. I will merge
the code when test is passed.
---
-
To unsubscribe
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21138
@mridulm I would treat the current fix as a workaround for SASL issue,
since it is a regression in 2.3.
For UGI refreshing issue (mainly cause STS long running failure, also lead
to SASL
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21088
ok to test.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21138
Hi @mridulm , thanks a lot for your comments.
UGI.loginUserFromKeytab is not existed any more in Spark 2.3+
(https://github.com/apache/spark/commit
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21138
[SPARK-24062][Thrift Server] Fix SASL encryption cannot enabled issue in
thrift server
## What changes were proposed in this pull request?
For the details of the exception please see
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20923
I would guess the test here doesn't actually run on Hadoop 3 profile. So we
actually doesn't test anything.
Also we still cannot use Hadoop3 even if we merge this because of Hive
issue
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21104
Jenkins, test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21104
Ahh, this looks like a bug I introduced.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21076
@SaddamKhan1490 would you please close this PR. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21098
Merging to master and branch 2.3.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21084
IIUC, null value will not be serialized (taskMemoryManager is only set in
executor side), maybe Java will leave some footprints, but the overhead should
be very small.
I'm +0 to fix
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21084
Seems OK to add `@transient`, but do you see any issue here without
`transient`?
---
-
To unsubscribe, e-mail: reviews
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21088
Should you also support this in kubernetes deploy mode?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21038
Thanks @koeninger for the review.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21038
Ping @koeninger , would you please help to review again. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21036#discussion_r181963024
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -323,7 +323,7 @@ package object config {
.internal
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21047
Merging to master. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21047
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21017
Thanks @jose-torres for your review.
@tdas would you please take a look at this PR?
---
-
To unsubscribe, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21036
Yes, this is already supported in Spark, seems like the PR is invalid.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21038
Thanks @koeninger , then I will just improve the exception message.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21038
Thanks @koeninger for your comments. I think your suggestion is valid, the
log here is just pasted from JIRA, but we also got the same issue from
customer's report.
Here in the PR
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20923
Ping @vanzin @gatorsmile , would like to hear your comments. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21038
@koeninger would you please help to review, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21038
[SPARK-22968][DStream] Fix Kafka connector partition revoked issue
## What changes were proposed in this pull request?
Kafka partitions can be revoked when new consumers joined
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21017
@jose-torres @tdas would you please help to review, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21017
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21017
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21017
[SPARK-23748][SS] Fix SS continuous process doesn't support SubqueryAlias
issue
## What changes were proposed in this pull request?
Current SS continuous doesn't support processing
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21009
Jenkins, test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21009
Jenkins, add to whitelist.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20923
I think you should also update "test-dependencies.sh" to make the new deps
file work.
---
-
To unsubscribe, e-mai
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20923
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20923
Sorry @steveloughran for the late response. most of the deps in file is
similar to "spark-deps-hadoop-2.7", so copy/rename it and run
"test-dependencies.sh" will show
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20958
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20958
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20944
Please list out the reason why do you need such change? If it is a UT bug,
why it didn't happen before?
---
-
To unsubscribe
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20958
@tdas , by thought about your suggestion about "failOnDataLoss" option, I
made a similar proposal on socket source, would you please review aga
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20923
Hi @steveloughran , I think you missed this comment. You need to create a
deps file under dev/deps and change the related script.
> Also I think we need to create a related spark-d
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20958
Thanks @tdas for your comments. I agree that socket source should only be
used in testing. But it doesn't mean that it can throw weird exception in
testing env. For example, if we're dumping
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20958
Not sure why the test is not triggered, maybe jenkins is down.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20958
Jenkins, test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/20958
[]Fix socket source honors recovered offsets issue
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20923
Also I think we need to create a related spark-deps-hadoop-3.x under
dev/deps and make dependency check work for Hadoop 3
201 - 300 of 2761 matches
Mail list logo
