GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/9412
[SPARK-11457][Streaming][YARN] Fix incorrect AM proxy filter conf recovery
from checkpoint
Currently Yarn AM proxy filter configuration is recovered from checkpoint
file when Spark Streaming
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/8669#discussion_r43827693
--- Diff: sbin/start-slaves.sh ---
@@ -52,11 +51,11 @@ if [ "$SPARK_MASTER_IP" = "" ]; then
fi
if [ "$START_T
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/8669#discussion_r43865494
--- Diff: sbin/spark-config.sh ---
@@ -19,21 +19,12 @@
# should not be executable directly
# also should not be passed any arguments, since we
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9476#discussion_r43981544
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/DStreamGraph.scala ---
@@ -167,7 +167,8 @@ final private[streaming] class DStreamGraph
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-153984512
Hi @mccheah sorry for late response, looks like this message is out of my
radar.
From Yarn's side, Yarn makes this preemption decision depends on stat
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/7410#issuecomment-154331499
Yeah, will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/9508#issuecomment-154334413
Thanks a lot @vazin, this suffers me a lot recent days.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22117
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/22164
[SPARK-23679][YARN] Fix AmIpFilter cannot work in RM HA scenario
## What changes were proposed in this pull request?
YARN `AmIpFilter` adds a new parameter "RM_HA_URLS" to sup
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22164
@vanzin @tgravescs would you please help to review, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22164
Gently ping again @vanzin @tgravescs . Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22213#discussion_r212530383
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2062,8 +2062,10 @@ private[spark] object Utils extends Logging {
try
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22199
ok to test.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22186
My local maven build also failed.
I think the problem is that`ShutdownHookManager` is implemented in Scala,
the complied method signature may be different when invoked from Java, I'
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22164
I think it should be related to this JIRA
(https://issues.apache.org/jira/browse/YARN-7269). Seems like a Hadoop 2.9/3.0+
issue
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22213#discussion_r212889779
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2062,8 +2062,10 @@ private[spark] object Utils extends Logging {
try
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22213#discussion_r213160007
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2062,8 +2062,10 @@ private[spark] object Utils extends Logging {
try
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22164#discussion_r213168025
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala
---
@@ -126,4 +136,21 @@ private[spark] class
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22164
Thanks @vanzin .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22186
The fix itself LGTM, but I don't think this could solve the STS shutdown
hook conflict problem with Hadoop.
---
---
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22213#discussion_r214231103
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
@@ -1144,6 +1144,46 @@ class SparkSubmitSuite
conf1.get
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22289#discussion_r214233802
--- Diff:
launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java ---
@@ -200,6 +200,7 @@ void addOptionString(List cmd, String
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22279
Hi @LucaCanali do you have an output current AM metrics? I would like to
know what kind of metrics will be output for now
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22279#discussion_r214234325
--- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
---
@@ -103,6 +103,14 @@ private[spark] class MetricsSystem private
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22186
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22186
I see. Thanks for the explain, I checked the code again, yes you're right.
Let me retrigger the test again, will merge it if everything is
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22213#discussion_r214244665
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
@@ -1144,6 +1144,46 @@ class SparkSubmitSuite
conf1.get
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22186
Merging to master branch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22299
Seems there's another similar PR #22296 .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For addit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21756
I think the use case here is quite specific, I'm not sure if it is a good
idea to make `SparkHadoopUtil` ServiceLoader-able to support your requirement.
Typically I don't think user
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18142
I see. Thanks for the note.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22372
Jackson version below 2.9.5 has CVE issues, I would suggest to upgrade to
2.9.6 as #21596 did.
---
-
To unsubscribe, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22372
Do we officially support hadoop3 in branch 2.4? If branch 2.4 doesn't
target to support Hadoop3 and this fix is only for Hadoop3, then I don't think
it is meaningful to hav
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22372
Btw, I don't think we can run current Spark with Hadoop 3.1 without any
change.
---
-
To unsubscribe, e-mail: re
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/22434
[SPARK-24685][BUILD][FOLLOWUP] Fix the nonexist profile name in release
script
## What changes were proposed in this pull request?
`without-hadoop` profile doesn't exist in
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22434
@cloud-fan @vanzin , please help to review, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22441
Is it possible to test this on Jenkins?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/23116
[SPARK-24553][UI][FOLLOWUP] Fix unnecessary UI redirect
## What changes were proposed in this pull request?
This PR is a follow-up PR of #21600 to fix the unnecessary UI redirect
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/23116
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/23116
@dongjoon-hyun , this should also be backported to branch 2.4, let me
create a backport PR.
---
-
To unsubscribe, e-mail
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/23121
[SPARK-24553][UI][FOLLOWUP][2.4 Backport] Fix unnecessary UI redirect
## What changes were proposed in this pull request?
This is a backport PR of #23116 .
This PR is a follow
Github user jerryshao closed the pull request at:
https://github.com/apache/spark/pull/23121
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/23121
OK, thanks for the reminder.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
Having issues tested with latest patch:
```
Exception in thread "main" java.lang.NoSuchM
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r194287473
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21523
I think here the problem is that `filters` are only added for those handler
who added before server starts.
This may not be the only problem for `ThriftserverTab`, I guess `SQLTable`
or
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21514#discussion_r194308977
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
---
@@ -100,7 +100,7 @@ private[spark] class
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21510#discussion_r194309780
--- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala ---
@@ -101,12 +101,12 @@ private[spark] abstract class WebUI
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21523
@mgaido91 Please fix the PR title and description to reflect the new
changes you made.
---
-
To unsubscribe, e-mail: reviews
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21533#discussion_r194927085
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1517,9 +1517,12 @@ class SparkContext(config: SparkConf) extends
Logging
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21548
[SPARK-24518] Using Hadoop credential provider API to store password
## What changes were proposed in this pull request?
Current Spark configs password in a plaintext way, like putting
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
CC @vanzin @tgravescs would you please help to review, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21533
Just take another look on this issue. I think the fix is just to make it
work, but not make it work correctly.
The fix here and the original code actually treats scheme "local"
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21533
"local" scheme was supported long ago for users who already deploy jars on
every node. HDI heavily uses th
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
Thanks @tgravescs I will add the docs about how to use it. I was thinking
to add a test case, but it looks like may not be easy to add one
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
Is there any left work, or everything is already done? @dbtsai
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21588
Curious how to test with Hadoop 3 profile in jenkins?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21588#discussion_r196689287
--- Diff: dev/run-tests.py ---
@@ -305,8 +305,8 @@ def get_hadoop_profiles(hadoop_version):
"""
sbt_maven_
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
> I just have an issue with your summary. It makes it sound like there's no
way to securely configure these things today, and that's not true.
>
> You can have the pas
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r196992852
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r196995267
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21548#discussion_r197033939
--- Diff: core/src/main/scala/org/apache/spark/SSLOptions.scala ---
@@ -179,9 +185,11 @@ private[spark] object SSLOptions extends Logging
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21533#discussion_r197068331
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1517,9 +1517,19 @@ class SparkContext(config: SparkConf) extends
Logging
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21548#discussion_r197327620
--- Diff: core/src/main/scala/org/apache/spark/SSLOptions.scala ---
@@ -179,9 +185,11 @@ private[spark] object SSLOptions extends Logging
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21588
@HyukjinKwon , I'm in favor of @vanzin 's comment, we should fix things
first and then back to this one.
---
-
To u
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
Gently ping @dbtsai , is there any left work?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21633
[SPARK-21917][CORE][FOLLOWUP] Minor change to
spark.yarn.dist.forceDownloadSchemes to support wildcard '*'
## What changes were proposed in this pull request?
In the case
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21633
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
OK, I'm going to merge it. We can fix the following issues if exists.
---
-
To unsubscribe, e-mail: reviews-uns
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21635#discussion_r198004697
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerSource.scala
---
@@ -0,0 +1,35
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21644
@mgaido91 , would you please check all other response to see if it returns
as expected, not only download link.
---
-
To
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
> The only change with this PR is that the welcome message will be printed
first, and then the Spark URL will be shown latter. It's a minor difference.
I think we should create a
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21533#discussion_r198682844
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1519,7 +1519,12 @@ class SparkContext(config: SparkConf) extends
Logging
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21533
I think maybe we could:
1) either ignore the files with "local" scheme, and let user to decide how
to fetch the files, like what current fix.
2) or copy the 'local&#
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
@dongjoon-hyun can you please create a JIRA to track this issue.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21664
Would you please explain how to reproduce this issue?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21633
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21633
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21709
Hi @jzhuge what is the purpose of supporting reading metrics conf from
HDFS/S3?
---
-
To unsubscribe, e-mail: reviews
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21709
If you want metrics conf to be centralized without needing to put it on to
different nodes, you can set it through `SparkConf` with prefix
"spark.metrics.conf.", MetricsSystem als
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21663
Can you please describe the usage scenario for this under standalone mode?
I know it is used in yarn mode, because Hadoop and Spark are two distributions,
they may build and run with different
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21709
From my point, I don't see a special reason that it cannot be achieved by
SparkConf. Also you can specify different local metrics conf file for different
users/purpose, I don't t
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21709#discussion_r200219590
--- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala
---
@@ -129,8 +131,11 @@ private[spark] class MetricsConfig(conf: SparkConf
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21664#discussion_r200220083
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1049,6 +1049,13 @@ class DAGScheduler(
abortStage(stage
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21633
Any more comment @vanzin ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21658
ok to test.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21658
Would you please add a UT for it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21658#discussion_r200226750
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -1569,7 +1569,7 @@ private[spark] object BlockManager
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21663
I see, thanks for verifying. I'm neutral to the fix, because I cannot see
the strong requirement of this feature compared to running on YARN, usually in
standalone mode we deploy stand
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21658#discussion_r200535022
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -1569,7 +1569,7 @@ private[spark] object BlockManager
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21666
Any update on this issue @felixcheung , this blocks 2.3.2 release, just
want to make sure if it is still in-progress
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21658#discussion_r200536325
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -1569,7 +1569,7 @@ private[spark] object BlockManager
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21666
Thanks @felixcheung .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21658
Please add the UTs as I mentioned before.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21658#discussion_r200586976
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -1569,7 +1570,8 @@ private[spark] object BlockManager
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21633
Thanks @jiangxb1987 , merging to master branch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
1401 - 1500 of 1537 matches
Mail list logo
