Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21664
The issue is not introduced by Spark itself, it is introduced by user code,
is it better to fix in the user side rather than in Spark? Besides, I'm not so
sure that Spark should take
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21659
I think we can treat this fix as a temp fix, once sphinx issue is resolved,
we can update in Spark accordingly. This is not a big problem, only affects
python doc, so I think it is safe to merge
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21734#discussion_r201322643
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
---
@@ -193,8 +193,7 @@ object
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21664
Please fix the JIRA number, you linked to wrong JIRA number in your PR
title.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21658
LGTM, merging to master branch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21658
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21658
Hi @sharkdtu , did you also verify this in your cluster, to see if the
locality is correct or not?
---
-
To unsubscribe, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21734
a) `spark.yarn.access.namenodes` is not used for such purpose, I don't
think it is meaningful to change this configuration.
b) `spark.yarn.access.namenodes` is already deprecated
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21734
Shall we fix this issue in `HadoopFSDelegationTokenProvider`, maybe we
should try catch the delegation token obtain process
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21731
This seems not a necessary fix. `master` can be configured via spark-submit
argument `--master`, not a best practice to set it in code
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21659#discussion_r200869393
--- Diff: python/docs/Makefile ---
@@ -1,19 +1,44 @@
# Makefile for Sphinx documentation
#
+ifndef SPHINXBUILD
+ifndef
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21659#discussion_r200869531
--- Diff: python/docs/Makefile ---
@@ -1,19 +1,44 @@
# Makefile for Sphinx documentation
#
+ifndef SPHINXBUILD
+ifndef
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21633
Thanks @jiangxb1987 , merging to master branch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21658#discussion_r200586976
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -1569,7 +1570,8 @@ private[spark] object BlockManager
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21658
Please add the UTs as I mentioned before.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21666
Thanks @felixcheung .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21658#discussion_r200536325
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -1569,7 +1569,7 @@ private[spark] object BlockManager
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21666
Any update on this issue @felixcheung , this blocks 2.3.2 release, just
want to make sure if it is still in-progress
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21658#discussion_r200535022
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -1569,7 +1569,7 @@ private[spark] object BlockManager
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21663
I see, thanks for verifying. I'm neutral to the fix, because I cannot see
the strong requirement of this feature compared to running on YARN, usually in
standalone mode we deploy standalone
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21658#discussion_r200226750
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -1569,7 +1569,7 @@ private[spark] object BlockManager
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21658
Would you please add a UT for it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21658
ok to test.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21633
Any more comment @vanzin ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21664#discussion_r200220083
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1049,6 +1049,13 @@ class DAGScheduler(
abortStage(stage
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21709#discussion_r200219590
--- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala
---
@@ -129,8 +131,11 @@ private[spark] class MetricsConfig(conf: SparkConf
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21709
From my point, I don't see a special reason that it cannot be achieved by
SparkConf. Also you can specify different local metrics conf file for different
users/purpose, I don't think
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21663
Can you please describe the usage scenario for this under standalone mode?
I know it is used in yarn mode, because Hadoop and Spark are two distributions,
they may build and run with different
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21709
If you want metrics conf to be centralized without needing to put it on to
different nodes, you can set it through `SparkConf` with prefix
"spark.metrics.conf.", MetricsSystem als
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21709
Hi @jzhuge what is the purpose of supporting reading metrics conf from
HDFS/S3?
---
-
To unsubscribe, e-mail: reviews
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21633
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21633
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21664
Would you please explain how to reproduce this issue?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
@dongjoon-hyun can you please create a JIRA to track this issue.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21533
I think maybe we could:
1) either ignore the files with "local" scheme, and let user to decide how
to fetch the files, like what current fix.
2) or copy the 'local' sc
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21533#discussion_r198682844
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1519,7 +1519,12 @@ class SparkContext(config: SparkConf) extends
Logging
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
> The only change with this PR is that the welcome message will be printed
first, and then the Spark URL will be shown latter. It's a minor difference.
I think we should create a J
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21644
@mgaido91 , would you please check all other response to see if it returns
as expected, not only download link
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21635#discussion_r198004697
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerSource.scala
---
@@ -0,0 +1,35
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
OK, I'm going to merge it. We can fix the following issues if exists.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21633
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21633
[SPARK-21917][CORE][FOLLOWUP] Minor change to
spark.yarn.dist.forceDownloadSchemes to support wildcard '*'
## What changes were proposed in this pull request?
In the case of getting
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
Gently ping @dbtsai , is there any left work?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21588
@HyukjinKwon , I'm in favor of @vanzin 's comment, we should fix things
first and then back to this one.
---
-
To unsubscribe
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21548#discussion_r197327620
--- Diff: core/src/main/scala/org/apache/spark/SSLOptions.scala ---
@@ -179,9 +185,11 @@ private[spark] object SSLOptions extends Logging
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21533#discussion_r197068331
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1517,9 +1517,19 @@ class SparkContext(config: SparkConf) extends
Logging
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21548#discussion_r197033939
--- Diff: core/src/main/scala/org/apache/spark/SSLOptions.scala ---
@@ -179,9 +185,11 @@ private[spark] object SSLOptions extends Logging
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r196995267
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r196992852
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
> I just have an issue with your summary. It makes it sound like there's no
way to securely configure these things today, and that's not true.
>
> You can have the password in t
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21588#discussion_r196689287
--- Diff: dev/run-tests.py ---
@@ -305,8 +305,8 @@ def get_hadoop_profiles(hadoop_version):
"""
sbt_maven_
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21588
Curious how to test with Hadoop 3 profile in jenkins?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
Is there any left work, or everything is already done? @dbtsai
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
Thanks @tgravescs I will add the docs about how to use it. I was thinking
to add a test case, but it looks like may not be easy to add one
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21533
"local" scheme was supported long ago for users who already deploy jars on
every node. HDI heavily uses th
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21533
Just take another look on this issue. I think the fix is just to make it
work, but not make it work correctly.
The fix here and the original code actually treats scheme "local"
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
CC @vanzin @tgravescs would you please help to review, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21548
[SPARK-24518] Using Hadoop credential provider API to store password
## What changes were proposed in this pull request?
Current Spark configs password in a plaintext way, like putting
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21533#discussion_r194927085
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1517,9 +1517,12 @@ class SparkContext(config: SparkConf) extends
Logging
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21523
@mgaido91 Please fix the PR title and description to reflect the new
changes you made.
---
-
To unsubscribe, e-mail: reviews
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21510#discussion_r194309780
--- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala ---
@@ -101,12 +101,12 @@ private[spark] abstract class WebUI
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21514#discussion_r194308977
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
---
@@ -100,7 +100,7 @@ private[spark] class
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21523
I think here the problem is that `filters` are only added for those handler
who added before server starts.
This may not be the only problem for `ThriftserverTab`, I guess `SQLTable
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21495#discussion_r194287473
--- Diff:
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala
---
@@ -21,8 +21,22 @@ import scala.collection.mutable
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21495
Having issues tested with latest patch:
```
Exception in thread "main" java.lang.NoSuchM
GitHub user jerryshao opened a pull request:
https://github.com/apache/hive/pull/364
HIVE-16391: Add a new classifier for hive-exec to be used by Spark
This fix adding a new classifier for hive-exec artifact (`core-spark`),
which is specifically used for Spark. Details in
[SPARK
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21453
I think we can close this one as a dup of #21495 .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21490
@arunmahadevan I have already proposed a fix for this issue
https://github.com/apache/spark/pull/20958, but seems it is not so necessary to
fix based on the comments, and pending to review
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21475
I think original description is OK, just not too detailed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21475
ok to test.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21466
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21466#discussion_r192033436
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/YARNHadoopDelegationTokenManager.scala
---
@@ -44,6 +44,10
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21466#discussion_r192028294
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/YARNHadoopDelegationTokenManager.scala
---
@@ -44,6 +44,8
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21455
Simply making these fields publicly accessible seems a little weird from
Spark's side. Maybe we can use reflection instead
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21453
Scala 2.11.12 cannot be built against with current Spark, due to some
method changes in REPL. We have tried internally
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21422
It is not valid for branch-2.3, 2.3 already removed the support of Scala
2.10.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21422
Do we still have a 2.2 release? If not, then this fix seems obsolete.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21430
Merging to master and branch 2.3.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21420
Thanks @HyukjinKwon !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21420
`--files` can be used, but user (Livy in our case) should differentiate
whether the added files are python dependency or just plaintext
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21422
Seems this paragraph is already removed in master branch
(https://issues.apache.org/jira/browse/SPARK-19810). So this is a branch 2.2-
issue only
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21422
And in master/Spark 2.3 code, since we don't support Scala 2.10, instead we
support 2.12, so we should change that part of doc accordingly
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21422
Yes, but the PR should be against master branch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21422
Why do you submit PR against branch 2.2?
Besides scala-2.10 profile is no longer valid after 2.3
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21426#discussion_r190803966
--- Diff: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala ---
@@ -153,4 +154,25 @@ object PythonRunner {
.map { p => formatP
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21420#discussion_r190783462
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -430,18 +430,15 @@ private[spark] class SparkSubmit extends Logging
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21426
Did you try remote py files, does it have similar issue?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21426#discussion_r190778192
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -372,8 +376,27 @@ private[spark] class SparkSubmit extends Logging
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21390#discussion_r190571272
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -97,6 +99,10 @@ private[deploy] class Worker(
private val
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21390
YARN will clean container local dirs when container (executor) is exited,
so this may not be a problem in YARN.
YARN has a useful configuration "yarn.nodemanager.delete.debug-dela
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21420
CC @HyukjinKwon @vanzin please help to review, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21420
[SPARK-24377][Spark Submit] make --py-files work in non pyspark application
## What changes were proposed in this pull request?
For some Spark applications, though they're a java program
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21396#discussion_r190112524
--- Diff:
core/src/main/scala/org/apache/spark/deploy/security/HiveDelegationTokenProvider.scala
---
@@ -85,7 +85,10 @@ private[spark] class
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21396#discussion_r190109928
--- Diff:
core/src/main/scala/org/apache/spark/deploy/security/HiveDelegationTokenProvider.scala
---
@@ -85,7 +85,10 @@ private[spark] class
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21396
Can you please describe your scenario @LantaoJin ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21216#discussion_r189255656
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
---
@@ -200,7 +200,27 @@ object
101 - 200 of 2761 matches
Mail list logo
