Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20923
I think we could separate cloud related stuffs to another PR, and fix only
build related stuff in this PR
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20923#discussion_r178251635
--- Diff: pom.xml ---
@@ -2671,6 +2671,15 @@
+
+ hadoop-3
+
+3.1.0-SNAPSHOT
--- End
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20920
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20860
Sorry I didn't notice it, will wait for the test finishing.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20860
Thanks, merging to master!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20922#discussion_r178027822
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamProvider.scala
---
@@ -0,0 +1,125
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20922#discussion_r177989371
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamProvider.scala
---
@@ -0,0 +1,125
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20860
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20920
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20812
@jinxing64 , I think using same name jars which contains different classes
seems practically is not a best practice. Ideally different udfs should be
packaged in different jars with different
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20860
LGTM. I'm also playing around with isolated hive classloader these days.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20860
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20920
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20928
It would be better to check other docs, not only in configurations here.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20922#discussion_r177953871
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamProvider.scala
---
@@ -0,0 +1,125
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20891
@mgaido91 what is the status of Hadoop, for example like YARN RM UI, will
it show apps which is run by other users, while this user doesn't have
permission to see
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20922#discussion_r177933081
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamProvider.scala
---
@@ -0,0 +1,125
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20922#discussion_r177932994
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamProvider.scala
---
@@ -0,0 +1,125
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20922
Thanks for the help @jose-torres .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20688
Thanks @tdas and @jose-torres .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20898
What's the relation between this pr and #20784 ? @wangyum
Are they just two fixes for the same issue
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20891
I think I fixed this issue before, but the community rejected such
proposal. They thought that listing all the applications for any user is OK
without checking the permission
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20895
Merging to master branch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20873#discussion_r176647902
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -776,6 +776,9 @@ object SparkSubmit extends CommandLineUtils with
Logging
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20873#discussion_r176646160
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -776,6 +776,9 @@ object SparkSubmit extends CommandLineUtils with
Logging
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20657
Thanks, merging to master branch!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20657
LGTM, just one small comment.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20657#discussion_r175638637
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosHadoopDelegationTokenManager.scala
---
@@ -105,7 +105,8
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20847
@mgaido91 this is already merged to branch 2.3. Please close this PR if it
is not closed automatically.
---
-
To unsubscribe
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20847
Thanks, merging to branch 2.3.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20657
I'm really sorry about the delay @vanzin @squito . I will take another
review today and back to you.
---
-
To unsubscribe, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20847
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20847
@mgaido91 would you please create a JIRA to track all the UT failures.
Hopes someone familiar with that part could fix them
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20847
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20845
@caneGuy it is not necessary to create a JIRA for such tiny fix.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20799
Thanks for the review, let me merge to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20812
Does it only fix the jars added by `sc.addJar` or using non-yarn mode?
Because yarn uses distributed cache at start, so it has a different code path,
right
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20794
@mgaido91 the PR has conflict with branch 2.3, so I don't cherry-pick it to
2.3. If you want to backport, please create another backport PR
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20794
Merging to master and brach 2.3. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20799#discussion_r174331279
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
---
@@ -247,6 +241,18 @@ private[yarn] class
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20803#discussion_r174109194
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala ---
@@ -34,6 +34,16 @@ object SQLExecution {
private val
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20803#discussion_r174038997
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala ---
@@ -34,6 +34,16 @@ object SQLExecution {
private val
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20807#discussion_r174035432
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -496,7 +497,7 @@ private[yarn] class
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20794
Ping @ajbozarth .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20659
@wangyum can you please create a relative JIRA, so that we discuss on the
JIRA. I think it is better to discuss and get a conclusion first
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20659
@gatorsmile @cloud-fan what is our plan for such thing? Currently we're
using our own forked version, which blocks the support of Hadoop 3
([SPARK-18673](https://issues.apache.org/jira/browse
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20785
I think if you're running on yarn, semantically
`spark.shuffle.service.port` is a yarn configuration specified in
yarn-site.xml. So it seems correct from semantic point
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20781
Still I'm not so sure about the root cause, but adding defensive code seems
no harm.
---
-
To unsubscribe, e-mail: reviews
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20781#discussion_r173709533
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -81,7 +81,7 @@ private[yarn] class YarnAllocator
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20799
@mridulm , would you please take a review. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/20799
[SPARK-23635][YARN] AM env variable should not overwrite same name env
variable set through spark.executorEnv.
## What changes were proposed in this pull request?
In the current Spark
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20781
This basically means that drive send multiple same kill requests to AM,
right? I'm wondering how this would happen, shall we also guarantee this in the
driver side
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20781
Does it happen only in dynamic allocation enabled scenario?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20657#discussion_r173383032
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -520,4 +520,16 @@ package object config {
.checkValue(v
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20657#discussion_r173380826
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMCredentialRenewer.scala
---
@@ -18,221 +18,160 @@ package
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20776
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20688
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20688
@tdas I tried different ways to keep history of rename, but seems git
always treat as "rm and add" for big changes
(https://coderwall.com/p/_csouq/renaming-and-changing-files-in-g
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20657#discussion_r173073703
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -520,4 +520,16 @@ package object config {
.checkValue(v
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20657#discussion_r173078047
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMCredentialRenewer.scala
---
@@ -18,221 +18,160 @@ package
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20735
Sorry my merge environment has some issues, please wait for a while.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20735
LGTM, merging to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20735#discussion_r172732010
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -736,7 +736,8 @@ private object YarnAllocator
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20688#discussion_r172730858
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/RateSourceSuite.scala
---
@@ -0,0 +1,344 @@
+/*
+ * Licensed
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20735#discussion_r172729670
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -736,7 +736,8 @@ private object YarnAllocator
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20688
@tdas @jose-torres can you please take a review when you have time, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20704
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20382#discussion_r171511528
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/TextSocketStreamSuite.scala
---
@@ -0,0 +1,300
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20382#discussion_r171506698
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/TextSocketStreamSuite.scala
---
@@ -0,0 +1,300
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20382#discussion_r171469866
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/TextSocketStreamSuite.scala
---
@@ -0,0 +1,300
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20382
Sure, I will do it today.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20683
> a extra boolean expression was added to test if a regex was present.
Can you please explain what's the meaning of "if a regex was present"?
Seems the fix is not
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/20688
[SPARK-23096][SS] Migrate rate source to V2
## What changes were proposed in this pull request?
This PR migrate micro batch rate source to V2 API and rewrite UTs to suite
V2 test
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20382
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20657
Will review it soon. ð
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20553
What is the default value if it is not configured, how do K8S control the
CPU usage by default?
Also it seems that user may configure how to differentiate between k8s
executor cores
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20382
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20382#discussion_r170178735
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/TextSocketStreamSuite.scala
---
@@ -0,0 +1,246
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20382
Sorry @tdas for the delay. I'm working on this, will push new changes soon.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20382
Hi @tdas, I'm on vacation this week, will update the code when I have time.
Sorry for the delay.
---
-
To unsubscribe, e-mail
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20596
Can you please elaborate the case to support your fix here?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20382
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20382#discussion_r167776323
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/TextSocketStreamSuite.scala
---
@@ -0,0 +1,246 @@
+/*
--- End
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20532
I would suggest to do it like what we have already done for block update
event. Since we already opened a door for block update event, it is also
acceptable to leave room for another event. User
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19775
My original intention is to expose MetricsSystem related interface in
#11994 , so that users can leverage such interface to build their own metrics
sink/source out of Spark. Unfortunately I'm
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20382
Hi @tdas , would you please help to review again, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20449#discussion_r166875113
--- Diff: core/src/test/scala/org/apache/spark/JobCancellationSuite.scala
---
@@ -320,6 +321,41 @@ class JobCancellationSuite extends SparkFunSuite
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20449
I see. Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20449
I understood your intention. I was wondering do we actually meet this issue
in production envs, or do you have a minimal reproduce code
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20449
@advancedxy did you see any issue or exception regarding to this issue?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20532#discussion_r166862037
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -53,10 +53,21 @@ package object config {
.booleanConf
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20512
Is it possible that TCP keepalive is disable by kernel, so that your
approach cannot be worked? I was thinking if it is better to add application
level heartbeat msg to detect lost workers
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20529#discussion_r166857279
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala
---
@@ -92,12 +92,14 @@ case class
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20532#discussion_r166855772
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -53,10 +53,21 @@ package object config {
.booleanConf
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20532#discussion_r166852617
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -53,10 +53,21 @@ package object config {
.booleanConf
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20532
I agree with @jiangxb1987 . @LantaoJin would you please elaborate the usage
scenario of dumping executor metrics to event log? Seems history server doesn't
leverage such information necessarily
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20532#discussion_r166818413
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -228,14 +231,23 @@ private[spark] class EventLoggingListener
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20532#discussion_r166805463
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -228,14 +231,23 @@ private[spark] class EventLoggingListener
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20532#discussion_r166805197
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -53,10 +53,21 @@ package object config {
.booleanConf
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20532#discussion_r166805138
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -53,10 +53,21 @@ package object config {
.booleanConf
301 - 400 of 2761 matches
Mail list logo
