[jira] [Assigned] (SPARK-28209) Shuffle Storage API: Writes

[ https://issues.apache.org/jira/browse/SPARK-28209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28209: -- Assignee: Matt Cheah > Shuffle Storage API: Wri

[jira] [Resolved] (SPARK-28525) Allow Launcher to be applied Java options

[ https://issues.apache.org/jira/browse/SPARK-28525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28525. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25265 [https

[jira] [Assigned] (SPARK-28042) Support mapping spark.local.dir to hostPath volume

[ https://issues.apache.org/jira/browse/SPARK-28042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28042: -- Assignee: Junjie Chen > Support mapping spark.local.dir to hostPath vol

[jira] [Resolved] (SPARK-28042) Support mapping spark.local.dir to hostPath volume

[ https://issues.apache.org/jira/browse/SPARK-28042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28042. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24879 [https

[jira] [Created] (SPARK-28535) Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"

Marcelo Vanzin created SPARK-28535: -- Summary: Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader" Key: SPARK-28535 URL: https://issues.apache.org/jira/browse/S

[jira] [Assigned] (SPARK-25285) Add executor task metrics to track the number of tasks started and of tasks successfully completed

[ https://issues.apache.org/jira/browse/SPARK-25285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25285: -- Assignee: Luca Canali > Add executor task metrics to track the number of ta

[jira] [Resolved] (SPARK-25285) Add executor task metrics to track the number of tasks started and of tasks successfully completed

[ https://issues.apache.org/jira/browse/SPARK-25285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25285. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22290 [https

[jira] [Resolved] (SPARK-28465) K8s integration tests fail due to missing ceph-nano image

[ https://issues.apache.org/jira/browse/SPARK-28465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28465. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25222 [https

[jira] [Assigned] (SPARK-28465) K8s integration tests fail due to missing ceph-nano image

[ https://issues.apache.org/jira/browse/SPARK-28465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28465: -- Assignee: Stavros Kontopoulos > K8s integration tests fail due to missing ceph-n

[jira] [Resolved] (SPARK-28496) Use branch name instead of tag during dry-run

[ https://issues.apache.org/jira/browse/SPARK-28496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28496. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25240 [https

[jira] [Assigned] (SPARK-28496) Use branch name instead of tag during dry-run

[ https://issues.apache.org/jira/browse/SPARK-28496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28496: -- Assignee: Dongjoon Hyun > Use branch name instead of tag during dry-

[jira] [Commented] (SPARK-28509) K8S integration tests are failing

[ https://issues.apache.org/jira/browse/SPARK-28509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16892079#comment-16892079 ] Marcelo Vanzin commented on SPARK-28509: [~shaneknapp] in case this is an infra issue. >

[jira] [Created] (SPARK-28509) K8S integration tests are failing

Marcelo Vanzin created SPARK-28509: -- Summary: K8S integration tests are failing Key: SPARK-28509 URL: https://issues.apache.org/jira/browse/SPARK-28509 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-25590) kubernetes-model-2.0.0.jar masks default Spark logging config

[ https://issues.apache.org/jira/browse/SPARK-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25590. Resolution: Duplicate > kubernetes-model-2.0.0.jar masks default Spark logging con

[jira] [Created] (SPARK-28488) Race in k8s scheduler shutdown can lead to misleading exceptions.

Marcelo Vanzin created SPARK-28488: -- Summary: Race in k8s scheduler shutdown can lead to misleading exceptions. Key: SPARK-28488 URL: https://issues.apache.org/jira/browse/SPARK-28488 Project: Spark

[jira] [Created] (SPARK-28487) K8S pod allocator behaves poorly with dynamic allocation

Marcelo Vanzin created SPARK-28487: -- Summary: K8S pod allocator behaves poorly with dynamic allocation Key: SPARK-28487 URL: https://issues.apache.org/jira/browse/SPARK-28487 Project: Spark

[jira] [Created] (SPARK-28455) Executor may be timed out too soon because of overflow in tracking code

Marcelo Vanzin created SPARK-28455: -- Summary: Executor may be timed out too soon because of overflow in tracking code Key: SPARK-28455 URL: https://issues.apache.org/jira/browse/SPARK-28455 Project

[jira] [Resolved] (SPARK-28417) Spark Submit does not use Proxy User Credentials to Resolve Path for Resources

[ https://issues.apache.org/jira/browse/SPARK-28417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28417. Resolution: Duplicate > Spark Submit does not use Proxy User Credentials to Resolve P

[jira] [Resolved] (SPARK-27963) Allow dynamic allocation without an external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27963. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24817 [https

[jira] [Assigned] (SPARK-27963) Allow dynamic allocation without an external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27963: -- Assignee: Marcelo Vanzin > Allow dynamic allocation without an external shuf

[jira] [Resolved] (SPARK-27959) Change YARN resource configs to use .amount

[ https://issues.apache.org/jira/browse/SPARK-27959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27959. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24989 [https

[jira] [Assigned] (SPARK-27959) Change YARN resource configs to use .amount

[ https://issues.apache.org/jira/browse/SPARK-27959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27959: -- Assignee: Thomas Graves > Change YARN resource configs to use .amo

[jira] [Resolved] (SPARK-28407) Support mapping spark.local.dir to hostPath volume

[ https://issues.apache.org/jira/browse/SPARK-28407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28407. Resolution: Duplicate This had already been cloned elsewhere. > Support mapp

[jira] [Commented] (SPARK-27499) Support mapping spark.local.dir to hostPath volume

[ https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885578#comment-16885578 ] Marcelo Vanzin commented on SPARK-27499: I can't see an option to reopen this, so I'll clone

[jira] [Created] (SPARK-28407) Support mapping spark.local.dir to hostPath volume

Marcelo Vanzin created SPARK-28407: -- Summary: Support mapping spark.local.dir to hostPath volume Key: SPARK-28407 URL: https://issues.apache.org/jira/browse/SPARK-28407 Project: Spark Issue

[jira] [Updated] (SPARK-28371) Parquet "starts with" filter is not null-safe

[ https://issues.apache.org/jira/browse/SPARK-28371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-28371: --- Description: I ran into this when running unit tests with Parquet 1.11. It seems that 1.10

[jira] [Created] (SPARK-28371) Parquet "starts with" filter is not null-safe

Marcelo Vanzin created SPARK-28371: -- Summary: Parquet "starts with" filter is not null-safe Key: SPARK-28371 URL: https://issues.apache.org/jira/browse/SPARK-28371 Project: Spark

[jira] [Assigned] (SPARK-23472) Add config properties for administrator JVM options

[ https://issues.apache.org/jira/browse/SPARK-23472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23472: -- Assignee: Gabor Somogyi > Add config properties for administrator JVM opti

[jira] [Resolved] (SPARK-23472) Add config properties for administrator JVM options

[ https://issues.apache.org/jira/browse/SPARK-23472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23472. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24804 [https

[jira] [Resolved] (SPARK-28055) Add delegation token custom AdminClient configurations.

[ https://issues.apache.org/jira/browse/SPARK-28055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28055. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24875 [https

[jira] [Assigned] (SPARK-28055) Add delegation token custom AdminClient configurations.

[ https://issues.apache.org/jira/browse/SPARK-28055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28055: -- Assignee: Gabor Somogyi > Add delegation token custom AdminClient configurati

[jira] [Created] (SPARK-28214) Flaky test: org.apache.spark.streaming.CheckpointSuite.basic rdd checkpoints + dstream graph checkpoint recovery

Marcelo Vanzin created SPARK-28214: -- Summary: Flaky test: org.apache.spark.streaming.CheckpointSuite.basic rdd checkpoints + dstream graph checkpoint recovery Key: SPARK-28214 URL: https://issues.apache.org

[jira] [Deleted] (SPARK-28207) https://rtatdotblog.wordpress.com/2019/05/30/rohit-travels-tours-rohit

[ https://issues.apache.org/jira/browse/SPARK-28207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin deleted SPARK-28207: --- > https://rtatdotblog.wordpress.com/2019/05/30/rohit-travels-tours-ro

[jira] [Assigned] (SPARK-28187) Add hadoop-cloud module to PR builders

[ https://issues.apache.org/jira/browse/SPARK-28187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28187: -- Assignee: Marcelo Vanzin > Add hadoop-cloud module to PR build

[jira] [Resolved] (SPARK-28187) Add hadoop-cloud module to PR builders

[ https://issues.apache.org/jira/browse/SPARK-28187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28187. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24987 [https

[jira] [Resolved] (SPARK-28150) Failure to create multiple contexts in same JVM with Kerberos auth

[ https://issues.apache.org/jira/browse/SPARK-28150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28150. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24955 [https

[jira] [Assigned] (SPARK-28150) Failure to create multiple contexts in same JVM with Kerberos auth

[ https://issues.apache.org/jira/browse/SPARK-28150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28150: -- Assignee: Marcelo Vanzin > Failure to create multiple contexts in same

[jira] [Created] (SPARK-28187) Add hadoop-cloud module to PR builders

Marcelo Vanzin created SPARK-28187: -- Summary: Add hadoop-cloud module to PR builders Key: SPARK-28187 URL: https://issues.apache.org/jira/browse/SPARK-28187 Project: Spark Issue Type

[jira] [Resolved] (SPARK-27622) Avoid the network when block manager fetches disk persisted RDD blocks from the same host

[ https://issues.apache.org/jira/browse/SPARK-27622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27622. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24554 [https

[jira] [Assigned] (SPARK-27622) Avoid the network when block manager fetches disk persisted RDD blocks from the same host

[ https://issues.apache.org/jira/browse/SPARK-27622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27622: -- Assignee: Attila Zsolt Piros > Avoid the network when block manager fetches d

[jira] [Assigned] (SPARK-24432) Add support for dynamic resource allocation

[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24432: -- Assignee: (was: Marcelo Vanzin) > Add support for dynamic resource allocat

[jira] [Assigned] (SPARK-24432) Add support for dynamic resource allocation

[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24432: -- Assignee: Marcelo Vanzin > Add support for dynamic resource allocat

[jira] [Updated] (SPARK-28150) Failure to create multiple contexts in same JVM with Kerberos auth

[ https://issues.apache.org/jira/browse/SPARK-28150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-28150: --- Description: Take the following small app that creates multiple contexts (not concurrently

[jira] [Created] (SPARK-28150) Failure to create multiple contexts in same JVM with Kerberos auth

Marcelo Vanzin created SPARK-28150: -- Summary: Failure to create multiple contexts in same JVM with Kerberos auth Key: SPARK-28150 URL: https://issues.apache.org/jira/browse/SPARK-28150 Project

[jira] [Commented] (SPARK-27963) Allow dynamic allocation without an external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857146#comment-16857146 ] Marcelo Vanzin commented on SPARK-27963: FYI I have a WIP patch to implement this that I plan

[jira] [Created] (SPARK-27963) Allow dynamic allocation without an external shuffle service

Marcelo Vanzin created SPARK-27963: -- Summary: Allow dynamic allocation without an external shuffle service Key: SPARK-27963 URL: https://issues.apache.org/jira/browse/SPARK-27963 Project: Spark

[jira] [Resolved] (SPARK-27748) Kafka consumer/producer password/token redaction

[ https://issues.apache.org/jira/browse/SPARK-27748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27748. Resolution: Fixed Assignee: Gabor Somogyi Fix Version/s: 3.0.0 > Ka

[jira] [Commented] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852420#comment-16852420 ] Marcelo Vanzin commented on SPARK-27891: Ok, the updated logs show the issue. But they're from

[jira] [Commented] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852408#comment-16852408 ] Marcelo Vanzin commented on SPARK-27891: {{container_e48_1559242207407_0001_02_01}} tells me

[jira] [Resolved] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27891. Resolution: Not A Problem You have to provide a keytab for Spark for this to work

[jira] [Resolved] (SPARK-27773) Add shuffle service metric for number of exceptions caught in ExternalShuffleBlockHandler

[ https://issues.apache.org/jira/browse/SPARK-27773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27773. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24645 [https

[jira] [Assigned] (SPARK-27773) Add shuffle service metric for number of exceptions caught in ExternalShuffleBlockHandler

[ https://issues.apache.org/jira/browse/SPARK-27773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27773: -- Assignee: Steven Rand > Add shuffle service metric for number of exceptions cau

[jira] [Resolved] (SPARK-27378) spark-submit requests GPUs in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-27378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27378. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24634 [https

[jira] [Assigned] (SPARK-27378) spark-submit requests GPUs in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-27378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27378: -- Assignee: Thomas Graves > spark-submit requests GPUs in YARN m

[jira] [Created] (SPARK-27868) Better document shuffle / RPC listen backlog

Marcelo Vanzin created SPARK-27868: -- Summary: Better document shuffle / RPC listen backlog Key: SPARK-27868 URL: https://issues.apache.org/jira/browse/SPARK-27868 Project: Spark Issue Type

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847926#comment-16847926 ] Marcelo Vanzin commented on SPARK-24149: bq. If they are unrelated the user explicitly provides

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847888#comment-16847888 ] Marcelo Vanzin commented on SPARK-24149: bq. I think we are duplicating the logic here Do you

[jira] [Assigned] (SPARK-27677) Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation

[ https://issues.apache.org/jira/browse/SPARK-27677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27677: -- Assignee: Attila Zsolt Piros > Disk-persisted RDD blocks served by shuffle serv

[jira] [Resolved] (SPARK-27677) Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation

[ https://issues.apache.org/jira/browse/SPARK-27677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27677. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24499 [https

[jira] [Resolved] (SPARK-27804) LiveListenerBus#addToQueue : create multiple AsyncEventQueues under race condition

[ https://issues.apache.org/jira/browse/SPARK-27804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27804. Resolution: Not A Problem > LiveListenerBus#addToQueue : create multiple AsyncEventQue

[jira] [Commented] (SPARK-27797) Shuffle service metric "registeredConnections" not tracked correctly

[ https://issues.apache.org/jira/browse/SPARK-27797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846068#comment-16846068 ] Marcelo Vanzin commented on SPARK-27797: You're right. Looks just like some dead code

[jira] [Created] (SPARK-27797) Shuffle service metric "registeredConnections" not tracked correctly

Marcelo Vanzin created SPARK-27797: -- Summary: Shuffle service metric "registeredConnections" not tracked correctly Key: SPARK-27797 URL: https://issues.apache.org/jira/browse/SPARK-27797

[jira] [Updated] (SPARK-27726) Performance of InMemoryStore suffers under load

[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-27726: --- Fix Version/s: 2.4.4 > Performance of InMemoryStore suffers under l

[jira] [Commented] (SPARK-27726) Performance of InMemoryStore suffers under load

[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845061#comment-16845061 ] Marcelo Vanzin commented on SPARK-27726: [~davidnavas] all of the sub-tasks were handled

[jira] [Assigned] (SPARK-27726) Performance of InMemoryStore suffers under load

[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27726: -- Assignee: David C Navas > Performance of InMemoryStore suffers under l

[jira] [Resolved] (SPARK-27726) Performance of InMemoryStore suffers under load

[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27726. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24616 [https

[jira] [Resolved] (SPARK-27745) build/mvn take wrong scala version when compile for scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-27745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27745. Resolution: Not A Bug You need to run {{./dev/change-scala-version.sh}} first. Pretty

[jira] [Assigned] (SPARK-27678) Support Knox user impersonation in UI

[ https://issues.apache.org/jira/browse/SPARK-27678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27678: -- Assignee: Marcelo Vanzin > Support Knox user impersonation in

[jira] [Resolved] (SPARK-27678) Support Knox user impersonation in UI

[ https://issues.apache.org/jira/browse/SPARK-27678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27678. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24582 [https

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839674#comment-16839674 ] Marcelo Vanzin commented on SPARK-27681: bq. 2043 in method parameters, 1488 in method return

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838799#comment-16838799 ] Marcelo Vanzin commented on SPARK-27681: bq. The case to consider is, roughly, where Seq

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838766#comment-16838766 ] Marcelo Vanzin commented on SPARK-27681: bq. But, user code that passes a non-immutable Seq

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838715#comment-16838715 ] Marcelo Vanzin commented on SPARK-27681: bq. varargs methods in Scala will change

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838693#comment-16838693 ] Marcelo Vanzin commented on SPARK-27681: bq. scala.Seq is just a type def

[jira] [Comment Edited] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837752#comment-16837752 ] Marcelo Vanzin edited comment on SPARK-27681 at 5/11/19 4:28 AM: - Also

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837753#comment-16837753 ] Marcelo Vanzin commented on SPARK-27681: bq. we don't need to do this until we support Scala

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837752#comment-16837752 ] Marcelo Vanzin commented on SPARK-27681: Also the change you're proposing likely would break

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837749#comment-16837749 ] Marcelo Vanzin commented on SPARK-27681: OOC, what happens if we do nothing? As far as I can

[jira] [Created] (SPARK-27678) Support Knox user impersonation in UI

Marcelo Vanzin created SPARK-27678: -- Summary: Support Knox user impersonation in UI Key: SPARK-27678 URL: https://issues.apache.org/jira/browse/SPARK-27678 Project: Spark Issue Type: New

[jira] [Resolved] (SPARK-26632) Separate Thread Configurations of Driver and Executor

[ https://issues.apache.org/jira/browse/SPARK-26632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26632. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23560 [https

[jira] [Assigned] (SPARK-26632) Separate Thread Configurations of Driver and Executor

[ https://issues.apache.org/jira/browse/SPARK-26632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26632: -- Assignee: jiafu zhang > Separate Thread Configurations of Driver and Execu

[jira] [Assigned] (SPARK-27294) Multi-cluster Kafka delegation token support

[ https://issues.apache.org/jira/browse/SPARK-27294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27294: -- Assignee: Gabor Somogyi > Multi-cluster Kafka delegation token supp

[jira] [Resolved] (SPARK-27294) Multi-cluster Kafka delegation token support

[ https://issues.apache.org/jira/browse/SPARK-27294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27294. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24305 [https

[jira] [Resolved] (SPARK-27610) Yarn external shuffle service fails to start when spark.shuffle.io.mode=EPOLL

[ https://issues.apache.org/jira/browse/SPARK-27610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27610. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24502 [https

[jira] [Assigned] (SPARK-27610) Yarn external shuffle service fails to start when spark.shuffle.io.mode=EPOLL

[ https://issues.apache.org/jira/browse/SPARK-27610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27610: -- Assignee: Adrian Muraru > Yarn external shuffle service fails to start w

[jira] [Assigned] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

[ https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27194: -- Assignee: (was: Marcelo Vanzin) > Job failures when task attempts do not cl

[jira] [Assigned] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

[ https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27194: -- Assignee: Marcelo Vanzin > Job failures when task attempts do not clean up sp

[jira] [Resolved] (SPARK-16367) Wheelhouse Support for PySpark

[ https://issues.apache.org/jira/browse/SPARK-16367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-16367. Resolution: Duplicate This is somewhat similar to SPARK-13587 so let's keep

[jira] [Assigned] (SPARK-13587) Support virtualenv in PySpark

[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-13587: -- Assignee: Marcelo Vanzin > Support virtualenv in PySp

[jira] [Updated] (SPARK-13587) Support virtualenv in PySpark

[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-13587: --- Target Version/s: (was: 3.0.0) > Support virtualenv in PySp

[jira] [Assigned] (SPARK-13587) Support virtualenv in PySpark

[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-13587: -- Assignee: (was: Marcelo Vanzin) > Support virtualenv in PySp

[jira] [Resolved] (SPARK-27575) Spark overwrites existing value of spark.yarn.dist.* instead of merging value

[ https://issues.apache.org/jira/browse/SPARK-27575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27575. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24465 [https

[jira] [Assigned] (SPARK-27575) Spark overwrites existing value of spark.yarn.dist.* instead of merging value

[ https://issues.apache.org/jira/browse/SPARK-27575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27575: -- Assignee: Jungtaek Lim > Spark overwrites existing value of spark.yarn.d

[jira] [Resolved] (SPARK-23014) Migrate MemorySink fully to v2

[ https://issues.apache.org/jira/browse/SPARK-23014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23014. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24403 [https

[jira] [Assigned] (SPARK-23014) Migrate MemorySink fully to v2

[ https://issues.apache.org/jira/browse/SPARK-23014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23014: -- Assignee: Gabor Somogyi > Migrate MemorySink fully to

[jira] [Assigned] (SPARK-27477) Kafka token provider should have provided dependency on Spark

[ https://issues.apache.org/jira/browse/SPARK-27477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27477: -- Assignee: koert kuipers > Kafka token provider should have provided depende

[jira] [Resolved] (SPARK-27477) Kafka token provider should have provided dependency on Spark

[ https://issues.apache.org/jira/browse/SPARK-27477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27477. Resolution: Fixed Issue resolved by pull request 24384 [https://github.com/apache/spark

Re: Is it possible to obtain the full command to be invoked by SparkLauncher?

BTW the SparkLauncher API has hooks to capture the stderr of the spark-submit process into the logging system of the parent process. Check the API javadocs since it's been forever since I looked at that. On Wed, Apr 24, 2019 at 1:58 PM Marcelo Vanzin wrote: > > S

Re: Is it possible to obtain the full command to be invoked by SparkLauncher?

Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the launcher env will make Spark code print the command to stderr. Not optimal but I think it's the only current option. On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans wrote: > > The org.apache.spark.launcher.SparkLauncher is used to

[jira] [Resolved] (SPARK-27515) [Deploy] When application master retry after a long time running, the hdfs delegation token may be expired

[ https://issues.apache.org/jira/browse/SPARK-27515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27515. Resolution: Duplicate > [Deploy] When application master retry after a long time runn

<    1   2   3   4   5   6   7   8   9   10   >