[jira] [Commented] (SPARK-3543) Write TaskContext in Java and expose it through a static accessor

[ https://issues.apache.org/jira/browse/SPARK-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135016#comment-14135016 ] Chengxiang Li commented on SPARK-3543: -- I think this would solve SPARK-2895 as well,

[jira] [Commented] (SPARK-3535) Spark on Mesos not correctly setting heap overhead

[ https://issues.apache.org/jira/browse/SPARK-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135073#comment-14135073 ] Tomas Barton commented on SPARK-3535: - I'm having the same issue in course grained

[jira] [Commented] (SPARK-2445) MesosExecutorBackend crashes in fine grained mode

[ https://issues.apache.org/jira/browse/SPARK-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135075#comment-14135075 ] Tomas Barton commented on SPARK-2445: - Good catch! Seems like Spark memory allocation

[jira] [Created] (SPARK-3544) SparkSQL thriftServer cannot release locks correctly in Zookeeper

Patrick Liu created SPARK-3544: -- Summary: SparkSQL thriftServer cannot release locks correctly in Zookeeper Key: SPARK-3544 URL: https://issues.apache.org/jira/browse/SPARK-3544 Project: Spark

[jira] [Updated] (SPARK-3544) SparkSQL thriftServer cannot release locks correctly in Zookeeper

[ https://issues.apache.org/jira/browse/SPARK-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Liu updated SPARK-3544: --- Component/s: (was: Spark Core) SQL Description: Bug description: The

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135116#comment-14135116 ] Hari Shreedharan commented on SPARK-3129: - It looks like Akka makes it difficult

[jira] [Commented] (SPARK-1405) parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib

[ https://issues.apache.org/jira/browse/SPARK-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135134#comment-14135134 ] Guoqiang Li commented on SPARK-1405: Here are some related papers: [Towards Topic

[jira] [Created] (SPARK-3545) Put HadoopRDD.getPartitions forward and put TaskScheduler.start back in SparkContext to reduce DAGScheduler.JobSubmitted processing time and shorten cluster resources occ

YanTang Zhai created SPARK-3545: --- Summary: Put HadoopRDD.getPartitions forward and put TaskScheduler.start back in SparkContext to reduce DAGScheduler.JobSubmitted processing time and shorten cluster resources occupation period

[jira] [Updated] (SPARK-3545) Put HadoopRDD.getPartitions forward and put TaskScheduler.start back in SparkContext to reduce DAGScheduler.JobSubmitted processing time and shorten cluster resources occ

[ https://issues.apache.org/jira/browse/SPARK-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YanTang Zhai updated SPARK-3545: Description: We have two problems: (1) HadoopRDD.getPartitions is lazyied to process in

[jira] [Updated] (SPARK-3545) Put HadoopRDD.getPartitions forward and put TaskScheduler.start back in SparkContext to reduce DAGScheduler.JobSubmitted processing time and shorten cluster resources occ

[ https://issues.apache.org/jira/browse/SPARK-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YanTang Zhai updated SPARK-3545: Description: We have two problems: (1) HadoopRDD.getPartitions is lazyied to process in

[jira] [Updated] (SPARK-3545) Put HadoopRDD.getPartitions forward and put TaskScheduler.start back in SparkContext to reduce DAGScheduler.JobSubmitted processing time and shorten cluster resources occ

[ https://issues.apache.org/jira/browse/SPARK-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YanTang Zhai updated SPARK-3545: Description: We have two problems: (1) HadoopRDD.getPartitions is lazyied to process in

[jira] [Updated] (SPARK-3545) Put HadoopRDD.getPartitions forward and put TaskScheduler.start back in SparkContext to reduce DAGScheduler.JobSubmitted processing time and shorten cluster resources occ

[ https://issues.apache.org/jira/browse/SPARK-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YanTang Zhai updated SPARK-3545: Description: We have two problems: (1) HadoopRDD.getPartitions is lazyied to process in

[jira] [Updated] (SPARK-3545) Put HadoopRDD.getPartitions forward and put TaskScheduler.start back in SparkContext to reduce DAGScheduler.JobSubmitted processing time and shorten cluster resources occ

[ https://issues.apache.org/jira/browse/SPARK-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YanTang Zhai updated SPARK-3545: Description: We have two problems: (1) HadoopRDD.getPartitions is lazyied to process in

[jira] [Updated] (SPARK-3545) Put HadoopRDD.getPartitions forward and put TaskScheduler.start back in SparkContext to reduce DAGScheduler.JobSubmitted processing time and shorten cluster resources occ

[ https://issues.apache.org/jira/browse/SPARK-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YanTang Zhai updated SPARK-3545: Description: We have two problems: (1) HadoopRDD.getPartitions is lazyied to process in

[jira] [Commented] (SPARK-3485) should check parameter type when find constructors

[ https://issues.apache.org/jira/browse/SPARK-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135228#comment-14135228 ] Apache Spark commented on SPARK-3485: - User 'adrian-wang' has created a pull request

[jira] [Commented] (SPARK-2594) Add CACHE TABLE name AS SELECT ...

[ https://issues.apache.org/jira/browse/SPARK-2594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135303#comment-14135303 ] Ravindra Pesala commented on SPARK-2594: [~marmbrus] There is a confusion over

[jira] [Created] (SPARK-3546) InputStream of ManagedBuffer does not close and causes running out of file descriptor

Kousuke Saruta created SPARK-3546: - Summary: InputStream of ManagedBuffer does not close and causes running out of file descriptor Key: SPARK-3546 URL: https://issues.apache.org/jira/browse/SPARK-3546

[jira] [Commented] (SPARK-3546) InputStream of ManagedBuffer does not close and causes running out of file descriptor

[ https://issues.apache.org/jira/browse/SPARK-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135358#comment-14135358 ] Apache Spark commented on SPARK-3546: - User 'sarutak' has created a pull request for

[jira] [Updated] (SPARK-3546) InputStream of ManagedBuffer is not closed and causes running out of file descriptor

[ https://issues.apache.org/jira/browse/SPARK-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3546: -- Summary: InputStream of ManagedBuffer is not closed and causes running out of file descriptor

[jira] [Created] (SPARK-3547) Maybe we should not simply make return code 1 equal to CLASS_NOT_FOUND

WangTaoTheTonic created SPARK-3547: -- Summary: Maybe we should not simply make return code 1 equal to CLASS_NOT_FOUND Key: SPARK-3547 URL: https://issues.apache.org/jira/browse/SPARK-3547 Project:

[jira] [Commented] (SPARK-3545) Put HadoopRDD.getPartitions forward and put TaskScheduler.start back in SparkContext to reduce DAGScheduler.JobSubmitted processing time and shorten cluster resources o

[ https://issues.apache.org/jira/browse/SPARK-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135421#comment-14135421 ] Apache Spark commented on SPARK-3545: - User 'YanTangZhai' has created a pull request

[jira] [Created] (SPARK-3549) Yarn client mode does not shutdown clearnly

Chen Avnery created SPARK-3549: -- Summary: Yarn client mode does not shutdown clearnly Key: SPARK-3549 URL: https://issues.apache.org/jira/browse/SPARK-3549 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-3548) Display cache hit ratio on WebUI

[ https://issues.apache.org/jira/browse/SPARK-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135575#comment-14135575 ] Apache Spark commented on SPARK-3548: - User 'sarutak' has created a pull request for

[jira] [Updated] (SPARK-3548) Display cache hit ratio on WebUI

[ https://issues.apache.org/jira/browse/SPARK-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3548: -- Summary: Display cache hit ratio on WebUI (was: Add cache hit ratio to WebUI) Display cache

[jira] [Created] (SPARK-3548) Add cache hit ratio to WebUI

Kousuke Saruta created SPARK-3548: - Summary: Add cache hit ratio to WebUI Key: SPARK-3548 URL: https://issues.apache.org/jira/browse/SPARK-3548 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-2022) Spark 1.0.0 is failing if mesos.coarse set to true

[ https://issues.apache.org/jira/browse/SPARK-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135592#comment-14135592 ] Timothy St. Clair commented on SPARK-2022: -- *this appears to be done, could we

[jira] [Commented] (SPARK-3533) Add saveAsTextFileByKey() method to RDDs

[ https://issues.apache.org/jira/browse/SPARK-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135597#comment-14135597 ] Nicholas Chammas commented on SPARK-3533: - [~kzhang] - I noticed you authored

[jira] [Created] (SPARK-3550) Disable automatic rdd caching in python api for relevant learners

Aaron Staple created SPARK-3550: --- Summary: Disable automatic rdd caching in python api for relevant learners Key: SPARK-3550 URL: https://issues.apache.org/jira/browse/SPARK-3550 Project: Spark

[jira] [Updated] (SPARK-3488) cache deserialized python RDDs before iterative learning

[ https://issues.apache.org/jira/browse/SPARK-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Staple updated SPARK-3488: Component/s: PySpark cache deserialized python RDDs before iterative learning

[jira] [Commented] (SPARK-3488) cache deserialized python RDDs before iterative learning

[ https://issues.apache.org/jira/browse/SPARK-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135613#comment-14135613 ] Aaron Staple commented on SPARK-3488: - After further discussion it's been decided

[jira] [Commented] (SPARK-3550) Disable automatic rdd caching in python api for relevant learners

[ https://issues.apache.org/jira/browse/SPARK-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135610#comment-14135610 ] Apache Spark commented on SPARK-3550: - User 'staple' has created a pull request for

[jira] [Commented] (SPARK-3508) annotate the Spark configs to indicate which ones are meant for the end user

[ https://issues.apache.org/jira/browse/SPARK-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135631#comment-14135631 ] Matthew Farrellee commented on SPARK-3508: -- documented == public is a good

[jira] [Updated] (SPARK-1389) Make numPartitions in Exchange configurable

[ https://issues.apache.org/jira/browse/SPARK-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1389: --- Component/s: SQL Make numPartitions in Exchange configurable

[jira] [Resolved] (SPARK-1069) Provide binary compatibility in Spark 1.X releases

[ https://issues.apache.org/jira/browse/SPARK-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1069. Resolution: Fixed We've implemented MIMA and a bunch of other things to help us do this, so

[jira] [Resolved] (SPARK-1172) Improve naming of the BlockManager classes

[ https://issues.apache.org/jira/browse/SPARK-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1172. Resolution: Won't Fix Okay let's close this until we do a braoder refactoring of the block

[jira] [Commented] (SPARK-2022) Spark 1.0.0 is failing if mesos.coarse set to true

[ https://issues.apache.org/jira/browse/SPARK-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135663#comment-14135663 ] Nicholas Chammas commented on SPARK-2022: - Pinging [~pwendell] about closing this

[jira] [Resolved] (SPARK-3069) Build instructions in README are outdated

[ https://issues.apache.org/jira/browse/SPARK-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3069. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Sean Owen Resolved by:

[jira] [Commented] (SPARK-3551) Remove redundant putting FetchResult which means Fetch Fail when Remote fetching

[ https://issues.apache.org/jira/browse/SPARK-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135666#comment-14135666 ] Apache Spark commented on SPARK-3551: - User 'sarutak' has created a pull request for

[jira] [Resolved] (SPARK-2182) Scalastyle rule blocking unicode operators

[ https://issues.apache.org/jira/browse/SPARK-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2182. Resolution: Fixed Fixed by Prashant in: https://github.com/apache/spark/pull/2358

[jira] [Commented] (SPARK-3542) Akka protocol authentication in plaintext

[ https://issues.apache.org/jira/browse/SPARK-3542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135689#comment-14135689 ] Patrick Wendell commented on SPARK-3542: Hey James. Are you using the standalone

[jira] [Updated] (SPARK-3542) If spark.authenticate.secret is set it's transferred in plain text

[ https://issues.apache.org/jira/browse/SPARK-3542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3542: --- Summary: If spark.authenticate.secret is set it's transferred in plain text (was: Akka

[jira] [Updated] (SPARK-1832) Executor UI improvement suggestions

[ https://issues.apache.org/jira/browse/SPARK-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1832: --- Fix Version/s: (was: 1.2.0) Executor UI improvement suggestions

[jira] [Updated] (SPARK-1832) Executor UI improvement suggestions

[ https://issues.apache.org/jira/browse/SPARK-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1832: --- Target Version/s: 1.2.0 Executor UI improvement suggestions

[jira] [Updated] (SPARK-3546) InputStream of ManagedBuffer is not closed and causes running out of file descriptor

[ https://issues.apache.org/jira/browse/SPARK-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3546: --- Target Version/s: 1.1.1, 1.2.0 (was: 1.2.0) InputStream of ManagedBuffer is not closed and

[jira] [Updated] (SPARK-3546) InputStream of ManagedBuffer is not closed and causes running out of file descriptor

[ https://issues.apache.org/jira/browse/SPARK-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3546: --- Assignee: Kousuke Saruta InputStream of ManagedBuffer is not closed and causes running out

[jira] [Commented] (SPARK-3539) Task description apply at Option.scala:120; no user code involved

[ https://issues.apache.org/jira/browse/SPARK-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135755#comment-14135755 ] Patrick Wendell commented on SPARK-3539: could you describe in more detail what

[jira] [Updated] (SPARK-1201) Do not materialize partitions whenever possible in BlockManager

[ https://issues.apache.org/jira/browse/SPARK-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1201: - Target Version/s: 1.2.0 Fix Version/s: (was: 1.2.0) Do not materialize partitions whenever

[jira] [Updated] (SPARK-1761) Add broadcast information on SparkUI storage tab

[ https://issues.apache.org/jira/browse/SPARK-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1761: - Target Version/s: 1.2.0 Fix Version/s: (was: 1.2.0) Add broadcast information on SparkUI

[jira] [Updated] (SPARK-1762) Add functionality to pin RDDs in cache

[ https://issues.apache.org/jira/browse/SPARK-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1762: - Target Version/s: 1.2.0 Fix Version/s: (was: 1.2.0) Add functionality to pin RDDs in cache

[jira] [Updated] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2984: --- Target Version/s: 1.2.0 FileNotFoundException on _temporary directory

[jira] [Updated] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2984: --- Component/s: Spark Core FileNotFoundException on _temporary directory

[jira] [Updated] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2984: -- Description: We've seen several stacktraces and threads on the user mailing list where people are

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135775#comment-14135775 ] Andrew Ash commented on SPARK-2984: --- [~gphil], S3 consistency varies based on the region

[jira] [Updated] (SPARK-3524) remove workaround to pickle array of float for Pyrolite

[ https://issues.apache.org/jira/browse/SPARK-3524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3524: --- Component/s: PySpark remove workaround to pickle array of float for Pyrolite

[jira] [Updated] (SPARK-1449) Please delete old releases from mirroring system

[ https://issues.apache.org/jira/browse/SPARK-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1449: --- Component/s: Project Infra Please delete old releases from mirroring system

[jira] [Updated] (SPARK-3528) Reading data from file:/// should be called NODE_LOCAL not PROCESS_LOCAL

[ https://issues.apache.org/jira/browse/SPARK-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3528: --- Priority: Critical (was: Major) Reading data from file:/// should be called NODE_LOCAL not

[jira] [Commented] (SPARK-3528) Reading data from file:/// should be called NODE_LOCAL not PROCESS_LOCAL

[ https://issues.apache.org/jira/browse/SPARK-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135781#comment-14135781 ] Andrew Ash commented on SPARK-3528: --- [~nchammas] it does look like S3 also has the

[jira] [Commented] (SPARK-3400) GraphX unit tests fail nondeterministically

[ https://issues.apache.org/jira/browse/SPARK-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135782#comment-14135782 ] Andrew Ash commented on SPARK-3400: --- [~ankurd] is there another ticket to track the root

[jira] [Updated] (SPARK-3542) If spark.authenticate.secret is set it's transferred in plain text

[ https://issues.apache.org/jira/browse/SPARK-3542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3542: --- Component/s: Spark Core If spark.authenticate.secret is set it's transferred in plain text

[jira] [Updated] (SPARK-3546) InputStream of ManagedBuffer is not closed and causes running out of file descriptor

[ https://issues.apache.org/jira/browse/SPARK-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3546: --- Component/s: (was: core) Spark Core InputStream of ManagedBuffer is not

[jira] [Updated] (SPARK-3546) InputStream of ManagedBuffer is not closed and causes running out of file descriptor

[ https://issues.apache.org/jira/browse/SPARK-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3546: --- Component/s: core InputStream of ManagedBuffer is not closed and causes running out of file

[jira] [Resolved] (SPARK-1201) Do not materialize partitions whenever possible in BlockManager

[ https://issues.apache.org/jira/browse/SPARK-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1201. Resolution: Duplicate This was solved by SPARK-1777. Do not materialize partitions

[jira] [Resolved] (SPARK-528) Provide a dist-like target that builds a binary distribution (JARs + scripts)

[ https://issues.apache.org/jira/browse/SPARK-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-528. --- Resolution: Fixed Yeah thanks this was fixed a long time ago. Provide a dist-like target

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135806#comment-14135806 ] Gregory Phillips commented on SPARK-2984: - [~aash] -- Thanks for bringing this to

[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark

[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135835#comment-14135835 ] Patrick Wendell commented on SPARK-2593: Hey [~helena] before you spend a lot of

[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark

[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135840#comment-14135840 ] Patrick Wendell commented on SPARK-2593: One potential alternative would be for

[jira] [Commented] (SPARK-3499) Create Spark-based distcp utility

[ https://issues.apache.org/jira/browse/SPARK-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135842#comment-14135842 ] Patrick Wendell commented on SPARK-3499: Yeah I think writing this in Spark would

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135841#comment-14135841 ] Hari Shreedharan commented on SPARK-3129: - As long as at least one executor

[jira] [Comment Edited] (SPARK-3499) Create Spark-based distcp utility

[ https://issues.apache.org/jira/browse/SPARK-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135842#comment-14135842 ] Patrick Wendell edited comment on SPARK-3499 at 9/16/14 6:03 PM:

[jira] [Commented] (SPARK-3539) Task description apply at Option.scala:120; no user code involved

[ https://issues.apache.org/jira/browse/SPARK-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135856#comment-14135856 ] John Salvatier commented on SPARK-3539: --- Sorry, in the Spark UI, a task appears with

[jira] [Commented] (SPARK-922) Update Spark AMI to Python 2.7

[ https://issues.apache.org/jira/browse/SPARK-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135860#comment-14135860 ] Josh Rosen commented on SPARK-922: -- [~nchammas] In the long run, it might be nice to

[jira] [Resolved] (SPARK-3527) Strip the physical plan message margin

[ https://issues.apache.org/jira/browse/SPARK-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-3527. - Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Cheng Hao Strip the

[jira] [Commented] (SPARK-3490) Alleviate port collisions during tests

[ https://issues.apache.org/jira/browse/SPARK-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135916#comment-14135916 ] Apache Spark commented on SPARK-3490: - User 'andrewor14' has created a pull request

[jira] [Resolved] (SPARK-3519) PySpark RDDs are missing the distinct(n) method

[ https://issues.apache.org/jira/browse/SPARK-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3519. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2383

[jira] [Resolved] (SPARK-3308) Ability to read JSON Arrays as tables

[ https://issues.apache.org/jira/browse/SPARK-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-3308. - Resolution: Fixed Fix Version/s: 1.2.0 Ability to read JSON Arrays as tables

[jira] [Resolved] (SPARK-2890) Spark SQL should allow SELECT with duplicated columns

[ https://issues.apache.org/jira/browse/SPARK-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-2890. - Resolution: Fixed Fix Version/s: 1.2.0 Spark SQL should allow SELECT with

[jira] [Resolved] (SPARK-2314) RDD actions are only overridden in Scala, not java or python

[ https://issues.apache.org/jira/browse/SPARK-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-2314. - Resolution: Fixed Fix Version/s: (was: 1.0.3) RDD actions are only overridden

[jira] [Commented] (SPARK-3223) runAsSparkUser cannot change HDFS write permission properly in mesos cluster mode

[ https://issues.apache.org/jira/browse/SPARK-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135965#comment-14135965 ] Timothy St. Clair commented on SPARK-3223: -- As [~tnachen] mentioned in the PR,

[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark

[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135968#comment-14135968 ] Andy Petrella commented on SPARK-2593: -- Couldn't we can translate the needs here to

[jira] [Commented] (SPARK-2691) Allow Spark on Mesos to be launched with Docker

[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135990#comment-14135990 ] Timothy St. Clair commented on SPARK-2691: -- +1 [~tnachen], I'd be happy to help

[jira] [Commented] (SPARK-1702) Mesos executor won't start because of a ClassNotFoundException

[ https://issues.apache.org/jira/browse/SPARK-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136015#comment-14136015 ] Timothy St. Clair commented on SPARK-1702: -- I don't believe so, there appears to

[jira] [Comment Edited] (SPARK-1702) Mesos executor won't start because of a ClassNotFoundException

[ https://issues.apache.org/jira/browse/SPARK-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136015#comment-14136015 ] Timothy St. Clair edited comment on SPARK-1702 at 9/16/14 7:17 PM:

[jira] [Commented] (SPARK-1807) Modify SPARK_EXECUTOR_URI to allow for script execution in Mesos.

[ https://issues.apache.org/jira/browse/SPARK-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136029#comment-14136029 ] Timothy St. Clair commented on SPARK-1807: -- Please close in favor of SPARK-2691

[jira] [Commented] (SPARK-2761) Merge similar code paths in ExternalSorter and EAOM

[ https://issues.apache.org/jira/browse/SPARK-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136056#comment-14136056 ] Apache Spark commented on SPARK-2761: - User 'jimjh' has created a pull request for

[jira] [Commented] (SPARK-2761) Merge similar code paths in ExternalSorter and EAOM

[ https://issues.apache.org/jira/browse/SPARK-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136061#comment-14136061 ] Jim Lim commented on SPARK-2761: I saw some duplicate code in `#maybeSpill` and `#spill` -

[jira] [Resolved] (SPARK-3546) InputStream of ManagedBuffer is not closed and causes running out of file descriptor

[ https://issues.apache.org/jira/browse/SPARK-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-3546. Resolution: Fixed Fix Version/s: 1.2.0 InputStream of ManagedBuffer is not closed and

[jira] [Commented] (SPARK-2883) Spark Support for ORCFile format

[ https://issues.apache.org/jira/browse/SPARK-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136102#comment-14136102 ] Zhan Zhang commented on SPARK-2883: --- There are several features to be supported. 1st:

[jira] [Commented] (SPARK-3400) GraphX unit tests fail nondeterministically

[ https://issues.apache.org/jira/browse/SPARK-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136109#comment-14136109 ] Ankur Dave commented on SPARK-3400: --- [~aash] There isn't another ticket yet. Would you

[jira] [Created] (SPARK-3552) Thrift server doesn't reset current database for each connection

Cheng Lian created SPARK-3552: - Summary: Thrift server doesn't reset current database for each connection Key: SPARK-3552 URL: https://issues.apache.org/jira/browse/SPARK-3552 Project: Spark

[jira] [Created] (SPARK-3553) Spark Streaming app streams files that have already being streamed in an endless loop

Ezequiel Bella created SPARK-3553: - Summary: Spark Streaming app streams files that have already being streamed in an endless loop Key: SPARK-3553 URL: https://issues.apache.org/jira/browse/SPARK-3553

[jira] [Updated] (SPARK-2761) Merge similar code paths in ExternalSorter and EAOM

[ https://issues.apache.org/jira/browse/SPARK-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2761: - Fix Version/s: (was: 1.2.0) Merge similar code paths in ExternalSorter and EAOM

[jira] [Updated] (SPARK-787) Add EC2 Script Option to Push EC2 Credentials to Spark Nodes

[ https://issues.apache.org/jira/browse/SPARK-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-787: -- Assignee: Dan Osipov (was: Patrick Cogan) Add EC2 Script Option to Push EC2 Credentials to

[jira] [Resolved] (SPARK-787) Add EC2 Script Option to Push EC2 Credentials to Spark Nodes

[ https://issues.apache.org/jira/browse/SPARK-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-787. --- Resolution: Fixed Fix Version/s: 1.2.0 This is fixed in

[jira] [Created] (SPARK-3554) handle large dataset in closure of PySpark

Davies Liu created SPARK-3554: - Summary: handle large dataset in closure of PySpark Key: SPARK-3554 URL: https://issues.apache.org/jira/browse/SPARK-3554 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-3554) handle large dataset in closure of PySpark

[ https://issues.apache.org/jira/browse/SPARK-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136250#comment-14136250 ] Apache Spark commented on SPARK-3554: - User 'davies' has created a pull request for

[jira] [Created] (SPARK-3555) UI port contention suite flakey

Patrick Wendell created SPARK-3555: -- Summary: UI port contention suite flakey Key: SPARK-3555 URL: https://issues.apache.org/jira/browse/SPARK-3555 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-3555) UI port contention suite flakey

[ https://issues.apache.org/jira/browse/SPARK-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136283#comment-14136283 ] Apache Spark commented on SPARK-3555: - User 'andrewor14' has created a pull request

[jira] [Created] (SPARK-3556) Monitoring and debugging improvements (Spark 1.2)

Josh Rosen created SPARK-3556: - Summary: Monitoring and debugging improvements (Spark 1.2) Key: SPARK-3556 URL: https://issues.apache.org/jira/browse/SPARK-3556 Project: Spark Issue Type:

[jira] [Commented] (SPARK-3535) Spark on Mesos not correctly setting heap overhead

[ https://issues.apache.org/jira/browse/SPARK-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136321#comment-14136321 ] Brenden Matthews commented on SPARK-3535: - I've updated the patch to include the

[jira] [Updated] (SPARK-3556) Monitoring and debugging improvements (Spark 1.2)

[ https://issues.apache.org/jira/browse/SPARK-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3556: -- Issue Type: Epic (was: Umbrella) Monitoring and debugging improvements (Spark 1.2)

  1   2   >