[jira] [Resolved] (SPARK-3682) Add helpful warnings to the UI

[ https://issues.apache.org/jira/browse/SPARK-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-3682. --- Resolution: Won't Fix > Add helpful warnings to the UI > -- > >

[jira] [Updated] (SPARK-5490) KMeans costs can be incorrect if tasks need to be rerun

[ https://issues.apache.org/jira/browse/SPARK-5490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-5490: -- Assignee: (was: Sandy Ryza) > KMeans costs can be incorrect if tasks need to be rerun >

[jira] [Resolved] (SPARK-7410) Add option to avoid broadcasting configuration with newAPIHadoopFile

[ https://issues.apache.org/jira/browse/SPARK-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-7410. --- Resolution: Won't Fix > Add option to avoid broadcasting configuration with newAPIHadoopFile >

[jira] [Commented] (SPARK-9999) Dataset API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15022634#comment-15022634 ] Sandy Ryza commented on SPARK-: --- [~nchammas] it's not clear that it makes sense to add a similar API

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980717#comment-14980717 ] Sandy Ryza commented on SPARK-2089: --- My opinion is that we should be moving towards dynamic allocation

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961513#comment-14961513 ] Sandy Ryza commented on SPARK-: --- So ClassTags would work for case classes and Avro specific records,

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957144#comment-14957144 ] Sandy Ryza commented on SPARK-: --- Maybe you all have thought through this as well, but I had some

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956341#comment-14956341 ] Sandy Ryza commented on SPARK-: --- Thanks for the explanation [~rxin] and [~marmbrus]. I understand

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958022#comment-14958022 ] Sandy Ryza commented on SPARK-: --- bq. The problem with doing this using a registry (like kryo in RDDs

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955286#comment-14955286 ] Sandy Ryza commented on SPARK-: --- To ask the obvious question: what are the reasons that the RDD API

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955320#comment-14955320 ] Sandy Ryza commented on SPARK-: --- [~rxin] where are the places where the API would need to break? >

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955840#comment-14955840 ] Sandy Ryza commented on SPARK-: --- If I understand correctly, it seems like there are ways to work

[jira] [Commented] (SPARK-10739) Add attempt window for long running Spark application on Yarn

[ https://issues.apache.org/jira/browse/SPARK-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902982#comment-14902982 ] Sandy Ryza commented on SPARK-10739: That's the one I was referring to as well. That's about

[jira] [Commented] (SPARK-10739) Add attempt window for long running Spark application on Yarn

[ https://issues.apache.org/jira/browse/SPARK-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901901#comment-14901901 ] Sandy Ryza commented on SPARK-10739: I recall there was a JIRA similar to this that avoided killing

[jira] [Resolved] (SPARK-4534) With YARN, JavaSparkContext provide to add preferredNodeLocalityData to SparkContext

[ https://issues.apache.org/jira/browse/SPARK-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-4534. --- Resolution: Won't Fix As SPARK-2089 is closed as "Won't Fix", also closing this. > With YARN,

[jira] [Updated] (SPARK-9782) Add support for YARN application tags running Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-9782: -- Assignee: Dennis Huo Add support for YARN application tags running Spark on YARN

[jira] [Resolved] (SPARK-9782) Add support for YARN application tags running Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-9782. --- Resolution: Fixed Fix Version/s: 1.6.0 Add support for YARN application tags running Spark on

[jira] [Updated] (SPARK-8674) 2-sample, 2-sided Kolmogorov Smirnov Test

[ https://issues.apache.org/jira/browse/SPARK-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8674: -- Summary: 2-sample, 2-sided Kolmogorov Smirnov Test (was: [WIP] 2-sample, 2-sided Kolmogorov Smirnov

[jira] [Updated] (SPARK-8674) [WIP] 2-sample, 2-sided Kolmogorov Smirnov Test Implementation

[ https://issues.apache.org/jira/browse/SPARK-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8674: -- Assignee: Jose Cambronero [WIP] 2-sample, 2-sided Kolmogorov Smirnov Test Implementation

[jira] [Commented] (SPARK-7707) User guide and example code for Statistics.kernelDensity

[ https://issues.apache.org/jira/browse/SPARK-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698634#comment-14698634 ] Sandy Ryza commented on SPARK-7707: --- [~mengxr] thoughts on which page this should land

[jira] [Updated] (SPARK-7707) User guide and example code for KernelDensity

[ https://issues.apache.org/jira/browse/SPARK-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-7707: -- Summary: User guide and example code for KernelDensity (was: User guide and example code for

[jira] [Assigned] (SPARK-7707) User guide and example code for KernelDensity

[ https://issues.apache.org/jira/browse/SPARK-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned SPARK-7707: - Assignee: Sandy Ryza User guide and example code for KernelDensity

[jira] [Commented] (SPARK-7707) User guide and example code for Statistics.kernelDensity

[ https://issues.apache.org/jira/browse/SPARK-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694413#comment-14694413 ] Sandy Ryza commented on SPARK-7707: --- Again, sorry for the long delay here. I'm

[jira] [Commented] (SPARK-9808) Remove hash shuffle file consolidation

[ https://issues.apache.org/jira/browse/SPARK-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681601#comment-14681601 ] Sandy Ryza commented on SPARK-9808: --- Have we considered removing the hash-based shuffle

[jira] [Commented] (SPARK-9808) Remove hash shuffle file consolidation

[ https://issues.apache.org/jira/browse/SPARK-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14692589#comment-14692589 ] Sandy Ryza commented on SPARK-9808: --- I don't have strong opinions here, but as a data

[jira] [Resolved] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-4352. --- Resolution: Fixed Fix Version/s: 1.5.0 Target Version/s: 1.5.0 Incorporate locality

[jira] [Resolved] (SPARK-1744) Document how to pass in preferredNodeLocationData

[ https://issues.apache.org/jira/browse/SPARK-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-1744. --- Resolution: Won't Fix Document how to pass in preferredNodeLocationData

[jira] [Commented] (SPARK-9092) Make --num-executors compatible with dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14635942#comment-14635942 ] Sandy Ryza commented on SPARK-9092: --- I had a brief discussion with [~andrewor14] about

[jira] [Resolved] (SPARK-1640) In yarn-client mode, pass preferred node locations to AM

[ https://issues.apache.org/jira/browse/SPARK-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-1640. --- Resolution: Invalid In yarn-client mode, pass preferred node locations to AM

[jira] [Commented] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603243#comment-14603243 ] Sandy Ryza commented on SPARK-8623: --- Am able to reproduce this locally. Looking into

[jira] [Commented] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603694#comment-14603694 ] Sandy Ryza commented on SPARK-8623: --- Figured out the issue - my patch omitted

[jira] [Updated] (SPARK-8623) Hadoop RDDs fail to properly serialize configuration

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8623: -- Summary: Hadoop RDDs fail to properly serialize configuration (was: Some queries in spark-sql lead to

[jira] [Assigned] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned SPARK-8623: - Assignee: Sandy Ryza Some queries in spark-sql lead to NullPointerException when using Yarn

[jira] [Updated] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8623: -- Component/s: (was: SQL) Spark Core Some queries in spark-sql lead to

[jira] [Commented] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14601593#comment-14601593 ] Sandy Ryza commented on SPARK-8623: --- Looking into it Some queries in spark-sql lead to

[jira] [Commented] (SPARK-7173) Support YARN node label expressions for the application master

[ https://issues.apache.org/jira/browse/SPARK-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576565#comment-14576565 ] Sandy Ryza commented on SPARK-7173: --- This requires additional work on top of SPARK-6470

[jira] [Updated] (SPARK-8099) In yarn-cluster mode, --executor-cores can't be setted into SparkConf

[ https://issues.apache.org/jira/browse/SPARK-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8099: -- Assignee: meiyoula In yarn-cluster mode, --executor-cores can't be setted into SparkConf

[jira] [Updated] (SPARK-8099) In yarn-cluster mode, --executor-cores can't be setted into SparkConf

[ https://issues.apache.org/jira/browse/SPARK-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8099: -- Assignee: (was: meiyoula) In yarn-cluster mode, --executor-cores can't be setted into SparkConf

[jira] [Resolved] (SPARK-8099) In yarn-cluster mode, --executor-cores can't be setted into SparkConf

[ https://issues.apache.org/jira/browse/SPARK-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-8099. --- Resolution: Fixed Fix Version/s: 1.5.0 Assignee: meiyoula In yarn-cluster mode,

[jira] [Resolved] (SPARK-7699) Dynamic allocation: initial executors may be canceled before first job

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-7699. --- Resolution: Fixed Fix Version/s: 1.5.0 Dynamic allocation: initial executors may be canceled

[jira] [Updated] (SPARK-8135) Don't load defaults when reconstituting Hadoop Configurations

[ https://issues.apache.org/jira/browse/SPARK-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8135: -- Summary: Don't load defaults when reconstituting Hadoop Configurations (was: In SerializableWritable,

[jira] [Commented] (SPARK-8135) In SerializableWritable, don't load defaults when instantiating Configuration

[ https://issues.apache.org/jira/browse/SPARK-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575397#comment-14575397 ] Sandy Ryza commented on SPARK-8135: --- CC [~joshrosen] In SerializableWritable, don't

[jira] [Created] (SPARK-8135) In SerializableWritable, don't load defaults when instantiating Configuration

Sandy Ryza created SPARK-8135: - Summary: In SerializableWritable, don't load defaults when instantiating Configuration Key: SPARK-8135 URL: https://issues.apache.org/jira/browse/SPARK-8135 Project: Spark

[jira] [Commented] (SPARK-8135) In SerializableWritable, don't load defaults when instantiating Configuration

[ https://issues.apache.org/jira/browse/SPARK-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575410#comment-14575410 ] Sandy Ryza commented on SPARK-8135: --- Your question made me think about the fact that,

[jira] [Commented] (SPARK-8135) Don't load defaults when reconstituting Hadoop Configurations

[ https://issues.apache.org/jira/browse/SPARK-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575463#comment-14575463 ] Sandy Ryza commented on SPARK-8135: --- Cool. Updated the PR with

[jira] [Updated] (SPARK-8136) AM link download test can be flaky

[ https://issues.apache.org/jira/browse/SPARK-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8136: -- Assignee: Hari Shreedharan AM link download test can be flaky --

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572245#comment-14572245 ] Sandy Ryza commented on SPARK-4352: --- I don't think it's abnormal. Consider joining the

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572356#comment-14572356 ] Sandy Ryza commented on SPARK-4352: --- Right, but once we have placed 5 executors on those

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572338#comment-14572338 ] Sandy Ryza commented on SPARK-4352: --- In your example, what would be the advantage of

[jira] [Commented] (SPARK-8062) NullPointerException in SparkHadoopUtil.getFileSystemThreadStatistics

[ https://issues.apache.org/jira/browse/SPARK-8062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570388#comment-14570388 ] Sandy Ryza commented on SPARK-8062: --- [~joshrosen] nothing sticks out to me past what

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569623#comment-14569623 ] Sandy Ryza commented on SPARK-4352: --- In the case where the task number = executor number

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14567755#comment-14567755 ] Sandy Ryza commented on SPARK-4352: --- [~jerryshao] I wouldn't say that the goal is

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568580#comment-14568580 ] Sandy Ryza commented on SPARK-4352: --- [~jerryshao] I think you're right that in the case

[jira] [Commented] (SPARK-7707) User guide and example code for Statistics.kernelDensity

[ https://issues.apache.org/jira/browse/SPARK-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565638#comment-14565638 ] Sandy Ryza commented on SPARK-7707: --- Sorry for the delayed response here. I will try to

[jira] [Commented] (SPARK-7896) IndexOutOfBoundsException in ChainedBuffer

[ https://issues.apache.org/jira/browse/SPARK-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561311#comment-14561311 ] Sandy Ryza commented on SPARK-7896: --- [~joshrosen] I'll take a look

[jira] [Commented] (SPARK-7896) IndexOutOfBoundsException in ChainedBuffer

[ https://issues.apache.org/jira/browse/SPARK-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561347#comment-14561347 ] Sandy Ryza commented on SPARK-7896: --- This must be because we're overflowing the 2 GB

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561656#comment-14561656 ] Sandy Ryza commented on SPARK-4352: --- I have a couple concerns about that approach. The

[jira] [Commented] (SPARK-7699) Number of executors can be reduced from initial before work is scheduled

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14560227#comment-14560227 ] Sandy Ryza commented on SPARK-7699: --- I think tying this the AM-RM heartbeat would just

[jira] [Commented] (SPARK-7699) Number of executors can be reduced from initial before work is scheduled

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558748#comment-14558748 ] Sandy Ryza commented on SPARK-7699: --- We can't wait only on the initial allocation being

[jira] [Commented] (SPARK-7699) Number of executors can be reduced from initial before work is scheduled

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558757#comment-14558757 ] Sandy Ryza commented on SPARK-7699: --- I think delaying releasing them is exactly the

[jira] [Updated] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-4352: -- Assignee: Saisai Shao Incorporate locality preferences in dynamic allocation requests

[jira] [Comment Edited] (SPARK-7699) Config spark.dynamicAllocation.initialExecutors has no effect

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557911#comment-14557911 ] Sandy Ryza edited comment on SPARK-7699 at 5/25/15 1:26 AM:

[jira] [Commented] (SPARK-7699) Config spark.dynamicAllocation.initialExecutors has no effect

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557911#comment-14557911 ] Sandy Ryza commented on SPARK-7699: --- [~sowen] I think the possible flaw in your argument

[jira] [Commented] (SPARK-7699) Config spark.dynamicAllocation.initialExecutors has no effect

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14555837#comment-14555837 ] Sandy Ryza commented on SPARK-7699: --- Sorry for the delay here. The desired behavior is

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551898#comment-14551898 ] Sandy Ryza commented on SPARK-4352: --- I don't think we should kill executors in order to

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551677#comment-14551677 ] Sandy Ryza commented on SPARK-4352: --- Thanks for posting this Saisai. Can you export and

[jira] [Commented] (SPARK-7579) User guide update for OneHotEncoder

[ https://issues.apache.org/jira/browse/SPARK-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541517#comment-14541517 ] Sandy Ryza commented on SPARK-7579: --- Ah. I was actually referring to the examples in

[jira] [Commented] (SPARK-5888) Add OneHotEncoder as a Transformer

[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541524#comment-14541524 ] Sandy Ryza commented on SPARK-5888: --- [~mengxr] that makes sense to me, but does that

[jira] [Commented] (SPARK-5888) Add OneHotEncoder as a Transformer

[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541339#comment-14541339 ] Sandy Ryza commented on SPARK-5888: --- Hi [~hvanhovell], I agree that this should work.

[jira] [Commented] (SPARK-5888) Add OneHotEncoder as a Transformer

[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541367#comment-14541367 ] Sandy Ryza commented on SPARK-5888: --- Right, but while the values are unknown at first,

[jira] [Commented] (SPARK-7579) User guide update for OneHotEncoder

[ https://issues.apache.org/jira/browse/SPARK-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541356#comment-14541356 ] Sandy Ryza commented on SPARK-7579: --- I can take this up. Any thoughts on how it should

[jira] [Commented] (SPARK-5888) Add OneHotEncoder as a Transformer

[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541351#comment-14541351 ] Sandy Ryza commented on SPARK-5888: --- The values of the nominal output attribute should

[jira] [Resolved] (SPARK-7515) Update documentation for PySpark on YARN with cluster mode

[ https://issues.apache.org/jira/browse/SPARK-7515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-7515. --- Resolution: Fixed Fix Version/s: 1.5.0 Target Version/s: (was: 1.4.0) Update

[jira] [Updated] (SPARK-7515) Update documentation for PySpark on YARN with cluster mode

[ https://issues.apache.org/jira/browse/SPARK-7515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-7515: -- Assignee: Kousuke Saruta Update documentation for PySpark on YARN with cluster mode

[jira] [Commented] (SPARK-7410) Add option to avoid broadcasting configuration with newAPIHadoopFile

[ https://issues.apache.org/jira/browse/SPARK-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538784#comment-14538784 ] Sandy Ryza commented on SPARK-7410: --- Thanks for the pointer, [~joshrosen]. Looked over

[jira] [Created] (SPARK-7533) Decrease spacing between AM-RM heartbeats.

Sandy Ryza created SPARK-7533: - Summary: Decrease spacing between AM-RM heartbeats. Key: SPARK-7533 URL: https://issues.apache.org/jira/browse/SPARK-7533 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-7410) Add option to avoid broadcasting configuration with newAPIHadoopFile

Sandy Ryza created SPARK-7410: - Summary: Add option to avoid broadcasting configuration with newAPIHadoopFile Key: SPARK-7410 URL: https://issues.apache.org/jira/browse/SPARK-7410 Project: Spark

[jira] [Resolved] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-4550. --- Resolution: Fixed Fix Version/s: 1.4.0 In sort-based shuffle, store map outputs in serialized

[jira] [Created] (SPARK-7311) Enable in-memory serialized map-side shuffle to work with SQL serializers

Sandy Ryza created SPARK-7311: - Summary: Enable in-memory serialized map-side shuffle to work with SQL serializers Key: SPARK-7311 URL: https://issues.apache.org/jira/browse/SPARK-7311 Project: Spark

[jira] [Commented] (SPARK-3655) Support sorting of values in addition to keys (i.e. secondary sort)

[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518286#comment-14518286 ] Sandy Ryza commented on SPARK-3655: --- My opinion is that a secondary sort operator in

[jira] [Created] (SPARK-7173) Support YARN node label expressions for the application master

Sandy Ryza created SPARK-7173: - Summary: Support YARN node label expressions for the application master Key: SPARK-7173 URL: https://issues.apache.org/jira/browse/SPARK-7173 Project: Spark

[jira] [Updated] (SPARK-6954) ExecutorAllocationManager can end up requesting a negative number of executors

[ https://issues.apache.org/jira/browse/SPARK-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-6954: -- Summary: ExecutorAllocationManager can end up requesting a negative number of executors (was: Dynamic

[jira] [Resolved] (SPARK-6891) ExecutorAllocationManager will request negative number executors

[ https://issues.apache.org/jira/browse/SPARK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-6891. --- Resolution: Duplicate ExecutorAllocationManager will request negative number executors

[jira] [Commented] (SPARK-6891) ExecutorAllocationManager will request negative number executors

[ https://issues.apache.org/jira/browse/SPARK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510550#comment-14510550 ] Sandy Ryza commented on SPARK-6891: --- This looks like a duplicate of SPARK-6954. While

[jira] [Commented] (SPARK-6954) Dynamic allocation: numExecutorsPending in ExecutorAllocationManager should never become negative

[ https://issues.apache.org/jira/browse/SPARK-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497465#comment-14497465 ] Sandy Ryza commented on SPARK-6954: --- Hi [~cheolsoo], are you running with a version of

[jira] [Assigned] (SPARK-5888) Add OneHotEncoder as a Transformer

[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned SPARK-5888: - Assignee: Sandy Ryza Add OneHotEncoder as a Transformer --

[jira] [Commented] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14487670#comment-14487670 ] Sandy Ryza commented on SPARK-6735: --- Hi [~twinkle], can you submit the PR against the

[jira] [Comment Edited] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14487670#comment-14487670 ] Sandy Ryza edited comment on SPARK-6735 at 4/9/15 5:06 PM: --- Hi

[jira] [Comment Edited] (SPARK-6700) flaky test: run Python application in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395135#comment-14395135 ] Sandy Ryza edited comment on SPARK-6700 at 4/3/15 9:54 PM: --- Does

[jira] [Commented] (SPARK-6700) flaky test: run Python application in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395135#comment-14395135 ] Sandy Ryza commented on SPARK-6700: --- Does this fail often? flaky test: run Python

[jira] [Commented] (SPARK-6646) Spark 2.0: Rearchitecting Spark for Mobile Platforms

[ https://issues.apache.org/jira/browse/SPARK-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390153#comment-14390153 ] Sandy Ryza commented on SPARK-6646: --- This seems like a good opportunity to finally add a

[jira] [Commented] (SPARK-6646) Spark 2.0: Rearchitecting Spark for Mobile Platforms

[ https://issues.apache.org/jira/browse/SPARK-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390196#comment-14390196 ] Sandy Ryza commented on SPARK-6646: --- [~srowen] I like the way you think. I know a lot

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383280#comment-14383280 ] Sandy Ryza commented on SPARK-4550: --- Java serialization appears to write out the full

[jira] [Commented] (SPARK-6479) Create off-heap block storage API (internal)

[ https://issues.apache.org/jira/browse/SPARK-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378011#comment-14378011 ] Sandy Ryza commented on SPARK-6479: --- I believe he means wrapping Spark's call-outs to

[jira] [Assigned] (SPARK-6470) Allow Spark apps to put YARN node labels in their requests

[ https://issues.apache.org/jira/browse/SPARK-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned SPARK-6470: - Assignee: Sandy Ryza Allow Spark apps to put YARN node labels in their requests

[jira] [Commented] (SPARK-6418) Add simple per-stage visualization to the UI

[ https://issues.apache.org/jira/browse/SPARK-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372144#comment-14372144 ] Sandy Ryza commented on SPARK-6418: --- I think this would be a great addition. One note

[jira] [Commented] (SPARK-6418) Add simple per-stage visualization to the UI

[ https://issues.apache.org/jira/browse/SPARK-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372178#comment-14372178 ] Sandy Ryza commented on SPARK-6418: --- Yeah, all of that is wishful thinking, definitely

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372232#comment-14372232 ] Sandy Ryza commented on SPARK-4550: --- I spoke briefly with Reynold about this offline,

[jira] [Updated] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-4550: -- Attachment: kryo-flush-benchmark.scala In sort-based shuffle, store map outputs in serialized form

[jira] [Created] (SPARK-6393) Extra RPC to the AM during killExecutor invocation

Sandy Ryza created SPARK-6393: - Summary: Extra RPC to the AM during killExecutor invocation Key: SPARK-6393 URL: https://issues.apache.org/jira/browse/SPARK-6393 Project: Spark Issue Type:

  1   2   3   4   5   >