[jira] [Resolved] (SPARK-7410) Add option to avoid broadcasting configuration with newAPIHadoopFile

[ https://issues.apache.org/jira/browse/SPARK-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-7410. --- Resolution: Won't Fix > Add option to avoid broadcasting configuration with newAPIHadoopFile > ---

[jira] [Resolved] (SPARK-3682) Add helpful warnings to the UI

[ https://issues.apache.org/jira/browse/SPARK-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-3682. --- Resolution: Won't Fix > Add helpful warnings to the UI > -- > >

[jira] [Updated] (SPARK-5490) KMeans costs can be incorrect if tasks need to be rerun

[ https://issues.apache.org/jira/browse/SPARK-5490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-5490: -- Assignee: (was: Sandy Ryza) > KMeans costs can be incorrect if tasks need to be rerun >

[jira] [Commented] (SPARK-9999) Dataset API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022634#comment-15022634 ] Sandy Ryza commented on SPARK-: --- [~nchammas] it's not clear that it makes sense to a

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14980717#comment-14980717 ] Sandy Ryza commented on SPARK-2089: --- My opinion is that we should be moving towards dyna

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14979186#comment-14979186 ] Sandy Ryza commented on SPARK-2089: --- Dynamic allocation may not currently be used for ba

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961513#comment-14961513 ] Sandy Ryza commented on SPARK-: --- So ClassTags would work for case classes and Avro s

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958022#comment-14958022 ] Sandy Ryza commented on SPARK-: --- bq. The problem with doing this using a registry (l

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957144#comment-14957144 ] Sandy Ryza commented on SPARK-: --- Maybe you all have thought through this as well, bu

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14956341#comment-14956341 ] Sandy Ryza commented on SPARK-: --- Thanks for the explanation [~rxin] and [~marmbrus].

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955840#comment-14955840 ] Sandy Ryza commented on SPARK-: --- If I understand correctly, it seems like there are

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955320#comment-14955320 ] Sandy Ryza commented on SPARK-: --- [~rxin] where are the places where the API would ne

[jira] [Commented] (SPARK-9999) RDD-like API on top of Catalyst/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955286#comment-14955286 ] Sandy Ryza commented on SPARK-: --- To ask the obvious question: what are the reasons t

[jira] [Commented] (SPARK-10739) Add attempt window for long running Spark application on Yarn

[ https://issues.apache.org/jira/browse/SPARK-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902982#comment-14902982 ] Sandy Ryza commented on SPARK-10739: That's the one I was referring to as well. That

[jira] [Commented] (SPARK-10739) Add attempt window for long running Spark application on Yarn

[ https://issues.apache.org/jira/browse/SPARK-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901901#comment-14901901 ] Sandy Ryza commented on SPARK-10739: I recall there was a JIRA similar to this that a

[jira] [Resolved] (SPARK-4534) With YARN, JavaSparkContext provide to add preferredNodeLocalityData to SparkContext

[ https://issues.apache.org/jira/browse/SPARK-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-4534. --- Resolution: Won't Fix As SPARK-2089 is closed as "Won't Fix", also closing this. > With YARN, JavaSpa

[jira] [Resolved] (SPARK-9782) Add support for YARN application tags running Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-9782. --- Resolution: Fixed Fix Version/s: 1.6.0 > Add support for YARN application tags running Spark on

[jira] [Updated] (SPARK-9782) Add support for YARN application tags running Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-9782: -- Assignee: Dennis Huo > Add support for YARN application tags running Spark on YARN > ---

[jira] [Updated] (SPARK-8674) [WIP] 2-sample, 2-sided Kolmogorov Smirnov Test Implementation

[ https://issues.apache.org/jira/browse/SPARK-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8674: -- Assignee: Jose Cambronero > [WIP] 2-sample, 2-sided Kolmogorov Smirnov Test Implementation > ---

[jira] [Updated] (SPARK-8674) 2-sample, 2-sided Kolmogorov Smirnov Test

[ https://issues.apache.org/jira/browse/SPARK-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8674: -- Summary: 2-sample, 2-sided Kolmogorov Smirnov Test (was: [WIP] 2-sample, 2-sided Kolmogorov Smirnov Tes

[jira] [Assigned] (SPARK-7707) User guide and example code for KernelDensity

[ https://issues.apache.org/jira/browse/SPARK-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned SPARK-7707: - Assignee: Sandy Ryza > User guide and example code for KernelDensity > --

[jira] [Updated] (SPARK-7707) User guide and example code for KernelDensity

[ https://issues.apache.org/jira/browse/SPARK-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-7707: -- Summary: User guide and example code for KernelDensity (was: User guide and example code for Statistics

[jira] [Commented] (SPARK-7707) User guide and example code for Statistics.kernelDensity

[ https://issues.apache.org/jira/browse/SPARK-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698634#comment-14698634 ] Sandy Ryza commented on SPARK-7707: --- [~mengxr] thoughts on which page this should land i

[jira] [Commented] (SPARK-7707) User guide and example code for Statistics.kernelDensity

[ https://issues.apache.org/jira/browse/SPARK-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694413#comment-14694413 ] Sandy Ryza commented on SPARK-7707: --- Again, sorry for the long delay here. I'm travelin

[jira] [Commented] (SPARK-9808) Remove hash shuffle file consolidation

[ https://issues.apache.org/jira/browse/SPARK-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14692589#comment-14692589 ] Sandy Ryza commented on SPARK-9808: --- I don't have strong opinions here, but as a data po

[jira] [Commented] (SPARK-9808) Remove hash shuffle file consolidation

[ https://issues.apache.org/jira/browse/SPARK-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681601#comment-14681601 ] Sandy Ryza commented on SPARK-9808: --- Have we considered removing the hash-based shuffle

[jira] [Resolved] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-4352. --- Resolution: Fixed Fix Version/s: 1.5.0 Target Version/s: 1.5.0 > Incorporate locality

[jira] [Resolved] (SPARK-1744) Document how to pass in preferredNodeLocationData

[ https://issues.apache.org/jira/browse/SPARK-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-1744. --- Resolution: Won't Fix > Document how to pass in preferredNodeLocationData > --

[jira] [Commented] (SPARK-9092) Make --num-executors compatible with dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635942#comment-14635942 ] Sandy Ryza commented on SPARK-9092: --- I had a brief discussion with [~andrewor14] about t

[jira] [Resolved] (SPARK-1640) In yarn-client mode, pass preferred node locations to AM

[ https://issues.apache.org/jira/browse/SPARK-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-1640. --- Resolution: Invalid > In yarn-client mode, pass preferred node locations to AM > -

[jira] [Updated] (SPARK-8623) Hadoop RDDs fail to properly serialize configuration

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8623: -- Summary: Hadoop RDDs fail to properly serialize configuration (was: Some queries in spark-sql lead to N

[jira] [Assigned] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned SPARK-8623: - Assignee: Sandy Ryza > Some queries in spark-sql lead to NullPointerException when using Yarn > -

[jira] [Updated] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8623: -- Component/s: (was: SQL) Spark Core > Some queries in spark-sql lead to NullPointerE

[jira] [Commented] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14603694#comment-14603694 ] Sandy Ryza commented on SPARK-8623: --- Figured out the issue - my patch omitted registerin

[jira] [Commented] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14603243#comment-14603243 ] Sandy Ryza commented on SPARK-8623: --- Am able to reproduce this locally. Looking into th

[jira] [Commented] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601644#comment-14601644 ] Sandy Ryza commented on SPARK-8623: --- I took a look at the line numbers and it seems like

[jira] [Commented] (SPARK-8623) Some queries in spark-sql lead to NullPointerException when using Yarn

[ https://issues.apache.org/jira/browse/SPARK-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601593#comment-14601593 ] Sandy Ryza commented on SPARK-8623: --- Looking into it > Some queries in spark-sql lead t

[jira] [Commented] (SPARK-7173) Support YARN node label expressions for the application master

[ https://issues.apache.org/jira/browse/SPARK-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576565#comment-14576565 ] Sandy Ryza commented on SPARK-7173: --- This requires additional work on top of SPARK-6470

[jira] [Updated] (SPARK-8136) AM link download test can be flaky

[ https://issues.apache.org/jira/browse/SPARK-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8136: -- Assignee: Hari Shreedharan > AM link download test can be flaky > -- > >

[jira] [Commented] (SPARK-8135) Don't load defaults when reconstituting Hadoop Configurations

[ https://issues.apache.org/jira/browse/SPARK-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575463#comment-14575463 ] Sandy Ryza commented on SPARK-8135: --- Cool. Updated the PR with SerializableConfiguratio

[jira] [Updated] (SPARK-8135) Don't load defaults when reconstituting Hadoop Configurations

[ https://issues.apache.org/jira/browse/SPARK-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8135: -- Summary: Don't load defaults when reconstituting Hadoop Configurations (was: In SerializableWritable, d

[jira] [Commented] (SPARK-8135) In SerializableWritable, don't load defaults when instantiating Configuration

[ https://issues.apache.org/jira/browse/SPARK-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575410#comment-14575410 ] Sandy Ryza commented on SPARK-8135: --- Your question made me think about the fact that, wh

[jira] [Commented] (SPARK-8135) In SerializableWritable, don't load defaults when instantiating Configuration

[ https://issues.apache.org/jira/browse/SPARK-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575397#comment-14575397 ] Sandy Ryza commented on SPARK-8135: --- CC [~joshrosen] > In SerializableWritable, don't l

[jira] [Created] (SPARK-8135) In SerializableWritable, don't load defaults when instantiating Configuration

Sandy Ryza created SPARK-8135: - Summary: In SerializableWritable, don't load defaults when instantiating Configuration Key: SPARK-8135 URL: https://issues.apache.org/jira/browse/SPARK-8135 Project: Spark

[jira] [Resolved] (SPARK-7699) Dynamic allocation: initial executors may be canceled before first job

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-7699. --- Resolution: Fixed Fix Version/s: 1.5.0 > Dynamic allocation: initial executors may be canceled

[jira] [Updated] (SPARK-8099) In yarn-cluster mode, "--executor-cores" can't be setted into SparkConf

[ https://issues.apache.org/jira/browse/SPARK-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8099: -- Assignee: meiyoula > In yarn-cluster mode, "--executor-cores" can't be setted into SparkConf > -

[jira] [Updated] (SPARK-8099) In yarn-cluster mode, "--executor-cores" can't be setted into SparkConf

[ https://issues.apache.org/jira/browse/SPARK-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-8099: -- Assignee: (was: meiyoula) > In yarn-cluster mode, "--executor-cores" can't be setted into SparkConf

[jira] [Resolved] (SPARK-8099) In yarn-cluster mode, "--executor-cores" can't be setted into SparkConf

[ https://issues.apache.org/jira/browse/SPARK-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-8099. --- Resolution: Fixed Fix Version/s: 1.5.0 Assignee: meiyoula > In yarn-cluster mode, "--e

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572356#comment-14572356 ] Sandy Ryza commented on SPARK-4352: --- Right, but once we have placed 5 executors on those

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572338#comment-14572338 ] Sandy Ryza commented on SPARK-4352: --- In your example, what would be the advantage of req

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572245#comment-14572245 ] Sandy Ryza commented on SPARK-4352: --- I don't think it's abnormal. Consider joining the

[jira] [Commented] (SPARK-8062) NullPointerException in SparkHadoopUtil.getFileSystemThreadStatistics

[ https://issues.apache.org/jira/browse/SPARK-8062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570388#comment-14570388 ] Sandy Ryza commented on SPARK-8062: --- [~joshrosen] nothing sticks out to me past what you

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569623#comment-14569623 ] Sandy Ryza commented on SPARK-4352: --- In the case where the task number <= executor numbe

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568580#comment-14568580 ] Sandy Ryza commented on SPARK-4352: --- [~jerryshao] I think you're right that in the case

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567755#comment-14567755 ] Sandy Ryza commented on SPARK-4352: --- [~jerryshao] I wouldn't say that the goal is necess

[jira] [Commented] (SPARK-7707) User guide and example code for Statistics.kernelDensity

[ https://issues.apache.org/jira/browse/SPARK-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565638#comment-14565638 ] Sandy Ryza commented on SPARK-7707: --- Sorry for the delayed response here. I will try to

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561656#comment-14561656 ] Sandy Ryza commented on SPARK-4352: --- I have a couple concerns about that approach. The

[jira] [Issue Comment Deleted] (SPARK-7896) IndexOutOfBoundsException in ChainedBuffer

[ https://issues.apache.org/jira/browse/SPARK-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-7896: -- Comment: was deleted (was: ChainedBuffer splits data into smaller buffers. The default size for these

[jira] [Commented] (SPARK-7896) IndexOutOfBoundsException in ChainedBuffer

[ https://issues.apache.org/jira/browse/SPARK-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561517#comment-14561517 ] Sandy Ryza commented on SPARK-7896: --- ChainedBuffer splits data into smaller buffers. Th

[jira] [Commented] (SPARK-7896) IndexOutOfBoundsException in ChainedBuffer

[ https://issues.apache.org/jira/browse/SPARK-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561516#comment-14561516 ] Sandy Ryza commented on SPARK-7896: --- ChainedBuffer splits data into smaller buffers. Th

[jira] [Commented] (SPARK-7896) IndexOutOfBoundsException in ChainedBuffer

[ https://issues.apache.org/jira/browse/SPARK-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561347#comment-14561347 ] Sandy Ryza commented on SPARK-7896: --- This must be because we're overflowing the 2 GB lim

[jira] [Commented] (SPARK-7896) IndexOutOfBoundsException in ChainedBuffer

[ https://issues.apache.org/jira/browse/SPARK-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561311#comment-14561311 ] Sandy Ryza commented on SPARK-7896: --- [~joshrosen] I'll take a look > IndexOutOfBoundsEx

[jira] [Commented] (SPARK-7699) Number of executors can be reduced from initial before work is scheduled

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560227#comment-14560227 ] Sandy Ryza commented on SPARK-7699: --- I think tying this the AM-RM heartbeat would just m

[jira] [Updated] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-4352: -- Assignee: Saisai Shao > Incorporate locality preferences in dynamic allocation requests > --

[jira] [Commented] (SPARK-7699) Number of executors can be reduced from initial before work is scheduled

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558757#comment-14558757 ] Sandy Ryza commented on SPARK-7699: --- I think delaying releasing them is exactly the poin

[jira] [Commented] (SPARK-7699) Number of executors can be reduced from initial before work is scheduled

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558748#comment-14558748 ] Sandy Ryza commented on SPARK-7699: --- We can't wait only on the initial allocation being

[jira] [Comment Edited] (SPARK-7699) Config "spark.dynamicAllocation.initialExecutors" has no effect

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14557911#comment-14557911 ] Sandy Ryza edited comment on SPARK-7699 at 5/25/15 1:26 AM: [~

[jira] [Commented] (SPARK-7699) Config "spark.dynamicAllocation.initialExecutors" has no effect

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14557911#comment-14557911 ] Sandy Ryza commented on SPARK-7699: --- [~sowen] I think the possible flaw in your argument

[jira] [Commented] (SPARK-7699) Config "spark.dynamicAllocation.initialExecutors" has no effect

[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555837#comment-14555837 ] Sandy Ryza commented on SPARK-7699: --- Sorry for the delay here. The desired behavior is

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14551898#comment-14551898 ] Sandy Ryza commented on SPARK-4352: --- I don't think we should kill executors in order to

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14551677#comment-14551677 ] Sandy Ryza commented on SPARK-4352: --- Thanks for posting this Saisai. Can you export and

[jira] [Commented] (SPARK-5888) Add OneHotEncoder as a Transformer

[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541524#comment-14541524 ] Sandy Ryza commented on SPARK-5888: --- [~mengxr] that makes sense to me, but does that add

[jira] [Commented] (SPARK-7579) User guide update for OneHotEncoder

[ https://issues.apache.org/jira/browse/SPARK-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541517#comment-14541517 ] Sandy Ryza commented on SPARK-7579: --- Ah. I was actually referring to the examples in th

[jira] [Commented] (SPARK-5888) Add OneHotEncoder as a Transformer

[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541367#comment-14541367 ] Sandy Ryza commented on SPARK-5888: --- Right, but while the values are unknown at first, t

[jira] [Commented] (SPARK-7579) User guide update for OneHotEncoder

[ https://issues.apache.org/jira/browse/SPARK-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541356#comment-14541356 ] Sandy Ryza commented on SPARK-7579: --- I can take this up. Any thoughts on how it should

[jira] [Commented] (SPARK-5888) Add OneHotEncoder as a Transformer

[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541351#comment-14541351 ] Sandy Ryza commented on SPARK-5888: --- The values of the nominal output attribute should b

[jira] [Commented] (SPARK-5888) Add OneHotEncoder as a Transformer

[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541339#comment-14541339 ] Sandy Ryza commented on SPARK-5888: --- Hi [~hvanhovell], I agree that this should work. [

[jira] [Commented] (SPARK-7410) Add option to avoid broadcasting configuration with newAPIHadoopFile

[ https://issues.apache.org/jira/browse/SPARK-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14538784#comment-14538784 ] Sandy Ryza commented on SPARK-7410: --- Thanks for the pointer, [~joshrosen]. Looked over

[jira] [Resolved] (SPARK-7515) Update documentation for PySpark on YARN with cluster mode

[ https://issues.apache.org/jira/browse/SPARK-7515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-7515. --- Resolution: Fixed Fix Version/s: 1.5.0 Target Version/s: (was: 1.4.0) > Update docu

[jira] [Updated] (SPARK-7515) Update documentation for PySpark on YARN with cluster mode

[ https://issues.apache.org/jira/browse/SPARK-7515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-7515: -- Assignee: Kousuke Saruta > Update documentation for PySpark on YARN with cluster mode >

[jira] [Resolved] (SPARK-6470) Allow Spark apps to put YARN node labels in their requests

[ https://issues.apache.org/jira/browse/SPARK-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-6470. --- Resolution: Fixed Fix Version/s: 1.5.0 Target Version/s: (was: 1.4.0) > Allow Spark

[jira] [Created] (SPARK-7533) Decrease spacing between AM-RM heartbeats.

Sandy Ryza created SPARK-7533: - Summary: Decrease spacing between AM-RM heartbeats. Key: SPARK-7533 URL: https://issues.apache.org/jira/browse/SPARK-7533 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-7410) Add option to avoid broadcasting configuration with newAPIHadoopFile

Sandy Ryza created SPARK-7410: - Summary: Add option to avoid broadcasting configuration with newAPIHadoopFile Key: SPARK-7410 URL: https://issues.apache.org/jira/browse/SPARK-7410 Project: Spark

[jira] [Commented] (SPARK-5581) When writing sorted map output file, avoid open / close between each partition

[ https://issues.apache.org/jira/browse/SPARK-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527636#comment-14527636 ] Sandy Ryza commented on SPARK-5581: --- [~joshrosen] agree with all of that. > When writin

[jira] [Created] (SPARK-7311) Enable in-memory serialized map-side shuffle to work with SQL serializers

Sandy Ryza created SPARK-7311: - Summary: Enable in-memory serialized map-side shuffle to work with SQL serializers Key: SPARK-7311 URL: https://issues.apache.org/jira/browse/SPARK-7311 Project: Spark

[jira] [Resolved] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-4550. --- Resolution: Fixed Fix Version/s: 1.4.0 > In sort-based shuffle, store map outputs in serialized

[jira] [Commented] (SPARK-3655) Support sorting of values in addition to keys (i.e. secondary sort)

[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518286#comment-14518286 ] Sandy Ryza commented on SPARK-3655: --- My opinion is that a secondary sort operator in cor

[jira] [Created] (SPARK-7173) Support YARN node label expressions for the application master

Sandy Ryza created SPARK-7173: - Summary: Support YARN node label expressions for the application master Key: SPARK-7173 URL: https://issues.apache.org/jira/browse/SPARK-7173 Project: Spark Issue

[jira] [Updated] (SPARK-6954) ExecutorAllocationManager can end up requesting a negative number of executors

[ https://issues.apache.org/jira/browse/SPARK-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-6954: -- Summary: ExecutorAllocationManager can end up requesting a negative number of executors (was: Dynamic a

[jira] [Resolved] (SPARK-6891) ExecutorAllocationManager will request negative number executors

[ https://issues.apache.org/jira/browse/SPARK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-6891. --- Resolution: Duplicate > ExecutorAllocationManager will request negative number executors > ---

[jira] [Commented] (SPARK-6891) ExecutorAllocationManager will request negative number executors

[ https://issues.apache.org/jira/browse/SPARK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510550#comment-14510550 ] Sandy Ryza commented on SPARK-6891: --- This looks like a duplicate of SPARK-6954. While t

[jira] [Commented] (SPARK-6954) Dynamic allocation: numExecutorsPending in ExecutorAllocationManager should never become negative

[ https://issues.apache.org/jira/browse/SPARK-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497465#comment-14497465 ] Sandy Ryza commented on SPARK-6954: --- Hi [~cheolsoo], are you running with a version of S

[jira] [Assigned] (SPARK-5888) Add OneHotEncoder as a Transformer

[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned SPARK-5888: - Assignee: Sandy Ryza > Add OneHotEncoder as a Transformer > -- >

[jira] [Commented] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14487670#comment-14487670 ] Sandy Ryza commented on SPARK-6735: --- Hi [~twinkle], can you submit the PR against the ma

[jira] [Comment Edited] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14487670#comment-14487670 ] Sandy Ryza edited comment on SPARK-6735 at 4/9/15 5:06 PM: --- Hi [

[jira] [Comment Edited] (SPARK-6700) flaky test: run Python application in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14395135#comment-14395135 ] Sandy Ryza edited comment on SPARK-6700 at 4/3/15 9:54 PM: --- Does

[jira] [Commented] (SPARK-6700) flaky test: run Python application in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14395135#comment-14395135 ] Sandy Ryza commented on SPARK-6700: --- Does this fail often? > flaky test: run Python app

[jira] [Commented] (SPARK-6646) Spark 2.0: Rearchitecting Spark for Mobile Platforms

[ https://issues.apache.org/jira/browse/SPARK-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14390196#comment-14390196 ] Sandy Ryza commented on SPARK-6646: --- [~srowen] I like the way you think. I know a lot o

[jira] [Commented] (SPARK-6646) Spark 2.0: Rearchitecting Spark for Mobile Platforms

[ https://issues.apache.org/jira/browse/SPARK-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14390153#comment-14390153 ] Sandy Ryza commented on SPARK-6646: --- This seems like a good opportunity to finally add a

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383280#comment-14383280 ] Sandy Ryza commented on SPARK-4550: --- Java serialization appears to write out the full cl

  1   2   3   4   5   6   >