[jira] [Created] (SPARK-3472) Option to take top n elements (unsorted)

2014-09-09 Thread Kanwaljit Singh (JIRA)
Kanwaljit Singh created SPARK-3472: -- Summary: Option to take top n elements (unsorted) Key: SPARK-3472 URL: https://issues.apache.org/jira/browse/SPARK-3472 Project: Spark Issue Type: New Fe

[jira] [Closed] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

2014-09-09 Thread Shay Rojansky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shay Rojansky closed SPARK-2972. Resolution: Won't Fix > APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[jira] [Created] (SPARK-3471) Automatic resource manager for SparkContext in Scala?

2014-09-09 Thread Shay Rojansky (JIRA)
Shay Rojansky created SPARK-3471: Summary: Automatic resource manager for SparkContext in Scala? Key: SPARK-3471 URL: https://issues.apache.org/jira/browse/SPARK-3471 Project: Spark Issue Typ

[jira] [Resolved] (SPARK-3395) {SQL] DSL uses incorrect attribute ids after a distinct()

2014-09-09 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-3395. - Resolution: Fixed Fix Version/s: 1.2.0 > {SQL] DSL uses incorrect attribute ids aft

[jira] [Commented] (SPARK-3469) All TaskCompletionListeners should be called even if some of them fail

2014-09-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128149#comment-14128149 ] Apache Spark commented on SPARK-3469: - User 'rxin' has created a pull request for this

[jira] [Created] (SPARK-3470) Have JavaSparkContext implement Closeable/AutoCloseable

2014-09-09 Thread Shay Rojansky (JIRA)
Shay Rojansky created SPARK-3470: Summary: Have JavaSparkContext implement Closeable/AutoCloseable Key: SPARK-3470 URL: https://issues.apache.org/jira/browse/SPARK-3470 Project: Spark Issue T

[jira] [Resolved] (SPARK-3282) It should support multiple receivers at one socketInputDStream

2014-09-09 Thread shenhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shenhong resolved SPARK-3282. - Resolution: Won't Fix > It should support multiple receivers at one socketInputDStream >

[jira] [Updated] (SPARK-3469) All TaskCompletionListeners should be called even if some of them fail

2014-09-09 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3469: --- Component/s: Spark Core > All TaskCompletionListeners should be called even if some of them fail > ---

[jira] [Updated] (SPARK-3469) All TaskCompletionListeners should be called even if some of them fail

2014-09-09 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3469: --- Summary: All TaskCompletionListeners should be called even if some of them fail (was: Make sure TaskC

[jira] [Created] (SPARK-3469) Make sure TaskCompletionListeners are called in presence of failures

2014-09-09 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-3469: -- Summary: Make sure TaskCompletionListeners are called in presence of failures Key: SPARK-3469 URL: https://issues.apache.org/jira/browse/SPARK-3469 Project: Spark

[jira] [Commented] (SPARK-3468) WebUI Timeline-View feature

2014-09-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128109#comment-14128109 ] Apache Spark commented on SPARK-3468: - User 'sarutak' has created a pull request for t

[jira] [Updated] (SPARK-3468) WebUI Timeline-View feature

2014-09-09 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3468: -- Attachment: taskDetails.png > WebUI Timeline-View feature > --- > >

[jira] [Updated] (SPARK-3468) WebUI Timeline-View feature

2014-09-09 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3468: -- Attachment: executors.png > WebUI Timeline-View feature > --- > >

[jira] [Updated] (SPARK-3468) WebUI Timeline-View feature

2014-09-09 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3468: -- Attachment: tasks.png > WebUI Timeline-View feature > --- > >

[jira] [Issue Comment Deleted] (SPARK-3468) WebUI Timeline-View feature

2014-09-09 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3468: -- Comment: was deleted (was: Stage's timeline view.) > WebUI Timeline-View feature >

[jira] [Updated] (SPARK-3468) WebUI Timeline-View feature

2014-09-09 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3468: -- Attachment: stages.png Stage's timeline view. > WebUI Timeline-View feature > -

[jira] [Created] (SPARK-3468) WebUI Timeline-View feature

2014-09-09 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-3468: - Summary: WebUI Timeline-View feature Key: SPARK-3468 URL: https://issues.apache.org/jira/browse/SPARK-3468 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-3160) Simplify DecisionTree data structure for training

2014-09-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128048#comment-14128048 ] Apache Spark commented on SPARK-3160: - User 'jkbradley' has created a pull request for

[jira] [Updated] (SPARK-3160) Simplify DecisionTree data structure for training

2014-09-09 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-3160: - Description: Improvement: code clarity Currently, we maintain a tree structure, a flat ar

[jira] [Commented] (SPARK-3212) Improve the clarity of caching semantics

2014-09-09 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127959#comment-14127959 ] Michael Armbrust commented on SPARK-3212: - [~matei] also points out that we should

[jira] [Resolved] (SPARK-3458) enable use of python's "with" statements for SparkContext management

2014-09-09 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-3458. Resolution: Fixed Fix Version/s: 1.2.0 > enable use of python's "with" statements for SparkCo

[jira] [Commented] (SPARK-3446) FutureAction should expose the job ID

2014-09-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127919#comment-14127919 ] Apache Spark commented on SPARK-3446: - User 'vanzin' has created a pull request for th

[jira] [Commented] (SPARK-3465) Task metrics are not aggregated correctly in local mode

2014-09-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127920#comment-14127920 ] Apache Spark commented on SPARK-3465: - User 'davies' has created a pull request for th

[jira] [Commented] (SPARK-3463) Show metrics about spilling in Python

2014-09-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127918#comment-14127918 ] Apache Spark commented on SPARK-3463: - User 'davies' has created a pull request for th

[jira] [Updated] (SPARK-3412) Add Missing Types for Row API

2014-09-09 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-3412: Assignee: Cheng Hao > Add Missing Types for Row API > - > >

[jira] [Commented] (SPARK-3412) Add Missing Types for Row API

2014-09-09 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127914#comment-14127914 ] Michael Armbrust commented on SPARK-3412: - Instead of adding methods for each data

[jira] [Resolved] (SPARK-3448) SpecificMutableRow.update doesn't check for null

2014-09-09 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-3448. - Resolution: Fixed Fix Version/s: 1.2.0 > SpecificMutableRow.update doesn't check fo

[jira] [Resolved] (SPARK-3176) Implement 'ABS' and 'LAST' for sql

2014-09-09 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-3176. - Resolution: Fixed > Implement 'ABS' and 'LAST' for sql > -

[jira] [Created] (SPARK-3467) Python BatchedSerializer should dynamically lower batch size for large objects

2014-09-09 Thread Matei Zaharia (JIRA)
Matei Zaharia created SPARK-3467: Summary: Python BatchedSerializer should dynamically lower batch size for large objects Key: SPARK-3467 URL: https://issues.apache.org/jira/browse/SPARK-3467 Project:

[jira] [Updated] (SPARK-3466) Limit size of results that a driver collects for each action

2014-09-09 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated SPARK-3466: - Issue Type: New Feature (was: Improvement) > Limit size of results that a driver collects for eac

[jira] [Created] (SPARK-3466) Limit size of results that a driver collects for each action

2014-09-09 Thread Matei Zaharia (JIRA)
Matei Zaharia created SPARK-3466: Summary: Limit size of results that a driver collects for each action Key: SPARK-3466 URL: https://issues.apache.org/jira/browse/SPARK-3466 Project: Spark I

[jira] [Closed] (SPARK-1919) In Windows, Spark shell cannot load classes in spark.jars (--jars)

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-1919. Resolution: Fixed Fix Version/s: 1.1.1 Ok, backported to 1.1 > In Windows, Spark shell cannot load c

[jira] [Assigned] (SPARK-1919) In Windows, Spark shell cannot load classes in spark.jars (--jars)

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reassigned SPARK-1919: Assignee: Andrew Or > In Windows, Spark shell cannot load classes in spark.jars (--jars) >

[jira] [Commented] (SPARK-3061) Maven build fails in Windows OS

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127677#comment-14127677 ] Andrew Or commented on SPARK-3061: -- Ok, backported. Thanks Josh. > Maven build fails in

[jira] [Closed] (SPARK-3061) Maven build fails in Windows OS

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3061. Resolution: Fixed Assignee: Josh Rosen (was: Andrew Or) > Maven build fails in Windows OS > -

[jira] [Updated] (SPARK-3061) Maven build fails in Windows OS

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3061: - Fix Version/s: 1.1.1 > Maven build fails in Windows OS > --- > >

[jira] [Updated] (SPARK-3345) Do correct parameters for ShuffleFileGroup

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3345: - Fix Version/s: 1.1.1 > Do correct parameters for ShuffleFileGroup > --

[jira] [Updated] (SPARK-3409) Avoid pulling in Exchange operator itself in Exchange's closures

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3409: - Fix Version/s: 1.1.1 > Avoid pulling in Exchange operator itself in Exchange's closures >

[jira] [Updated] (SPARK-3409) Avoid pulling in Exchange operator itself in Exchange's closures

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3409: - Affects Version/s: 1.1.0 > Avoid pulling in Exchange operator itself in Exchange's closures >

[jira] [Updated] (SPARK-3465) Task metrics are not aggregated correctly in local mode

2014-09-09 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-3465: -- Description: In local mode, after onExecutorMetricsUpdate(), t.taskMetrics will be the same object with

[jira] [Updated] (SPARK-3465) Task metrics are not aggregated correctly in local mode

2014-09-09 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-3465: -- Description: In local mode, after onExecutorMetricsUpdate(), t.taskMetrics will be the same object with

[jira] [Updated] (SPARK-3465) Task metrics are not aggregated correctly in local mode

2014-09-09 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-3465: -- Description: In local mode, after onExecutorMetricsUpdate(), t.taskMetrics will be the same object with

[jira] [Updated] (SPARK-3465) Task metrics are not aggregated correctly in local mode

2014-09-09 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-3465: -- Description: In local mode, after onExecutorMetricsUpdate(), t.taskMetrics will be the same object with

[jira] [Created] (SPARK-3465) Task metrics are not aggregated correctly in local mode

2014-09-09 Thread Davies Liu (JIRA)
Davies Liu created SPARK-3465: - Summary: Task metrics are not aggregated correctly in local mode Key: SPARK-3465 URL: https://issues.apache.org/jira/browse/SPARK-3465 Project: Spark Issue Type: B

[jira] [Resolved] (SPARK-3404) SparkSubmitSuite fails with "spark-submit exits with code 1"

2014-09-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-3404. -- Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Tests are now failing due to Hive

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-09-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127564#comment-14127564 ] Marcelo Vanzin commented on SPARK-3215: --- For those following, I moved the prototype

[jira] [Commented] (SPARK-3450) Enable specifiying the --jars CLI option multiple times

2014-09-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127533#comment-14127533 ] Marcelo Vanzin commented on SPARK-3450: --- My only concern is that adding this would p

[jira] [Commented] (SPARK-1985) SPARK_HOME shouldn't be required when spark.executor.uri is provided

2014-09-09 Thread Chip Senkbeil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127455#comment-14127455 ] Chip Senkbeil commented on SPARK-1985: -- Does anyone know what the status of this is?

[jira] [Commented] (SPARK-3174) Under YARN, add and remove executors based on load

2014-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127446#comment-14127446 ] Thomas Graves commented on SPARK-3174: -- {quote} Yeah so how about we create a sub-tas

[jira] [Updated] (SPARK-3464) Graceful decommission of executors

2014-09-09 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-3464: -- Description: In most cases, even when an application is utilizing only a small fraction of its available

[jira] [Updated] (SPARK-3460) Discard executors

2014-09-09 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-3460: -- Summary: Discard executors (was: Graceful decommission of idle YARN sessions) > Discard executors > --

[jira] [Updated] (SPARK-3460) Under YARN, discard executors that have been idle

2014-09-09 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-3460: -- Summary: Under YARN, discard executors that have been idle (was: Discard executors) > Under YARN, dis

[jira] [Created] (SPARK-3464) Graceful decommission of executors

2014-09-09 Thread Sandy Ryza (JIRA)
Sandy Ryza created SPARK-3464: - Summary: Graceful decommission of executors Key: SPARK-3464 URL: https://issues.apache.org/jira/browse/SPARK-3464 Project: Spark Issue Type: Sub-task R

[jira] [Commented] (SPARK-3174) Under YARN, add and remove executors based on load

2014-09-09 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127415#comment-14127415 ] Sandy Ryza commented on SPARK-3174: --- bq. Since you mention the graceful decommission as

[jira] [Closed] (SPARK-3193) output error info when Process exitcode not zero

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3193. Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Assignee: wangfei > output e

[jira] [Created] (SPARK-3463) Show metrics about spilling in Python

2014-09-09 Thread Davies Liu (JIRA)
Davies Liu created SPARK-3463: - Summary: Show metrics about spilling in Python Key: SPARK-3463 URL: https://issues.apache.org/jira/browse/SPARK-3463 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-2425) Standalone Master is too aggressive in removing Applications

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127322#comment-14127322 ] Andrew Or commented on SPARK-2425: -- Reopened a few times to change the fixed version. The

[jira] [Closed] (SPARK-2425) Standalone Master is too aggressive in removing Applications

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-2425. Resolution: Fixed > Standalone Master is too aggressive in removing Applications > -

[jira] [Updated] (SPARK-2425) Standalone Master is too aggressive in removing Applications

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2425: - Fix Version/s: 1.1.1 > Standalone Master is too aggressive in removing Applications >

[jira] [Reopened] (SPARK-2425) Standalone Master is too aggressive in removing Applications

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reopened SPARK-2425: -- > Standalone Master is too aggressive in removing Applications > ---

[jira] [Reopened] (SPARK-2425) Standalone Master is too aggressive in removing Applications

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reopened SPARK-2425: -- > Standalone Master is too aggressive in removing Applications > ---

[jira] [Closed] (SPARK-2425) Standalone Master is too aggressive in removing Applications

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-2425. Resolution: Fixed > Standalone Master is too aggressive in removing Applications > -

[jira] [Commented] (SPARK-3445) Deprecate and later remove YARN alpha support

2014-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127318#comment-14127318 ] Thomas Graves commented on SPARK-3445: -- We are in progress of moving and the timeline

[jira] [Updated] (SPARK-2425) Standalone Master is too aggressive in removing Applications

2014-09-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2425: - Fix Version/s: (was: 1.1.1) > Standalone Master is too aggressive in removing Applications > -

[jira] [Created] (SPARK-3462) parquet pushdown for unionAll

2014-09-09 Thread Cody Koeninger (JIRA)
Cody Koeninger created SPARK-3462: - Summary: parquet pushdown for unionAll Key: SPARK-3462 URL: https://issues.apache.org/jira/browse/SPARK-3462 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-3445) Deprecate and later remove YARN alpha support

2014-09-09 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127301#comment-14127301 ] Reynold Xin commented on SPARK-3445: [~tgraves] when is Yahoo moving? (or was that alr

[jira] [Closed] (SPARK-3459) MulticlassMetrics is not serializable

2014-09-09 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng closed SPARK-3459. Resolution: Cannot Reproduce > MulticlassMetrics is not serializable > -

[jira] [Updated] (SPARK-3461) Support external groupBy

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3461: --- Component/s: Spark Core > Support external groupBy > > >

[jira] [Updated] (SPARK-3461) Support external groupBy

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3461: --- Description: Given that we have SPARK-2978, it seems like we could support an external group

[jira] [Updated] (SPARK-3461) Support external groupBy

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3461: --- Summary: Support external groupBy (was: Support external group-by) > Support external groupBy

[jira] [Created] (SPARK-3461) Support external group-by

2014-09-09 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3461: -- Summary: Support external group-by Key: SPARK-3461 URL: https://issues.apache.org/jira/browse/SPARK-3461 Project: Spark Issue Type: Bug Repor

[jira] [Updated] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3426: --- Priority: Blocker (was: Critical) > Sort-based shuffle compression behavior is inconsistent >

[jira] [Commented] (SPARK-2182) Scalastyle rule blocking unicode operators

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127248#comment-14127248 ] Patrick Wendell commented on SPARK-2182: [~prashant_] as our resident expert on sc

[jira] [Updated] (SPARK-2182) Scalastyle rule blocking unicode operators

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2182: --- Assignee: Prashant Sharma > Scalastyle rule blocking unicode operators > -

[jira] [Updated] (SPARK-3438) Support for accessing secured HDFS in Standalone Mode

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3438: --- Description: Secured HDFS is supported in YARN currently, but not in standalone mode. The tric

[jira] [Updated] (SPARK-3438) Support for accessing secured HDFS in Standalone Mode

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3438: --- Summary: Support for accessing secured HDFS in Standalone Mode (was: Adding support for acces

[jira] [Updated] (SPARK-3438) Support for accessing secured HDFS in Standalone Mode

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3438: --- Component/s: Deploy > Support for accessing secured HDFS in Standalone Mode >

[jira] [Comment Edited] (SPARK-3174) Under YARN, add and remove executors based on load

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127232#comment-14127232 ] Patrick Wendell edited comment on SPARK-3174 at 9/9/14 5:08 PM:

[jira] [Created] (SPARK-3460) Graceful decommission of idle YARN sessions

2014-09-09 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3460: -- Summary: Graceful decommission of idle YARN sessions Key: SPARK-3460 URL: https://issues.apache.org/jira/browse/SPARK-3460 Project: Spark Issue Type: Sub

[jira] [Commented] (SPARK-3174) Under YARN, add and remove executors based on load

2014-09-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127232#comment-14127232 ] Patrick Wendell commented on SPARK-3174: Yeah so how about we create a sub-task th

[jira] [Comment Edited] (SPARK-3174) Under YARN, add and remove executors based on load

2014-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127188#comment-14127188 ] Thomas Graves edited comment on SPARK-3174 at 9/9/14 4:51 PM: --

[jira] [Commented] (SPARK-3458) enable use of python's "with" statements for SparkContext management

2014-09-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127210#comment-14127210 ] Apache Spark commented on SPARK-3458: - User 'mattf' has created a pull request for thi

[jira] [Created] (SPARK-3459) MulticlassMetrics is not serializable

2014-09-09 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-3459: Summary: MulticlassMetrics is not serializable Key: SPARK-3459 URL: https://issues.apache.org/jira/browse/SPARK-3459 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-3445) Deprecate and later remove YARN alpha support

2014-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127204#comment-14127204 ] Thomas Graves commented on SPARK-3445: -- Yahoo moving to 2.x and I'm +1 for this propo

[jira] [Commented] (SPARK-3174) Under YARN, add and remove executors based on load

2014-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127188#comment-14127188 ] Thomas Graves commented on SPARK-3174: -- Since you mention the graceful decommission a

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

2014-09-09 Thread Matthew Farrellee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127187#comment-14127187 ] Matthew Farrellee commented on SPARK-2972: -- +1 close this and open 2 feature requ

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

2014-09-09 Thread Shay Rojansky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127171#comment-14127171 ] Shay Rojansky commented on SPARK-2972: -- I'd love to help on this, but I know 0 Scala

[jira] [Updated] (SPARK-3458) enable use of python's "with" statements for SparkContext management

2014-09-09 Thread Matthew Farrellee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee updated SPARK-3458: - Description: best practice for managing SparkContexts involves exception handling, e.g. {

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

2014-09-09 Thread Matthew Farrellee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127137#comment-14127137 ] Matthew Farrellee commented on SPARK-2972: -- fyi - i've created https://issues.apa

[jira] [Created] (SPARK-3458) enable use of python's "with" statements for SparkContext management

2014-09-09 Thread Matthew Farrellee (JIRA)
Matthew Farrellee created SPARK-3458: Summary: enable use of python's "with" statements for SparkContext management Key: SPARK-3458 URL: https://issues.apache.org/jira/browse/SPARK-3458 Project: S

[jira] [Updated] (SPARK-3457) ConcurrentModificationException starting up pyspark

2014-09-09 Thread Shay Rojansky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shay Rojansky updated SPARK-3457: - Description: Just downloaded Spark 1.1.0-rc4. Launching pyspark for the very first time in yarn-c

[jira] [Created] (SPARK-3457) ConcurrentModificationException starting up pyspark

2014-09-09 Thread Shay Rojansky (JIRA)
Shay Rojansky created SPARK-3457: Summary: ConcurrentModificationException starting up pyspark Key: SPARK-3457 URL: https://issues.apache.org/jira/browse/SPARK-3457 Project: Spark Issue Type:

[jira] [Commented] (SPARK-3456) YarnAllocator can lose container requests to RM

2014-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127088#comment-14127088 ] Thomas Graves commented on SPARK-3456: -- Note this is only a problem on yarn alpha bec

[jira] [Created] (SPARK-3456) YarnAllocator can lose container requests to RM

2014-09-09 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-3456: Summary: YarnAllocator can lose container requests to RM Key: SPARK-3456 URL: https://issues.apache.org/jira/browse/SPARK-3456 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

2014-09-09 Thread Matthew Farrellee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127016#comment-14127016 ] Matthew Farrellee commented on SPARK-2972: -- > I suggest having context implement

[jira] [Commented] (SPARK-3455) **HotFix** Unit test failed due to can not resolve the attribute references

2014-09-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126909#comment-14126909 ] Apache Spark commented on SPARK-3455: - User 'chenghao-intel' has created a pull reques

[jira] [Created] (SPARK-3455) **HotFix** Unit test failed due to can not resolve the attribute references

2014-09-09 Thread Cheng Hao (JIRA)
Cheng Hao created SPARK-3455: Summary: **HotFix** Unit test failed due to can not resolve the attribute references Key: SPARK-3455 URL: https://issues.apache.org/jira/browse/SPARK-3455 Project: Spark

[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI

2014-09-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126854#comment-14126854 ] Apache Spark commented on SPARK-3454: - User 'sarutak' has created a pull request for t

[jira] [Created] (SPARK-3454) Expose JSON expression of data shown in WebUI

2014-09-09 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-3454: - Summary: Expose JSON expression of data shown in WebUI Key: SPARK-3454 URL: https://issues.apache.org/jira/browse/SPARK-3454 Project: Spark Issue Type: Imp

[jira] [Updated] (SPARK-3454) Expose JSON representation of data shown in WebUI

2014-09-09 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3454: -- Summary: Expose JSON representation of data shown in WebUI (was: Expose JSON expression of data

  1   2   >