[jira] [Updated] (SPARK-9426) Job page DAG visualization is not shown

2015-07-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9426: - Attachment: Screen Shot 2015-07-28 at 7.03.09 PM.png > Job page DAG visualization is not shown > -

[jira] [Created] (SPARK-9426) Job page DAG visualization is not shown

2015-07-28 Thread Andrew Or (JIRA)
Andrew Or created SPARK-9426: Summary: Job page DAG visualization is not shown Key: SPARK-9426 URL: https://issues.apache.org/jira/browse/SPARK-9426 Project: Spark Issue Type: Bug Compo

[jira] [Commented] (SPARK-9092) Make --num-executors compatible with dynamic allocation

2015-07-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644809#comment-14644809 ] Andrew Or commented on SPARK-9092: -- To add to [~sandyryza]'s comment, I think there's act

[jira] [Comment Edited] (SPARK-7268) [Spark SQL] Throw 'Shutdown hooks cannot be modified during shutdown' on YARN

2015-07-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644774#comment-14644774 ] Andrew Or edited comment on SPARK-7268 at 7/28/15 6:04 PM: --- By t

[jira] [Commented] (SPARK-7268) [Spark SQL] Throw 'Shutdown hooks cannot be modified during shutdown' on YARN

2015-07-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644773#comment-14644773 ] Andrew Or commented on SPARK-7268: -- I just re-closed this as "Cannot reproduce" since tha

[jira] [Commented] (SPARK-7268) [Spark SQL] Throw 'Shutdown hooks cannot be modified during shutdown' on YARN

2015-07-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644774#comment-14644774 ] Andrew Or commented on SPARK-7268: -- By the way [~yizhou] is this still an issue with the

[jira] [Reopened] (SPARK-7268) [Spark SQL] Throw 'Shutdown hooks cannot be modified during shutdown' on YARN

2015-07-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reopened SPARK-7268: -- > [Spark SQL] Throw 'Shutdown hooks cannot be modified during shutdown' on YARN > --

[jira] [Closed] (SPARK-7268) [Spark SQL] Throw 'Shutdown hooks cannot be modified during shutdown' on YARN

2015-07-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-7268. Resolution: Cannot Reproduce > [Spark SQL] Throw 'Shutdown hooks cannot be modified during shutdown' on YARN

[jira] [Closed] (SPARK-3684) Can't configure local dirs in Yarn mode

2015-07-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3684. Resolution: Won't Fix > Can't configure local dirs in Yarn mode > --- >

[jira] [Commented] (SPARK-3684) Can't configure local dirs in Yarn mode

2015-07-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644767#comment-14644767 ] Andrew Or commented on SPARK-3684: -- OK, let's close this then. My original use case was f

[jira] [Closed] (SPARK-9353) Standalone scheduling memory requirement incorrect if cores per executor is not set

2015-07-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-9353. Resolution: Fixed Fix Version/s: 1.5.0 1.4.2 Target Version/s: 1.4.2,

[jira] [Closed] (SPARK-9352) Add tests for standalone scheduling code

2015-07-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-9352. Resolution: Fixed Fix Version/s: 1.5.0 1.4.2 Target Version/s: 1.4.2,

[jira] [Created] (SPARK-9353) Standalone scheduling memory requirement incorrect if cores per executor is not set

2015-07-25 Thread Andrew Or (JIRA)
Andrew Or created SPARK-9353: Summary: Standalone scheduling memory requirement incorrect if cores per executor is not set Key: SPARK-9353 URL: https://issues.apache.org/jira/browse/SPARK-9353 Project: Sp

[jira] [Created] (SPARK-9352) Add tests for standalone scheduling code

2015-07-25 Thread Andrew Or (JIRA)
Andrew Or created SPARK-9352: Summary: Add tests for standalone scheduling code Key: SPARK-9352 URL: https://issues.apache.org/jira/browse/SPARK-9352 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-8881) Standalone mode scheduling fails because cores assignment is not atomic

2015-07-25 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8881: - Fix Version/s: 1.5.0 1.4.2 > Standalone mode scheduling fails because cores assignment

[jira] [Closed] (SPARK-8881) Standalone mode scheduling fails because cores assignment is not atomic

2015-07-25 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-8881. Resolution: Fixed > Standalone mode scheduling fails because cores assignment is not atomic > --

[jira] [Closed] (SPARK-9260) Standalone scheduling can overflow a worker with cores

2015-07-25 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-9260. Resolution: Fixed Fix Version/s: 1.5.0 1.4.2 Target Version/s: 1.4.2,

[jira] [Updated] (SPARK-9267) Remove highly unnecessary accumulators stringify methods

2015-07-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9267: - Description: {code} def stringifyPartialValue(partialValue: Any): String = "%s".format(partialValue) def s

[jira] [Updated] (SPARK-9267) Remove highly unnecessary accumulators stringify methods

2015-07-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9267: - Assignee: (was: Andrew Or) > Remove highly unnecessary accumulators stringify methods > --

[jira] [Updated] (SPARK-9267) Remove highly unnecessary accumulators stringify methods

2015-07-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9267: - Description: {code} def stringifyPartialValue(partialValue: Any): String = "%s".format(partialValue) def s

[jira] [Updated] (SPARK-9267) Remove highly unnecessary accumulators stringify methods

2015-07-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9267: - Description: {code} def stringifyPartialValue(partialValue: Any): String = "%s".format(partialValue) def s

[jira] [Created] (SPARK-9267) Remove highly unnecessary accumulators stringify methods

2015-07-22 Thread Andrew Or (JIRA)
Andrew Or created SPARK-9267: Summary: Remove highly unnecessary accumulators stringify methods Key: SPARK-9267 URL: https://issues.apache.org/jira/browse/SPARK-9267 Project: Spark Issue Type: Bu

[jira] [Updated] (SPARK-9260) Standalone scheduling can overflow a worker with cores

2015-07-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9260: - Attachment: Screen Shot 2015-07-22 at 12.06.21 PM.png > Standalone scheduling can overflow a worker with c

[jira] [Created] (SPARK-9260) Standalone scheduling can overflow a worker with cores

2015-07-22 Thread Andrew Or (JIRA)
Andrew Or created SPARK-9260: Summary: Standalone scheduling can overflow a worker with cores Key: SPARK-9260 URL: https://issues.apache.org/jira/browse/SPARK-9260 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-9092) Make --num-executors compatible with dynamic allocation

2015-07-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9092: - Affects Version/s: 1.2.0 > Make --num-executors compatible with dynamic allocation > -

[jira] [Updated] (SPARK-9092) Make --num-executors compatible with dynamic allocation

2015-07-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9092: - Priority: Major (was: Minor) > Make --num-executors compatible with dynamic allocation >

[jira] [Created] (SPARK-9232) Duplicate code in JSONRelation

2015-07-21 Thread Andrew Or (JIRA)
Andrew Or created SPARK-9232: Summary: Duplicate code in JSONRelation Key: SPARK-9232 URL: https://issues.apache.org/jira/browse/SPARK-9232 Project: Spark Issue Type: Improvement Compon

[jira] [Closed] (SPARK-5423) ExternalAppendOnlyMap won't delete temp spilled file if some exception happens during using it

2015-07-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-5423. Resolution: Fixed Fix Version/s: 1.5.0 Target Version/s: 1.5.0 > ExternalAppendOnlyMap won'

[jira] [Closed] (SPARK-4598) Paginate stage page to avoid OOM with > 100,000 tasks

2015-07-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-4598. Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 1.5.0 Target Version/s:

[jira] [Updated] (SPARK-7171) Allow for more flexible use of metric sources

2015-07-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-7171: - Assignee: Jacek Lewandowski > Allow for more flexible use of metric sources >

[jira] [Closed] (SPARK-7171) Allow for more flexible use of metric sources

2015-07-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-7171. Resolution: Fixed Fix Version/s: 1.5.0 Target Version/s: 1.5.0 > Allow for more flexible us

[jira] [Closed] (SPARK-9128) Get outerclasses and objects at the same time in ClosureCleaner

2015-07-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-9128. Resolution: Fixed Fix Version/s: 1.5.0 Target Version/s: 1.5.0 > Get outerclasses and objec

[jira] [Closed] (SPARK-9036) SparkListenerExecutorMetricsUpdate messages not included in JsonProtocol

2015-07-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-9036. Resolution: Fixed Fix Version/s: 1.5.0 Target Version/s: 1.5.0 > SparkListenerExecutorMetri

[jira] [Updated] (SPARK-9193) Avoid assigning tasks to executors under killing

2015-07-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9193: - Priority: Major (was: Minor) > Avoid assigning tasks to executors under killing > ---

[jira] [Updated] (SPARK-9193) Avoid assigning tasks to executors under killing

2015-07-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9193: - Assignee: Jie Huang > Avoid assigning tasks to executors under killing > -

[jira] [Updated] (SPARK-9193) Avoid assigning tasks to executors under killing

2015-07-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9193: - Target Version/s: 1.4.2, 1.5.0 > Avoid assigning tasks to executors under killing > --

[jira] [Closed] (SPARK-4922) Support dynamic allocation for coarse-grained Mesos

2015-07-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-4922. Resolution: Fixed > Support dynamic allocation for coarse-grained Mesos > --

[jira] [Closed] (SPARK-4922) Support dynamic allocation for coarse-grained Mesos

2015-07-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-4922. Resolution: Duplicate > Support dynamic allocation for coarse-grained Mesos > --

[jira] [Reopened] (SPARK-4922) Support dynamic allocation for coarse-grained Mesos

2015-07-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reopened SPARK-4922: -- > Support dynamic allocation for coarse-grained Mesos > ---

[jira] [Closed] (SPARK-5349) Spark standalone should support dynamic resource scaling

2015-07-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-5349. Resolution: Duplicate > Spark standalone should support dynamic resource scaling > -

[jira] [Updated] (SPARK-4751) Support dynamic allocation for standalone mode

2015-07-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4751: - Target Version/s: 1.5.0 > Support dynamic allocation for standalone mode > ---

[jira] [Updated] (SPARK-9128) Get outerclasses and objects at the same time in ClosureCleaner

2015-07-17 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9128: - Assignee: Liang-Chi Hsieh > Get outerclasses and objects at the same time in ClosureCleaner >

[jira] [Updated] (SPARK-9128) Get outerclasses and objects at the same time in ClosureCleaner

2015-07-17 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9128: - Affects Version/s: 1.0.0 > Get outerclasses and objects at the same time in ClosureCleaner > -

[jira] [Updated] (SPARK-8881) Standalone mode scheduling fails because cores assignment is not atomic

2015-07-17 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8881: - Description: Current scheduling algorithm (in Master.scala) has two issues: 1. cores are allocated one at

[jira] [Updated] (SPARK-8881) Standalone mode scheduling fails because cores assignment is not atomic

2015-07-17 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8881: - Summary: Standalone mode scheduling fails because cores assignment is not atomic (was: Standalone mode sc

[jira] [Updated] (SPARK-8881) Standalone mode scheduling fails if num_executors < num_workers

2015-07-17 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8881: - Priority: Critical (was: Major) > Standalone mode scheduling fails if num_executors < num_workers > -

[jira] [Closed] (SPARK-8166) Consistently failing test: o.a.s.sql.sources.FilteredScanSuite

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-8166. Resolution: Cannot Reproduce > Consistently failing test: o.a.s.sql.sources.FilteredScanSuite >

[jira] [Commented] (SPARK-8166) Consistently failing test: o.a.s.sql.sources.FilteredScanSuite

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630748#comment-14630748 ] Andrew Or commented on SPARK-8166: -- Ok, sounds good. > Consistently failing test: o.a.s.

[jira] [Updated] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8119: - Fix Version/s: 1.5.0 > HeartbeatReceiver should not adjust application executor resources > --

[jira] [Closed] (SPARK-6284) Support framework authentication and role in Mesos framework

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-6284. Resolution: Fixed Fix Version/s: 1.5.0 > Support framework authentication and role in Mesos framework

[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630687#comment-14630687 ] Andrew Or commented on SPARK-6284: -- This is now merged and will appear in Spark 1.5! Than

[jira] [Comment Edited] (SPARK-6284) Support framework authentication and role in Mesos framework

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630687#comment-14630687 ] Andrew Or edited comment on SPARK-6284 at 7/17/15 2:38 AM: --- This

[jira] [Closed] (SPARK-8646) PySpark does not run on YARN if master not provided in command line

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-8646. Resolution: Fixed Fix Version/s: 1.5.0 Target Version/s: 1.5.0 > PySpark does not run on YA

[jira] [Updated] (SPARK-8646) PySpark does not run on YARN if master not provided in command line

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8646: - Assignee: Lianhui Wang > PySpark does not run on YARN if master not provided in command line > ---

[jira] [Updated] (SPARK-8881) Standalone mode scheduling fails if num_executors < num_workers

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8881: - Target Version/s: 1.4.2, 1.5.0 > Standalone mode scheduling fails if num_executors < num_workers > ---

[jira] [Updated] (SPARK-8881) Standalone mode scheduling fails if num_executors < num_workers

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8881: - Assignee: Nishkam Ravi > Standalone mode scheduling fails if num_executors < num_workers > ---

[jira] [Updated] (SPARK-4134) Dynamic allocation: tone down scary executor lost messages when killing on purpose

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4134: - Summary: Dynamic allocation: tone down scary executor lost messages when killing on purpose (was: Tone do

[jira] [Commented] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630545#comment-14630545 ] Andrew Or commented on SPARK-8119: -- Hi [~roji], yes, we should fix it for 1.4.2 as well.

[jira] [Updated] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8119: - Target Version/s: 1.4.2, 1.5.0 (was: 1.5.0) > HeartbeatReceiver should not adjust application executor re

[jira] [Updated] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-07-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8119: - Target Version/s: 1.5.0 > HeartbeatReceiver should not adjust application executor resources > ---

[jira] [Commented] (SPARK-9025) Storage tab shows no blocks for cached RDDs

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627533#comment-14627533 ] Andrew Or commented on SPARK-9025: -- Yeah, me neither. > Storage tab shows no blocks for

[jira] [Closed] (SPARK-9025) Storage tab shows no blocks for cached RDDs

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-9025. Resolution: Cannot Reproduce > Storage tab shows no blocks for cached RDDs > ---

[jira] [Commented] (SPARK-1855) Provide memory-and-local-disk RDD checkpointing

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627470#comment-14627470 ] Andrew Or commented on SPARK-1855: -- I just found this old issue. I've been working on a d

[jira] [Updated] (SPARK-1855) Provide memory-and-local-disk RDD checkpointing

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1855: - Assignee: Andrew Or > Provide memory-and-local-disk RDD checkpointing > --

[jira] [Updated] (SPARK-1855) Provide memory-and-local-disk RDD checkpointing

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1855: - Target Version/s: 1.5.0 > Provide memory-and-local-disk RDD checkpointing > --

[jira] [Closed] (SPARK-7292) Provide operator to truncate lineage without persisting RDD's

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-7292. Resolution: Duplicate > Provide operator to truncate lineage without persisting RDD's >

[jira] [Updated] (SPARK-8881) Standalone mode scheduling fails if num_executors < num_workers

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8881: - Summary: Standalone mode scheduling fails if num_executors < num_workers (was: Scheduling fails if num_ex

[jira] [Closed] (SPARK-8374) Job frequently hangs after YARN preemption

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-8374. Resolution: Duplicate > Job frequently hangs after YARN preemption > ---

[jira] [Updated] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8119: - Assignee: Andrew Or > HeartbeatReceiver should not adjust application executor resources > ---

[jira] [Updated] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8119: - Priority: Critical (was: Major) > HeartbeatReceiver should not adjust application executor resources > --

[jira] [Closed] (SPARK-8911) Local mode endless heartbeat warnings

2015-07-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-8911. Resolution: Fixed Fix Version/s: 1.5.0 > Local mode endless heartbeat warnings >

[jira] [Created] (SPARK-9025) Storage tab shows no blocks for cached RDDs

2015-07-13 Thread Andrew Or (JIRA)
Andrew Or created SPARK-9025: Summary: Storage tab shows no blocks for cached RDDs Key: SPARK-9025 URL: https://issues.apache.org/jira/browse/SPARK-9025 Project: Spark Issue Type: Bug C

[jira] [Created] (SPARK-8987) Increase test coverage of DAGScheduler

2015-07-10 Thread Andrew Or (JIRA)
Andrew Or created SPARK-8987: Summary: Increase test coverage of DAGScheduler Key: SPARK-8987 URL: https://issues.apache.org/jira/browse/SPARK-8987 Project: Spark Issue Type: Bug Compon

[jira] [Closed] (SPARK-8675) Executors created by LocalBackend won't get the same classpath as other executor backends

2015-07-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-8675. Resolution: Fixed Assignee: Min Zhou Fix Version/s: 1.5.0 Target Version/s: 1.5.

[jira] [Closed] (SPARK-8958) Dynamic allocation: change cached executor timeout to infinity

2015-07-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-8958. Resolution: Fixed Fix Version/s: 1.5.0 > Dynamic allocation: change cached executor timeout to infini

[jira] [Commented] (SPARK-6287) Add support for dynamic allocation in the Mesos coarse-grained scheduler

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621204#comment-14621204 ] Andrew Or commented on SPARK-6287: -- Note that this feature is not complete. The shuffle f

[jira] [Updated] (SPARK-8873) Support cleaning up shuffle files when using shuffle service in Mesos

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8873: - Affects Version/s: (was: 1.5.0) 1.2.0 > Support cleaning up shuffle files when

[jira] [Updated] (SPARK-8873) Support cleaning up shuffle files when using shuffle service in Mesos

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8873: - Affects Version/s: 1.5.0 > Support cleaning up shuffle files when using shuffle service in Mesos > ---

[jira] [Updated] (SPARK-8873) Support cleaning up shuffle files when using shuffle service in Mesos

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8873: - Target Version/s: 1.5.0 > Support cleaning up shuffle files when using shuffle service in Mesos >

[jira] [Updated] (SPARK-8873) Support cleaning up shuffle files when using shuffle service in Mesos

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8873: - Assignee: Timothy Chen > Support cleaning up shuffle files when using shuffle service in Mesos > -

[jira] [Updated] (SPARK-8873) Support cleaning up shuffle files when using shuffle service in Mesos

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8873: - Summary: Support cleaning up shuffle files when using shuffle service in Mesos (was: Support cleaning up

[jira] [Closed] (SPARK-6287) Add support for dynamic allocation in the Mesos coarse-grained scheduler

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-6287. Resolution: Fixed Assignee: Iulian Dragos Fix Version/s: 1.5.0 Target Version/s:

[jira] [Updated] (SPARK-8873) Support cleaning up shuffle files for drivers launched with Mesos

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8873: - Priority: Critical (was: Minor) > Support cleaning up shuffle files for drivers launched with Mesos > ---

[jira] [Updated] (SPARK-6287) Add support for dynamic allocation in the Mesos coarse-grained scheduler

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-6287: - Affects Version/s: 1.2.0 > Add support for dynamic allocation in the Mesos coarse-grained scheduler >

[jira] [Updated] (SPARK-2017) web ui stage page becomes unresponsive when the number of tasks is large

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2017: - Affects Version/s: 1.0.0 > web ui stage page becomes unresponsive when the number of tasks is large >

[jira] [Closed] (SPARK-2017) web ui stage page becomes unresponsive when the number of tasks is large

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-2017. Resolution: Fixed Assignee: Andrew Or Target Version/s: 1.4.2, 1.5.0 > web ui stage pa

[jira] [Comment Edited] (SPARK-2017) web ui stage page becomes unresponsive when the number of tasks is large

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621196#comment-14621196 ] Andrew Or edited comment on SPARK-2017 at 7/9/15 8:24 PM: -- Resolv

[jira] [Updated] (SPARK-2017) web ui stage page becomes unresponsive when the number of tasks is large

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2017: - Fix Version/s: 1.5.0 1.4.2 > web ui stage page becomes unresponsive when the number of

[jira] [Commented] (SPARK-2017) web ui stage page becomes unresponsive when the number of tasks is large

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621196#comment-14621196 ] Andrew Or commented on SPARK-2017: -- Partially resolved by https://github.com/apache/spark

[jira] [Closed] (SPARK-7419) Flaky test: o.a.s.streaming.CheckpointSuite

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-7419. Resolution: Fixed Fix Version/s: 1.5.0 1.4.2 Target Version/s: 1.4.2,

[jira] [Updated] (SPARK-7419) Flaky test: o.a.s.streaming.CheckpointSuite

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-7419: - Assignee: Shixiong Zhu (was: Tathagata Das) > Flaky test: o.a.s.streaming.CheckpointSuite > -

[jira] [Closed] (SPARK-8953) SPARK_EXECUTOR_CORES has no effect to dynamic executor allocation function

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-8953. Resolution: Fixed Assignee: meiyoula Fix Version/s: 1.5.0 Target Version/s: 1.5.

[jira] [Updated] (SPARK-8953) SPARK_EXECUTOR_CORES is not read in SparkSubmit

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8953: - Priority: Trivial (was: Minor) > SPARK_EXECUTOR_CORES is not read in SparkSubmit > --

[jira] [Updated] (SPARK-8953) SPARK_EXECUTOR_CORES is not read in SparkSubmit

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8953: - Summary: SPARK_EXECUTOR_CORES is not read in SparkSubmit (was: SPARK_EXECUTOR_CORES has no effect to dyn

[jira] [Updated] (SPARK-8953) SPARK_EXECUTOR_CORES has no effect to dynamic executor allocation function

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8953: - Affects Version/s: 1.0.0 > SPARK_EXECUTOR_CORES has no effect to dynamic executor allocation function > -

[jira] [Commented] (SPARK-8939) YARN EC2 default setting fails with IllegalArgumentException

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621117#comment-14621117 ] Andrew Or commented on SPARK-8939: -- It's just the default. No particular reason from my u

[jira] [Created] (SPARK-8958) Dynamic allocation: change cached executor timeout to infinity

2015-07-09 Thread Andrew Or (JIRA)
Andrew Or created SPARK-8958: Summary: Dynamic allocation: change cached executor timeout to infinity Key: SPARK-8958 URL: https://issues.apache.org/jira/browse/SPARK-8958 Project: Spark Issue T

[jira] [Commented] (SPARK-8939) YARN EC2 default setting fails with IllegalArgumentException

2015-07-09 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621021#comment-14621021 ] Andrew Or commented on SPARK-8939: -- [~shivaram] > YARN EC2 default setting fails with Il

[jira] [Updated] (SPARK-8939) YARN EC2 does not

2015-07-08 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8939: - Environment: (was: I just set it up from scratch using the spark-ec2 script. Then I ran {code} bin/spa

[jira] [Updated] (SPARK-8939) YARN EC2 does not

2015-07-08 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8939: - Summary: YARN EC2 does not (was: YARN EC2 ) > YARN EC2 does not > -- > >

<    7   8   9   10   11   12   13   14   15   16   >