[jira] [Updated] (SPARK-11097) Add connection established callback to lower level RPC layer so we don't need to check for new connections in NettyRpcHandler.receive

2015-12-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-11097: -- Assignee: Shixiong Zhu (was: Apache Spark) > Add connection established callback to lower level RPC la

[jira] [Resolved] (SPARK-12411) Reconsider executor heartbeats rpc timeout

2015-12-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12411. --- Resolution: Fixed Assignee: Nong Li Fix Version/s: 2.0.0 Target Version/s

[jira] [Resolved] (SPARK-11097) Add connection established callback to lower level RPC layer so we don't need to check for new connections in NettyRpcHandler.receive

2015-12-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-11097. --- Resolution: Fixed Fix Version/s: 2.0.0 > Add connection established callback to lower level RP

[jira] [Updated] (SPARK-12413) Mesos ZK persistence throws a NotSerializableException

2015-12-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12413: -- Fix Version/s: (was: 2.0.0) > Mesos ZK persistence throws a NotSerializableException >

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

2015-12-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065047#comment-15065047 ] Andrew Or commented on SPARK-12345: --- For those who are following: There are 4 patches

[jira] [Created] (SPARK-12466) Harmless Master NPE in tests

2015-12-21 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12466: - Summary: Harmless Master NPE in tests Key: SPARK-12466 URL: https://issues.apache.org/jira/browse/SPARK-12466 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-12464) Remove spark.deploy.mesos.zookeeper.url and use spark.deploy.zookeeper.url

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067062#comment-15067062 ] Andrew Or commented on SPARK-12464: --- By the way for future reference you probably don't

[jira] [Resolved] (SPARK-2331) SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T]

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-2331. -- Resolution: Fixed Fix Version/s: 2.0.0 > SparkContext.emptyRDD should return RDD[T] not EmptyRDD[

[jira] [Resolved] (SPARK-12466) Harmless Master NPE in tests

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12466. --- Resolution: Fixed > Harmless Master NPE in tests > > > K

[jira] [Resolved] (SPARK-12339) NullPointerException on stage kill from web UI

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12339. --- Resolution: Fixed Assignee: Alex Bozarth (was: Apache Spark) Fix Version/s:

[jira] [Updated] (SPARK-12339) NullPointerException on stage kill from web UI

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12339: -- Affects Version/s: (was: 1.6.0) 2.0.0 > NullPointerException on stage kill f

[jira] [Resolved] (SPARK-12392) Optimize a location order of broadcast blocks by considering preferred local hosts

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12392. --- Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.0.0 Target

[jira] [Commented] (SPARK-12339) NullPointerException on stage kill from web UI

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067154#comment-15067154 ] Andrew Or commented on SPARK-12339: --- I've updated the affected version to 2.0 since SPA

[jira] [Resolved] (SPARK-5882) Add a test for GraphLoader.edgeListFile

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-5882. -- Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.0.0 Target Ver

[jira] [Updated] (SPARK-12414) Remove closure serializer

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12414: -- Issue Type: Sub-task (was: Bug) Parent: SPARK-11806 > Remove closure serializer >

[jira] [Created] (SPARK-12473) Reuse serializer instances for performance

2015-12-21 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12473: - Summary: Reuse serializer instances for performance Key: SPARK-12473 URL: https://issues.apache.org/jira/browse/SPARK-12473 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-12473) Reuse serializer instances for performance

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12473: -- Description: After commit de02782 of page rank regressed from 242s to 260s, about 7%. The commit added

[jira] [Updated] (SPARK-12473) Reuse serializer instances for performance

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12473: -- Description: After commit de02782 of page rank regressed from 242s to 260s, about 7%. Although current

[jira] [Updated] (SPARK-12473) Reuse serializer instances for performance

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12473: -- Description: After commit de02782 of page rank regressed from 242s to 260s, about 7%. Although current

[jira] [Updated] (SPARK-12485) Rename "dynamic allocation" to "elastic scaling"

2015-12-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12485: -- Target Version/s: 2.0.0 > Rename "dynamic allocation" to "elastic scaling" > --

[jira] [Created] (SPARK-12485) Rename "dynamic allocation" to "elastic scaling"

2015-12-22 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12485: - Summary: Rename "dynamic allocation" to "elastic scaling" Key: SPARK-12485 URL: https://issues.apache.org/jira/browse/SPARK-12485 Project: Spark Issue Type: Sub-ta

[jira] [Assigned] (SPARK-12414) Remove closure serializer

2015-12-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reassigned SPARK-12414: - Assignee: Andrew Or > Remove closure serializer > - > >

[jira] [Updated] (SPARK-12411) Reconsider executor heartbeats rpc timeout

2015-12-23 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12411: -- Target Version/s: 1.6.1, 2.0.0 (was: 2.0.0) > Reconsider executor heartbeats rpc timeout > ---

[jira] [Updated] (SPARK-12411) Reconsider executor heartbeats rpc timeout

2015-12-23 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12411: -- Fix Version/s: 1.6.1 > Reconsider executor heartbeats rpc timeout > ---

[jira] [Created] (SPARK-12559) Standalone cluster mode doesn't work with --packages

2015-12-29 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12559: - Summary: Standalone cluster mode doesn't work with --packages Key: SPARK-12559 URL: https://issues.apache.org/jira/browse/SPARK-12559 Project: Spark Issue Type: Bu

[jira] [Updated] (SPARK-12559) Standalone cluster mode doesn't work with --packages

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12559: -- Description: >From the mailing list: {quote} Another problem I ran into that you also might is that --p

[jira] [Updated] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12554: -- Priority: Major (was: Critical) > Standalone app scheduler will hang when app.coreToAssign < minCoresP

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074189#comment-15074189 ] Andrew Or commented on SPARK-12554: --- I've downgraded the issue. I also think the behav

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074198#comment-15074198 ] Andrew Or commented on SPARK-12554: --- If anything I would say it's the calculation of {

[jira] [Closed] (SPARK-12415) Do not use closure serializer to serialize task result

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-12415. - Resolution: Won't Fix > Do not use closure serializer to serialize task result >

[jira] [Commented] (SPARK-12415) Do not use closure serializer to serialize task result

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074201#comment-15074201 ] Andrew Or commented on SPARK-12415: --- Closed as Won't Fix. We can't use Kryo to serializ

[jira] [Created] (SPARK-12561) Remove JobLogger

2015-12-29 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12561: - Summary: Remove JobLogger Key: SPARK-12561 URL: https://issues.apache.org/jira/browse/SPARK-12561 Project: Spark Issue Type: Sub-task Components: Spark C

[jira] [Created] (SPARK-12565) Document standalone mode jar distribution more clearly

2015-12-29 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12565: - Summary: Document standalone mode jar distribution more clearly Key: SPARK-12565 URL: https://issues.apache.org/jira/browse/SPARK-12565 Project: Spark Issue Type:

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075353#comment-15075353 ] Andrew Or commented on SPARK-12554: --- [~jerrylead] The change you are proposing changes

[jira] [Updated] (SPARK-12554) Standalone mode may hang if max cores is not a multiple of executor cores

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12554: -- Summary: Standalone mode may hang if max cores is not a multiple of executor cores (was: Standalone ap

[jira] [Updated] (SPARK-12554) Standalone mode may hang if max cores is not a multiple of executor cores

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12554: -- Priority: Minor (was: Major) > Standalone mode may hang if max cores is not a multiple of executor cor

[jira] [Updated] (SPARK-12582) IndexShuffleBlockResolverSuite fails in windows

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12582: -- Assignee: (was: Apache Spark) Target Version/s: 1.6.1, 2.0.0 > IndexShuffleBlockResolve

[jira] [Updated] (SPARK-12582) IndexShuffleBlockResolverSuite fails in windows

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12582: -- Assignee: yucai (was: Andrew Or) > IndexShuffleBlockResolverSuite fails in windows > -

[jira] [Assigned] (SPARK-12582) IndexShuffleBlockResolverSuite fails in windows

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reassigned SPARK-12582: - Assignee: Andrew Or > IndexShuffleBlockResolverSuite fails in windows >

[jira] [Updated] (SPARK-12582) IndexShuffleBlockResolverSuite fails in windows

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12582: -- Component/s: (was: Shuffle) Windows Tests > IndexShuffleBlockReso

[jira] [Commented] (SPARK-12414) Remove closure serializer

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075814#comment-15075814 ] Andrew Or commented on SPARK-12414: --- It's also for code cleanup. Right now SparkEnv has

[jira] [Comment Edited] (SPARK-12414) Remove closure serializer

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075814#comment-15075814 ] Andrew Or edited comment on SPARK-12414 at 12/31/15 7:51 AM: -

[jira] [Assigned] (SPARK-12627) Remove SparkContext preferred locations constructor

2016-01-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reassigned SPARK-12627: - Assignee: Andrew Or > Remove SparkContext preferred locations constructor >

[jira] [Created] (SPARK-12627) Remove SparkContext preferred locations constructor

2016-01-04 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12627: - Summary: Remove SparkContext preferred locations constructor Key: SPARK-12627 URL: https://issues.apache.org/jira/browse/SPARK-12627 Project: Spark Issue Type: Sub

[jira] [Created] (SPARK-12628) SparkUI: weird formatting on additional metrics tooltip

2016-01-04 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12628: - Summary: SparkUI: weird formatting on additional metrics tooltip Key: SPARK-12628 URL: https://issues.apache.org/jira/browse/SPARK-12628 Project: Spark Issue Type:

[jira] [Updated] (SPARK-12628) SparkUI: weird formatting on additional metrics tooltip

2016-01-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12628: -- Attachment: Screen Shot 2016-01-04 at 1.18.16 PM.png > SparkUI: weird formatting on additional metrics

[jira] [Commented] (SPARK-12627) Remove SparkContext preferred locations constructor

2016-01-05 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15084438#comment-15084438 ] Andrew Or commented on SPARK-12627: --- Resolved through https://github.com/apache/spark/p

[jira] [Resolved] (SPARK-12627) Remove SparkContext preferred locations constructor

2016-01-05 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12627. --- Resolution: Fixed Assignee: Reynold Xin (was: Andrew Or) Fix Version/s: 2.0.0 > Remo

[jira] [Updated] (SPARK-2105) SparkUI doesn't remove active stages that failed

2014-06-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2105: - Affects Version/s: 1.0.0 > SparkUI doesn't remove active stages that failed > ---

[jira] [Updated] (SPARK-2105) SparkUI doesn't remove active stages that failed

2014-06-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2105: - Component/s: Spark Core > SparkUI doesn't remove active stages that failed >

[jira] [Created] (SPARK-2105) SparkUI doesn't remove active stages that failed

2014-06-10 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2105: Summary: SparkUI doesn't remove active stages that failed Key: SPARK-2105 URL: https://issues.apache.org/jira/browse/SPARK-2105 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-2105) SparkUI doesn't remove active stages that failed

2014-06-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2105: - Description: If a stage fails because its tasks cannot be serialized, for instance, the failed stage rema

[jira] [Updated] (SPARK-1201) Do not materialize partitions whenever possible in BlockManager

2014-06-13 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1201: - Description: This is a slightly more complex version of SPARK-942 where we try to avoid unrolling iterato

[jira] [Created] (SPARK-2144) SparkUI Executors tab displays incorrect RDD blocks

2014-06-13 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2144: Summary: SparkUI Executors tab displays incorrect RDD blocks Key: SPARK-2144 URL: https://issues.apache.org/jira/browse/SPARK-2144 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-1201) Do not materialize partitions whenever possible in BlockManager

2014-06-13 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14031425#comment-14031425 ] Andrew Or commented on SPARK-1201: -- Depends on when we release 1.0.1. I am actually worki

[jira] [Commented] (SPARK-1201) Do not materialize partitions whenever possible in BlockManager

2014-06-13 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14031431#comment-14031431 ] Andrew Or commented on SPARK-1201: -- It will mainly be a bug fix that doesn't change the A

[jira] [Updated] (SPARK-2147) Master UI forgets about Executors when application exits cleanly

2014-06-16 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2147: - Description: When an application exits cleanly, the Master will remove all executors from the applicatio

[jira] [Created] (SPARK-2159) Spark shell exit() does not stop SparkContext

2014-06-16 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2159: Summary: Spark shell exit() does not stop SparkContext Key: SPARK-2159 URL: https://issues.apache.org/jira/browse/SPARK-2159 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-2161) UI should remember executors that have been removed

2014-06-16 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2161: Summary: UI should remember executors that have been removed Key: SPARK-2161 URL: https://issues.apache.org/jira/browse/SPARK-2161 Project: Spark Issue Type: Improve

[jira] [Updated] (SPARK-1777) Pass "cached" blocks directly to disk if memory is not large enough

2014-06-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1777: - Attachment: spark-1777-design-doc.pdf > Pass "cached" blocks directly to disk if memory is not large enou

[jira] [Updated] (SPARK-2242) Running sc.parallelize(..).count() hangs pyspark

2014-06-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2242: - Priority: Blocker (was: Critical) > Running sc.parallelize(..).count() hangs pyspark > -

[jira] [Created] (SPARK-2258) Worker UI displays zombie executors

2014-06-24 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2258: Summary: Worker UI displays zombie executors Key: SPARK-2258 URL: https://issues.apache.org/jira/browse/SPARK-2258 Project: Spark Issue Type: Bug Component

[jira] [Updated] (SPARK-2258) Worker UI displays zombie executors

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2258: - Attachment: Screen Shot 2014-06-24 at 9.23.18 AM.png > Worker UI displays zombie executors >

[jira] [Created] (SPARK-2259) Spark submit documentation for --deploy-mode is highly misleading

2014-06-24 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2259: Summary: Spark submit documentation for --deploy-mode is highly misleading Key: SPARK-2259 URL: https://issues.apache.org/jira/browse/SPARK-2259 Project: Spark Issu

[jira] [Created] (SPARK-2260) Spark submit standalone-cluster mode is broken

2014-06-24 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2260: Summary: Spark submit standalone-cluster mode is broken Key: SPARK-2260 URL: https://issues.apache.org/jira/browse/SPARK-2260 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-1911) Warn users that jars should be built with Java 6 for YARN

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1911: - Summary: Warn users that jars should be built with Java 6 for YARN (was: Warn users that jars should be

[jira] [Reopened] (SPARK-1911) Warn users that jars should be built with Java 6 for YARN

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reopened SPARK-1911: -- Looks like we still haven't fixed this. At the very least we should explicitly add this warning in the doc

[jira] [Updated] (SPARK-1911) Warn users if their jars are not built with Java 6

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1911: - Fix Version/s: (was: 1.0.0) 1.1.0 > Warn users if their jars are not built with Ja

[jira] [Updated] (SPARK-1911) Warn users if their jars are not built with Java 6

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1911: - Affects Version/s: 1.1.0 > Warn users if their jars are not built with Java 6 > -

[jira] [Updated] (SPARK-1911) Warn users if their jars are not built with Java 6

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1911: - Description: Currently we warn users only in make-distribution.sh, but most users build the jars direc

[jira] [Updated] (SPARK-1911) Warn users that jars should be built with Java 6

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1911: - Summary: Warn users that jars should be built with Java 6 (was: Warn users that jars should be built wit

[jira] [Updated] (SPARK-1911) Warn users if their assembly jars are not built with Java 6

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1911: - Summary: Warn users if their assembly jars are not built with Java 6 (was: Warn users if their jars are

[jira] [Updated] (SPARK-1911) Warn users if their assembly jars are not built with Java 6

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1911: - Description: The root cause of the problem is detailed in: https://issues.apache.org/jira/browse/SPARK-1

[jira] [Updated] (SPARK-1911) Warn users if their jars are not built with Java 6

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1911: - Summary: Warn users if their jars are not built with Java 6 (was: Warn users that jars should be built w

[jira] [Updated] (SPARK-1911) Warn users if their assembly jars are not built with Java 6

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1911: - Description: The root cause of the problem is detailed in: https://issues.apache.org/jira/browse/SPARK-1

[jira] [Updated] (SPARK-1911) Warn users if their assembly jars are not built with Java 6

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1911: - Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-1351) > Warn users if their assembly jar

[jira] [Commented] (SPARK-2259) Spark submit documentation for --deploy-mode is highly misleading

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042811#comment-14042811 ] Andrew Or commented on SPARK-2259: -- https://github.com/apache/spark/pull/1200 > Spark su

[jira] [Updated] (SPARK-2266) Log page on Worker UI displays "Some(app-id)"

2014-06-24 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2266: - Attachment: Screen Shot 2014-06-24 at 5.07.54 PM.png > Log page on Worker UI displays "Some(app-id)" > --

[jira] [Created] (SPARK-2266) Log page on Worker UI displays "Some(app-id)"

2014-06-24 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2266: Summary: Log page on Worker UI displays "Some(app-id)" Key: SPARK-2266 URL: https://issues.apache.org/jira/browse/SPARK-2266 Project: Spark Issue Type: Bug Affec

[jira] [Updated] (SPARK-2258) Worker UI displays zombie executors

2014-06-25 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2258: - Fix Version/s: 1.1.0 > Worker UI displays zombie executors > --- > >

[jira] [Resolved] (SPARK-2258) Worker UI displays zombie executors

2014-06-25 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-2258. -- Resolution: Fixed https://github.com/apache/spark/pull/1213 > Worker UI displays zombie executors > --

[jira] [Created] (SPARK-2296) Refactor util.JsonProtocol for evolvability

2014-06-26 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2296: Summary: Refactor util.JsonProtocol for evolvability Key: SPARK-2296 URL: https://issues.apache.org/jira/browse/SPARK-2296 Project: Spark Issue Type: Bug C

[jira] [Commented] (SPARK-2300) PySpark shell hides stderr output

2014-06-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045342#comment-14045342 ] Andrew Or commented on SPARK-2300: -- https://github.com/apache/spark/pull/1178 > PySpark

[jira] [Created] (SPARK-2300) PySpark shell hides stderr output

2014-06-26 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2300: Summary: PySpark shell hides stderr output Key: SPARK-2300 URL: https://issues.apache.org/jira/browse/SPARK-2300 Project: Spark Issue Type: Bug Components:

[jira] [Resolved] (SPARK-2300) PySpark shell hides stderr output

2014-06-26 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-2300. -- Resolution: Fixed > PySpark shell hides stderr output > - > >

[jira] [Updated] (SPARK-2307) SparkUI Storage page cached statuses incorrect

2014-06-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2307: - Attachment: Screen Shot 2014-06-27 at 11.09.54 AM.png > SparkUI Storage page cached statuses incorrect >

[jira] [Updated] (SPARK-2307) SparkUI Storage page cached statuses incorrect

2014-06-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2307: - Description: See attached: the executor has 512MB, but somehow it has cached (279 + 27 + 279 + 27) = 612M

[jira] [Created] (SPARK-2307) SparkUI Storage page cached statuses incorrect

2014-06-27 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2307: Summary: SparkUI Storage page cached statuses incorrect Key: SPARK-2307 URL: https://issues.apache.org/jira/browse/SPARK-2307 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-1392) Local spark-shell Runs Out of Memory With Default Settings

2014-06-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046374#comment-14046374 ] Andrew Or commented on SPARK-1392: -- and by "fixed by SPARK-1777" he means https://github.

[jira] [Commented] (SPARK-1777) Pass "cached" blocks directly to disk if memory is not large enough

2014-06-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046388#comment-14046388 ] Andrew Or commented on SPARK-1777: -- An easy way reproduce this: run spark shell local mod

[jira] [Comment Edited] (SPARK-1777) Pass "cached" blocks directly to disk if memory is not large enough

2014-06-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046388#comment-14046388 ] Andrew Or edited comment on SPARK-1777 at 6/27/14 8:56 PM: --- An e

[jira] [Comment Edited] (SPARK-1777) Pass "cached" blocks directly to disk if memory is not large enough

2014-06-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046388#comment-14046388 ] Andrew Or edited comment on SPARK-1777 at 6/27/14 8:56 PM: --- An e

[jira] [Commented] (SPARK-1777) Pass "cached" blocks directly to disk if memory is not large enough

2014-06-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046392#comment-14046392 ] Andrew Or commented on SPARK-1777: -- Fix: https://github.com/apache/spark/pull/1165 > Pas

[jira] [Comment Edited] (SPARK-1777) Pass "cached" blocks directly to disk if memory is not large enough

2014-06-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046388#comment-14046388 ] Andrew Or edited comment on SPARK-1777 at 6/27/14 8:56 PM: --- An e

[jira] [Updated] (SPARK-2340) Resolve paths properly for History Server

2014-07-01 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2340: - Description: We resolved paths for --jars and --files for spark submit when releasing Spark 1.0. The path

[jira] [Created] (SPARK-2340) Resolve paths properly for History Server

2014-07-01 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2340: Summary: Resolve paths properly for History Server Key: SPARK-2340 URL: https://issues.apache.org/jira/browse/SPARK-2340 Project: Spark Issue Type: Bug Com

[jira] [Updated] (SPARK-2340) Resolve paths properly for event logging / history server

2014-07-01 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2340: - Summary: Resolve paths properly for event logging / history server (was: Resolve paths properly for Hist

[jira] [Commented] (SPARK-1850) Bad exception if multiple jars exist when running PySpark

2014-07-02 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14050318#comment-14050318 ] Andrew Or commented on SPARK-1850: -- Ye, I will change it. > Bad exception if multiple ja

[jira] [Closed] (SPARK-1850) Bad exception if multiple jars exist when running PySpark

2014-07-02 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-1850. Resolution: Fixed > Bad exception if multiple jars exist when running PySpark > ---

[jira] [Created] (SPARK-2349) Fix NPE in ExternalAppendOnlyMap

2014-07-02 Thread Andrew Or (JIRA)
Andrew Or created SPARK-2349: Summary: Fix NPE in ExternalAppendOnlyMap Key: SPARK-2349 URL: https://issues.apache.org/jira/browse/SPARK-2349 Project: Spark Issue Type: Bug Components:

<    6   7   8   9   10   11   12   13   14   15   >