[jira] [Resolved] (SPARK-13088) DAG viz does not work with latest version of chrome

2016-01-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-13088. --- Resolution: Fixed Fix Version/s: 2.0.0 1.6.1 1.5.3

[jira] [Resolved] (SPARK-13096) Make AccumulatorSuite#verifyPeakExecutionMemorySet less flaky

2016-01-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-13096. --- Resolution: Fixed Fix Version/s: 2.0.0 > Make AccumulatorSuite#verifyPeakExecutionMemorySet

[jira] [Updated] (SPARK-13096) Make AccumulatorSuite#verifyPeakExecutionMemorySet less flaky

2016-01-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13096: -- Affects Version/s: 1.6.0 Target Version/s: 2.0.0 > Make

[jira] [Created] (SPARK-13096) Make AccumulatorSuite#verifyPeakExecutionMemorySet less flaky

2016-01-29 Thread Andrew Or (JIRA)
Andrew Or created SPARK-13096: - Summary: Make AccumulatorSuite#verifyPeakExecutionMemorySet less flaky Key: SPARK-13096 URL: https://issues.apache.org/jira/browse/SPARK-13096 Project: Spark

[jira] [Resolved] (SPARK-10620) Look into whether accumulator mechanism can replace TaskMetrics

2016-01-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10620. --- Resolution: Fixed Fix Version/s: 2.0.0 > Look into whether accumulator mechanism can replace

[jira] [Created] (SPARK-13088) DAG viz does not work with latest version of chrome

2016-01-29 Thread Andrew Or (JIRA)
Andrew Or created SPARK-13088: - Summary: DAG viz does not work with latest version of chrome Key: SPARK-13088 URL: https://issues.apache.org/jira/browse/SPARK-13088 Project: Spark Issue Type:

[jira] [Updated] (SPARK-13088) DAG viz does not work with latest version of chrome

2016-01-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13088: -- Attachment: Screen Shot 2016-01-29 at 10.54.14 AM.png > DAG viz does not work with latest version of

[jira] [Created] (SPARK-13065) streaming-twitter pass twitter4j.FilterQuery argument to TwitterUtils.createStream()

2016-01-28 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-13065: --- Summary: streaming-twitter pass twitter4j.FilterQuery argument to TwitterUtils.createStream() Key: SPARK-13065 URL: https://issues.apache.org/jira/browse/SPARK-13065

[jira] [Updated] (SPARK-13055) SQLHistoryListener throws ClassCastException

2016-01-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13055: -- Component/s: SQL > SQLHistoryListener throws ClassCastException >

[jira] [Updated] (SPARK-13071) Coalescing HadoopRDD overwrites existing input metrics

2016-01-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13071: -- Description: Currently if we do the following in Hadoop 2.5+: {code} sc.textFile(...,

[jira] [Created] (SPARK-13071) Coalescing HadoopRDD overwrites existing input metrics

2016-01-28 Thread Andrew Or (JIRA)
Andrew Or created SPARK-13071: - Summary: Coalescing HadoopRDD overwrites existing input metrics Key: SPARK-13071 URL: https://issues.apache.org/jira/browse/SPARK-13071 Project: Spark Issue Type:

[jira] [Created] (SPARK-13054) Always post TaskEnd event for tasks in cancelled stages

2016-01-27 Thread Andrew Or (JIRA)
Andrew Or created SPARK-13054: - Summary: Always post TaskEnd event for tasks in cancelled stages Key: SPARK-13054 URL: https://issues.apache.org/jira/browse/SPARK-13054 Project: Spark Issue

[jira] [Created] (SPARK-13053) Rectify ignored tests in InternalAccumulatorSuite

2016-01-27 Thread Andrew Or (JIRA)
Andrew Or created SPARK-13053: - Summary: Rectify ignored tests in InternalAccumulatorSuite Key: SPARK-13053 URL: https://issues.apache.org/jira/browse/SPARK-13053 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-13053) Rectify ignored tests in InternalAccumulatorSuite

2016-01-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13053: -- Target Version/s: 2.0.0 > Rectify ignored tests in InternalAccumulatorSuite >

[jira] [Updated] (SPARK-13053) Rectify ignored tests in InternalAccumulatorSuite

2016-01-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13053: -- Description: {code} // TODO: these two tests are incorrect; they don't actually trigger stage

[jira] [Created] (SPARK-13051) Do not maintain global singleton map for accumulators

2016-01-27 Thread Andrew Or (JIRA)
Andrew Or created SPARK-13051: - Summary: Do not maintain global singleton map for accumulators Key: SPARK-13051 URL: https://issues.apache.org/jira/browse/SPARK-13051 Project: Spark Issue Type:

[jira] [Updated] (SPARK-13053) Rectify ignored tests in InternalAccumulatorSuite

2016-01-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13053: -- Affects Version/s: 2.0.0 > Rectify ignored tests in InternalAccumulatorSuite >

[jira] [Created] (SPARK-13055) SQLHistoryListener throws ClassCastException

2016-01-27 Thread Andrew Or (JIRA)
Andrew Or created SPARK-13055: - Summary: SQLHistoryListener throws ClassCastException Key: SPARK-13055 URL: https://issues.apache.org/jira/browse/SPARK-13055 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-13009) spark-streaming-twitter_2.10 does not make it possible to access the raw twitter json

2016-01-26 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-13009: --- Summary: spark-streaming-twitter_2.10 does not make it possible to access the raw twitter json Key: SPARK-13009 URL: https://issues.apache.org/jira/browse/SPARK-13009

[jira] [Commented] (SPARK-12911) Cacheing a dataframe causes array comparisons to fail (in filter / where) after 1.6

2016-01-20 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109269#comment-15109269 ] Andrew Ray commented on SPARK-12911: In the current master this happens even without caching. The

[jira] [Updated] (SPARK-12790) Remove HistoryServer old multiple files format

2016-01-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12790: -- Assignee: Felix Cheung > Remove HistoryServer old multiple files format >

[jira] [Commented] (SPARK-12790) Remove HistoryServer old multiple files format

2016-01-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107167#comment-15107167 ] Andrew Or commented on SPARK-12790: --- also updating all the tests that rely on the old format. If you

[jira] [Commented] (SPARK-12790) Remove HistoryServer old multiple files format

2016-01-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107169#comment-15107169 ] Andrew Or commented on SPARK-12790: --- I've assigned this to you. > Remove HistoryServer old multiple

[jira] [Resolved] (SPARK-12887) Do not expose var's in TaskMetrics

2016-01-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12887. --- Resolution: Fixed Fix Version/s: 2.0.0 > Do not expose var's in TaskMetrics >

[jira] [Commented] (SPARK-12485) Rename "dynamic allocation" to "elastic scaling"

2016-01-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107151#comment-15107151 ] Andrew Or commented on SPARK-12485: --- [~srowen] to answer your question no I don't feel super strongly

[jira] [Updated] (SPARK-10620) Look into whether accumulator mechanism can replace TaskMetrics

2016-01-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10620: -- Attachment: AccumulatorsandTaskMetricsinSpark2.0.pdf > Look into whether accumulator mechanism can

[jira] [Updated] (SPARK-10620) Look into whether accumulator mechanism can replace TaskMetrics

2016-01-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10620: -- Attachment: accums-and-task-metrics.pdf > Look into whether accumulator mechanism can replace

[jira] [Updated] (SPARK-10620) Look into whether accumulator mechanism can replace TaskMetrics

2016-01-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10620: -- Attachment: (was: AccumulatorsandTaskMetricsinSpark2.0.pdf) > Look into whether accumulator

[jira] [Created] (SPARK-12895) Implement TaskMetrics using accumulators

2016-01-18 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12895: - Summary: Implement TaskMetrics using accumulators Key: SPARK-12895 URL: https://issues.apache.org/jira/browse/SPARK-12895 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-12895) Implement TaskMetrics using accumulators

2016-01-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12895: -- Description: We need to first do this before we can avoid sending TaskMetrics from the executors to

[jira] [Updated] (SPARK-12896) Send only accumulator updates, not TaskMetrics, to the driver

2016-01-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12896: -- Description: Currently in Spark there are two different things that the executors send to the driver,

[jira] [Created] (SPARK-12896) Send only accumulator updates, not TaskMetrics, to the driver

2016-01-18 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12896: - Summary: Send only accumulator updates, not TaskMetrics, to the driver Key: SPARK-12896 URL: https://issues.apache.org/jira/browse/SPARK-12896 Project: Spark

[jira] [Created] (SPARK-12885) Rename 3 fields in ShuffleWriteMetrics

2016-01-18 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12885: - Summary: Rename 3 fields in ShuffleWriteMetrics Key: SPARK-12885 URL: https://issues.apache.org/jira/browse/SPARK-12885 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-10985) Avoid passing evicted blocks throughout BlockManager / CacheManager

2016-01-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10985. --- Resolution: Fixed Assignee: Josh Rosen Fix Version/s: 2.0.0 Target

[jira] [Issue Comment Deleted] (SPARK-10620) Look into whether accumulator mechanism can replace TaskMetrics

2016-01-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10620: -- Comment: was deleted (was: User 'andrewor14' has created a pull request for this issue:

[jira] [Issue Comment Deleted] (SPARK-10620) Look into whether accumulator mechanism can replace TaskMetrics

2016-01-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10620: -- Comment: was deleted (was: User 'andrewor14' has created a pull request for this issue:

[jira] [Created] (SPARK-12887) Do not expose var's in TaskMetrics

2016-01-18 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12887: - Summary: Do not expose var's in TaskMetrics Key: SPARK-12887 URL: https://issues.apache.org/jira/browse/SPARK-12887 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-12884) Move *Metrics and *Accum* classes to their own files

2016-01-18 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12884: - Summary: Move *Metrics and *Accum* classes to their own files Key: SPARK-12884 URL: https://issues.apache.org/jira/browse/SPARK-12884 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-12174) Slow test: BlockManagerSuite."SPARK-9591: getRemoteBytes from another location when Exception throw"

2016-01-14 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12174. --- Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: 2.0.0 > Slow test:

[jira] [Created] (SPARK-12790) Remove HistoryServer old multiple files format

2016-01-12 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12790: - Summary: Remove HistoryServer old multiple files format Key: SPARK-12790 URL: https://issues.apache.org/jira/browse/SPARK-12790 Project: Spark Issue Type:

[jira] [Commented] (SPARK-12627) Remove SparkContext preferred locations constructor

2016-01-05 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084438#comment-15084438 ] Andrew Or commented on SPARK-12627: --- Resolved through https://github.com/apache/spark/pull/10569/files

[jira] [Resolved] (SPARK-12627) Remove SparkContext preferred locations constructor

2016-01-05 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12627. --- Resolution: Fixed Assignee: Reynold Xin (was: Andrew Or) Fix Version/s: 2.0.0 >

[jira] [Created] (SPARK-12627) Remove SparkContext preferred locations constructor

2016-01-04 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12627: - Summary: Remove SparkContext preferred locations constructor Key: SPARK-12627 URL: https://issues.apache.org/jira/browse/SPARK-12627 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-12627) Remove SparkContext preferred locations constructor

2016-01-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reassigned SPARK-12627: - Assignee: Andrew Or > Remove SparkContext preferred locations constructor >

[jira] [Updated] (SPARK-12628) SparkUI: weird formatting on additional metrics tooltip

2016-01-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12628: -- Attachment: Screen Shot 2016-01-04 at 1.18.16 PM.png > SparkUI: weird formatting on additional metrics

[jira] [Created] (SPARK-12628) SparkUI: weird formatting on additional metrics tooltip

2016-01-04 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12628: - Summary: SparkUI: weird formatting on additional metrics tooltip Key: SPARK-12628 URL: https://issues.apache.org/jira/browse/SPARK-12628 Project: Spark Issue

[jira] [Created] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?

2016-01-02 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-12606: --- Summary: Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ? Key: SPARK-12606 URL:

[jira] [Comment Edited] (SPARK-12414) Remove closure serializer

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075814#comment-15075814 ] Andrew Or edited comment on SPARK-12414 at 12/31/15 7:51 AM: - It's also for

[jira] [Commented] (SPARK-12414) Remove closure serializer

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075814#comment-15075814 ] Andrew Or commented on SPARK-12414: --- It's also for code cleanup. Right now SparkEnv has a "closure

[jira] [Assigned] (SPARK-12582) IndexShuffleBlockResolverSuite fails in windows

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reassigned SPARK-12582: - Assignee: Andrew Or > IndexShuffleBlockResolverSuite fails in windows >

[jira] [Updated] (SPARK-12582) IndexShuffleBlockResolverSuite fails in windows

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12582: -- Component/s: (was: Shuffle) Windows Tests >

[jira] [Updated] (SPARK-12582) IndexShuffleBlockResolverSuite fails in windows

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12582: -- Assignee: (was: Apache Spark) Target Version/s: 1.6.1, 2.0.0 >

[jira] [Updated] (SPARK-12582) IndexShuffleBlockResolverSuite fails in windows

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12582: -- Assignee: yucai (was: Andrew Or) > IndexShuffleBlockResolverSuite fails in windows >

[jira] [Updated] (SPARK-12554) Standalone mode may hang if max cores is not a multiple of executor cores

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12554: -- Priority: Minor (was: Major) > Standalone mode may hang if max cores is not a multiple of executor

[jira] [Updated] (SPARK-12554) Standalone mode may hang if max cores is not a multiple of executor cores

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12554: -- Summary: Standalone mode may hang if max cores is not a multiple of executor cores (was: Standalone

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

2015-12-30 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075353#comment-15075353 ] Andrew Or commented on SPARK-12554: --- [~jerrylead] The change you are proposing changes the semantics

[jira] [Updated] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12554: -- Priority: Major (was: Critical) > Standalone app scheduler will hang when app.coreToAssign <

[jira] [Created] (SPARK-12559) Standalone cluster mode doesn't work with --packages

2015-12-29 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12559: - Summary: Standalone cluster mode doesn't work with --packages Key: SPARK-12559 URL: https://issues.apache.org/jira/browse/SPARK-12559 Project: Spark Issue Type:

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074189#comment-15074189 ] Andrew Or commented on SPARK-12554: --- I've downgraded the issue. I also think the behavior is by

[jira] [Updated] (SPARK-12559) Standalone cluster mode doesn't work with --packages

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12559: -- Description: >From the mailing list: {quote} Another problem I ran into that you also might is that

[jira] [Created] (SPARK-12565) Document standalone mode jar distribution more clearly

2015-12-29 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12565: - Summary: Document standalone mode jar distribution more clearly Key: SPARK-12565 URL: https://issues.apache.org/jira/browse/SPARK-12565 Project: Spark Issue Type:

[jira] [Closed] (SPARK-12415) Do not use closure serializer to serialize task result

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-12415. - Resolution: Won't Fix > Do not use closure serializer to serialize task result >

[jira] [Commented] (SPARK-12415) Do not use closure serializer to serialize task result

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074201#comment-15074201 ] Andrew Or commented on SPARK-12415: --- Closed as Won't Fix. We can't use Kryo to serialize task result

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

2015-12-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074198#comment-15074198 ] Andrew Or commented on SPARK-12554: --- If anything I would say it's the calculation of

[jira] [Created] (SPARK-12561) Remove JobLogger

2015-12-29 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12561: - Summary: Remove JobLogger Key: SPARK-12561 URL: https://issues.apache.org/jira/browse/SPARK-12561 Project: Spark Issue Type: Sub-task Components: Spark

[jira] [Updated] (SPARK-12411) Reconsider executor heartbeats rpc timeout

2015-12-23 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12411: -- Target Version/s: 1.6.1, 2.0.0 (was: 2.0.0) > Reconsider executor heartbeats rpc timeout >

[jira] [Updated] (SPARK-12411) Reconsider executor heartbeats rpc timeout

2015-12-23 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12411: -- Fix Version/s: 1.6.1 > Reconsider executor heartbeats rpc timeout >

[jira] [Created] (SPARK-12483) Data Frame as() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-12483: --- Summary: Data Frame as() does not work in Java Key: SPARK-12483 URL: https://issues.apache.org/jira/browse/SPARK-12483 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-12484) DataFrame withColumn() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068696#comment-15068696 ] Andrew Davidson commented on SPARK-12484: - What I am really trying to do is rewrite the following

[jira] [Updated] (SPARK-12485) Rename "dynamic allocation" to "elastic scaling"

2015-12-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12485: -- Target Version/s: 2.0.0 > Rename "dynamic allocation" to "elastic scaling" >

[jira] [Updated] (SPARK-12483) Data Frame as() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Davidson updated SPARK-12483: Attachment: SPARK_12483_unitTest.java added a unit test file > Data Frame as() does not

[jira] [Updated] (SPARK-12484) DataFrame withColumn() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Davidson updated SPARK-12484: Attachment: UDFTest.java Add a unit test file > DataFrame withColumn() does not work in

[jira] [Commented] (SPARK-12484) DataFrame withColumn() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068698#comment-15068698 ] Andrew Davidson commented on SPARK-12484: - releated issue

[jira] [Commented] (SPARK-12483) Data Frame as() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069027#comment-15069027 ] Andrew Davidson commented on SPARK-12483: - Hi Xiao thanks for looking at the issue is there a

[jira] [Commented] (SPARK-12484) DataFrame withColumn() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069061#comment-15069061 ] Andrew Davidson commented on SPARK-12484: - Hi Xiao thanks for looking into this Andy >

[jira] [Updated] (SPARK-12483) Data Frame as() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Davidson updated SPARK-12483: Attachment: SPARK_12483_unitTest.java add a unit test file > Data Frame as() does not

[jira] [Created] (SPARK-12484) DataFrame withColumn() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-12484: --- Summary: DataFrame withColumn() does not work in Java Key: SPARK-12484 URL: https://issues.apache.org/jira/browse/SPARK-12484 Project: Spark Issue

[jira] [Commented] (SPARK-12484) DataFrame withColumn() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068684#comment-15068684 ] Andrew Davidson commented on SPARK-12484: - you can find some more back ground on the email thread

[jira] [Issue Comment Deleted] (SPARK-12483) Data Frame as() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Davidson updated SPARK-12483: Comment: was deleted (was: add a unit test file ) > Data Frame as() does not work in Java

[jira] [Updated] (SPARK-12483) Data Frame as() does not work in Java

2015-12-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Davidson updated SPARK-12483: Attachment: (was: SPARK_12483_unitTest.java) > Data Frame as() does not work in Java >

[jira] [Created] (SPARK-12485) Rename "dynamic allocation" to "elastic scaling"

2015-12-22 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12485: - Summary: Rename "dynamic allocation" to "elastic scaling" Key: SPARK-12485 URL: https://issues.apache.org/jira/browse/SPARK-12485 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-12414) Remove closure serializer

2015-12-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reassigned SPARK-12414: - Assignee: Andrew Or > Remove closure serializer > - > >

[jira] [Commented] (SPARK-12464) Remove spark.deploy.mesos.zookeeper.url and use spark.deploy.zookeeper.url

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067062#comment-15067062 ] Andrew Or commented on SPARK-12464: --- By the way for future reference you probably don't need a separate

[jira] [Created] (SPARK-12466) Harmless Master NPE in tests

2015-12-21 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12466: - Summary: Harmless Master NPE in tests Key: SPARK-12466 URL: https://issues.apache.org/jira/browse/SPARK-12466 Project: Spark Issue Type: Bug Components:

[jira] [Resolved] (SPARK-5882) Add a test for GraphLoader.edgeListFile

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-5882. -- Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.0.0 Target

[jira] [Updated] (SPARK-12414) Remove closure serializer

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12414: -- Issue Type: Sub-task (was: Bug) Parent: SPARK-11806 > Remove closure serializer >

[jira] [Updated] (SPARK-12473) Reuse serializer instances for performance

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12473: -- Description: After commit de02782 of page rank regressed from 242s to 260s, about 7%. Although

[jira] [Commented] (SPARK-12339) NullPointerException on stage kill from web UI

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067154#comment-15067154 ] Andrew Or commented on SPARK-12339: --- I've updated the affected version to 2.0 since SPARK-11206 was

[jira] [Updated] (SPARK-12339) NullPointerException on stage kill from web UI

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12339: -- Affects Version/s: (was: 1.6.0) 2.0.0 > NullPointerException on stage kill

[jira] [Resolved] (SPARK-12392) Optimize a location order of broadcast blocks by considering preferred local hosts

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12392. --- Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.0.0 Target

[jira] [Updated] (SPARK-12473) Reuse serializer instances for performance

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12473: -- Description: After commit de02782 of page rank regressed from 242s to 260s, about 7%. Although

[jira] [Resolved] (SPARK-12466) Harmless Master NPE in tests

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12466. --- Resolution: Fixed > Harmless Master NPE in tests > > >

[jira] [Resolved] (SPARK-2331) SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T]

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-2331. -- Resolution: Fixed Fix Version/s: 2.0.0 > SparkContext.emptyRDD should return RDD[T] not

[jira] [Resolved] (SPARK-12339) NullPointerException on stage kill from web UI

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12339. --- Resolution: Fixed Assignee: Alex Bozarth (was: Apache Spark) Fix Version/s:

[jira] [Updated] (SPARK-12473) Reuse serializer instances for performance

2015-12-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12473: -- Description: After commit de02782 of page rank regressed from 242s to 260s, about 7%. The commit

[jira] [Created] (SPARK-12473) Reuse serializer instances for performance

2015-12-21 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12473: - Summary: Reuse serializer instances for performance Key: SPARK-12473 URL: https://issues.apache.org/jira/browse/SPARK-12473 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-11097) Add connection established callback to lower level RPC layer so we don't need to check for new connections in NettyRpcHandler.receive

2015-12-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-11097. --- Resolution: Fixed Fix Version/s: 2.0.0 > Add connection established callback to lower level

[jira] [Updated] (SPARK-11097) Add connection established callback to lower level RPC layer so we don't need to check for new connections in NettyRpcHandler.receive

2015-12-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-11097: -- Assignee: Shixiong Zhu (was: Apache Spark) > Add connection established callback to lower level RPC

[jira] [Resolved] (SPARK-12411) Reconsider executor heartbeats rpc timeout

2015-12-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12411. --- Resolution: Fixed Assignee: Nong Li Fix Version/s: 2.0.0 Target

[jira] [Updated] (SPARK-12413) Mesos ZK persistence throws a NotSerializableException

2015-12-18 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12413: -- Fix Version/s: (was: 2.0.0) > Mesos ZK persistence throws a NotSerializableException >

<    4   5   6   7   8   9   10   11   12   13   >