[jira] [Commented] (SPARK-10382) Make example code in user guide testable

2015-09-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740299#comment-14740299 ] Xiangrui Meng commented on SPARK-10382: --- On one hand it would be really nice to aut

[jira] [Assigned] (SPARK-10529) When creating multiple HiveContext objects in one jvm, jdbc connections to metastore cann't be released and it may cause PermGen OutOfMemoryError.

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10529: Assignee: Apache Spark > When creating multiple HiveContext objects in one jvm, jdbc conne

[jira] [Commented] (SPARK-10529) When creating multiple HiveContext objects in one jvm, jdbc connections to metastore cann't be released and it may cause PermGen OutOfMemoryError.

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740291#comment-14740291 ] Apache Spark commented on SPARK-10529: -- User 'GavinGavinNo1' has created a pull requ

[jira] [Assigned] (SPARK-10529) When creating multiple HiveContext objects in one jvm, jdbc connections to metastore cann't be released and it may cause PermGen OutOfMemoryError.

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10529: Assignee: (was: Apache Spark) > When creating multiple HiveContext objects in one jvm,

[jira] [Commented] (SPARK-10557) Publish Spark 1.5.0 on Maven central

2015-09-10 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740288#comment-14740288 ] Jacek Laskowski commented on SPARK-10557: - Also, as part of the release process t

[jira] [Assigned] (SPARK-10559) DataFrame schema ArrayType should accept ResultIterable

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10559: Assignee: (was: Apache Spark) > DataFrame schema ArrayType should accept ResultIterabl

[jira] [Assigned] (SPARK-10559) DataFrame schema ArrayType should accept ResultIterable

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10559: Assignee: Apache Spark > DataFrame schema ArrayType should accept ResultIterable > ---

[jira] [Commented] (SPARK-10559) DataFrame schema ArrayType should accept ResultIterable

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740278#comment-14740278 ] Apache Spark commented on SPARK-10559: -- User 'maver1ck' has created a pull request f

[jira] [Created] (SPARK-10559) DataFrame schema ArrayType should accept ResultIterable

2015-09-10 Thread JIRA
Maciej Bryński created SPARK-10559: -- Summary: DataFrame schema ArrayType should accept ResultIterable Key: SPARK-10559 URL: https://issues.apache.org/jira/browse/SPARK-10559 Project: Spark I

[jira] [Created] (SPARK-10558) Wrong executor state in standalone master because of wrong state transition

2015-09-10 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-10558: --- Summary: Wrong executor state in standalone master because of wrong state transition Key: SPARK-10558 URL: https://issues.apache.org/jira/browse/SPARK-10558 Project: Sp

[jira] [Commented] (SPARK-10538) java.lang.NegativeArraySizeException during join

2015-09-10 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740250#comment-14740250 ] Maciej Bryński commented on SPARK-10538: I tried also to adjust spark.shuffle.sor

[jira] [Commented] (SPARK-10110) StringIndexer lacks of parameter "handleInvalid".

2015-09-10 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740235#comment-14740235 ] Yanbo Liang commented on SPARK-10110: - Since this issue is fixed by SPARK-10027, do y

[jira] [Created] (SPARK-10557) Publish Spark 1.5.0 on Maven central

2015-09-10 Thread Marko Asplund (JIRA)
Marko Asplund created SPARK-10557: - Summary: Publish Spark 1.5.0 on Maven central Key: SPARK-10557 URL: https://issues.apache.org/jira/browse/SPARK-10557 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-6724) Model import/export for FPGrowth

2015-09-10 Thread Meethu Mathew (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740160#comment-14740160 ] Meethu Mathew commented on SPARK-6724: -- [~josephkb] I will take a look into it and up

[jira] [Commented] (SPARK-10552) Connection String for SparkR to Cassandra

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740167#comment-14740167 ] Sean Owen commented on SPARK-10552: --- Can you provide a description? this doesn't specif

[jira] [Updated] (SPARK-10027) Add Python API missing methods for ml.feature

2015-09-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-10027: -- Assignee: Yanbo Liang > Add Python API missing methods for ml.feature > ---

[jira] [Resolved] (SPARK-10027) Add Python API missing methods for ml.feature

2015-09-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10027. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8313 [https://gi

[jira] [Updated] (SPARK-7770) Should GBT validationTol be relative tolerance?

2015-09-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-7770: - Shepherd: Joseph K. Bradley > Should GBT validationTol be relative tolerance? > --

[jira] [Updated] (SPARK-7770) Should GBT validationTol be relative tolerance?

2015-09-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-7770: - Target Version/s: 1.6.0 > Should GBT validationTol be relative tolerance? > --

[jira] [Updated] (SPARK-7770) Should GBT validationTol be relative tolerance?

2015-09-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-7770: - Assignee: Yanbo Liang > Should GBT validationTol be relative tolerance? >

[jira] [Resolved] (SPARK-10023) Unified DecisionTreeParams "checkpointInterval" between Scala and Python API.

2015-09-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10023. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8528 [https://gi

[jira] [Commented] (SPARK-10050) Support collecting data of MapType in DataFrame

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740089#comment-14740089 ] Apache Spark commented on SPARK-10050: -- User 'sun-rui' has created a pull request fo

[jira] [Assigned] (SPARK-10050) Support collecting data of MapType in DataFrame

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10050: Assignee: Apache Spark > Support collecting data of MapType in DataFrame > ---

[jira] [Assigned] (SPARK-10050) Support collecting data of MapType in DataFrame

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10050: Assignee: (was: Apache Spark) > Support collecting data of MapType in DataFrame >

[jira] [Updated] (SPARK-10529) When creating multiple HiveContext objects in one jvm, jdbc connections to metastore cann't be released and it may cause PermGen OutOfMemoryError.

2015-09-10 Thread ZhengYaofeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhengYaofeng updated SPARK-10529: - Attachment: (was: IsolatedClientLoader.scala) > When creating multiple HiveContext objects in

[jira] [Commented] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739938#comment-14739938 ] Apache Spark commented on SPARK-10548: -- User 'andrewor14' has created a pull request

[jira] [Assigned] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10548: Assignee: Apache Spark (was: Andrew Or) > Concurrent execution in SQL does not work > ---

[jira] [Assigned] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10548: Assignee: Andrew Or (was: Apache Spark) > Concurrent execution in SQL does not work > ---

[jira] [Updated] (SPARK-10532) Added new option to specify "user profile" of AWS credentials in spark/spark-ec2.py

2015-09-10 Thread teramonagi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] teramonagi updated SPARK-10532: --- Description: AWS users want to use "Named Profiles" sometimes. - http://docs.aws.amazon.com/cli/lat

[jira] [Commented] (SPARK-9213) Improve regular expression performance (via joni)

2015-09-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739920#comment-14739920 ] Reynold Xin commented on SPARK-9213: [~waterman] are you still working on this? It is

[jira] [Assigned] (SPARK-10556) SBT build explicitly sets Scala version, which can conflict with SBT's own scala version

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10556: Assignee: (was: Apache Spark) > SBT build explicitly sets Scala version, which can con

[jira] [Commented] (SPARK-10556) SBT build explicitly sets Scala version, which can conflict with SBT's own scala version

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739914#comment-14739914 ] Apache Spark commented on SPARK-10556: -- User 'ahirreddy' has created a pull request

[jira] [Assigned] (SPARK-10556) SBT build explicitly sets Scala version, which can conflict with SBT's own scala version

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10556: Assignee: Apache Spark > SBT build explicitly sets Scala version, which can conflict with

[jira] [Created] (SPARK-10556) SBT build explicitly sets Scala version, which can conflict with SBT's own scala version

2015-09-10 Thread Ahir Reddy (JIRA)
Ahir Reddy created SPARK-10556: -- Summary: SBT build explicitly sets Scala version, which can conflict with SBT's own scala version Key: SPARK-10556 URL: https://issues.apache.org/jira/browse/SPARK-10556

[jira] [Resolved] (SPARK-9043) Serialize key, value and combiner classes in ShuffleDependency

2015-09-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-9043. Resolution: Fixed Fix Version/s: 1.6.0 > Serialize key, value and combiner classes in Shuffle

[jira] [Closed] (SPARK-10553) Allow Ctrl-C in pyspark shell to kill running job

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10553. -- Resolution: Duplicate > Allow Ctrl-C in pyspark shell to kill running job > ---

[jira] [Comment Edited] (SPARK-10489) GraphX dataframe wrapper

2015-09-10 Thread Feynman Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735206#comment-14735206 ] Feynman Liang edited comment on SPARK-10489 at 9/10/15 11:52 PM: --

[jira] [Created] (SPARK-10555) Add INotifyDStream to Spark Streaming

2015-09-10 Thread Vinoth Chandar (JIRA)
Vinoth Chandar created SPARK-10555: -- Summary: Add INotifyDStream to Spark Streaming Key: SPARK-10555 URL: https://issues.apache.org/jira/browse/SPARK-10555 Project: Spark Issue Type: New Fea

[jira] [Comment Edited] (SPARK-10251) Some internal spark classes are not registered with kryo

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739781#comment-14739781 ] Josh Rosen edited comment on SPARK-10251 at 9/10/15 10:49 PM: -

[jira] [Commented] (SPARK-10251) Some internal spark classes are not registered with kryo

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739788#comment-14739788 ] Josh Rosen commented on SPARK-10251: Ah, right; forgot about that. > Some internal s

[jira] [Commented] (SPARK-10251) Some internal spark classes are not registered with kryo

2015-09-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739784#comment-14739784 ] Reynold Xin commented on SPARK-10251: - But the tests by default run Java serializatio

[jira] [Commented] (SPARK-10553) Allow Ctrl-C in pyspark shell to kill running job

2015-09-10 Thread Ashwin Shankar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739783#comment-14739783 ] Ashwin Shankar commented on SPARK-10553: Hi [~davies], this jira is a duplicate o

[jira] [Commented] (SPARK-10251) Some internal spark classes are not registered with kryo

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739781#comment-14739781 ] Josh Rosen commented on SPARK-10251: Just to be clear, my suggestion was that we chan

[jira] [Commented] (SPARK-10251) Some internal spark classes are not registered with kryo

2015-09-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739770#comment-14739770 ] Reynold Xin commented on SPARK-10251: - we should have one test suite for that -- just

[jira] [Updated] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10548: -- Description: >From the mailing list: {code} future { df1.count() } future { df2.count() } java.lang.

[jira] [Commented] (SPARK-10554) Potential NPE with ShutdownHook

2015-09-10 Thread Nithin Asokan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739714#comment-14739714 ] Nithin Asokan commented on SPARK-10554: --- The fix could be as easy as checking {{blo

[jira] [Updated] (SPARK-10554) Potential NPE with ShutdownHook

2015-09-10 Thread Nithin Asokan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nithin Asokan updated SPARK-10554: -- Description: Originally posted in user mailing list [here|http://apache-spark-user-list.100156

[jira] [Created] (SPARK-10554) Potential NPE with ShutdownHook

2015-09-10 Thread Nithin Asokan (JIRA)
Nithin Asokan created SPARK-10554: - Summary: Potential NPE with ShutdownHook Key: SPARK-10554 URL: https://issues.apache.org/jira/browse/SPARK-10554 Project: Spark Issue Type: Bug

[jira] [Closed] (SPARK-10514) Minimum ratio of registered resources [ spark.scheduler.minRegisteredResourcesRatio] is not enabled for Mesos Coarse Grained mode

2015-09-10 Thread Akash Mishra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akash Mishra closed SPARK-10514. Patch is successfully merged in master. > Minimum ratio of registered resources [ > spark.scheduler.

[jira] [Created] (SPARK-10553) Allow Ctrl-C in pyspark shell to kill running job

2015-09-10 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10553: -- Summary: Allow Ctrl-C in pyspark shell to kill running job Key: SPARK-10553 URL: https://issues.apache.org/jira/browse/SPARK-10553 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-10489) GraphX dataframe wrapper

2015-09-10 Thread Michael Malak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739681#comment-14739681 ] Michael Malak commented on SPARK-10489: --- Feynman Liang: Link https://github.com/da

[jira] [Updated] (SPARK-10543) Peak Execution Memory Quantile should be Per-task Basis

2015-09-10 Thread Sen Fang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sen Fang updated SPARK-10543: - Summary: Peak Execution Memory Quantile should be Per-task Basis (was: Peak Execution Memory Quantile sh

[jira] [Resolved] (SPARK-4534) With YARN, JavaSparkContext provide to add preferredNodeLocalityData to SparkContext

2015-09-10 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-4534. --- Resolution: Won't Fix As SPARK-2089 is closed as "Won't Fix", also closing this. > With YARN, JavaSpa

[jira] [Reopened] (SPARK-6350) Make mesosExecutorCores configurable in mesos "fine-grained" mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reopened SPARK-6350: -- > Make mesosExecutorCores configurable in mesos "fine-grained" mode > --

[jira] [Commented] (SPARK-6350) Make mesosExecutorCores configurable in mesos "fine-grained" mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739645#comment-14739645 ] Andrew Or commented on SPARK-6350: -- re-opening this because it needs to be back ported in

[jira] [Updated] (SPARK-6350) Make mesosExecutorCores configurable in mesos "fine-grained" mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-6350: - Labels: backport-needed (was: ) > Make mesosExecutorCores configurable in mesos "fine-grained" mode > ---

[jira] [Updated] (SPARK-6350) Make mesosExecutorCores configurable in mesos "fine-grained" mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-6350: - Fix Version/s: (was: 1.5.1) > Make mesosExecutorCores configurable in mesos "fine-grained" mode >

[jira] [Created] (SPARK-10552) Connection String for SparkR to Cassandra

2015-09-10 Thread Austin Trombley (JIRA)
Austin Trombley created SPARK-10552: --- Summary: Connection String for SparkR to Cassandra Key: SPARK-10552 URL: https://issues.apache.org/jira/browse/SPARK-10552 Project: Spark Issue Type: T

[jira] [Commented] (SPARK-10551) Successful task-end event after task failed due to executor loss

2015-09-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739630#comment-14739630 ] Ryan Williams commented on SPARK-10551: --- Something else I just noticed: both of the

[jira] [Updated] (SPARK-10551) Successful task-end event after task failed due to executor loss

2015-09-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-10551: -- Description: Doing forensics on some failed Spark applications and seeing nonsensical things i

[jira] [Commented] (SPARK-10551) Successful task-end event after task failed due to executor loss

2015-09-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739619#comment-14739619 ] Ryan Williams commented on SPARK-10551: --- The same behavior is observable on a secon

[jira] [Commented] (SPARK-10551) Successful task-end event after task failed due to executor loss

2015-09-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739615#comment-14739615 ] Ryan Williams commented on SPARK-10551: --- Here is the full event log: https://www.d

[jira] [Closed] (SPARK-10397) Make Python's SparkContext self-descriptive on "print sc"

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10397. -- Resolution: Won't Fix > Make Python's SparkContext self-descriptive on "print sc" > ---

[jira] [Commented] (SPARK-10397) Make Python's SparkContext self-descriptive on "print sc"

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739603#comment-14739603 ] Davies Liu commented on SPARK-10397: I'd like to stick with current approach, that's

[jira] [Created] (SPARK-10551) Successful task-end event after task failed due to executor loss

2015-09-10 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-10551: - Summary: Successful task-end event after task failed due to executor loss Key: SPARK-10551 URL: https://issues.apache.org/jira/browse/SPARK-10551 Project: Spark

[jira] [Commented] (SPARK-8939) YARN EC2 default setting fails with IllegalArgumentException

2015-09-10 Thread Heji Kim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739578#comment-14739578 ] Heji Kim commented on SPARK-8939: - I was trying to upgrade to 1.5 today and could not subm

[jira] [Resolved] (SPARK-10056) PySpark Row - Support for row["columnName"] syntax

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10056. Resolution: Fixed Assignee: Yanbo Liang Fix Version/s: 1.6.0 > PySpark Row - Suppor

[jira] [Resolved] (SPARK-7544) pyspark.sql.types.Row should implement __getitem__

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-7544. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8333 [https://github.com/

[jira] [Commented] (SPARK-9990) Create local hash join operator

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739567#comment-14739567 ] Apache Spark commented on SPARK-9990: - User 'andrewor14' has created a pull request fo

[jira] [Created] (SPARK-10550) SQLListener error constructing extended SQLContext

2015-09-10 Thread shao lo (JIRA)
shao lo created SPARK-10550: --- Summary: SQLListener error constructing extended SQLContext Key: SPARK-10550 URL: https://issues.apache.org/jira/browse/SPARK-10550 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-10443) Refactor SortMergeOuterJoin to reduce duplication

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10443. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8596 [https://github.c

[jira] [Created] (SPARK-10549) scala 2.11 spark on yarn with security - Repl doesn't work

2015-09-10 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-10549: - Summary: scala 2.11 spark on yarn with security - Repl doesn't work Key: SPARK-10549 URL: https://issues.apache.org/jira/browse/SPARK-10549 Project: Spark

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-09-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739516#comment-14739516 ] Marcelo Vanzin commented on SPARK-10528: I think a lot of code in this area chang

[jira] [Commented] (SPARK-9790) [YARN] Expose in WebUI if NodeManager is the reason why executors were killed.

2015-09-10 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739502#comment-14739502 ] Mark Grover commented on SPARK-9790: I was waiting on SPARK-8167 to get committed. Tha

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-09-10 Thread Aliaksei Belablotski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739499#comment-14739499 ] Aliaksei Belablotski commented on SPARK-10528: -- Thanks a lot Marcelo. Yes, W

[jira] [Updated] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10548: -- Assignee: Andrew Or > Concurrent execution in SQL does not work > -

[jira] [Created] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-10 Thread Andrew Or (JIRA)
Andrew Or created SPARK-10548: - Summary: Concurrent execution in SQL does not work Key: SPARK-10548 URL: https://issues.apache.org/jira/browse/SPARK-10548 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-10544) Serialization of Python namedtuple subclasses in functions / closures is broken

2015-09-10 Thread Doug Bateman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739492#comment-14739492 ] Doug Bateman commented on SPARK-10544: -- This also fails in Spark 1.5 pyspark sc

[jira] [Commented] (SPARK-10056) PySpark Row - Support for row["columnName"] syntax

2015-09-10 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-10056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739482#comment-14739482 ] Maciej Bryński commented on SPARK-10056: [~davies] Is there a chance that PR from

[jira] [Commented] (SPARK-10547) Streamline / improve style of Java API tests

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739464#comment-14739464 ] Apache Spark commented on SPARK-10547: -- User 'srowen' has created a pull request for

[jira] [Assigned] (SPARK-10542) The PySpark 1.5 closure serializer can't serialize a namedtuple instance.

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10542: Assignee: Apache Spark (was: Davies Liu) > The PySpark 1.5 closure serializer can't seri

[jira] [Commented] (SPARK-10542) The PySpark 1.5 closure serializer can't serialize a namedtuple instance.

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739468#comment-14739468 ] Apache Spark commented on SPARK-10542: -- User 'davies' has created a pull request for

[jira] [Assigned] (SPARK-10542) The PySpark 1.5 closure serializer can't serialize a namedtuple instance.

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10542: Assignee: Davies Liu (was: Apache Spark) > The PySpark 1.5 closure serializer can't seri

[jira] [Assigned] (SPARK-10547) Streamline / improve style of Java API tests

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10547: Assignee: Sean Owen (was: Apache Spark) > Streamline / improve style of Java API tests >

[jira] [Assigned] (SPARK-10547) Streamline / improve style of Java API tests

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10547: Assignee: Apache Spark (was: Sean Owen) > Streamline / improve style of Java API tests >

[jira] [Created] (SPARK-10547) Streamline / improve style of Java API tests

2015-09-10 Thread Sean Owen (JIRA)
Sean Owen created SPARK-10547: - Summary: Streamline / improve style of Java API tests Key: SPARK-10547 URL: https://issues.apache.org/jira/browse/SPARK-10547 Project: Spark Issue Type: Improvemen

[jira] [Updated] (SPARK-10049) Support collecting data of ArraryType in DataFrame

2015-09-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-10049: -- Assignee: Sun Rui > Support collecting data of ArraryType in DataFrame > --

[jira] [Resolved] (SPARK-10049) Support collecting data of ArraryType in DataFrame

2015-09-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-10049. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request

[jira] [Assigned] (SPARK-10546) Check partitionId's range in ExternalSorter#spill()

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10546: Assignee: (was: Apache Spark) > Check partitionId's range in ExternalSorter#spill() >

[jira] [Commented] (SPARK-10546) Check partitionId's range in ExternalSorter#spill()

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739411#comment-14739411 ] Apache Spark commented on SPARK-10546: -- User 'tedyu' has created a pull request for

[jira] [Assigned] (SPARK-10546) Check partitionId's range in ExternalSorter#spill()

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10546: Assignee: Apache Spark > Check partitionId's range in ExternalSorter#spill() > ---

[jira] [Created] (SPARK-10546) Check partitionId's range in ExternalSorter#spill()

2015-09-10 Thread Ted Yu (JIRA)
Ted Yu created SPARK-10546: -- Summary: Check partitionId's range in ExternalSorter#spill() Key: SPARK-10546 URL: https://issues.apache.org/jira/browse/SPARK-10546 Project: Spark Issue Type: Improveme

[jira] [Closed] (SPARK-10544) Serialization of Python namedtuple subclasses in functions / closures is broken

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10544. -- Resolution: Duplicate Fix Version/s: (was: 1.5.1) Target Version/s: 1.5.1 > Serial

[jira] [Resolved] (SPARK-9990) Create local hash join operator

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-9990. -- Resolution: Fixed Fix Version/s: 1.6.0 > Create local hash join operator > --

[jira] [Created] (SPARK-10545) HiveMetastoreTypes.toMetastoreType should handle interval type

2015-09-10 Thread Yin Huai (JIRA)
Yin Huai created SPARK-10545: Summary: HiveMetastoreTypes.toMetastoreType should handle interval type Key: SPARK-10545 URL: https://issues.apache.org/jira/browse/SPARK-10545 Project: Spark Issue

[jira] [Updated] (SPARK-10514) Minimum ratio of registered resources [ spark.scheduler.minRegisteredResourcesRatio] is not enabled for Mesos Coarse Grained mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10514: -- Assignee: Akash Mishra > Minimum ratio of registered resources [ > spark.scheduler.minRegisteredResour

[jira] [Updated] (SPARK-10514) Minimum ratio of registered resources [ spark.scheduler.minRegisteredResourcesRatio] is not enabled for Mesos Coarse Grained mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10514: -- Component/s: (was: Spark Core) Mesos > Minimum ratio of registered resources [ >

[jira] [Resolved] (SPARK-10514) Minimum ratio of registered resources [ spark.scheduler.minRegisteredResourcesRatio] is not enabled for Mesos Coarse Grained mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10514. --- Resolution: Fixed Fix Version/s: 1.6.0 Target Version/s: 1.6.0 > Minimum ratio of re

[jira] [Updated] (SPARK-10544) Serialization of Python namedtuple subclasses in functions / closures is broken

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-10544: --- Description: The following example works on Spark 1.4.1 but not in 1.5: {code} from collections impo

[jira] [Updated] (SPARK-10544) Serialization of Python namedtuple subclasses in functions / closures is broken

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-10544: --- Summary: Serialization of Python namedtuple subclasses in functions / closures is broken (was: Seria

  1   2   3   >