[jira] [Assigned] (SPARK-10535) Support for recommendUsersForProducts and recommendProductsForUsers in matrix factorization model for PySpark

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10535: Assignee: Apache Spark > Support for recommendUsersForProducts and

[jira] [Assigned] (SPARK-10535) Support for recommendUsersForProducts and recommendProductsForUsers in matrix factorization model for PySpark

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10535: Assignee: (was: Apache Spark) > Support for recommendUsersForProducts and

[jira] [Commented] (SPARK-10535) Support for recommendUsersForProducts and recommendProductsForUsers in matrix factorization model for PySpark

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738947#comment-14738947 ] Apache Spark commented on SPARK-10535: -- User 'smartkiwi' has created a pull request for this issue:

[jira] [Resolved] (SPARK-10497) Release utils does not work with new version of jira-python library

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-10497. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8661

[jira] [Updated] (SPARK-10497) Release utils does not work with new version of jira-python library

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-10497: -- Assignee: holdenk > Release utils does not work with new version of jira-python library >

[jira] [Resolved] (SPARK-2015) Spark UI issues at scale

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-2015. -- Resolution: Done Target Version/s: (was: 1.5.0) This is a stale umbrella, which just had

[jira] [Resolved] (SPARK-2064) web ui should not remove executors if they are dead

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-2064. -- Resolution: Duplicate > web ui should not remove executors if they are dead >

[jira] [Updated] (SPARK-8414) Ensure ContextCleaner actually triggers clean ups

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-8414: - Target Version/s: 1.5.1 (was: 1.5.0) > Ensure ContextCleaner actually triggers clean ups >

[jira] [Updated] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-8119: - Target Version/s: 1.5.1 (was: 1.4.2, 1.5.0) > HeartbeatReceiver should not adjust application executor

[jira] [Updated] (SPARK-8447) Test external shuffle service with all shuffle managers

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-8447: - Target Version/s: 1.5.1 (was: 1.5.0) > Test external shuffle service with all shuffle managers >

[jira] [Updated] (SPARK-10538) java.lang.NegativeArraySizeException during join

2015-09-10 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-10538: --- Attachment: screenshot-1.png > java.lang.NegativeArraySizeException during join >

[jira] [Commented] (SPARK-10493) reduceByKey not returning distinct results

2015-09-10 Thread Glenn Strycker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738794#comment-14738794 ] Glenn Strycker commented on SPARK-10493: Unfortunately we don't have anything past 1.3.0. We're

[jira] [Updated] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-8119: - Target Version/s: 1.4.2, 1.5.1 (was: 1.5.1) > HeartbeatReceiver should not adjust application executor

[jira] [Updated] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-8119: - Target Version/s: 1.4.2 (was: 1.4.2, 1.5.1) > HeartbeatReceiver should not adjust application executor

[jira] [Updated] (SPARK-5659) Flaky test: o.a.s.streaming.ReceiverSuite.block

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-5659: - Target Version/s: (was: 1.5.0) > Flaky test: o.a.s.streaming.ReceiverSuite.block >

[jira] [Commented] (SPARK-10537) Document LIBSVM data source options in public doc and minor improvements

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738891#comment-14738891 ] Apache Spark commented on SPARK-10537: -- User 'mengxr' has created a pull request for this issue:

[jira] [Assigned] (SPARK-10537) Document LIBSVM data source options in public doc and minor improvements

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10537: Assignee: Apache Spark (was: Xiangrui Meng) > Document LIBSVM data source options in

[jira] [Assigned] (SPARK-10537) Document LIBSVM data source options in public doc and minor improvements

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10537: Assignee: Xiangrui Meng (was: Apache Spark) > Document LIBSVM data source options in

[jira] [Created] (SPARK-10538) java.lang.NegativeArraySizeException during join

2015-09-10 Thread JIRA
Maciej Bryński created SPARK-10538: -- Summary: java.lang.NegativeArraySizeException during join Key: SPARK-10538 URL: https://issues.apache.org/jira/browse/SPARK-10538 Project: Spark Issue

[jira] [Updated] (SPARK-10538) java.lang.NegativeArraySizeException during join

2015-09-10 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-10538: --- Description: Hi, I've got a problem during joining tables in PySpark. (in my example 20 of

[jira] [Commented] (SPARK-10382) Make example code in user guide testable

2015-09-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739042#comment-14739042 ] Xiangrui Meng commented on SPARK-10382: --- [~holdenk] I think this might be related to testing

[jira] [Resolved] (SPARK-10065) Avoid triple copy of var-length objects in Array in tungsten projection

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10065. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8496

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-09-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739096#comment-14739096 ] Marcelo Vanzin commented on SPARK-10528: Ok, if you don't have hdfs-site.xml you're probably

[jira] [Commented] (SPARK-9924) checkForLogs and cleanLogs are scheduled at fixed rate and can get piled up

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739114#comment-14739114 ] Apache Spark commented on SPARK-9924: - User 'tgravescs' has created a pull request for this issue:

[jira] [Commented] (SPARK-9924) checkForLogs and cleanLogs are scheduled at fixed rate and can get piled up

2015-09-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739115#comment-14739115 ] Thomas Graves commented on SPARK-9924: -- https://github.com/apache/spark/pull/8701 pr to cherry-pick

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739166#comment-14739166 ] Sean Owen commented on SPARK-10528: --- Despite the error, I wonder if the problem is that you show that

[jira] [Commented] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-09-10 Thread Dan Shechter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738995#comment-14738995 ] Dan Shechter commented on SPARK-8119: - Why was the target version moved to 1.5.1? Wasn't this already

[jira] [Commented] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739030#comment-14739030 ] Sean Owen commented on SPARK-8119: -- No, it's marked as Fixed for 1.5.0 which remains true. I did a bulk

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-09-10 Thread Aliaksei Belablotski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739153#comment-14739153 ] Aliaksei Belablotski commented on SPARK-10528: -- Hi Marcelo, thanks for response. * Yes, I

[jira] [Comment Edited] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-09-10 Thread Aliaksei Belablotski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739153#comment-14739153 ] Aliaksei Belablotski edited comment on SPARK-10528 at 9/10/15 5:10 PM:

[jira] [Commented] (SPARK-10536) filtered POJOs replaced by other instances after collect()

2015-09-10 Thread Erik Schmiegelow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739077#comment-14739077 ] Erik Schmiegelow commented on SPARK-10536: -- We were discussing the implication of InputFormat as

[jira] [Updated] (SPARK-10538) java.lang.NegativeArraySizeException during join

2015-09-10 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-10538: --- Description: Hi, I've got a problem during joining tables in PySpark. (in my example 20 of

[jira] [Commented] (SPARK-10536) filtered POJOs replaced by other instances after collect()

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739056#comment-14739056 ] Sean Owen commented on SPARK-10536: --- Tough one -- I think the behavior is not a surprise to people that

[jira] [Resolved] (SPARK-7142) Minor enhancement to BooleanSimplification Optimizer rule

2015-09-10 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-7142. - Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 5700

[jira] [Updated] (SPARK-7142) Minor enhancement to BooleanSimplification Optimizer rule

2015-09-10 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-7142: Assignee: Yash Datta > Minor enhancement to BooleanSimplification Optimizer rule >

[jira] [Commented] (SPARK-10501) support UUID as an atomic type

2015-09-10 Thread Alex Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739228#comment-14739228 ] Alex Liu commented on SPARK-10501: -- Cassandra has InetAddress and BigInteger types. Can we add those as

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-09-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739302#comment-14739302 ] Marcelo Vanzin commented on SPARK-10528: Thanks! That is really a bug in Hive then: {code}

[jira] [Created] (SPARK-10542) The PySpark 1.5 closure serializer can't serialize a namedtuple instance.

2015-09-10 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10542: -- Summary: The PySpark 1.5 closure serializer can't serialize a namedtuple instance. Key: SPARK-10542 URL: https://issues.apache.org/jira/browse/SPARK-10542 Project:

[jira] [Commented] (SPARK-10539) Intersection Optimization is Wrong

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739363#comment-14739363 ] Josh Rosen commented on SPARK-10539: Per discussion offline, it looks like this might be caused by

[jira] [Updated] (SPARK-10525) Add Python example for VectorSlicer to user guide

2015-09-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-10525: -- Target Version/s: (was: 1.6.0) > Add Python example for VectorSlicer to user guide >

[jira] [Resolved] (SPARK-8167) Tasks that fail due to YARN preemption can cause job failure

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-8167. -- Resolution: Fixed Fix Version/s: 1.6.0 > Tasks that fail due to YARN preemption can cause job

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-09-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739222#comment-14739222 ] Marcelo Vanzin commented on SPARK-10528: Hmmm, that might be a bug in Hive then. Can you execute

[jira] [Commented] (SPARK-10382) Make example code in user guide testable

2015-09-10 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739256#comment-14739256 ] holdenk commented on SPARK-10382: - My experience has mostly been that we just manually cut & paste over

[jira] [Created] (SPARK-10541) Allow ApplicationHistoryProviders to provide their own text when there aren't any complete apps

2015-09-10 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-10541: -- Summary: Allow ApplicationHistoryProviders to provide their own text when there aren't any complete apps Key: SPARK-10541 URL:

[jira] [Assigned] (SPARK-10540) HadoopFsRelationTest's "test all data types" is flaky

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10540: Assignee: Apache Spark > HadoopFsRelationTest's "test all data types" is flaky >

[jira] [Resolved] (SPARK-10527) evaluate debugString only when log level is debug

2015-09-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-10527. --- Resolution: Won't Fix > evaluate debugString only when log level is debug >

[jira] [Created] (SPARK-10544) Serialization of Python namedtuples in functions / closures is broken

2015-09-10 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-10544: -- Summary: Serialization of Python namedtuples in functions / closures is broken Key: SPARK-10544 URL: https://issues.apache.org/jira/browse/SPARK-10544 Project: Spark

[jira] [Updated] (SPARK-10544) Serialization of Python namedtuple subclasses in functions / closures is broken

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-10544: --- Summary: Serialization of Python namedtuple subclasses in functions / closures is broken (was:

[jira] [Updated] (SPARK-10544) Serialization of Python namedtuple subclasses in functions / closures is broken

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-10544: --- Description: The following example works on Spark 1.4.1 but not in 1.5: {code} from collections

[jira] [Created] (SPARK-10540) HadoopFsRelationTest's "test all data types" is flaky

2015-09-10 Thread Yin Huai (JIRA)
Yin Huai created SPARK-10540: Summary: HadoopFsRelationTest's "test all data types" is flaky Key: SPARK-10540 URL: https://issues.apache.org/jira/browse/SPARK-10540 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-09-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739302#comment-14739302 ] Marcelo Vanzin edited comment on SPARK-10528 at 9/10/15 6:29 PM: - Thanks!

[jira] [Resolved] (SPARK-10466) UnsafeRow exception in Sort-Based Shuffle with data spill

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10466. --- Resolution: Fixed Assignee: Cheng Hao Fix Version/s: 1.5.1

[jira] [Resolved] (SPARK-10469) Document tungsten-sort

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10469. --- Resolution: Fixed Assignee: holdenk Fix Version/s: 1.5.1

[jira] [Commented] (SPARK-10525) Add Python example for VectorSlicer to user guide

2015-09-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739364#comment-14739364 ] Joseph K. Bradley commented on SPARK-10525: --- True, that's a good point. I may not assign it

[jira] [Commented] (SPARK-10251) Some internal spark classes are not registered with kryo

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739367#comment-14739367 ] Josh Rosen commented on SPARK-10251: I don't think that we properly test Spark with

[jira] [Created] (SPARK-10543) Peak Execution Memory Quantile should be Pre-task Basis

2015-09-10 Thread Sen Fang (JIRA)
Sen Fang created SPARK-10543: Summary: Peak Execution Memory Quantile should be Pre-task Basis Key: SPARK-10543 URL: https://issues.apache.org/jira/browse/SPARK-10543 Project: Spark Issue Type:

[jira] [Updated] (SPARK-6350) Make mesosExecutorCores configurable in mesos "fine-grained" mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-6350: - Affects Version/s: 1.5.0 > Make mesosExecutorCores configurable in mesos "fine-grained" mode >

[jira] [Resolved] (SPARK-10514) Minimum ratio of registered resources [ spark.scheduler.minRegisteredResourcesRatio] is not enabled for Mesos Coarse Grained mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10514. --- Resolution: Fixed Fix Version/s: 1.6.0 Target Version/s: 1.6.0 > Minimum ratio of

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-09-10 Thread Aliaksei Belablotski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739198#comment-14739198 ] Aliaksei Belablotski commented on SPARK-10528: -- Thanks Sean, Already tried :) Set "Full

[jira] [Commented] (SPARK-10382) Make example code in user guide testable

2015-09-10 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739260#comment-14739260 ] holdenk commented on SPARK-10382: - I'd love to help with this if your looking for people to help :) >

[jira] [Commented] (SPARK-10540) HadoopFsRelationTest's "test all data types" is flaky

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739274#comment-14739274 ] Apache Spark commented on SPARK-10540: -- User 'yhuai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-10540) HadoopFsRelationTest's "test all data types" is flaky

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10540: Assignee: (was: Apache Spark) > HadoopFsRelationTest's "test all data types" is flaky

[jira] [Resolved] (SPARK-6931) python: struct.pack('!q', value) in write_long(value, stream) in serializers.py require int(but doesn't raise exceptions in common cases)

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-6931. --- Resolution: Fixed Fix Version/s: 1.2.3 1.3.2 Issue resolved by pull request

[jira] [Commented] (SPARK-10540) HadoopFsRelationTest's "test all data types" is flaky

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739351#comment-14739351 ] Apache Spark commented on SPARK-10540: -- User 'yhuai' has created a pull request for this issue:

[jira] [Resolved] (SPARK-6350) Make mesosExecutorCores configurable in mesos "fine-grained" mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-6350. -- Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 Target Version/s:

[jira] [Updated] (SPARK-10514) Minimum ratio of registered resources [ spark.scheduler.minRegisteredResourcesRatio] is not enabled for Mesos Coarse Grained mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10514: -- Component/s: (was: Spark Core) Mesos > Minimum ratio of registered resources [ >

[jira] [Created] (SPARK-10545) HiveMetastoreTypes.toMetastoreType should handle interval type

2015-09-10 Thread Yin Huai (JIRA)
Yin Huai created SPARK-10545: Summary: HiveMetastoreTypes.toMetastoreType should handle interval type Key: SPARK-10545 URL: https://issues.apache.org/jira/browse/SPARK-10545 Project: Spark

[jira] [Updated] (SPARK-10514) Minimum ratio of registered resources [ spark.scheduler.minRegisteredResourcesRatio] is not enabled for Mesos Coarse Grained mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10514: -- Assignee: Akash Mishra > Minimum ratio of registered resources [ >

[jira] [Assigned] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10548: Assignee: Andrew Or (was: Apache Spark) > Concurrent execution in SQL does not work >

[jira] [Assigned] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10548: Assignee: Apache Spark (was: Andrew Or) > Concurrent execution in SQL does not work >

[jira] [Updated] (SPARK-10529) When creating multiple HiveContext objects in one jvm, jdbc connections to metastore cann't be released and it may cause PermGen OutOfMemoryError.

2015-09-10 Thread ZhengYaofeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhengYaofeng updated SPARK-10529: - Attachment: (was: IsolatedClientLoader.scala) > When creating multiple HiveContext objects

[jira] [Closed] (SPARK-10514) Minimum ratio of registered resources [ spark.scheduler.minRegisteredResourcesRatio] is not enabled for Mesos Coarse Grained mode

2015-09-10 Thread Akash Mishra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akash Mishra closed SPARK-10514. Patch is successfully merged in master. > Minimum ratio of registered resources [ >

[jira] [Updated] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10548: -- Description: >From the mailing list: {code} future { df1.count() } future { df2.count() }

[jira] [Commented] (SPARK-10554) Potential NPE with ShutdownHook

2015-09-10 Thread Nithin Asokan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739714#comment-14739714 ] Nithin Asokan commented on SPARK-10554: --- The fix could be as easy as checking {{blockManagerId}} is

[jira] [Commented] (SPARK-10251) Some internal spark classes are not registered with kryo

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739781#comment-14739781 ] Josh Rosen commented on SPARK-10251: Just to be clear, my suggestion was that we change the defaults

[jira] [Updated] (SPARK-6350) Make mesosExecutorCores configurable in mesos "fine-grained" mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-6350: - Labels: backport-needed (was: ) > Make mesosExecutorCores configurable in mesos "fine-grained" mode >

[jira] [Reopened] (SPARK-6350) Make mesosExecutorCores configurable in mesos "fine-grained" mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reopened SPARK-6350: -- > Make mesosExecutorCores configurable in mesos "fine-grained" mode >

[jira] [Commented] (SPARK-6350) Make mesosExecutorCores configurable in mesos "fine-grained" mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739645#comment-14739645 ] Andrew Or commented on SPARK-6350: -- re-opening this because it needs to be back ported into branch-1.5 >

[jira] [Updated] (SPARK-6350) Make mesosExecutorCores configurable in mesos "fine-grained" mode

2015-09-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-6350: - Fix Version/s: (was: 1.5.1) > Make mesosExecutorCores configurable in mesos "fine-grained" mode >

[jira] [Created] (SPARK-10552) Connection String for SparkR to Cassandra

2015-09-10 Thread Austin Trombley (JIRA)
Austin Trombley created SPARK-10552: --- Summary: Connection String for SparkR to Cassandra Key: SPARK-10552 URL: https://issues.apache.org/jira/browse/SPARK-10552 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-10251) Some internal spark classes are not registered with kryo

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739781#comment-14739781 ] Josh Rosen edited comment on SPARK-10251 at 9/10/15 10:49 PM: -- Just to be

[jira] [Commented] (SPARK-9213) Improve regular expression performance (via joni)

2015-09-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739920#comment-14739920 ] Reynold Xin commented on SPARK-9213: [~waterman] are you still working on this? It is fine if you are

[jira] [Updated] (SPARK-10532) Added new option to specify "user profile" of AWS credentials in spark/spark-ec2.py

2015-09-10 Thread teramonagi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] teramonagi updated SPARK-10532: --- Description: AWS users want to use "Named Profiles" sometimes. -

[jira] [Updated] (SPARK-10543) Peak Execution Memory Quantile should be Per-task Basis

2015-09-10 Thread Sen Fang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sen Fang updated SPARK-10543: - Summary: Peak Execution Memory Quantile should be Per-task Basis (was: Peak Execution Memory Quantile

[jira] [Commented] (SPARK-10251) Some internal spark classes are not registered with kryo

2015-09-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739788#comment-14739788 ] Josh Rosen commented on SPARK-10251: Ah, right; forgot about that. > Some internal spark classes are

[jira] [Commented] (SPARK-10251) Some internal spark classes are not registered with kryo

2015-09-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739784#comment-14739784 ] Reynold Xin commented on SPARK-10251: - But the tests by default run Java serialization, not Kryo? >

[jira] [Resolved] (SPARK-9043) Serialize key, value and combiner classes in ShuffleDependency

2015-09-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-9043. Resolution: Fixed Fix Version/s: 1.6.0 > Serialize key, value and combiner classes in

[jira] [Commented] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739938#comment-14739938 ] Apache Spark commented on SPARK-10548: -- User 'andrewor14' has created a pull request for this issue:

[jira] [Created] (SPARK-10553) Allow Ctrl-C in pyspark shell to kill running job

2015-09-10 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10553: -- Summary: Allow Ctrl-C in pyspark shell to kill running job Key: SPARK-10553 URL: https://issues.apache.org/jira/browse/SPARK-10553 Project: Spark Issue Type:

[jira] [Created] (SPARK-10554) Potential NPE with ShutdownHook

2015-09-10 Thread Nithin Asokan (JIRA)
Nithin Asokan created SPARK-10554: - Summary: Potential NPE with ShutdownHook Key: SPARK-10554 URL: https://issues.apache.org/jira/browse/SPARK-10554 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-10553) Allow Ctrl-C in pyspark shell to kill running job

2015-09-10 Thread Ashwin Shankar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739783#comment-14739783 ] Ashwin Shankar commented on SPARK-10553: Hi [~davies], this jira is a duplicate of SPARK-8170. >

[jira] [Commented] (SPARK-10556) SBT build explicitly sets Scala version, which can conflict with SBT's own scala version

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739914#comment-14739914 ] Apache Spark commented on SPARK-10556: -- User 'ahirreddy' has created a pull request for this issue:

[jira] [Assigned] (SPARK-10556) SBT build explicitly sets Scala version, which can conflict with SBT's own scala version

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10556: Assignee: Apache Spark > SBT build explicitly sets Scala version, which can conflict with

[jira] [Created] (SPARK-10556) SBT build explicitly sets Scala version, which can conflict with SBT's own scala version

2015-09-10 Thread Ahir Reddy (JIRA)
Ahir Reddy created SPARK-10556: -- Summary: SBT build explicitly sets Scala version, which can conflict with SBT's own scala version Key: SPARK-10556 URL: https://issues.apache.org/jira/browse/SPARK-10556

[jira] [Resolved] (SPARK-4534) With YARN, JavaSparkContext provide to add preferredNodeLocalityData to SparkContext

2015-09-10 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-4534. --- Resolution: Won't Fix As SPARK-2089 is closed as "Won't Fix", also closing this. > With YARN,

[jira] [Commented] (SPARK-10489) GraphX dataframe wrapper

2015-09-10 Thread Michael Malak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739681#comment-14739681 ] Michael Malak commented on SPARK-10489: --- Feynman Liang: Link

[jira] [Updated] (SPARK-10554) Potential NPE with ShutdownHook

2015-09-10 Thread Nithin Asokan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nithin Asokan updated SPARK-10554: -- Description: Originally posted in user mailing list

[jira] [Created] (SPARK-10555) Add INotifyDStream to Spark Streaming

2015-09-10 Thread Vinoth Chandar (JIRA)
Vinoth Chandar created SPARK-10555: -- Summary: Add INotifyDStream to Spark Streaming Key: SPARK-10555 URL: https://issues.apache.org/jira/browse/SPARK-10555 Project: Spark Issue Type: New

[jira] [Closed] (SPARK-10553) Allow Ctrl-C in pyspark shell to kill running job

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10553. -- Resolution: Duplicate > Allow Ctrl-C in pyspark shell to kill running job >

[jira] [Assigned] (SPARK-10556) SBT build explicitly sets Scala version, which can conflict with SBT's own scala version

2015-09-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10556: Assignee: (was: Apache Spark) > SBT build explicitly sets Scala version, which can

<    1   2   3   >