[jira] [Commented] (SPARK-10631) Add missing API doc in pyspark.mllib.linalg.Vector

2015-09-15 Thread Vinod KC (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746994#comment-14746994 ] Vinod KC commented on SPARK-10631: -- I'm working on this > Add missing API doc in pyspar

[jira] [Commented] (SPARK-10630) createDataFrame from a Java List

2015-09-15 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746968#comment-14746968 ] holdenk commented on SPARK-10630: - Sounds good to me :) I'll give it a shot :) > createD

[jira] [Resolved] (SPARK-10516) Add values as a property to DenseVector in PySpark

2015-09-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10516. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8682 [https://gi

[jira] [Created] (SPARK-10631) Add missing API doc in pyspark.mllib.linalg.Vector

2015-09-15 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-10631: - Summary: Add missing API doc in pyspark.mllib.linalg.Vector Key: SPARK-10631 URL: https://issues.apache.org/jira/browse/SPARK-10631 Project: Spark Issue Ty

[jira] [Created] (SPARK-10630) createDataFrame from a Java List

2015-09-15 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-10630: - Summary: createDataFrame from a Java List Key: SPARK-10630 URL: https://issues.apache.org/jira/browse/SPARK-10630 Project: Spark Issue Type: New Feature

[jira] [Comment Edited] (SPARK-10474) Aggregation failed with unable to acquire memory

2015-09-15 Thread Yi Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746929#comment-14746929 ] Yi Zhou edited comment on SPARK-10474 at 9/16/15 5:55 AM: -- BTW,

[jira] [Commented] (SPARK-10474) Aggregation failed with unable to acquire memory

2015-09-15 Thread Yi Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746929#comment-14746929 ] Yi Zhou commented on SPARK-10474: - BTW, the "spark.shuffle.safetyFraction" is not public

[jira] [Commented] (SPARK-10474) Aggregation failed with unable to acquire memory

2015-09-15 Thread Yi Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746881#comment-14746881 ] Yi Zhou commented on SPARK-10474: - Thanks [~chenghao]. It's better not to throw such exce

[jira] [Commented] (SPARK-9963) ML RandomForest cleanup: replace predictNodeIndex with predictImpl

2015-09-15 Thread Luvsandondov Lkhamsuren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746813#comment-14746813 ] Luvsandondov Lkhamsuren commented on SPARK-9963: Implementing {code:title

[jira] [Updated] (SPARK-10629) Gradient boosted trees: mapPartitions input size increasing

2015-09-15 Thread Wenmin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenmin Wu updated SPARK-10629: -- Description: First of all, I think my problem is quite different from https://issues.apache.org/jira/b

[jira] [Resolved] (SPARK-10595) Various ML programming guide cleanups post 1.5

2015-09-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10595. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8752 [https://gi

[jira] [Updated] (SPARK-9078) Use of non-standard LIMIT keyword in JDBC tableExists code

2015-09-15 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-9078: Assignee: Suresh Thalamati > Use of non-standard LIMIT keyword in JDBC tableExists code > --

[jira] [Updated] (SPARK-9078) Use of non-standard LIMIT keyword in JDBC tableExists code

2015-09-15 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-9078: Assignee: (was: Suresh Thalamati) > Use of non-standard LIMIT keyword in JDBC tableExists code > ---

[jira] [Resolved] (SPARK-9078) Use of non-standard LIMIT keyword in JDBC tableExists code

2015-09-15 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-9078. - Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8676 [https://github.com/apac

[jira] [Updated] (SPARK-9078) Use of non-standard LIMIT keyword in JDBC tableExists code

2015-09-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-9078: --- Assignee: Suresh Thalamati > Use of non-standard LIMIT keyword in JDBC tableExists code >

[jira] [Commented] (SPARK-10577) [PySpark] DataFrame hint for broadcast join

2015-09-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746692#comment-14746692 ] Apache Spark commented on SPARK-10577: -- User 'Jianfeng-chs' has created a pull reque

[jira] [Updated] (SPARK-10629) Gradient boosted trees: mapPartitions input size increasing

2015-09-15 Thread Wenmin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenmin Wu updated SPARK-10629: -- Description: First of all, I think my problem is quite different from https://issues.apache.org/jira/b

[jira] [Updated] (SPARK-10629) Gradient boosted trees: mapPartitions input size increasing

2015-09-15 Thread Wenmin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenmin Wu updated SPARK-10629: -- Description: First of all, I think my problem is quite different from https://issues.apache.org/jira/b

[jira] [Created] (SPARK-10629) Gradient boosted trees: mapPartitions input size increasing

2015-09-15 Thread Wenmin Wu (JIRA)
Wenmin Wu created SPARK-10629: - Summary: Gradient boosted trees: mapPartitions input size increasing Key: SPARK-10629 URL: https://issues.apache.org/jira/browse/SPARK-10629 Project: Spark Issue

[jira] [Updated] (SPARK-8673) Launcher: add support for monitoring launched applications

2015-09-15 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-8673: - Assignee: Marcelo Vanzin > Launcher: add support for monitoring launched applications > --

[jira] [Commented] (SPARK-10584) Documentation about the compatible Hive version is wrong.

2015-09-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746660#comment-14746660 ] Apache Spark commented on SPARK-10584: -- User 'sarutak' has created a pull request fo

[jira] [Updated] (SPARK-10584) Documentation about the compatible Hive version is wrong.

2015-09-15 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-10584: --- Description: In Spark 1.5.0, Spark SQL is compatible with Hive 0.12.0 through 1.2.1 but the

[jira] [Updated] (SPARK-10584) Documentation about the compatible Hive version is wrong.

2015-09-15 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-10584: --- Summary: Documentation about the compatible Hive version is wrong. (was: Documentation about

[jira] [Created] (SPARK-10628) Add support for arbitrary RandomRDD generation to PySparkAPI

2015-09-15 Thread holdenk (JIRA)
holdenk created SPARK-10628: --- Summary: Add support for arbitrary RandomRDD generation to PySparkAPI Key: SPARK-10628 URL: https://issues.apache.org/jira/browse/SPARK-10628 Project: Spark Issue Typ

[jira] [Commented] (SPARK-4226) SparkSQL - Add support for subqueries in predicates

2015-09-15 Thread Cheng Hao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746642#comment-14746642 ] Cheng Hao commented on SPARK-4226: -- Thank you [~brooks], you're right! I meant it will ma

[jira] [Commented] (SPARK-10627) Regularization for artificial neural networks

2015-09-15 Thread Alexander Ulanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746636#comment-14746636 ] Alexander Ulanov commented on SPARK-10627: -- Dropout WIP refactoring for the new

[jira] [Created] (SPARK-10627) Regularization for artificial neural networks

2015-09-15 Thread Alexander Ulanov (JIRA)
Alexander Ulanov created SPARK-10627: Summary: Regularization for artificial neural networks Key: SPARK-10627 URL: https://issues.apache.org/jira/browse/SPARK-10627 Project: Spark Issue T

[jira] [Resolved] (SPARK-7192) Pyspark casts hive bigint to int

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-7192. --- Resolution: Not A Problem I'm resolving as "Not a Problem." Please comment / re-open if you can explai

[jira] [Created] (SPARK-10626) Create a Java friendly method for randomRDD & RandomDataGenerator on RandomRDDs.

2015-09-15 Thread holdenk (JIRA)
holdenk created SPARK-10626: --- Summary: Create a Java friendly method for randomRDD & RandomDataGenerator on RandomRDDs. Key: SPARK-10626 URL: https://issues.apache.org/jira/browse/SPARK-10626 Project: Spark

[jira] [Resolved] (SPARK-10381) Infinite loop when OutputCommitCoordination is enabled and OutputCommitter.commitTask throws exception

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-10381. Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 Issue resolved by pull reque

[jira] [Commented] (SPARK-8786) Create a wrapper for BinaryType

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746588#comment-14746588 ] Josh Rosen commented on SPARK-8786: --- And to confirm, the case given above does work in 1

[jira] [Resolved] (SPARK-8786) Create a wrapper for BinaryType

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-8786. --- Resolution: Fixed Fix Version/s: 1.5.0 This should have been fixed in 1.5.0. I believe that it

[jira] [Resolved] (SPARK-10624) TakeOrderedAndProjectNode output is not ordered

2015-09-15 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10624. --- Resolution: Fixed Fix Version/s: 1.6.0 > TakeOrderedAndProjectNode output is not ordered > ---

[jira] [Resolved] (SPARK-10613) Reduce LocalNode tests dependency on SQLContext

2015-09-15 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10613. --- Resolution: Fixed Fix Version/s: 1.6.0 > Reduce LocalNode tests dependency on SQLContext > ---

[jira] [Updated] (SPARK-9642) LinearRegression should supported weighted data

2015-09-15 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-9642: --- Shepherd: DB Tsai > LinearRegression should supported weighted data >

[jira] [Resolved] (SPARK-10575) Wrap RDD.takeSample with scope

2015-09-15 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10575. --- Resolution: Fixed Fix Version/s: 1.6.0 > Wrap RDD.takeSample with scope >

[jira] [Commented] (SPARK-10504) aggregate where NULL is defined as the value expression aborts when SUM used

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746524#comment-14746524 ] Josh Rosen commented on SPARK-10504: [~marmbrus], I just checked and this is fixed in

[jira] [Updated] (SPARK-10504) aggregate where NULL is defined as the value expression aborts when SUM used

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-10504: --- Affects Version/s: 1.4.1 > aggregate where NULL is defined as the value expression aborts when SUM us

[jira] [Updated] (SPARK-10503) incorrect predicate evaluation involving NULL value

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-10503: --- Description: Query an ORC table in Hive using the following SQL statement via the SPARKSQL thrift-se

[jira] [Updated] (SPARK-9642) LinearRegression should supported weighted data

2015-09-15 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-9642: --- Assignee: Meihua Wu > LinearRegression should supported weighted data > --

[jira] [Resolved] (SPARK-10612) Add prepare to LocalNode

2015-09-15 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10612. --- Resolution: Fixed Fix Version/s: 1.6.0 > Add prepare to LocalNode > >

[jira] [Resolved] (SPARK-10508) incorrect evaluation of searched case expression

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-10508. Resolution: Fixed Fix Version/s: 1.4.1 1.5.0 > incorrect evaluation of se

[jira] [Commented] (SPARK-10508) incorrect evaluation of searched case expression

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746515#comment-14746515 ] Josh Rosen commented on SPARK-10508: I managed to reproduce this on 1.3.1 as well, us

[jira] [Resolved] (SPARK-10548) Concurrent execution in SQL does not work

2015-09-15 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10548. --- Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 > Concurrent execution in SQL

[jira] [Commented] (SPARK-10563) SparkContext's local properties should be cloned when inherited

2015-09-15 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746512#comment-14746512 ] Andrew Or commented on SPARK-10563: --- NOTE: there's a subtle difference in behavior betw

[jira] [Resolved] (SPARK-10563) SparkContext's local properties should be cloned when inherited

2015-09-15 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-10563. --- Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 > SparkContext's local propert

[jira] [Updated] (SPARK-10508) incorrect evaluation of searched case expression

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-10508: --- Description: The following case expression never evaluates to 'test1' when cdec is -1 or 10 as it wi

[jira] [Updated] (SPARK-10538) java.lang.NegativeArraySizeException during join

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-10538: --- Target Version/s: 1.5.1 > java.lang.NegativeArraySizeException during join >

[jira] [Resolved] (SPARK-10460) fieldIndex method missing on spark.sql.Row

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-10460. Resolution: Cannot Reproduce Row definitely has a fieldIndex method as of 1.4.1 (https://github.co

[jira] [Resolved] (SPARK-5919) Enable broadcast joins for Parquet files

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-5919. --- Resolution: Fixed This is now supported; ParquetRelation now implements proper statistics support. >

[jira] [Resolved] (SPARK-6321) Adapt the number of partitions used by the Exchange rule to the cluster specifications

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-6321. --- Resolution: Won't Fix I'm going to resolve this specific issue as "Won't Fix." While we do plan to do

[jira] [Commented] (SPARK-10625) Spark SQL JDBC read/write is unable to handle JDBC Drivers that adds unserializable objects into connection properties

2015-09-15 Thread Peng Cheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746450#comment-14746450 ] Peng Cheng commented on SPARK-10625: branch: https://github.com/Schedule1/spark/tree/

[jira] [Commented] (SPARK-6715) Eliminate duplicate filters from pushdown predicates

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746442#comment-14746442 ] Josh Rosen commented on SPARK-6715: --- Per discussion on the PR, I believe that this is "W

[jira] [Resolved] (SPARK-6102) Create a SparkSQL DataSource API implementation for Redshift

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-6102. --- Resolution: Done The spark-redshift library did this: https://github.com/databricks/spark-redshift >

[jira] [Created] (SPARK-10625) Spark SQL JDBC read/write is unable to handle JDBC Drivers that adds unserializable objects into connection properties

2015-09-15 Thread Peng Cheng (JIRA)
Peng Cheng created SPARK-10625: -- Summary: Spark SQL JDBC read/write is unable to handle JDBC Drivers that adds unserializable objects into connection properties Key: SPARK-10625 URL: https://issues.apache.org/jira/br

[jira] [Commented] (SPARK-10300) Use tags to control which tests to run depending on changes being tested

2015-09-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746429#comment-14746429 ] Apache Spark commented on SPARK-10300: -- User 'vanzin' has created a pull request for

[jira] [Resolved] (SPARK-5624) Can't find new column

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-5624. --- Resolution: Fixed Resolving as "Fixed" per claim that this works in a newer release. > Can't find new

[jira] [Resolved] (SPARK-2337) String Interpolation for SparkSQL queries

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2337. --- Resolution: Won't Fix Resolving as "Won't Fix" per PR discussion. > String Interpolation for SparkSQL

[jira] [Commented] (SPARK-10624) TakeOrderedAndProjectNode output is not ordered

2015-09-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746419#comment-14746419 ] Apache Spark commented on SPARK-10624: -- User 'andrewor14' has created a pull request

[jira] [Resolved] (SPARK-4576) Add concatenation operator

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4576. --- Resolution: Won't Fix Going to resolve this as "Won't Fix" for now, since it seems like {{concat}} ha

[jira] [Assigned] (SPARK-10624) TakeOrderedAndProjectNode output is not ordered

2015-09-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10624: Assignee: Andrew Or (was: Apache Spark) > TakeOrderedAndProjectNode output is not ordered

[jira] [Assigned] (SPARK-10624) TakeOrderedAndProjectNode output is not ordered

2015-09-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10624: Assignee: Apache Spark (was: Andrew Or) > TakeOrderedAndProjectNode output is not ordered

[jira] [Resolved] (SPARK-7685) Handle high imbalanced data and apply weights to different samples in Logistic Regression

2015-09-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-7685. -- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 7884 [https://githu

[jira] [Updated] (SPARK-4684) Add a script to run JDBC server on Windows

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4684: -- Component/s: Windows > Add a script to run JDBC server on Windows >

[jira] [Resolved] (SPARK-4450) SparkSQL producing incorrect answer when using --master yarn

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4450. --- Resolution: Cannot Reproduce Resolving as "Cannot Reproduce" since this is targeted against an old Spa

[jira] [Updated] (SPARK-4450) SparkSQL producing incorrect answer when using --master yarn

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4450: -- Description: A simple summary program using spark-submit --master local MyJob.py vs. spark-submit -

[jira] [Created] (SPARK-10624) TakeOrderedAndProjectNode output is not ordered

2015-09-15 Thread Andrew Or (JIRA)
Andrew Or created SPARK-10624: - Summary: TakeOrderedAndProjectNode output is not ordered Key: SPARK-10624 URL: https://issues.apache.org/jira/browse/SPARK-10624 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-9033) scala.MatchError: interface java.util.Map (of class java.lang.Class) with Spark SQL

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-9033. --- Resolution: Fixed Fix Version/s: 1.4.0 This should have been fixed for maps and arrays in SPARK

[jira] [Updated] (SPARK-9033) scala.MatchError: interface java.util.Map (of class java.lang.Class) with Spark SQL

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-9033: -- Description: I've a java.util.Map field in a POJO class and I'm trying to use it to createDataFrame (1.

[jira] [Resolved] (SPARK-7175) Upgrade Hive to 1.1.0

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-7175. --- Resolution: Duplicate We added support for connecting to Hive 1.1 in SPARK-8067 > Upgrade Hive to 1.1

[jira] [Updated] (SPARK-6942) Umbrella: UI Visualizations for Core and Dataframes

2015-09-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6942: --- Assignee: Andrew Or (was: Patrick Wendell) > Umbrella: UI Visualizations for Core and Datafra

[jira] [Commented] (SPARK-6513) Add zipWithUniqueId (and other RDD APIs) to RDDApi

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746365#comment-14746365 ] Josh Rosen commented on SPARK-6513: --- [~marmbrus], safe to say that this is "Won't Fix" g

[jira] [Resolved] (SPARK-9032) scala.MatchError in DataFrameReader.json(String path)

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-9032. --- Resolution: Fixed Fix Version/s: 1.4.1 Just confirmed that this is fixed in 1.4.1. > scala.Mat

[jira] [Updated] (SPARK-9032) scala.MatchError in DataFrameReader.json(String path)

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-9032: -- Description: Executing read().json() of SQLContext e.g. DataFrameReader raises a MatchError with a stac

[jira] [Resolved] (SPARK-9188) make open hash set/table APIs public

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-9188. --- Resolution: Won't Fix This has been proposed previously. I'm closing this issue as "Won't Fix" becaus

[jira] [Resolved] (SPARK-6632) Optimize the parquetSchema to metastore schema reconciliation, so that the process is delegated to each map task itself

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-6632. - Resolution: Fixed Fix Version/s: 1.5.0 Starting with Spark 1.5 I believe all footer

[jira] [Commented] (SPARK-8152) Dataframe Join Ignores Condition

2015-09-15 Thread Eric Doi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746337#comment-14746337 ] Eric Doi commented on SPARK-8152: - Thanks. Will reopen if I'm able to reproduce in a clea

[jira] [Closed] (SPARK-2789) Apply names to RDD to becoming SchemaRDD

2015-09-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen closed SPARK-2789. - Resolution: Won't Fix > Apply names to RDD to becoming SchemaRDD > ---

[jira] [Commented] (SPARK-5391) SparkSQL fails to create tables with custom JSON SerDe

2015-09-15 Thread David Ross (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746320#comment-14746320 ] David Ross commented on SPARK-5391: --- Haven't tried native JSON but looks promising, so t

[jira] [Resolved] (SPARK-5194) ADD JAR doesn't update classpath until reconnect

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5194. - Resolution: Cannot Reproduce Closing as cannot reproduce. Please reopen if you can on Spa

[jira] [Resolved] (SPARK-5236) java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.MutableAny cannot be cast to org.apache.spark.sql.catalyst.expressions.MutableInt

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5236. - Resolution: Cannot Reproduce Closing as cannot reproduce, please reopen if you can. > jav

[jira] [Resolved] (SPARK-5305) Using a field in a WHERE clause that is not in the schema does not throw an exception.

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5305. - Resolution: Cannot Reproduce Marking as cannot reproduce, please reopen if you can. > Usi

[jira] [Resolved] (SPARK-5302) Add support for SQLContext "partition" columns

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5302. - Resolution: Fixed Fix Version/s: 1.4.0 This is supported now. > Add support for SQ

[jira] [Resolved] (SPARK-5306) Support for a NotEqualsFilter in the filter PrunedFilteredScan pushdown

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5306. - Resolution: Fixed Fix Version/s: 1.3.0 We added {{Not}} in Spark 1.3. > Support fo

[jira] [Resolved] (SPARK-5314) java.lang.OutOfMemoryError in SparkSQL with GROUP BY

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5314. - Resolution: Fixed Fix Version/s: 1.5.0 1.5 now uses sort based aggregation with spi

[jira] [Commented] (SPARK-5391) SparkSQL fails to create tables with custom JSON SerDe

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746285#comment-14746285 ] Michael Armbrust commented on SPARK-5391: - Is this still a problem? Is there a re

[jira] [Resolved] (SPARK-5397) Assigning aliases to several return values of an UDF

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5397. - Resolution: Fixed Fix Version/s: 1.5.0 Tested in 1.5 and this query now parses corr

[jira] [Resolved] (SPARK-5410) Error parsing scientific notation in a select statement

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5410. - Resolution: Fixed I tested and in Spark 1.5 the HiveContext can parse {{2.2E10}}. HiveCon

[jira] [Resolved] (SPARK-5421) SparkSql throw OOM at shuffle

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5421. - Resolution: Fixed Fix Version/s: 1.5.0 Spark 1.5 should have taken care of this. P

[jira] [Commented] (SPARK-8165) sqlContext.createDataFrame(dataWithoutHeader, csvSchema) type conversion error after .cache

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746269#comment-14746269 ] Michael Armbrust commented on SPARK-8165: - I'd suggest looking at http://spark-pa

[jira] [Resolved] (SPARK-8109) TestSQLContext's static initialization is run during MiMa tests, causing SparkContexts to be created

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-8109. - Resolution: Fixed I think [~andrewor14] has removed TestSQLContext so I'm closing this iss

[jira] [Resolved] (SPARK-8165) sqlContext.createDataFrame(dataWithoutHeader, csvSchema) type conversion error after .cache

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-8165. - Resolution: Not A Problem This is perhaps confusing, but the issue here is that caching an

[jira] [Resolved] (SPARK-8152) Dataframe Join Ignores Condition

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-8152. - Resolution: Cannot Reproduce I can't reproduce this in Spark 1.5, but please reopen if you

[jira] [Resolved] (SPARK-4794) Wrong parse of GROUP BY query

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-4794. - Resolution: Cannot Reproduce I think we have fixed our resolution logic here, but please r

[jira] [Resolved] (SPARK-4838) StackOverflowError when serialization task

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-4838. - Resolution: Cannot Reproduce I'm going to close this barring any new information on how to

[jira] [Resolved] (SPARK-4869) The variable names in IF statement of Spark SQL doesn't resolve to its value.

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-4869. - Resolution: Cannot Reproduce I can't reproduce this in Spark 1.5. Please reopen if you ca

[jira] [Resolved] (SPARK-4278) SparkSQL job failing with java.lang.ClassCastException

2015-09-15 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-4278. - Resolution: Fixed Assignee: Yin Huai Fix Version/s: 1.5.0 In Spark 1.5, if the type of dat

[jira] [Resolved] (SPARK-5030) Approximated cardinality with HyperLogLog UDAF

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5030. - Resolution: Duplicate > Approximated cardinality with HyperLogLog UDAF > -

[jira] [Resolved] (SPARK-5060) Spark driver main thread hanging after SQL insert in Parquet file

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5060. - Resolution: Cannot Reproduce This code has changed a lot in Spark 1.5, so I'm going to clo

[jira] [Resolved] (SPARK-5109) Loading multiple parquet files into a single SchemaRDD

2015-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5109. - Resolution: Fixed Fix Version/s: 1.5.0 Methods for passing a list of files is now s

  1   2   3   >