[jira] [Commented] (SPARK-10942) Not all cached RDDs are unpersisted

2015-10-06 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-10942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946316#comment-14946316 ] Jean-Baptiste Onofré commented on SPARK-10942: -- Let me try it on one of my environment. >

[jira] [Updated] (SPARK-10967) Incorrect Join behavior in filter conditions

2015-10-06 Thread RaviShankar KS (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RaviShankar KS updated SPARK-10967: --- Target Version/s: 1.5.1, 1.4.1 (was: 1.5.1, 1.6.0) > Incorrect Join behavior in filter

[jira] [Updated] (SPARK-10967) Incorrect Join behavior in filter conditions

2015-10-06 Thread RaviShankar KS (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RaviShankar KS updated SPARK-10967: --- Environment: RHEL (was: Ubuntu on AWS) > Incorrect Join behavior in filter conditions >

[jira] [Updated] (SPARK-10967) Incorrect Join behavior in filter conditions

2015-10-06 Thread RaviShankar KS (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RaviShankar KS updated SPARK-10967: --- Description: We notice that the join conditions are not working as expected in the case of

[jira] [Updated] (SPARK-10967) Incorrect Join behavior in filter conditions

2015-10-06 Thread RaviShankar KS (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RaviShankar KS updated SPARK-10967: --- Description: We notice that the join conditions are not working as expected in the case of

[jira] [Updated] (SPARK-10967) Incorrect Join behavior in filter conditions

2015-10-06 Thread RaviShankar KS (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RaviShankar KS updated SPARK-10967: --- Description: We notice that the join conditions are not working as expected in the case of

[jira] [Updated] (SPARK-10967) Incorrect Join behavior in filter conditions

2015-10-06 Thread RaviShankar KS (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RaviShankar KS updated SPARK-10967: --- Description: We notice that the join conditions are not working as expected in the case of

[jira] [Created] (SPARK-10967) Incorrect Join behavior in filter conditions

2015-10-06 Thread RaviShankar KS (JIRA)
RaviShankar KS created SPARK-10967: -- Summary: Incorrect Join behavior in filter conditions Key: SPARK-10967 URL: https://issues.apache.org/jira/browse/SPARK-10967 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-10937) java.lang.NoSuchMethodError when instantiating sqlContext in spark-shell using hive 0.12.x, 0.13.x

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945224#comment-14945224 ] Curtis Wilde edited comment on SPARK-10937 at 10/6/15 3:47 PM: --- {{val props

[jira] [Comment Edited] (SPARK-10937) java.lang.NoSuchMethodError when instantiating sqlContext in spark-shell using hive 0.12.x, 0.13.x

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945224#comment-14945224 ] Curtis Wilde edited comment on SPARK-10937 at 10/6/15 3:47 PM: --- {{val props

[jira] [Comment Edited] (SPARK-10937) java.lang.NoSuchMethodError when instantiating sqlContext in spark-shell using hive 0.12.x, 0.13.x

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945224#comment-14945224 ] Curtis Wilde edited comment on SPARK-10937 at 10/6/15 3:46 PM: --- {{val props

[jira] [Comment Edited] (SPARK-10937) java.lang.NoSuchMethodError when instantiating sqlContext in spark-shell using hive 0.12.x, 0.13.x

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945224#comment-14945224 ] Curtis Wilde edited comment on SPARK-10937 at 10/6/15 3:46 PM: --- {{val props

[jira] [Resolved] (SPARK-10938) Remove typeId in columnar cache

2015-10-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10938. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8989

[jira] [Commented] (SPARK-10937) java.lang.NoSuchMethodError when instantiating sqlContext in spark-shell using hive 0.12.x, 0.13.x

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945224#comment-14945224 ] Curtis Wilde commented on SPARK-10937: -- {{val props = sys.props("java.class.path") val sysprops =

[jira] [Comment Edited] (SPARK-10937) java.lang.NoSuchMethodError when instantiating sqlContext in spark-shell using hive 0.12.x, 0.13.x

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945224#comment-14945224 ] Curtis Wilde edited comment on SPARK-10937 at 10/6/15 3:46 PM: --- {{val props

[jira] [Commented] (SPARK-5569) Checkpoints cannot reference classes defined outside of Spark's assembly

2015-10-06 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945091#comment-14945091 ] Cody Koeninger commented on SPARK-5569: --- The gist I originally posted, linked at the top of this

[jira] [Commented] (SPARK-10945) GraphX computes Pagerank with NaN (with some datasets)

2015-10-06 Thread Khaled Ammar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945127#comment-14945127 ] Khaled Ammar commented on SPARK-10945: -- Actually, I am not sure. I did not know this is a condition

[jira] [Commented] (SPARK-10306) sbt hive/update issue

2015-10-06 Thread Kevin Cox (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945090#comment-14945090 ] Kevin Cox commented on SPARK-10306: --- Could the solution be shared. There are a couple of people running

[jira] [Commented] (SPARK-10945) GraphX computes Pagerank with NaN (with some datasets)

2015-10-06 Thread Khaled Ammar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945097#comment-14945097 ] Khaled Ammar commented on SPARK-10945: -- Hi, I found what might seem as a good starting point.

[jira] [Commented] (SPARK-10185) Spark SQL does not handle comma separates paths on Hadoop FileSystem

2015-10-06 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945180#comment-14945180 ] koert kuipers commented on SPARK-10185: --- someone else is also running into this:

[jira] [Commented] (SPARK-10945) GraphX computes Pagerank with NaN (with some datasets)

2015-10-06 Thread Khaled Ammar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945155#comment-14945155 ] Khaled Ammar commented on SPARK-10945: -- Thank you Sean, The file looks like sorted, based on first

[jira] [Commented] (SPARK-10945) GraphX computes Pagerank with NaN (with some datasets)

2015-10-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945136#comment-14945136 ] Sean Owen commented on SPARK-10945: --- I'm not 100% sure it's required, but I see the reader does not

[jira] [Created] (SPARK-10950) ApplicationHistoryInfo to include spark version; History Server to report incompatibility with later versions

2015-10-06 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-10950: -- Summary: ApplicationHistoryInfo to include spark version; History Server to report incompatibility with later versions Key: SPARK-10950 URL:

[jira] [Commented] (SPARK-10950) ApplicationHistoryInfo to include spark version; History Server to report incompatibility with later versions

2015-10-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945161#comment-14945161 ] Steve Loughran commented on SPARK-10950: There's a less elegant solution which wouldn't address

[jira] [Comment Edited] (SPARK-10937) java.lang.NoSuchMethodError when instantiating sqlContext in spark-shell using hive 0.12.x, 0.13.x

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945224#comment-14945224 ] Curtis Wilde edited comment on SPARK-10937 at 10/6/15 3:55 PM: --- {{val props

[jira] [Updated] (SPARK-10967) Incorrect Join behavior in filter conditions

2015-10-06 Thread RaviShankar KS (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RaviShankar KS updated SPARK-10967: --- Description: (was: According to the [Hive Language

[jira] [Assigned] (SPARK-10963) Make KafkaCluster api public

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10963: Assignee: (was: Apache Spark) > Make KafkaCluster api public >

[jira] [Commented] (SPARK-10963) Make KafkaCluster api public

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946183#comment-14946183 ] Apache Spark commented on SPARK-10963: -- User 'koeninger' has created a pull request for this issue:

[jira] [Commented] (SPARK-9478) Add class weights to Random Forest

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946201#comment-14946201 ] Apache Spark commented on SPARK-9478: - User 'rotationsymmetry' has created a pull request for this

[jira] [Assigned] (SPARK-9478) Add class weights to Random Forest

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9478: --- Assignee: (was: Apache Spark) > Add class weights to Random Forest >

[jira] [Assigned] (SPARK-9478) Add class weights to Random Forest

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9478: --- Assignee: Apache Spark > Add class weights to Random Forest >

[jira] [Updated] (SPARK-5565) LDA wrapper for spark.ml package

2015-10-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-5565: - Assignee: Joseph K. Bradley > LDA wrapper for spark.ml package >

[jira] [Commented] (SPARK-5565) LDA wrapper for spark.ml package

2015-10-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946209#comment-14946209 ] Joseph K. Bradley commented on SPARK-5565: -- I'm writing a design doc and will post it soon. >

[jira] [Commented] (SPARK-9325) Support `collect` on DataFrame columns

2015-10-06 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945446#comment-14945446 ] Felix Cheung commented on SPARK-9325: - My comment was on collect(df$Age). As I've stated,

[jira] [Commented] (SPARK-10812) Spark Hadoop Util does not support stopping a non-yarn Spark Context & starting a Yarn spark context.

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945492#comment-14945492 ] Apache Spark commented on SPARK-10812: -- User 'vanzin' has created a pull request for this issue:

[jira] [Updated] (SPARK-10810) Improve session management for SQL

2015-10-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10810: --- Attachment: Session management in Spark SQL 1.6.pdf Design doc > Improve session management for SQL

[jira] [Commented] (SPARK-10794) Spark-SQL- select query on table column with binary Data Type displays error message- java.lang.ClassCastException: java.lang.String cannot be cast to [B

2015-10-06 Thread Jia Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945567#comment-14945567 ] Jia Li commented on SPARK-10794: I tried the repro steps on spark 1.5.0 for hadoop 2.6 and did not see

[jira] [Commented] (SPARK-10902) Hive UDF current_database() does not work

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945530#comment-14945530 ] Apache Spark commented on SPARK-10902: -- User 'davies' has created a pull request for this issue:

[jira] [Commented] (SPARK-10953) Benchmark codegen vs. hand-written code for univariate statistics

2015-10-06 Thread Jihong MA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945532#comment-14945532 ] Jihong MA commented on SPARK-10953: --- [~mengxr] do you mean comparing an implementation which operate

[jira] [Assigned] (SPARK-10902) Hive UDF current_database() does not work

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10902: Assignee: Apache Spark > Hive UDF current_database() does not work >

[jira] [Assigned] (SPARK-10902) Hive UDF current_database() does not work

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10902: Assignee: (was: Apache Spark) > Hive UDF current_database() does not work >

[jira] [Updated] (SPARK-10794) Spark-SQL- select query on table column with binary Data Type displays error message- java.lang.ClassCastException: java.lang.String cannot be cast to [B

2015-10-06 Thread Jia Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Li updated SPARK-10794: --- Attachment: spark_1_5_0.png > Spark-SQL- select query on table column with binary Data Type displays error

[jira] [Assigned] (SPARK-10955) Disable dynamic allocation for Streaming jobs

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10955: Assignee: Apache Spark > Disable dynamic allocation for Streaming jobs >

[jira] [Assigned] (SPARK-10955) Disable dynamic allocation for Streaming jobs

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10955: Assignee: (was: Apache Spark) > Disable dynamic allocation for Streaming jobs >

[jira] [Commented] (SPARK-10326) Cannot launch YARN job on Windows

2015-10-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945598#comment-14945598 ] Marcelo Vanzin commented on SPARK-10326: Can you post the actual exception? > Cannot launch YARN

[jira] [Commented] (SPARK-10326) Cannot launch YARN job on Windows

2015-10-06 Thread Jose Antonio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945606#comment-14945606 ] Jose Antonio commented on SPARK-10326: -- Ok, give a moment its about half past Spai (night) here in

[jira] [Created] (SPARK-10956) Introduce common memory management interface for execution and storage

2015-10-06 Thread Andrew Or (JIRA)
Andrew Or created SPARK-10956: - Summary: Introduce common memory management interface for execution and storage Key: SPARK-10956 URL: https://issues.apache.org/jira/browse/SPARK-10956 Project: Spark

[jira] [Commented] (SPARK-10798) JsonMappingException with Spark Context Parallelize

2015-10-06 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945445#comment-14945445 ] Miao Wang commented on SPARK-10798: --- @Sean, Thanks for pointing it out. I come from C,C++ and I am

[jira] [Commented] (SPARK-10914) Incorrect empty join sets when executor-memory >= 32g

2015-10-06 Thread Ben Moran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945443#comment-14945443 ] Ben Moran commented on SPARK-10914: --- Ah, I did it the wrong way around! With a *small* heap and the

[jira] [Commented] (SPARK-10940) Too many open files Spark Shuffle

2015-10-06 Thread Sandeep Pal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945449#comment-14945449 ] Sandeep Pal commented on SPARK-10940: - Ok sure, how much do you suggest to increase further. > Too

[jira] [Updated] (SPARK-10951) Support private S3 repositories using spark-submit via --repositories flag

2015-10-06 Thread Jerry Lam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry Lam updated SPARK-10951: -- Summary: Support private S3 repositories using spark-submit via --repositories flag (was: Support

[jira] [Commented] (SPARK-9443) Expose sampleByKey in SparkR

2015-10-06 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945593#comment-14945593 ] Felix Cheung commented on SPARK-9443: - is this sampleBy

[jira] [Updated] (SPARK-10956) Introduce common memory management interface for execution and storage

2015-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10956: -- Description: Memory management in Spark is currently broken down into two disjoint regions: one for

[jira] [Assigned] (SPARK-10956) Introduce common memory management interface for execution and storage

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10956: Assignee: Andrew Or (was: Apache Spark) > Introduce common memory management interface

[jira] [Commented] (SPARK-10956) Introduce common memory management interface for execution and storage

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945629#comment-14945629 ] Apache Spark commented on SPARK-10956: -- User 'andrewor14' has created a pull request for this issue:

[jira] [Assigned] (SPARK-10956) Introduce common memory management interface for execution and storage

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10956: Assignee: Apache Spark (was: Andrew Or) > Introduce common memory management interface

[jira] [Commented] (SPARK-10913) Add attach() function for DataFrame

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945510#comment-14945510 ] Apache Spark commented on SPARK-10913: -- User 'adrian555' has created a pull request for this issue:

[jira] [Created] (SPARK-10955) Disable dynamic allocation for Streaming jobs

2015-10-06 Thread Hari Shreedharan (JIRA)
Hari Shreedharan created SPARK-10955: Summary: Disable dynamic allocation for Streaming jobs Key: SPARK-10955 URL: https://issues.apache.org/jira/browse/SPARK-10955 Project: Spark Issue

[jira] [Created] (SPARK-10954) Parquet version in the "created_by" metadata field of Parquet files written by Spark 1.5 and 1.6 is wrong

2015-10-06 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-10954: -- Summary: Parquet version in the "created_by" metadata field of Parquet files written by Spark 1.5 and 1.6 is wrong Key: SPARK-10954 URL:

[jira] [Commented] (SPARK-10955) Disable dynamic allocation for Streaming jobs

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945576#comment-14945576 ] Apache Spark commented on SPARK-10955: -- User 'harishreedharan' has created a pull request for this

[jira] [Commented] (SPARK-10326) Cannot launch YARN job on Windows

2015-10-06 Thread Jose Antonio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945594#comment-14945594 ] Jose Antonio commented on SPARK-10326: -- Hi Marcelo. I see the same exact error reported in the

[jira] [Updated] (SPARK-10946) JDBC - Use Statement.executeUpdate instead of PreparedStatement.executeUpdate for DDLs

2015-10-06 Thread Pallavi Priyadarshini (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pallavi Priyadarshini updated SPARK-10946: -- Summary: JDBC - Use Statement.executeUpdate instead of

[jira] [Commented] (SPARK-10942) Not all cached RDDs are unpersisted

2015-10-06 Thread Rekha Joshi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944569#comment-14944569 ] Rekha Joshi commented on SPARK-10942: - great, thanks [~pnpritchard].In anycase will keep an eye open

[jira] [Created] (SPARK-10945) GraphX computes Pagerank with NaN (with some datasets)

2015-10-06 Thread Khaled Ammar (JIRA)
Khaled Ammar created SPARK-10945: Summary: GraphX computes Pagerank with NaN (with some datasets) Key: SPARK-10945 URL: https://issues.apache.org/jira/browse/SPARK-10945 Project: Spark Issue

[jira] [Created] (SPARK-10946) JDBC - Use Statement.execute instead of PreparedStatement.execute for DDLs

2015-10-06 Thread Pallavi Priyadarshini (JIRA)
Pallavi Priyadarshini created SPARK-10946: - Summary: JDBC - Use Statement.execute instead of PreparedStatement.execute for DDLs Key: SPARK-10946 URL: https://issues.apache.org/jira/browse/SPARK-10946

[jira] [Commented] (SPARK-10513) Springleaf Marketing Response

2015-10-06 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944634#comment-14944634 ] Yanbo Liang commented on SPARK-10513: - [~mengxr] I'm on vacation until 8th, Oct. The coding almost

[jira] [Commented] (SPARK-10937) java.lang.NoSuchMethodError when instantiating sqlContext in spark-shell using hive 0.12.x, 0.13.x

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945647#comment-14945647 ] Curtis Wilde commented on SPARK-10937: -- Sorry, I must have misunderstood. I removed the jars from

[jira] [Created] (SPARK-10957) setParams changes quantileProbabilities unexpectly in PySpark's AFTSurvivalRegression

2015-10-06 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-10957: - Summary: setParams changes quantileProbabilities unexpectly in PySpark's AFTSurvivalRegression Key: SPARK-10957 URL: https://issues.apache.org/jira/browse/SPARK-10957

[jira] [Updated] (SPARK-10940) Too many open files Spark Shuffle

2015-10-06 Thread Sandeep Pal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-10940: Description: Executing terasort by Spark-SQL on the data generated by teragen in hadoop. Data

[jira] [Commented] (SPARK-10914) Incorrect empty join sets when executor-memory >= 32g

2015-10-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945638#comment-14945638 ] Sean Owen commented on SPARK-10914: --- Hm, it could be a valid lead after all. The size estimator code is

[jira] [Commented] (SPARK-10940) Too many open files Spark Shuffle

2015-10-06 Thread Sandeep Pal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945652#comment-14945652 ] Sandeep Pal commented on SPARK-10940: - Ok, I will. One more important think I for got to mention. The

[jira] [Comment Edited] (SPARK-10937) java.lang.NoSuchMethodError when instantiating sqlContext in spark-shell using hive 0.12.x, 0.13.x

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945647#comment-14945647 ] Curtis Wilde edited comment on SPARK-10937 at 10/6/15 7:46 PM: --- It seems I

[jira] [Comment Edited] (SPARK-10940) Too many open files Spark Shuffle

2015-10-06 Thread Sandeep Pal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945652#comment-14945652 ] Sandeep Pal edited comment on SPARK-10940 at 10/6/15 7:55 PM: -- Ok, I will.

[jira] [Resolved] (SPARK-10688) Python API for AFTSurvivalRegression

2015-10-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10688. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8926

[jira] [Commented] (SPARK-10940) Too many open files Spark Shuffle

2015-10-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945642#comment-14945642 ] Sean Owen commented on SPARK-10940: --- I mean, just for the sake of testing, 100K? if that's not enough

[jira] [Updated] (SPARK-10688) Python API for AFTSurvivalRegression

2015-10-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-10688: -- Assignee: Kai Jiang > Python API for AFTSurvivalRegression >

[jira] [Comment Edited] (SPARK-10940) Too many open files Spark Shuffle

2015-10-06 Thread Sandeep Pal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945652#comment-14945652 ] Sandeep Pal edited comment on SPARK-10940 at 10/6/15 7:54 PM: -- Ok, I will.

[jira] [Assigned] (SPARK-10957) setParams changes quantileProbabilities unexpectly in PySpark's AFTSurvivalRegression

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10957: Assignee: Xiangrui Meng (was: Apache Spark) > setParams changes quantileProbabilities

[jira] [Commented] (SPARK-10957) setParams changes quantileProbabilities unexpectly in PySpark's AFTSurvivalRegression

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945665#comment-14945665 ] Apache Spark commented on SPARK-10957: -- User 'mengxr' has created a pull request for this issue:

[jira] [Assigned] (SPARK-10957) setParams changes quantileProbabilities unexpectly in PySpark's AFTSurvivalRegression

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10957: Assignee: Apache Spark (was: Xiangrui Meng) > setParams changes quantileProbabilities

[jira] [Created] (SPARK-10958) Upgrade json4s version to 3.3.0. Formats are now serializable.

2015-10-06 Thread Tyler Prete (JIRA)
Tyler Prete created SPARK-10958: --- Summary: Upgrade json4s version to 3.3.0. Formats are now serializable. Key: SPARK-10958 URL: https://issues.apache.org/jira/browse/SPARK-10958 Project: Spark

[jira] [Assigned] (SPARK-10959) PySpark StreamingLogisticRegressionWithSGD does not train with given regParam and convergenceTol parameters

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10959: Assignee: (was: Apache Spark) > PySpark StreamingLogisticRegressionWithSGD does not

[jira] [Commented] (SPARK-9443) Expose sampleByKey in SparkR

2015-10-06 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945816#comment-14945816 ] Shivaram Venkataraman commented on SPARK-9443: -- Yes. I think so > Expose sampleByKey in

[jira] [Assigned] (SPARK-10959) PySpark StreamingLogisticRegressionWithSGD does not train with given regParam and convergenceTol parameters

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10959: Assignee: Apache Spark > PySpark StreamingLogisticRegressionWithSGD does not train with

[jira] [Commented] (SPARK-10959) PySpark StreamingLogisticRegressionWithSGD does not train with given regParam and convergenceTol parameters

2015-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945819#comment-14945819 ] Apache Spark commented on SPARK-10959: -- User 'BryanCutler' has created a pull request for this

[jira] [Created] (SPARK-10960) SQL with windowing function cannot reference column in inner select block

2015-10-06 Thread David Wong (JIRA)
David Wong created SPARK-10960: -- Summary: SQL with windowing function cannot reference column in inner select block Key: SPARK-10960 URL: https://issues.apache.org/jira/browse/SPARK-10960 Project: Spark

[jira] [Updated] (SPARK-10958) Upgrade json4s version to 3.3.0 to eliminate common serialization issues with Formats.

2015-10-06 Thread Tyler Prete (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Prete updated SPARK-10958: Summary: Upgrade json4s version to 3.3.0 to eliminate common serialization issues with Formats.

[jira] [Updated] (SPARK-10961) Specified metastore 0.12.0 but spark-shell still using metastore classes for 0.13+

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Curtis Wilde updated SPARK-10961: - Description: After setting metastore to 0.12.0 in {{spark-defaults.conf}}, spark-shell still

[jira] [Commented] (SPARK-9318) Add `merge` as synonym for join

2015-10-06 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945835#comment-14945835 ] Hossein Falaki commented on SPARK-9318: --- I agree with the issue being discussed. SparkR should have

[jira] [Issue Comment Deleted] (SPARK-10474) TungstenAggregation cannot acquire memory for pointer array after switching to sort-based

2015-10-06 Thread Hans van den Bogert (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans van den Bogert updated SPARK-10474: Comment: was deleted (was: One more debug println for the calculated cores (in

[jira] [Updated] (SPARK-10064) Decision tree continuous feature binning is slow in large feature spaces

2015-10-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-10064: -- Shepherd: Joseph K. Bradley Assignee: Nathan Howell > Decision tree continuous

[jira] [Updated] (SPARK-10064) Decision tree continuous feature binning is slow in large feature spaces

2015-10-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-10064: -- Priority: Minor (was: Major) > Decision tree continuous feature binning is slow in

[jira] [Updated] (SPARK-10960) SQL with windowing function cannot reference column in inner select block

2015-10-06 Thread David Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Wong updated SPARK-10960: --- Description: There seems to be a bug in the Spark SQL parser when I use windowing functions.

[jira] [Updated] (SPARK-10382) Make example code in user guide testable

2015-10-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-10382: -- Assignee: Xusen Yin (was: Xusen Yin) > Make example code in user guide testable >

[jira] [Commented] (SPARK-10382) Make example code in user guide testable

2015-10-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945917#comment-14945917 ] Xiangrui Meng commented on SPARK-10382: --- Assigned. Please keep the initial implementation as simple

[jira] [Commented] (SPARK-10943) NullType Column cannot be written to Parquet

2015-10-06 Thread Jason C Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945783#comment-14945783 ] Jason C Lee commented on SPARK-10943: - I'd like to work on this. Thanx > NullType Column cannot be

[jira] [Updated] (SPARK-10961) Specified metastore 0.12.0 but spark-shell still using metastore classes for 0.13+

2015-10-06 Thread Curtis Wilde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Curtis Wilde updated SPARK-10961: - Description: After setting metastore to 0.12.0 in {{spark-defaults.conf}}, spark-shell still

[jira] [Updated] (SPARK-10779) Set initialModel for KMeans model in PySpark (spark.mllib)

2015-10-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-10779: -- Shepherd: Joseph K. Bradley > Set initialModel for KMeans model in PySpark

[jira] [Created] (SPARK-10961) Specified metastore 0.12.0 but spark-shell still using metastore classes for 0.13+

2015-10-06 Thread Curtis Wilde (JIRA)
Curtis Wilde created SPARK-10961: Summary: Specified metastore 0.12.0 but spark-shell still using metastore classes for 0.13+ Key: SPARK-10961 URL: https://issues.apache.org/jira/browse/SPARK-10961

[jira] [Created] (SPARK-10962) DataFrame "except

2015-10-06 Thread Abhijit Deb (JIRA)
Abhijit Deb created SPARK-10962: --- Summary: DataFrame "except Key: SPARK-10962 URL: https://issues.apache.org/jira/browse/SPARK-10962 Project: Spark Issue Type: Improvement

  1   2   3   >