[jira] [Commented] (SPARK-15748) Replace inefficient foldLeft() call in PartitionStatistics

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313619#comment-15313619 ] Apache Spark commented on SPARK-15748: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Resolved] (SPARK-14959) ​Problem Reading partitioned ORC or Parquet files

2016-06-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14959. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13463

[jira] [Updated] (SPARK-14959) ​Problem Reading partitioned ORC or Parquet files

2016-06-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14959: --- Assignee: Xin Wu > ​Problem Reading partitioned ORC or Parquet files >

[jira] [Assigned] (SPARK-15748) Replace inefficient foldLeft() call in PartitionStatistics

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15748: Assignee: Josh Rosen (was: Apache Spark) > Replace inefficient foldLeft() call in

[jira] [Assigned] (SPARK-15748) Replace inefficient foldLeft() call in PartitionStatistics

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15748: Assignee: Apache Spark (was: Josh Rosen) > Replace inefficient foldLeft() call in

[jira] [Commented] (SPARK-14146) Imported implicits can't be found in Spark REPL in some cases

2016-06-02 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313618#comment-15313618 ] Prashant Sharma commented on SPARK-14146: - Thanks for the reproducers, I am looking into it. >

[jira] [Resolved] (SPARK-15733) Makes the explain output less verbose by hiding some verbose output like None, null, empty List, and etc..

2016-06-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-15733. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13470

[jira] [Updated] (SPARK-15733) Makes the explain output less verbose by hiding some verbose output like None, null, empty List, and etc..

2016-06-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-15733: --- Assignee: Sean Zhong > Makes the explain output less verbose by hiding some verbose output like >

[jira] [Created] (SPARK-15748) Replace inefficient foldLeft() call in PartitionStatistics

2016-06-02 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-15748: -- Summary: Replace inefficient foldLeft() call in PartitionStatistics Key: SPARK-15748 URL: https://issues.apache.org/jira/browse/SPARK-15748 Project: Spark Issue

[jira] [Updated] (SPARK-15747) Support SPARK_CONF_DIR/spark-defaults.d/*.conf drop-in style config files

2016-06-02 Thread Terry Moschou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terry Moschou updated SPARK-15747: -- Description: Feature request to automatically source all files in

[jira] [Created] (SPARK-15747) Support SPARK_CONF_DIR/spark-defaults.d/*.conf drop-in style config files

2016-06-02 Thread Terry Moschou (JIRA)
Terry Moschou created SPARK-15747: - Summary: Support SPARK_CONF_DIR/spark-defaults.d/*.conf drop-in style config files Key: SPARK-15747 URL: https://issues.apache.org/jira/browse/SPARK-15747 Project:

[jira] [Updated] (SPARK-15724) Add benchmarks for performance over wide schemas

2016-06-02 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-15724: --- Assignee: Eric Liang > Add benchmarks for performance over wide schemas >

[jira] [Resolved] (SPARK-15724) Add benchmarks for performance over wide schemas

2016-06-02 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-15724. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13456

[jira] [Commented] (SPARK-15723) SimpleDateParamSuite test is locale-fragile and relies on deprecated short TZ name

2016-06-02 Thread Brett Randall (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313479#comment-15313479 ] Brett Randall commented on SPARK-15723: --- Yes the fix is to the test only. I looked at where

[jira] [Updated] (SPARK-15746) SchemaUtils.checkColumnType with VectorUDT prints instance details in error message

2016-06-02 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-15746: --- Summary: SchemaUtils.checkColumnType with VectorUDT prints instance details in error message

[jira] [Created] (SPARK-15746) SchemaUtils.checkColumnType with VectorUDT prints instance details

2016-06-02 Thread Nick Pentreath (JIRA)
Nick Pentreath created SPARK-15746: -- Summary: SchemaUtils.checkColumnType with VectorUDT prints instance details Key: SPARK-15746 URL: https://issues.apache.org/jira/browse/SPARK-15746 Project:

[jira] [Commented] (SPARK-15722) Wrong data when CTAS specifies schema

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313438#comment-15313438 ] Apache Spark commented on SPARK-15722: -- User 'andrewor14' has created a pull request for this issue:

[jira] [Resolved] (SPARK-15732) Dataset generated code "generated.java" Fails with Certain Case Classes

2016-06-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-15732. Resolution: Fixed Fix Version/s: 2.0.0 Resolved by

[jira] [Updated] (SPARK-15732) Dataset generated code "generated.java" Fails with Certain Case Classes

2016-06-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-15732: --- Assignee: Wenchen Fan > Dataset generated code "generated.java" Fails with Certain Case Classes >

[jira] [Assigned] (SPARK-15723) SimpleDateParamSuite test is locale-fragile and relies on deprecated short TZ name

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15723: Assignee: Apache Spark > SimpleDateParamSuite test is locale-fragile and relies on

[jira] [Assigned] (SPARK-15723) SimpleDateParamSuite test is locale-fragile and relies on deprecated short TZ name

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15723: Assignee: (was: Apache Spark) > SimpleDateParamSuite test is locale-fragile and

[jira] [Commented] (SPARK-15723) SimpleDateParamSuite test is locale-fragile and relies on deprecated short TZ name

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313427#comment-15313427 ] Apache Spark commented on SPARK-15723: -- User 'javabrett' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15745) Use classloader's getResource() for reading resource files in HiveTests

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15745: Assignee: Apache Spark > Use classloader's getResource() for reading resource files in

[jira] [Commented] (SPARK-15745) Use classloader's getResource() for reading resource files in HiveTests

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313421#comment-15313421 ] Apache Spark commented on SPARK-15745: -- User 'sameeragarwal' has created a pull request for this

[jira] [Assigned] (SPARK-15745) Use classloader's getResource() for reading resource files in HiveTests

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15745: Assignee: (was: Apache Spark) > Use classloader's getResource() for reading resource

[jira] [Created] (SPARK-15745) Use classloader's getResource() for reading resource files in HiveTests

2016-06-02 Thread Sameer Agarwal (JIRA)
Sameer Agarwal created SPARK-15745: -- Summary: Use classloader's getResource() for reading resource files in HiveTests Key: SPARK-15745 URL: https://issues.apache.org/jira/browse/SPARK-15745 Project:

[jira] [Resolved] (SPARK-15736) Gracefully handle loss of DiskStore files

2016-06-02 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15736. --- Resolution: Fixed Fix Version/s: 2.0.0 1.6.2 > Gracefully handle loss of

[jira] [Assigned] (SPARK-15744) Rename two TungstenAggregation*Suites and update error messages/comments

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15744: Assignee: (was: Apache Spark) > Rename two TungstenAggregation*Suites and update

[jira] [Commented] (SPARK-15744) Rename two TungstenAggregation*Suites and update error messages/comments

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313405#comment-15313405 ] Apache Spark commented on SPARK-15744: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-15744) Rename two TungstenAggregation*Suites and update error messages/comments

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15744: Assignee: Apache Spark > Rename two TungstenAggregation*Suites and update error

[jira] [Comment Edited] (SPARK-13868) Random forest accuracy exploration

2016-06-02 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313400#comment-15313400 ] Xusen Yin edited comment on SPARK-13868 at 6/3/16 12:40 AM: [~josephkb]

[jira] [Resolved] (SPARK-15718) better error message for writing bucketing data

2016-06-02 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15718. --- Resolution: Fixed Fix Version/s: 2.0.0 > better error message for writing bucketing data >

[jira] [Updated] (SPARK-15744) Rename two TungstenAggregation*Suites and update error messages/comments

2016-06-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-15744: -- Description: For consistency, this issue updates some remaining

[jira] [Commented] (SPARK-13868) Random forest accuracy exploration

2016-06-02 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313400#comment-15313400 ] Xusen Yin commented on SPARK-13868: --- [~josephkb] [~tanwanirahul] Here is what I found: 1. Dataset

[jira] [Created] (SPARK-15744) Rename two TungstenAggregation*Suites and update error messages/comments

2016-06-02 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-15744: - Summary: Rename two TungstenAggregation*Suites and update error messages/comments Key: SPARK-15744 URL: https://issues.apache.org/jira/browse/SPARK-15744 Project:

[jira] [Assigned] (SPARK-15743) Prevent saving with all-column partitioning

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15743: Assignee: (was: Apache Spark) > Prevent saving with all-column partitioning >

[jira] [Commented] (SPARK-15743) Prevent saving with all-column partitioning

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313381#comment-15313381 ] Apache Spark commented on SPARK-15743: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-15743) Prevent saving with all-column partitioning

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15743: Assignee: Apache Spark > Prevent saving with all-column partitioning >

[jira] [Commented] (SPARK-15741) PySpark Cleanup of _setDefault with seed=None

2016-06-02 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313376#comment-15313376 ] Bryan Cutler commented on SPARK-15741: -- >From what I gathered, explicitly setting a seed to {{None}}

[jira] [Created] (SPARK-15743) Prevent saving with all-column partitioning

2016-06-02 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-15743: - Summary: Prevent saving with all-column partitioning Key: SPARK-15743 URL: https://issues.apache.org/jira/browse/SPARK-15743 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-15732) Dataset generated code "generated.java" Fails with Certain Case Classes

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15732: Assignee: Apache Spark > Dataset generated code "generated.java" Fails with Certain Case

[jira] [Assigned] (SPARK-15732) Dataset generated code "generated.java" Fails with Certain Case Classes

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15732: Assignee: (was: Apache Spark) > Dataset generated code "generated.java" Fails with

[jira] [Commented] (SPARK-15742) Reduce collections allocations in Catalyst tree transformation methods

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313355#comment-15313355 ] Apache Spark commented on SPARK-15742: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15742) Reduce collections allocations in Catalyst tree transformation methods

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15742: Assignee: Josh Rosen (was: Apache Spark) > Reduce collections allocations in Catalyst

[jira] [Assigned] (SPARK-15742) Reduce collections allocations in Catalyst tree transformation methods

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15742: Assignee: Apache Spark (was: Josh Rosen) > Reduce collections allocations in Catalyst

[jira] [Commented] (SPARK-15732) Dataset generated code "generated.java" Fails with Certain Case Classes

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313354#comment-15313354 ] Apache Spark commented on SPARK-15732: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-15742) Reduce collections allocations in Catalyst tree transformation methods

2016-06-02 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-15742: -- Summary: Reduce collections allocations in Catalyst tree transformation methods Key: SPARK-15742 URL: https://issues.apache.org/jira/browse/SPARK-15742 Project: Spark

[jira] [Commented] (SPARK-15732) Dataset generated code "generated.java" Fails with Certain Case Classes

2016-06-02 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313330#comment-15313330 ] Reynold Xin commented on SPARK-15732: - Yea we definitely should put a better error message for 2.0.

[jira] [Closed] (SPARK-15741) PySpark Cleanup of _setDefault with seed=None

2016-06-02 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler closed SPARK-15741. Resolution: Invalid Looks like I jumped the gun here, None values are not ignored and seems like

[jira] [Commented] (SPARK-15732) Dataset generated code "generated.java" Fails with Certain Case Classes

2016-06-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313321#comment-15313321 ] Wenchen Fan commented on SPARK-15732: - It's really hard to support it, and I don't think this corner

[jira] [Resolved] (SPARK-15668) ml.feature: update check schema to avoid confusion when user use MLlib.vector as input type

2016-06-02 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-15668. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13411

[jira] [Assigned] (SPARK-15688) RelationalGroupedDataset.toDF should not add group by expressions that are already added in the aggregate expressions.

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15688: Assignee: Apache Spark > RelationalGroupedDataset.toDF should not add group by

[jira] [Assigned] (SPARK-15688) RelationalGroupedDataset.toDF should not add group by expressions that are already added in the aggregate expressions.

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15688: Assignee: (was: Apache Spark) > RelationalGroupedDataset.toDF should not add group by

[jira] [Commented] (SPARK-15688) RelationalGroupedDataset.toDF should not add group by expressions that are already added in the aggregate expressions.

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313306#comment-15313306 ] Apache Spark commented on SPARK-15688: -- User 'dilipbiswal' has created a pull request for this

[jira] [Updated] (SPARK-15741) PySpark Cleanup of _setDefault with seed=None

2016-06-02 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-15741: - Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-14771) > PySpark

[jira] [Created] (SPARK-15741) PySpark Cleanup of _setDefault with seed=None

2016-06-02 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-15741: Summary: PySpark Cleanup of _setDefault with seed=None Key: SPARK-15741 URL: https://issues.apache.org/jira/browse/SPARK-15741 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15725: Assignee: (was: Apache Spark) > Dynamic allocation hangs YARN app when executors time

[jira] [Commented] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313285#comment-15313285 ] Apache Spark commented on SPARK-15725: -- User 'rdblue' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15725: Assignee: Apache Spark > Dynamic allocation hangs YARN app when executors time out >

[jira] [Resolved] (SPARK-15734) Avoids printing internal row in explain output

2016-06-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-15734. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13471

[jira] [Updated] (SPARK-15139) PySpark TreeEnsemble missing methods

2016-06-02 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-15139: --- Assignee: holdenk > PySpark TreeEnsemble missing methods >

[jira] [Resolved] (SPARK-15719) Disable writing Parquet summary files by default

2016-06-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-15719. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13455

[jira] [Comment Edited] (SPARK-15705) Spark won't read ORC schema from metastore for partitioned tables

2016-06-02 Thread Xin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313257#comment-15313257 ] Xin Wu edited comment on SPARK-15705 at 6/2/16 11:15 PM: - I can recreate it now.

[jira] [Commented] (SPARK-15705) Spark won't read ORC schema from metastore for partitioned tables

2016-06-02 Thread Xin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313257#comment-15313257 ] Xin Wu commented on SPARK-15705: I can recreate it now. and will look into it. > Spark won't read ORC

[jira] [Resolved] (SPARK-15139) PySpark TreeEnsemble missing methods

2016-06-02 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-15139. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12919

[jira] [Resolved] (SPARK-15092) toDebugString missing from ML DecisionTreeClassifier

2016-06-02 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-15092. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12919

[jira] [Commented] (SPARK-15732) Dataset generated code "generated.java" Fails with Certain Case Classes

2016-06-02 Thread Bo Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313230#comment-15313230 ] Bo Meng commented on SPARK-15732: - There is no easy way to work around this issue since "abstract" is a

[jira] [Commented] (SPARK-15623) 2.0 python coverage ml.feature

2016-06-02 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313226#comment-15313226 ] Bryan Cutler commented on SPARK-15623: -- I took another spin through this and linked a couple of

[jira] [Assigned] (SPARK-15738) PySpark ml.feature RFormula missing string representation displaying formula

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15738: Assignee: (was: Apache Spark) > PySpark ml.feature RFormula missing string

[jira] [Assigned] (SPARK-15738) PySpark ml.feature RFormula missing string representation displaying formula

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15738: Assignee: Apache Spark > PySpark ml.feature RFormula missing string representation

[jira] [Commented] (SPARK-15738) PySpark ml.feature RFormula missing string representation displaying formula

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313222#comment-15313222 ] Apache Spark commented on SPARK-15738: -- User 'BryanCutler' has created a pull request for this

[jira] [Comment Edited] (SPARK-14811) ML, Graph 2.0 QA: API: New Scala APIs, docs

2016-06-02 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313208#comment-15313208 ] Nick Pentreath edited comment on SPARK-14811 at 6/2/16 10:31 PM: -

[jira] [Comment Edited] (SPARK-14811) ML, Graph 2.0 QA: API: New Scala APIs, docs

2016-06-02 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313208#comment-15313208 ] Nick Pentreath edited comment on SPARK-14811 at 6/2/16 10:31 PM: -

[jira] [Commented] (SPARK-14811) ML, Graph 2.0 QA: API: New Scala APIs, docs

2016-06-02 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313208#comment-15313208 ] Nick Pentreath commented on SPARK-14811: Question on this - we seem to be inconsistent with the

[jira] [Commented] (SPARK-15740) Word2VecSuite "big model load / save" caused OOM in maven jenkins builds

2016-06-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313198#comment-15313198 ] Xiangrui Meng commented on SPARK-15740: --- [~tmnd91] Could you run the test and estimate how much ram

[jira] [Comment Edited] (SPARK-15740) Word2VecSuite "big model load / save" caused OOM in maven jenkins builds

2016-06-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313198#comment-15313198 ] Xiangrui Meng edited comment on SPARK-15740 at 6/2/16 10:24 PM: [~tmnd91]

[jira] [Commented] (SPARK-15736) Gracefully handle loss of DiskStore files

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313196#comment-15313196 ] Apache Spark commented on SPARK-15736: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Updated] (SPARK-15736) Gracefully handle loss of DiskStore files

2016-06-02 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-15736: --- Affects Version/s: 1.6.0 > Gracefully handle loss of DiskStore files >

[jira] [Updated] (SPARK-15736) Gracefully handle loss of DiskStore files

2016-06-02 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-15736: --- Target Version/s: 1.6.2, 2.0.0 (was: 2.0.0) > Gracefully handle loss of DiskStore files >

[jira] [Assigned] (SPARK-15740) Word2VecSuite "big model load / save" caused OOM in maven jenkins builds

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15740: Assignee: (was: Apache Spark) > Word2VecSuite "big model load / save" caused OOM in

[jira] [Commented] (SPARK-15740) Word2VecSuite "big model load / save" caused OOM in maven jenkins builds

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313195#comment-15313195 ] Apache Spark commented on SPARK-15740: -- User 'mengxr' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15740) Word2VecSuite "big model load / save" caused OOM in maven jenkins builds

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15740: Assignee: Apache Spark > Word2VecSuite "big model load / save" caused OOM in maven

[jira] [Commented] (SPARK-15710) Exception with WHERE clause in SQL for non-default Hive database

2016-06-02 Thread Xin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313193#comment-15313193 ] Xin Wu commented on SPARK-15710: hmm.. after another rebase of the master. it seems that the problem is

[jira] [Updated] (SPARK-15740) Word2VecSuite "big model load / save" caused OOM in maven jenkins builds

2016-06-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-15740: -- Description: [~andrewor14] noticed some OOM errors caused by "test big model load / save" in

[jira] [Created] (SPARK-15740) Word2VecSuite "big model load / save" caused OOM in maven jenkins builds

2016-06-02 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-15740: - Summary: Word2VecSuite "big model load / save" caused OOM in maven jenkins builds Key: SPARK-15740 URL: https://issues.apache.org/jira/browse/SPARK-15740 Project:

[jira] [Commented] (SPARK-15739) Expose aggregateMessagesWithActiveSet to users.

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313188#comment-15313188 ] Apache Spark commented on SPARK-15739: -- User 'adeandrade' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15739) Expose aggregateMessagesWithActiveSet to users.

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15739: Assignee: Apache Spark > Expose aggregateMessagesWithActiveSet to users. >

[jira] [Assigned] (SPARK-15739) Expose aggregateMessagesWithActiveSet to users.

2016-06-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15739: Assignee: (was: Apache Spark) > Expose aggregateMessagesWithActiveSet to users. >

[jira] [Updated] (SPARK-15739) Expose aggregateMessagesWithActiveSet to users.

2016-06-02 Thread Anderson de Andrade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anderson de Andrade updated SPARK-15739: Description: The current version of Pregel has some flaws: * Each iteration

[jira] [Created] (SPARK-15739) Expose aggregateMessagesWithActiveSet to users.

2016-06-02 Thread Anderson de Andrade (JIRA)
Anderson de Andrade created SPARK-15739: --- Summary: Expose aggregateMessagesWithActiveSet to users. Key: SPARK-15739 URL: https://issues.apache.org/jira/browse/SPARK-15739 Project: Spark

[jira] [Created] (SPARK-15738) PySpark ml.feature RFormula missing string representation displaying formula

2016-06-02 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-15738: Summary: PySpark ml.feature RFormula missing string representation displaying formula Key: SPARK-15738 URL: https://issues.apache.org/jira/browse/SPARK-15738

[jira] [Updated] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-02 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Chen updated SPARK-15716: - Summary: Memory usage of driver keeps growing up in Spark Streaming (was: Memory usage of driver keep

[jira] [Comment Edited] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-02 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313116#comment-15313116 ] Yan Chen edited comment on SPARK-15716 at 6/2/16 9:59 PM: -- I tried to run it

[jira] [Updated] (SPARK-15716) Memory usage of driver keep growing up in Spark Streaming

2016-06-02 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Chen updated SPARK-15716: - Summary: Memory usage of driver keep growing up in Spark Streaming (was: Memory usage keep growing up

[jira] [Comment Edited] (SPARK-15716) Memory usage keep growing up in Spark Streaming

2016-06-02 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15312450#comment-15312450 ] Yan Chen edited comment on SPARK-15716 at 6/2/16 9:58 PM: -- We actually ran jmap

[jira] [Comment Edited] (SPARK-15716) Memory usage keep growing up in Spark Streaming

2016-06-02 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313116#comment-15313116 ] Yan Chen edited comment on SPARK-15716 at 6/2/16 9:57 PM: -- I tried to run it

[jira] [Comment Edited] (SPARK-15716) Memory usage keep growing up in Spark Streaming

2016-06-02 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313116#comment-15313116 ] Yan Chen edited comment on SPARK-15716 at 6/2/16 9:51 PM: -- I tried to run it

[jira] [Comment Edited] (SPARK-15716) Memory usage keep growing up in Spark Streaming

2016-06-02 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313116#comment-15313116 ] Yan Chen edited comment on SPARK-15716 at 6/2/16 9:45 PM: -- I tried to run it

[jira] [Issue Comment Deleted] (SPARK-15722) Wrong data when CTAS specifies schema

2016-06-02 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15722: -- Comment: was deleted (was: User 'andrewor14' has created a pull request for this issue:

[jira] [Updated] (SPARK-15716) Memory usage keep growing up in Spark Streaming

2016-06-02 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Chen updated SPARK-15716: - Description: Code: {code:java} import org.apache.hadoop.io.LongWritable; import

  1   2   3   >