[jira] [Created] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-10 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-9814: --- Summary: EqualNotNull not passing to data sources Key: SPARK-9814 URL: https://issues.apache.org/jira/browse/SPARK-9814 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9340. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8070

[jira] [Updated] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9814: Description: When data sources (such as Parquet) tries to filter data when reading from HDFS (not

[jira] [Commented] (SPARK-9813) Incorrect UNION ALL behavior

2015-08-10 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681246#comment-14681246 ] Herman van Hovell commented on SPARK-9813: -- So I am not to sure if we want to

[jira] [Assigned] (SPARK-9815) Rename PlatformDependent.UNSAFE - Platform

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9815: --- Assignee: Apache Spark (was: Reynold Xin) Rename PlatformDependent.UNSAFE - Platform

[jira] [Created] (SPARK-9815) Rename PlatformDependent.UNSAFE - Platform

2015-08-10 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-9815: -- Summary: Rename PlatformDependent.UNSAFE - Platform Key: SPARK-9815 URL: https://issues.apache.org/jira/browse/SPARK-9815 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-9815) Rename PlatformDependent.UNSAFE - Platform

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681250#comment-14681250 ] Apache Spark commented on SPARK-9815: - User 'rxin' has created a pull request for this

[jira] [Assigned] (SPARK-9815) Rename PlatformDependent.UNSAFE - Platform

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9815: --- Assignee: Reynold Xin (was: Apache Spark) Rename PlatformDependent.UNSAFE - Platform

[jira] [Updated] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9814: Description: When data sources (such as Parquet) tries to filter data when reading from HDFS (not

[jira] [Commented] (SPARK-9790) [YARN] Expose in WebUI if NodeManager is the reason why executors were killed.

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681249#comment-14681249 ] Apache Spark commented on SPARK-9790: - User 'markgrover' has created a pull request

[jira] [Assigned] (SPARK-9790) [YARN] Expose in WebUI if NodeManager is the reason why executors were killed.

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9790: --- Assignee: (was: Apache Spark) [YARN] Expose in WebUI if NodeManager is the reason why

[jira] [Assigned] (SPARK-9790) [YARN] Expose in WebUI if NodeManager is the reason why executors were killed.

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9790: --- Assignee: Apache Spark [YARN] Expose in WebUI if NodeManager is the reason why executors

[jira] [Resolved] (SPARK-9363) SortMergeJoin operator should support UnsafeRow

2015-08-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-9363. Resolution: Fixed Fix Version/s: 1.5.0 SortMergeJoin operator should support UnsafeRow

[jira] [Resolved] (SPARK-9729) Sort Merge Join for Left and Right Outer Join

2015-08-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-9729. Resolution: Fixed Fix Version/s: 1.5.0 Sort Merge Join for Left and Right Outer Join

[jira] [Updated] (SPARK-9790) [YARN] Expose in WebUI if NodeManager is the reason why executors were killed.

2015-08-10 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated SPARK-9790: --- Attachment: error_showing_in_UI.png Attaching an image of what the error message in the UI would now

[jira] [Comment Edited] (SPARK-9790) [YARN] Expose in WebUI if NodeManager is the reason why executors were killed.

2015-08-10 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681263#comment-14681263 ] Mark Grover edited comment on SPARK-9790 at 8/11/15 5:17 AM: -

[jira] [Updated] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9814: Environment: (was: Centos 6.6) EqualNotNull not passing to data sources

[jira] [Updated] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9814: Component/s: (was: Input/Output) SQL EqualNotNull not passing to data sources

[jira] [Commented] (SPARK-7726) Maven Install Breaks When Upgrading Scala 2.11.2--[2.11.3 or higher]

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681277#comment-14681277 ] Apache Spark commented on SPARK-7726: - User 'pwendell' has created a pull request for

[jira] [Resolved] (SPARK-9743) Scanning a HadoopFsRelation shouldn't requrire refreshing

2015-08-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-9743. - Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8035

[jira] [Updated] (SPARK-9714) Cannot insert into a table using pySpark

2015-08-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-9714: Sprint: Spark 1.5 doc/QA sprint Cannot insert into a table using pySpark

[jira] [Updated] (SPARK-9781) KCL Workers should be configurable from Spark configuration

2015-08-10 Thread Anton Nekhaev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Nekhaev updated SPARK-9781: - Description: Currently the KinesisClientLibConfiguration for KCL Workers is created within the

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680355#comment-14680355 ] Ryan Blue commented on SPARK-9340: -- Sorry to jump in late on this issue... I think you're

[jira] [Updated] (SPARK-9600) DataFrameWriter.saveAsTable always writes data to /user/hive/warehouse

2015-08-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-9600: Priority: Blocker (was: Critical) DataFrameWriter.saveAsTable always writes data to /user/hive/warehouse

[jira] [Created] (SPARK-9782) Add support for YARN application tags running Spark on YARN

2015-08-10 Thread Dennis Huo (JIRA)
Dennis Huo created SPARK-9782: - Summary: Add support for YARN application tags running Spark on YARN Key: SPARK-9782 URL: https://issues.apache.org/jira/browse/SPARK-9782 Project: Spark Issue

[jira] [Commented] (SPARK-7751) Add @since to stable and experimental methods in MLlib

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680423#comment-14680423 ] Joseph K. Bradley commented on SPARK-7751: -- [~mengxr] I haven't reviewed these

[jira] [Updated] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9783: -- Sprint: Spark 1.5 doc/QA sprint Environment: (was: PR #8035 made a quick fix for SPARK-9743

[jira] [Commented] (SPARK-9622) DecisionTreeRegressor: provide variance of prediction

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680510#comment-14680510 ] Joseph K. Bradley commented on SPARK-9622: -- OK, before you do though, it'd be

[jira] [Updated] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9340: -- Sprint: Spark 1.5 doc/QA sprint Target Version/s: 1.5.0 CatalystSchemaConverter and

[jira] [Created] (SPARK-9785) HashPartitioning guarantees / compatibleWith violate those methods' contracts

2015-08-10 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-9785: - Summary: HashPartitioning guarantees / compatibleWith violate those methods' contracts Key: SPARK-9785 URL: https://issues.apache.org/jira/browse/SPARK-9785 Project: Spark

[jira] [Created] (SPARK-9788) LDA docConcentration, gammaShape 1.5 binary incompatibility fixes

2015-08-10 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-9788: Summary: LDA docConcentration, gammaShape 1.5 binary incompatibility fixes Key: SPARK-9788 URL: https://issues.apache.org/jira/browse/SPARK-9788 Project:

[jira] [Commented] (SPARK-9782) Add support for YARN application tags running Spark on YARN

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680413#comment-14680413 ] Apache Spark commented on SPARK-9782: - User 'dennishuo' has created a pull request for

[jira] [Updated] (SPARK-9755) Add method documentation to MultivariateOnlineSummarizer

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-9755: - Shepherd: Joseph K. Bradley Assignee: Feynman Liang Add method documentation to

[jira] [Commented] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680457#comment-14680457 ] Cheng Lian commented on SPARK-9783: --- cc [~yhuai] Use SqlNewHadoopRDD in JSONRelation

[jira] [Updated] (SPARK-9720) spark.ml Identifiable types should have UID in toString methods

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-9720: - Description: It would be nice to include the UID (instance name) in toString methods.

[jira] [Updated] (SPARK-9720) spark.ml Identifiable types should have UID in toString methods

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-9720: - Assignee: Bertrand Dechoux spark.ml Identifiable types should have UID in toString

[jira] [Commented] (SPARK-9720) spark.ml Identifiable types should have UID in toString methods

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680520#comment-14680520 ] Joseph K. Bradley commented on SPARK-9720: -- Oh sorry! I shouldn't have said

[jira] [Assigned] (SPARK-9785) HashPartitioning compatibility should consider expression ordering

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9785: --- Assignee: Apache Spark (was: Josh Rosen) HashPartitioning compatibility should consider

[jira] [Created] (SPARK-9790) [YARN] Expose in WebUI if NodeManager is the reason why executors were killed.

2015-08-10 Thread Mark Grover (JIRA)
Mark Grover created SPARK-9790: -- Summary: [YARN] Expose in WebUI if NodeManager is the reason why executors were killed. Key: SPARK-9790 URL: https://issues.apache.org/jira/browse/SPARK-9790 Project:

[jira] [Commented] (SPARK-9570) Consistent recommendation for submitting spark apps to YARN, -master yarn --deploy-mode x vs -master yarn-x'.

2015-08-10 Thread Guru Medasani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680379#comment-14680379 ] Guru Medasani commented on SPARK-9570: -- I won't have time to look at this. Neelesh

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680385#comment-14680385 ] Apache Spark commented on SPARK-9340: - User 'liancheng' has created a pull request for

[jira] [Assigned] (SPARK-9570) Consistent recommendation for submitting spark apps to YARN, -master yarn --deploy-mode x vs -master yarn-x'.

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9570: --- Assignee: (was: Apache Spark) Consistent recommendation for submitting spark apps to

[jira] [Commented] (SPARK-9570) Consistent recommendation for submitting spark apps to YARN, -master yarn --deploy-mode x vs -master yarn-x'.

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680392#comment-14680392 ] Apache Spark commented on SPARK-9570: - User 'nssalian' has created a pull request for

[jira] [Assigned] (SPARK-9570) Consistent recommendation for submitting spark apps to YARN, -master yarn --deploy-mode x vs -master yarn-x'.

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9570: --- Assignee: Apache Spark Consistent recommendation for submitting spark apps to YARN, -master

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680418#comment-14680418 ] Cheng Lian commented on SPARK-9340: --- Thanks for the clarification. In [PR

[jira] [Updated] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9340: -- Summary: CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields

[jira] [Created] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

2015-08-10 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-9783: - Summary: Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call Key: SPARK-9783 URL: https://issues.apache.org/jira/browse/SPARK-9783 Project: Spark

[jira] [Assigned] (SPARK-9784) Exchange.isUnsafe should check whether codegen and unsafe are enabled

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9784: --- Assignee: Apache Spark (was: Josh Rosen) Exchange.isUnsafe should check whether codegen

[jira] [Commented] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680523#comment-14680523 ] Cheng Lian commented on SPARK-9340: --- Great, would you mind to leave a LGTM on the GitHub

[jira] [Updated] (SPARK-9785) HashPartitioning compatibility should be sensitive to expression ordering

2015-08-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-9785: -- Summary: HashPartitioning compatibility should be sensitive to expression ordering (was:

[jira] [Assigned] (SPARK-9574) Review the contents of uber JARs spark-streaming-XXX-assembly

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9574: --- Assignee: Apache Spark (was: Shixiong Zhu) Review the contents of uber JARs

[jira] [Commented] (SPARK-9574) Review the contents of uber JARs spark-streaming-XXX-assembly

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680362#comment-14680362 ] Apache Spark commented on SPARK-9574: - User 'zsxwing' has created a pull request for

[jira] [Assigned] (SPARK-9782) Add support for YARN application tags running Spark on YARN

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9782: --- Assignee: (was: Apache Spark) Add support for YARN application tags running Spark on

[jira] [Assigned] (SPARK-9782) Add support for YARN application tags running Spark on YARN

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9782: --- Assignee: Apache Spark Add support for YARN application tags running Spark on YARN

[jira] [Commented] (SPARK-9782) Add support for YARN application tags running Spark on YARN

2015-08-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680415#comment-14680415 ] Sean Owen commented on SPARK-9782: -- This is different from

[jira] [Assigned] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-9340: - Assignee: Cheng Lian CatalystSchemaConverter and CatalystRowConverter don't handle unannotated

[jira] [Resolved] (SPARK-9450) HashedRelation.get() could return an Iterator[Row] instead of Seq[Row]

2015-08-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-9450. --- Resolution: Invalid I'm going to resolve this as Invalid, since it turns out that we need to return

[jira] [Created] (SPARK-9784) Exchange.isUnsafe should check whether codegen and unsafe are enabled

2015-08-10 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-9784: - Summary: Exchange.isUnsafe should check whether codegen and unsafe are enabled Key: SPARK-9784 URL: https://issues.apache.org/jira/browse/SPARK-9784 Project: Spark

[jira] [Comment Edited] (SPARK-9658) ML 1.5 QA: API: Binary incompatible changes

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14659182#comment-14659182 ] Joseph K. Bradley edited comment on SPARK-9658 at 8/10/15 6:49 PM:

[jira] [Updated] (SPARK-9785) HashPartitioning compatibility should consider expression ordering

2015-08-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-9785: -- Summary: HashPartitioning compatibility should consider expression ordering (was: HashPartitioning

[jira] [Created] (SPARK-9789) Reinstate LogisticRegression threshold Param

2015-08-10 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-9789: Summary: Reinstate LogisticRegression threshold Param Key: SPARK-9789 URL: https://issues.apache.org/jira/browse/SPARK-9789 Project: Spark Issue

[jira] [Resolved] (SPARK-9710) RPackageUtilsSuite fails if R is not installed

2015-08-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-9710. -- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request

[jira] [Updated] (SPARK-9710) RPackageUtilsSuite fails if R is not installed

2015-08-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-9710: - Assignee: Marcelo Vanzin RPackageUtilsSuite fails if R is not installed

[jira] [Resolved] (SPARK-7536) Audit MLlib Python API for 1.4

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-7536. -- Resolution: Done Thanks [~yanboliang] for putting this (very successful) list together

[jira] [Commented] (SPARK-9784) Exchange.isUnsafe should check whether codegen and unsafe are enabled

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680482#comment-14680482 ] Apache Spark commented on SPARK-9784: - User 'JoshRosen' has created a pull request for

[jira] [Assigned] (SPARK-9784) Exchange.isUnsafe should check whether codegen and unsafe are enabled

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9784: --- Assignee: Josh Rosen (was: Apache Spark) Exchange.isUnsafe should check whether codegen

[jira] [Commented] (SPARK-7778) Add standard deviation aggregate expression

2015-08-10 Thread Rakesh Chalasani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680491#comment-14680491 ] Rakesh Chalasani commented on SPARK-7778: - Hi SsuTing: The aggregate expression

[jira] [Updated] (SPARK-9711) Unable to run spark after restarting cluster with spark-ec2

2015-08-10 Thread Guangyang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangyang Li updated SPARK-9711: Description: With Spark 1.4.1 and YARN client mode, my application works at the first time the

[jira] [Comment Edited] (SPARK-9663) ML Python API coverage issues found during 1.5 QA

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14659207#comment-14659207 ] Joseph K. Bradley edited comment on SPARK-9663 at 8/10/15 6:32 PM:

[jira] [Assigned] (SPARK-9574) Review the contents of uber JARs spark-streaming-XXX-assembly

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9574: --- Assignee: Shixiong Zhu (was: Apache Spark) Review the contents of uber JARs

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680389#comment-14680389 ] Cheng Lian commented on SPARK-9340: --- [~damianguy] Would you mind to help reviewing [PR

[jira] [Commented] (SPARK-9782) Add support for YARN application tags running Spark on YARN

2015-08-10 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680431#comment-14680431 ] Dennis Huo commented on SPARK-9782: --- Correct, from what I understand, the node labels

[jira] [Updated] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9340: -- Description: SPARK-6776 and SPARK-6777 followed {{parquet-avro}} to implement backwards-compatibility

[jira] [Updated] (SPARK-9450) [INVALID] HashedRelation.get() could return an Iterator[Row] instead of Seq[Row]

2015-08-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-9450: -- Summary: [INVALID] HashedRelation.get() could return an Iterator[Row] instead of Seq[Row] (was:

[jira] [Resolved] (SPARK-9755) Add method documentation to MultivariateOnlineSummarizer

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-9755. -- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8045

[jira] [Commented] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Damian Guy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680492#comment-14680492 ] Damian Guy commented on SPARK-9340: --- Code looks good and it works as expected. Tests

[jira] [Created] (SPARK-9786) Test backpressure

2015-08-10 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-9786: Summary: Test backpressure Key: SPARK-9786 URL: https://issues.apache.org/jira/browse/SPARK-9786 Project: Spark Issue Type: Sub-task Components:

[jira] [Updated] (SPARK-9787) Test for memory leaks using the streaming tests in spark-perf

2015-08-10 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-9787: - Assignee: Shixiong Zhu Test for memory leaks using the streaming tests in spark-perf

[jira] [Created] (SPARK-9787) Test for memory leaks using the streaming tests in spark-perf

2015-08-10 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-9787: Summary: Test for memory leaks using the streaming tests in spark-perf Key: SPARK-9787 URL: https://issues.apache.org/jira/browse/SPARK-9787 Project: Spark

[jira] [Commented] (SPARK-9785) HashPartitioning compatibility should consider expression ordering

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680550#comment-14680550 ] Apache Spark commented on SPARK-9785: - User 'JoshRosen' has created a pull request for

[jira] [Assigned] (SPARK-9785) HashPartitioning compatibility should consider expression ordering

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9785: --- Assignee: Josh Rosen (was: Apache Spark) HashPartitioning compatibility should consider

[jira] [Updated] (SPARK-9750) SparseMatrix should override equals

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-9750: - Assignee: Feynman Liang SparseMatrix should override equals

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680277#comment-14680277 ] Cheng Lian commented on SPARK-9340: --- Ah, thanks a lot! I see the problem now.

[jira] [Created] (SPARK-9781) KCL Workers should be configurable from Spark configuration

2015-08-10 Thread Anton Nekhaev (JIRA)
Anton Nekhaev created SPARK-9781: Summary: KCL Workers should be configurable from Spark configuration Key: SPARK-9781 URL: https://issues.apache.org/jira/browse/SPARK-9781 Project: Spark

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Damian Guy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680281#comment-14680281 ] Damian Guy commented on SPARK-9340: --- Thanks. I'm sure there is a simpler solution to

[jira] [Commented] (SPARK-9788) LDA docConcentration, gammaShape 1.5 binary incompatibility fixes

2015-08-10 Thread Feynman Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680568#comment-14680568 ] Feynman Liang commented on SPARK-9788: -- Assign to me LDA docConcentration,

[jira] [Commented] (SPARK-9790) [YARN] Expose in WebUI if NodeManager is the reason why executors were killed.

2015-08-10 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680570#comment-14680570 ] Mark Grover commented on SPARK-9790: I am working on this, will file a

[jira] [Created] (SPARK-9793) PySpark DenseVector, SparseVector should override __eq__

2015-08-10 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-9793: Summary: PySpark DenseVector, SparseVector should override __eq__ Key: SPARK-9793 URL: https://issues.apache.org/jira/browse/SPARK-9793 Project: Spark

[jira] [Created] (SPARK-9795) Dynamic allocation: avoid double counting when killing same executor

2015-08-10 Thread Andrew Or (JIRA)
Andrew Or created SPARK-9795: Summary: Dynamic allocation: avoid double counting when killing same executor Key: SPARK-9795 URL: https://issues.apache.org/jira/browse/SPARK-9795 Project: Spark

[jira] [Commented] (SPARK-9788) LDA docConcentration, gammaShape 1.5 binary incompatibility fixes

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680666#comment-14680666 ] Joseph K. Bradley commented on SPARK-9788: -- Yeah, I guess we should revert

[jira] [Resolved] (SPARK-9784) Exchange.isUnsafe should check whether codegen and unsafe are enabled

2015-08-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-9784. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8073

[jira] [Updated] (SPARK-9794) ISO DateTime parser is too strict

2015-08-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-9794: -- Affects Version/s: 1.2.2 1.3.1 1.4.1 ISO DateTime parser

[jira] [Commented] (SPARK-9794) ISO DateTime parser is too strict

2015-08-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680688#comment-14680688 ] Josh Rosen commented on SPARK-9794: --- The same code exists in 1.4.0 and 1.4.1:

[jira] [Updated] (SPARK-9795) Dynamic allocation: avoid double counting when killing same executor twice

2015-08-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-9795: - Summary: Dynamic allocation: avoid double counting when killing same executor twice (was: Dynamic

[jira] [Assigned] (SPARK-9795) Dynamic allocation: avoid double counting when killing same executor

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9795: --- Assignee: Andrew Or (was: Apache Spark) Dynamic allocation: avoid double counting when

[jira] [Commented] (SPARK-9795) Dynamic allocation: avoid double counting when killing same executor

2015-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680738#comment-14680738 ] Apache Spark commented on SPARK-9795: - User 'andrewor14' has created a pull request

[jira] [Created] (SPARK-9791) Review API for developer and experimental tags

2015-08-10 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-9791: Summary: Review API for developer and experimental tags Key: SPARK-9791 URL: https://issues.apache.org/jira/browse/SPARK-9791 Project: Spark Issue Type:

[jira] [Updated] (SPARK-9750) DenseMatrix, SparseMatrix should override equals

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-9750: - Description:

[jira] [Updated] (SPARK-9750) DenseMatrix, SparseMatrix should override equals

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-9750: - Summary: DenseMatrix, SparseMatrix should override equals (was: SparseMatrix should

[jira] [Updated] (SPARK-9766) check and add missing docs for PySpark ML

2015-08-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-9766: - Target Version/s: 1.5.0 check and add missing docs for PySpark ML

  1   2   3   >