[jira] [Assigned] (SPARK-24803) add support for numeric

[ https://issues.apache.org/jira/browse/SPARK-24803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24803: Assignee: (was: Apache Spark) > add support for numeric > --- >

[jira] [Commented] (SPARK-24803) add support for numeric

[ https://issues.apache.org/jira/browse/SPARK-24803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544000#comment-16544000 ] Apache Spark commented on SPARK-24803: -- User 'wangtao605' has created a pull request for this

[jira] [Assigned] (SPARK-24803) add support for numeric

[ https://issues.apache.org/jira/browse/SPARK-24803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24803: Assignee: Apache Spark > add support for numeric > --- > >

[jira] [Created] (SPARK-24803) add support for numeric

wangtao93 created SPARK-24803: - Summary: add support for numeric Key: SPARK-24803 URL: https://issues.apache.org/jira/browse/SPARK-24803 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-12850) Support bucket pruning (predicate pushdown for bucketed tables)

[ https://issues.apache.org/jira/browse/SPARK-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543988#comment-16543988 ] Hyukjin Kwon commented on SPARK-12850: -- Fixed in https://github.com/apache/spark/pull/20915 at

[jira] [Updated] (SPARK-12850) Support bucket pruning (predicate pushdown for bucketed tables)

[ https://issues.apache.org/jira/browse/SPARK-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-12850: - Fix Version/s: (was: 2.0.0) 2.4.0 > Support bucket pruning (predicate

[jira] [Assigned] (SPARK-24802) Optimization Rule Exclusion

[ https://issues.apache.org/jira/browse/SPARK-24802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24802: Assignee: (was: Apache Spark) > Optimization Rule Exclusion >

[jira] [Assigned] (SPARK-24802) Optimization Rule Exclusion

[ https://issues.apache.org/jira/browse/SPARK-24802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24802: Assignee: Apache Spark > Optimization Rule Exclusion > --- > >

[jira] [Commented] (SPARK-24802) Optimization Rule Exclusion

[ https://issues.apache.org/jira/browse/SPARK-24802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543752#comment-16543752 ] Apache Spark commented on SPARK-24802: -- User 'maryannxue' has created a pull request for this

[jira] [Updated] (SPARK-24802) Optimization Rule Exclusion

[ https://issues.apache.org/jira/browse/SPARK-24802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-24802: Description: Since Spark has provided fairly clear interfaces for adding user-defined

[jira] [Created] (SPARK-24802) Optimization Rule Exclusion

Maryann Xue created SPARK-24802: --- Summary: Optimization Rule Exclusion Key: SPARK-24802 URL: https://issues.apache.org/jira/browse/SPARK-24802 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-24766) CreateHiveTableAsSelect and InsertIntoHiveDir won't generate decimal column stats in parquet

[ https://issues.apache.org/jira/browse/SPARK-24766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543734#comment-16543734 ] Yuming Wang commented on SPARK-24766: - cc [~smilegator] [~cloud_fan] > CreateHiveTableAsSelect and

[jira] [Resolved] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

[ https://issues.apache.org/jira/browse/SPARK-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23831. - Resolution: Fixed Assignee: Yuming Wang Fix Version/s: 2.4.0 > Add org.apache.derby to

[jira] [Created] (SPARK-24801) Empty byte[] arrays in spark.network.sasl.SaslEncryption$EncryptedMessage can waste a lot of memory

Misha Dmitriev created SPARK-24801: -- Summary: Empty byte[] arrays in spark.network.sasl.SaslEncryption$EncryptedMessage can waste a lot of memory Key: SPARK-24801 URL:

[jira] [Assigned] (SPARK-23528) Add numIter to ClusteringSummary

[ https://issues.apache.org/jira/browse/SPARK-23528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-23528: --- Assignee: Marco Gaido > Add numIter to ClusteringSummary > > >

[jira] [Resolved] (SPARK-23528) Add numIter to ClusteringSummary

[ https://issues.apache.org/jira/browse/SPARK-23528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-23528. - Resolution: Fixed Fix Version/s: 2.4.0 Thanks! > Add numIter to ClusteringSummary >

[jira] [Updated] (SPARK-23528) Add numIter to ClusteringSummary

[ https://issues.apache.org/jira/browse/SPARK-23528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-23528: Description: Spark ML should expose vital statistics of the GMM model: * *Number of iterations* (actual,

[jira] [Updated] (SPARK-23528) Add numIter to ClusteringSummary

[ https://issues.apache.org/jira/browse/SPARK-23528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-23528: Summary: Add numIter to ClusteringSummary (was: Expose vital statistics of GaussianMixtureModel) > Add

[jira] [Resolved] (SPARK-24797) Analyzer should respect spark.sql.hive.convertMetastoreOrc/Parquet when build the data source table

[ https://issues.apache.org/jira/browse/SPARK-24797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nan Zhu resolved SPARK-24797. - Resolution: Won't Fix > Analyzer should respect spark.sql.hive.convertMetastoreOrc/Parquet when build

[jira] [Reopened] (SPARK-24776) AVRO unit test: use SQLTestUtils and Replace deprecated methods

[ https://issues.apache.org/jira/browse/SPARK-24776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reopened SPARK-24776: - > AVRO unit test: use SQLTestUtils and Replace deprecated methods >

[jira] [Commented] (SPARK-24800) Refactor Avro Serializer and Deserializer

[ https://issues.apache.org/jira/browse/SPARK-24800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543428#comment-16543428 ] Apache Spark commented on SPARK-24800: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-24800) Refactor Avro Serializer and Deserializer

[ https://issues.apache.org/jira/browse/SPARK-24800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24800: Assignee: Apache Spark > Refactor Avro Serializer and Deserializer >

[jira] [Assigned] (SPARK-24800) Refactor Avro Serializer and Deserializer

[ https://issues.apache.org/jira/browse/SPARK-24800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24800: Assignee: (was: Apache Spark) > Refactor Avro Serializer and Deserializer >

[jira] [Created] (SPARK-24800) Refactor Avro Serializer and Deserializer

Gengliang Wang created SPARK-24800: -- Summary: Refactor Avro Serializer and Deserializer Key: SPARK-24800 URL: https://issues.apache.org/jira/browse/SPARK-24800 Project: Spark Issue Type:

[jira] [Commented] (SPARK-21791) ORC should support column names with dot

[ https://issues.apache.org/jira/browse/SPARK-21791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543390#comment-16543390 ] Dongjoon Hyun commented on SPARK-21791: --- Hi, [~fpin]. This is resolved via new ORC file format.

[jira] [Resolved] (SPARK-24776) AVRO unit test: use SQLTestUtils and Replace deprecated methods

[ https://issues.apache.org/jira/browse/SPARK-24776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24776. - Resolution: Fixed Assignee: Gengliang Wang Fix Version/s: 2.4.0 > AVRO unit test: use

[jira] [Commented] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543372#comment-16543372 ] Xiao Li commented on SPARK-24741: - Please follow the umbrella JIRA:

[jira] [Commented] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543367#comment-16543367 ] Marek Novotny commented on SPARK-24741: --- Hi [~smilegator], The main motivation for creation of

[jira] [Resolved] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.

[ https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24781. - Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.4.0

[jira] [Commented] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.

[ https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543342#comment-16543342 ] Xiao Li commented on SPARK-24781: - cc [~jerryshao] The fix is merged. Please start the next rc of Spark

[jira] [Commented] (SPARK-24498) Add JDK compiler for runtime codegen

[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543336#comment-16543336 ] Marco Gaido commented on SPARK-24498: - [~maropu] yes, I remembered I had some troubles compiling the

[jira] [Assigned] (SPARK-24771) Upgrade AVRO version from 1.7.7 to 1.8

[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24771: Assignee: (was: Apache Spark) > Upgrade AVRO version from 1.7.7 to 1.8 >

[jira] [Assigned] (SPARK-24771) Upgrade AVRO version from 1.7.7 to 1.8

[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24771: Assignee: Apache Spark > Upgrade AVRO version from 1.7.7 to 1.8 >

[jira] [Commented] (SPARK-24771) Upgrade AVRO version from 1.7.7 to 1.8

[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543319#comment-16543319 ] Apache Spark commented on SPARK-24771: -- User 'gengliangwang' has created a pull request for this

[jira] [Commented] (SPARK-21791) ORC should support column names with dot

[ https://issues.apache.org/jira/browse/SPARK-21791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543291#comment-16543291 ] Furcy Pin commented on SPARK-21791: --- Hi, this bug is in fact only half-solved. Here is a minimal

[jira] [Comment Edited] (SPARK-24798) sortWithinPartitions(xx) will failed in java.lang.NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-24798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543204#comment-16543204 ] mahmoud mehdi edited comment on SPARK-24798 at 7/13/18 2:14 PM: Hello

[jira] [Commented] (SPARK-24798) sortWithinPartitions(xx) will failed in java.lang.NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-24798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543204#comment-16543204 ] mahmoud mehdi commented on SPARK-24798: --- Hello [~sypark212], Replacing Some by Option will solve

[jira] [Resolved] (SPARK-24713) AppMatser of spark streaming kafka OOM if there are hundreds of topics consumed

[ https://issues.apache.org/jira/browse/SPARK-24713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger resolved SPARK-24713. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21690

[jira] [Assigned] (SPARK-24713) AppMatser of spark streaming kafka OOM if there are hundreds of topics consumed

[ https://issues.apache.org/jira/browse/SPARK-24713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger reassigned SPARK-24713: -- Assignee: Yuanbo Liu > AppMatser of spark streaming kafka OOM if there are hundreds

[jira] [Comment Edited] (SPARK-24498) Add JDK compiler for runtime codegen

[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543025#comment-16543025 ] Takeshi Yamamuro edited comment on SPARK-24498 at 7/13/18 12:57 PM:

[jira] [Commented] (SPARK-24498) Add JDK compiler for runtime codegen

[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543025#comment-16543025 ] Takeshi Yamamuro commented on SPARK-24498: -- I found some weird gen'd code that jdk java

[jira] [Commented] (SPARK-24799) A solution of dealing with data skew in left,right,inner join

[ https://issues.apache.org/jira/browse/SPARK-24799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542974#comment-16542974 ] Takeshi Yamamuro commented on SPARK-24799: -- This topic seems to be related to

[jira] [Commented] (SPARK-24776) AVRO unit test: use SQLTestUtils and Replace deprecated methods

[ https://issues.apache.org/jira/browse/SPARK-24776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542940#comment-16542940 ] Apache Spark commented on SPARK-24776: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-24776) AVRO unit test: use SQLTestUtils and Replace deprecated methods

[ https://issues.apache.org/jira/browse/SPARK-24776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24776: Assignee: Apache Spark > AVRO unit test: use SQLTestUtils and Replace deprecated methods

[jira] [Assigned] (SPARK-24776) AVRO unit test: use SQLTestUtils and Replace deprecated methods

[ https://issues.apache.org/jira/browse/SPARK-24776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24776: Assignee: (was: Apache Spark) > AVRO unit test: use SQLTestUtils and Replace

[jira] [Created] (SPARK-24799) A solution of dealing with data skew in left,right,inner join

marymwu created SPARK-24799: --- Summary: A solution of dealing with data skew in left,right,inner join Key: SPARK-24799 URL: https://issues.apache.org/jira/browse/SPARK-24799 Project: Spark Issue

[jira] [Created] (SPARK-24798) sortWithinPartitions(xx) will failed in java.lang.NullPointerException

shengyao piao created SPARK-24798: - Summary: sortWithinPartitions(xx) will failed in java.lang.NullPointerException Key: SPARK-24798 URL: https://issues.apache.org/jira/browse/SPARK-24798 Project: