[jira] [Created] (SPARK-24635) Remove Blocks class

2018-06-23 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-24635: --- Summary: Remove Blocks class Key: SPARK-24635 URL: https://issues.apache.org/jira/browse/SPARK-24635 Project: Spark Issue Type: Improvement C

[jira] [Assigned] (SPARK-24635) Remove Blocks class

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24635: Assignee: (was: Apache Spark) > Remove Blocks class > --- > >

[jira] [Assigned] (SPARK-24635) Remove Blocks class

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24635: Assignee: Apache Spark > Remove Blocks class > --- > > Ke

[jira] [Commented] (SPARK-24635) Remove Blocks class

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520996#comment-16520996 ] Apache Spark commented on SPARK-24635: -- User 'viirya' has created a pull request fo

[jira] [Created] (SPARK-24636) Type Coercion of Arrays for array_join Function

2018-06-23 Thread Marek Novotny (JIRA)
Marek Novotny created SPARK-24636: - Summary: Type Coercion of Arrays for array_join Function Key: SPARK-24636 URL: https://issues.apache.org/jira/browse/SPARK-24636 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24636) Type Coercion of Arrays for array_join Function

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521037#comment-16521037 ] Apache Spark commented on SPARK-24636: -- User 'mn-mikke' has created a pull request

[jira] [Assigned] (SPARK-24636) Type Coercion of Arrays for array_join Function

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24636: Assignee: Apache Spark > Type Coercion of Arrays for array_join Function > --

[jira] [Assigned] (SPARK-24636) Type Coercion of Arrays for array_join Function

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24636: Assignee: (was: Apache Spark) > Type Coercion of Arrays for array_join Function > ---

[jira] [Updated] (SPARK-24636) Type Coercion of Arrays for array_join Function

2018-06-23 Thread Marek Novotny (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marek Novotny updated SPARK-24636: -- Description: The description of SPARK-23916 refers to the implementation of {{array_join}} fu

[jira] [Assigned] (SPARK-24633) arrays_zip function's code generator splits input processing incorrectly

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24633: Assignee: (was: Apache Spark) > arrays_zip function's code generator splits input pro

[jira] [Assigned] (SPARK-24633) arrays_zip function's code generator splits input processing incorrectly

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24633: Assignee: Apache Spark > arrays_zip function's code generator splits input processing inc

[jira] [Commented] (SPARK-24633) arrays_zip function's code generator splits input processing incorrectly

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521068#comment-16521068 ] Apache Spark commented on SPARK-24633: -- User 'mgaido91' has created a pull request

[jira] [Created] (SPARK-24637) Add metrics regarding state and watermark to dropwizard metrics

2018-06-23 Thread Jungtaek Lim (JIRA)
Jungtaek Lim created SPARK-24637: Summary: Add metrics regarding state and watermark to dropwizard metrics Key: SPARK-24637 URL: https://issues.apache.org/jira/browse/SPARK-24637 Project: Spark

[jira] [Updated] (SPARK-24637) Add metrics regarding state and watermark to dropwizard metrics

2018-06-23 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-24637: - Description: Though Spark provides option to enable stream metrics into Dropwizard, it only exp

[jira] [Commented] (SPARK-24637) Add metrics regarding state and watermark to dropwizard metrics

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521077#comment-16521077 ] Apache Spark commented on SPARK-24637: -- User 'HeartSaVioR' has created a pull reque

[jira] [Assigned] (SPARK-24637) Add metrics regarding state and watermark to dropwizard metrics

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24637: Assignee: (was: Apache Spark) > Add metrics regarding state and watermark to dropwiza

[jira] [Assigned] (SPARK-24637) Add metrics regarding state and watermark to dropwizard metrics

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24637: Assignee: Apache Spark > Add metrics regarding state and watermark to dropwizard metrics

[jira] [Created] (SPARK-24638) StringStartsWith support push down

2018-06-23 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-24638: --- Summary: StringStartsWith support push down Key: SPARK-24638 URL: https://issues.apache.org/jira/browse/SPARK-24638 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-24639) Add three configs in the doc

2018-06-23 Thread xueyu (JIRA)
xueyu created SPARK-24639: - Summary: Add three configs in the doc Key: SPARK-24639 URL: https://issues.apache.org/jira/browse/SPARK-24639 Project: Spark Issue Type: Improvement Components:

[jira] [Assigned] (SPARK-24638) StringStartsWith support push down

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24638: Assignee: Apache Spark > StringStartsWith support push down > ---

[jira] [Assigned] (SPARK-24638) StringStartsWith support push down

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24638: Assignee: (was: Apache Spark) > StringStartsWith support push down >

[jira] [Commented] (SPARK-24638) StringStartsWith support push down

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521099#comment-16521099 ] Apache Spark commented on SPARK-24638: -- User 'wangyum' has created a pull request f

[jira] [Updated] (SPARK-24639) Add three configs in the doc

2018-06-23 Thread xueyu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xueyu updated SPARK-24639: -- External issue URL: https://github.com/apache/spark/pull/21624 > Add three configs in the doc > --

[jira] [Assigned] (SPARK-24639) Add three configs in the doc

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24639: Assignee: Apache Spark > Add three configs in the doc > > >

[jira] [Commented] (SPARK-24639) Add three configs in the doc

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521102#comment-16521102 ] Apache Spark commented on SPARK-24639: -- User 'xueyumusic' has created a pull reques

[jira] [Assigned] (SPARK-24639) Add three configs in the doc

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24639: Assignee: (was: Apache Spark) > Add three configs in the doc > --

[jira] [Commented] (SPARK-23710) Upgrade Hive to 2.3.2

2018-06-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521200#comment-16521200 ] Sean Owen commented on SPARK-23710: --- My lightly-informed take is: * The fork must go

[jira] [Commented] (SPARK-20202) Remove references to org.spark-project.hive

2018-06-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521203#comment-16521203 ] Hyukjin Kwon commented on SPARK-20202: -- Hi all, what do you guys think about eplaci

[jira] [Comment Edited] (SPARK-20202) Remove references to org.spark-project.hive

2018-06-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521203#comment-16521203 ] Hyukjin Kwon edited comment on SPARK-20202 at 6/23/18 3:53 PM: ---

[jira] [Issue Comment Deleted] (SPARK-18755) Add Randomized Grid Search to Spark ML

2018-06-23 Thread Teng Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teng Peng updated SPARK-18755: -- Comment: was deleted (was: [~yuhaoyan] Is this what you are looking for: after we build the grid, we

[jira] [Updated] (SPARK-24625) put all the backward compatible behavior change configs under spark.sql.legacy.*

2018-06-23 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24625: Target Version/s: 3.0.0 > put all the backward compatible behavior change configs under > spark.sql.legac

[jira] [Created] (SPARK-24640) size(null) returns null

2018-06-23 Thread Xiao Li (JIRA)
Xiao Li created SPARK-24640: --- Summary: size(null) returns null Key: SPARK-24640 URL: https://issues.apache.org/jira/browse/SPARK-24640 Project: Spark Issue Type: Improvement Components:

[jira] [Updated] (SPARK-24640) size(null) returns null

2018-06-23 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24640: Labels: api (was: ) > size(null) returns null > > > Key: SPARK-

[jira] [Updated] (SPARK-17427) function SIZE should return -1 when parameter is null

2018-06-23 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-17427: Labels: api releasenotes (was: releasenotes) > function SIZE should return -1 when parameter is null > --

[jira] [Updated] (SPARK-23507) Migrate file-based data sources to data source v2

2018-06-23 Thread Teng Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teng Peng updated SPARK-23507: -- Issue Type: Umbrella (was: Improvement) > Migrate file-based data sources to data source v2 > ---

[jira] [Resolved] (SPARK-24190) lineSep shouldn't be required in JSON write

2018-06-23 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24190. - Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 2.4.0 > lineSep shouldn't be requi

[jira] [Resolved] (SPARK-24324) Pandas Grouped Map UserDefinedFunction mixes column labels

2018-06-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24324. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21427 [https://gi

[jira] [Assigned] (SPARK-24324) Pandas Grouped Map UserDefinedFunction mixes column labels

2018-06-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24324: Assignee: Bryan Cutler > Pandas Grouped Map UserDefinedFunction mixes column labels > ---

[jira] [Commented] (SPARK-24206) Improve DataSource benchmark code for read and pushdown

2018-06-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521355#comment-16521355 ] Apache Spark commented on SPARK-24206: -- User 'maropu' has created a pull request fo

[jira] [Commented] (SPARK-24631) Cannot up cast column from bigint to smallint as it may truncate

2018-06-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521393#comment-16521393 ] Hyukjin Kwon commented on SPARK-24631: -- [~HadoopSiva], you mean it's resolved in th

[jira] [Updated] (SPARK-24568) Code refactoring for DataType equalsXXX methods

2018-06-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24568: - Fix Version/s: (was: 2.4.0) > Code refactoring for DataType equalsXXX methods >

[jira] [Commented] (SPARK-24568) Code refactoring for DataType equalsXXX methods

2018-06-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521394#comment-16521394 ] Hyukjin Kwon commented on SPARK-24568: -- (Please avoid to set the fix version which

[jira] [Resolved] (SPARK-24517) Bug in loading unstructured data

2018-06-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24517. -- Resolution: Invalid >From my look, it sounds specific to the datasource. Let me resolve this u

[jira] [Commented] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521401#comment-16521401 ] Hyukjin Kwon commented on SPARK-24530: -- Probably it's related with Sphinx's version