[jira] [Commented] (SPARK-21748) Migrate the implementation of HashingTF from MLlib to ML

[ https://issues.apache.org/jira/browse/SPARK-21748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133947#comment-16133947 ] Yan Facai (颜发才) commented on SPARK-21748: - There seems to be something wrong with

[jira] [Comment Edited] (SPARK-21748) Migrate the implementation of HashingTF from MLlib to ML

[ https://issues.apache.org/jira/browse/SPARK-21748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133947#comment-16133947 ] Yan Facai (颜发才) edited comment on SPARK-21748 at 8/19/17 4:43 AM: -

[jira] [Commented] (SPARK-21720) Filter predicate with many conditions throw stackoverflow error

[ https://issues.apache.org/jira/browse/SPARK-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133878#comment-16133878 ] poplav commented on SPARK-21720: [~kiszk], Thank you for the update. I am following your

[jira] [Resolved] (SPARK-21566) Python method for summary

[ https://issues.apache.org/jira/browse/SPARK-21566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-21566. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18762 [https://github.com/ap

[jira] [Assigned] (SPARK-21566) Python method for summary

[ https://issues.apache.org/jira/browse/SPARK-21566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-21566: --- Assignee: Andrew Ray > Python method for summary > - > > Key

[jira] [Created] (SPARK-21788) Handle more exceptions when stopping a streaming query

Shixiong Zhu created SPARK-21788: Summary: Handle more exceptions when stopping a streaming query Key: SPARK-21788 URL: https://issues.apache.org/jira/browse/SPARK-21788 Project: Spark Issue

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:29 PM: -

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:19 PM: -

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:13 PM: -

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:12 PM: -

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:12 PM: -

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:11 PM: -

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133565#comment-16133565 ] Stavros Kontopoulos commented on SPARK-21752: - [~jsnowacki] What I am doing i

[jira] [Commented] (SPARK-21787) Support for pushing down filters for date types in ORC

[ https://issues.apache.org/jira/browse/SPARK-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133522#comment-16133522 ] Dongjoon Hyun commented on SPARK-21787: --- I'll make a PR for this. > Support for pu

[jira] [Commented] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133493#comment-16133493 ] Sean Owen commented on SPARK-21770: --- Yeah good point, the better question is perhaps wh

[jira] [Commented] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133483#comment-16133483 ] Joseph K. Bradley commented on SPARK-21770: --- I vaguely recall discussing this b

[jira] [Commented] (SPARK-21784) Add ALTER TABLE ADD CONSTRANT DDL to support defining primary key and foreign keys

[ https://issues.apache.org/jira/browse/SPARK-21784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133381#comment-16133381 ] Xiao Li commented on SPARK-21784: - https://github.com/apache/spark/pull/18994 > Add ALTE

[jira] [Commented] (SPARK-21787) Support for pushing down filters for date types in ORC

[ https://issues.apache.org/jira/browse/SPARK-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133378#comment-16133378 ] Dongjoon Hyun commented on SPARK-21787: --- Thank you for reporting! I linked this to

[jira] [Commented] (SPARK-21772) HiveException unable to move results from srcf to destf in InsertIntoHiveTable

[ https://issues.apache.org/jira/browse/SPARK-21772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133271#comment-16133271 ] Xiao Li commented on SPARK-21772: - https://github.com/apache/spark/pull/18985 > HiveExc

[jira] [Resolved] (SPARK-21213) Support collecting partition-level statistics: rowCount and sizeInBytes

[ https://issues.apache.org/jira/browse/SPARK-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21213. - Resolution: Fixed Fix Version/s: 2.3.0 > Support collecting partition-level statistics: rowCount a

[jira] [Assigned] (SPARK-21213) Support collecting partition-level statistics: rowCount and sizeInBytes

[ https://issues.apache.org/jira/browse/SPARK-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-21213: --- Assignee: Maria > Support collecting partition-level statistics: rowCount and sizeInBytes >

[jira] [Commented] (SPARK-17423) Support IGNORE NULLS option in Window functions

[ https://issues.apache.org/jira/browse/SPARK-17423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133095#comment-16133095 ] Gaurav Shah commented on SPARK-17423: - is there a way to do second non-null value ?

[jira] [Commented] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.

[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133091#comment-16133091 ] Hyukjin Kwon commented on SPARK-21677: -- Issue resolved by pull request 18930 https:/

[jira] [Resolved] (SPARK-21778) Simpler Dataset.sample API in Scala / Java

[ https://issues.apache.org/jira/browse/SPARK-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21778. -- Resolution: Resolved Assignee: Reynold Xin Fix Version/s: 2.3.0 Issue resolved

[jira] [Created] (SPARK-21787) Support for pushing down filters for date types in ORC

Stefan de Koning created SPARK-21787: Summary: Support for pushing down filters for date types in ORC Key: SPARK-21787 URL: https://issues.apache.org/jira/browse/SPARK-21787 Project: Spark

[jira] [Resolved] (SPARK-21768) spark.csv.read Empty String Parsed as NULL when nullValue is Set

[ https://issues.apache.org/jira/browse/SPARK-21768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21768. --- Resolution: Duplicate > spark.csv.read Empty String Parsed as NULL when nullValue is Set > --

[jira] [Commented] (SPARK-21768) spark.csv.read Empty String Parsed as NULL when nullValue is Set

[ https://issues.apache.org/jira/browse/SPARK-21768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16132191#comment-16132191 ] Marco Gaido commented on SPARK-21768: - This is a duplicate of SPARK-17916. > spark.c

[jira] [Resolved] (SPARK-4981) Add a streaming singular value decomposition

[ https://issues.apache.org/jira/browse/SPARK-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4981. -- Resolution: Won't Fix > Add a streaming singular value decomposition > -

[jira] [Commented] (SPARK-4981) Add a streaming singular value decomposition

[ https://issues.apache.org/jira/browse/SPARK-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16132118#comment-16132118 ] Nick Pentreath commented on SPARK-4981: --- Hey folks, as interesting as this would be,

[jira] [Created] (SPARK-21786) The 'spark.sql.parquet.compression.codec' configuration doesn't take effect on tables with partition field(s)

Jinhua Fu created SPARK-21786: - Summary: The 'spark.sql.parquet.compression.codec' configuration doesn't take effect on tables with partition field(s) Key: SPARK-21786 URL: https://issues.apache.org/jira/browse/SPARK-

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131945#comment-16131945 ] Jakub Nowacki edited comment on SPARK-21752 at 8/18/17 9:11 AM: ---

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131945#comment-16131945 ] Jakub Nowacki commented on SPARK-21752: --- [~skonto] What you are doing is in fact st

[jira] [Commented] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131921#comment-16131921 ] StanZhai commented on SPARK-21774: -- I've opened a PR (https://github.com/apache/spark/pu

[jira] [Commented] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131917#comment-16131917 ] Sean Owen commented on SPARK-21770: --- All 0 outputs mean the model says all classes are

[jira] [Updated] (SPARK-21785) Support create table from a file schema

[ https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-21785: - Issue Type: Wish (was: New Feature) > Support create table from a file schema >

[jira] [Comment Edited] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131908#comment-16131908 ] Feng Zhu edited comment on SPARK-21774 at 8/18/17 8:41 AM: --- Thi

[jira] [Comment Edited] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131908#comment-16131908 ] Feng Zhu edited comment on SPARK-21774 at 8/18/17 8:41 AM: --- Thi

[jira] [Comment Edited] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131908#comment-16131908 ] Feng Zhu edited comment on SPARK-21774 at 8/18/17 8:39 AM: --- Thi

[jira] [Created] (SPARK-21785) Support create table from a file schema

liupengcheng created SPARK-21785: Summary: Support create table from a file schema Key: SPARK-21785 URL: https://issues.apache.org/jira/browse/SPARK-21785 Project: Spark Issue Type: New Featu

[jira] [Commented] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131908#comment-16131908 ] Feng Zhu commented on SPARK-21774: -- This is introduced by the PR-15880 (https://github

[jira] [Commented] (SPARK-21784) Add ALTER TABLE ADD CONSTRANT DDL to support defining primary key and foreign keys

[ https://issues.apache.org/jira/browse/SPARK-21784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131900#comment-16131900 ] Suresh Thalamati commented on SPARK-21784: -- I am working on implementation of th

[jira] [Created] (SPARK-21784) Add ALTER TABLE ADD CONSTRANT DDL to support defining primary key and foreign keys

Suresh Thalamati created SPARK-21784: Summary: Add ALTER TABLE ADD CONSTRANT DDL to support defining primary key and foreign keys Key: SPARK-21784 URL: https://issues.apache.org/jira/browse/SPARK-21784

[jira] [Commented] (SPARK-21763) InferSchema option does not infer the correct schema (timestamp) from xlsx file.

[ https://issues.apache.org/jira/browse/SPARK-21763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131890#comment-16131890 ] Liang-Chi Hsieh commented on SPARK-21763: - I'm afraid that the inferSchema is don

[jira] [Commented] (SPARK-19109) ORC metadata section can sometimes exceed protobuf message size limit

[ https://issues.apache.org/jira/browse/SPARK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131877#comment-16131877 ] Dongjoon Hyun commented on SPARK-19109: --- Hi, [~nseggert] and [~wangchao2017]. Could

[jira] [Comment Edited] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131868#comment-16131868 ] Siddharth Murching edited comment on SPARK-21770 at 8/18/17 7:48 AM: --

[jira] [Commented] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131868#comment-16131868 ] Siddharth Murching commented on SPARK-21770: Good question: * Predictions on

[jira] [Commented] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131865#comment-16131865 ] Sergey Serebryakov commented on SPARK-21782: I played with Scala's `hashing.b

[jira] [Updated] (SPARK-21783) Turn on ORC filter push-down by default

[ https://issues.apache.org/jira/browse/SPARK-21783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21783: -- Description: Like Parquet (SPARK-9207), it would be great to turn on ORC option, too. This opti

[jira] [Created] (SPARK-21783) Turn on ORC filter push-down by default

Dongjoon Hyun created SPARK-21783: - Summary: Turn on ORC filter push-down by default Key: SPARK-21783 URL: https://issues.apache.org/jira/browse/SPARK-21783 Project: Spark Issue Type: Improve

[jira] [Commented] (SPARK-20984) Reading back from ORC format gives error on big endian systems.

[ https://issues.apache.org/jira/browse/SPARK-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131837#comment-16131837 ] Dongjoon Hyun commented on SPARK-20984: --- Hi, All. I added this as a blocker of SPAR

[jira] [Commented] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

[ https://issues.apache.org/jira/browse/SPARK-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131828#comment-16131828 ] Kent Yao commented on SPARK-21771: -- https://github.com/apache/spark/pull/18983 git webho

[jira] [Comment Edited] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

[ https://issues.apache.org/jira/browse/SPARK-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131828#comment-16131828 ] Kent Yao edited comment on SPARK-21771 at 8/18/17 7:05 AM: --- htt

[jira] [Commented] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131818#comment-16131818 ] Sean Owen commented on SPARK-21782: --- I think scrambling the seed a bit is the smallest