[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2017-08-17 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131796#comment-16131796 ] cen yuhai commented on SPARK-16188: --- [~xianlongZhang] yes, you are right, I has impleme

[jira] [Commented] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

2017-08-17 Thread Sergey Serebryakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131794#comment-16131794 ] Sergey Serebryakov commented on SPARK-21782: Your understanding is correct. E

[jira] [Updated] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

2017-08-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21771: -- Issue Type: Improvement (was: Bug) > SparkSQLEnv creates a useless meta hive client >

[jira] [Commented] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

2017-08-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131784#comment-16131784 ] Sean Owen commented on SPARK-21782: --- Is the problem summary just: with a power of 2 bou

[jira] [Commented] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

2017-08-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131775#comment-16131775 ] Sean Owen commented on SPARK-21770: --- What's the current behavior for the prediction? I

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

2017-08-17 Thread xianlongZhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131772#comment-16131772 ] xianlongZhang commented on SPARK-16188: --- cen yuhai,thanks for your advice, but my

[jira] [Resolved] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21776. --- Resolution: Invalid > How to use the memory-mapped file on Spark?? >

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21776: -- Priority: Trivial (was: Blocker) > How to use the memory-mapped file on Spark?? >

[jira] [Updated] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

2017-08-17 Thread Sergey Serebryakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Serebryakov updated SPARK-21782: --- Attachment: Screen Shot 2017-08-16 at 3.40.01 PM.png Distribution of partition sizes

[jira] [Created] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

2017-08-17 Thread Sergey Serebryakov (JIRA)
Sergey Serebryakov created SPARK-21782: -- Summary: Repartition creates skews when numPartitions is a power of 2 Key: SPARK-21782 URL: https://issues.apache.org/jira/browse/SPARK-21782 Project: Spa

[jira] [Resolved] (SPARK-21739) timestamp partition would fail in v2.2.0

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21739. - Resolution: Fixed Assignee: Feng Zhu Fix Version/s: 2.3.0 2.2.1 > time

[jira] [Comment Edited] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131749#comment-16131749 ] zhaP524 edited comment on SPARK-21776 at 8/18/17 5:36 AM: -- [~kis

[jira] [Commented] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131749#comment-16131749 ] zhaP524 commented on SPARK-21776: - @Kazuaki Ishizaki I see , I have changed the type o

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaP524 updated SPARK-21776: Priority: Blocker (was: Major) Issue Type: Improvement (was: Bug) > How to use the memory-mapped fi

[jira] [Created] (SPARK-21781) Modify DataSourceScanExec to use concrete ColumnVector type.

2017-08-17 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-21781: - Summary: Modify DataSourceScanExec to use concrete ColumnVector type. Key: SPARK-21781 URL: https://issues.apache.org/jira/browse/SPARK-21781 Project: Spark

[jira] [Updated] (SPARK-21778) Simpler Dataset.sample API in Scala / Java

2017-08-17 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-21778: Summary: Simpler Dataset.sample API in Scala / Java (was: Simpler Dataset.sample API in Scala) >

[jira] [Created] (SPARK-21779) Simpler Dataset.sample API in Python

2017-08-17 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-21779: --- Summary: Simpler Dataset.sample API in Python Key: SPARK-21779 URL: https://issues.apache.org/jira/browse/SPARK-21779 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-21780) Simpler Dataset.sample API in R

2017-08-17 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-21780: --- Summary: Simpler Dataset.sample API in R Key: SPARK-21780 URL: https://issues.apache.org/jira/browse/SPARK-21780 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-21778) Simpler Dataset.sample API in Scala

2017-08-17 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-21778: --- Summary: Simpler Dataset.sample API in Scala Key: SPARK-21778 URL: https://issues.apache.org/jira/browse/SPARK-21778 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-21777) Simpler Dataset.sample API

2017-08-17 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-21777: --- Summary: Simpler Dataset.sample API Key: SPARK-21777 URL: https://issues.apache.org/jira/browse/SPARK-21777 Project: Spark Issue Type: New Feature Co

[jira] [Updated] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

2017-08-17 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-21774: - External issue URL: (was: https://github.com/apache/spark/pull/18986) > The rule PromoteStrings cast st

[jira] [Updated] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

2017-08-17 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-21774: - External issue URL: https://github.com/apache/spark/pull/18986 > The rule PromoteStrings cast string to a

[jira] [Updated] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

2017-08-17 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-21774: - External issue ID: (was: SPARK-21646) > The rule PromoteStrings cast string to a wrong data type >

[jira] [Updated] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

2017-08-17 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-21774: - Description: Data {code} create temporary view tb as select * from values ("0", 1), ("-0.1", 2), ("1", 3)

[jira] [Commented] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131681#comment-16131681 ] Kazuaki Ishizaki commented on SPARK-21776: -- Is this a question? It this is a kin

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaP524 updated SPARK-21776: Component/s: Spark Core Issue Type: Bug (was: Question) > How to use the memory-mapped file on Spark?

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaP524 updated SPARK-21776: Description: In generation, we have to use the Spark full quantity loaded HBase table based on one d

[jira] [Commented] (SPARK-21776) How to use the memory-mapped file on Spark???

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131672#comment-16131672 ] zhaP524 commented on SPARK-21776: - !screenshot-2.png! > How to use the memory-mapped fil

[jira] [Commented] (SPARK-21776) How to use the memory-mapped file on Spark???

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131671#comment-16131671 ] zhaP524 commented on SPARK-21776: - !screenshot-1.png! > How to use the memory-mapped fil

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark???

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaP524 updated SPARK-21776: Attachment: screenshot-1.png > How to use the memory-mapped file on Spark??? >

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark???

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaP524 updated SPARK-21776: Attachment: screenshot-2.png > How to use the memory-mapped file on Spark??? >

[jira] [Created] (SPARK-21776) How to use the memory-mapped file on Spark???

2017-08-17 Thread zhaP524 (JIRA)
zhaP524 created SPARK-21776: --- Summary: How to use the memory-mapped file on Spark??? Key: SPARK-21776 URL: https://issues.apache.org/jira/browse/SPARK-21776 Project: Spark Issue Type: Question

[jira] [Commented] (SPARK-5073) "spark.storage.memoryMapThreshold" has two default values

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131667#comment-16131667 ] zhaP524 commented on SPARK-5073: I wonder what this parameter is for?Also want to know if

[jira] [Updated] (SPARK-21775) Dynamic Log Level Settings for executors

2017-08-17 Thread LvDongrong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LvDongrong updated SPARK-21775: --- Attachment: web.PNG terminal.PNG I changed the loglevel of driver to debug and take

[jira] [Created] (SPARK-21775) Dynamic Log Level Settings for executors

2017-08-17 Thread LvDongrong (JIRA)
LvDongrong created SPARK-21775: -- Summary: Dynamic Log Level Settings for executors Key: SPARK-21775 URL: https://issues.apache.org/jira/browse/SPARK-21775 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

2017-08-17 Thread StanZhai (JIRA)
StanZhai created SPARK-21774: Summary: The rule PromoteStrings cast string to a wrong data type Key: SPARK-21774 URL: https://issues.apache.org/jira/browse/SPARK-21774 Project: Spark Issue Type:

[jira] [Created] (SPARK-21773) Should Install mkdocs if missing in the path in SQL documentation build

2017-08-17 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-21773: Summary: Should Install mkdocs if missing in the path in SQL documentation build Key: SPARK-21773 URL: https://issues.apache.org/jira/browse/SPARK-21773 Project: Spar

[jira] [Created] (SPARK-21772) HiveException unable to move results from srcf to destf in InsertIntoHiveTable

2017-08-17 Thread liupengcheng (JIRA)
liupengcheng created SPARK-21772: Summary: HiveException unable to move results from srcf to destf in InsertIntoHiveTable Key: SPARK-21772 URL: https://issues.apache.org/jira/browse/SPARK-21772 Proje

[jira] [Created] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

2017-08-17 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21771: Summary: SparkSQLEnv creates a useless meta hive client Key: SPARK-21771 URL: https://issues.apache.org/jira/browse/SPARK-21771 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

2017-08-17 Thread Siddharth Murching (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Murching updated SPARK-21770: --- Description: Given an n-element raw prediction vector of all-zeros, ProbabilisticClas

[jira] [Created] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

2017-08-17 Thread Siddharth Murching (JIRA)
Siddharth Murching created SPARK-21770: -- Summary: ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions Key: SPARK-21770 URL: https://issues.apache.org/jira/browse/SPARK-21770

[jira] [Comment Edited] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131537#comment-16131537 ] George Pongracz edited comment on SPARK-21702 at 8/18/17 12:55 AM:

[jira] [Comment Edited] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131537#comment-16131537 ] George Pongracz edited comment on SPARK-21702 at 8/18/17 12:55 AM:

[jira] [Comment Edited] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131537#comment-16131537 ] George Pongracz edited comment on SPARK-21702 at 8/18/17 12:54 AM:

[jira] [Comment Edited] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131537#comment-16131537 ] George Pongracz edited comment on SPARK-21702 at 8/18/17 12:47 AM:

[jira] [Updated] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Pongracz updated SPARK-21702: Summary: Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when Partit

[jira] [Comment Edited] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Applied when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131537#comment-16131537 ] George Pongracz edited comment on SPARK-21702 at 8/18/17 12:43 AM:

[jira] [Commented] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Applied when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131537#comment-16131537 ] George Pongracz commented on SPARK-21702: - *Update:* The data bearing files (fil

[jira] [Commented] (SPARK-21685) Params isSet in scala Transformer triggered by _setDefault in pyspark

2017-08-17 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131512#comment-16131512 ] Bryan Cutler commented on SPARK-21685: -- I believe the problem is during the call to

[jira] [Commented] (SPARK-21759) In.checkInputDataTypes should not wrongly report unresolved plans for IN correlated subquery

2017-08-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131493#comment-16131493 ] Liang-Chi Hsieh commented on SPARK-21759: - Submitted PR at https://github.com/apa

[jira] [Updated] (SPARK-21759) In.checkInputDataTypes should not wrongly report unresolved plans for IN correlated subquery

2017-08-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-21759: Description: With the check for structural integrity proposed in SPARK-21726, I found that

[jira] [Updated] (SPARK-21759) In.checkInputDataTypes should not wrongly report unresolved plans for IN correlated subquery

2017-08-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-21759: Summary: In.checkInputDataTypes should not wrongly report unresolved plans for IN correlate

[jira] [Resolved] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21677. - Resolution: Fixed Fix Version/s: 2.3.0 > json_tuple throws NullPointException when column is null

[jira] [Assigned] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-21677: --- Assignee: Jen-Ming Chung > json_tuple throws NullPointException when column is null as string type.

[jira] [Resolved] (SPARK-21767) Add Decimal Test For Avro in VersionSuite

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21767. - Resolution: Fixed Fix Version/s: 2.3.0 > Add Decimal Test For Avro in VersionSuite > -

[jira] [Updated] (SPARK-21769) Add a table property for Hive-serde tables to control Spark always respecting schemas inferred by Spark SQL

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21769: Summary: Add a table property for Hive-serde tables to control Spark always respecting schemas inferred by

[jira] [Updated] (SPARK-21769) Add a table property for Hive-serde tables to controlling Spark always respecting schemas inferred by Spark SQL

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21769: Issue Type: Improvement (was: Bug) > Add a table property for Hive-serde tables to controlling Spark alway

[jira] [Created] (SPARK-21769) Add a table property for Hive-serde tables to controlling Spark always respecting schemas inferred by Spark SQL

2017-08-17 Thread Xiao Li (JIRA)
Xiao Li created SPARK-21769: --- Summary: Add a table property for Hive-serde tables to controlling Spark always respecting schemas inferred by Spark SQL Key: SPARK-21769 URL: https://issues.apache.org/jira/browse/SPARK-21

[jira] [Updated] (SPARK-21769) Add a table property for Hive-serde tables to make Spark always respect schemas inferred by Spark SQL

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21769: Summary: Add a table property for Hive-serde tables to make Spark always respect schemas inferred by Spark

[jira] [Resolved] (SPARK-16742) Kerberos support for Spark on Mesos

2017-08-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-16742. Resolution: Fixed Assignee: Arthur Rand Fix Version/s: 2.3.0 [~arand] I don

[jira] [Commented] (SPARK-19747) Consolidate code in ML aggregators

2017-08-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131340#comment-16131340 ] Joseph K. Bradley commented on SPARK-19747: --- Just saying: Thanks a lot for doin

[jira] [Commented] (SPARK-4131) Support "Writing data into the filesystem from queries"

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131315#comment-16131315 ] Xiao Li commented on SPARK-4131: https://github.com/apache/spark/pull/18975 > Support "Wr

[jira] [Updated] (SPARK-4131) Support "Writing data into the filesystem from queries"

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-4131: --- Target Version/s: 2.3.0 > Support "Writing data into the filesystem from queries" > --

[jira] [Resolved] (SPARK-18394) Executing the same query twice in a row results in CodeGenerator cache misses

2017-08-17 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-18394. --- Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.3.0 > E

[jira] [Created] (SPARK-21768) spark.csv.read Empty String Parsed as NULL when nullValue is Set

2017-08-17 Thread Andrew Gross (JIRA)
Andrew Gross created SPARK-21768: Summary: spark.csv.read Empty String Parsed as NULL when nullValue is Set Key: SPARK-21768 URL: https://issues.apache.org/jira/browse/SPARK-21768 Project: Spark

[jira] [Commented] (SPARK-21762) FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file isn't yet visible

2017-08-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131190#comment-16131190 ] Steve Loughran commented on SPARK-21762: SPARK-20703 simplifies this, especially

[jira] [Comment Edited] (SPARK-21762) FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file isn't yet visible

2017-08-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131190#comment-16131190 ] Steve Loughran edited comment on SPARK-21762 at 8/17/17 7:41 PM: --

[jira] [Updated] (SPARK-21762) FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file isn't yet visible

2017-08-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-21762: --- Summary: FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file i

[jira] [Commented] (SPARK-21767) Add Decimal Test For Avro in VersionSuite

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131028#comment-16131028 ] Xiao Li commented on SPARK-21767: - https://github.com/apache/spark/pull/18977 > Add Dec

[jira] [Updated] (SPARK-21767) Add Decimal Test For Avro in VersionSuite

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21767: Description: Decimal is a logical type of AVRO. We need to ensure the support of Hive's AVRO serde works we

[jira] [Created] (SPARK-21767) Add Decimal Test For Avro in VersionSuite

2017-08-17 Thread Xiao Li (JIRA)
Xiao Li created SPARK-21767: --- Summary: Add Decimal Test For Avro in VersionSuite Key: SPARK-21767 URL: https://issues.apache.org/jira/browse/SPARK-21767 Project: Spark Issue Type: Bug Com

[jira] [Updated] (SPARK-21766) DataFrame toPandas() raises ValueError with nullable int columns

2017-08-17 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-21766: - Summary: DataFrame toPandas() raises ValueError with nullable int columns (was: DataFrame toPand

[jira] [Created] (SPARK-21766) DataFrame toPandas()

2017-08-17 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-21766: Summary: DataFrame toPandas() Key: SPARK-21766 URL: https://issues.apache.org/jira/browse/SPARK-21766 Project: Spark Issue Type: Bug Components: P

[jira] [Comment Edited] (SPARK-15689) Data source API v2

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130863#comment-16130863 ] Wenchen Fan edited comment on SPARK-15689 at 8/17/17 5:25 PM: -

[jira] [Commented] (SPARK-15689) Data source API v2

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130863#comment-16130863 ] Wenchen Fan commented on SPARK-15689: - good doc attached! > Data source API v2 > ---

[jira] [Commented] (SPARK-17414) Set type is not supported for creating data frames

2017-08-17 Thread Alexander Bessonov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130852#comment-16130852 ] Alexander Bessonov commented on SPARK-17414: Fixed in SPARK-21204 > Set type

[jira] [Created] (SPARK-21765) Mark all streaming plans as isStreaming

2017-08-17 Thread Jose Torres (JIRA)
Jose Torres created SPARK-21765: --- Summary: Mark all streaming plans as isStreaming Key: SPARK-21765 URL: https://issues.apache.org/jira/browse/SPARK-21765 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-21764) Tests failures on Windows: resources not being closed and incorrect paths

2017-08-17 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-21764: Summary: Tests failures on Windows: resources not being closed and incorrect paths Key: SPARK-21764 URL: https://issues.apache.org/jira/browse/SPARK-21764 Project: Sp

[jira] [Created] (SPARK-21763) InferSchema option does not infer the correct schema (timestamp) from xlsx file.

2017-08-17 Thread ANSHUMAN (JIRA)
ANSHUMAN created SPARK-21763: Summary: InferSchema option does not infer the correct schema (timestamp) from xlsx file. Key: SPARK-21763 URL: https://issues.apache.org/jira/browse/SPARK-21763 Project: Spa

[jira] [Assigned] (SPARK-21428) CliSessionState never be recognized because of IsolatedClientLoader

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21428: --- Assignee: Kent Yao > CliSessionState never be recognized because of IsolatedClientLoader > -

[jira] [Resolved] (SPARK-21428) CliSessionState never be recognized because of IsolatedClientLoader

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21428. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18648 [https://githu

[jira] [Created] (SPARK-21762) FileFormatWriter metrics collection fails if a newly close()d file isn't yet visible

2017-08-17 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-21762: -- Summary: FileFormatWriter metrics collection fails if a newly close()d file isn't yet visible Key: SPARK-21762 URL: https://issues.apache.org/jira/browse/SPARK-21762

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130525#comment-16130525 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/17/17 3:15 PM: -

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130525#comment-16130525 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/17/17 3:12 PM: -

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130525#comment-16130525 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/17/17 3:11 PM: -

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130525#comment-16130525 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/17/17 3:02 PM: -

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130525#comment-16130525 ] Stavros Kontopoulos commented on SPARK-21752: - [~jsnowacki] I dont think I am

[jira] [Commented] (SPARK-15689) Data source API v2

2017-08-17 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130494#comment-16130494 ] Dongjoon Hyun commented on SPARK-15689: --- Thank you for the document, too! > Data s

[jira] [Commented] (SPARK-21063) Spark return an empty result from remote hadoop cluster

2017-08-17 Thread Varene Olivier (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130472#comment-16130472 ] Varene Olivier commented on SPARK-21063: Hi, I am experiencing the same issue wit

[jira] [Comment Edited] (SPARK-21063) Spark return an empty result from remote hadoop cluster

2017-08-17 Thread Varene Olivier (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130472#comment-16130472 ] Varene Olivier edited comment on SPARK-21063 at 8/17/17 2:25 PM: --

[jira] [Assigned] (SPARK-21642) Use FQDN for DRIVER_HOST_ADDRESS instead of ip address

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21642: --- Assignee: Aki Tanaka (was: Hideaki Tanaka) > Use FQDN for DRIVER_HOST_ADDRESS instead of ip

[jira] [Assigned] (SPARK-21642) Use FQDN for DRIVER_HOST_ADDRESS instead of ip address

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21642: --- Assignee: Hideaki Tanaka > Use FQDN for DRIVER_HOST_ADDRESS instead of ip address >

[jira] [Resolved] (SPARK-21642) Use FQDN for DRIVER_HOST_ADDRESS instead of ip address

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21642. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18846 [https://githu

[jira] [Commented] (SPARK-21743) top-most limit should not cause memory leak

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130387#comment-16130387 ] Wenchen Fan commented on SPARK-21743: - issue resolved by https://github.com/apache/sp

[jira] [Commented] (SPARK-15689) Data source API v2

2017-08-17 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130355#comment-16130355 ] Russell Spitzer commented on SPARK-15689: - Thanks [~cloud_fan] for posting the de

[jira] [Updated] (SPARK-15689) Data source API v2

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-15689: Attachment: SPIP Data Source API V2.pdf > Data source API v2 > -- > >

[jira] [Created] (SPARK-21761) [Core] Add the application's final state for SparkListenerApplicationEnd event

2017-08-17 Thread lishuming (JIRA)
lishuming created SPARK-21761: - Summary: [Core] Add the application's final state for SparkListenerApplicationEnd event Key: SPARK-21761 URL: https://issues.apache.org/jira/browse/SPARK-21761 Project: Spa

[jira] [Commented] (SPARK-21758) `SHOW TBLPROPERTIES` can not get properties start with spark.sql.*

2017-08-17 Thread Feng Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130288#comment-16130288 ] Feng Zhu commented on SPARK-21758: -- I can't reproduce this issue in 2.1 and master branc

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Jakub Nowacki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130282#comment-16130282 ] Jakub Nowacki commented on SPARK-21752: --- [~skonto] Well, I'm not sure where you're

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Jakub Nowacki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130274#comment-16130274 ] Jakub Nowacki commented on SPARK-21752: --- OK I get the point. I think we should only

  1   2   >