[jira] [Updated] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

2017-08-17 Thread Sergey Serebryakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Serebryakov updated SPARK-21782: --- Attachment: Screen Shot 2017-08-16 at 3.40.01 PM.png Distribution of partition sizes

[jira] [Created] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

2017-08-17 Thread Sergey Serebryakov (JIRA)
Sergey Serebryakov created SPARK-21782: -- Summary: Repartition creates skews when numPartitions is a power of 2 Key: SPARK-21782 URL: https://issues.apache.org/jira/browse/SPARK-21782 Project:

[jira] [Resolved] (SPARK-21739) timestamp partition would fail in v2.2.0

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21739. - Resolution: Fixed Assignee: Feng Zhu Fix Version/s: 2.3.0 2.2.1 >

[jira] [Comment Edited] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131749#comment-16131749 ] zhaP524 edited comment on SPARK-21776 at 8/18/17 5:36 AM: -- [~kiszk] I see , I

[jira] [Commented] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131749#comment-16131749 ] zhaP524 commented on SPARK-21776: - @Kazuaki Ishizaki I see , I have changed the type of question,This

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaP524 updated SPARK-21776: Priority: Blocker (was: Major) Issue Type: Improvement (was: Bug) > How to use the memory-mapped

[jira] [Created] (SPARK-21781) Modify DataSourceScanExec to use concrete ColumnVector type.

2017-08-17 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-21781: - Summary: Modify DataSourceScanExec to use concrete ColumnVector type. Key: SPARK-21781 URL: https://issues.apache.org/jira/browse/SPARK-21781 Project: Spark

[jira] [Updated] (SPARK-21778) Simpler Dataset.sample API in Scala / Java

2017-08-17 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-21778: Summary: Simpler Dataset.sample API in Scala / Java (was: Simpler Dataset.sample API in Scala) >

[jira] [Created] (SPARK-21779) Simpler Dataset.sample API in Python

2017-08-17 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-21779: --- Summary: Simpler Dataset.sample API in Python Key: SPARK-21779 URL: https://issues.apache.org/jira/browse/SPARK-21779 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-21780) Simpler Dataset.sample API in R

2017-08-17 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-21780: --- Summary: Simpler Dataset.sample API in R Key: SPARK-21780 URL: https://issues.apache.org/jira/browse/SPARK-21780 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-21778) Simpler Dataset.sample API in Scala

2017-08-17 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-21778: --- Summary: Simpler Dataset.sample API in Scala Key: SPARK-21778 URL: https://issues.apache.org/jira/browse/SPARK-21778 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-21777) Simpler Dataset.sample API

2017-08-17 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-21777: --- Summary: Simpler Dataset.sample API Key: SPARK-21777 URL: https://issues.apache.org/jira/browse/SPARK-21777 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

2017-08-17 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-21774: - External issue URL: (was: https://github.com/apache/spark/pull/18986) > The rule PromoteStrings cast

[jira] [Updated] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

2017-08-17 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-21774: - External issue URL: https://github.com/apache/spark/pull/18986 > The rule PromoteStrings cast string to

[jira] [Updated] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

2017-08-17 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-21774: - External issue ID: (was: SPARK-21646) > The rule PromoteStrings cast string to a wrong data type >

[jira] [Updated] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

2017-08-17 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-21774: - Description: Data {code} create temporary view tb as select * from values ("0", 1), ("-0.1", 2), ("1",

[jira] [Commented] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131681#comment-16131681 ] Kazuaki Ishizaki commented on SPARK-21776: -- Is this a question? It this is a kind of questions,

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaP524 updated SPARK-21776: Component/s: Spark Core Issue Type: Bug (was: Question) > How to use the memory-mapped file on

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark??

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaP524 updated SPARK-21776: Description: In generation, we have to use the Spark full quantity loaded HBase table based on one

[jira] [Commented] (SPARK-21776) How to use the memory-mapped file on Spark???

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131672#comment-16131672 ] zhaP524 commented on SPARK-21776: - !screenshot-2.png! > How to use the memory-mapped file on Spark??? >

[jira] [Commented] (SPARK-21776) How to use the memory-mapped file on Spark???

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131671#comment-16131671 ] zhaP524 commented on SPARK-21776: - !screenshot-1.png! > How to use the memory-mapped file on Spark??? >

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark???

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaP524 updated SPARK-21776: Attachment: screenshot-1.png > How to use the memory-mapped file on Spark??? >

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark???

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaP524 updated SPARK-21776: Attachment: screenshot-2.png > How to use the memory-mapped file on Spark??? >

[jira] [Created] (SPARK-21776) How to use the memory-mapped file on Spark???

2017-08-17 Thread zhaP524 (JIRA)
zhaP524 created SPARK-21776: --- Summary: How to use the memory-mapped file on Spark??? Key: SPARK-21776 URL: https://issues.apache.org/jira/browse/SPARK-21776 Project: Spark Issue Type: Question

[jira] [Commented] (SPARK-5073) "spark.storage.memoryMapThreshold" has two default values

2017-08-17 Thread zhaP524 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131667#comment-16131667 ] zhaP524 commented on SPARK-5073: I wonder what this parameter is for?Also want to know if this parameter

[jira] [Updated] (SPARK-21775) Dynamic Log Level Settings for executors

2017-08-17 Thread LvDongrong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LvDongrong updated SPARK-21775: --- Attachment: web.PNG terminal.PNG I changed the loglevel of driver to debug and take

[jira] [Created] (SPARK-21775) Dynamic Log Level Settings for executors

2017-08-17 Thread LvDongrong (JIRA)
LvDongrong created SPARK-21775: -- Summary: Dynamic Log Level Settings for executors Key: SPARK-21775 URL: https://issues.apache.org/jira/browse/SPARK-21775 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

2017-08-17 Thread StanZhai (JIRA)
StanZhai created SPARK-21774: Summary: The rule PromoteStrings cast string to a wrong data type Key: SPARK-21774 URL: https://issues.apache.org/jira/browse/SPARK-21774 Project: Spark Issue Type:

[jira] [Created] (SPARK-21773) Should Install mkdocs if missing in the path in SQL documentation build

2017-08-17 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-21773: Summary: Should Install mkdocs if missing in the path in SQL documentation build Key: SPARK-21773 URL: https://issues.apache.org/jira/browse/SPARK-21773 Project:

[jira] [Created] (SPARK-21772) HiveException unable to move results from srcf to destf in InsertIntoHiveTable

2017-08-17 Thread liupengcheng (JIRA)
liupengcheng created SPARK-21772: Summary: HiveException unable to move results from srcf to destf in InsertIntoHiveTable Key: SPARK-21772 URL: https://issues.apache.org/jira/browse/SPARK-21772

[jira] [Created] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

2017-08-17 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21771: Summary: SparkSQLEnv creates a useless meta hive client Key: SPARK-21771 URL: https://issues.apache.org/jira/browse/SPARK-21771 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

2017-08-17 Thread Siddharth Murching (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Murching updated SPARK-21770: --- Description: Given an n-element raw prediction vector of all-zeros,

[jira] [Created] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

2017-08-17 Thread Siddharth Murching (JIRA)
Siddharth Murching created SPARK-21770: -- Summary: ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions Key: SPARK-21770 URL: https://issues.apache.org/jira/browse/SPARK-21770

[jira] [Comment Edited] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131537#comment-16131537 ] George Pongracz edited comment on SPARK-21702 at 8/18/17 12:55 AM: ---

[jira] [Comment Edited] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131537#comment-16131537 ] George Pongracz edited comment on SPARK-21702 at 8/18/17 12:55 AM: ---

[jira] [Comment Edited] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131537#comment-16131537 ] George Pongracz edited comment on SPARK-21702 at 8/18/17 12:54 AM: ---

[jira] [Comment Edited] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131537#comment-16131537 ] George Pongracz edited comment on SPARK-21702 at 8/18/17 12:47 AM: ---

[jira] [Updated] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Pongracz updated SPARK-21702: Summary: Structured Streaming S3A SSE Encryption Not Visible through AWS S3 GUI when

[jira] [Comment Edited] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Applied when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131537#comment-16131537 ] George Pongracz edited comment on SPARK-21702 at 8/18/17 12:43 AM: ---

[jira] [Commented] (SPARK-21702) Structured Streaming S3A SSE Encryption Not Applied when PartitionBy Used

2017-08-17 Thread George Pongracz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131537#comment-16131537 ] George Pongracz commented on SPARK-21702: - *Update:* The data bearing files (files that contain

[jira] [Commented] (SPARK-21685) Params isSet in scala Transformer triggered by _setDefault in pyspark

2017-08-17 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131512#comment-16131512 ] Bryan Cutler commented on SPARK-21685: -- I believe the problem is during the call to transform, the

[jira] [Commented] (SPARK-21759) In.checkInputDataTypes should not wrongly report unresolved plans for IN correlated subquery

2017-08-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131493#comment-16131493 ] Liang-Chi Hsieh commented on SPARK-21759: - Submitted PR at

[jira] [Updated] (SPARK-21759) In.checkInputDataTypes should not wrongly report unresolved plans for IN correlated subquery

2017-08-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-21759: Description: With the check for structural integrity proposed in SPARK-21726, I found that

[jira] [Updated] (SPARK-21759) In.checkInputDataTypes should not wrongly report unresolved plans for IN correlated subquery

2017-08-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-21759: Summary: In.checkInputDataTypes should not wrongly report unresolved plans for IN

[jira] [Resolved] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21677. - Resolution: Fixed Fix Version/s: 2.3.0 > json_tuple throws NullPointException when column is null

[jira] [Assigned] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-21677: --- Assignee: Jen-Ming Chung > json_tuple throws NullPointException when column is null as string type.

[jira] [Resolved] (SPARK-21767) Add Decimal Test For Avro in VersionSuite

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21767. - Resolution: Fixed Fix Version/s: 2.3.0 > Add Decimal Test For Avro in VersionSuite >

[jira] [Updated] (SPARK-21769) Add a table property for Hive-serde tables to control Spark always respecting schemas inferred by Spark SQL

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21769: Summary: Add a table property for Hive-serde tables to control Spark always respecting schemas inferred by

[jira] [Updated] (SPARK-21769) Add a table property for Hive-serde tables to controlling Spark always respecting schemas inferred by Spark SQL

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21769: Issue Type: Improvement (was: Bug) > Add a table property for Hive-serde tables to controlling Spark

[jira] [Created] (SPARK-21769) Add a table property for Hive-serde tables to controlling Spark always respecting schemas inferred by Spark SQL

2017-08-17 Thread Xiao Li (JIRA)
Xiao Li created SPARK-21769: --- Summary: Add a table property for Hive-serde tables to controlling Spark always respecting schemas inferred by Spark SQL Key: SPARK-21769 URL:

[jira] [Updated] (SPARK-21769) Add a table property for Hive-serde tables to make Spark always respect schemas inferred by Spark SQL

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21769: Summary: Add a table property for Hive-serde tables to make Spark always respect schemas inferred by Spark

[jira] [Resolved] (SPARK-16742) Kerberos support for Spark on Mesos

2017-08-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-16742. Resolution: Fixed Assignee: Arthur Rand Fix Version/s: 2.3.0 [~arand] I

[jira] [Commented] (SPARK-19747) Consolidate code in ML aggregators

2017-08-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131340#comment-16131340 ] Joseph K. Bradley commented on SPARK-19747: --- Just saying: Thanks a lot for doing this reorg!

[jira] [Commented] (SPARK-4131) Support "Writing data into the filesystem from queries"

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131315#comment-16131315 ] Xiao Li commented on SPARK-4131: https://github.com/apache/spark/pull/18975 > Support "Writing data into

[jira] [Updated] (SPARK-4131) Support "Writing data into the filesystem from queries"

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-4131: --- Target Version/s: 2.3.0 > Support "Writing data into the filesystem from queries" >

[jira] [Resolved] (SPARK-18394) Executing the same query twice in a row results in CodeGenerator cache misses

2017-08-17 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-18394. --- Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.3.0 >

[jira] [Created] (SPARK-21768) spark.csv.read Empty String Parsed as NULL when nullValue is Set

2017-08-17 Thread Andrew Gross (JIRA)
Andrew Gross created SPARK-21768: Summary: spark.csv.read Empty String Parsed as NULL when nullValue is Set Key: SPARK-21768 URL: https://issues.apache.org/jira/browse/SPARK-21768 Project: Spark

[jira] [Commented] (SPARK-21762) FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file isn't yet visible

2017-08-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131190#comment-16131190 ] Steve Loughran commented on SPARK-21762: SPARK-20703 simplifies this, especially testing, as it's

[jira] [Comment Edited] (SPARK-21762) FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file isn't yet visible

2017-08-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131190#comment-16131190 ] Steve Loughran edited comment on SPARK-21762 at 8/17/17 7:41 PM: -

[jira] [Updated] (SPARK-21762) FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file isn't yet visible

2017-08-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-21762: --- Summary: FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file

[jira] [Commented] (SPARK-21767) Add Decimal Test For Avro in VersionSuite

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131028#comment-16131028 ] Xiao Li commented on SPARK-21767: - https://github.com/apache/spark/pull/18977 > Add Decimal Test For

[jira] [Updated] (SPARK-21767) Add Decimal Test For Avro in VersionSuite

2017-08-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21767: Description: Decimal is a logical type of AVRO. We need to ensure the support of Hive's AVRO serde works

[jira] [Created] (SPARK-21767) Add Decimal Test For Avro in VersionSuite

2017-08-17 Thread Xiao Li (JIRA)
Xiao Li created SPARK-21767: --- Summary: Add Decimal Test For Avro in VersionSuite Key: SPARK-21767 URL: https://issues.apache.org/jira/browse/SPARK-21767 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-21766) DataFrame toPandas() raises ValueError with nullable int columns

2017-08-17 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-21766: - Summary: DataFrame toPandas() raises ValueError with nullable int columns (was: DataFrame

[jira] [Created] (SPARK-21766) DataFrame toPandas()

2017-08-17 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-21766: Summary: DataFrame toPandas() Key: SPARK-21766 URL: https://issues.apache.org/jira/browse/SPARK-21766 Project: Spark Issue Type: Bug Components:

[jira] [Comment Edited] (SPARK-15689) Data source API v2

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130863#comment-16130863 ] Wenchen Fan edited comment on SPARK-15689 at 8/17/17 5:25 PM: -- google doc

[jira] [Commented] (SPARK-15689) Data source API v2

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130863#comment-16130863 ] Wenchen Fan commented on SPARK-15689: - good doc attached! > Data source API v2 > --

[jira] [Commented] (SPARK-17414) Set type is not supported for creating data frames

2017-08-17 Thread Alexander Bessonov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130852#comment-16130852 ] Alexander Bessonov commented on SPARK-17414: Fixed in SPARK-21204 > Set type is not

[jira] [Created] (SPARK-21765) Mark all streaming plans as isStreaming

2017-08-17 Thread Jose Torres (JIRA)
Jose Torres created SPARK-21765: --- Summary: Mark all streaming plans as isStreaming Key: SPARK-21765 URL: https://issues.apache.org/jira/browse/SPARK-21765 Project: Spark Issue Type:

[jira] [Created] (SPARK-21764) Tests failures on Windows: resources not being closed and incorrect paths

2017-08-17 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-21764: Summary: Tests failures on Windows: resources not being closed and incorrect paths Key: SPARK-21764 URL: https://issues.apache.org/jira/browse/SPARK-21764 Project:

[jira] [Created] (SPARK-21763) InferSchema option does not infer the correct schema (timestamp) from xlsx file.

2017-08-17 Thread ANSHUMAN (JIRA)
ANSHUMAN created SPARK-21763: Summary: InferSchema option does not infer the correct schema (timestamp) from xlsx file. Key: SPARK-21763 URL: https://issues.apache.org/jira/browse/SPARK-21763 Project:

[jira] [Assigned] (SPARK-21428) CliSessionState never be recognized because of IsolatedClientLoader

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21428: --- Assignee: Kent Yao > CliSessionState never be recognized because of IsolatedClientLoader >

[jira] [Resolved] (SPARK-21428) CliSessionState never be recognized because of IsolatedClientLoader

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21428. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18648

[jira] [Created] (SPARK-21762) FileFormatWriter metrics collection fails if a newly close()d file isn't yet visible

2017-08-17 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-21762: -- Summary: FileFormatWriter metrics collection fails if a newly close()d file isn't yet visible Key: SPARK-21762 URL: https://issues.apache.org/jira/browse/SPARK-21762

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130525#comment-16130525 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/17/17 3:15 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130525#comment-16130525 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/17/17 3:12 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130525#comment-16130525 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/17/17 3:11 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130525#comment-16130525 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/17/17 3:02 PM:

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130525#comment-16130525 ] Stavros Kontopoulos commented on SPARK-21752: - [~jsnowacki] I dont think I am doing anything

[jira] [Commented] (SPARK-15689) Data source API v2

2017-08-17 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130494#comment-16130494 ] Dongjoon Hyun commented on SPARK-15689: --- Thank you for the document, too! > Data source API v2 >

[jira] [Commented] (SPARK-21063) Spark return an empty result from remote hadoop cluster

2017-08-17 Thread Varene Olivier (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130472#comment-16130472 ] Varene Olivier commented on SPARK-21063: Hi, I am experiencing the same issue with Spark 2.2.0

[jira] [Comment Edited] (SPARK-21063) Spark return an empty result from remote hadoop cluster

2017-08-17 Thread Varene Olivier (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130472#comment-16130472 ] Varene Olivier edited comment on SPARK-21063 at 8/17/17 2:25 PM: - Hi, I

[jira] [Assigned] (SPARK-21642) Use FQDN for DRIVER_HOST_ADDRESS instead of ip address

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21642: --- Assignee: Aki Tanaka (was: Hideaki Tanaka) > Use FQDN for DRIVER_HOST_ADDRESS instead of

[jira] [Assigned] (SPARK-21642) Use FQDN for DRIVER_HOST_ADDRESS instead of ip address

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21642: --- Assignee: Hideaki Tanaka > Use FQDN for DRIVER_HOST_ADDRESS instead of ip address >

[jira] [Resolved] (SPARK-21642) Use FQDN for DRIVER_HOST_ADDRESS instead of ip address

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21642. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18846

[jira] [Commented] (SPARK-21743) top-most limit should not cause memory leak

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130387#comment-16130387 ] Wenchen Fan commented on SPARK-21743: - issue resolved by https://github.com/apache/spark/pull/18955

[jira] [Commented] (SPARK-15689) Data source API v2

2017-08-17 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130355#comment-16130355 ] Russell Spitzer commented on SPARK-15689: - Thanks [~cloud_fan] for posting the design doc it was

[jira] [Updated] (SPARK-15689) Data source API v2

2017-08-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-15689: Attachment: SPIP Data Source API V2.pdf > Data source API v2 > -- > >

[jira] [Created] (SPARK-21761) [Core] Add the application's final state for SparkListenerApplicationEnd event

2017-08-17 Thread lishuming (JIRA)
lishuming created SPARK-21761: - Summary: [Core] Add the application's final state for SparkListenerApplicationEnd event Key: SPARK-21761 URL: https://issues.apache.org/jira/browse/SPARK-21761 Project:

[jira] [Commented] (SPARK-21758) `SHOW TBLPROPERTIES` can not get properties start with spark.sql.*

2017-08-17 Thread Feng Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130288#comment-16130288 ] Feng Zhu commented on SPARK-21758: -- I can't reproduce this issue in 2.1 and master branch, could you

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Jakub Nowacki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130282#comment-16130282 ] Jakub Nowacki commented on SPARK-21752: --- [~skonto] Well, I'm not sure where you're failing here. If

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Jakub Nowacki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130274#comment-16130274 ] Jakub Nowacki commented on SPARK-21752: --- OK I get the point. I think we should only consider this

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130259#comment-16130259 ] Stavros Kontopoulos commented on SPARK-21752: - [~jsnowacki] Do you have a step by step script

[jira] [Commented] (SPARK-21720) Filter predicate with many conditions throw stackoverflow error

2017-08-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130231#comment-16130231 ] Kazuaki Ishizaki commented on SPARK-21720: -- I identified issues in {{predicates.scala}}. I am

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130161#comment-16130161 ] Sean Owen commented on SPARK-21752: --- I think this is, in any event, not something that's intended to

[jira] [Resolved] (SPARK-21749) Add comments for MessageEncoder to explain the wire format

2017-08-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21749. --- Resolution: Not A Problem > Add comments for MessageEncoder to explain the wire format >

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Jakub Nowacki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130150#comment-16130150 ] Jakub Nowacki commented on SPARK-21752: --- [~skonto] Jupyter is not passing many environmental

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130132#comment-16130132 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/17/17 9:14 AM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130132#comment-16130132 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/17/17 9:14 AM:

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130132#comment-16130132 ] Stavros Kontopoulos commented on SPARK-21752: - [~jerryshao] That is true. I was curious which

  1   2   >