[jira] [Updated] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-28546: --- Description: My code is as follows: {code:java} Dataset dataset =

[jira] [Updated] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-28546: --- Description: My code is as follows: {code:java} Dataset dataset =

[jira] [Updated] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-28546: --- Description: My code is as follows: {code:java} Dataset dataset =

[jira] [Updated] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-28546: --- Description: My code is as follows: {code:java} Dataset dataset =

[jira] [Comment Edited] (SPARK-22063) Upgrade lintr to latest commit sha1 ID

2019-07-27 Thread Manu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16892595#comment-16892595 ] Manu Zhang edited comment on SPARK-22063 at 7/28/19 4:19 AM: -

[jira] [Updated] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-28546: --- Description: My code is as follows: {code:java} Dataset dataset =

[jira] [Updated] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-28546: --- Description: My code is as follows: {code:java} Dataset dataset =

[jira] [Updated] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-28546: --- Description: My code is as follows: {code:java} Dataset dataset =

[jira] [Updated] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-28546: --- Description: My code is as follows: {code:java} Dataset dataset =

[jira] [Updated] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-28546: --- Description: My code is as follows: {code:java} Dataset dataset =

[jira] [Updated] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-28546: --- Description: My code is as follows: {code:java} Dataset dataset =

[jira] [Created] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-27 Thread tommy duan (JIRA)
tommy duan created SPARK-28546: -- Summary: Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation? Key: SPARK-28546 URL:

[jira] [Commented] (SPARK-28476) Support ALTER DATABASE SET LOCATION

2019-07-27 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894612#comment-16894612 ] Weichen Xu commented on SPARK-28476: I am working on this. :) > Support ALTER DATABASE SET LOCATION

[jira] [Resolved] (SPARK-28545) Add the hash map size to the directional log of ObjectAggregationIterator

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28545. --- Resolution: Fixed Fix Version/s: 2.4.4 2.3.4

[jira] [Assigned] (SPARK-28545) Add the hash map size to the directional log of ObjectAggregationIterator

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28545: - Assignee: Dongjoon Hyun > Add the hash map size to the directional log of

[jira] [Resolved] (SPARK-28544) Update zstd-jni to 1.4.2-1

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28544. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25275

[jira] [Assigned] (SPARK-28544) Update zstd-jni to 1.4.2-1

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28544: - Assignee: Dongjoon Hyun > Update zstd-jni to 1.4.2-1 > -- > >

[jira] [Updated] (SPARK-25474) Size in bytes of the query is coming in EB in case of parquet datasource

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25474: -- Affects Version/s: 2.4.3 > Size in bytes of the query is coming in EB in case of parquet

[jira] [Assigned] (SPARK-28460) Port test from HIVE-11835

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28460: - Assignee: Yuming Wang > Port test from HIVE-11835 > - > >

[jira] [Resolved] (SPARK-28460) Port test from HIVE-11835

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28460. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25212

[jira] [Created] (SPARK-28545) Add the hash map size to the directional log of ObjectAggregationIterator

2019-07-27 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-28545: - Summary: Add the hash map size to the directional log of ObjectAggregationIterator Key: SPARK-28545 URL: https://issues.apache.org/jira/browse/SPARK-28545 Project:

[jira] [Created] (SPARK-28544) Update zstd-jni to 1.4.2-1

2019-07-27 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-28544: - Summary: Update zstd-jni to 1.4.2-1 Key: SPARK-28544 URL: https://issues.apache.org/jira/browse/SPARK-28544 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-28424) Support typed interval expression

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28424: - Assignee: Yuming Wang > Support typed interval expression >

[jira] [Resolved] (SPARK-28424) Support typed interval expression

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28424. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25241

[jira] [Assigned] (SPARK-28471) Formatting dates with negative years

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28471: - Assignee: Maxim Gekk > Formatting dates with negative years >

[jira] [Commented] (SPARK-28519) Tests failed on aarch64 due the value of math.log and power function is different

2019-07-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894524#comment-16894524 ] Sean Owen commented on SPARK-28519: --- A summary of the dev@ discussion: This is almost surely because

[jira] [Created] (SPARK-28543) Document Spark Jobs page

2019-07-27 Thread Xiao Li (JIRA)
Xiao Li created SPARK-28543: --- Summary: Document Spark Jobs page Key: SPARK-28543 URL: https://issues.apache.org/jira/browse/SPARK-28543 Project: Spark Issue Type: Sub-task Components:

[jira] [Created] (SPARK-28542) Document Stages page

2019-07-27 Thread Xiao Li (JIRA)
Xiao Li created SPARK-28542: --- Summary: Document Stages page Key: SPARK-28542 URL: https://issues.apache.org/jira/browse/SPARK-28542 Project: Spark Issue Type: Sub-task Components: SQL,

[jira] [Created] (SPARK-28541) Document Storage page

2019-07-27 Thread Xiao Li (JIRA)
Xiao Li created SPARK-28541: --- Summary: Document Storage page Key: SPARK-28541 URL: https://issues.apache.org/jira/browse/SPARK-28541 Project: Spark Issue Type: Sub-task Components: SQL,

[jira] [Created] (SPARK-28540) Document Environment page

2019-07-27 Thread Xiao Li (JIRA)
Xiao Li created SPARK-28540: --- Summary: Document Environment page Key: SPARK-28540 URL: https://issues.apache.org/jira/browse/SPARK-28540 Project: Spark Issue Type: Sub-task Components:

[jira] [Created] (SPARK-28539) Document Executors page

2019-07-27 Thread Xiao Li (JIRA)
Xiao Li created SPARK-28539: --- Summary: Document Executors page Key: SPARK-28539 URL: https://issues.apache.org/jira/browse/SPARK-28539 Project: Spark Issue Type: Sub-task Components: SQL

[jira] [Created] (SPARK-28538) Document SQL page

2019-07-27 Thread Xiao Li (JIRA)
Xiao Li created SPARK-28538: --- Summary: Document SQL page Key: SPARK-28538 URL: https://issues.apache.org/jira/browse/SPARK-28538 Project: Spark Issue Type: Sub-task Components: SQL, Web

[jira] [Updated] (SPARK-28520) WholeStageCodegen does not work property for LocalTableScanExec

2019-07-27 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-28520: --- Issue Type: Bug (was: Improvement) > WholeStageCodegen does not work property for

[jira] [Resolved] (SPARK-28535) Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28535. --- Resolution: Fixed Fix Version/s: 2.4.4 2.3.4

[jira] [Commented] (SPARK-28373) Document JDBC/ODBC Server page

2019-07-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894513#comment-16894513 ] Xiao Li commented on SPARK-28373: - ping [~yumwang] > Document JDBC/ODBC Server page >

[jira] [Assigned] (SPARK-28535) Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28535: - Assignee: Marcelo Vanzin > Flaky test: JobCancellationSuite."interruptible iterator of

[jira] [Resolved] (SPARK-23158) Move HadoopFsRelationTest test suites to from sql/hive to sql/core

2019-07-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23158. - Resolution: Later > Move HadoopFsRelationTest test suites to from sql/hive to sql/core >

[jira] [Resolved] (SPARK-22759) Filters can be combined iff both are deterministic

2019-07-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22759. - Resolution: Fixed Fix Version/s: 2.3.0 > Filters can be combined iff both are deterministic >

[jira] [Resolved] (SPARK-22931) Add deterministic parameter to registerFunction

2019-07-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22931. - Resolution: Won't Fix > Add deterministic parameter to registerFunction >

[jira] [Resolved] (SPARK-18543) SaveAsTable(CTAS) using overwrite could change table definition

2019-07-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-18543. - Resolution: Won't Fix > SaveAsTable(CTAS) using overwrite could change table definition >

[jira] [Resolved] (SPARK-22965) Add deterministic parameter to registerJavaFunction

2019-07-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22965. - Resolution: Won't Fix > Add deterministic parameter to registerJavaFunction >

[jira] [Resolved] (SPARK-28536) Reduce shuffle partitions in Python UDF tests in SQLQueryTestSuite

2019-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28536. --- Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 3.0.0 This is

[jira] [Created] (SPARK-28537) DebugExec cannot debug broadcast or columnar related queries.

2019-07-27 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-28537: -- Summary: DebugExec cannot debug broadcast or columnar related queries. Key: SPARK-28537 URL: https://issues.apache.org/jira/browse/SPARK-28537 Project: Spark

[jira] [Commented] (SPARK-28522) Pass dynamic parameters to custom file input format

2019-07-27 Thread Ayan Mukherjee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894478#comment-16894478 ] Ayan Mukherjee commented on SPARK-28522: Please share an example. > Pass dynamic parameters to

[jira] [Commented] (SPARK-28515) to_timestamp returns null for summer time switch dates

2019-07-27 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-28515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894422#comment-16894422 ] Andreas Költringer commented on SPARK-28515: I used it with Python 3.7. But I don't think

[jira] [Commented] (SPARK-24036) Stateful operators in continuous processing

2019-07-27 Thread Kevin Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894323#comment-16894323 ] Kevin Zhang commented on SPARK-24036: - Hi [~joseph.torres], is there any update on this work? Will

[jira] [Updated] (SPARK-28519) Tests failed on aarch64 due the value of math.log and power function is different

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28519: - Description: Sorry to disturb again, we ran unit tests on arm64 instance, and there are other

[jira] [Commented] (SPARK-28519) Tests failed on aarch64 due the value of math.log and power function is different

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894302#comment-16894302 ] Hyukjin Kwon commented on SPARK-28519: -- There are multiple issues related with it. Can you search

[jira] [Resolved] (SPARK-28522) Pass dynamic parameters to custom file input format

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28522. -- Resolution: Invalid > Pass dynamic parameters to custom file input format >

[jira] [Commented] (SPARK-28522) Pass dynamic parameters to custom file input format

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894301#comment-16894301 ] Hyukjin Kwon commented on SPARK-28522: -- You can use Hadoop configuration for hadoop input/output

[jira] [Resolved] (SPARK-28526) Kinesis Metric Sink

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28526. -- Resolution: Won't Fix > Kinesis Metric Sink > --- > > Key:

[jira] [Commented] (SPARK-28526) Kinesis Metric Sink

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894300#comment-16894300 ] Hyukjin Kwon commented on SPARK-28526: -- I think that should better be existed as a thridparty

[jira] [Resolved] (SPARK-28523) Cant Select from a partitioned table using the Hive Warehouse Connector

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28523. -- Resolution: Invalid This looks an issue in HWC > Cant Select from a partitioned table using

[jira] [Resolved] (SPARK-28533) PySpark datatype casting error

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28533. -- Resolution: Invalid > PySpark datatype casting error > -- > >

[jira] [Commented] (SPARK-28533) PySpark datatype casting error

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894294#comment-16894294 ] Hyukjin Kwon commented on SPARK-28533: -- That's because you specified the output type as \{{array}}

[jira] [Updated] (SPARK-28533) PySpark datatype casting error

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28533: - Description: Hello, I have faced an issue while casting the datatype of a column in pyspark

[jira] [Updated] (SPARK-28533) PySpark datatype casting error

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28533: - Description: Hello, I have faced an issue while casting the datatype of a column in pyspark

[jira] [Updated] (SPARK-28533) PySpark datatype casting error

2019-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28533: - Description: Hello, I have faced an issue while casting the datatype of a column in pyspark