[jira] [Created] (SPARK-27438) Increase precision of to_timestamp

Maxim Gekk created SPARK-27438: -- Summary: Increase precision of to_timestamp Key: SPARK-27438 URL: https://issues.apache.org/jira/browse/SPARK-27438 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-27437) check the parquet column format

xzh_dz created SPARK-27437: -- Summary: check the parquet column format Key: SPARK-27437 URL: https://issues.apache.org/jira/browse/SPARK-27437 Project: Spark Issue Type: Test Components:

[jira] [Created] (SPARK-27436) Add spark.sql.optimizer.nonExcludedRules

Dongjoon Hyun created SPARK-27436: - Summary: Add spark.sql.optimizer.nonExcludedRules Key: SPARK-27436 URL: https://issues.apache.org/jira/browse/SPARK-27436 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is cleared

[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815028#comment-16815028 ] Lantao Jin edited comment on SPARK-27337 at 4/11/19 2:47 AM: - [~vinooganesh]

[jira] [Commented] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is cleared

[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815028#comment-16815028 ] Lantao Jin commented on SPARK-27337: [~vinooganesh] Could you point me the title of mail thread in

[jira] [Commented] (SPARK-26568) Too many partitions may cause thriftServer frequently Full GC

[ https://issues.apache.org/jira/browse/SPARK-26568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815016#comment-16815016 ] zhoukang commented on SPARK-26568: -- It is hive client we used in spark cause this problem [~srowen] >

[jira] [Commented] (SPARK-26703) Hive record writer will always depends on parquet-1.6 writer should fix it

[ https://issues.apache.org/jira/browse/SPARK-26703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815017#comment-16815017 ] zhoukang commented on SPARK-26703: -- I can make a pr for this [~hyukjin.kwon] > Hive record writer will

[jira] [Commented] (SPARK-26533) Support query auto cancel on thriftserver

[ https://issues.apache.org/jira/browse/SPARK-26533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815015#comment-16815015 ] zhoukang commented on SPARK-26533: -- I will work on this > Support query auto cancel on thriftserver >

[jira] [Commented] (SPARK-27068) Support failed jobs ui and completed jobs ui use different queue

[ https://issues.apache.org/jira/browse/SPARK-27068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815010#comment-16815010 ] shahid commented on SPARK-27068: Already failed jobs is in a single table right?. I meant, For running

[jira] [Commented] (SPARK-27068) Support failed jobs ui and completed jobs ui use different queue

[ https://issues.apache.org/jira/browse/SPARK-27068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815005#comment-16815005 ] zhoukang commented on SPARK-27068: -- For long-running application like spark sql,showing failed jobs in

[jira] [Assigned] (SPARK-27088) Apply conf "spark.sql.optimizer.planChangeLog.level" to batch plan change in RuleExecutor

[ https://issues.apache.org/jira/browse/SPARK-27088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27088: Assignee: Chakravarthi > Apply conf "spark.sql.optimizer.planChangeLog.level" to batch

[jira] [Resolved] (SPARK-27088) Apply conf "spark.sql.optimizer.planChangeLog.level" to batch plan change in RuleExecutor

[ https://issues.apache.org/jira/browse/SPARK-27088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27088. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24136

[jira] [Updated] (SPARK-27394) The staleness of UI may last minutes or hours when no tasks start or finish

[ https://issues.apache.org/jira/browse/SPARK-27394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27394: - Fix Version/s: 2.4.2 > The staleness of UI may last minutes or hours when no tasks start or

[jira] [Updated] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is cleared

[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoo Ganesh updated SPARK-27337: - Summary: QueryExecutionListener never cleans up listeners from the bus after SparkSession is

[jira] [Commented] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is cleared

[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814885#comment-16814885 ] Vinoo Ganesh commented on SPARK-27337: -- Clearing the spark session* Updated the ticket title, and

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814883#comment-16814883 ] Robert Joseph Evans commented on SPARK-27396: - This SPIP has been up for 5 days and I see 10

[jira] [Updated] (SPARK-27421) RuntimeException when querying a view on a partitioned parquet table

[ https://issues.apache.org/jira/browse/SPARK-27421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Maynard updated SPARK-27421: - Affects Version/s: 2.4.1 > RuntimeException when querying a view on a partitioned parquet table

[jira] [Commented] (SPARK-27421) RuntimeException when querying a view on a partitioned parquet table

[ https://issues.apache.org/jira/browse/SPARK-27421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814872#comment-16814872 ] Eric Maynard commented on SPARK-27421: -- [~shivuson...@gmail.com] Any hiccup confirming the issue

[jira] [Commented] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.6

[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814762#comment-16814762 ] shane knapp commented on SPARK-25079: - pyarrow 0.12.1 is out, i've updated my conda test environment

[jira] [Commented] (SPARK-26218) Throw exception on overflow for integers

[ https://issues.apache.org/jira/browse/SPARK-26218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814709#comment-16814709 ] Reynold Xin commented on SPARK-26218: - The no-exception is by design. Imagine you have an ETL job

[jira] [Created] (SPARK-27435) Support schema pruning in file source V2

Gengliang Wang created SPARK-27435: -- Summary: Support schema pruning in file source V2 Key: SPARK-27435 URL: https://issues.apache.org/jira/browse/SPARK-27435 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-27419) When setting spark.executor.heartbeatInterval to a value less than 1 seconds, it will always fail

[ https://issues.apache.org/jira/browse/SPARK-27419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-27419. -- Resolution: Fixed Fix Version/s: 2.4.2 > When setting spark.executor.heartbeatInterval

[jira] [Updated] (SPARK-27434) memory leak in spark driver

[ https://issues.apache.org/jira/browse/SPARK-27434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryne Yang updated SPARK-27434: -- Description: we got a OOM exception on the driver after driver has completed multiple jobs(we are

[jira] [Updated] (SPARK-27434) memory leak in spark driver

[ https://issues.apache.org/jira/browse/SPARK-27434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryne Yang updated SPARK-27434: -- Attachment: Screen Shot 2019-04-10 at 12.11.35 PM.png > memory leak in spark driver >

[jira] [Created] (SPARK-27434) memory leak in spark driver

Ryne Yang created SPARK-27434: - Summary: memory leak in spark driver Key: SPARK-27434 URL: https://issues.apache.org/jira/browse/SPARK-27434 Project: Spark Issue Type: Bug Components:

[jira] [Created] (SPARK-27433) Spark Structured Streaming left outer joins returns outer nulls for already matched rows

Binit created SPARK-27433: - Summary: Spark Structured Streaming left outer joins returns outer nulls for already matched rows Key: SPARK-27433 URL: https://issues.apache.org/jira/browse/SPARK-27433 Project:

[jira] [Updated] (SPARK-27432) Spark job stuck when no jobs/stages are pending

[ https://issues.apache.org/jira/browse/SPARK-27432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajashekar updated SPARK-27432: --- Attachment: jobs.pdf stages.pdf > Spark job stuck when no jobs/stages are pending >

[jira] [Created] (SPARK-27432) Spark job stuck when no jobs/stages are pending

Rajashekar created SPARK-27432: -- Summary: Spark job stuck when no jobs/stages are pending Key: SPARK-27432 URL: https://issues.apache.org/jira/browse/SPARK-27432 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-27423) Cast DATE to/from TIMESTAMP according to SQL standard

[ https://issues.apache.org/jira/browse/SPARK-27423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27423: --- Assignee: Maxim Gekk > Cast DATE to/from TIMESTAMP according to SQL standard >

[jira] [Resolved] (SPARK-27423) Cast DATE to/from TIMESTAMP according to SQL standard

[ https://issues.apache.org/jira/browse/SPARK-27423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27423. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24332

[jira] [Assigned] (SPARK-27422) CurrentDate should return local date

[ https://issues.apache.org/jira/browse/SPARK-27422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27422: --- Assignee: Maxim Gekk > CurrentDate should return local date >

[jira] [Resolved] (SPARK-27422) CurrentDate should return local date

[ https://issues.apache.org/jira/browse/SPARK-27422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27422. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24330

[jira] [Resolved] (SPARK-27412) Add a new shuffle manager to use Persistent Memory as shuffle and spilling storage

[ https://issues.apache.org/jira/browse/SPARK-27412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27412. --- Resolution: Won't Fix > Add a new shuffle manager to use Persistent Memory as shuffle and spilling

[jira] [Created] (SPARK-27431) move HashedRelation to global UnifiedMemoryManager and enable offheap

Xiaoju Wu created SPARK-27431: - Summary: move HashedRelation to global UnifiedMemoryManager and enable offheap Key: SPARK-27431 URL: https://issues.apache.org/jira/browse/SPARK-27431 Project: Spark

[jira] [Updated] (SPARK-27428) Test "metrics StatsD sink with Timer " fails on BigEndian

[ https://issues.apache.org/jira/browse/SPARK-27428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-27428: -- Description: Test case "metrics StatsD sink with Timer *** FAILED ***" fails with error

[jira] [Updated] (SPARK-27428) Test "metrics StatsD sink with Timer " fails on BigEndian

[ https://issues.apache.org/jira/browse/SPARK-27428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-27428: -- Description: Test case "metrics StatsD sink with Timer *** FAILED ***" fails with error

[jira] [Updated] (SPARK-27428) Test "metrics StatsD sink with Timer " fails on BigEndian

[ https://issues.apache.org/jira/browse/SPARK-27428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-27428: -- Description: Test case "metrics StatsD sink with Timer *** FAILED ***" fails with error

[jira] [Updated] (SPARK-27428) Test "metrics StatsD sink with Timer " fails on BigEndian

[ https://issues.apache.org/jira/browse/SPARK-27428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-27428: -- Description: Test case "metrics StatsD sink with Timer *** FAILED ***" fails with error

[jira] [Updated] (SPARK-27430) BroadcastNestedLoopJoinExec can build any side whatever the join type is

[ https://issues.apache.org/jira/browse/SPARK-27430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-27430: Summary: BroadcastNestedLoopJoinExec can build any side whatever the join type is (was:

[jira] [Created] (SPARK-27430) BroadcastNestedLoopJoinExec should support all join types

Wenchen Fan created SPARK-27430: --- Summary: BroadcastNestedLoopJoinExec should support all join types Key: SPARK-27430 URL: https://issues.apache.org/jira/browse/SPARK-27430 Project: Spark

[jira] [Created] (SPARK-27429) [SQL] to_timestamp function with additional argument flag that will allow exception if value could not be cast

t oo created SPARK-27429: Summary: [SQL] to_timestamp function with additional argument flag that will allow exception if value could not be cast Key: SPARK-27429 URL: https://issues.apache.org/jira/browse/SPARK-27429

[jira] [Resolved] (SPARK-26012) Dynamic partition will fail when both '' and null values are taken as dynamic partition values simultaneously.

[ https://issues.apache.org/jira/browse/SPARK-26012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26012. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24334

[jira] [Assigned] (SPARK-26012) Dynamic partition will fail when both '' and null values are taken as dynamic partition values simultaneously.

[ https://issues.apache.org/jira/browse/SPARK-26012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26012: --- Assignee: eaton > Dynamic partition will fail when both '' and null values are taken as

[jira] [Created] (SPARK-27428) Test "metrics StatsD sink with Timer " fails on BigEndian

Anuja Jakhade created SPARK-27428: - Summary: Test "metrics StatsD sink with Timer " fails on BigEndian Key: SPARK-27428 URL: https://issues.apache.org/jira/browse/SPARK-27428 Project: Spark

[jira] [Created] (SPARK-27427) cannot change list of packages from PYSPARK_SUBMIT_ARGS after restarting context

Natalino Busa created SPARK-27427: - Summary: cannot change list of packages from PYSPARK_SUBMIT_ARGS after restarting context Key: SPARK-27427 URL: https://issues.apache.org/jira/browse/SPARK-27427

[jira] [Commented] (SPARK-27409) Micro-batch support for Kafka Source in Spark 2.3

[ https://issues.apache.org/jira/browse/SPARK-27409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814314#comment-16814314 ] Gabor Somogyi commented on SPARK-27409: --- ping [~pbharaj] > Micro-batch support for Kafka Source

[jira] [Updated] (SPARK-27425) Add count_if functions

[ https://issues.apache.org/jira/browse/SPARK-27425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaerim Yeo updated SPARK-27425: Description: Add aggregation function which returns the number of records satisfying a given

[jira] [Updated] (SPARK-27425) Add count_if functions

[ https://issues.apache.org/jira/browse/SPARK-27425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaerim Yeo updated SPARK-27425: Description: Add aggregation function which returns the number of records satisfying a given

[jira] [Updated] (SPARK-27425) Add count_if functions

[ https://issues.apache.org/jira/browse/SPARK-27425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaerim Yeo updated SPARK-27425: Summary: Add count_if functions (was: SQL count_if functions) > Add count_if functions >

[jira] [Commented] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed

[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814247#comment-16814247 ] Lantao Jin commented on SPARK-27337: {quote} This means that if you close and remake spark sessions

[jira] [Created] (SPARK-27426) SparkAppHandle states not getting updated in Kubernetes

Nishant Ranjan created SPARK-27426: -- Summary: SparkAppHandle states not getting updated in Kubernetes Key: SPARK-27426 URL: https://issues.apache.org/jira/browse/SPARK-27426 Project: Spark

[jira] [Created] (SPARK-27425) SQL count_if functions

Chaerim Yeo created SPARK-27425: --- Summary: SQL count_if functions Key: SPARK-27425 URL: https://issues.apache.org/jira/browse/SPARK-27425 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-27406) UnsafeArrayData serialization breaks when two machines have different Oops size

[ https://issues.apache.org/jira/browse/SPARK-27406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27406. - Resolution: Fixed Assignee: peng bo Fix Version/s: 3.0.0

[jira] [Assigned] (SPARK-24872) Remove the symbol “||” of the “OR” operation

[ https://issues.apache.org/jira/browse/SPARK-24872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24872: --- Assignee: hantiantian > Remove the symbol “||” of the “OR” operation >

[jira] [Resolved] (SPARK-24872) Remove the symbol “||” of the “OR” operation

[ https://issues.apache.org/jira/browse/SPARK-24872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24872. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21826

[jira] [Resolved] (SPARK-27414) make it clear that date type is timezone independent

[ https://issues.apache.org/jira/browse/SPARK-27414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27414. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24325

[jira] [Commented] (SPARK-27424) Joining of one stream against the most recent update in another stream

[ https://issues.apache.org/jira/browse/SPARK-27424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814120#comment-16814120 ] Thilo Schneider commented on SPARK-27424: - Attached is a - not fully detailed - sketch of the

[jira] [Updated] (SPARK-27424) Joining of one stream against the most recent update in another stream

[ https://issues.apache.org/jira/browse/SPARK-27424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thilo Schneider updated SPARK-27424: Attachment: join-last-update-design.pdf > Joining of one stream against the most recent

[jira] [Created] (SPARK-27424) Joining of one stream against the most recent update in another stream

Thilo Schneider created SPARK-27424: --- Summary: Joining of one stream against the most recent update in another stream Key: SPARK-27424 URL: https://issues.apache.org/jira/browse/SPARK-27424

[jira] [Resolved] (SPARK-27181) Add public expression and transform API for DSv2 partitioning

[ https://issues.apache.org/jira/browse/SPARK-27181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27181. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24117

[jira] [Assigned] (SPARK-27181) Add public expression and transform API for DSv2 partitioning

[ https://issues.apache.org/jira/browse/SPARK-27181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27181: --- Assignee: Ryan Blue > Add public expression and transform API for DSv2 partitioning >

[jira] [Commented] (SPARK-27289) spark-submit explicit configuration does not take effect but Spark UI shows it's effective

[ https://issues.apache.org/jira/browse/SPARK-27289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814094#comment-16814094 ] KaiXu commented on SPARK-27289: --- I have verified that the intermediate data is written to spark.local.dir