[jira] [Created] (SPARK-38919) Filter stale event log before parse logs

2022-04-15 Thread shuyouZZ (Jira)
shuyouZZ created SPARK-38919: Summary: Filter stale event log before parse logs Key: SPARK-38919 URL: https://issues.apache.org/jira/browse/SPARK-38919 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-38824) Bug in async commit of Kafka offset in DirectKafkaInputDStream

2022-04-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522996#comment-17522996 ] L. C. Hsieh commented on SPARK-38824: - Ideally, I think we might need to call commitAll in stop

[jira] [Commented] (SPARK-38824) Bug in async commit of Kafka offset in DirectKafkaInputDStream

2022-04-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522995#comment-17522995 ] L. C. Hsieh commented on SPARK-38824: - Hmm, I think I can get the idea you want to point out. The

[jira] [Commented] (SPARK-38918) Nested column pruning should filter out attributes that do not belong to the current relation

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522990#comment-17522990 ] Apache Spark commented on SPARK-38918: -- User 'allisonwang-db' has created a pull request for this

[jira] [Assigned] (SPARK-38918) Nested column pruning should filter out attributes that do not belong to the current relation

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38918: Assignee: (was: Apache Spark) > Nested column pruning should filter out attributes

[jira] [Assigned] (SPARK-38918) Nested column pruning should filter out attributes that do not belong to the current relation

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38918: Assignee: Apache Spark > Nested column pruning should filter out attributes that do not

[jira] [Commented] (SPARK-38918) Nested column pruning should filter out attributes that do not belong to the current relation

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522989#comment-17522989 ] Apache Spark commented on SPARK-38918: -- User 'allisonwang-db' has created a pull request for this

[jira] [Commented] (SPARK-38917) StreamingQuery.processAllAvailable() blocks forever on queries containing mapGroupsWithState(GroupStateTimeout.ProcessingTimeTimeout())

2022-04-15 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522951#comment-17522951 ] Jungtaek Lim commented on SPARK-38917: -- Would you mind attaching thread dump, please? That would

[jira] [Created] (SPARK-38918) Nested column pruning should filter out attributes that do not belong to the current relation

2022-04-15 Thread Allison Wang (Jira)
Allison Wang created SPARK-38918: Summary: Nested column pruning should filter out attributes that do not belong to the current relation Key: SPARK-38918 URL: https://issues.apache.org/jira/browse/SPARK-38918

[jira] [Created] (SPARK-38917) StreamingQuery.processAllAvailable() blocks forever on queries containing mapGroupsWithState(GroupStateTimeout.ProcessingTimeTimeout())

2022-04-15 Thread Trevor Christman (Jira)
Trevor Christman created SPARK-38917: Summary: StreamingQuery.processAllAvailable() blocks forever on queries containing mapGroupsWithState(GroupStateTimeout.ProcessingTimeTimeout()) Key: SPARK-38917 URL:

[jira] [Resolved] (SPARK-38724) Test the error class: DIVIDE_BY_ZERO

2022-04-15 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-38724. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36193

[jira] [Assigned] (SPARK-38724) Test the error class: DIVIDE_BY_ZERO

2022-04-15 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-38724: Assignee: panbingkun > Test the error class: DIVIDE_BY_ZERO >

[jira] [Created] (SPARK-38916) Tasks not killed caused by race conditions between killTask() and launchTask()

2022-04-15 Thread Wei Xue (Jira)
Wei Xue created SPARK-38916: --- Summary: Tasks not killed caused by race conditions between killTask() and launchTask() Key: SPARK-38916 URL: https://issues.apache.org/jira/browse/SPARK-38916 Project: Spark

[jira] [Commented] (SPARK-38884) java.util.NoSuchElementException: key not found: numPartitions

2022-04-15 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522885#comment-17522885 ] Bruce Robbins commented on SPARK-38884: --- {quote}what`s self-contained reproducer?{quote}

[jira] [Assigned] (SPARK-38573) Support Auto Partition Statistics Collection

2022-04-15 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned SPARK-38573: Assignee: Kazuyuki Tanimura > Support Auto Partition Statistics Collection >

[jira] [Resolved] (SPARK-38573) Support Auto Partition Statistics Collection

2022-04-15 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-38573. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36067

[jira] [Created] (SPARK-38915) Log executed SQL queries

2022-04-15 Thread Jira
Gérald Quintana created SPARK-38915: --- Summary: Log executed SQL queries Key: SPARK-38915 URL: https://issues.apache.org/jira/browse/SPARK-38915 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-38914) Allow user to insert specified columns into insertable view

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522811#comment-17522811 ] Apache Spark commented on SPARK-38914: -- User 'morvenhuang' has created a pull request for this

[jira] [Assigned] (SPARK-38914) Allow user to insert specified columns into insertable view

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38914: Assignee: Apache Spark > Allow user to insert specified columns into insertable view >

[jira] [Assigned] (SPARK-38914) Allow user to insert specified columns into insertable view

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38914: Assignee: (was: Apache Spark) > Allow user to insert specified columns into

[jira] [Updated] (SPARK-38914) Allow user to insert specified columns into insertable view

2022-04-15 Thread morvenhuang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] morvenhuang updated SPARK-38914: Description: The option `spark.sql.defaultColumn.useNullsForMissingDefautValues` allows us to

[jira] [Created] (SPARK-38914) Allow user to insert specified columns into insertable view

2022-04-15 Thread morvenhuang (Jira)
morvenhuang created SPARK-38914: --- Summary: Allow user to insert specified columns into insertable view Key: SPARK-38914 URL: https://issues.apache.org/jira/browse/SPARK-38914 Project: Spark

[jira] [Resolved] (SPARK-38892) Fix the UT of schema equal assert

2022-04-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38892. -- Fix Version/s: 3.3.0 3.0.4 3.2.2

[jira] [Assigned] (SPARK-38892) Fix the UT of schema equal assert

2022-04-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38892: Assignee: YuanGuanhu > Fix the UT of schema equal assert >

[jira] [Commented] (SPARK-38738) Test the error class: INVALID_FRACTION_OF_SECOND

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522799#comment-17522799 ] Apache Spark commented on SPARK-38738: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-38738) Test the error class: INVALID_FRACTION_OF_SECOND

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38738: Assignee: Apache Spark > Test the error class: INVALID_FRACTION_OF_SECOND >

[jira] [Commented] (SPARK-38738) Test the error class: INVALID_FRACTION_OF_SECOND

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522798#comment-17522798 ] Apache Spark commented on SPARK-38738: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-38738) Test the error class: INVALID_FRACTION_OF_SECOND

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38738: Assignee: (was: Apache Spark) > Test the error class: INVALID_FRACTION_OF_SECOND >

[jira] [Resolved] (SPARK-38908) Provide query context in runtime error of Casting from String to Number/Date/Timestamp/Boolean

2022-04-15 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-38908. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 36206

[jira] [Updated] (SPARK-38910) Clean sparkStaging dir should before unregister()

2022-04-15 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-38910: -- Description: {code:java} ShutdownHookManager.addShutdownHook(priority) { () => try {

[jira] [Updated] (SPARK-38910) Clean sparkStaging dir should before unregister()

2022-04-15 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-38910: -- Description: {code:java} {code} Not clean the staging dir when match case {code:jave}

[jira] [Updated] (SPARK-38910) Clean sparkStaging dir should before unregister()

2022-04-15 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-38910: -- Summary: Clean sparkStaging dir should before unregister() (was: Clean sparkStaging dir should bef)

[jira] [Updated] (SPARK-38910) Clean sparkStaging dir should bef

2022-04-15 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-38910: -- Summary: Clean sparkStaging dir should bef (was: Clean sparkStaging dir when WAIT_FOR_APP_COMPLETION

[jira] [Assigned] (SPARK-38913) Output identifiers in error messages in SQL style

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38913: Assignee: Max Gekk (was: Apache Spark) > Output identifiers in error messages in SQL

[jira] [Assigned] (SPARK-38913) Output identifiers in error messages in SQL style

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38913: Assignee: Apache Spark (was: Max Gekk) > Output identifiers in error messages in SQL

[jira] [Commented] (SPARK-38913) Output identifiers in error messages in SQL style

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522750#comment-17522750 ] Apache Spark commented on SPARK-38913: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Updated] (SPARK-38913) Output identifiers in error messages in SQL style

2022-04-15 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38913: - Description: All identifiers like table names should be printed in SQL style in error messages. For

[jira] [Created] (SPARK-38913) Output identifiers in error messages in SQL style

2022-04-15 Thread Max Gekk (Jira)
Max Gekk created SPARK-38913: Summary: Output identifiers in error messages in SQL style Key: SPARK-38913 URL: https://issues.apache.org/jira/browse/SPARK-38913 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-38697) Extend SparkSessionExtensions to inject rules into AQE Optimizer

2022-04-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38697: --- Assignee: XiDuo You > Extend SparkSessionExtensions to inject rules into AQE Optimizer >

[jira] [Resolved] (SPARK-38697) Extend SparkSessionExtensions to inject rules into AQE Optimizer

2022-04-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38697. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36011

[jira] [Assigned] (SPARK-38750) Test the error class: SECOND_FUNCTION_ARGUMENT_NOT_INTEGER

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38750: Assignee: Apache Spark > Test the error class: SECOND_FUNCTION_ARGUMENT_NOT_INTEGER >

[jira] [Assigned] (SPARK-38750) Test the error class: SECOND_FUNCTION_ARGUMENT_NOT_INTEGER

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38750: Assignee: (was: Apache Spark) > Test the error class:

[jira] [Commented] (SPARK-38750) Test the error class: SECOND_FUNCTION_ARGUMENT_NOT_INTEGER

2022-04-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522719#comment-17522719 ] Apache Spark commented on SPARK-38750: -- User 'panbingkun' has created a pull request for this

[jira] [Updated] (SPARK-38901) DS V2 supports push down misc functions

2022-04-15 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-38901: --- Summary: DS V2 supports push down misc functions (was: Support for more misc functions to push

[jira] [Updated] (SPARK-38900) DS V2 supports push down collection functions

2022-04-15 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-38900: --- Summary: DS V2 supports push down collection functions (was: Support for more collection functions

[jira] [Updated] (SPARK-38899) DS V2 supports push down datetime functions

2022-04-15 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-38899: --- Summary: DS V2 supports push down datetime functions (was: Support for more datetime functions to

[jira] [Updated] (SPARK-38897) DS V2 supports push down string functions

2022-04-15 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-38897: --- Summary: DS V2 supports push down string functions (was: Support for more string functions to push

[jira] [Commented] (SPARK-38900) Support for more collection functions to push down to data source

2022-04-15 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522716#comment-17522716 ] jiaan.geng commented on SPARK-38900: Could you do the job? > Support for more collection functions

[jira] [Commented] (SPARK-38899) Support for more datetime functions to push down to data source

2022-04-15 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522715#comment-17522715 ] jiaan.geng commented on SPARK-38899: [~Zhixiong Chen] Could you do the job? > Support for more

[jira] [Commented] (SPARK-38901) Support for more misc functions to push down to data source

2022-04-15 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522717#comment-17522717 ] jiaan.geng commented on SPARK-38901: Could you do the job? > Support for more misc functions to

[jira] [Commented] (SPARK-38897) Support for more string functions to push down to data source

2022-04-15 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522711#comment-17522711 ] jiaan.geng commented on SPARK-38897: [~Zhixiong Chen] Please do the job. > Support for more string

[jira] [Created] (SPARK-38912) Clean "classproperty" workaround in pyspark.sql.session once support for Python 3.8 is dropped

2022-04-15 Thread Furcy Pin (Jira)
Furcy Pin created SPARK-38912: - Summary: Clean "classproperty" workaround in pyspark.sql.session once support for Python 3.8 is dropped Key: SPARK-38912 URL: https://issues.apache.org/jira/browse/SPARK-38912

[jira] [Resolved] (SPARK-38722) Test the error class: CAST_CAUSES_OVERFLOW

2022-04-15 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-38722. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36199

[jira] [Assigned] (SPARK-38722) Test the error class: CAST_CAUSES_OVERFLOW

2022-04-15 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-38722: Assignee: panbingkun > Test the error class: CAST_CAUSES_OVERFLOW >