[jira] [Commented] (SPARK-29245) CCE during creating HiveMetaStoreClient

[ https://issues.apache.org/jira/browse/SPARK-29245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081174#comment-17081174 ] Fokko Driesprong commented on SPARK-29245: -- Thanks, on Iceberg we have a similar issue: 

[jira] [Created] (SPARK-31421) transformAllExpressions can not transform expressions in ScalarSubquery

Yuming Wang created SPARK-31421: --- Summary: transformAllExpressions can not transform expressions in ScalarSubquery Key: SPARK-31421 URL: https://issues.apache.org/jira/browse/SPARK-31421 Project: Spark

[jira] [Issue Comment Deleted] (SPARK-31256) Dropna doesn't work for struct columns

[ https://issues.apache.org/jira/browse/SPARK-31256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JinxinTang updated SPARK-31256: --- Comment: was deleted (was: I will try to solve it) > Dropna doesn't work for struct columns >

[jira] [Commented] (SPARK-31256) Dropna doesn't work for struct columns

[ https://issues.apache.org/jira/browse/SPARK-31256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081084#comment-17081084 ] JinxinTang commented on SPARK-31256: I will try to solve it > Dropna doesn't work for struct

[jira] [Resolved] (SPARK-25154) Support NOT IN sub-queries inside nested OR conditions.

[ https://issues.apache.org/jira/browse/SPARK-25154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-25154. -- Fix Version/s: 3.1.0 Assignee: Dilip Biswal Resolution: Fixed

[jira] [Updated] (SPARK-25154) Support NOT IN sub-queries inside nested OR conditions.

[ https://issues.apache.org/jira/browse/SPARK-25154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-25154: - Issue Type: Improvement (was: Bug) > Support NOT IN sub-queries inside nested OR

[jira] [Commented] (SPARK-31420) Infinite timeline redraw in job details page

[ https://issues.apache.org/jira/browse/SPARK-31420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081017#comment-17081017 ] Gengliang Wang commented on SPARK-31420: [~sarutak] I looked into it but I am not familiar with

[jira] [Created] (SPARK-31420) Infinite timeline redraw in job details page

Gengliang Wang created SPARK-31420: -- Summary: Infinite timeline redraw in job details page Key: SPARK-31420 URL: https://issues.apache.org/jira/browse/SPARK-31420 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31420) Infinite timeline redraw in job details page

[ https://issues.apache.org/jira/browse/SPARK-31420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-31420: --- Attachment: timeline.mov > Infinite timeline redraw in job details page >

[jira] [Created] (SPARK-31419) Document Table-valued Function and Inline Table

Huaxin Gao created SPARK-31419: -- Summary: Document Table-valued Function and Inline Table Key: SPARK-31419 URL: https://issues.apache.org/jira/browse/SPARK-31419 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31417) Allow broadcast variables when using isin code

[ https://issues.apache.org/jira/browse/SPARK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florentino Sainz updated SPARK-31417: - Description: *I would like some help to explore if the following feature makes sense

[jira] [Updated] (SPARK-31417) Allow broadcast variables when using isin code

[ https://issues.apache.org/jira/browse/SPARK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florentino Sainz updated SPARK-31417: - Description: *I would like some help to explore if the following feature makes sense

[jira] [Updated] (SPARK-31417) Allow broadcast variables when using isin code

[ https://issues.apache.org/jira/browse/SPARK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florentino Sainz updated SPARK-31417: - Description: *I would like some help to explore if the following feature makes sense

[jira] [Updated] (SPARK-31417) Allow broadcast variables when using isin code

[ https://issues.apache.org/jira/browse/SPARK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florentino Sainz updated SPARK-31417: - Description: *I would like some help to explore if the following feature makes sense

[jira] [Updated] (SPARK-31417) Allow broadcast variables when using isin code

[ https://issues.apache.org/jira/browse/SPARK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florentino Sainz updated SPARK-31417: - Description: *I would like some help to explore if the following feature makes sense

[jira] [Updated] (SPARK-31417) Allow broadcast variables when using isin code

[ https://issues.apache.org/jira/browse/SPARK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florentino Sainz updated SPARK-31417: - Description: *I would like to for someone to explore if the following feature makes

[jira] [Commented] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-31418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080942#comment-17080942 ] Venkata krishnan Sowrirajan commented on SPARK-31418: - Currently, I'm working on

[jira] [Updated] (SPARK-31417) Allow broadcast variables when using isin code

[ https://issues.apache.org/jira/browse/SPARK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florentino Sainz updated SPARK-31417: - Description: *I would like to for someone to explore if the following feature makes

[jira] [Updated] (SPARK-31417) Allow broadcast variables when using isin code

[ https://issues.apache.org/jira/browse/SPARK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florentino Sainz updated SPARK-31417: - Description: *I would like to for someone to explore if the following feature makes

[jira] [Updated] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-31418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata krishnan Sowrirajan updated SPARK-31418: Description: With Spark blacklisting, if a task fails on an

[jira] [Updated] (SPARK-31417) Allow broadcast variables when using isin code

[ https://issues.apache.org/jira/browse/SPARK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florentino Sainz updated SPARK-31417: - Description: *I would like to for someone to explore if the following feature makes

[jira] [Created] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

Venkata krishnan Sowrirajan created SPARK-31418: --- Summary: Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation Key: SPARK-31418 URL:

[jira] [Created] (SPARK-31417) Allow broadcast variables when using isin code

Florentino Sainz created SPARK-31417: Summary: Allow broadcast variables when using isin code Key: SPARK-31417 URL: https://issues.apache.org/jira/browse/SPARK-31417 Project: Spark Issue

[jira] [Created] (SPARK-31416) Check more strictly that a field name can be used as a valid Java identifier for codegen

Kousuke Saruta created SPARK-31416: -- Summary: Check more strictly that a field name can be used as a valid Java identifier for codegen Key: SPARK-31416 URL: https://issues.apache.org/jira/browse/SPARK-31416

[jira] [Updated] (SPARK-31404) backward compatibility issues after switching to Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-31404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31404: -- Priority: Blocker (was: Major) > backward compatibility issues after switching to Proleptic

[jira] [Updated] (SPARK-31404) backward compatibility issues after switching to Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-31404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31404: -- Target Version/s: 3.0.0 > backward compatibility issues after switching to Proleptic

[jira] [Assigned] (SPARK-31415) builtin date-time functions/operations improvement

[ https://issues.apache.org/jira/browse/SPARK-31415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31415: - Assignee: Maxim Gekk > builtin date-time functions/operations improvement >

[jira] [Assigned] (SPARK-31306) rand() function documentation suggests an inclusive upper bound of 1.0

[ https://issues.apache.org/jira/browse/SPARK-31306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-31306: Assignee: Bryan Cutler > rand() function documentation suggests an inclusive upper bound

[jira] [Assigned] (SPARK-31306) rand() function documentation suggests an inclusive upper bound of 1.0

[ https://issues.apache.org/jira/browse/SPARK-31306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-31306: Assignee: (was: Bryan Cutler) > rand() function documentation suggests an inclusive

[jira] [Resolved] (SPARK-31306) rand() function documentation suggests an inclusive upper bound of 1.0

[ https://issues.apache.org/jira/browse/SPARK-31306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-31306. -- Resolution: Fixed Issue resolved by pull request 28071

[jira] [Commented] (SPARK-31399) closure cleaner is broken in Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-31399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080627#comment-17080627 ] Dongjoon Hyun commented on SPARK-31399: --- According to the above information, - I added the failure

[jira] [Updated] (SPARK-31399) closure cleaner is broken in Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-31399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31399: -- Affects Version/s: (was: 2.4.6) 2.4.5 > closure cleaner is broken

[jira] [Updated] (SPARK-31399) closure cleaner is broken in Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-31399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31399: -- Affects Version/s: 2.4.6 > closure cleaner is broken in Scala 2.12 >

[jira] [Updated] (SPARK-31399) closure cleaner is broken in Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-31399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31399: -- Description: The `ClosureCleaner` only support Scala functions and it uses the following

[jira] [Updated] (SPARK-31399) closure cleaner is broken in Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-31399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31399: -- Summary: closure cleaner is broken in Scala 2.12 (was: closure cleaner is broken in Spark

[jira] [Updated] (SPARK-31377) Add unit tests for "number of output rows" metric for joins in SQLMetricsSuite

[ https://issues.apache.org/jira/browse/SPARK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-31377: -- Description: For some combinations of join algorithm and join

[jira] [Updated] (SPARK-31377) Add unit tests for "number of output rows" metric for joins in SQLMetricsSuite

[ https://issues.apache.org/jira/browse/SPARK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-31377: -- Description: For some combinations of join algorithm and join

[jira] [Updated] (SPARK-30953) InsertAdaptiveSparkPlan should apply AQE on child plan of write commands

[ https://issues.apache.org/jira/browse/SPARK-30953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-30953: - Description: Apply AQE on write commands with child plan will expose  {{LogicalQueryStage}} to {{Analyzer}} 

[jira] [Updated] (SPARK-30953) InsertAdaptiveSparkPlan should apply AQE on child plan of write commands

[ https://issues.apache.org/jira/browse/SPARK-30953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-30953: - Summary: InsertAdaptiveSparkPlan should apply AQE on child plan of write commands (was:

[jira] [Resolved] (SPARK-31415) builtin date-time functions/operations improvement

[ https://issues.apache.org/jira/browse/SPARK-31415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31415. - Fix Version/s: 3.0.0 Resolution: Fixed > builtin date-time functions/operations

[jira] [Updated] (SPARK-27898) Support 4 date operators(date + integer, integer + date, date - integer and date - date)

[ https://issues.apache.org/jira/browse/SPARK-27898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-27898: Parent Issue: SPARK-31415 (was: SPARK-27764) > Support 4 date operators(date + integer, integer

[jira] [Updated] (SPARK-29387) Support `*` and `/` operators for intervals

[ https://issues.apache.org/jira/browse/SPARK-29387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29387: Parent Issue: SPARK-31415 (was: SPARK-27764) > Support `*` and `/` operators for intervals >

[jira] [Updated] (SPARK-29365) Support date and timestamp subtraction

[ https://issues.apache.org/jira/browse/SPARK-29365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29365: Parent Issue: SPARK-31415 (was: SPARK-27764) > Support date and timestamp subtraction >

[jira] [Updated] (SPARK-31415) builtin date-time functions/operations improvement

[ https://issues.apache.org/jira/browse/SPARK-31415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31415: Summary: builtin date-time functions/operations improvement (was: builtin date-time functions

[jira] [Updated] (SPARK-29774) Date and Timestamp type +/- null should be null as Postgres

[ https://issues.apache.org/jira/browse/SPARK-29774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29774: Parent Issue: SPARK-31415 (was: SPARK-27764) > Date and Timestamp type +/- null should be null

[jira] [Updated] (SPARK-29486) CalendarInterval should have 3 fields: months, days and microseconds

[ https://issues.apache.org/jira/browse/SPARK-29486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29486: Parent: SPARK-31415 Issue Type: Sub-task (was: Bug) > CalendarInterval should have 3

[jira] [Updated] (SPARK-29364) Return intervals from date subtracts

[ https://issues.apache.org/jira/browse/SPARK-29364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29364: Parent: SPARK-31415 Issue Type: Sub-task (was: Improvement) > Return intervals from date

[jira] [Updated] (SPARK-28420) Date/Time Functions: date_part for intervals

[ https://issues.apache.org/jira/browse/SPARK-28420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28420: Parent Issue: SPARK-31415 (was: SPARK-30375) > Date/Time Functions: date_part for intervals >

[jira] [Updated] (SPARK-29355) Support timestamps subtraction

[ https://issues.apache.org/jira/browse/SPARK-29355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29355: Parent Issue: SPARK-31415 (was: SPARK-27764) > Support timestamps subtraction >

[jira] [Updated] (SPARK-29187) Return null from `date_part()` for the null `field`

[ https://issues.apache.org/jira/browse/SPARK-29187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29187: Parent Issue: SPARK-31415 (was: SPARK-30375) > Return null from `date_part()` for the null

[jira] [Updated] (SPARK-28141) Date type can not accept special values

[ https://issues.apache.org/jira/browse/SPARK-28141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28141: Parent Issue: SPARK-31415 (was: SPARK-27764) > Date type can not accept special values >

[jira] [Updated] (SPARK-29012) Timestamp type can not accept special values

[ https://issues.apache.org/jira/browse/SPARK-29012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29012: Parent Issue: SPARK-31415 (was: SPARK-27764) > Timestamp type can not accept special values >

[jira] [Updated] (SPARK-28690) Date/Time Functions: date_part for timestamps

[ https://issues.apache.org/jira/browse/SPARK-28690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28690: Parent Issue: SPARK-31415 (was: SPARK-30375) > Date/Time Functions: date_part for timestamps >

[jira] [Updated] (SPARK-28687) Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`

[ https://issues.apache.org/jira/browse/SPARK-28687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28687: Parent Issue: SPARK-31415 (was: SPARK-30375) > Support `epoch`, `isoyear`, `milliseconds` and

[jira] [Updated] (SPARK-28700) make_timestamp loses seconds fractions

[ https://issues.apache.org/jira/browse/SPARK-28700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28700: Parent: SPARK-31415 Issue Type: Sub-task (was: Bug) > make_timestamp loses seconds

[jira] [Updated] (SPARK-28656) Support `millennium`, `century` and `decade` at `extract()`

[ https://issues.apache.org/jira/browse/SPARK-28656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28656: Parent Issue: SPARK-31415 (was: SPARK-30375) > Support `millennium`, `century` and `decade` at

[jira] [Updated] (SPARK-28017) Enhance DATE_TRUNC

[ https://issues.apache.org/jira/browse/SPARK-28017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28017: Parent Issue: SPARK-31415 (was: SPARK-30375) > Enhance DATE_TRUNC > -- > >

[jira] [Updated] (SPARK-28459) Date/Time Functions: make_timestamp

[ https://issues.apache.org/jira/browse/SPARK-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28459: Parent Issue: SPARK-31415 (was: SPARK-30375) > Date/Time Functions: make_timestamp >

[jira] [Updated] (SPARK-28432) Add `make_date` function

[ https://issues.apache.org/jira/browse/SPARK-28432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28432: Parent Issue: SPARK-31415 (was: SPARK-30375) > Add `make_date` function >

[jira] [Created] (SPARK-31415) builtin date-time functions improvement

Wenchen Fan created SPARK-31415: --- Summary: builtin date-time functions improvement Key: SPARK-31415 URL: https://issues.apache.org/jira/browse/SPARK-31415 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31121) catalog plugin API

[ https://issues.apache.org/jira/browse/SPARK-31121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31121: Summary: catalog plugin API (was: Spark API for Table Metadata) > catalog plugin API >

[jira] [Updated] (SPARK-27576) table capabilty to skip the output column resolution

[ https://issues.apache.org/jira/browse/SPARK-27576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-27576: Parent: SPARK-25390 Issue Type: Sub-task (was: Improvement) > table capabilty to skip

[jira] [Updated] (SPARK-27521) move data source v2 API to catalyst module

[ https://issues.apache.org/jira/browse/SPARK-27521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-27521: Parent: SPARK-25390 Issue Type: Sub-task (was: Improvement) > move data source v2 API to

[jira] [Updated] (SPARK-26811) Add DataSourceV2 capabilities to check support for batch append, overwrite, truncate during analysis.

[ https://issues.apache.org/jira/browse/SPARK-26811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-26811: Parent: SPARK-25390 Issue Type: Sub-task (was: Bug) > Add DataSourceV2 capabilities to

[jira] [Updated] (SPARK-27190) Add DataSourceV2 capabilities for streaming

[ https://issues.apache.org/jira/browse/SPARK-27190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-27190: Parent: SPARK-25390 Issue Type: Sub-task (was: Improvement) > Add DataSourceV2

[jira] [Updated] (SPARK-24882) data source v2 API improvement

[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24882: Parent Issue: SPARK-25390 (was: SPARK-22386) > data source v2 API improvement >

[jira] [Resolved] (SPARK-31406) ThriftServerQueryTestSuite: Sharing test data and test tables among multiple test cases.

[ https://issues.apache.org/jira/browse/SPARK-31406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31406. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28180

[jira] [Assigned] (SPARK-31406) ThriftServerQueryTestSuite: Sharing test data and test tables among multiple test cases.

[ https://issues.apache.org/jira/browse/SPARK-31406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31406: --- Assignee: jiaan.geng > ThriftServerQueryTestSuite: Sharing test data and test tables among

[jira] [Updated] (SPARK-31124) change the default value of minPartitionNum in AQE

[ https://issues.apache.org/jira/browse/SPARK-31124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31124: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > change the default value

[jira] [Created] (SPARK-31414) Performance regression with new TimestampFormatter for json and csv

Kent Yao created SPARK-31414: Summary: Performance regression with new TimestampFormatter for json and csv Key: SPARK-31414 URL: https://issues.apache.org/jira/browse/SPARK-31414 Project: Spark

[jira] [Updated] (SPARK-31413) Accessing the sequence number and partition id for records in Kinesis adapter

[ https://issues.apache.org/jira/browse/SPARK-31413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] varun senthilnathan updated SPARK-31413: Description: We are using spark 2.4.3 in java. We would like to log the partition

[jira] [Resolved] (SPARK-31412) New Adaptive Query Execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-31412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31412. - Fix Version/s: 3.0.0 Resolution: Fixed > New Adaptive Query Execution in Spark SQL >

[jira] [Updated] (SPARK-31413) Accessing the sequence number and partition id for records in Kinesis adapter

[ https://issues.apache.org/jira/browse/SPARK-31413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] varun senthilnathan updated SPARK-31413: Description: We are using spark 2.4.3 in java. We would like to log the partition

[jira] [Updated] (SPARK-30864) Add the user guide for Adaptive Query Execution

[ https://issues.apache.org/jira/browse/SPARK-30864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30864: Parent: SPARK-31412 Issue Type: Sub-task (was: Bug) > Add the user guide for Adaptive

[jira] [Updated] (SPARK-30918) improve the splitting of skewed partitions

[ https://issues.apache.org/jira/browse/SPARK-30918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30918: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > improve the splitting of

[jira] [Updated] (SPARK-28560) Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

[ https://issues.apache.org/jira/browse/SPARK-28560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28560: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > Optimize shuffle reader to

[jira] [Updated] (SPARK-29893) Improve the local reader performance by changing the task number from 1 to multi

[ https://issues.apache.org/jira/browse/SPARK-29893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29893: Parent: (was: SPARK-28560) Issue Type: Improvement (was: Sub-task) > Improve the

[jira] [Updated] (SPARK-31413) Accessing the sequence number and partition id for records in Kinesis adapter

[ https://issues.apache.org/jira/browse/SPARK-31413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] varun senthilnathan updated SPARK-31413: Environment: (was: We are using spark 2.4.3 in java. We would like to log the

[jira] [Updated] (SPARK-31413) Accessing the sequence number and partition id for records in Kinesis adapter

[ https://issues.apache.org/jira/browse/SPARK-31413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] varun senthilnathan updated SPARK-31413: Description: We are using spark 2.4.3 in java. We would like to log the partition

[jira] [Updated] (SPARK-31413) Accessing the sequence number and partition id for records in Kinesis adapter

[ https://issues.apache.org/jira/browse/SPARK-31413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] varun senthilnathan updated SPARK-31413: Description: We are using spark 2.4.3 in java. We would like to log the partition

[jira] [Created] (SPARK-31413) Accessing the sequence number and partition id for records in Kinesis adapter

varun senthilnathan created SPARK-31413: --- Summary: Accessing the sequence number and partition id for records in Kinesis adapter Key: SPARK-31413 URL: https://issues.apache.org/jira/browse/SPARK-31413

[jira] [Updated] (SPARK-28739) Add a simple cost check for Adaptive Query Execution

[ https://issues.apache.org/jira/browse/SPARK-28739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28739: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > Add a simple cost check

[jira] [Updated] (SPARK-28753) Dynamically reuse subqueries in AQE

[ https://issues.apache.org/jira/browse/SPARK-28753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28753: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > Dynamically reuse

[jira] [Updated] (SPARK-31070) make skew join split skewed partitions more evenly

[ https://issues.apache.org/jira/browse/SPARK-31070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31070: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > make skew join split

[jira] [Updated] (SPARK-31037) refine AQE config names

[ https://issues.apache.org/jira/browse/SPARK-31037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31037: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > refine AQE config names >

[jira] [Updated] (SPARK-31134) optimize skew join after shuffle partitions are coalesced

[ https://issues.apache.org/jira/browse/SPARK-31134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31134: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > optimize skew join after

[jira] [Updated] (SPARK-31201) add an individual config for skewed partition threshold

[ https://issues.apache.org/jira/browse/SPARK-31201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31201: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > add an individual config

[jira] [Updated] (SPARK-31253) add metrics to shuffle reader

[ https://issues.apache.org/jira/browse/SPARK-31253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31253: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > add metrics to shuffle

[jira] [Updated] (SPARK-31253) add metrics to AQE shuffle reader

[ https://issues.apache.org/jira/browse/SPARK-31253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31253: Summary: add metrics to AQE shuffle reader (was: add metrics to shuffle reader) > add metrics

[jira] [Updated] (SPARK-29544) Optimize skewed join at runtime with new Adaptive Execution

[ https://issues.apache.org/jira/browse/SPARK-29544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29544: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > Optimize skewed join at

[jira] [Updated] (SPARK-30524) Disable OptimizeSkewJoin rule if introducing additional shuffle.

[ https://issues.apache.org/jira/browse/SPARK-30524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30524: Parent Issue: SPARK-31412 (was: SPARK-29544) > Disable OptimizeSkewJoin rule if introducing

[jira] [Updated] (SPARK-29954) collect the runtime statistics of row count in map stage

[ https://issues.apache.org/jira/browse/SPARK-29954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29954: Parent Issue: SPARK-31412 (was: SPARK-29544) > collect the runtime statistics of row count in

[jira] [Updated] (SPARK-28177) Adjust post shuffle partition number in adaptive execution

[ https://issues.apache.org/jira/browse/SPARK-28177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28177: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > Adjust post shuffle

[jira] [Updated] (SPARK-31412) New Adaptive Query Execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-31412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31412: Summary: New Adaptive Query Execution in Spark SQL (was: Adaptive Query Execution in Spark SQL)

[jira] [Updated] (SPARK-31412) Adaptive Query Execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-31412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31412: Description: SPARK-9850 proposed the basic idea of adaptive execution in Spark. In DAGScheduler,

[jira] [Updated] (SPARK-23128) The basic framework for the new Adaptive Query Execution

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23128: Summary: The basic framework for the new Adaptive Query Execution (was: A new approach to do

[jira] [Updated] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23128: Parent: SPARK-31412 Issue Type: Sub-task (was: Improvement) > A new approach to do

[jira] [Created] (SPARK-31412) Adaptive Query Execution in Spark SQL

Wenchen Fan created SPARK-31412: --- Summary: Adaptive Query Execution in Spark SQL Key: SPARK-31412 URL: https://issues.apache.org/jira/browse/SPARK-31412 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-31399) closure cleaner is broken in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-31399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080359#comment-17080359 ] Josh Rosen commented on SPARK-31399: h3. My rough first impression I think the problem is that

[jira] [Assigned] (SPARK-31384) NPE in OptimizeSkewedJoin when there's a inputRDD of plan has 0 partition

[ https://issues.apache.org/jira/browse/SPARK-31384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31384: --- Assignee: wuyi > NPE in OptimizeSkewedJoin when there's a inputRDD of plan has 0 partition

  1   2   >