[jira] [Created] (SPARK-45510) Replace `scala.collection.generic.Growable` to `scala.collection.mutable.Growable`

2023-10-11 Thread Jia Fan (Jira)
Jia Fan created SPARK-45510: --- Summary: Replace `scala.collection.generic.Growable` to `scala.collection.mutable.Growable` Key: SPARK-45510 URL: https://issues.apache.org/jira/browse/SPARK-45510 Project:

[jira] [Commented] (SPARK-45424) Regression in CSV schema inference when timestamps do not match specified timestampFormat

2023-10-05 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772442#comment-17772442 ] Jia Fan commented on SPARK-45424: - Thanks [~andygrove] , I found the reason of the bug. Let me create a

[jira] [Updated] (SPARK-45433) CSV/JSON schema inference when timestamps do not match specified timestampFormat with only one row on each partition report error

2023-10-05 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-45433: Description: CSV/JSON schema inference when timestamps do not match specified timestampFormat with `only

[jira] [Created] (SPARK-45433) CSV/JSON schema inference when timestamps do not match specified timestampFormat with only one row on each partition report error

2023-10-05 Thread Jia Fan (Jira)
Jia Fan created SPARK-45433: --- Summary: CSV/JSON schema inference when timestamps do not match specified timestampFormat with only one row on each partition report error Key: SPARK-45433 URL:

[jira] [Updated] (SPARK-45413) Add warning for prepare drop LevelDB support

2023-10-04 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-45413: Summary: Add warning for prepare drop LevelDB support (was: Drop leveldb support for

[jira] [Commented] (SPARK-44223) Drop leveldb support

2023-10-04 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17771824#comment-17771824 ] Jia Fan commented on SPARK-44223: - Drop leveldb support for `spark.shuffle.service.db.backend` will be

[jira] [Created] (SPARK-45413) Drop leveldb support for `spark.history.store.hybridStore.diskBackend`

2023-10-04 Thread Jia Fan (Jira)
Jia Fan created SPARK-45413: --- Summary: Drop leveldb support for `spark.history.store.hybridStore.diskBackend` Key: SPARK-45413 URL: https://issues.apache.org/jira/browse/SPARK-45413 Project: Spark

[jira] [Created] (SPARK-45351) Change RocksDB as default shuffle service db backend

2023-09-26 Thread Jia Fan (Jira)
Jia Fan created SPARK-45351: --- Summary: Change RocksDB as default shuffle service db backend Key: SPARK-45351 URL: https://issues.apache.org/jira/browse/SPARK-45351 Project: Spark Issue Type:

[jira] [Created] (SPARK-45338) Remove scala.collection.JavaConverters

2023-09-26 Thread Jia Fan (Jira)
Jia Fan created SPARK-45338: --- Summary: Remove scala.collection.JavaConverters Key: SPARK-45338 URL: https://issues.apache.org/jira/browse/SPARK-45338 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-45116) Add some comment for param of JdbcDialect createTable

2023-09-11 Thread Jia Fan (Jira)
Jia Fan created SPARK-45116: --- Summary: Add some comment for param of JdbcDialect createTable Key: SPARK-45116 URL: https://issues.apache.org/jira/browse/SPARK-45116 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45075) Alter table with invalid default value will not report error

2023-09-04 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-45075: Summary: Alter table with invalid default value will not report error (was: Alter table with invaild

[jira] [Created] (SPARK-45075) Alter table with invaild default value will not report error

2023-09-04 Thread Jia Fan (Jira)
Jia Fan created SPARK-45075: --- Summary: Alter table with invaild default value will not report error Key: SPARK-45075 URL: https://issues.apache.org/jira/browse/SPARK-45075 Project: Spark Issue

[jira] [Updated] (SPARK-45059) Add try_reflect to Scala and Python

2023-09-01 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-45059: Summary: Add try_reflect to Scala and Python (was: Add to_reflect to Scala and Python) > Add

[jira] [Commented] (SPARK-45059) Add to_reflect to Scala and Python

2023-09-01 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761434#comment-17761434 ] Jia Fan commented on SPARK-45059: - I'm working on it > Add to_reflect to Scala and Python >

[jira] [Created] (SPARK-45059) Add to_reflect to Scala and Python

2023-09-01 Thread Jia Fan (Jira)
Jia Fan created SPARK-45059: --- Summary: Add to_reflect to Scala and Python Key: SPARK-45059 URL: https://issues.apache.org/jira/browse/SPARK-45059 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-44735) Log a warning when inserting columns with the same name by row that don't match up

2023-08-31 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760971#comment-17760971 ] Jia Fan commented on SPARK-44735: - It's a good improve, let me implement it. > Log a warning when

[jira] [Commented] (SPARK-44988) Parquet INT64 (TIMESTAMP(NANOS,false)) throwing Illegal Parquet type

2023-08-31 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760931#comment-17760931 ] Jia Fan commented on SPARK-44988: - Have you tried setting spark.sql.legacy.parquet.nanosAsLong to true?

[jira] [Updated] (SPARK-44978) Fix SQLQueryTestSuite unable create table normally

2023-08-27 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-44978: Description: When we repeatedly execute SQLQueryTestSuite to generate the golden file, the warehouse

[jira] [Updated] (SPARK-44978) Fix SQLQueryTestSuite unable create table normally

2023-08-27 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-44978: Attachment: image-2023-08-27-14-25-21-843.png > Fix SQLQueryTestSuite unable create table normally >

[jira] [Created] (SPARK-44978) Fix SQLQueryTestSuite unable create table normally

2023-08-27 Thread Jia Fan (Jira)
Jia Fan created SPARK-44978: --- Summary: Fix SQLQueryTestSuite unable create table normally Key: SPARK-44978 URL: https://issues.apache.org/jira/browse/SPARK-44978 Project: Spark Issue Type:

[jira] [Created] (SPARK-44975) BinaryArithmetic with useless override resolved field

2023-08-25 Thread Jia Fan (Jira)
Jia Fan created SPARK-44975: --- Summary: BinaryArithmetic with useless override resolved field Key: SPARK-44975 URL: https://issues.apache.org/jira/browse/SPARK-44975 Project: Spark Issue Type:

[jira] [Created] (SPARK-44869) Add doc for insert by name statement

2023-08-18 Thread Jia Fan (Jira)
Jia Fan created SPARK-44869: --- Summary: Add doc for insert by name statement Key: SPARK-44869 URL: https://issues.apache.org/jira/browse/SPARK-44869 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-42132) DeduplicateRelations rule breaks plan when co-grouping the same DataFrame

2023-08-10 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan resolved SPARK-42132. - Fix Version/s: 3.5.0 Resolution: Fixed > DeduplicateRelations rule breaks plan when co-grouping

[jira] [Updated] (SPARK-44754) Improve DeduplicateRelations rewriteAttrs compatibility

2023-08-10 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-44754: Description: {{Follow [https://github.com/apache/spark/pull/41554,] we should add test for }}

[jira] [Created] (SPARK-44754) Improve DeduplicateRelations rewriteAttrs compatibility

2023-08-10 Thread Jia Fan (Jira)
Jia Fan created SPARK-44754: --- Summary: Improve DeduplicateRelations rewriteAttrs compatibility Key: SPARK-44754 URL: https://issues.apache.org/jira/browse/SPARK-44754 Project: Spark Issue Type:

[jira] [Updated] (SPARK-44685) Remove deprecated Catalog#createExternalTable

2023-08-05 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-44685: Docs Text: The deprecated methods `createExternalTable` have been removed. Use `createTable` instead. >

[jira] [Updated] (SPARK-44685) Remove deprecated Catalog#createExternalTable

2023-08-05 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-44685: Labels: release-notes (was: ) > Remove deprecated Catalog#createExternalTable >

[jira] [Updated] (SPARK-44685) Remove deprecated Catalog#createExternalTable

2023-08-05 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-44685: Affects Version/s: 4.0.0 (was: 3.5.0) > Remove deprecated

[jira] [Created] (SPARK-44685) Remove deprecated Catalog#createExternalTable

2023-08-04 Thread Jia Fan (Jira)
Jia Fan created SPARK-44685: --- Summary: Remove deprecated Catalog#createExternalTable Key: SPARK-44685 URL: https://issues.apache.org/jira/browse/SPARK-44685 Project: Spark Issue Type: Improvement

[jira] [Closed] (SPARK-44668) ObjectMapper are threadsafe, we can reuse it in Object

2023-08-03 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan closed SPARK-44668. --- > ObjectMapper are threadsafe, we can reuse it in Object >

[jira] [Resolved] (SPARK-44668) ObjectMapper are threadsafe, we can reuse it in Object

2023-08-03 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan resolved SPARK-44668. - Resolution: Invalid > ObjectMapper are threadsafe, we can reuse it in Object >

[jira] [Created] (SPARK-44668) ObjectMapper are threadsafe, we can reuse it in Object

2023-08-03 Thread Jia Fan (Jira)
Jia Fan created SPARK-44668: --- Summary: ObjectMapper are threadsafe, we can reuse it in Object Key: SPARK-44668 URL: https://issues.apache.org/jira/browse/SPARK-44668 Project: Spark Issue Type:

[jira] [Updated] (SPARK-41636) DataSourceStrategy#selectFilters returns predicates in non-deterministic order

2023-08-02 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-41636: Affects Version/s: 3.4.1 3.4.0 > DataSourceStrategy#selectFilters returns

[jira] [Commented] (SPARK-44577) INSERT BY NAME returns non-sensical error message

2023-07-28 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748772#comment-17748772 ] Jia Fan commented on SPARK-44577: - https://github.com/apache/spark/pull/42220 > INSERT BY NAME returns

[jira] [Commented] (SPARK-44577) INSERT BY NAME returns non-sensical error message

2023-07-28 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748769#comment-17748769 ] Jia Fan commented on SPARK-44577: - Let me fix this, thanks for report. > INSERT BY NAME returns

[jira] [Commented] (SPARK-42118) Wrong result when parsing a multiline JSON file with differing types for same column

2023-07-20 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17745345#comment-17745345 ] Jia Fan commented on SPARK-42118: - Seem like already fixed on master branch. > Wrong result when

[jira] [Resolved] (SPARK-43778) RewriteCorrelatedScalarSubquery should handle duplicate attributes

2023-07-19 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan resolved SPARK-43778. - Fix Version/s: 4.0.0 Resolution: Fixed > RewriteCorrelatedScalarSubquery should handle duplicate

[jira] [Comment Edited] (SPARK-43778) RewriteCorrelatedScalarSubquery should handle duplicate attributes

2023-07-19 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744875#comment-17744875 ] Jia Fan edited comment on SPARK-43778 at 7/20/23 5:07 AM: -- This ticket already

[jira] [Commented] (SPARK-43778) RewriteCorrelatedScalarSubquery should handle duplicate attributes

2023-07-19 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744875#comment-17744875 ] Jia Fan commented on SPARK-43778: - This ticket already fixed by

[jira] [Created] (SPARK-44487) KubernetesSuite report NPE when not set spark.kubernetes.test.unpackSparkDir

2023-07-19 Thread Jia Fan (Jira)
Jia Fan created SPARK-44487: --- Summary: KubernetesSuite report NPE when not set spark.kubernetes.test.unpackSparkDir Key: SPARK-44487 URL: https://issues.apache.org/jira/browse/SPARK-44487 Project: Spark

[jira] [Resolved] (SPARK-44428) Add test case for all PartitionEvaluator API

2023-07-17 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan resolved SPARK-44428. - Resolution: Not A Problem > Add test case for all PartitionEvaluator API >

[jira] [Commented] (SPARK-44443) Use PartitionEvaluator API in CoGroupExec, DeserializeToObjectExec, ExternalRDDScanExec

2023-07-15 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743353#comment-17743353 ] Jia Fan commented on SPARK-3: - I'm working on it. > Use PartitionEvaluator API in CoGroupExec,

[jira] [Created] (SPARK-44443) Use PartitionEvaluator API in CoGroupExec, DeserializeToObjectExec, ExternalRDDScanExec

2023-07-15 Thread Jia Fan (Jira)
Jia Fan created SPARK-3: --- Summary: Use PartitionEvaluator API in CoGroupExec, DeserializeToObjectExec, ExternalRDDScanExec Key: SPARK-3 URL: https://issues.apache.org/jira/browse/SPARK-3

[jira] [Created] (SPARK-44428) Add test case for all PartitionEvaluator API

2023-07-14 Thread Jia Fan (Jira)
Jia Fan created SPARK-44428: --- Summary: Add test case for all PartitionEvaluator API Key: SPARK-44428 URL: https://issues.apache.org/jira/browse/SPARK-44428 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-44427) Use PartitionEvaluator API in MapElementsExec, MapGroupsExec, MapPartitionsExec

2023-07-14 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743119#comment-17743119 ] Jia Fan commented on SPARK-44427: - I'm working on this. > Use PartitionEvaluator API in

[jira] [Created] (SPARK-44427) Use PartitionEvaluator API in MapElementsExec, MapGroupsExec, MapPartitionsExec

2023-07-14 Thread Jia Fan (Jira)
Jia Fan created SPARK-44427: --- Summary: Use PartitionEvaluator API in MapElementsExec, MapGroupsExec, MapPartitionsExec Key: SPARK-44427 URL: https://issues.apache.org/jira/browse/SPARK-44427 Project: Spark

[jira] [Commented] (SPARK-44162) Support G1GC in `spark.eventLog.gcMetrics.*` without warning

2023-07-14 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743112#comment-17743112 ] Jia Fan commented on SPARK-44162: - https://github.com/apache/spark/pull/41808 > Support G1GC in

[jira] [Commented] (SPARK-44385) Use PartitionEvaluator API in MergingSessionsExec & UpdatingSessionsExec

2023-07-12 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742402#comment-17742402 ] Jia Fan commented on SPARK-44385: - I'm working on it. > Use PartitionEvaluator API in

[jira] [Commented] (SPARK-44386) Use PartitionEvaluator API in HashAggregateExec, ObjectHashAggregateExec, SortAggregateExec

2023-07-12 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742403#comment-17742403 ] Jia Fan commented on SPARK-44386: - I'm working on it. > Use PartitionEvaluator API in

[jira] [Created] (SPARK-44386) Use PartitionEvaluator API in HashAggregateExec, ObjectHashAggregateExec, SortAggregateExec

2023-07-12 Thread Jia Fan (Jira)
Jia Fan created SPARK-44386: --- Summary: Use PartitionEvaluator API in HashAggregateExec, ObjectHashAggregateExec, SortAggregateExec Key: SPARK-44386 URL: https://issues.apache.org/jira/browse/SPARK-44386

[jira] [Created] (SPARK-44385) Use PartitionEvaluator API in MergingSessionsExec & UpdatingSessionsExec

2023-07-12 Thread Jia Fan (Jira)
Jia Fan created SPARK-44385: --- Summary: Use PartitionEvaluator API in MergingSessionsExec & UpdatingSessionsExec Key: SPARK-44385 URL: https://issues.apache.org/jira/browse/SPARK-44385 Project: Spark

[jira] [Created] (SPARK-44375) Use PartitionEvaluator API in DebugExec

2023-07-11 Thread Jia Fan (Jira)
Jia Fan created SPARK-44375: --- Summary: Use PartitionEvaluator API in DebugExec Key: SPARK-44375 URL: https://issues.apache.org/jira/browse/SPARK-44375 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-44375) Use PartitionEvaluator API in DebugExec

2023-07-11 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742004#comment-17742004 ] Jia Fan commented on SPARK-44375: - I'm working on it. > Use PartitionEvaluator API in DebugExec >

[jira] [Created] (SPARK-44370) Migrate Buf remote generation alpha to remote plugins

2023-07-11 Thread Jia Fan (Jira)
Jia Fan created SPARK-44370: --- Summary: Migrate Buf remote generation alpha to remote plugins Key: SPARK-44370 URL: https://issues.apache.org/jira/browse/SPARK-44370 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-44262) JdbcUtils hardcodes some SQL statements

2023-07-03 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739515#comment-17739515 ] Jia Fan commented on SPARK-44262: - I got it. I will try to create a PR for this. Thanks for your

[jira] [Commented] (SPARK-44262) JdbcUtils hardcodes some SQL statements

2023-07-03 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739501#comment-17739501 ] Jia Fan commented on SPARK-44262: - Maybe you should check

[jira] [Commented] (SPARK-44262) JdbcUtils hardcodes some SQL statements

2023-07-01 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739339#comment-17739339 ] Jia Fan commented on SPARK-44262: - why not use datasoure v2? > JdbcUtils hardcodes some SQL statements

[jira] [Created] (SPARK-44268) Add tests to ensure error-classes.json and docs are in sync

2023-07-01 Thread Jia Fan (Jira)
Jia Fan created SPARK-44268: --- Summary: Add tests to ensure error-classes.json and docs are in sync Key: SPARK-44268 URL: https://issues.apache.org/jira/browse/SPARK-44268 Project: Spark Issue

[jira] [Created] (SPARK-44236) Even `spark.sql.codegen.factoryMode` is NO_CODEGEN, the WholeStageCodegen also will be generated.

2023-06-28 Thread Jia Fan (Jira)
Jia Fan created SPARK-44236: --- Summary: Even `spark.sql.codegen.factoryMode` is NO_CODEGEN, the WholeStageCodegen also will be generated. Key: SPARK-44236 URL: https://issues.apache.org/jira/browse/SPARK-44236

[jira] [Commented] (SPARK-43999) Data is still fetched even though result was returned

2023-06-26 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737421#comment-17737421 ] Jia Fan commented on SPARK-43999: - https://github.com/apache/spark/pull/41755 > Data is still fetched

[jira] [Comment Edited] (SPARK-43999) Data is still fetched even though result was returned

2023-06-26 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737026#comment-17737026 ] Jia Fan edited comment on SPARK-43999 at 6/26/23 10:50 AM: --- The reason are

[jira] [Commented] (SPARK-43999) Data is still fetched even though result was returned

2023-06-26 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737045#comment-17737045 ] Jia Fan commented on SPARK-43999: - > When do you plan to fix this behaviour?   I'm doing this. > Data

[jira] [Commented] (SPARK-43999) Data is still fetched even though result was returned

2023-06-26 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737026#comment-17737026 ] Jia Fan commented on SPARK-43999: - The reason are when AQE on, the small empty table alway return faster

[jira] [Created] (SPARK-44188) Remove useless code `resetAllPartitions` in ActiveJob

2023-06-26 Thread Jia Fan (Jira)
Jia Fan created SPARK-44188: --- Summary: Remove useless code `resetAllPartitions` in ActiveJob Key: SPARK-44188 URL: https://issues.apache.org/jira/browse/SPARK-44188 Project: Spark Issue Type:

[jira] [Commented] (SPARK-43201) Inconsistency between from_avro and from_json function

2023-06-15 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733281#comment-17733281 ] Jia Fan commented on SPARK-43201: - If avroSchema1 not equals avroSchema2, the dataframe's schema would

[jira] [Commented] (SPARK-43486) number of files read is incorrect if it is bucket table

2023-06-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17732320#comment-17732320 ] Jia Fan commented on SPARK-43486: - I didn't reproduce it too.:( > number of files read is incorrect if

[jira] [Commented] (SPARK-43486) number of files read is incorrect if it is bucket table

2023-06-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17732045#comment-17732045 ] Jia Fan commented on SPARK-43486: - [~panbingkun] Hi, any update for this? > number of files read is

[jira] [Commented] (SPARK-43891) Support SHOW VIEWS IN . when not is not the current selected catalog

2023-06-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17732031#comment-17732031 ] Jia Fan commented on SPARK-43891: - cc [~cloud_fan] [~dongjoon]  > Support SHOW VIEWS IN . when not is

[jira] (SPARK-43891) Support SHOW VIEWS IN . when not is not the current selected catalog

2023-06-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43891 ] Jia Fan deleted comment on SPARK-43891: - was (Author: fanjia): I can work for this. > Support SHOW VIEWS IN . when not is not the > current selected catalog >

[jira] [Commented] (SPARK-43891) Support SHOW VIEWS IN . when not is not the current selected catalog

2023-06-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17732030#comment-17732030 ] Jia Fan commented on SPARK-43891: - Hi [~amaliujia] , I have a question about view, I find Spark add

[jira] [Commented] (SPARK-43753) Incorrect result of MINUS in spark sql.

2023-06-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17732012#comment-17732012 ] Jia Fan commented on SPARK-43753: - Seem already fixed on the master branch. > Incorrect result of MINUS

[jira] [Commented] (SPARK-43891) Support SHOW VIEWS IN . when not is not the current selected catalog

2023-06-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17732006#comment-17732006 ] Jia Fan commented on SPARK-43891: - I can work for this. > Support SHOW VIEWS IN . when not is not the

[jira] [Updated] (SPARK-43781) IllegalStateException when cogrouping two datasets derived from the same source

2023-06-12 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-43781: Affects Version/s: 3.4.0 > IllegalStateException when cogrouping two datasets derived from the same >

[jira] [Commented] (SPARK-42290) Spark Driver hangs on OOM during Broadcast when AQE is enabled

2023-06-11 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731331#comment-17731331 ] Jia Fan commented on SPARK-42290: - Thanks [~dongjoon]  > Spark Driver hangs on OOM during Broadcast

[jira] [Updated] (SPARK-40637) Spark-shell can correctly encode BINARY type but Spark-sql cannot

2023-06-08 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-40637: Affects Version/s: 3.4.0 > Spark-shell can correctly encode BINARY type but Spark-sql cannot >

[jira] [Commented] (SPARK-42290) Spark Driver hangs on OOM during Broadcast when AQE is enabled

2023-06-08 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17730775#comment-17730775 ] Jia Fan commented on SPARK-42290: - [~dongjoon] Seem the assigning people not right. > Spark Driver

[jira] [Commented] (SPARK-43203) Fix DROP table behavior in session catalog

2023-05-30 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17727795#comment-17727795 ] Jia Fan commented on SPARK-43203: - https://github.com/apache/spark/pull/41348 > Fix DROP table behavior

[jira] [Commented] (SPARK-43521) Support CREATE TABLE LIKE FILE for PARQUET

2023-05-30 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17727793#comment-17727793 ] Jia Fan commented on SPARK-43521: - https://github.com/apache/spark/pull/41251 > Support CREATE TABLE

[jira] [Commented] (SPARK-43838) Subquery on single table with having clause can't be optimized

2023-05-30 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17727794#comment-17727794 ] Jia Fan commented on SPARK-43838: - https://github.com/apache/spark/pull/41347 > Subquery on single

[jira] [Created] (SPARK-43882) Assign a name to the error class _LEGACY_ERROR_TEMP_2122

2023-05-30 Thread Jia Fan (Jira)
Jia Fan created SPARK-43882: --- Summary: Assign a name to the error class _LEGACY_ERROR_TEMP_2122 Key: SPARK-43882 URL: https://issues.apache.org/jira/browse/SPARK-43882 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-43805) Support SELECT * EXCEPT AND SELECT * REPLACE

2023-05-28 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726627#comment-17726627 ] Jia Fan edited comment on SPARK-43805 at 5/29/23 12:55 AM: --- Tell the truth, I

[jira] [Created] (SPARK-43838) Subquery on single table with having clause can't be optimized

2023-05-27 Thread Jia Fan (Jira)
Jia Fan created SPARK-43838: --- Summary: Subquery on single table with having clause can't be optimized Key: SPARK-43838 URL: https://issues.apache.org/jira/browse/SPARK-43838 Project: Spark Issue

[jira] [Commented] (SPARK-43805) Support SELECT * EXCEPT AND SELECT * REPLACE

2023-05-26 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726627#comment-17726627 ] Jia Fan commented on SPARK-43805: - Tell the truth, I'm don't know will spark accept this statement? It

[jira] [Commented] (SPARK-43338) Support modify the SESSION_CATALOG_NAME value

2023-05-22 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724939#comment-17724939 ] Jia Fan commented on SPARK-43338: - `Assign each hms a unique catalogname only so that the meta tableId

[jira] [Commented] (SPARK-37351) Supports write data flow control

2023-05-22 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724934#comment-17724934 ] Jia Fan commented on SPARK-37351: - Do you want data flow control on micro-batch or batch mode?  This is

[jira] [Commented] (SPARK-43338) Support modify the SESSION_CATALOG_NAME value

2023-05-22 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724927#comment-17724927 ] Jia Fan commented on SPARK-43338: - `If the same hive database has parquet and hudi table, does

[jira] [Comment Edited] (SPARK-43521) Support CREATE TABLE LIKE FILE for PARQUET

2023-05-18 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723900#comment-17723900 ] Jia Fan edited comment on SPARK-43521 at 5/19/23 1:31 AM: -- I'm working on this!

[jira] [Commented] (SPARK-43521) Support CREATE TABLE LIKE FILE for PARQUET

2023-05-18 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723900#comment-17723900 ] Jia Fan commented on SPARK-43521: - I'm working for this! > Support CREATE TABLE LIKE FILE for PARQUET >

[jira] [Commented] (SPARK-43488) bitmap function

2023-05-17 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723772#comment-17723772 ] Jia Fan commented on SPARK-43488: - Hi, [~cloud_fan] If we want achieve this feature. Are we should

[jira] [Commented] (SPARK-43522) Creating struct column occurs error 'org.apache.spark.sql.AnalysisException [DATATYPE_MISMATCH.CREATE_NAMED_STRUCT_WITHOUT_FOLDABLE_STRING]'

2023-05-16 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723158#comment-17723158 ] Jia Fan commented on SPARK-43522: - https://github.com/apache/spark/pull/41187 > Creating struct column

[jira] [Updated] (SPARK-40189) Support json_array_get function

2023-05-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-40189: Description: presto provides  json_array_get function,frequently used

[jira] [Updated] (SPARK-40189) Support json_array_get function

2023-05-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-40189: Summary: Support json_array_get function (was: Support json_array_get/json_array_length function) >

[jira] [Commented] (SPARK-40189) Support json_array_get/json_array_length function

2023-05-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722443#comment-17722443 ] Jia Fan commented on SPARK-40189: - Already have json_array_length function in SPARK-31008. > Support

[jira] (SPARK-43492) Define the DATE_ADD and DATE_DIFF functions with 3-args

2023-05-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43492 ] Jia Fan deleted comment on SPARK-43492: - was (Author: fanjia): I can fix this. > Define the DATE_ADD and DATE_DIFF functions with 3-args >

[jira] [Commented] (SPARK-43492) Define the DATE_ADD and DATE_DIFF functions with 3-args

2023-05-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722439#comment-17722439 ] Jia Fan commented on SPARK-43492: - I can fix this. > Define the DATE_ADD and DATE_DIFF functions with

[jira] [Updated] (SPARK-41401) spark3 stagedir can't be change

2023-05-13 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-41401: Summary: spark3 stagedir can't be change (was: spark2 stagedir can't be change ) > spark3 stagedir

[jira] [Commented] (SPARK-40129) Decimal multiply can produce the wrong answer because it rounds twice

2023-05-12 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1774#comment-1774 ] Jia Fan commented on SPARK-40129: - https://github.com/apache/spark/pull/41156 > Decimal multiply can

[jira] [Commented] (SPARK-43267) Support creating data frame from a Postgres table that contains user-defined array column

2023-05-12 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722029#comment-17722029 ] Jia Fan commented on SPARK-43267: - https://github.com/apache/spark/pull/40953 > Support creating data

[jira] [Commented] (SPARK-39420) Support ANALYZE TABLE on v2 tables

2023-05-12 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722027#comment-17722027 ] Jia Fan commented on SPARK-39420: - https://github.com/apache/spark/pull/4 > Support ANALYZE TABLE

[jira] [Created] (SPARK-43443) Add benchmark for Timestamp type inference when use invalid value

2023-05-10 Thread Jia Fan (Jira)
Jia Fan created SPARK-43443: --- Summary: Add benchmark for Timestamp type inference when use invalid value Key: SPARK-43443 URL: https://issues.apache.org/jira/browse/SPARK-43443 Project: Spark

  1   2   >