[jira] [Created] (SPARK-43078) Separate test into `pyspark-conenct-pandas` and `pyspark-connect-pandas-slow`

Haejoon Lee created SPARK-43078: --- Summary: Separate test into `pyspark-conenct-pandas` and `pyspark-connect-pandas-slow` Key: SPARK-43078 URL: https://issues.apache.org/jira/browse/SPARK-43078 Project:

[jira] [Assigned] (SPARK-43065) Set job description for tpcds queries

[ https://issues.apache.org/jira/browse/SPARK-43065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43065: Assignee: caican > Set job description for tpcds queries > --

[jira] [Resolved] (SPARK-43065) Set job description for tpcds queries

[ https://issues.apache.org/jira/browse/SPARK-43065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43065. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40700 [https://gi

[jira] [Resolved] (SPARK-43057) Migrate Spark Connect Column errors into error class

[ https://issues.apache.org/jira/browse/SPARK-43057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43057. -- Assignee: Haejoon Lee Resolution: Fixed Fixed in https://github.com/apache/spark/pull/40

[jira] [Resolved] (SPARK-43059) Migrate TypeError from DataFrame(Reader|Writer) into error class

[ https://issues.apache.org/jira/browse/SPARK-43059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43059. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40706 [https://gi

[jira] [Assigned] (SPARK-43059) Migrate TypeError from DataFrame(Reader|Writer) into error class

[ https://issues.apache.org/jira/browse/SPARK-43059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43059: Assignee: Haejoon Lee > Migrate TypeError from DataFrame(Reader|Writer) into error class

[jira] [Resolved] (SPARK-43056) RocksDB state store commit should continue background work in finally only if its paused

[ https://issues.apache.org/jira/browse/SPARK-43056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-43056. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40696 [https://gi

[jira] [Assigned] (SPARK-43056) RocksDB state store commit should continue background work in finally only if its paused

[ https://issues.apache.org/jira/browse/SPARK-43056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-43056: Assignee: Anish Shrigondekar > RocksDB state store commit should continue background work

[jira] [Commented] (SPARK-42948) Execution plan error, unable to obtain desired results

[ https://issues.apache.org/jira/browse/SPARK-42948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710048#comment-17710048 ] miaowang commented on SPARK-42948: -- [~gurwls223] A new image has been added. Please che

[jira] [Updated] (SPARK-42948) Execution plan error, unable to obtain desired results

[ https://issues.apache.org/jira/browse/SPARK-42948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] miaowang updated SPARK-42948: - Attachment: image-2023-04-10-11-39-06-501.png > Execution plan error, unable to obtain desired results >

[jira] [Updated] (SPARK-42948) Execution plan error, unable to obtain desired results

[ https://issues.apache.org/jira/browse/SPARK-42948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] miaowang updated SPARK-42948: - Attachment: image-2023-04-10-11-39-33-658.png > Execution plan error, unable to obtain desired results >

[jira] [Updated] (SPARK-42948) Execution plan error, unable to obtain desired results

[ https://issues.apache.org/jira/browse/SPARK-42948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] miaowang updated SPARK-42948: - Description: A jar is packaged using SparkSession to submit Spark SQL： {code:java} //SparkSession.builde

[jira] [Resolved] (SPARK-42860) Add analysed logical mode in org.apache.spark.sql.execution.ExplainMode

[ https://issues.apache.org/jira/browse/SPARK-42860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42860. -- Resolution: Won't Fix > Add analysed logical mode in org.apache.spark.sql.execution.ExplainMod

[jira] [Resolved] (SPARK-42910) Generic annotation of class attribute in abstract class is NOT initalized in inherited classes

[ https://issues.apache.org/jira/browse/SPARK-42910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42910. -- Resolution: Duplicate > Generic annotation of class attribute in abstract class is NOT initali

[jira] [Commented] (SPARK-42916) JDBCCatalog Keep Char/Varchar meta information on the read-side

[ https://issues.apache.org/jira/browse/SPARK-42916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710047#comment-17710047 ] Hyukjin Kwon commented on SPARK-42916: -- https://github.com/apache/spark/pull/40543

[jira] [Commented] (SPARK-42923) Delayed scheduling doesn’t work in some situations in local mode if different localities present in loaded files leading to tasks getting stuck

[ https://issues.apache.org/jira/browse/SPARK-42923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710046#comment-17710046 ] Hyukjin Kwon commented on SPARK-42923: -- [~dolmio] does "local mode" mean master="lo

[jira] [Commented] (SPARK-42932) Spark 3.3.2, with hadoop3, Error with java.io.IOException: Mkdirs failed to create file

[ https://issues.apache.org/jira/browse/SPARK-42932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710045#comment-17710045 ] Hyukjin Kwon commented on SPARK-42932: -- I can't reproduce with Hadoop 3. Would be g

[jira] [Commented] (SPARK-42948) Execution plan error, unable to obtain desired results

[ https://issues.apache.org/jira/browse/SPARK-42948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710044#comment-17710044 ] Hyukjin Kwon commented on SPARK-42948: -- [~muser]the last image is broken. Mind reup

[jira] [Commented] (SPARK-42950) Add exit code in SparkListenerApplicationEnd

[ https://issues.apache.org/jira/browse/SPARK-42950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710043#comment-17710043 ] Hyukjin Kwon commented on SPARK-42950: -- https://github.com/apache/spark/pull/40591

[jira] [Commented] (SPARK-42975) Cast result type to timestamp type for string +/- interval

[ https://issues.apache.org/jira/browse/SPARK-42975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710042#comment-17710042 ] Hyukjin Kwon commented on SPARK-42975: -- https://github.com/apache/spark/pull/40601

[jira] [Assigned] (SPARK-42987) Correct code highlights in SQL protobuf documentation

[ https://issues.apache.org/jira/browse/SPARK-42987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42987: Assignee: Lucas Pompeu Neves > Correct code highlights in SQL protobuf documentation > --

[jira] [Resolved] (SPARK-42987) Correct code highlights in SQL protobuf documentation

[ https://issues.apache.org/jira/browse/SPARK-42987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42987. -- Fix Version/s: 3.5.0 Resolution: Fixed Fixed in https://github.com/apache/spark/pull/40

[jira] [Commented] (SPARK-42988) Spark Sql insert into hive table dynamic partitions slow

[ https://issues.apache.org/jira/browse/SPARK-42988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710040#comment-17710040 ] Hyukjin Kwon commented on SPARK-42988: -- Would be great if there's a reproducer and

[jira] [Resolved] (SPARK-42989) When Spark is going to end support of Hadoop2

[ https://issues.apache.org/jira/browse/SPARK-42989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42989. -- Resolution: Invalid Most likely from the next Spark version (3.5.0) > When Spark is going to

[jira] [Updated] (SPARK-42987) Correct code highlights in SQL protobuf documentation

[ https://issues.apache.org/jira/browse/SPARK-42987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-42987: - Target Version/s: (was: 3.3.2) > Correct code highlights in SQL protobuf documentation > -

[jira] [Updated] (SPARK-42987) Correct code highlights in SQL protobuf documentation

[ https://issues.apache.org/jira/browse/SPARK-42987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-42987: - Fix Version/s: (was: 3.3.3) > Correct code highlights in SQL protobuf documentation > --

[jira] [Commented] (SPARK-43000) Do not cast to double type if one side is AnsiIntervalType in BinaryArithmetic

[ https://issues.apache.org/jira/browse/SPARK-43000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710038#comment-17710038 ] Hyukjin Kwon commented on SPARK-43000: -- https://github.com/apache/spark/pull/40633

[jira] [Updated] (SPARK-43001) Spark last window dont flush in append mode

[ https://issues.apache.org/jira/browse/SPARK-43001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43001: - Priority: Major (was: Critical) > Spark last window dont flush in append mode > ---

[jira] [Updated] (SPARK-43001) Spark last window dont flush in append mode

[ https://issues.apache.org/jira/browse/SPARK-43001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43001: - Component/s: Structured Streaming (was: Spark Core) > Spark last window don

[jira] [Commented] (SPARK-43012) Name based access of accumulators from tasks

[ https://issues.apache.org/jira/browse/SPARK-43012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710037#comment-17710037 ] Hyukjin Kwon commented on SPARK-43012: -- Not sure. Doesn't look to me that it's very

[jira] [Resolved] (SPARK-43029) PySpark Breaks with Pandas 2.0

[ https://issues.apache.org/jira/browse/SPARK-43029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43029. -- Resolution: Cannot Reproduce Yup, it's fixed in 3.4. > PySpark Breaks with Pandas 2.0 > -

[jira] [Commented] (SPARK-43033) Avoid task retries due to AssertNotNull checks

[ https://issues.apache.org/jira/browse/SPARK-43033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710035#comment-17710035 ] Hyukjin Kwon commented on SPARK-43033: -- https://github.com/apache/spark/pull/40707

[jira] [Commented] (SPARK-43038) Support the CBC mode by aes_encrypt()/aes_decrypt()

[ https://issues.apache.org/jira/browse/SPARK-43038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710034#comment-17710034 ] Hyukjin Kwon commented on SPARK-43038: -- https://github.com/apache/spark/pull/40704

[jira] [Commented] (SPARK-43039) Support custom fields in the file source _metadata column

[ https://issues.apache.org/jira/browse/SPARK-43039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710033#comment-17710033 ] Hyukjin Kwon commented on SPARK-43039: -- https://github.com/apache/spark/pull/40677

[jira] [Updated] (SPARK-43039) Support custom fields in the file source _metadata column

[ https://issues.apache.org/jira/browse/SPARK-43039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43039: - Target Version/s: (was: 3.5.0) > Support custom fields in the file source _metadata column > -

[jira] [Commented] (SPARK-43050) Fix construct aggregate expressions by replacing grouping functions

[ https://issues.apache.org/jira/browse/SPARK-43050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710031#comment-17710031 ] Hyukjin Kwon commented on SPARK-43050: -- https://github.com/apache/spark/pull/40685

[jira] [Updated] (SPARK-43046) Implement dropDuplicatesWithinWatermark in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-43046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43046: - Epic Link: SPARK-42938 > Implement dropDuplicatesWithinWatermark in Spark Connect >

[jira] [Resolved] (SPARK-43053) Possible logic issue

[ https://issues.apache.org/jira/browse/SPARK-43053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43053. -- Resolution: Invalid > Possible logic issue > > > Key: SPA

[jira] [Commented] (SPARK-43052) Handle stacktrace with null file name in event log

[ https://issues.apache.org/jira/browse/SPARK-43052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710030#comment-17710030 ] Hyukjin Kwon commented on SPARK-43052: -- https://github.com/apache/spark/pull/40687

[jira] [Updated] (SPARK-43060) Spark JDBC rate limitation

[ https://issues.apache.org/jira/browse/SPARK-43060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43060: - Priority: Minor (was: Major) > Spark JDBC rate limitation > -- > >

[jira] [Updated] (SPARK-43060) [SQL] Spark JDBC rate limitation

[ https://issues.apache.org/jira/browse/SPARK-43060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43060: - Component/s: SQL (was: Spark Core) > [SQL] Spark JDBC rate limitation > ---

[jira] [Updated] (SPARK-43060) Spark JDBC rate limitation

[ https://issues.apache.org/jira/browse/SPARK-43060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43060: - Summary: Spark JDBC rate limitation (was: [SQL] Spark JDBC rate limitation) > Spark JDBC rate l

[jira] [Updated] (SPARK-43060) Spark JDBC rate limitation

[ https://issues.apache.org/jira/browse/SPARK-43060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43060: - Issue Type: Improvement (was: Bug) > Spark JDBC rate limitation > -- >

[jira] [Commented] (SPARK-43061) Introduce TaskEvaluator for SQL operator execution

[ https://issues.apache.org/jira/browse/SPARK-43061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710029#comment-17710029 ] Hyukjin Kwon commented on SPARK-43061: -- https://github.com/apache/spark/pull/40697

[jira] [Commented] (SPARK-43063) `df.show` handle null should print NULL instead of null

[ https://issues.apache.org/jira/browse/SPARK-43063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710028#comment-17710028 ] Hyukjin Kwon commented on SPARK-43063: -- https://github.com/apache/spark/pull/40699

[jira] [Commented] (SPARK-43065) Set job description for tpcds queries

[ https://issues.apache.org/jira/browse/SPARK-43065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710026#comment-17710026 ] Hyukjin Kwon commented on SPARK-43065: -- https://github.com/apache/spark/pull/40700

[jira] [Commented] (SPARK-43064) Spark SQL CLI SQL tab should only show once statement once

[ https://issues.apache.org/jira/browse/SPARK-43064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710027#comment-17710027 ] Hyukjin Kwon commented on SPARK-43064: -- https://github.com/apache/spark/pull/40701

[jira] [Resolved] (SPARK-43074) Add the function without constant parameters of `SessionState#executePlan`

[ https://issues.apache.org/jira/browse/SPARK-43074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43074. -- Resolution: Won't Fix This isn't an API. > Add the function without constant parameters of `S

[jira] [Created] (SPARK-43077) Improve the error message of UNRECOGNIZED_SQL_TYPE

Kent Yao created SPARK-43077: Summary: Improve the error message of UNRECOGNIZED_SQL_TYPE Key: SPARK-43077 URL: https://issues.apache.org/jira/browse/SPARK-43077 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-43072) Include TIMESTAMP_NTZ in ANSI Compliance doc

[ https://issues.apache.org/jira/browse/SPARK-43072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43072. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40711 [https://gi

[jira] [Updated] (SPARK-43072) Include TIMESTAMP_NTZ in ANSI Compliance doc

[ https://issues.apache.org/jira/browse/SPARK-43072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43072: - Fix Version/s: 3.4.1 (was: 3.4.0) > Include TIMESTAMP_NTZ in ANSI Complia

[jira] [Resolved] (SPARK-43073) Add proto types constants

[ https://issues.apache.org/jira/browse/SPARK-43073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43073. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40712 [https://

[jira] [Assigned] (SPARK-43073) Add proto types constants

[ https://issues.apache.org/jira/browse/SPARK-43073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43073: - Assignee: Ruifeng Zheng > Add proto types constants > - > >

[jira] [Commented] (SPARK-43076) Removing the dependency on `grpcio` when remote session is not used.

[ https://issues.apache.org/jira/browse/SPARK-43076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710008#comment-17710008 ] Haejoon Lee commented on SPARK-43076: - I'm working on it > Removing the dependency

[jira] [Created] (SPARK-43076) Removing the dependency on `grpcio` when remote session is not used.

Haejoon Lee created SPARK-43076: --- Summary: Removing the dependency on `grpcio` when remote session is not used. Key: SPARK-43076 URL: https://issues.apache.org/jira/browse/SPARK-43076 Project: Spark