[jira] [Updated] (SPARK-46985) Move _NoValue from pyspark.* to pyspark.sql.*

[ https://issues.apache.org/jira/browse/SPARK-46985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46985: --- Labels: pull-request-available (was: ) > Move _NoValue from pyspark.* to pyspark.sql.* > --

[jira] [Updated] (SPARK-46679) Encoders with multiple inheritance - Key not found: T

[ https://issues.apache.org/jira/browse/SPARK-46679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andoni Teso updated SPARK-46679: Affects Version/s: 4.0.0 > Encoders with multiple inheritance - Key not found: T > ---

[jira] [Updated] (SPARK-46679) Encoders with multiple inheritance - Key not found: T

[ https://issues.apache.org/jira/browse/SPARK-46679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andoni Teso updated SPARK-46679: Priority: Critical (was: Blocker) > Encoders with multiple inheritance - Key not found: T > -

[jira] [Updated] (SPARK-46984) Remove pyspark.copy_func

[ https://issues.apache.org/jira/browse/SPARK-46984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46984: - Priority: Minor (was: Major) > Remove pyspark.copy_func > > >

[jira] [Created] (SPARK-46985) Move _NoValue from pyspark.* to pyspark.sql.*

Hyukjin Kwon created SPARK-46985: Summary: Move _NoValue from pyspark.* to pyspark.sql.* Key: SPARK-46985 URL: https://issues.apache.org/jira/browse/SPARK-46985 Project: Spark Issue Type: Sub

[jira] [Updated] (SPARK-46984) Remove pyspark.copy_func

[ https://issues.apache.org/jira/browse/SPARK-46984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46984: --- Labels: pull-request-available (was: ) > Remove pyspark.copy_func > ---

[jira] [Created] (SPARK-46984) Remove pyspark.copy_func

Hyukjin Kwon created SPARK-46984: Summary: Remove pyspark.copy_func Key: SPARK-46984 URL: https://issues.apache.org/jira/browse/SPARK-46984 Project: Spark Issue Type: Sub-task Compo

[jira] [Created] (SPARK-46983) Decouple module dependencies between PySpark modules

Hyukjin Kwon created SPARK-46983: Summary: Decouple module dependencies between PySpark modules Key: SPARK-46983 URL: https://issues.apache.org/jira/browse/SPARK-46983 Project: Spark Issue Ty

[jira] [Updated] (SPARK-46170) Support inject adaptive query post planner strategy rules in SparkSessionExtensions

[ https://issues.apache.org/jira/browse/SPARK-46170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-46170: - Fix Version/s: 3.5.1 > Support inject adaptive query post planner strategy rules in > SparkSessionExten

[jira] [Updated] (SPARK-46982) Remove _LEGACY_ERROR_TEMP_2187 in favor of CANNOT_RECOGNIZE_HIVE_TYPE

[ https://issues.apache.org/jira/browse/SPARK-46982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46982: --- Labels: pull-request-available (was: ) > Remove _LEGACY_ERROR_TEMP_2187 in favor of CANNOT_

[jira] [Updated] (SPARK-46981) Driver OOM happens in query planning phase with empty tables

[ https://issues.apache.org/jira/browse/SPARK-46981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noritaka Sekiyama updated SPARK-46981: -- Description: We have observed that Driver OOM happens in query planning phase with emp

[jira] [Created] (SPARK-46982) Remove _LEGACY_ERROR_TEMP_2187 in favor of CANNOT_RECOGNIZE_HIVE_TYPE

Kent Yao created SPARK-46982: Summary: Remove _LEGACY_ERROR_TEMP_2187 in favor of CANNOT_RECOGNIZE_HIVE_TYPE Key: SPARK-46982 URL: https://issues.apache.org/jira/browse/SPARK-46982 Project: Spark

[jira] [Updated] (SPARK-46981) Driver OOM happens in query planning phase with empty tables

[ https://issues.apache.org/jira/browse/SPARK-46981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noritaka Sekiyama updated SPARK-46981: -- Description: We have observed that Driver OOM happens in query planning phase with emp

[jira] [Updated] (SPARK-46981) Driver OOM happens in query planning phase with empty tables

[ https://issues.apache.org/jira/browse/SPARK-46981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noritaka Sekiyama updated SPARK-46981: -- Attachment: test_and_twodays_simplified.sql > Driver OOM happens in query planning pha

[jira] [Created] (SPARK-46981) Driver OOM happens in query planning phase with empty tables

Noritaka Sekiyama created SPARK-46981: - Summary: Driver OOM happens in query planning phase with empty tables Key: SPARK-46981 URL: https://issues.apache.org/jira/browse/SPARK-46981 Project: Spark

[jira] [Updated] (SPARK-46981) Driver OOM happens in query planning phase with empty tables

[ https://issues.apache.org/jira/browse/SPARK-46981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noritaka Sekiyama updated SPARK-46981: -- Attachment: create_sanitized_tables.py > Driver OOM happens in query planning phase wi

[jira] [Resolved] (SPARK-46958) missing timezone to coerce default values

[ https://issues.apache.org/jira/browse/SPARK-46958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-46958. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45000 [https://github.com

[jira] [Updated] (SPARK-46934) Unable to create Hive View from certain Spark Dataframe StructType

[ https://issues.apache.org/jira/browse/SPARK-46934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46934: --- Labels: pull-request-available (was: ) > Unable to create Hive View from certain Spark Data

[jira] [Updated] (SPARK-46979) Add support for defining state encoder for key/value and col family independently

[ https://issues.apache.org/jira/browse/SPARK-46979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46979: --- Labels: pull-request-available (was: ) > Add support for defining state encoder for key/val

[jira] [Commented] (SPARK-46934) Unable to create Hive View from certain Spark Dataframe StructType

[ https://issues.apache.org/jira/browse/SPARK-46934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814588#comment-17814588 ] Kent Yao commented on SPARK-46934: -- Hi [~yutinglin], How can I create an element named

[jira] [Assigned] (SPARK-46960) Testing Multiple Input Streams for TransformWithState operator

[ https://issues.apache.org/jira/browse/SPARK-46960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-46960: Assignee: Eric Marnadi > Testing Multiple Input Streams for TransformWithState operator >

[jira] [Resolved] (SPARK-46960) Testing Multiple Input Streams for TransformWithState operator

[ https://issues.apache.org/jira/browse/SPARK-46960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-46960. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45004 [https://gi

[jira] [Updated] (SPARK-46960) Testing Multiple Input Streams for TransformWithState operator

[ https://issues.apache.org/jira/browse/SPARK-46960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46960: --- Labels: pull-request-available (was: ) > Testing Multiple Input Streams for TransformWithSt

[jira] [Updated] (SPARK-45599) Percentile can produce a wrong answer if -0.0 and 0.0 are mixed in the dataset

[ https://issues.apache.org/jira/browse/SPARK-45599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45599: --- Labels: correctness pull-request-available (was: correctness) > Percentile can produce a wr

[jira] [Updated] (SPARK-46980) Avoid using internal APIs in dataframe end-to-end tests

[ https://issues.apache.org/jira/browse/SPARK-46980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46980: --- Labels: pull-request-available (was: ) > Avoid using internal APIs in dataframe end-to-end

[jira] [Resolved] (SPARK-46980) Avoid using internal APIs in dataframe end-to-end tests

[ https://issues.apache.org/jira/browse/SPARK-46980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-46980. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45034 [https://gith

[jira] [Created] (SPARK-46980) Avoid using internal APIs in tests

Wenchen Fan created SPARK-46980: --- Summary: Avoid using internal APIs in tests Key: SPARK-46980 URL: https://issues.apache.org/jira/browse/SPARK-46980 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-46980) Avoid using internal APIs in dataframe end-to-end tests

[ https://issues.apache.org/jira/browse/SPARK-46980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-46980: Summary: Avoid using internal APIs in dataframe end-to-end tests (was: Avoid using internal APIs

[jira] [Comment Edited] (SPARK-39441) Speed up DeduplicateRelations

[ https://issues.apache.org/jira/browse/SPARK-39441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17749321#comment-17749321 ] Mitesh edited comment on SPARK-39441 at 2/5/24 11:28 PM: - After

[jira] [Created] (SPARK-46979) Add support for defining state encoder for key/value and col family independently

Anish Shrigondekar created SPARK-46979: -- Summary: Add support for defining state encoder for key/value and col family independently Key: SPARK-46979 URL: https://issues.apache.org/jira/browse/SPARK-46979

[jira] [Resolved] (SPARK-46977) A failed request to obtain a token from one NameNode should not block subsequent token requests

[ https://issues.apache.org/jira/browse/SPARK-46977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46977. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45030 [https://

[jira] [Assigned] (SPARK-46977) A failed request to obtain a token from one NameNode should not block subsequent token requests

[ https://issues.apache.org/jira/browse/SPARK-46977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46977: - Assignee: Cheng Pan > A failed request to obtain a token from one NameNode should not b

[jira] [Resolved] (SPARK-46972) Asymmetrical replacement for char/varchar in V2SessionCatalog.createTable

[ https://issues.apache.org/jira/browse/SPARK-46972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46972. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45019 [https://

[jira] [Resolved] (SPARK-46978) Refine docstring of `sum_distinct/array_agg/count_if`

[ https://issues.apache.org/jira/browse/SPARK-46978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46978. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45031 [https://

[jira] [Assigned] (SPARK-46978) Refine docstring of `sum_distinct/array_agg/count_if`

[ https://issues.apache.org/jira/browse/SPARK-46978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46978: - Assignee: Yang Jie > Refine docstring of `sum_distinct/array_agg/count_if` > --

[jira] [Comment Edited] (SPARK-39441) Speed up DeduplicateRelations

[ https://issues.apache.org/jira/browse/SPARK-39441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17749321#comment-17749321 ] Mitesh edited comment on SPARK-39441 at 2/5/24 7:01 PM: After ap

[jira] [Comment Edited] (SPARK-39441) Speed up DeduplicateRelations

[ https://issues.apache.org/jira/browse/SPARK-39441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17749321#comment-17749321 ] Mitesh edited comment on SPARK-39441 at 2/5/24 7:00 PM: After ap

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814432#comment-17814432 ] Gaétan CACACE edited comment on SPARK-46032 at 2/5/24 4:37 PM: ---

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814432#comment-17814432 ] Gaétan CACACE commented on SPARK-46032: --- Hello there,   Just coming to give some

[jira] [Assigned] (SPARK-46833) Using ICU library for collation tracking

[ https://issues.apache.org/jira/browse/SPARK-46833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-46833: --- Assignee: Aleksandar Tomic > Using ICU library for collation tracking > ---

[jira] [Resolved] (SPARK-46833) Using ICU library for collation tracking

[ https://issues.apache.org/jira/browse/SPARK-46833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-46833. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44968 [https://gith

[jira] [Commented] (SPARK-46810) Clarify error class terminology

[ https://issues.apache.org/jira/browse/SPARK-46810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814406#comment-17814406 ] Nicholas Chammas commented on SPARK-46810: -- [~cloud_fan], [~LuciferYang], [~bel

[jira] [Comment Edited] (SPARK-24815) Structured Streaming should support dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763940#comment-17763940 ] Krystal Mitchell edited comment on SPARK-24815 at 2/5/24 3:32 PM:

[jira] [Updated] (SPARK-46978) Refine docstring of `sum_distinct/array_agg/count_if`

[ https://issues.apache.org/jira/browse/SPARK-46978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46978: --- Labels: pull-request-available (was: ) > Refine docstring of `sum_distinct/array_agg/count_

[jira] [Updated] (SPARK-46977) A failed request to obtain a token from one NameNode should not block subsequent token requests

[ https://issues.apache.org/jira/browse/SPARK-46977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46977: --- Labels: pull-request-available (was: ) > A failed request to obtain a token from one NameNo

[jira] [Created] (SPARK-46978) Refine docstring of `sum_distinct/array_agg/count_if`

Yang Jie created SPARK-46978: Summary: Refine docstring of `sum_distinct/array_agg/count_if` Key: SPARK-46978 URL: https://issues.apache.org/jira/browse/SPARK-46978 Project: Spark Issue Type: Sub

[jira] [Created] (SPARK-46977) A failed request to obtain a token from one NameNode should not block subsequent token requests

Cheng Pan created SPARK-46977: - Summary: A failed request to obtain a token from one NameNode should not block subsequent token requests Key: SPARK-46977 URL: https://issues.apache.org/jira/browse/SPARK-46977

[jira] [Assigned] (SPARK-46975) Move to_{hdf, feather, stata} to the fallback list

[ https://issues.apache.org/jira/browse/SPARK-46975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46975: -- Assignee: (was: Apache Spark) > Move to_{hdf, feather, stata} to the fallback lis

[jira] [Assigned] (SPARK-46975) Move to_{hdf, feather, stata} to the fallback list

[ https://issues.apache.org/jira/browse/SPARK-46975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46975: -- Assignee: Apache Spark > Move to_{hdf, feather, stata} to the fallback list > ---

[jira] [Assigned] (SPARK-46976) Implement `DataFrameGroupBy.corr`

[ https://issues.apache.org/jira/browse/SPARK-46976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46976: -- Assignee: (was: Apache Spark) > Implement `DataFrameGroupBy.corr` > -

[jira] [Assigned] (SPARK-46976) Implement `DataFrameGroupBy.corr`

[ https://issues.apache.org/jira/browse/SPARK-46976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46976: -- Assignee: Apache Spark > Implement `DataFrameGroupBy.corr` >

[jira] [Assigned] (SPARK-46976) Implement `DataFrameGroupBy.corr`

[ https://issues.apache.org/jira/browse/SPARK-46976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46976: -- Assignee: Apache Spark > Implement `DataFrameGroupBy.corr` >

[jira] [Assigned] (SPARK-46976) Implement `DataFrameGroupBy.corr`

[ https://issues.apache.org/jira/browse/SPARK-46976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46976: -- Assignee: (was: Apache Spark) > Implement `DataFrameGroupBy.corr` > -