[jira] [Updated] (SPARK-34463) toPandas failed with error: buffer source array is read-only when Arrow with self-destruct is enabled

[ https://issues.apache.org/jira/browse/SPARK-34463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34463: - Affects Version/s: (was: 3.0.2) 3.2.0 > toPandas failed with error:

[jira] [Commented] (SPARK-34463) toPandas failed with error: buffer source array is read-only when Arrow with self-destruct is enabled

[ https://issues.apache.org/jira/browse/SPARK-34463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287863#comment-17287863 ] Hyukjin Kwon commented on SPARK-34463: -- I think it only fails in the master branch because

[jira] [Updated] (SPARK-34463) toPandas failed with error: buffer source array is read-only when Arrow with self-destruct is enabled

[ https://issues.apache.org/jira/browse/SPARK-34463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34463: - Summary: toPandas failed with error: buffer source array is read-only when Arrow with

[jira] [Updated] (SPARK-34463) toPandas failed with error: buffer source array is read-only

[ https://issues.apache.org/jira/browse/SPARK-34463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34463: - Description: Environment: apache/spark master pandas version > 1.0.5 Reproduce code:

[jira] [Created] (SPARK-34489) my personal blog

Prabhu ram created SPARK-34489: -- Summary: my personal blog Key: SPARK-34489 URL: https://issues.apache.org/jira/browse/SPARK-34489 Project: Spark Issue Type: Blog - New Blog Request

[jira] [Updated] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages

[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-26399: --- Summary: Define query parameters to support various filtering conditions in REST API for overall stages

[jira] [Updated] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages

[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-26399: --- Description: [~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for endpoint

[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34488: --- Summary: Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a

[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage

[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34488: --- Description: For a specific stage, it is useful to show the task metrics in percentile distribution.  This

[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage

[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34488: --- Description: For a specific stage, it is useful to show the task metrics in percentile distribution.  This

[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage

[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34488: --- Description: For a specific stage, it is useful to show the task metrics in percentile distribution.  This

[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage

[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34488: --- Attachment: executorMetricsDistributions.json > Support task Metrics Distributions and executor Metrics

[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage

[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34488: --- Attachment: taskMetricsDistributions.json > Support task Metrics Distributions and executor Metrics

[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage

[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34488: --- Description: For a specific stage, it is useful to show the task metrics in percentile distribution.  This

[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage

[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34488: --- Description: For a specific stage, it is useful to show the task metrics in percentile distribution. This

[jira] [Created] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage

Ron Hu created SPARK-34488: -- Summary: Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage Key: SPARK-34488 URL:

[jira] [Updated] (SPARK-34092) support filtering by task status in REST API call for a specific stage

[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34092: --- Summary: support filtering by task status in REST API call for a specific stage (was: Stage level RestFul

[jira] [Updated] (SPARK-34092) Stage level RestFul API support filter by task status

[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34092: --- Description: Query parameter taskStatus can be used to filter the tasks meeting a specific status in the

[jira] [Updated] (SPARK-34092) Stage level RestFul API support filter by task status

[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34092: --- Description: Query parameter taskStatus can be used to filter the tasks meeting a specific status in the

[jira] [Assigned] (SPARK-20977) NPE in CollectionAccumulator

[ https://issues.apache.org/jira/browse/SPARK-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-20977: Assignee: Gera Shegalov > NPE in CollectionAccumulator > >

[jira] [Resolved] (SPARK-20977) NPE in CollectionAccumulator

[ https://issues.apache.org/jira/browse/SPARK-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-20977. -- Fix Version/s: 3.1.2 3.2.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-34487) K8s integration test should use the runtime Hadoop Version

[ https://issues.apache.org/jira/browse/SPARK-34487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34487: Assignee: (was: Apache Spark) > K8s integration test should use the runtime Hadoop

[jira] [Assigned] (SPARK-34487) K8s integration test should use the runtime Hadoop Version

[ https://issues.apache.org/jira/browse/SPARK-34487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34487: Assignee: Apache Spark > K8s integration test should use the runtime Hadoop Version >

[jira] [Commented] (SPARK-34487) K8s integration test should use the runtime Hadoop Version

[ https://issues.apache.org/jira/browse/SPARK-34487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287842#comment-17287842 ] Apache Spark commented on SPARK-34487: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-34487) K8s integration test should use the runtime Hadoop Version

[ https://issues.apache.org/jira/browse/SPARK-34487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34487: -- Issue Type: Bug (was: Improvement) > K8s integration test should use the runtime Hadoop

[jira] [Created] (SPARK-34487) K8s integration test should use the runtime Hadoop Version

Dongjoon Hyun created SPARK-34487: - Summary: K8s integration test should use the runtime Hadoop Version Key: SPARK-34487 URL: https://issues.apache.org/jira/browse/SPARK-34487 Project: Spark

[jira] [Resolved] (SPARK-34481) Refactor dataframe reader/writer path option logic

[ https://issues.apache.org/jira/browse/SPARK-34481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34481. --- Fix Version/s: 3.2.0 Assignee: Yuchen Huo Resolution: Fixed This is

[jira] [Commented] (SPARK-34029) Add OrcEncryptionSuite and FakeKeyProvider

[ https://issues.apache.org/jira/browse/SPARK-34029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287828#comment-17287828 ] Apache Spark commented on SPARK-34029: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-34029) Add OrcEncryptionSuite and FakeKeyProvider

[ https://issues.apache.org/jira/browse/SPARK-34029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287827#comment-17287827 ] Apache Spark commented on SPARK-34029: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-34486) Upgrade kubernetes-client to 4.13.2

[ https://issues.apache.org/jira/browse/SPARK-34486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287766#comment-17287766 ] Apache Spark commented on SPARK-34486: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-34486) Upgrade kubernetes-client to 4.13.2

[ https://issues.apache.org/jira/browse/SPARK-34486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287765#comment-17287765 ] Apache Spark commented on SPARK-34486: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-34486) Upgrade kubernetes-client to 4.13.2

[ https://issues.apache.org/jira/browse/SPARK-34486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34486: Assignee: Apache Spark > Upgrade kubernetes-client to 4.13.2 >

[jira] [Assigned] (SPARK-34486) Upgrade kubernetes-client to 4.13.2

[ https://issues.apache.org/jira/browse/SPARK-34486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34486: Assignee: (was: Apache Spark) > Upgrade kubernetes-client to 4.13.2 >

[jira] [Created] (SPARK-34486) Upgrade kubernetes-client to 4.13.2

Dongjoon Hyun created SPARK-34486: - Summary: Upgrade kubernetes-client to 4.13.2 Key: SPARK-34486 URL: https://issues.apache.org/jira/browse/SPARK-34486 Project: Spark Issue Type:

[jira] [Commented] (SPARK-7768) Make user-defined type (UDT) API public

[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287739#comment-17287739 ] Patrick Grandjean commented on SPARK-7768: -- Awesome! Thank you!!! > Make user-defined type

[jira] [Updated] (SPARK-34485) GeneratedIterator grows beyond 64 KB in Spark Structured Streaming application

[ https://issues.apache.org/jira/browse/SPARK-34485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kim Min Woo updated SPARK-34485: Description: I ran the following Spark Structured Streaming application, but failed with  ERROR

[jira] [Updated] (SPARK-34485) GeneratedIterator grows beyond 64 KB in Spark Structured Streaming application

[ https://issues.apache.org/jira/browse/SPARK-34485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kim Min Woo updated SPARK-34485: Description: I ran the following Spark Structured Streaming application, but failed with  ERROR

[jira] [Created] (SPARK-34485) GeneratedIterator grows beyond 64 KB in Spark Structured Streaming application

Kim Min Woo created SPARK-34485: --- Summary: GeneratedIterator grows beyond 64 KB in Spark Structured Streaming application Key: SPARK-34485 URL: https://issues.apache.org/jira/browse/SPARK-34485

[jira] [Commented] (SPARK-34484) Introduce a new syntax to represent attributes with the Catalyst DSL

[ https://issues.apache.org/jira/browse/SPARK-34484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287709#comment-17287709 ] Apache Spark commented on SPARK-34484: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34484) Introduce a new syntax to represent attributes with the Catalyst DSL

[ https://issues.apache.org/jira/browse/SPARK-34484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34484: Assignee: Apache Spark (was: Kousuke Saruta) > Introduce a new syntax to represent

[jira] [Assigned] (SPARK-34484) Introduce a new syntax to represent attributes with the Catalyst DSL

[ https://issues.apache.org/jira/browse/SPARK-34484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34484: Assignee: Kousuke Saruta (was: Apache Spark) > Introduce a new syntax to represent

[jira] [Commented] (SPARK-34484) Introduce a new syntax to represent attributes with the Catalyst DSL

[ https://issues.apache.org/jira/browse/SPARK-34484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287708#comment-17287708 ] Apache Spark commented on SPARK-34484: -- User 'sarutak' has created a pull request for this issue:

[jira] [Updated] (SPARK-34484) Introduce a new syntax to represent attributes with the Catalyst DSL

[ https://issues.apache.org/jira/browse/SPARK-34484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-34484: --- Summary: Introduce a new syntax to represent attributes with the Catalyst DSL (was:

[jira] [Created] (SPARK-34484) Introduce a new syntax attr() to represent attributes with Catalyst DSL

Kousuke Saruta created SPARK-34484: -- Summary: Introduce a new syntax attr() to represent attributes with Catalyst DSL Key: SPARK-34484 URL: https://issues.apache.org/jira/browse/SPARK-34484 Project:

[jira] [Updated] (SPARK-34484) Introduce a new syntax to represent attributes with Catalyst DSL

[ https://issues.apache.org/jira/browse/SPARK-34484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-34484: --- Summary: Introduce a new syntax to represent attributes with Catalyst DSL (was: Introduce

[jira] [Resolved] (SPARK-34379) Map JDBC RowID to StringType rather than LongType

[ https://issues.apache.org/jira/browse/SPARK-34379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-34379. Fix Version/s: 3.2.0 Resolution: Fixed This issue is resolved by

[jira] [Assigned] (SPARK-7768) Make user-defined type (UDT) API public

[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-7768: --- Assignee: Sean R. Owen > Make user-defined type (UDT) API public >

[jira] [Resolved] (SPARK-7768) Make user-defined type (UDT) API public

[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-7768. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31461

[jira] [Commented] (SPARK-34479) Add zstandard codec to spark.sql.avro.compression.codec

[ https://issues.apache.org/jira/browse/SPARK-34479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287656#comment-17287656 ] Yuming Wang commented on SPARK-34479: - cc [~dongjoon] > Add zstandard codec to

[jira] [Updated] (SPARK-34447) Refactor the unified v1 and v2 command tests

[ https://issues.apache.org/jira/browse/SPARK-34447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-34447: --- Description: The ticket aims to gather potential improvements for the unified tests. 1. Remove

[jira] [Commented] (SPARK-34479) Add zstandard codec to spark.sql.avro.compression.codec

[ https://issues.apache.org/jira/browse/SPARK-34479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287639#comment-17287639 ] Ismaël Mejía commented on SPARK-34479: -- Argh a pity that we cannot overwrite the version it was me

[jira] [Commented] (SPARK-34483) Make check point compress codec configurable

[ https://issues.apache.org/jira/browse/SPARK-34483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287637#comment-17287637 ] angerszhu commented on SPARK-34483: --- Make a pr soon   > Make check point compress codec configurable

[jira] [Created] (SPARK-34483) Make check point compress codec configurable

angerszhu created SPARK-34483: - Summary: Make check point compress codec configurable Key: SPARK-34483 URL: https://issues.apache.org/jira/browse/SPARK-34483 Project: Spark Issue Type:

[jira] [Commented] (SPARK-34482) Correct the active SparkSession for streaming query

[ https://issues.apache.org/jira/browse/SPARK-34482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287597#comment-17287597 ] Apache Spark commented on SPARK-34482: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34482) Correct the active SparkSession for streaming query

[ https://issues.apache.org/jira/browse/SPARK-34482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34482: Assignee: Apache Spark > Correct the active SparkSession for streaming query >

[jira] [Assigned] (SPARK-34482) Correct the active SparkSession for streaming query

[ https://issues.apache.org/jira/browse/SPARK-34482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34482: Assignee: (was: Apache Spark) > Correct the active SparkSession for streaming query

[jira] [Updated] (SPARK-34482) Correct the active SparkSession for streaming query

[ https://issues.apache.org/jira/browse/SPARK-34482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-34482: - Description:

[jira] [Created] (SPARK-34482) Correct the active SparkSession for streaming query

wuyi created SPARK-34482: Summary: Correct the active SparkSession for streaming query Key: SPARK-34482 URL: https://issues.apache.org/jira/browse/SPARK-34482 Project: Spark Issue Type: Improvement