[jira] [Commented] (SPARK-27030) DataFrameWriter.insertInto fails when writing in parallel to a hive table

[ https://issues.apache.org/jira/browse/SPARK-27030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719158#comment-17719158 ] Shrikant Prasad commented on SPARK-27030: - [~lev] [~shivuson...@gmail.com] Did you find any

[jira] [Created] (SPARK-43374) Protobuf Licensed under BSD-3 not BSD-2 clause

Kent Yao created SPARK-43374: Summary: Protobuf Licensed under BSD-3 not BSD-2 clause Key: SPARK-43374 URL: https://issues.apache.org/jira/browse/SPARK-43374 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-43051) Allow emitting zero values when deserializing protobuf messages

[ https://issues.apache.org/jira/browse/SPARK-43051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43051. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40686

[jira] [Assigned] (SPARK-43051) Allow emitting zero values when deserializing protobuf messages

[ https://issues.apache.org/jira/browse/SPARK-43051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43051: Assignee: Parth Upadhyay > Allow emitting zero values when deserializing protobuf

[jira] [Commented] (SPARK-43342) Spark in Kubernetes mode throws IllegalArgumentException when using static PVC

[ https://issues.apache.org/jira/browse/SPARK-43342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719137#comment-17719137 ] Dongjoon Hyun commented on SPARK-43342: --- BTW, thank you for reporting, [~ofrenkel] . > Spark in

[jira] [Commented] (SPARK-43342) Spark in Kubernetes mode throws IllegalArgumentException when using static PVC

[ https://issues.apache.org/jira/browse/SPARK-43342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719136#comment-17719136 ] Dongjoon Hyun commented on SPARK-43342: --- Thank you for pinging me, [~dcoliversun] . BTW, Apache

[jira] [Commented] (SPARK-43373) Revert [SPARK-39203][SQL] Rewrite table location to absolute URI based on database URI

[ https://issues.apache.org/jira/browse/SPARK-43373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719130#comment-17719130 ] Wenchen Fan commented on SPARK-43373: - https://github.com/apache/spark/pull/40871 > Revert

[jira] [Resolved] (SPARK-43373) Revert [SPARK-39203][SQL] Rewrite table location to absolute URI based on database URI

[ https://issues.apache.org/jira/browse/SPARK-43373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43373. - Fix Version/s: 3.4.1 Assignee: Wenchen Fan Resolution: Fixed > Revert

[jira] [Created] (SPARK-43373) Revert [SPARK-39203][SQL] Rewrite table location to absolute URI based on database URI

Wenchen Fan created SPARK-43373: --- Summary: Revert [SPARK-39203][SQL] Rewrite table location to absolute URI based on database URI Key: SPARK-43373 URL: https://issues.apache.org/jira/browse/SPARK-43373

[jira] [Created] (SPARK-43372) Use ; instead of && when enable set -ex

Yikun Jiang created SPARK-43372: --- Summary: Use ; instead of && when enable set -ex Key: SPARK-43372 URL: https://issues.apache.org/jira/browse/SPARK-43372 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-43365) Refactor Dockerfile and workflow based on base image

[ https://issues.apache.org/jira/browse/SPARK-43365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719129#comment-17719129 ] Yikun Jiang commented on SPARK-43365: - I'm working on this > Refactor Dockerfile and workflow based

[jira] [Assigned] (SPARK-43365) Refactor Dockerfile and workflow based on base image

[ https://issues.apache.org/jira/browse/SPARK-43365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang reassigned SPARK-43365: --- Assignee: Yikun Jiang > Refactor Dockerfile and workflow based on base image >

[jira] [Created] (SPARK-43371) minimize duplication across layers for chmod

Yikun Jiang created SPARK-43371: --- Summary: minimize duplication across layers for chmod Key: SPARK-43371 URL: https://issues.apache.org/jira/browse/SPARK-43371 Project: Spark Issue Type:

[jira] [Created] (SPARK-43370) Switch spark user only when run driver and executor

Yikun Jiang created SPARK-43370: --- Summary: Switch spark user only when run driver and executor Key: SPARK-43370 URL: https://issues.apache.org/jira/browse/SPARK-43370 Project: Spark Issue

[jira] [Created] (SPARK-43369) Address comments about /etc/pam.d/su

Yikun Jiang created SPARK-43369: --- Summary: Address comments about /etc/pam.d/su Key: SPARK-43369 URL: https://issues.apache.org/jira/browse/SPARK-43369 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-43368) Address DOI comments about /etc/passwd

Yikun Jiang created SPARK-43368: --- Summary: Address DOI comments about /etc/passwd Key: SPARK-43368 URL: https://issues.apache.org/jira/browse/SPARK-43368 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-43367) Recover sh support in Dockerfiles

Yikun Jiang created SPARK-43367: --- Summary: Recover sh support in Dockerfiles Key: SPARK-43367 URL: https://issues.apache.org/jira/browse/SPARK-43367 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-43366) Spark Driver Bind Address is off-by-one

Derek Brown created SPARK-43366: --- Summary: Spark Driver Bind Address is off-by-one Key: SPARK-43366 URL: https://issues.apache.org/jira/browse/SPARK-43366 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-43365) Refactor Dockerfile and workflow based on base image

Yikun Jiang created SPARK-43365: --- Summary: Refactor Dockerfile and workflow based on base image Key: SPARK-43365 URL: https://issues.apache.org/jira/browse/SPARK-43365 Project: Spark Issue

[jira] [Commented] (SPARK-43342) Spark in Kubernetes mode throws IllegalArgumentException when using static PVC

[ https://issues.apache.org/jira/browse/SPARK-43342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719121#comment-17719121 ] Qian Sun commented on SPARK-43342: -- [~dongjoon] [~yikunkero] It seems like a regression caused by 

[jira] [Commented] (SPARK-43364) Add docs for rocksdb state store memory mgmt to structured streaming guide

[ https://issues.apache.org/jira/browse/SPARK-43364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719103#comment-17719103 ] Anish Shrigondekar commented on SPARK-43364: PR here:

[jira] [Created] (SPARK-43364) Add docs for rocksdb state store memory mgmt to structured streaming guide

Anish Shrigondekar created SPARK-43364: -- Summary: Add docs for rocksdb state store memory mgmt to structured streaming guide Key: SPARK-43364 URL: https://issues.apache.org/jira/browse/SPARK-43364

[jira] [Created] (SPARK-43363) Remove a workaround for pandas categorical type for pyarrow

Takuya Ueshin created SPARK-43363: - Summary: Remove a workaround for pandas categorical type for pyarrow Key: SPARK-43363 URL: https://issues.apache.org/jira/browse/SPARK-43363 Project: Spark

[jira] [Created] (SPARK-43362) Special handling of JSON type for MySQL connector

Tianhan Hu created SPARK-43362: -- Summary: Special handling of JSON type for MySQL connector Key: SPARK-43362 URL: https://issues.apache.org/jira/browse/SPARK-43362 Project: Spark Issue Type:

[jira] [Updated] (SPARK-43361) Allow deserializing protobuf enum fields as integers

[ https://issues.apache.org/jira/browse/SPARK-43361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Upadhyay updated SPARK-43361: --- Description: When deserializing protobuf enum fields, the spark-protobuf library will

[jira] [Created] (SPARK-43361) Allow deserializing protobuf enum fields as integers

Parth Upadhyay created SPARK-43361: -- Summary: Allow deserializing protobuf enum fields as integers Key: SPARK-43361 URL: https://issues.apache.org/jira/browse/SPARK-43361 Project: Spark

[jira] [Created] (SPARK-43360) Scala Connect: Add StreamingQueryManager API

Wei Liu created SPARK-43360: --- Summary: Scala Connect: Add StreamingQueryManager API Key: SPARK-43360 URL: https://issues.apache.org/jira/browse/SPARK-43360 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-43359) DELETE from Hive table result in INTERNAL error

Serge Rielau created SPARK-43359: Summary: DELETE from Hive table result in INTERNAL error Key: SPARK-43359 URL: https://issues.apache.org/jira/browse/SPARK-43359 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-43319) Remove usage of deprecated DefaultKubernetesClient

[ https://issues.apache.org/jira/browse/SPARK-43319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43319. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40994

[jira] [Assigned] (SPARK-43319) Remove usage of deprecated DefaultKubernetesClient

[ https://issues.apache.org/jira/browse/SPARK-43319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43319: - Assignee: Cheng Pan > Remove usage of deprecated DefaultKubernetesClient >

[jira] [Assigned] (SPARK-43347) Remove Python 3.7 Support

[ https://issues.apache.org/jira/browse/SPARK-43347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43347: - Assignee: Dongjoon Hyun > Remove Python 3.7 Support > - > >

[jira] [Resolved] (SPARK-43347) Remove Python 3.7 Support

[ https://issues.apache.org/jira/browse/SPARK-43347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43347. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41023

[jira] [Created] (SPARK-43358) Cover the resolution of insertion with the SupportsCustomSchemaWrite interface

Daniel created SPARK-43358: -- Summary: Cover the resolution of insertion with the SupportsCustomSchemaWrite interface Key: SPARK-43358 URL: https://issues.apache.org/jira/browse/SPARK-43358 Project: Spark

[jira] [Resolved] (SPARK-43250) Assign a name to the error class _LEGACY_ERROR_TEMP_2014

[ https://issues.apache.org/jira/browse/SPARK-43250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-43250. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40951

[jira] [Assigned] (SPARK-43250) Assign a name to the error class _LEGACY_ERROR_TEMP_2014

[ https://issues.apache.org/jira/browse/SPARK-43250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-43250: Assignee: Atour Mousavi Gourabi > Assign a name to the error class _LEGACY_ERROR_TEMP_2014 >

[jira] [Resolved] (SPARK-43352) `setPythonSparkConfProperties` should use conf instead of sparkAppConf

[ https://issues.apache.org/jira/browse/SPARK-43352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43352. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41030

[jira] [Updated] (SPARK-43352) Inline DepsTestsSuite#setPythonSparkConfProperties

[ https://issues.apache.org/jira/browse/SPARK-43352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43352: -- Summary: Inline DepsTestsSuite#setPythonSparkConfProperties (was:

[jira] [Updated] (SPARK-43352) Inline DepsTestsSuite#setPythonSparkConfProperties

[ https://issues.apache.org/jira/browse/SPARK-43352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43352: -- Priority: Trivial (was: Major) > Inline DepsTestsSuite#setPythonSparkConfProperties >

[jira] [Assigned] (SPARK-43352) `setPythonSparkConfProperties` should use conf instead of sparkAppConf

[ https://issues.apache.org/jira/browse/SPARK-43352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43352: - Assignee: Cheng Pan > `setPythonSparkConfProperties` should use conf instead of

[jira] [Resolved] (SPARK-43331) Spark Connect - SparkSession interruptAll

[ https://issues.apache.org/jira/browse/SPARK-43331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-43331. --- Fix Version/s: 3.5.0 Assignee: Juliusz Sompolski Resolution: Fixed

[jira] [Resolved] (SPARK-43350) Upgrade scalafmt from 3.7.2 to 3.7.3

[ https://issues.apache.org/jira/browse/SPARK-43350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43350. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41029

[jira] [Assigned] (SPARK-43350) Upgrade scalafmt from 3.7.2 to 3.7.3

[ https://issues.apache.org/jira/browse/SPARK-43350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43350: - Assignee: BingKun Pan > Upgrade scalafmt from 3.7.2 to 3.7.3 >

[jira] [Resolved] (SPARK-43355) Upgrade fabric8io/kubernetes-client 6.6.0

[ https://issues.apache.org/jira/browse/SPARK-43355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43355. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41034

[jira] [Assigned] (SPARK-43355) Upgrade fabric8io/kubernetes-client 6.6.0

[ https://issues.apache.org/jira/browse/SPARK-43355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43355: - Assignee: Cheng Pan > Upgrade fabric8io/kubernetes-client 6.6.0 >

[jira] [Updated] (SPARK-43356) Migrate deprecated createOrReplace to serverSideApply

[ https://issues.apache.org/jira/browse/SPARK-43356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43356: -- Description:     {{public interface CreateOrReplaceable extends Replaceable {}} {{  /**}} {{   *

[jira] [Updated] (SPARK-43356) Migrate deprecated createOrReplace to serverSideApply

[ https://issues.apache.org/jira/browse/SPARK-43356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43356: -- Description:     {{public interface CreateOrReplaceable extends Replaceable {}}{{  /**}} {{   *

[jira] [Created] (SPARK-43357) Spark AWS Glue date partition push down broken

Stijn De Haes created SPARK-43357: - Summary: Spark AWS Glue date partition push down broken Key: SPARK-43357 URL: https://issues.apache.org/jira/browse/SPARK-43357 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-43348) Support Python 3.8 in PyPy3

[ https://issues.apache.org/jira/browse/SPARK-43348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43348. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41024

[jira] [Created] (SPARK-43356) Migrate deprecated createOrReplace to serverSideApply

Cheng Pan created SPARK-43356: - Summary: Migrate deprecated createOrReplace to serverSideApply Key: SPARK-43356 URL: https://issues.apache.org/jira/browse/SPARK-43356 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-43349) Fix flaky test for `DataFrame` creation

[ https://issues.apache.org/jira/browse/SPARK-43349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43349. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41025

[jira] [Assigned] (SPARK-43349) Fix flaky test for `DataFrame` creation

[ https://issues.apache.org/jira/browse/SPARK-43349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43349: Assignee: Haejoon Lee > Fix flaky test for `DataFrame` creation >

[jira] [Created] (SPARK-43355) Upgrade fabric8io/kubernetes-client 6.6.0

Cheng Pan created SPARK-43355: - Summary: Upgrade fabric8io/kubernetes-client 6.6.0 Key: SPARK-43355 URL: https://issues.apache.org/jira/browse/SPARK-43355 Project: Spark Issue Type: Dependency

[jira] [Resolved] (SPARK-42845) Assign a name to the error class _LEGACY_ERROR_TEMP_2010

[ https://issues.apache.org/jira/browse/SPARK-42845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-42845. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40817

[jira] [Assigned] (SPARK-42845) Assign a name to the error class _LEGACY_ERROR_TEMP_2010

[ https://issues.apache.org/jira/browse/SPARK-42845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-42845: Assignee: Liang Yan > Assign a name to the error class _LEGACY_ERROR_TEMP_2010 >

[jira] [Created] (SPARK-43354) Re-enable test_create_dataframe_from_pandas_with_day_time_interval

Dongjoon Hyun created SPARK-43354: - Summary: Re-enable test_create_dataframe_from_pandas_with_day_time_interval Key: SPARK-43354 URL: https://issues.apache.org/jira/browse/SPARK-43354 Project: Spark

[jira] [Commented] (SPARK-37942) Use error classes in the compilation errors of properties

[ https://issues.apache.org/jira/browse/SPARK-37942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17718857#comment-17718857 ] ASF GitHub Bot commented on SPARK-37942: User 'NarekDW' has created a pull request for this

[jira] [Commented] (SPARK-43341) StructType.toDDL does not pick up on non-nullability of column in nested struct

[ https://issues.apache.org/jira/browse/SPARK-43341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17718856#comment-17718856 ] ASF GitHub Bot commented on SPARK-43341: User 'BramBoog' has created a pull request for this

[jira] [Commented] (SPARK-16484) Incremental Cardinality estimation operations with Hyperloglog

[ https://issues.apache.org/jira/browse/SPARK-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17718855#comment-17718855 ] ASF GitHub Bot commented on SPARK-16484: User 'mkaravel' has created a pull request for this

[jira] [Commented] (SPARK-43296) Migrate Spark Connect session errors into error class

[ https://issues.apache.org/jira/browse/SPARK-43296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17718799#comment-17718799 ] Hudson commented on SPARK-43296: User 'itholic' has created a pull request for this issue:

[jira] [Commented] (SPARK-43324) DataSource V2: Handle UPDATE commands for delta-based sources

[ https://issues.apache.org/jira/browse/SPARK-43324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17718798#comment-17718798 ] Hudson commented on SPARK-43324: User 'aokolnychyi' has created a pull request for this issue: