[jira] [Assigned] (SPARK-50340) unwrap UDT in INSERT input query

[ https://issues.apache.org/jira/browse/SPARK-50340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-50340: --- Assignee: Wenchen Fan > unwrap UDT in INSERT input query >

[jira] [Resolved] (SPARK-50340) unwrap UDT in INSERT input query

[ https://issues.apache.org/jira/browse/SPARK-50340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-50340. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48881 [https://gith

[jira] [Updated] (SPARK-50323) Add missing schema check for createDataFrame from numpy ndarray on Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-50323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-50323: - Summary: Add missing schema check for createDataFrame from numpy ndarray on Spark Connect (was:

[jira] [Updated] (SPARK-50323) Add missing schema check for createDataFrame from numpy ndarray

[ https://issues.apache.org/jira/browse/SPARK-50323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50323: --- Labels: pull-request-available (was: ) > Add missing schema check for createDataFrame from

[jira] [Updated] (SPARK-50323) Add missing schema check for createDataFrame from numpy ndarray

[ https://issues.apache.org/jira/browse/SPARK-50323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-50323: - Summary: Add missing schema check for createDataFrame from numpy ndarray (was: Enforce schema f

[jira] [Created] (SPARK-50346) TransformWithState timers should compare expiration with less than or equal to

Neil Ramaswamy created SPARK-50346: -- Summary: TransformWithState timers should compare expiration with less than or equal to Key: SPARK-50346 URL: https://issues.apache.org/jira/browse/SPARK-50346 Pr

[jira] [Assigned] (SPARK-50335) Refine docstrings for aggregation functions - part 2

[ https://issues.apache.org/jira/browse/SPARK-50335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-50335: Assignee: Ruifeng Zheng > Refine docstrings for aggregation functions - part 2 >

[jira] [Assigned] (SPARK-50298) Implement verifySchema parameter of createDataFrame in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-50298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-50298: Assignee: Xinrong Meng > Implement verifySchema parameter of createDataFrame in Spark Con

[jira] [Resolved] (SPARK-50298) Implement verifySchema parameter of createDataFrame in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-50298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-50298. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48841 [https://gi

[jira] [Resolved] (SPARK-50328) Add a separate docker file for SparkR

[ https://issues.apache.org/jira/browse/SPARK-50328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-50328. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48859 [https://

[jira] [Resolved] (SPARK-50335) Refine docstrings for aggregation functions - part 2

[ https://issues.apache.org/jira/browse/SPARK-50335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-50335. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48877 [https://gi

[jira] [Updated] (SPARK-49566) EXTEND operator

[ https://issues.apache.org/jira/browse/SPARK-49566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel updated SPARK-49566: --- Summary: EXTEND operator (was: EXTEND + SET + DROP + AS operators) > EXTEND operator > --- > >

[jira] [Created] (SPARK-50344) AS operator

Daniel created SPARK-50344: -- Summary: AS operator Key: SPARK-50344 URL: https://issues.apache.org/jira/browse/SPARK-50344 Project: Spark Issue Type: Sub-task Components: SQL Affects Ve

[jira] [Updated] (SPARK-49566) EXTEND operator

[ https://issues.apache.org/jira/browse/SPARK-49566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel updated SPARK-49566: --- Description: The EXTEND operator computes a new expression and appends it as a new column after the existing

[jira] [Created] (SPARK-50342) SET operator

Daniel created SPARK-50342: -- Summary: SET operator Key: SPARK-50342 URL: https://issues.apache.org/jira/browse/SPARK-50342 Project: Spark Issue Type: Sub-task Components: SQL Affects V

[jira] [Created] (SPARK-50343) DROP operator

Daniel created SPARK-50343: -- Summary: DROP operator Key: SPARK-50343 URL: https://issues.apache.org/jira/browse/SPARK-50343 Project: Spark Issue Type: Sub-task Components: SQL Affects

[jira] [Commented] (SPARK-49566) EXTEND operator

[ https://issues.apache.org/jira/browse/SPARK-49566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17899284#comment-17899284 ] Daniel commented on SPARK-49566: I am updating this to just cover the EXTEND operator, s

[jira] [Created] (SPARK-50334) Extract common logic for reading PB files

BingKun Pan created SPARK-50334: --- Summary: Extract common logic for reading PB files Key: SPARK-50334 URL: https://issues.apache.org/jira/browse/SPARK-50334 Project: Spark Issue Type: Improveme

[jira] [Updated] (SPARK-50341) PySpark - Use UDS for communication between JVM and Python worker

[ https://issues.apache.org/jira/browse/SPARK-50341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50341: --- Labels: pull-request-available (was: ) > PySpark - Use UDS for communication between JVM an

[jira] [Updated] (SPARK-50340) unwrap UDT in INSERT input query

[ https://issues.apache.org/jira/browse/SPARK-50340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50340: --- Labels: pull-request-available (was: ) > unwrap UDT in INSERT input query > ---

[jira] [Created] (SPARK-50340) unwrap UDT in INSERT input query

Wenchen Fan created SPARK-50340: --- Summary: unwrap UDT in INSERT input query Key: SPARK-50340 URL: https://issues.apache.org/jira/browse/SPARK-50340 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-49907) [Connect] Support spark.ml on Connect

[ https://issues.apache.org/jira/browse/SPARK-49907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-49907: -- Assignee: Apache Spark > [Connect] Support spark.ml on Connect >

[jira] [Updated] (SPARK-50339) Enable changelog to store lineage information

[ https://issues.apache.org/jira/browse/SPARK-50339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50339: --- Labels: pull-request-available (was: ) > Enable changelog to store lineage information > --

[jira] [Created] (SPARK-50339) Enable changelog to store lineage information

Wei Liu created SPARK-50339: --- Summary: Enable changelog to store lineage information Key: SPARK-50339 URL: https://issues.apache.org/jira/browse/SPARK-50339 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-50332) migrate kafka consumer offset information in spark to new MSK cluster

[ https://issues.apache.org/jira/browse/SPARK-50332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-50332: Description: I have spark job that reads messages from kafka , using                  {{```spa

[jira] [Updated] (SPARK-50332) migrate kafka consumer offset information in spark to new MSK cluster

[ https://issues.apache.org/jira/browse/SPARK-50332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-50332: Description: I have spark job that reads messages from kafka , using                  {{```spa

[jira] [Assigned] (SPARK-49907) [Connect] Support spark.ml on Connect

[ https://issues.apache.org/jira/browse/SPARK-49907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-49907: -- Assignee: (was: Apache Spark) > [Connect] Support spark.ml on Connect > -

[jira] [Updated] (SPARK-50335) Refine docstrings for aggregation functions - part 2

[ https://issues.apache.org/jira/browse/SPARK-50335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50335: --- Labels: pull-request-available (was: ) > Refine docstrings for aggregation functions - part

[jira] [Updated] (SPARK-50336) bash -c execute command risk.

[ https://issues.apache.org/jira/browse/SPARK-50336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ShengQiangLi updated SPARK-50336: - Description: bash -c executing user input is risky, does Spark need to be guarded? For example,

[jira] [Created] (SPARK-50337) Error logged due to race condition when shutting down kubernetes client

Rocco Verhoef created SPARK-50337: - Summary: Error logged due to race condition when shutting down kubernetes client Key: SPARK-50337 URL: https://issues.apache.org/jira/browse/SPARK-50337 Project: Sp

[jira] [Created] (SPARK-50335) Refine docstrings for aggregation functions - part 2

Ruifeng Zheng created SPARK-50335: - Summary: Refine docstrings for aggregation functions - part 2 Key: SPARK-50335 URL: https://issues.apache.org/jira/browse/SPARK-50335 Project: Spark Issue

[jira] [Assigned] (SPARK-49907) [Connect] Support spark.ml on Connect

[ https://issues.apache.org/jira/browse/SPARK-49907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-49907: -- Assignee: Apache Spark > [Connect] Support spark.ml on Connect >

[jira] [Assigned] (SPARK-49907) [Connect] Support spark.ml on Connect

[ https://issues.apache.org/jira/browse/SPARK-49907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-49907: -- Assignee: (was: Apache Spark) > [Connect] Support spark.ml on Connect > -

[jira] [Assigned] (SPARK-49907) [Connect] Support spark.ml on Connect

[ https://issues.apache.org/jira/browse/SPARK-49907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-49907: -- Assignee: Apache Spark > [Connect] Support spark.ml on Connect >

[jira] [Assigned] (SPARK-49907) [Connect] Support spark.ml on Connect

[ https://issues.apache.org/jira/browse/SPARK-49907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-49907: -- Assignee: (was: Apache Spark) > [Connect] Support spark.ml on Connect > -

[jira] [Updated] (SPARK-50334) Extract common logic for reading the descriptor of PB file

[ https://issues.apache.org/jira/browse/SPARK-50334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-50334: Summary: Extract common logic for reading the descriptor of PB file (was: Extract common logic fo

[jira] [Updated] (SPARK-50334) Extract common logic for reading PB files

[ https://issues.apache.org/jira/browse/SPARK-50334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50334: --- Labels: pull-request-available (was: ) > Extract common logic for reading PB files > --