[jira] [Created] (SPARK-43159) Refine `column_op` to use lambda function instead of Column API.

2023-04-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-43159: --- Summary: Refine `column_op` to use lambda function instead of Column API. Key: SPARK-43159 URL: https://issues.apache.org/jira/browse/SPARK-43159 Project: Spark

[jira] [Assigned] (SPARK-43042) Spark Connect: Streaming readerwriter table() API

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43042: Assignee: Wei Liu > Spark Connect: Streaming readerwriter table() API > -

[jira] [Resolved] (SPARK-43042) Spark Connect: Streaming readerwriter table() API

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43042. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40797 [https://gi

[jira] [Resolved] (SPARK-43151) Update the prerequisites for generating Python API docs

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43151. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40804 [https://gi

[jira] [Resolved] (SPARK-43130) Move InternalType to PhysicalDataType

2023-04-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43130. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40784 [https://gith

[jira] [Commented] (SPARK-43158) Set upperbound of pandas version in binder integrations

2023-04-16 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712879#comment-17712879 ] Snoot.io commented on SPARK-43158: -- User 'HyukjinKwon' has created a pull request for t

[jira] [Updated] (SPARK-43158) Set upperbound of pandas version in binder integrations

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43158: - Fix Version/s: 3.3.3 > Set upperbound of pandas version in binder integrations > ---

[jira] [Commented] (SPARK-43122) Reenable TorchDistributorLocalUnitTestsOnConnect and TorchDistributorLocalUnitTestsIIOnConnect

2023-04-16 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712876#comment-17712876 ] Snoot.io commented on SPARK-43122: -- User 'zhengruifeng' has created a pull request for

[jira] [Assigned] (SPARK-43099) `Class.getCanonicalName` return null for anonymous class on JDK15+, impacting function registry

2023-04-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43099: --- Assignee: Alex Jing > `Class.getCanonicalName` return null for anonymous class on JDK15+, i

[jira] [Commented] (SPARK-43140) Override computeStats in DummyLeafNode

2023-04-16 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712869#comment-17712869 ] Snoot.io commented on SPARK-43140: -- User 'wangyum' has created a pull request for this

[jira] [Resolved] (SPARK-43099) `Class.getCanonicalName` return null for anonymous class on JDK15+, impacting function registry

2023-04-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43099. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40747 [https://gith

[jira] [Assigned] (SPARK-43140) Override computeStats in DummyLeafNode

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43140: Assignee: Yuming Wang > Override computeStats in DummyLeafNode >

[jira] [Resolved] (SPARK-43140) Override computeStats in DummyLeafNode

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43140. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40791 [https://gi

[jira] [Assigned] (SPARK-43147) Python lint local config

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43147: Assignee: Wei Liu > Python lint local config > > >

[jira] [Resolved] (SPARK-43147) Python lint local config

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43147. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40801 [https://gi

[jira] [Updated] (SPARK-43141) Ignore generated Java files in checkstyle

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43141: - Fix Version/s: 3.4.1 > Ignore generated Java files in checkstyle > -

[jira] [Commented] (SPARK-43099) `Class.getCanonicalName` return null for anonymous class on JDK15+, impacting function registry

2023-04-16 Thread GridGain Integration (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712855#comment-17712855 ] GridGain Integration commented on SPARK-43099: -- User 'alexjinghn' has creat

[jira] [Updated] (SPARK-43158) Set upperbound of pandas version in binder integrations

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43158: - Fix Version/s: 3.4.0 (was: 3.4.1) > Set upperbound of pandas version in b

[jira] [Resolved] (SPARK-43158) Set upperbound of pandas version in binder integrations

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43158. -- Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-43158) Set upperbound of pandas version in binder integrations

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43158: - Summary: Set upperbound of pandas version in binder integrations (was: Set upperbound of pandas

[jira] [Updated] (SPARK-43158) Set upperbound of pandas version in binder integrations

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43158: - Component/s: Documentation (was: Pandas API on Spark) (

[jira] [Assigned] (SPARK-43158) Set upperbound of pandas version in binder integartions

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43158: Assignee: Hyukjin Kwon > Set upperbound of pandas version in binder integartions > --

[jira] [Updated] (SPARK-43158) Set upperbound of pandas version in binder integartions

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43158: - Summary: Set upperbound of pandas version in binder integartions (was: Set lowerbound of pandas

[jira] [Created] (SPARK-43158) Set lowerbound of pandas version in 3.4.0

2023-04-16 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-43158: Summary: Set lowerbound of pandas version in 3.4.0 Key: SPARK-43158 URL: https://issues.apache.org/jira/browse/SPARK-43158 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-43157) TreeNode tags can become corrupted and hang driver when the dataset is cached

2023-04-16 Thread Rob Reeves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Reeves updated SPARK-43157: --- Description: If a cached dataset is used by multiple other datasets materialized in separate thread

[jira] [Updated] (SPARK-43157) TreeNode tags can become corrupted and hang driver when the dataset is cached

2023-04-16 Thread Rob Reeves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Reeves updated SPARK-43157: --- Description: If a cached dataset is used by multiple other datasets materialized in separate thread

[jira] [Updated] (SPARK-43157) TreeNode tags can become corrupted and hang driver when the dataset is cached

2023-04-16 Thread Rob Reeves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Reeves updated SPARK-43157: --- Description: If a cached dataset is used by multiple other datasets materialized in separate thread

[jira] [Updated] (SPARK-43157) TreeNode tags can become corrupted and hang driver when the dataset is cached

2023-04-16 Thread Rob Reeves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Reeves updated SPARK-43157: --- Description: If a cached dataset is used by multiple other datasets materialized in separate thread

[jira] [Created] (SPARK-43157) TreeNode tags can become corrupted and hang driver when the dataset is cached

2023-04-16 Thread Rob Reeves (Jira)
Rob Reeves created SPARK-43157: -- Summary: TreeNode tags can become corrupted and hang driver when the dataset is cached Key: SPARK-43157 URL: https://issues.apache.org/jira/browse/SPARK-43157 Project: Sp

[jira] [Updated] (SPARK-43156) Correctness COUNT bug in correlated scalar subselect with `COUNT(*) is null`

2023-04-16 Thread Jack Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Chen updated SPARK-43156: -- Description: Example query: {code:java} spark.sql("select *, (select (count(1)) is null from t1 where

[jira] [Updated] (SPARK-43156) Correctness COUNT bug in correlated scalar subselect with `COUNT(*) is null`

2023-04-16 Thread Jack Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Chen updated SPARK-43156: -- Description: Example query: {code:java} spark.sql("select *, (select (count(1)) is null from t1 where

[jira] [Created] (SPARK-43156) Correctness COUNT bug in correlated scalar subselect with `COUNT(*) is null`

2023-04-16 Thread Jack Chen (Jira)
Jack Chen created SPARK-43156: - Summary: Correctness COUNT bug in correlated scalar subselect with `COUNT(*) is null` Key: SPARK-43156 URL: https://issues.apache.org/jira/browse/SPARK-43156 Project: Spark

[jira] [Commented] (SPARK-42923) Delayed scheduling doesn’t work in some situations in local mode if different localities present in loaded files leading to tasks getting stuck

2023-04-16 Thread Juho Salmio (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712831#comment-17712831 ] Juho Salmio commented on SPARK-42923: - [~gurwls223]: yeah > Delayed scheduling does

[jira] [Updated] (SPARK-43155) DataSourceV2 is hard to be implemented without following V1

2023-04-16 Thread PEIYUAN SUN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PEIYUAN SUN updated SPARK-43155: Description: h1. Description The current interface of DataSourceV2 becomes overly complicated tha

[jira] [Updated] (SPARK-43155) DataSourceV2 is hard to be implemented without following V1

2023-04-16 Thread PEIYUAN SUN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PEIYUAN SUN updated SPARK-43155: Description: h1. Description The current interface of DataSourceV2 becomes overly complicated tha

[jira] [Updated] (SPARK-43155) DataSourceV2 is hard to be implemented without following V1

2023-04-16 Thread PEIYUAN SUN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PEIYUAN SUN updated SPARK-43155: Description: h1. Description The current interface of DataSourceV2 becomes overly complicated tha

[jira] [Created] (SPARK-43155) DataSourceV2 is hard to be implemented without following V1

2023-04-16 Thread PEIYUAN SUN (Jira)
PEIYUAN SUN created SPARK-43155: --- Summary: DataSourceV2 is hard to be implemented without following V1 Key: SPARK-43155 URL: https://issues.apache.org/jira/browse/SPARK-43155 Project: Spark Is

[jira] [Commented] (SPARK-43138) ClassNotFoundException during RDD block replication/migration

2023-04-16 Thread Emil Ejbyfeldt (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712781#comment-17712781 ] Emil Ejbyfeldt commented on SPARK-43138: No. The class `com.class.from.user.jar.

[jira] [Commented] (SPARK-43154) Pyspark 3.4 fails when running "pivot" function on a dataframe using the values arguement

2023-04-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712774#comment-17712774 ] Yuming Wang commented on SPARK-43154: - cc [~gurwls223] > Pyspark 3.4 fails when run

[jira] [Assigned] (SPARK-43141) Ignore generated Java files in checkstyle

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43141: Assignee: Hyukjin Kwon > Ignore generated Java files in checkstyle >

[jira] [Resolved] (SPARK-43141) Ignore generated Java files in checkstyle

2023-04-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43141. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40792 [https://gi

[jira] [Created] (SPARK-43154) Pyspark 3.4 fails when running "pivot" function on a dataframe using the values arguement

2023-04-16 Thread Ofri Kleinfeld (Jira)
Ofri Kleinfeld created SPARK-43154: -- Summary: Pyspark 3.4 fails when running "pivot" function on a dataframe using the values arguement Key: SPARK-43154 URL: https://issues.apache.org/jira/browse/SPARK-43154

[jira] [Resolved] (SPARK-43139) Bug in INSERT INTO documentation

2023-04-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-43139. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed Issue resolved by pull re

[jira] [Assigned] (SPARK-43139) Bug in INSERT INTO documentation

2023-04-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-43139: --- Assignee: Yuming Wang > Bug in INSERT INTO documentation >