[jira] [Created] (SPARK-43234) Migrate ValueError from Connect DataFrame into error class

Haejoon Lee created SPARK-43234: --- Summary: Migrate ValueError from Connect DataFrame into error class Key: SPARK-43234 URL: https://issues.apache.org/jira/browse/SPARK-43234 Project: Spark

[jira] [Assigned] (SPARK-43082) Arrow-optimized Python UDFs in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-43082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43082: Assignee: Xinrong Meng > Arrow-optimized Python UDFs in Spark Connect >

[jira] [Resolved] (SPARK-43082) Arrow-optimized Python UDFs in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-43082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43082. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40725

[jira] [Resolved] (SPARK-43046) Implement dropDuplicatesWithinWatermark in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-43046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43046. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40834

[jira] [Assigned] (SPARK-43174) Fix SparkSQLCLIDriver completer

[ https://issues.apache.org/jira/browse/SPARK-43174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-43174: --- Assignee: Yuming Wang > Fix SparkSQLCLIDriver completer > ---

[jira] [Resolved] (SPARK-43174) Fix SparkSQLCLIDriver completer

[ https://issues.apache.org/jira/browse/SPARK-43174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-43174. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40838

[jira] [Commented] (SPARK-38114) Spark build fails in Windows

[ https://issues.apache.org/jira/browse/SPARK-38114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715180#comment-17715180 ] Felipe commented on SPARK-38114: Hi, this seems a big issue. Anybody found a workaround? > Spark build

[jira] [Updated] (SPARK-43206) Connect Better StreamingQueryException

[ https://issues.apache.org/jira/browse/SPARK-43206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-43206: Summary: Connect Better StreamingQueryException (was: Streaming query exception() also include stack

[jira] [Commented] (SPARK-43206) Connect Better StreamingQueryException

[ https://issues.apache.org/jira/browse/SPARK-43206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715178#comment-17715178 ] Wei Liu commented on SPARK-43206: - Also cause, offsets, stack trace...   > Connect Better

[jira] [Updated] (SPARK-43233) Before batch reading from Kafka, log topic partition, offset range, etc, for debuggin

[ https://issues.apache.org/jira/browse/SPARK-43233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-43233: Issue Type: Task (was: Improvement) > Before batch reading from Kafka, log topic partition,

[jira] [Created] (SPARK-43233) Before batch reading from Kafka, log topic partition, offset range, etc, for debuggin

Siying Dong created SPARK-43233: --- Summary: Before batch reading from Kafka, log topic partition, offset range, etc, for debuggin Key: SPARK-43233 URL: https://issues.apache.org/jira/browse/SPARK-43233

[jira] [Commented] (SPARK-43206) Streaming query exception() also include stack trace

[ https://issues.apache.org/jira/browse/SPARK-43206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715171#comment-17715171 ] Wei Liu commented on SPARK-43206: - I'll work on this. To myself: don't forget jvm exceptions   >

[jira] [Commented] (SPARK-43032) Add StreamingQueryManager API

[ https://issues.apache.org/jira/browse/SPARK-43032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715147#comment-17715147 ] Wei Liu commented on SPARK-43032: - [https://github.com/apache/spark/pull/40861] still draft   > Add

[jira] [Commented] (SPARK-43143) Scala: Add StreamingQuery awaitTermination() API

[ https://issues.apache.org/jira/browse/SPARK-43143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715149#comment-17715149 ] Wei Liu commented on SPARK-43143: - I'm working on this   > Scala: Add StreamingQuery

[jira] [Commented] (SPARK-43134) Add streaming query exception API in Scala

[ https://issues.apache.org/jira/browse/SPARK-43134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715148#comment-17715148 ] Wei Liu commented on SPARK-43134: - I'm working on this   > Add streaming query exception API in Scala

[jira] [Assigned] (SPARK-43179) Add option for applications to control saving of metadata in the External Shuffle Service LevelDB

[ https://issues.apache.org/jira/browse/SPARK-43179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-43179: --- Assignee: Chandni Singh > Add option for applications to control saving of

[jira] [Resolved] (SPARK-43179) Add option for applications to control saving of metadata in the External Shuffle Service LevelDB

[ https://issues.apache.org/jira/browse/SPARK-43179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-43179. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request

[jira] [Updated] (SPARK-43142) DSL expressions fail on attribute with special characters

[ https://issues.apache.org/jira/browse/SPARK-43142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-43142: Fix Version/s: (was: 3.5.0) > DSL expressions fail on attribute with special characters >

[jira] [Reopened] (SPARK-43142) DSL expressions fail on attribute with special characters

[ https://issues.apache.org/jira/browse/SPARK-43142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reopened SPARK-43142: - > DSL expressions fail on attribute with special characters >

[jira] [Commented] (SPARK-43196) Replace reflection w/ direct calling for `ContainerLaunchContext#setTokensConf`

[ https://issues.apache.org/jira/browse/SPARK-43196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715030#comment-17715030 ] Ignite TC Bot commented on SPARK-43196: --- User 'pan3793' has created a pull request for this issue:

[jira] [Commented] (SPARK-42330) Assign name to _LEGACY_ERROR_TEMP_2175

[ https://issues.apache.org/jira/browse/SPARK-42330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714992#comment-17714992 ] Koray Beyaz commented on SPARK-42330: - Working on this issue > Assign name to

[jira] [Assigned] (SPARK-43142) DSL expressions fail on attribute with special characters

[ https://issues.apache.org/jira/browse/SPARK-43142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43142: --- Assignee: Willi Raschkowski > DSL expressions fail on attribute with special characters >

[jira] [Resolved] (SPARK-43142) DSL expressions fail on attribute with special characters

[ https://issues.apache.org/jira/browse/SPARK-43142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43142. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40794

[jira] [Commented] (SPARK-42780) Upgrade google Tink to 1.9.0

[ https://issues.apache.org/jira/browse/SPARK-42780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714971#comment-17714971 ] Bjørn Jørgensen commented on SPARK-42780: - https://github.com/apache/spark/pull/40878 > Upgrade

[jira] [Updated] (SPARK-42780) Upgrade google Tink to 1.9.0

[ https://issues.apache.org/jira/browse/SPARK-42780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bjørn Jørgensen updated SPARK-42780: Summary: Upgrade google Tink to 1.9.0 (was: Upgrade google Tink from 1.7.0 to 1.8.0) >

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance

[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Description: The `ObjectHashAggregateExec` has two preformance issues: - heavy overhead of scala

[jira] [Created] (SPARK-43232) Improve ObjectHashAggregateExec performance

XiDuo You created SPARK-43232: - Summary: Improve ObjectHashAggregateExec performance Key: SPARK-43232 URL: https://issues.apache.org/jira/browse/SPARK-43232 Project: Spark Issue Type:

[jira] [Commented] (SPARK-43199) Make InlineCTE idempotent

[ https://issues.apache.org/jira/browse/SPARK-43199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714910#comment-17714910 ] ASF GitHub Bot commented on SPARK-43199: User 'peter-toth' has created a pull request for this

[jira] [Created] (SPARK-43231) Reduce the memory requirement in torch-related tests

Ruifeng Zheng created SPARK-43231: - Summary: Reduce the memory requirement in torch-related tests Key: SPARK-43231 URL: https://issues.apache.org/jira/browse/SPARK-43231 Project: Spark Issue

[jira] [Commented] (SPARK-43228) Join keys also match PartitioningCollection

[ https://issues.apache.org/jira/browse/SPARK-43228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714909#comment-17714909 ] ASF GitHub Bot commented on SPARK-43228: User 'wangyum' has created a pull request for this

[jira] [Commented] (SPARK-43228) Join keys also match PartitioningCollection

[ https://issues.apache.org/jira/browse/SPARK-43228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714907#comment-17714907 ] Yuming Wang commented on SPARK-43228: - https://github.com/apache/spark/pull/40897 > Join keys also

[jira] [Commented] (SPARK-43230) Simplify `DataFrameNaFunctions.fillna`

[ https://issues.apache.org/jira/browse/SPARK-43230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714906#comment-17714906 ] ASF GitHub Bot commented on SPARK-43230: User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-43230) Simplify `DataFrameNaFunctions.fillna`

Ruifeng Zheng created SPARK-43230: - Summary: Simplify `DataFrameNaFunctions.fillna` Key: SPARK-43230 URL: https://issues.apache.org/jira/browse/SPARK-43230 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-43128) Streaming progress struct (especially in Scala)

[ https://issues.apache.org/jira/browse/SPARK-43128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714904#comment-17714904 ] ASF GitHub Bot commented on SPARK-43128: User 'bogao007' has created a pull request for this

[jira] [Commented] (SPARK-43229) Support Barrier Python UDF

[ https://issues.apache.org/jira/browse/SPARK-43229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714902#comment-17714902 ] ASF GitHub Bot commented on SPARK-43229: User 'zhengruifeng' has created a pull request for this

[jira] [Commented] (SPARK-43156) Correctness COUNT bug in correlated scalar subselect with `COUNT(*) is null`

[ https://issues.apache.org/jira/browse/SPARK-43156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714903#comment-17714903 ] ASF GitHub Bot commented on SPARK-43156: User 'Hisoka-X' has created a pull request for this

[jira] [Created] (SPARK-43229) Support Barrier Python UDF

Ruifeng Zheng created SPARK-43229: - Summary: Support Barrier Python UDF Key: SPARK-43229 URL: https://issues.apache.org/jira/browse/SPARK-43229 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-43228) Join keys also match PartitioningCollection

Yuming Wang created SPARK-43228: --- Summary: Join keys also match PartitioningCollection Key: SPARK-43228 URL: https://issues.apache.org/jira/browse/SPARK-43228 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-42288) Expose file path if reading failed

[ https://issues.apache.org/jira/browse/SPARK-42288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikaifei resolved SPARK-42288. -- Resolution: Duplicate > Expose file path if reading failed > -- > >

[jira] [Assigned] (SPARK-43192) Spark connect's user agent validations are too restrictive

[ https://issues.apache.org/jira/browse/SPARK-43192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43192: Assignee: Niranjan Jayakar > Spark connect's user agent validations are too restrictive

[jira] [Resolved] (SPARK-43192) Spark connect's user agent validations are too restrictive

[ https://issues.apache.org/jira/browse/SPARK-43192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43192. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40853

[jira] [Resolved] (SPARK-43222) Remove check of `isHadoop3`

[ https://issues.apache.org/jira/browse/SPARK-43222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43222. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40882

[jira] [Assigned] (SPARK-43222) Remove check of `isHadoop3`

[ https://issues.apache.org/jira/browse/SPARK-43222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43222: Assignee: Yang Jie > Remove check of `isHadoop3` > --- > >