[jira] [Created] (SPARK-43978) Dropping blocks from memory to disk may result in heartbeat loss

Igor Konev created SPARK-43978: -- Summary: Dropping blocks from memory to disk may result in heartbeat loss Key: SPARK-43978 URL: https://issues.apache.org/jira/browse/SPARK-43978 Project: Spark

[jira] [Resolved] (SPARK-43973) Structured Streaming UI should display failed queries correctly

[ https://issues.apache.org/jira/browse/SPARK-43973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-43973. Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed Issue resolved by

[jira] [Assigned] (SPARK-43973) Structured Streaming UI should display failed queries correctly

[ https://issues.apache.org/jira/browse/SPARK-43973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-43973: -- Assignee: Kris Mok > Structured Streaming UI should display failed queries correctly

[jira] [Commented] (SPARK-43977) bad case of connect-jvm-client-mima-check

[ https://issues.apache.org/jira/browse/SPARK-43977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729575#comment-17729575 ] Snoot.io commented on SPARK-43977: -- User 'LuciferYang' has created a pull request for this issue:

[jira] [Created] (SPARK-43977) bad case of connect-jvm-client-mima-check

Yang Jie created SPARK-43977: Summary: bad case of connect-jvm-client-mima-check Key: SPARK-43977 URL: https://issues.apache.org/jira/browse/SPARK-43977 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-43203) Fix DROP table behavior in session catalog

[ https://issues.apache.org/jira/browse/SPARK-43203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729573#comment-17729573 ] Snoot.io commented on SPARK-43203: -- User 'Hisoka-X' has created a pull request for this issue:

[jira] [Commented] (SPARK-43935) Add xpath_* functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729572#comment-17729572 ] Snoot.io commented on SPARK-43935: -- User 'panbingkun' has created a pull request for this issue:

[jira] [Commented] (SPARK-43615) Enable DataFrameSlowParityTests.test_eval

[ https://issues.apache.org/jira/browse/SPARK-43615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729570#comment-17729570 ] Snoot.io commented on SPARK-43615: -- User 'zhengruifeng' has created a pull request for this issue:

[jira] [Commented] (SPARK-43930) Add unix_* functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729569#comment-17729569 ] Snoot.io commented on SPARK-43930: -- User 'panbingkun' has created a pull request for this issue:

[jira] [Updated] (SPARK-43911) Use toSet to deduplicate the iterator data to prevent the creation of large Array

[ https://issues.apache.org/jira/browse/SPARK-43911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-43911: Fix Version/s: (was: 3.5.0) (was: 3.4.1) > Use toSet to deduplicate

[jira] [Reopened] (SPARK-43911) Use toSet to deduplicate the iterator data to prevent the creation of large Array

[ https://issues.apache.org/jira/browse/SPARK-43911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reopened SPARK-43911: - Assignee: (was: mcdull_zhang) > Use toSet to deduplicate the iterator data to prevent the

[jira] [Created] (SPARK-43976) Handle the case where modifiedConfigs doesn't exist in event logs

Dongjoon Hyun created SPARK-43976: - Summary: Handle the case where modifiedConfigs doesn't exist in event logs Key: SPARK-43976 URL: https://issues.apache.org/jira/browse/SPARK-43976 Project: Spark

[jira] [Updated] (SPARK-41958) Disallow arbitrary custom classpath with proxy user in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-41958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41958: -- Issue Type: Bug (was: Improvement) > Disallow arbitrary custom classpath with proxy user in

[jira] [Commented] (SPARK-43933) Add linear regression aggregate functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729549#comment-17729549 ] Ruifeng Zheng commented on SPARK-43933: --- [~beliefer] many thanks, please go ahead > Add linear

[jira] [Updated] (SPARK-43933) Add linear regression aggregate functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-43933: --- Summary: Add linear regression aggregate functions to Scala and Python (was: Add regression

[jira] [Updated] (SPARK-43933) Add regression aggregate functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-43933: --- Summary: Add regression aggregate functions to Scala and Python (was: Add regr_* functions to

[jira] [Resolved] (SPARK-42626) Add Destructive Iterator for SparkResult

[ https://issues.apache.org/jira/browse/SPARK-42626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-42626. --- Fix Version/s: 3.5.0 Assignee: Tengfei Huang Resolution: Fixed >

[jira] [Commented] (SPARK-43938) Add to_* functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729539#comment-17729539 ] Ruifeng Zheng commented on SPARK-43938: --- [~panbingkun] thanks, please go ahead > Add to_*

[jira] [Commented] (SPARK-43931) Add make_* functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729538#comment-17729538 ] Ruifeng Zheng commented on SPARK-43931: --- [~panbingkun] many thanks, please go ahead > Add make_*

[jira] [Commented] (SPARK-43938) Add to_* functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729537#comment-17729537 ] BingKun Pan commented on SPARK-43938: - I work on it > Add to_* functions to Scala and Python >

[jira] [Assigned] (SPARK-43788) Enable SummarizerTests.test_summarize_dataframe for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu reassigned SPARK-43788: -- Assignee: Weichen Xu > Enable SummarizerTests.test_summarize_dataframe for pandas 2.0.0. >

[jira] [Resolved] (SPARK-43788) Enable SummarizerTests.test_summarize_dataframe for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu resolved SPARK-43788. Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41456

[jira] [Resolved] (SPARK-43784) Enable FeatureTests.test_max_abs_scaler for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu resolved SPARK-43784. Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41456

[jira] [Assigned] (SPARK-43784) Enable FeatureTests.test_max_abs_scaler for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu reassigned SPARK-43784: -- Assignee: Weichen Xu > Enable FeatureTests.test_max_abs_scaler for pandas 2.0.0. >

[jira] [Resolved] (SPARK-43783) Enable FeatureTests.test_standard_scaler for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu resolved SPARK-43783. Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41456

[jira] [Assigned] (SPARK-43783) Enable FeatureTests.test_standard_scaler for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu reassigned SPARK-43783: -- Assignee: Weichen Xu > Enable FeatureTests.test_standard_scaler for pandas 2.0.0. >

[jira] [Commented] (SPARK-41958) Disallow arbitrary custom classpath with proxy user in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-41958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729525#comment-17729525 ] Dongjoon Hyun commented on SPARK-41958: --- This is backported to branch-3.3 via

[jira] [Updated] (SPARK-41958) Disallow arbitrary custom classpath with proxy user in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-41958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41958: -- Fix Version/s: 3.3.3 > Disallow arbitrary custom classpath with proxy user in cluster mode >

[jira] [Created] (SPARK-43975) DataSource V2: Handle UPDATE commands for group-based sources

Anton Okolnychyi created SPARK-43975: Summary: DataSource V2: Handle UPDATE commands for group-based sources Key: SPARK-43975 URL: https://issues.apache.org/jira/browse/SPARK-43975 Project: Spark

[jira] [Updated] (SPARK-43975) DataSource V2: Handle UPDATE commands for group-based sources

[ https://issues.apache.org/jira/browse/SPARK-43975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-43975: - Description: We need to handle UPDATE commands for group-based sources. (was: We need

[jira] [Created] (SPARK-43974) Upgrade buf to v1.21.0

BingKun Pan created SPARK-43974: --- Summary: Upgrade buf to v1.21.0 Key: SPARK-43974 URL: https://issues.apache.org/jira/browse/SPARK-43974 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-43973) Structured Streaming UI should display failed queries correctly

Kris Mok created SPARK-43973: Summary: Structured Streaming UI should display failed queries correctly Key: SPARK-43973 URL: https://issues.apache.org/jira/browse/SPARK-43973 Project: Spark

[jira] [Commented] (SPARK-43523) Memory leak in Spark UI

[ https://issues.apache.org/jira/browse/SPARK-43523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729480#comment-17729480 ] ci-cassandra.apache.org commented on SPARK-43523: - User 'aminebag' has created a pull

[jira] [Updated] (SPARK-43972) Tests never succeed on pyspark 3.4.0 (work OK on pyspark 3.3.2)

[ https://issues.apache.org/jira/browse/SPARK-43972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jamie updated SPARK-43972: -- Description: I have a project that uses pyspark. The tests have always run fine on pyspark versions prior to

[jira] [Created] (SPARK-43972) Tests never succeed on pyspark 3.4.0 (work OK on pyspark 3.3.2)

Jamie created SPARK-43972: - Summary: Tests never succeed on pyspark 3.4.0 (work OK on pyspark 3.3.2) Key: SPARK-43972 URL: https://issues.apache.org/jira/browse/SPARK-43972 Project: Spark Issue

[jira] [Resolved] (SPARK-42299) Assign name to _LEGACY_ERROR_TEMP_2206

[ https://issues.apache.org/jira/browse/SPARK-42299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-42299. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41387

[jira] [Assigned] (SPARK-42299) Assign name to _LEGACY_ERROR_TEMP_2206

[ https://issues.apache.org/jira/browse/SPARK-42299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-42299: Assignee: Amanda Liu > Assign name to _LEGACY_ERROR_TEMP_2206 >

[jira] [Commented] (SPARK-37369) Avoid redundant ColumnarToRow transistion on InMemoryTableScan

[ https://issues.apache.org/jira/browse/SPARK-37369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729375#comment-17729375 ] Filipe Oliveira commented on SPARK-37369: - [~viirya] InMemoryTableScanExec assumes the

[jira] [Commented] (SPARK-43376) Improve reuse subquery with table cache

[ https://issues.apache.org/jira/browse/SPARK-43376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729336#comment-17729336 ] ci-cassandra.apache.org commented on SPARK-43376: - User 'ulysses-you' has created a pull

[jira] [Commented] (SPARK-43783) Enable FeatureTests.test_standard_scaler for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729335#comment-17729335 ] ci-cassandra.apache.org commented on SPARK-43783: - User 'WeichenXu123' has created a

[jira] [Created] (SPARK-43971) Support Python's createDataFrame in streaming manner

Max Gekk created SPARK-43971: Summary: Support Python's createDataFrame in streaming manner Key: SPARK-43971 URL: https://issues.apache.org/jira/browse/SPARK-43971 Project: Spark Issue Type: New

[jira] [Resolved] (SPARK-38315) Add a config to control decoding of datetime as Java 8 classes

[ https://issues.apache.org/jira/browse/SPARK-38315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-38315. -- Resolution: Won't Do > Add a config to control decoding of datetime as Java 8 classes >

[jira] [Resolved] (SPARK-39275) Pass SQL config values as parameters of error classes

[ https://issues.apache.org/jira/browse/SPARK-39275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-39275. -- Resolution: Won't Do > Pass SQL config values as parameters of error classes >

[jira] [Commented] (SPARK-43935) Add xpath_* functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729296#comment-17729296 ] BingKun Pan commented on SPARK-43935: - I work on it. > Add xpath_* functions to Scala and Python >

[jira] [Commented] (SPARK-43931) Add make_* functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729294#comment-17729294 ] BingKun Pan commented on SPARK-43931: - I work on it. > Add make_* functions to Scala and Python >

[jira] [Commented] (SPARK-43970) Hide unsupported dataframe methods from auto-completion

[ https://issues.apache.org/jira/browse/SPARK-43970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729253#comment-17729253 ] ASF GitHub Bot commented on SPARK-43970: User 'zhengruifeng' has created a pull request for this

[jira] [Commented] (SPARK-43970) Hide unsupported dataframe methods from auto-completion

[ https://issues.apache.org/jira/browse/SPARK-43970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729252#comment-17729252 ] ASF GitHub Bot commented on SPARK-43970: User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-43970) Hide unsupported dataframe methods from auto-completion

Ruifeng Zheng created SPARK-43970: - Summary: Hide unsupported dataframe methods from auto-completion Key: SPARK-43970 URL: https://issues.apache.org/jira/browse/SPARK-43970 Project: Spark

[jira] [Commented] (SPARK-43923) [CONNECT] Post listenerBus events during ExecutePlanRequest

[ https://issues.apache.org/jira/browse/SPARK-43923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729250#comment-17729250 ] ASF GitHub Bot commented on SPARK-43923: User 'jdesjean' has created a pull request for this

[jira] [Commented] (SPARK-43291) Match behavior for DataFrame.cov on string DataFrame

[ https://issues.apache.org/jira/browse/SPARK-43291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729234#comment-17729234 ] Bjørn Jørgensen commented on SPARK-43291: - [~dongjoon] FYI > Match behavior for DataFrame.cov

[jira] [Commented] (SPARK-43930) Add unix_* functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729231#comment-17729231 ] BingKun Pan commented on SPARK-43930: - I work on it. > Add unix_* functions to Scala and Python >

[jira] [Updated] (SPARK-43969) Refactor & Assign names to the error class _LEGACY_ERROR_TEMP_1170

[ https://issues.apache.org/jira/browse/SPARK-43969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-43969: Description: - Refactor `PreWriteCheck` to use error framework. - Make

[jira] [Commented] (SPARK-43242) diagnoseCorruption should not throw Unexpected type of BlockId for ShuffleBlockBatchId

[ https://issues.apache.org/jira/browse/SPARK-43242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729205#comment-17729205 ] xiangxiang Shen commented on SPARK-43242: - @cloud-fan , [~John Zhang]  [~davidonlaptop]   

[jira] [Created] (SPARK-43969) Refactor & Assign names to the error class _LEGACY_ERROR_TEMP_1170

BingKun Pan created SPARK-43969: --- Summary: Refactor & Assign names to the error class _LEGACY_ERROR_TEMP_1170 Key: SPARK-43969 URL: https://issues.apache.org/jira/browse/SPARK-43969 Project: Spark

[jira] [Commented] (SPARK-43929) Add date time functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729197#comment-17729197 ] Ruifeng Zheng commented on SPARK-43929: --- [~mich.talebza...@gmail.com] I think some already exists,

[jira] [Commented] (SPARK-43106) Data lost from the table if the INSERT OVERWRITE query fails

[ https://issues.apache.org/jira/browse/SPARK-43106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729193#comment-17729193 ] Vaibhav Beriwala commented on SPARK-43106: -- [~dongjoon] Did you get a chance to take a look at

[jira] [Created] (SPARK-43968) Add more compile-time checks when creating Python UDTFs

Allison Wang created SPARK-43968: Summary: Add more compile-time checks when creating Python UDTFs Key: SPARK-43968 URL: https://issues.apache.org/jira/browse/SPARK-43968 Project: Spark