Weichen Xu created SPARK-52470:
--
Summary: Support `model.summary` when model is evicted from Spark
driver memory
Key: SPARK-52470
URL: https://issues.apache.org/jira/browse/SPARK-52470
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-52470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-52470:
--
Assignee: Weichen Xu
> Support `model.summary` when model is evicted from Spark driver memory
[
https://issues.apache.org/jira/browse/SPARK-52259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-52259.
Fix Version/s: 4.1.0
4.0.1
Resolution: Fixed
Issue resolved by pull requ
[
https://issues.apache.org/jira/browse/SPARK-52259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-52259:
--
Assignee: Weichen Xu
> Fix Param class binary compatibility
> ---
Weichen Xu created SPARK-52259:
--
Summary: Fix Param class binary compatibility
Key: SPARK-52259
URL: https://issues.apache.org/jira/browse/SPARK-52259
Project: Spark
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/SPARK-52229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-52229.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50955
[https://github
[
https://issues.apache.org/jira/browse/SPARK-52229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-52229:
--
Assignee: Weichen Xu
> Improve model size estimation
> -
>
>
[
https://issues.apache.org/jira/browse/SPARK-52229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-52229:
---
Description:
Improve model size estimation:
For models which contain `modelSummary`, we shouldn
[
https://issues.apache.org/jira/browse/SPARK-52192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-52192.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50923
[https://github
[
https://issues.apache.org/jira/browse/SPARK-52192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-52192:
--
Assignee: Weichen Xu
> MLCache loading path check
> --
>
>
[
https://issues.apache.org/jira/browse/SPARK-52191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-52191.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50922
[https://github
[
https://issues.apache.org/jira/browse/SPARK-52191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-52191:
--
Assignee: Weichen Xu
> Remove Java deserializer in model local path loader
>
Weichen Xu created SPARK-52192:
--
Summary: MLCache loading path check
Key: SPARK-52192
URL: https://issues.apache.org/jira/browse/SPARK-52192
Project: Spark
Issue Type: Sub-task
Compone
[
https://issues.apache.org/jira/browse/SPARK-52191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-52191:
---
Summary: Remove Java deserializer in model local path loader (was: Remove
Java deserializer in mode
[
https://issues.apache.org/jira/browse/SPARK-52191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-52191:
---
Description:
Remove Java deserializer in model local path loader.
Java deserializer is unsafe, remo
Weichen Xu created SPARK-52191:
--
Summary: Remove Java deserializer in model reader from local path
Key: SPARK-52191
URL: https://issues.apache.org/jira/browse/SPARK-52191
Project: Spark
Issue Ty
[
https://issues.apache.org/jira/browse/SPARK-52122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-52122:
--
Assignee: Weichen Xu
> Fix DefaultParamsReader RCE vulnerability
> --
[
https://issues.apache.org/jira/browse/SPARK-52122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-52122.
Fix Version/s: 4.1.0
4.0.0
Resolution: Fixed
Issue resolved by pull requ
[
https://issues.apache.org/jira/browse/SPARK-52130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-52130.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50894
[https://github
[
https://issues.apache.org/jira/browse/SPARK-52130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-52130:
--
Assignee: Weichen Xu
> Refine error message, and hide internal spark config
> ---
Weichen Xu created SPARK-52130:
--
Summary: Refine error message, and hide internal spark config
Key: SPARK-52130
URL: https://issues.apache.org/jira/browse/SPARK-52130
Project: Spark
Issue Type:
Weichen Xu created SPARK-52122:
--
Summary: Fix DefaultParamsReader RCE vulnerability
Key: SPARK-52122
URL: https://issues.apache.org/jira/browse/SPARK-52122
Project: Spark
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/SPARK-52057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-52057.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50848
[https://github
[
https://issues.apache.org/jira/browse/SPARK-52057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-52057:
--
Assignee: Weichen Xu
> Collect Tree size limit warning messages to client
> -
[
https://issues.apache.org/jira/browse/SPARK-52051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-52051:
--
Assignee: Weichen Xu
> Enable model summary when memory control is enabled
>
[
https://issues.apache.org/jira/browse/SPARK-52051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-52051.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50843
[https://github
Weichen Xu created SPARK-52057:
--
Summary: Collect Tree size limit warning messages to client
Key: SPARK-52057
URL: https://issues.apache.org/jira/browse/SPARK-52057
Project: Spark
Issue Type: Su
[
https://issues.apache.org/jira/browse/SPARK-52056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-52056.
Resolution: Not A Problem
it is already disabled by default
> Disable tuning algorithm sub-models
Weichen Xu created SPARK-52056:
--
Summary: Disable tuning algorithm sub-models in spark connect
Key: SPARK-52056
URL: https://issues.apache.org/jira/browse/SPARK-52056
Project: Spark
Issue Type:
Weichen Xu created SPARK-52051:
--
Summary: Enable model summary when memory control is enabled
Key: SPARK-52051
URL: https://issues.apache.org/jira/browse/SPARK-52051
Project: Spark
Issue Type: S
[
https://issues.apache.org/jira/browse/SPARK-52013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-52013.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50832
[https://github
[
https://issues.apache.org/jira/browse/SPARK-52013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-52013:
--
Assignee: Weichen Xu
> Fix SparkConnectClient ml_caches
>
>
[
https://issues.apache.org/jira/browse/SPARK-51473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17950265#comment-17950265
]
Weichen Xu edited comment on SPARK-51473 at 5/8/25 2:32 PM:
[
https://issues.apache.org/jira/browse/SPARK-51473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reopened SPARK-51473:
> ML transformed dataframe keep a reference to the model
> ---
[
https://issues.apache.org/jira/browse/SPARK-51473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17950265#comment-17950265
]
Weichen Xu commented on SPARK-51473:
I found an issue:
{code:java}
model_a = esti
[
https://issues.apache.org/jira/browse/SPARK-51974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-51974.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50751
[https://github
[
https://issues.apache.org/jira/browse/SPARK-51974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-51974:
--
Assignee: Weichen Xu
> Limit model size and per-session model cache size
> --
Weichen Xu created SPARK-52013:
--
Summary: Fix SparkConnectClient ml_caches
Key: SPARK-52013
URL: https://issues.apache.org/jira/browse/SPARK-52013
Project: Spark
Issue Type: Sub-task
C
[
https://issues.apache.org/jira/browse/SPARK-51947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-51947.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50752
[https://github
Weichen Xu created SPARK-51974:
--
Summary: Limit model size and per-session model cache size
Key: SPARK-51974
URL: https://issues.apache.org/jira/browse/SPARK-51974
Project: Spark
Issue Type: Sub
[
https://issues.apache.org/jira/browse/SPARK-51867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-51867.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50665
[https://github
[
https://issues.apache.org/jira/browse/SPARK-51867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-51867:
--
Assignee: Weichen Xu
> Make scala model supporting save / load against local filesystem path
[
https://issues.apache.org/jira/browse/SPARK-51873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-51873:
--
Assignee: Weichen Xu
> For OneVsRest algorithm, allow using save / load to replace cache
> --
[
https://issues.apache.org/jira/browse/SPARK-51873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-51873.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50672
[https://github
Weichen Xu created SPARK-51873:
--
Summary: For OneVsRest algorithm, allow using save / load to
replace cache
Key: SPARK-51873
URL: https://issues.apache.org/jira/browse/SPARK-51873
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-51856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-51856.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50652
[https://github
Weichen Xu created SPARK-51867:
--
Summary: Make scala model supporting save / load against local
filesystem path
Key: SPARK-51867
URL: https://issues.apache.org/jira/browse/SPARK-51867
Project: Spark
Weichen Xu created SPARK-51856:
--
Summary: Add API for estimate saved model size
Key: SPARK-51856
URL: https://issues.apache.org/jira/browse/SPARK-51856
Project: Spark
Issue Type: Sub-task
Weichen Xu created SPARK-51666:
--
Summary: Fix sparkStageCompleted executorRunTime metric
calculation
Key: SPARK-51666
URL: https://issues.apache.org/jira/browse/SPARK-51666
Project: Spark
Issu
[
https://issues.apache.org/jira/browse/SPARK-51666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-51666:
---
Description:
Fix sparkStageCompleted executorRunTime metric calculation:
In case of when a spark ta
[
https://issues.apache.org/jira/browse/SPARK-51666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-51666:
--
Assignee: Weichen Xu
> Fix sparkStageCompleted executorRunTime metric calculation
>
[
https://issues.apache.org/jira/browse/SPARK-51551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-51551.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50324
[https://github
[
https://issues.apache.org/jira/browse/SPARK-51551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-51551:
---
Summary: For tuning algorithm, allow using save / load to replace cache
(was: For tuning algorithm
Weichen Xu created SPARK-51551:
--
Summary: For tuning algorithm, allow using save / load to replace
persist
Key: SPARK-51551
URL: https://issues.apache.org/jira/browse/SPARK-51551
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-51536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-51536.
Resolution: Not A Problem
> whitelist VectorAssembler
> -
>
>
[
https://issues.apache.org/jira/browse/SPARK-51340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-51340:
--
Assignee: Weichen Xu
> Model size estimation for linear classification & regression models
>
[
https://issues.apache.org/jira/browse/SPARK-51340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-51340.
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50278
[https://github
[
https://issues.apache.org/jira/browse/SPARK-51536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-51536:
--
Assignee: Weichen Xu
> whitelist VectorAssembler
> -
>
>
Weichen Xu created SPARK-51536:
--
Summary: whitelist VectorAssembler
Key: SPARK-51536
URL: https://issues.apache.org/jira/browse/SPARK-51536
Project: Spark
Issue Type: Sub-task
Componen
[
https://issues.apache.org/jira/browse/SPARK-50281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-50281:
---
Description:
pyspark local session `spark.jars` configuration does not work
Reproducing code:
[
https://issues.apache.org/jira/browse/SPARK-50281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-50281:
---
Description:
pyspark local session `spark.jars` configuration does not work
Reproducing code:
Weichen Xu created SPARK-50281:
--
Summary: pyspark local session `spark.jars` configuration does not
work
Key: SPARK-50281
URL: https://issues.apache.org/jira/browse/SPARK-50281
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-49615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reopened SPARK-49615:
> Feature transformers are case sensitive when unintented
> --
[
https://issues.apache.org/jira/browse/SPARK-49615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890345#comment-17890345
]
Weichen Xu edited comment on SPARK-49615 at 10/17/24 7:13 AM:
[
https://issues.apache.org/jira/browse/SPARK-49615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890345#comment-17890345
]
Weichen Xu commented on SPARK-49615:
[~chhavibansal]
You need to test against spar
[
https://issues.apache.org/jira/browse/SPARK-49793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-49793:
--
Assignee: Weichen Xu
> Enable PredictBatchUDFTests.test_caching for NumPy 2
> ---
[
https://issues.apache.org/jira/browse/SPARK-49615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-49615.
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 48398
[https://github
[
https://issues.apache.org/jira/browse/SPARK-49615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-49615:
--
Assignee: Weichen Xu
> Feature transformers are case sensitive when unintented
>
[
https://issues.apache.org/jira/browse/SPARK-48970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-48970.
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 47453
[https://github
[
https://issues.apache.org/jira/browse/SPARK-48970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-48970:
--
Assignee: Weichen Xu
> Avoid using SparkSession.getActiveSession in spark ML reader/writer
>
[
https://issues.apache.org/jira/browse/SPARK-48970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-48970:
---
Issue Type: Bug (was: Improvement)
> Avoid using SparkSession.getActiveSession in spark ML reader/w
Weichen Xu created SPARK-48970:
--
Summary: Avoid using SparkSession.getActiveSession in spark ML
reader/writer
Key: SPARK-48970
URL: https://issues.apache.org/jira/browse/SPARK-48970
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-48941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-48941:
--
Assignee: Weichen Xu
> PySparkML: Replace RDD read / write API invocation with Dataframe read
[
https://issues.apache.org/jira/browse/SPARK-48941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-48941.
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 47411
[https://github
Weichen Xu created SPARK-48941:
--
Summary: PySparkML: Replace RDD read / write API invocation with
Dataframe read / write API
Key: SPARK-48941
URL: https://issues.apache.org/jira/browse/SPARK-48941
Proje
[
https://issues.apache.org/jira/browse/SPARK-48883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-48883:
--
Assignee: Weichen Xu
> In spark ML, replace RDD read / write API invocation with Dataframe re
[
https://issues.apache.org/jira/browse/SPARK-48883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-48883.
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 47328
[https://github
Weichen Xu created SPARK-48883:
--
Summary: In spark ML, replace RDD read / write API invocation with
Dataframe read / write API
Key: SPARK-48883
URL: https://issues.apache.org/jira/browse/SPARK-48883
Proj
[
https://issues.apache.org/jira/browse/SPARK-48463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856713#comment-17856713
]
Weichen Xu commented on SPARK-48463:
I will try to do it this sprint. (and then cher
[
https://issues.apache.org/jira/browse/SPARK-48463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reopened SPARK-48463:
Assignee: Weichen Xu
> MLLib function unable to handle nested data
> ---
[
https://issues.apache.org/jira/browse/SPARK-48463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-48463.
Resolution: Not A Problem
> MLLib function unable to handle nested data
>
[
https://issues.apache.org/jira/browse/SPARK-48463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17854214#comment-17854214
]
Weichen Xu commented on SPARK-48463:
ah got it. then it is not supported :)
as
[
https://issues.apache.org/jira/browse/SPARK-48463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17854053#comment-17854053
]
Weichen Xu commented on SPARK-48463:
I think you don’t need to flatten the original
[
https://issues.apache.org/jira/browse/SPARK-48084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844112#comment-17844112
]
Weichen Xu commented on SPARK-48084:
This test error {{pyspark.ml.connect.evaluation
[
https://issues.apache.org/jira/browse/SPARK-48083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844111#comment-17844111
]
Weichen Xu commented on SPARK-48083:
this is not an issue,
{{copyFromLocalToFs}} req
[
https://issues.apache.org/jira/browse/SPARK-47663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-47663.
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45794
[https://github
[
https://issues.apache.org/jira/browse/SPARK-47663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-47663:
--
Assignee: Bobby Wang
> Add an end to end tests for checking if spark task works well with res
[
https://issues.apache.org/jira/browse/SPARK-46812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-46812.
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 44852
[https://github
[
https://issues.apache.org/jira/browse/SPARK-46812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-46812:
--
Assignee: Bobby Wang
> Make `mapInPandas` / mapInArrow` support ResourceProfile (Stage-Level
[
https://issues.apache.org/jira/browse/SPARK-46361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-46361.
Resolution: Won't Do
> Add spark dataset chunk read API (python only)
> --
[
https://issues.apache.org/jira/browse/SPARK-46361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-46361:
---
Description:
*Design doc:*
h1.
[https://docs.google.com/document/d/1LHzwCjm2SluHkta_08cM3jxFSgfF-ni
[
https://issues.apache.org/jira/browse/SPARK-46361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-46361:
--
Assignee: Weichen Xu
> Add spark dataset chunk read API (python only)
> -
[
https://issues.apache.org/jira/browse/SPARK-46361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-46361:
---
Description:
*Proposed API:*
{code:java}
def persist_dataframe_as_chunks(dataframe: DataFrame) -> li
Weichen Xu created SPARK-46361:
--
Summary: Add spark dataset chunk read API (python only)
Key: SPARK-46361
URL: https://issues.apache.org/jira/browse/SPARK-46361
Project: Spark
Issue Type: Improv
[
https://issues.apache.org/jira/browse/SPARK-45397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu resolved SPARK-45397.
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43199
[https://github
[
https://issues.apache.org/jira/browse/SPARK-45397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu reassigned SPARK-45397:
--
Assignee: Weichen Xu
> Add vector assembler feature transformer
> ---
Weichen Xu created SPARK-45397:
--
Summary: Add vector assembler feature transformer
Key: SPARK-45397
URL: https://issues.apache.org/jira/browse/SPARK-45397
Project: Spark
Issue Type: Sub-task
Weichen Xu created SPARK-45396:
--
Summary: Add doc entry for `pyspark.ml.connect` module
Key: SPARK-45396
URL: https://issues.apache.org/jira/browse/SPARK-45396
Project: Spark
Issue Type: Sub-tas
Weichen Xu created SPARK-45130:
--
Summary: Avoid Spark connect ML model to change input pandas
dataframe
Key: SPARK-45130
URL: https://issues.apache.org/jira/browse/SPARK-45130
Project: Spark
Is
[
https://issues.apache.org/jira/browse/SPARK-45130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-45130:
---
Description: Currently,
> Avoid Spark connect ML model to change input pandas dataframe
> -
1 - 100 of 708 matches
Mail list logo
