[jira] [Updated] (SPARK-38503) Add warn for getAdditionalPreKubernetesResources in executor side

2022-03-14 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38503: Summary: Add warn for getAdditionalPreKubernetesResources in executor side (was: Add driver `PodG

[jira] [Updated] (SPARK-38524) [TEST] Fix Volcano weight to be positive integer and use cpu capability instead

2022-03-13 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38524: Summary: [TEST] Fix Volcano weight to be positive integer and use cpu capability instead (was: [T

[jira] [Updated] (SPARK-38524) Fix Volcano weight to be positive integer and use cpu capability instead

2022-03-13 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38524: Summary: Fix Volcano weight to be positive integer and use cpu capability instead (was: [TEST] Fi

[jira] [Updated] (SPARK-38524) [TEST] Change disable queue to capability limit way

2022-03-13 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38524: Component/s: Tests > [TEST] Change disable queue to capability limit way > ---

[jira] [Closed] (SPARK-38525) Check resource after resource creation

2022-03-13 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang closed SPARK-38525. --- > Check resource after resource creation > -- > > Ke

[jira] [Resolved] (SPARK-38525) Check resource after resource creation

2022-03-13 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-38525. - Resolution: Won't Fix > Check resource after resource creation > ---

[jira] [Updated] (SPARK-38525) Check resource after resource creation

2022-03-13 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38525: Summary: Check resource after resource creation (was: [TEST] Check resource after resource creati

[jira] [Created] (SPARK-38543) [CI] Introduce Spark on Kubernetes CI into Volcano community

2022-03-13 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38543: --- Summary: [CI] Introduce Spark on Kubernetes CI into Volcano community Key: SPARK-38543 URL: https://issues.apache.org/jira/browse/SPARK-38543 Project: Spark I

[jira] [Created] (SPARK-38525) [TEST] Check resource after resource creation

2022-03-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38525: --- Summary: [TEST] Check resource after resource creation Key: SPARK-38525 URL: https://issues.apache.org/jira/browse/SPARK-38525 Project: Spark Issue Type: Sub-t

[jira] [Updated] (SPARK-38524) [TEST] Change disable queue to capability limit way

2022-03-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38524: Summary: [TEST] Change disable queue to capability limit way (was: Change disable queue to capabi

[jira] [Created] (SPARK-38524) Change disable queue to capability limit way

2022-03-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38524: --- Summary: Change disable queue to capability limit way Key: SPARK-38524 URL: https://issues.apache.org/jira/browse/SPARK-38524 Project: Spark Issue Type: Sub-ta

[jira] [Commented] (SPARK-38515) Volcano queue is not deleted

2022-03-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504838#comment-17504838 ] Yikun Jiang commented on SPARK-38515: - docker rmi volcanosh/vc-scheduler:latest doc

[jira] [Commented] (SPARK-38508) Volcano feature doesn't work on EKS graviton instances

2022-03-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504836#comment-17504836 ] Yikun Jiang commented on SPARK-38508: - docker rmi volcanosh/vc-scheduler-arm64:lates

[jira] [Created] (SPARK-38503) Add driver `PodGroup` templates doc and cleanup executor PodGroup templates

2022-03-10 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38503: --- Summary: Add driver `PodGroup` templates doc and cleanup executor PodGroup templates Key: SPARK-38503 URL: https://issues.apache.org/jira/browse/SPARK-38503 Project: Sp

[jira] [Commented] (SPARK-38488) Spark doc build not work on Mac OS M1

2022-03-09 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503978#comment-17503978 ] Yikun Jiang commented on SPARK-38488: - [~hyukjin.kwon] Sure, let me check and test i

[jira] [Updated] (SPARK-38488) Spark doc build not work on Mac OS M1

2022-03-09 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38488: Description:   {code:java} diff --git a/docs/.bundle/config b/docs/.bundle/config index b13821f801

[jira] [Created] (SPARK-38488) Spark doc build not work on Mac OS M1

2022-03-09 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38488: --- Summary: Spark doc build not work on Mac OS M1 Key: SPARK-38488 URL: https://issues.apache.org/jira/browse/SPARK-38488 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-36060) Support backing off dynamic allocation increases if resources are "stuck"

2022-03-09 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-36060. - Resolution: Fixed Now minRes and queue supported have been supported by https://issues.apache.o

[jira] [Updated] (SPARK-38453) Make Volcano IT can be completed in 4U6G env

2022-03-08 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38453: Summary: Make Volcano IT can be completed in 4U6G env (was: Improve Volcano IT can be completed i

[jira] [Created] (SPARK-38453) Improve Volcano IT can be completed in 4U6G env

2022-03-08 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38453: --- Summary: Improve Volcano IT can be completed in 4U6G env Key: SPARK-38453 URL: https://issues.apache.org/jira/browse/SPARK-38453 Project: Spark Issue Type: Sub

[jira] [Created] (SPARK-38452) Support K8s IT with pyDockerfile and rDockerfile

2022-03-08 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38452: --- Summary: Support K8s IT with pyDockerfile and rDockerfile Key: SPARK-38452 URL: https://issues.apache.org/jira/browse/SPARK-38452 Project: Spark Issue Type: Su

[jira] [Updated] (SPARK-38452) Support pyDockerfile and rDockerfile in SBT K8s IT

2022-03-08 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38452: Summary: Support pyDockerfile and rDockerfile in SBT K8s IT (was: Support K8s IT with pyDockerf

[jira] [Commented] (SPARK-38416) Duplicate functions

2022-03-06 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17501907#comment-17501907 ] Yikun Jiang commented on SPARK-38416: - Yep, this might be a bad copy paste, are you

[jira] [Commented] (SPARK-38189) Add priority scheduling doc for Spark on K8S

2022-03-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17501867#comment-17501867 ] Yikun Jiang commented on SPARK-38189: - [~dongjoon] Thanks for information, I re-grea

[jira] [Created] (SPARK-38423) Support priority scheduling with volcano implementations

2022-03-05 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38423: --- Summary: Support priority scheduling with volcano implementations Key: SPARK-38423 URL: https://issues.apache.org/jira/browse/SPARK-38423 Project: Spark Issue

[jira] [Resolved] (SPARK-38135) Introduce `spark.kubernetes.job` sheduling related configurations

2022-03-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-38135. - Resolution: Invalid > Introduce `spark.kubernetes.job` sheduling related configurations > -

[jira] [Updated] (SPARK-38189) Add priority scheduling doc for Spark on K8S

2022-03-03 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38189: Component/s: Documentation > Add priority scheduling doc for Spark on K8S > --

[jira] [Updated] (SPARK-38189) Add priority scheduling doc for Spark on K8S

2022-03-03 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38189: Summary: Add priority scheduling doc for Spark on K8S (was: Support priority scheduling (Introduc

[jira] [Resolved] (SPARK-38403) `running-on-kubernetes` page render bad in v3.2.1(latest) website

2022-03-03 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-38403. - Resolution: Fixed master: [https://github.com/apache/spark/pull/35572] 3.2 backport: https://

[jira] [Commented] (SPARK-38403) `running-on-kubernetes` page render bad in v3.2.1(latest) website

2022-03-03 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500604#comment-17500604 ] Yikun Jiang commented on SPARK-38403: - Ah, yes, it's this fix, I also do a compare v

[jira] [Created] (SPARK-38403) `running-on-kubernetes` page render bad in v3.2.1(latest) website

2022-03-02 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38403: --- Summary: `running-on-kubernetes` page render bad in v3.2.1(latest) website Key: SPARK-38403 URL: https://issues.apache.org/jira/browse/SPARK-38403 Project: Spark

[jira] [Updated] (SPARK-36057) Support Customized Kubernetes Schedulers

2022-02-23 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36057: Description: This is an umbrella issue for tracking the work for supporting Volcano & Yunikorn on

[jira] [Updated] (SPARK-38189) Support priority scheduling (Introduce priorityClass) with volcano implementations

2022-02-23 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38189: Summary: Support priority scheduling (Introduce priorityClass) with volcano implementations (was:

[jira] [Commented] (SPARK-36060) Support backing off dynamic allocation increases if resources are "stuck"

2022-02-22 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17495987#comment-17495987 ] Yikun Jiang commented on SPARK-36060: - Yep, agree with [~wwei] . it can be also res

[jira] [Resolved] (SPARK-36593) [Deprecated] Support the Volcano Job API

2022-02-22 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-36593. - Resolution: Abandoned > [Deprecated] Support the Volcano Job API > -

[jira] [Created] (SPARK-38272) Fix wrong context name in spark k8s integration test on docker-desktop

2022-02-21 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38272: --- Summary: Fix wrong context name in spark k8s integration test on docker-desktop Key: SPARK-38272 URL: https://issues.apache.org/jira/browse/SPARK-38272 Project: Spark

[jira] [Created] (SPARK-38244) Upgrade kubernetes-client to 5.12.1

2022-02-17 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38244: --- Summary: Upgrade kubernetes-client to 5.12.1 Key: SPARK-38244 URL: https://issues.apache.org/jira/browse/SPARK-38244 Project: Spark Issue Type: Bug C

[jira] [Updated] (SPARK-36059) Add the ability to specify a scheduler

2022-02-12 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36059: Summary: Add the ability to specify a scheduler (was: Add the ability to specify a scheduler & qu

[jira] [Updated] (SPARK-38190) Add fair-share scheduling integration test

2022-02-12 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38190: Summary: Add fair-share scheduling integration test (was: Support fair-share scheduling) > Add f

[jira] [Updated] (SPARK-36061) Add `volcano` module and feature step

2022-02-12 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36061: Summary: Add `volcano` module and feature step (was: Add Volcano feature step) > Add `volcano` m

[jira] [Updated] (SPARK-38189) Support preemption scheduling (Introduce priorityClass) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38189: Summary: Support preemption scheduling (Introduce priorityClass) with volcano implementations (wa

[jira] [Created] (SPARK-38190) Support fair-share scheduling

2022-02-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38190: --- Summary: Support fair-share scheduling Key: SPARK-38190 URL: https://issues.apache.org/jira/browse/SPARK-38190 Project: Spark Issue Type: Sub-task Co

[jira] [Created] (SPARK-38189) Preemption scheduling (Introduce priorityClass) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38189: --- Summary: Preemption scheduling (Introduce priorityClass) with volcano implementations Key: SPARK-38189 URL: https://issues.apache.org/jira/browse/SPARK-38189 Project: S

[jira] [Created] (SPARK-38188) Support queue scheduling (Introduce queue) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38188: --- Summary: Support queue scheduling (Introduce queue) with volcano implementations Key: SPARK-38188 URL: https://issues.apache.org/jira/browse/SPARK-38188 Project: Spark

[jira] [Updated] (SPARK-38187) Support resource reservation (Introduce minCPU/minMemory) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38187: Summary: Support resource reservation (Introduce minCPU/minMemory) with volcano implementations (

[jira] [Updated] (SPARK-38187) Resource reservation (Introduce minCPU/minMemory) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38187: Parent: SPARK-36057 Issue Type: Sub-task (was: Bug) > Resource reservation (Introduce min

[jira] [Created] (SPARK-38187) Resource reservation (Introduce minCPU/minMemory) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38187: --- Summary: Resource reservation (Introduce minCPU/minMemory) with volcano implementations Key: SPARK-38187 URL: https://issues.apache.org/jira/browse/SPARK-38187 Project:

[jira] [Created] (SPARK-38135) Introduce `spark.kubernetes.job` sheduling related configurations

2022-02-07 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38135: --- Summary: Introduce `spark.kubernetes.job` sheduling related configurations Key: SPARK-38135 URL: https://issues.apache.org/jira/browse/SPARK-38135 Project: Spark

[jira] [Updated] (SPARK-38135) Introduce `spark.kubernetes.job` sheduling related configurations

2022-02-07 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38135: Description: spark.kubernetes.job.minCPU:  the minimum cpu resources for running job spark.kuberne

[jira] [Created] (SPARK-37998) Failed to create rbac in latest kubernetes (minikube setup)

2022-01-24 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-37998: --- Summary: Failed to create rbac in latest kubernetes (minikube setup) Key: SPARK-37998 URL: https://issues.apache.org/jira/browse/SPARK-37998 Project: Spark Is

[jira] [Updated] (SPARK-37988) dev/reformat version required warning is confused

2022-01-22 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37988: Component/s: Build PySpark (was: Project Infra) > dev/reform

[jira] [Created] (SPARK-37988) dev/reformat version required warning is confused

2022-01-22 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-37988: --- Summary: dev/reformat version required warning is confused Key: SPARK-37988 URL: https://issues.apache.org/jira/browse/SPARK-37988 Project: Spark Issue Type: I

[jira] [Updated] (SPARK-36057) Support volcano/alternative schedulers

2022-01-20 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36057: Shepherd: Holden Karau Affects Version/s: 3.3.0 (was: 3.2.0

[jira] [Comment Edited] (SPARK-37930) Fix DataFrame select subset with duplicated columns

2022-01-17 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477581#comment-17477581 ] Yikun Jiang edited comment on SPARK-37930 at 1/18/22, 5:46 AM: ---

[jira] [Comment Edited] (SPARK-37930) Fix DataFrame select subset with duplicated columns

2022-01-17 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477581#comment-17477581 ] Yikun Jiang edited comment on SPARK-37930 at 1/18/22, 5:41 AM: ---

[jira] [Comment Edited] (SPARK-37930) Fix DataFrame select subset with duplicated columns

2022-01-17 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477581#comment-17477581 ] Yikun Jiang edited comment on SPARK-37930 at 1/18/22, 5:34 AM: ---

[jira] [Comment Edited] (SPARK-37930) Fix DataFrame select subset with duplicated columns

2022-01-17 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477581#comment-17477581 ] Yikun Jiang edited comment on SPARK-37930 at 1/18/22, 5:30 AM: ---

[jira] [Commented] (SPARK-37930) Fix DataFrame select subset with duplicated columns

2022-01-17 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477581#comment-17477581 ] Yikun Jiang commented on SPARK-37930: - [~dchvn] Thanks for investigation Also update

[jira] (SPARK-37930) Fix DataFrame select subset with duplicated columns

2022-01-17 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37930 ] Yikun Jiang deleted comment on SPARK-37930: - was (Author: yikunkero): https://github.com/apache/spark/blob/df7447bc62052e3d7391ba23d7220fb8c9b923fd/python/pyspark/pandas/frame.py#L12268 FYI:

[jira] [Comment Edited] (SPARK-37930) Fix DataFrame select subset with duplicated columns

2022-01-17 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477561#comment-17477561 ] Yikun Jiang edited comment on SPARK-37930 at 1/18/22, 4:31 AM: ---

[jira] [Commented] (SPARK-37930) Fix DataFrame select subset with duplicated columns

2022-01-17 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477561#comment-17477561 ] Yikun Jiang commented on SPARK-37930: - https://github.com/apache/spark/blob/df7447bc

[jira] [Created] (SPARK-37908) Refactoring on pod label test in BasicFeatureStepSuite

2022-01-14 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-37908: --- Summary: Refactoring on pod label test in BasicFeatureStepSuite Key: SPARK-37908 URL: https://issues.apache.org/jira/browse/SPARK-37908 Project: Spark Issue Ty

[jira] [Updated] (SPARK-37886) Use ComparisonTestBase to reduce redundant test code

2022-01-12 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37886: Environment: (was: We have many testcase are using same logic to covert pdf to psdf, we can us

[jira] [Updated] (SPARK-37886) Use ComparisonTestBase to reduce redundant test code

2022-01-12 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37886: Description: We have many testcase are using same logic to covert pdf to psdf, we can use Comparis

[jira] [Created] (SPARK-37886) Use ComparisonTestBase to reduce redundant test code

2022-01-12 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-37886: --- Summary: Use ComparisonTestBase to reduce redundant test code Key: SPARK-37886 URL: https://issues.apache.org/jira/browse/SPARK-37886 Project: Spark Issue Type

[jira] [Updated] (SPARK-37258) Upgrade kubernetes-client >= 5.11.0 to support Volcano

2022-01-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37258: Summary: Upgrade kubernetes-client >= 5.11.0 to support Volcano (was: Upgrade kubernetes-client

[jira] [Created] (SPARK-37868) Skip equal check in LocIndexer._select_cols_by_iterable if eager_check disable

2022-01-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-37868: --- Summary: Skip equal check in LocIndexer._select_cols_by_iterable if eager_check disable Key: SPARK-37868 URL: https://issues.apache.org/jira/browse/SPARK-37868 Project:

[jira] (SPARK-37772) [PYSPARK] Publish ApacheSparkGitHubActionImage arm64 docker image

2022-01-06 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37772 ] Yikun Jiang deleted comment on SPARK-37772: - was (Author: yikunkero): To speed up the installation on pypy (aarch64) https://github.com/MacPython/numpy-wheels/issues/143 https://github.com/M

[jira] [Commented] (SPARK-37772) [PYSPARK] Publish ApacheSparkGitHubActionImage arm64 docker image

2022-01-06 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470309#comment-17470309 ] Yikun Jiang commented on SPARK-37772: - Add the PR in https://github.com/dongjoon-hy

[jira] [Commented] (SPARK-37772) [PYSPARK] Publish ApacheSparkGitHubActionImage arm64 docker image

2022-01-06 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470295#comment-17470295 ] Yikun Jiang commented on SPARK-37772: - To speed up the installation on pypy (aarch64

[jira] [Updated] (SPARK-35607) Migrate Spark Arm Job from Jenkins to GitHub Actions

2022-01-06 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-35607: Description: There were two Arm CI jobs in AMP lab: [https://amplab.cs.berkeley.edu/jenkins/job/s

[jira] [Updated] (SPARK-35607) Migrate Spark Arm Job from Jenkins to GitHub Actions

2022-01-06 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-35607: Description: There were two Arm CI jobs in AMP lab: [https://amplab.cs.berkeley.edu/jenkins/job/s

[jira] [Updated] (SPARK-35607) Migrate Spark Arm Job from Jenkins to GitHub Actions

2022-01-06 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-35607: Description: There were two Arm CI jobs in AMP lab: [https://amplab.cs.berkeley.edu/jenkins/job/s

[jira] [Updated] (SPARK-37759) Add ARM based self-hosted runner to Apache Spark

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37759: Description: For Spark Test Runner: {code:bash} # Prepare the custom runner mkdir actions-runner

[jira] [Updated] (SPARK-37759) Add ARM based self-hosted runner to Apache Spark

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37759: Description: {code:bash} # Prepare the custom runner mkdir actions-runner && cd actions-runner cur

[jira] [Updated] (SPARK-37759) Add ARM based self-hosted runner to Apache Spark

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37759: Description: {code:shell} # Prepare the custom runner mkdir actions-runner && cd actions-runner c

[jira] [Updated] (SPARK-37759) Add ARM based self-hosted runner to Apache Spark

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37759: Description: # Prepare the custom runner mkdir actions-runner && cd actions-runner curl -o actions

[jira] [Commented] (SPARK-37758) [PYSPARK] Enable PySpark scheduled job on ARM based self-hosted runner

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469654#comment-17469654 ] Yikun Jiang commented on SPARK-37758: - {code:java} sudo apt install docker.io sudo

[jira] [Comment Edited] (SPARK-37757) [Scala] Enable Spark test scheduled job on ARM based self-hosted runner

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469653#comment-17469653 ] Yikun Jiang edited comment on SPARK-37757 at 1/6/22, 3:34 AM:

[jira] [Commented] (SPARK-37757) [Scala] Enable Spark test scheduled job on ARM based self-hosted runner

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469653#comment-17469653 ] Yikun Jiang commented on SPARK-37757: -   {code:java} sudo apt update sudo apt-get in

[jira] [Commented] (SPARK-37757) [Scala] Enable Spark test scheduled job on ARM based self-hosted runner

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469646#comment-17469646 ] Yikun Jiang commented on SPARK-37757: - `apt install libssl-dev`, then it's stable to

[jira] [Updated] (SPARK-36593) [Deprecated] Support the Volcano Job API

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36593: Summary: [Deprecated] Support the Volcano Job API (was: Support the Volcano Job API) > [Deprecat

[jira] [Updated] (SPARK-36061) Add Volcano feature step

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36061: Description: Create a PodGroup with user specified minimum resources required > Add Volcano featu

[jira] [Updated] (SPARK-36061) Add Volcano feature step

2022-01-05 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36061: Summary: Add Volcano feature step (was: Create a PodGroup with user specified minimum resources r

[jira] [Updated] (SPARK-37331) Add the ability to create resources before driver pod

2022-01-04 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37331: Summary: Add the ability to create resources before driver pod (was: Add the ability to creating

[jira] [Commented] (SPARK-37772) [PYSPARK] Publish ApacheSparkGitHubActionImage arm64 docker image

2022-01-02 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17467764#comment-17467764 ] Yikun Jiang commented on SPARK-37772: - [~hyukjin.kwon] Thanks, if I remembered corre

[jira] [Created] (SPARK-37801) List PyPy3 installed libraries in build_and_test workflow

2022-01-02 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-37801: --- Summary: List PyPy3 installed libraries in build_and_test workflow Key: SPARK-37801 URL: https://issues.apache.org/jira/browse/SPARK-37801 Project: Spark Issue

[jira] [Commented] (SPARK-37772) [PYSPARK] Publish ApacheSparkGitHubActionImage arm64 docker image

2021-12-30 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17467061#comment-17467061 ] Yikun Jiang commented on SPARK-37772: - For CPython(such like, python 3.8, python 3.9

[jira] [Commented] (SPARK-37772) [PYSPARK] Publish ApacheSparkGitHubActionImage arm64 docker image

2021-12-30 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466848#comment-17466848 ] Yikun Jiang commented on SPARK-37772: - [https://pypi.org/project/oldest-supported-nu

[jira] [Commented] (SPARK-37758) [PYSPARK] Enable PySpark scheduled job on ARM based self-hosted runner

2021-12-29 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466717#comment-17466717 ] Yikun Jiang commented on SPARK-37758: - cpython 3.9 passed: https://github.com/Yikun

[jira] [Updated] (SPARK-37758) [PYSPARK] Enable PySpark scheduled job on ARM based self-hosted runner

2021-12-29 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37758: Description: cpython 3.9 pypy3 > [PYSPARK] Enable PySpark scheduled job on ARM based self-hosted

[jira] [Commented] (SPARK-37757) [Scala] Enable Spark test scheduled job on ARM based self-hosted runner

2021-12-29 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466716#comment-17466716 ] Yikun Jiang commented on SPARK-37757: - * DistributedSuite.caching in memory, replica

[jira] (SPARK-37758) [PYSPARK] Enable PySpark scheduled job on ARM based self-hosted runner

2021-12-29 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37758 ] Yikun Jiang deleted comment on SPARK-37758: - was (Author: yikunkero): * DistributedSuite.caching in memory, replicated (encryption = on) * DistributedSuite.caching on disk, replicated 2 (enc

[jira] [Comment Edited] (SPARK-37772) [PYSPARK] Publish ApacheSparkGitHubActionImage arm64 docker image

2021-12-29 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466715#comment-17466715 ] Yikun Jiang edited comment on SPARK-37772 at 12/30/21, 7:45 AM: --

[jira] [Commented] (SPARK-37772) [PYSPARK] Publish ApacheSparkGitHubActionImage arm64 docker image

2021-12-29 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466715#comment-17466715 ] Yikun Jiang commented on SPARK-37772: - There are some depdency conflict on: [https:

[jira] [Created] (SPARK-37775) [PYSPARK] Fix mlflow doctest

2021-12-28 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-37775: --- Summary: [PYSPARK] Fix mlflow doctest Key: SPARK-37775 URL: https://issues.apache.org/jira/browse/SPARK-37775 Project: Spark Issue Type: Sub-task Com

[jira] [Updated] (SPARK-37775) [PYSPARK] Fix mlflow doctest

2021-12-28 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37775: Description: {code:java} ** 13

[jira] [Updated] (SPARK-37775) [PYSPARK] Fix mlflow doctest

2021-12-28 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37775: Environment: (was: {code:java}

[jira] [Updated] (SPARK-37758) [PYSPARK] Enable PySpark scheduled job on ARM based self-hosted runner

2021-12-28 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37758: Summary: [PYSPARK] Enable PySpark scheduled job on ARM based self-hosted runner (was: [Scala] Ena

[jira] [Updated] (SPARK-37757) [Scala] Enable Spark test scheduled job on ARM based self-hosted runner

2021-12-28 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37757: Summary: [Scala] Enable Spark test scheduled job on ARM based self-hosted runner (was: Enable Spa

<    1   2   3   4   5   6   7   >