[
https://issues.apache.org/jira/browse/SPARK-54364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-54364:
------------------------------------
Assignee: Tian Gao
> Separate test_install_spark out from our CI
> -------------------------------------------
>
> Key: SPARK-54364
> URL: https://issues.apache.org/jira/browse/SPARK-54364
> Project: Spark
> Issue Type: Test
> Components: PySpark
> Affects Versions: 4.1.0
> Reporter: Tian Gao
> Assignee: Tian Gao
> Priority: Minor
> Labels: pull-request-available
>
> test_install_spark can take 2000~3000s based on the network, but it's not
> testing any local code. It basically download the spark from remote and try
> to unpack it. Now it's part of pyspark-core which means any commit that
> changes pyspark core will trigger this test in our CI. e.g.
> [https://github.com/apache/spark/actions/runs/19347901449/job/55352935128]
> This wastes a lot of our limited CI resources.
> I think we should at least disable this on main commit triggered CI - so
> leave it only for scheduled CIs.
> If we are confident about how this works, we can have a single CI that run
> this test once per day.
> I think we are probably spending more than 10 hours every day to run this
> test which does not give us any meaningful coverage.
> [~dongjoon]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
