nchammas commented on code in PR #59250:
URL: https://github.com/apache/spark/pull/59250#discussion_r4437493354
##########
dev/sparktestsupport/utils.py:
##########
@@ -154,6 +156,27 @@ def determine_modules_to_test(changed_modules,
deduplicated=True):
)
+def determine_shell_script_test_goals(
+ test_modules: Sequence[modules.Module], stage: modules.TestStage
+) -> tuple[modules.ShellScriptTestGoal, ...]:
+ """
+ Return the shell script test goals that the given modules declare for a
stage.
+
+ Goals are returned in the order determined by the modules passed in,
+ preserving each module's declaration order. Each goal is returned only
once:
+
+ >>> from pathlib import Path
+ >>> goals = determine_shell_script_test_goals(
+ ... [modules.build, modules.root], modules.TestStage.PRE_BUILD)
+ >>> [g.path for g in goals].count(Path("dev/test-dependencies.sh"))
+ 1
+ """
+ declared_goals = itertools.chain.from_iterable(m.shell_script_test_goals
for m in test_modules)
+ stage_goals = (goal for goal in declared_goals if goal.stage == stage)
Review Comment:
I think the added type check is a minor improvement, but I also believe we
are overusing doctests and explicit type checks. The Pythonic way to do this
IMO is to instrument a type checker and run it on `sparktestsupport`. I will do
that in a separate PR.
##########
dev/run-tests.py:
##########
@@ -602,9 +618,10 @@ def main():
with titled_block("Running R style checks"):
run_sparkr_style_checks()
- if any(m.should_run_build_tests for m in test_modules):
- with titled_block("Running build tests"):
- run_build_tests()
+ run_shell_script_tests(
Review Comment:
Tested this locally and updated the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
