Copilot commented on code in PR #13132:
URL: https://github.com/apache/gravitino/pull/13132#discussion_r4002840774
##########
core/src/test/java/org/apache/gravitino/job/local/TestLocalJobExecutor.java:
##########
@@ -159,6 +160,41 @@ public void testSubmitJobFailure() throws IOException {
Assertions.assertEquals(JobHandle.Status.FAILED,
jobExecutor.getJobStatus(jobId));
}
+ @Test
+ public void testSubmitSparkJobRejectedWhenSparkSubmitIsNotAvailable() throws
IOException {
+ File sparkHome = new File(workingDir, "spark");
+ LocalJobExecutor exec = new LocalJobExecutor();
+ exec.initialize(
+ ImmutableMap.of(LocalJobExecutorConfigs.SPARK_HOME,
sparkHome.getAbsolutePath()));
+
+ try {
+ SparkJobTemplate template =
+ SparkJobTemplate.builder()
+ .withName("spark-job")
+ .withExecutable(new File(workingDir,
"spark-demo.jar").getAbsolutePath())
+ .withClassName("com.example.MainClass")
+ .build();
+
+ // spark-submit does not exist, the job is rejected at submission
instead of being queued.
+ IllegalArgumentException e =
+ Assertions.assertThrows(IllegalArgumentException.class, () ->
exec.submitJob(template));
+ Assertions.assertTrue(e.getMessage().contains("spark-submit is not found
or not executable"));
+
+ // Once spark-submit is available, the same job is accepted.
+ File sparkSubmit = new File(sparkHome, "bin/spark-submit");
+ FileUtils.writeStringToFile(sparkSubmit, "#!/bin/sh\nexit 0\n", "UTF-8");
Review Comment:
`sparkHome` is only a path under `workingDir`, and neither it nor its `bin`
subdirectory is created. Commons IO's `writeStringToFile` does not create
missing parents, so this test fails with `FileNotFoundException` before it can
verify that a previously rejected job is accepted; create the parent directory
first.
##########
core/src/test/java/org/apache/gravitino/job/local/TestSparkProcessBuilder.java:
##########
@@ -144,6 +149,52 @@ public void testGenerateSparkSubmitCommand() {
Assertions.assertTrue(command4.contains("arg2"));
}
+ @Test
+ public void testResolveSparkSubmit() throws IOException {
+ File validSparkHome =
Files.createTempDirectory("gravitino-test-spark-home").toFile();
+ File invalidSparkHome =
Files.createTempDirectory("gravitino-test-no-spark-home").toFile();
+ try {
+ File sparkSubmit = new File(validSparkHome, "bin/spark-submit");
+ FileUtils.writeStringToFile(sparkSubmit, "#!/bin/sh\n", "UTF-8");
Review Comment:
The temporary Spark home only contains the top-level directory, so its `bin`
parent does not exist. Commons IO's `writeStringToFile` does not create missing
parent directories and this test will throw `FileNotFoundException` at this
line before exercising `resolveSparkSubmit`; create the parent directory first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
