Copilot commented on code in PR #12617: URL: https://github.com/apache/gluten/pull/12617#discussion_r3656323345
########## gluten-core/src/test/scala/org/apache/spark/util/SparkResourceUtilSuite.scala: ########## @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.util + +import org.apache.spark.SparkConf + +import org.scalatest.funsuite.AnyFunSuite + +class SparkResourceUtilSuite extends AnyFunSuite { + + test("getTaskSlots floors at one when task cpus exceed executor cores") { + // spark.task.cpus > executor cores is an invalid config that Spark rejects later, but Gluten + // reads task slots during plugin init and divides by the result. Returning 0 here makes that + // division throw ArithmeticException before Spark's validateTaskCpusLargeEnough can report the + // real misconfiguration. + val conf = new SparkConf(false) + .set("spark.master", "local[1]") + .set("spark.task.cpus", "2") + assert(SparkResourceUtil.getTaskSlots(conf) == 1) + } + + test("getTaskSlots fails fast when task cpus is zero") { + // spark.task.cpus is read via raw conf.getInt, which bypasses Spark's checkValue(_ > 0) (a + // check that only exists on Spark >= 4.2), so a zero value must not reach the division. Fail + // fast with a clear message instead of an opaque "/ by zero" ArithmeticException. + val conf = new SparkConf(false) + .set("spark.master", "local[8]") + .set("spark.task.cpus", "0") + val e = intercept[IllegalArgumentException](SparkResourceUtil.getTaskSlots(conf)) + assert(e.getMessage.contains("spark.task.cpus should be positive")) + } Review Comment: The suite currently codifies a Gluten-side fail-fast for `spark.task.cpus <= 0` (expecting `IllegalArgumentException`). That conflicts with the PR description, which states `getTaskSlots` should return a safe slot count (1) and let Spark's own validation surface the error message later. Please align the tests with the chosen behavior (either assert a safe return value and no exception, or update the PR description/issue rationale to match the intended early validation). ########## gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala: ########## @@ -77,8 +77,17 @@ object SparkResourceUtil extends Logging { def getTaskSlots(conf: SparkConf): Int = { val executorCores = SparkResourceUtil.getExecutorCores(conf) + // spark.task.cpus is read raw here, which bypasses Spark's own checkValue(_ > 0) (and on + // Spark < 4.2 that positivity check does not exist at all). getTaskSlots runs during driver + // plugin init, before Spark validates the value, so fail fast on a non-positive setting rather + // than dividing by it: a zero would throw an opaque "/ by zero" ArithmeticException and a + // negative would silently produce negative task slots and off-heap budgets. val taskCores = conf.getInt("spark.task.cpus", 1) - executorCores / taskCores + require(taskCores > 0, s"spark.task.cpus should be positive, but was $taskCores") Review Comment: `getTaskSlots` now throws via `require(taskCores > 0, ...)` for `spark.task.cpus <= 0`. That still fails during Gluten plugin init (albeit with a clearer message), whereas the PR description/issue motivation says the goal is to avoid a Gluten-side failure and *defer to Spark's own validation/error message* by returning a safe slot count. Please either (a) change this path to floor non-positive values to 1 (and rely on Spark to reject the config later), or (b) update the PR description/issue rationale and tests to reflect an intentional fail-fast behavior in Gluten. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
