sunchao commented on code in PR #57332:
URL: https://github.com/apache/spark/pull/57332#discussion_r3604984634
##########
core/src/main/scala/org/apache/spark/resource/TaskResourceRequest.scala:
##########
@@ -28,17 +28,28 @@ import org.apache.spark.annotation.{Since, Stable}
*
* @param resourceName Resource name
* @param amount Amount requesting as a Double to support fractional resource
requests.
- * Valid values are less than or equal to 1.0 or whole numbers.
This essentially
- * lets you configure X number of tasks to run on a single
resource,
- * ie amount equals 0.5 translates into 2 tasks per resource
address.
+ * For custom resources, valid values are less than or equal to
1.0 or whole
+ * numbers, since a task's amount must map onto discrete
resource addresses -
+ * ie amount equals 0.5 translates into 2 tasks per resource
address. CPUs
+ * (resource name "cpus") are a plain quantity drawn from the
executor's core
+ * pool rather than an addressable resource, so any amount of at
least 1e-9 is
+ * valid, e.g. 1.5.
*/
@Stable
@Since("3.1.0")
class TaskResourceRequest(val resourceName: String, val amount: Double)
extends Serializable {
- assert(amount <= 1.0 || amount % 1 == 0,
- s"The resource amount ${amount} must be either <= 1.0, or a whole number.")
+ if (resourceName == ResourceProfile.CPUS) {
+ // A positive amount below half of the internal accounting scale (1e-9)
would silently
+ // round to zero cpus downstream; reject it here where the original value
is still visible.
+ assert(!amount.isNaN && !amount.isInfinity &&
+ CpuAmount.normalize(BigDecimal(amount.toString)).signum > 0,
Review Comment:
[P2] Reject CPU requests that would be rounded down
This predicate validates only the sign after normalization, so the new
public `cpus(Double)` API accepts values with more precision than the `1e-9`
accounting scale and can allocate less CPU than requested. For example,
`cpus(1.0000000004)` is accepted, but `getTaskCpus` normalizes it to
`1.000000000`; a 10-core executor then gets 10 slots even though `floor(10 /
1.0000000004) = 9`. The config path already rejects excess precision, so please
validate the unnormalized decimal against the same scale and bounds (or round
upward, never downward).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
