Mridul Muralidharan created SPARK-40902:
-------------------------------------------
Summary: Quick submission of drivers in tests to mesos scheduler
results in dropping drivers
Key: SPARK-40902
URL: https://issues.apache.org/jira/browse/SPARK-40902
Project: Spark
Issue Type: Bug
Components: Mesos
Affects Versions: 3.0.3, 2.4.8, 3.4.0
Reporter: Mridul Muralidharan
Queued drivers in MesosClusterScheduler are ordered based on
MesosDriverDescription - and the default ordering checks for priority, followed
by submission time. For two driver submissions with same priority and if made
in quick succession (such that submission time is same due to millisecond
granularity of Date), this results in dropping the second
MesosDriverDescription from the queuedDrivers - as driverOrdering returns 0
when comparing the descriptions. This jira fixes the more immediate issue with
tests, but we do need to relook at this for mess scheduler in general later.
Currently, this affects tests - for example, in the latest VOTE for 3.3.1 [1] -
and is not consistently reproducible unless on a fast machine.
[1] https://lists.apache.org/thread/jof098qxp0s6qqmt9qwv52f9665b1pjg
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
