[jira] [Commented] (SPARK-40902) Quick submission of drivers in tests to mesos scheduler results in dropping drivers
[ https://issues.apache.org/jira/browse/SPARK-40902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623290#comment-17623290 ] Apache Spark commented on SPARK-40902: -- User 'mridulm' has created a pull request for this issue: https://github.com/apache/spark/pull/38378 > Quick submission of drivers in tests to mesos scheduler results in dropping > drivers > --- > > Key: SPARK-40902 > URL: https://issues.apache.org/jira/browse/SPARK-40902 > Project: Spark > Issue Type: Bug > Components: Mesos >Affects Versions: 2.4.8, 3.0.3, 3.4.0 >Reporter: Mridul Muralidharan >Priority: Minor > > Queued drivers in MesosClusterScheduler are ordered based on > MesosDriverDescription - and the default ordering checks for priority, > followed by submission time. For two driver submissions with same priority > and if made in quick succession (such that submission time is same due to > millisecond granularity of Date), this results in dropping the second > MesosDriverDescription from the queuedDrivers - as driverOrdering returns 0 > when comparing the descriptions. This jira fixes the more immediate issue > with tests, but we do need to relook at this for mess scheduler in general > later. > Currently, this affects tests - for example, in the latest VOTE for 3.3.1 [1] > - and is not consistently reproducible unless on a fast machine. > [1] https://lists.apache.org/thread/jof098qxp0s6qqmt9qwv52f9665b1pjg -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40902) Quick submission of drivers in tests to mesos scheduler results in dropping drivers
[ https://issues.apache.org/jira/browse/SPARK-40902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623288#comment-17623288 ] Apache Spark commented on SPARK-40902: -- User 'mridulm' has created a pull request for this issue: https://github.com/apache/spark/pull/38378 > Quick submission of drivers in tests to mesos scheduler results in dropping > drivers > --- > > Key: SPARK-40902 > URL: https://issues.apache.org/jira/browse/SPARK-40902 > Project: Spark > Issue Type: Bug > Components: Mesos >Affects Versions: 2.4.8, 3.0.3, 3.4.0 >Reporter: Mridul Muralidharan >Priority: Major > > Queued drivers in MesosClusterScheduler are ordered based on > MesosDriverDescription - and the default ordering checks for priority, > followed by submission time. For two driver submissions with same priority > and if made in quick succession (such that submission time is same due to > millisecond granularity of Date), this results in dropping the second > MesosDriverDescription from the queuedDrivers - as driverOrdering returns 0 > when comparing the descriptions. This jira fixes the more immediate issue > with tests, but we do need to relook at this for mess scheduler in general > later. > Currently, this affects tests - for example, in the latest VOTE for 3.3.1 [1] > - and is not consistently reproducible unless on a fast machine. > [1] https://lists.apache.org/thread/jof098qxp0s6qqmt9qwv52f9665b1pjg -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40902) Quick submission of drivers in tests to mesos scheduler results in dropping drivers
[ https://issues.apache.org/jira/browse/SPARK-40902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623270#comment-17623270 ] Mridul Muralidharan commented on SPARK-40902: - +CC [~dongjoon], [~yumwang] > Quick submission of drivers in tests to mesos scheduler results in dropping > drivers > --- > > Key: SPARK-40902 > URL: https://issues.apache.org/jira/browse/SPARK-40902 > Project: Spark > Issue Type: Bug > Components: Mesos >Affects Versions: 2.4.8, 3.0.3, 3.4.0 >Reporter: Mridul Muralidharan >Priority: Major > > Queued drivers in MesosClusterScheduler are ordered based on > MesosDriverDescription - and the default ordering checks for priority, > followed by submission time. For two driver submissions with same priority > and if made in quick succession (such that submission time is same due to > millisecond granularity of Date), this results in dropping the second > MesosDriverDescription from the queuedDrivers - as driverOrdering returns 0 > when comparing the descriptions. This jira fixes the more immediate issue > with tests, but we do need to relook at this for mess scheduler in general > later. > Currently, this affects tests - for example, in the latest VOTE for 3.3.1 [1] > - and is not consistently reproducible unless on a fast machine. > [1] https://lists.apache.org/thread/jof098qxp0s6qqmt9qwv52f9665b1pjg -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
