[
https://issues.apache.org/jira/browse/MESOS-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17178186#comment-17178186
]
Puneet Kumar commented on MESOS-10143:
--------------------------------------
We reduced average latency of Scheduler::resourceOffers from ~50 ms to ~15ms by
eliminating non-essential workload from our java application that implements
Scheduler callbacks. After that, we haven't observed this issue till date. Our
application is handling ~400 offers per second at average latency of 15 ms for
Scheduler::resourceOffers.
[~bmahler] Thanks for describing how to obtain metrics for scheduler library.
> Outstanding Offers accumulating
> -------------------------------
>
> Key: MESOS-10143
> URL: https://issues.apache.org/jira/browse/MESOS-10143
> Project: Mesos
> Issue Type: Bug
> Components: master, scheduler driver
> Affects Versions: 1.7.0
> Environment: Mesos Version 1.7.0
> JDK 8.0
> Reporter: Puneet Kumar
> Priority: Minor
>
> We manage an Apache Mesos cluster version 1.7.0. We have written a framework
> in Java that schedules tasks to Mesos master at a rate of 300 TPS. Everything
> works fine for almost 24 hours but then outstanding offers accumulate &
> saturate within 15 minutes. Outstanding offers aren't reclaimed by Mesos
> master. We observe "RescindOffer" messages in verbose (GLOG v=3) framework
> logs but outstanding offers don't reduce. New resources aren't offered to
> framework when outstanding offers saturate. We have to restart the scheduler
> to reset outstanding offers to zero.
> Any suggestions to debug this issue are welcome.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
