[
https://issues.apache.org/jira/browse/MESOS-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166505#comment-17166505
]
Benjamin Mahler commented on MESOS-10143:
-----------------------------------------
[~puneetku287] It looks like the scheduler native library is getting
backlogged, this can happen when the scheduler cannot process messages as fast
as they come in from the master. (In your example I see
Scheduler::resourceOffers took 6.8 ms which is long).
If you want to check this while it's happening the next time, you can hit
{{http://IP:PORT/metrics/snapshot}} of the scheduler library, and it shouldn't
return because the scheduler metrics are not able to get computed in a timely
manner. You can also specify a timeout via
{{http://IP:PORT/metrics/snapshot?timeout=10secs}} and should see a response
without the {{scheduler/event_queue_messages}} metric present.
You may want to fix the port of the scheduler library in order to do this, by
setting LIBPROCESS_PORT=X in your environment before instantiating the library.
> Outstanding Offers accumulating
> -------------------------------
>
> Key: MESOS-10143
> URL: https://issues.apache.org/jira/browse/MESOS-10143
> Project: Mesos
> Issue Type: Bug
> Components: master, scheduler driver
> Affects Versions: 1.7.0
> Environment: Mesos Version 1.7.0
> JDK 8.0
> Reporter: Puneet Kumar
> Priority: Minor
>
> We manage an Apache Mesos cluster version 1.7.0. We have written a framework
> in Java that schedules tasks to Mesos master at a rate of 300 TPS. Everything
> works fine for almost 24 hours but then outstanding offers accumulate &
> saturate within 15 minutes. Outstanding offers aren't reclaimed by Mesos
> master. We observe "RescindOffer" messages in verbose (GLOG v=3) framework
> logs but outstanding offers don't reduce. New resources aren't offered to
> framework when outstanding offers saturate. We have to restart the scheduler
> to reset outstanding offers to zero.
> Any suggestions to debug this issue are welcome.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
