[
https://issues.apache.org/jira/browse/MESOS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088372#comment-15088372
]
Adam B commented on MESOS-3307:
-------------------------------
Please note that each agent tracks its own list of completed tasks/frameworks,
and does not poll the master's state.json.
And I wouldn't worry about adding more flags to master, since those are usually
managed by config files or scripts, and each flag should have a sane default so
most users don't need to change them.
Since the only consumers of this data are CLI clients and other scripts, the
real question is whether to limit the length of the lists in-memory in the
master, which could save some memory, but affects all clients equally; or do
the filtering per-request (query-style), which consumes more memory, takes
extra time to filter when generating (the already-slow) state.json, but allows
each client to control how long of a history, if any, it wants from its request.
I'm in favor of the simple cmd-line flag for now, and we can consider more
intelligent query filtering as a separate issue, namely MESOS-2258
> Configurable size of completed task / framework history
> -------------------------------------------------------
>
> Key: MESOS-3307
> URL: https://issues.apache.org/jira/browse/MESOS-3307
> Project: Mesos
> Issue Type: Bug
> Reporter: Ian Babrou
> Assignee: Kevin Klues
> Labels: mesosphere
>
> We try to make Mesos work with multiple frameworks and mesos-dns at the same
> time. The goal is to have set of frameworks per team / project on a single
> Mesos cluster.
> At this point our mesos state.json is at 4mb and it takes a while to
> assembly. 5 mesos-dns instances hit state.json every 5 seconds, effectively
> pushing mesos-master CPU usage through the roof. It's at 100%+ all the time.
> Here's the problem:
> {noformat}
> mesos λ curl -s http://mesos-master:5050/master/state.json | jq
> .frameworks[].completed_tasks[].framework_id | sort | uniq -c | sort -n
> 1 "20150606-001827-252388362-5050-5982-0003"
> 16 "20150606-001827-252388362-5050-5982-0005"
> 18 "20150606-001827-252388362-5050-5982-0029"
> 73 "20150606-001827-252388362-5050-5982-0007"
> 141 "20150606-001827-252388362-5050-5982-0009"
> 154 "20150820-154817-302720010-5050-15320-0000"
> 289 "20150606-001827-252388362-5050-5982-0004"
> 510 "20150606-001827-252388362-5050-5982-0012"
> 666 "20150606-001827-252388362-5050-5982-0028"
> 923 "20150116-002612-269165578-5050-32204-0003"
> 1000 "20150606-001827-252388362-5050-5982-0001"
> 1000 "20150606-001827-252388362-5050-5982-0006"
> 1000 "20150606-001827-252388362-5050-5982-0010"
> 1000 "20150606-001827-252388362-5050-5982-0011"
> 1000 "20150606-001827-252388362-5050-5982-0027"
> mesos λ fgrep 1000 -r src/master
> src/master/constants.cpp:const size_t MAX_REMOVED_SLAVES = 100000;
> src/master/constants.cpp:const uint32_t MAX_COMPLETED_TASKS_PER_FRAMEWORK =
> 1000;
> {noformat}
> Active tasks are just 6% of state.json response:
> {noformat}
> mesos λ cat ~/temp/mesos-state.json | jq -c . | wc
> 1 14796 4138942
> mesos λ cat ~/temp/mesos-state.json | jq .frameworks[].tasks | jq -c . | wc
> 16 37 252774
> {noformat}
> I see four options that can improve the situation:
> 1. Add query string param to exclude completed tasks from state.json and use
> it in mesos-dns and similar tools. There is no need for mesos-dns to know
> about completed tasks, it's just extra load on master and mesos-dns.
> 2. Make history size configurable.
> 3. Make JSON serialization faster. With 10000s of tasks even without history
> it would take a lot of time to serialize tasks for mesos-dns. Doing it every
> 60 seconds instead of every 5 seconds isn't really an option.
> 4. Create event bus for mesos master. Marathon has it and it'd be nice to
> have it in Mesos. This way mesos-dns could avoid polling master state and
> switch to listening for events.
> All can be done independently.
> Note to mesosphere folks: please start distributing debug symbols with your
> distribution. I was asking for it for a while and it is really helpful:
> https://github.com/mesosphere/marathon/issues/1497#issuecomment-104182501
> Perf report for leading master:
> !http://i.imgur.com/iz7C3o0.png!
> I'm on 0.23.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
