[
https://issues.apache.org/jira/browse/MESOS-3589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14951100#comment-14951100
]
James Mulcahy commented on MESOS-3589:
--------------------------------------
As above, I've back ported the patch and the results are as follows:
Random Ports
~5.8-10.4s (results were: 10.4, 5.8, 8.3, 8.0)
Sequential Ports
~3.5-8.3s (results were: 6, 7.7, 8.3, 3.5)
In summary, the patch makes a huge improvement to the Value_Range handling.
Schedulers which randomize ports still see worse state.json fetch times, but
it's no longer ~4x slower, it's closer to ~1.5x slower.
I'll leave it to the mesos team to judge whether this should be marked as
fixed, or whether they want to keep it open to track the potential for further
improvements here?
> Mesos tasks with many ports cause significant master performance problems
> -------------------------------------------------------------------------
>
> Key: MESOS-3589
> URL: https://issues.apache.org/jira/browse/MESOS-3589
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: James Mulcahy
> Attachments: mesos-master-perf-top-call-expanded.txt,
> mesos-master-perf.txt
>
>
> Today, I used a framework to fire off some tasks which each requested a lot
> of ports. When doing so, the meson-master performance drops heavily.
> With 70 tasks each requesting 100 ports, the master seems to spend the
> majority (~90%) of its time dealing with merging and coalescing Value_Range
> objects.
> I'll attach some text views of the output of 'perf' record/report from the
> meson-master during this. The call-graph trace for the most frequently
> sampled call is:
> {code}
> - 4.42% mesos-master mesos-master [.]
> google::protobuf::internal::RepeatedPtrFieldBase::size() const
>
>
> - google::protobuf::internal::RepeatedPtrFieldBase::size() const
>
>
>
> - 37.05% mesos::Value_Ranges::range_size() const
>
>
>
> - 91.07% mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Range
> const&)
>
>
> - mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Ranges
> const&)
>
>
> - 68.35% mesos::operator+=(mesos::Value_Ranges&,
> mesos::Value_Ranges const&)
>
>
> - 99.46% Option<mesos::Value_Ranges>
> mesos::Resources::get<mesos::Value_Ranges>(std::string const&) const
>
>
> - mesos::internal::model(mesos::Resources const&)
>
>
>
> - 97.58% mesos::internal::model(mesos::internal::Task
> const&)
>
>
>
> mesos::internal::master::model(mesos::internal::master::Framework const&)
>
>
>
>
> mesos::internal::master::Master::Http::state(process::http::Request const&)
> const
>
>
>
> mesos::internal::master::Master::initialize()::{lambda(process::http::Request
> const&)#9}::operator()(process::http::Request const&) const
>
>
>
> std::_Function_handler<process::Future<process::http::Response>
> (process::http::Request const&),
> mesos::internal::master::Master::initialize()::{lambda(process::http::Request
> const&)#9}>::_M_invoke(std::_Any_data const&, process::http::Request const&)
>
>
> std::function<process::Future<process::http::Response>
> (process::http::Request const&)>::operator()(process::http::Request const&)
> const
>
> process::ProcessBase::visit(process::HttpEvent
> const&)
>
>
>
> process::HttpEvent::visit(process::EventVisitor*) const
>
>
>
> process::ProcessBase::serve(process::Event
> const&)
>
>
>
> process::ProcessManager::resume(process::ProcessBase*)
>
>
>
> process::schedule(void*)
>
>
>
> start_thread
>
>
>
> {code}
> The top ~80 call sides pretty much all relate to Value_Range handling, and
> account for 90% of the runtime of the mesos-master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
