[
https://issues.apache.org/jira/browse/MESOS-3589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-3589:
----------------------------
Description:
Today, I used a framework to fire off some tasks which each requested a lot of
ports. When doing so, the meson-master performance drops heavily.
With 70 tasks each requesting 100 ports, the master seems to spend the majority
(~90%) of its time dealing with merging and coalescing Value_Range objects.
I'll attach some text views of the output of 'perf' record/report from the
meson-master during this. The call-graph trace for the most frequently sampled
call is:
{code}
- 4.42% mesos-master mesos-master [.]
google::protobuf::internal::RepeatedPtrFieldBase::size() const
- google::protobuf::internal::RepeatedPtrFieldBase::size() const
- 37.05% mesos::Value_Ranges::range_size() const
- 91.07% mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Range
const&)
- mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Ranges const&)
- 68.35% mesos::operator+=(mesos::Value_Ranges&,
mesos::Value_Ranges const&)
- 99.46% Option<mesos::Value_Ranges>
mesos::Resources::get<mesos::Value_Ranges>(std::string const&) const
- mesos::internal::model(mesos::Resources const&)
- 97.58% mesos::internal::model(mesos::internal::Task
const&)
mesos::internal::master::model(mesos::internal::master::Framework const&)
mesos::internal::master::Master::Http::state(process::http::Request const&)
const
mesos::internal::master::Master::initialize()::{lambda(process::http::Request
const&)#9}::operator()(process::http::Request const&) const
std::_Function_handler<process::Future<process::http::Response>
(process::http::Request const&),
mesos::internal::master::Master::initialize()::{lambda(process::http::Request
const&)#9}>::_M_invoke(std::_Any_data const&, process::http::Request const&)
std::function<process::Future<process::http::Response> (process::http::Request
const&)>::operator()(process::http::Request const&) const
process::ProcessBase::visit(process::HttpEvent
const&)
process::HttpEvent::visit(process::EventVisitor*)
const
process::ProcessBase::serve(process::Event const&)
process::ProcessManager::resume(process::ProcessBase*)
process::schedule(void*)
start_thread
{code}
The top ~80 call sides pretty much all relate to Value_Range handling, and
account for 90% of the runtime of the mesos-master.
was:
Today, I used a framework to fire off some tasks which each requested a lot of
ports. When doing so, the meson-master performance drops heavily.
With 70 tasks each requesting 100 ports, the master seems to spend the majority
(~90%) of its time dealing with merging and coalescing Value_Range objects.
I'll attach some text views of the output of 'perf' record/report from the
meson-master during this. The call-graph trace for the most frequently sampled
call is:
{{
- 4.42% mesos-master mesos-master [.]
google::protobuf::internal::RepeatedPtrFieldBase::size() const
- google::protobuf::internal::RepeatedPtrFieldBase::size() const
- 37.05% mesos::Value_Ranges::range_size() const
- 91.07% mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Range
const&)
- mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Ranges const&)
- 68.35% mesos::operator+=(mesos::Value_Ranges&,
mesos::Value_Ranges const&)
- 99.46% Option<mesos::Value_Ranges>
mesos::Resources::get<mesos::Value_Ranges>(std::string const&) const
- mesos::internal::model(mesos::Resources const&)
- 97.58% mesos::internal::model(mesos::internal::Task
const&)
mesos::internal::master::model(mesos::internal::master::Framework const&)
mesos::internal::master::Master::Http::state(process::http::Request const&)
const
mesos::internal::master::Master::initialize()::{lambda(process::http::Request
const&)#9}::operator()(process::http::Request const&) const
std::_Function_handler<process::Future<process::http::Response>
(process::http::Request const&),
mesos::internal::master::Master::initialize()::{lambda(process::http::Request
const&)#9}>::_M_invoke(std::_Any_data const&, process::http::Request const&)
std::function<process::Future<process::http::Response> (process::http::Request
const&)>::operator()(process::http::Request const&) const
process::ProcessBase::visit(process::HttpEvent
const&)
process::HttpEvent::visit(process::EventVisitor*)
const
process::ProcessBase::serve(process::Event const&)
process::ProcessManager::resume(process::ProcessBase*)
process::schedule(void*)
start_thread
}}
The top ~80 call sides pretty much all relate to Value_Range handling, and
account for 90% of the runtime of the mesos-master.
> Mesos tasks with many ports cause significant master performance problems
> -------------------------------------------------------------------------
>
> Key: MESOS-3589
> URL: https://issues.apache.org/jira/browse/MESOS-3589
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: James Mulcahy
> Attachments: mesos-master-perf-top-call-expanded.txt,
> mesos-master-perf.txt
>
>
> Today, I used a framework to fire off some tasks which each requested a lot
> of ports. When doing so, the meson-master performance drops heavily.
> With 70 tasks each requesting 100 ports, the master seems to spend the
> majority (~90%) of its time dealing with merging and coalescing Value_Range
> objects.
> I'll attach some text views of the output of 'perf' record/report from the
> meson-master during this. The call-graph trace for the most frequently
> sampled call is:
> {code}
> - 4.42% mesos-master mesos-master [.]
> google::protobuf::internal::RepeatedPtrFieldBase::size() const
>
>
> - google::protobuf::internal::RepeatedPtrFieldBase::size() const
>
>
>
> - 37.05% mesos::Value_Ranges::range_size() const
>
>
>
> - 91.07% mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Range
> const&)
>
>
> - mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Ranges
> const&)
>
>
> - 68.35% mesos::operator+=(mesos::Value_Ranges&,
> mesos::Value_Ranges const&)
>
>
> - 99.46% Option<mesos::Value_Ranges>
> mesos::Resources::get<mesos::Value_Ranges>(std::string const&) const
>
>
> - mesos::internal::model(mesos::Resources const&)
>
>
>
> - 97.58% mesos::internal::model(mesos::internal::Task
> const&)
>
>
>
> mesos::internal::master::model(mesos::internal::master::Framework const&)
>
>
>
>
> mesos::internal::master::Master::Http::state(process::http::Request const&)
> const
>
>
>
> mesos::internal::master::Master::initialize()::{lambda(process::http::Request
> const&)#9}::operator()(process::http::Request const&) const
>
>
>
> std::_Function_handler<process::Future<process::http::Response>
> (process::http::Request const&),
> mesos::internal::master::Master::initialize()::{lambda(process::http::Request
> const&)#9}>::_M_invoke(std::_Any_data const&, process::http::Request const&)
>
>
> std::function<process::Future<process::http::Response>
> (process::http::Request const&)>::operator()(process::http::Request const&)
> const
>
> process::ProcessBase::visit(process::HttpEvent
> const&)
>
>
>
> process::HttpEvent::visit(process::EventVisitor*) const
>
>
>
> process::ProcessBase::serve(process::Event
> const&)
>
>
>
> process::ProcessManager::resume(process::ProcessBase*)
>
>
>
> process::schedule(void*)
>
>
>
> start_thread
>
>
>
> {code}
> The top ~80 call sides pretty much all relate to Value_Range handling, and
> account for 90% of the runtime of the mesos-master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
