[jira] [Commented] (MESOS-9935) The agent crashes after the disk du isolator supporting rootfs checks.
[
https://issues.apache.org/jira/browse/MESOS-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905794#comment-16905794
]
James Peach commented on MESOS-9935:
This reproduces if you run a task without any disk resource.
> The agent crashes after the disk du isolator supporting rootfs checks.
> --
>
> Key: MESOS-9935
> URL: https://issues.apache.org/jira/browse/MESOS-9935
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Gilbert Song
>Assignee: James Peach
>Priority: Blocker
>
> This issue was broken by this patch:
> https://github.com/apache/mesos/commit/8ba0682521c6051b42f33b3dd96a37f4d46a290d#diff-33089e53bdf9f646cdb9317c212eda02
> A task can be launched without disk resource. However, after this patch, if
> the disk resource does not exist, the agent crashes - because the info->paths
> only add an entry 'path' when there is a quota and the quota comes from the
> disk resource.
> {noformat}
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: F0809 14:54:00.017730 15498 process.cpp:3057] Aborting
> libprocess: 'posix-disk-isolator(1)@172.12.2.196:5051' threw exception:
> _Map_base::at
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: *** Check failure stack trace: ***
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7d585cd google::LogMessage::Fail()
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7d5a828 google::LogMessage::SendToLog()
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7d58163 google::LogMessage::Flush()
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7d5b169
> google::LogMessageFatal::~LogMessageFatal()
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7cb8dbd process::ProcessManager::resume()
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7cbe926
> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f3976070 (unknown)
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f3194e25 start_thread
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f2ebebad __clone
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (MESOS-9935) The agent crashes after the disk du isolator supporting rootfs checks.
[
https://issues.apache.org/jira/browse/MESOS-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Peach reassigned MESOS-9935:
--
Assignee: James Peach
> The agent crashes after the disk du isolator supporting rootfs checks.
> --
>
> Key: MESOS-9935
> URL: https://issues.apache.org/jira/browse/MESOS-9935
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Gilbert Song
>Assignee: James Peach
>Priority: Blocker
>
> This issue was broken by this patch:
> https://github.com/apache/mesos/commit/8ba0682521c6051b42f33b3dd96a37f4d46a290d#diff-33089e53bdf9f646cdb9317c212eda02
> A task can be launched without disk resource. However, after this patch, if
> the disk resource does not exist, the agent crashes - because the info->paths
> only add an entry 'path' when there is a quota and the quota comes from the
> disk resource.
> {noformat}
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: F0809 14:54:00.017730 15498 process.cpp:3057] Aborting
> libprocess: 'posix-disk-isolator(1)@172.12.2.196:5051' threw exception:
> _Map_base::at
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: *** Check failure stack trace: ***
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7d585cd google::LogMessage::Fail()
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7d5a828 google::LogMessage::SendToLog()
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7d58163 google::LogMessage::Flush()
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7d5b169
> google::LogMessageFatal::~LogMessageFatal()
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7cb8dbd process::ProcessManager::resume()
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f7cbe926
> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f3976070 (unknown)
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f3194e25 start_thread
> Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal
> mesos-agent[15492]: @ 0x7f65f2ebebad __clone
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (MESOS-9935) The agent crashes after the disk du isolator supporting rootfs checks.
Gilbert Song created MESOS-9935:
---
Summary: The agent crashes after the disk du isolator supporting
rootfs checks.
Key: MESOS-9935
URL: https://issues.apache.org/jira/browse/MESOS-9935
Project: Mesos
Issue Type: Bug
Components: containerization
Reporter: Gilbert Song
This issue was broken by this patch:
https://github.com/apache/mesos/commit/8ba0682521c6051b42f33b3dd96a37f4d46a290d#diff-33089e53bdf9f646cdb9317c212eda02
A task can be launched without disk resource. However, after this patch, if the
disk resource does not exist, the agent crashes - because the info->paths only
add an entry 'path' when there is a quota and the quota comes from the disk
resource.
{noformat}
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
F0809 14:54:00.017730 15498 process.cpp:3057] Aborting libprocess:
'posix-disk-isolator(1)@172.12.2.196:5051' threw exception: _Map_base::at
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
*** Check failure stack trace: ***
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
@ 0x7f65f7d585cd google::LogMessage::Fail()
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
@ 0x7f65f7d5a828 google::LogMessage::SendToLog()
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
@ 0x7f65f7d58163 google::LogMessage::Flush()
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
@ 0x7f65f7d5b169 google::LogMessageFatal::~LogMessageFatal()
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
@ 0x7f65f7cb8dbd process::ProcessManager::resume()
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
@ 0x7f65f7cbe926
_ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
@ 0x7f65f3976070 (unknown)
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
@ 0x7f65f3194e25 start_thread
Aug 09 14:54:00 ip-172-12-2-196.us-west-2.compute.internal mesos-agent[15492]:
@ 0x7f65f2ebebad __clone
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (MESOS-9934) Master does not handle returning unreachable agents as draining/deactivated
Joseph Wu created MESOS-9934:
Summary: Master does not handle returning unreachable agents as
draining/deactivated
Key: MESOS-9934
URL: https://issues.apache.org/jira/browse/MESOS-9934
Project: Mesos
Issue Type: Bug
Components: master
Reporter: Joseph Wu
Assignee: Joseph Wu
The master has two code paths for handling agent reregistration messages, one
culminating in {{Master::___reregisterSlave}} and the other in
{{Master::}}{{__reregisterSlave}}. The two paths are not continuations of each
other. Looks like we missed the double-underscore case in the initial
implementation. This is the path that unreachable agents take, when/if they
come back to the cluster. The result is that when unreachable agents are
marked for draining, they do not get sent the appropriate message unless they
are forced to reregister again (i.e. restarted manually).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (MESOS-9339) SSL (TLS) peer reverse DNS lookup can block the event loop thread.
[ https://issues.apache.org/jira/browse/MESOS-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905433#comment-16905433 ] Benno Evers edited comment on MESOS-9339 at 8/12/19 5:54 PM: - This is partially resolved in Mesos 1.9 by https://reviews.apache.org/r/70749/ , which eliminates rDNS lookups for incoming TLS connections when setting `LIBPROCESS_SSL_HOSTNAME_VALIDATION_SCHEME=openssl`. We can probably close this once we change the default for that setting from `legacy` to `openssl`. was (Author: bennoe): This is partially resolved in Mesos 1.9 by https://reviews.apache.org/r/70749/ , which eliminates rDNS lookups for incoming TLS connections when setting `LIBPROCESS_SSL_HOSTNAME_VALIDATION_SCHEME=openssl`. We can probably close this once we change the default for that ticket from `legacy` to `openssl`. > SSL (TLS) peer reverse DNS lookup can block the event loop thread. > -- > > Key: MESOS-9339 > URL: https://issues.apache.org/jira/browse/MESOS-9339 > Project: Mesos > Issue Type: Bug > Components: libprocess >Reporter: Benjamin Mahler >Priority: Major > Labels: foundations > > We currently look up the peer hostname in order to perform certificate > verification while accepting SSL (TLS) connections. This blocks the event > loop thread in cases where it has to go over the network. We saw one issue > where a misconfiguration meant that this would block for 15 seconds. > Once we add asynchronous DNS lookup facilities (MESOS-9338), we can use them > to avoid blocking the event loop thread. > We should consider logging slow DNS reverse lookups and adding timing metrics > for them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MESOS-9339) SSL (TLS) peer reverse DNS lookup can block the event loop thread.
[ https://issues.apache.org/jira/browse/MESOS-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905433#comment-16905433 ] Benno Evers commented on MESOS-9339: This is partially resolved in Mesos 1.9 by https://reviews.apache.org/r/70749/ , which eliminates rDNS lookups for incoming TLS connections when setting `LIBPROCESS_SSL_HOSTNAME_VALIDATION_SCHEME=openssl`. We can probably close this once we change the default for that ticket from `legacy` to `openssl`. > SSL (TLS) peer reverse DNS lookup can block the event loop thread. > -- > > Key: MESOS-9339 > URL: https://issues.apache.org/jira/browse/MESOS-9339 > Project: Mesos > Issue Type: Bug > Components: libprocess >Reporter: Benjamin Mahler >Priority: Major > Labels: foundations > > We currently look up the peer hostname in order to perform certificate > verification while accepting SSL (TLS) connections. This blocks the event > loop thread in cases where it has to go over the network. We saw one issue > where a misconfiguration meant that this would block for 15 seconds. > Once we add asynchronous DNS lookup facilities (MESOS-9338), we can use them > to avoid blocking the event loop thread. > We should consider logging slow DNS reverse lookups and adding timing metrics > for them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MESOS-9852) Slow memory growth in master due to deferred deletion of offer filters and timers.
[
https://issues.apache.org/jira/browse/MESOS-9852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905316#comment-16905316
]
Benjamin Mahler commented on MESOS-9852:
{quote}
Do you mean max_*_tasks_per_framework? Would this history take hundreds of MBs?
I'll try...
{quote}
Yes, for task history:
{noformat}
--max_completed_frameworks
--max_completed_tasks_per_framework
{noformat}
{quote}
I found that every terminated(no matter completed or unreachable) task would be
put into slaves.unreachableTasks and would only be erased in _doRegistryGc.
{quote}
This will only happen for unreachable agents. Please file a ticket if you see
otherwise. cc [~greggomann] [~vinodkone]
At this point I don't see the leak described in this ticket in the memory
profiling data, so we can continue the discussion on the mailing list or in
slack, to avoid spamming the watchers of this ticket.
> Slow memory growth in master due to deferred deletion of offer filters and
> timers.
> --
>
> Key: MESOS-9852
> URL: https://issues.apache.org/jira/browse/MESOS-9852
> Project: Mesos
> Issue Type: Bug
> Components: allocation, master
>Reporter: Benjamin Mahler
>Assignee: Benjamin Mahler
>Priority: Critical
> Labels: resource-management
> Fix For: 1.5.4, 1.6.3, 1.7.3, 1.8.1, 1.9.0
>
> Attachments: _tmp_libprocess.Do1MrG_profile (1).dump,
> _tmp_libprocess.Do1MrG_profile (1).svg, _tmp_libprocess.Do1MrG_profile
> 24hours.dump, _tmp_libprocess.Do1MrG_profile 24hours.svg, screenshot-1.png,
> statistics
>
>
> The allocator does not keep a handle to the offer filter timer, which means
> it cannot remove the timer overhead (in this case memory) when removing the
> offer filter earlier (e.g. due to revive):
> https://github.com/apache/mesos/blob/1.8.0/src/master/allocator/mesos/hierarchical.cpp#L1338-L1352
> In addition, the offer filter is allocated on the heap but not deleted until
> the timer fires (which might take forever!):
> https://github.com/apache/mesos/blob/1.8.0/src/master/allocator/mesos/hierarchical.cpp#L1321
> https://github.com/apache/mesos/blob/1.8.0/src/master/allocator/mesos/hierarchical.cpp#L1408-L1413
> https://github.com/apache/mesos/blob/1.8.0/src/master/allocator/mesos/hierarchical.cpp#L2249
> We'll need to try to backport this to all active release branches.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MESOS-9560) ContentType/AgentAPITest.MarkResourceProviderGone/1 is flaky
[
https://issues.apache.org/jira/browse/MESOS-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905214#comment-16905214
]
Benjamin Bannier commented on MESOS-9560:
-
Reopening since we are still observing similar failures.
> ContentType/AgentAPITest.MarkResourceProviderGone/1 is flaky
>
>
> Key: MESOS-9560
> URL: https://issues.apache.org/jira/browse/MESOS-9560
> Project: Mesos
> Issue Type: Bug
> Components: test
>Reporter: Benjamin Bannier
>Assignee: Benjamin Bannier
>Priority: Critical
> Labels: flaky, flaky-test, mesosphere, storage, test
> Fix For: 1.9.0
>
> Attachments: consoleText.txt
>
>
> We observed a segfault in
> {{ContentType/AgentAPITest.MarkResourceProviderGone/1}} on test teardown.
> {noformat}
> I0131 23:55:59.378453 6798 slave.cpp:923] Agent terminating
> I0131 23:55:59.378813 31143 master.cpp:1269] Agent
> a27bcaba-70cc-4ec3-9786-38f9512c61fd-S0 at slave(1112)@172.16.10.236:43229
> (ip-172-16-10-236.ec2.internal) disconnected
> I0131 23:55:59.378831 31143 master.cpp:3272] Disconnecting agent
> a27bcaba-70cc-4ec3-9786-38f9512c61fd-S0 at slave(1112)@172.16.10.236:43229
> (ip-172-16-10-236.ec2.internal)
> I0131 23:55:59.378846 31143 master.cpp:3291] Deactivating agent
> a27bcaba-70cc-4ec3-9786-38f9512c61fd-S0 at slave(1112)@172.16.10.236:43229
> (ip-172-16-10-236.ec2.internal)
> I0131 23:55:59.378891 31143 hierarchical.cpp:793] Agent
> a27bcaba-70cc-4ec3-9786-38f9512c61fd-S0 deactivated
> F0131 23:55:59.378891 31149 logging.cpp:67] RAW: Pure virtual method called
> @ 0x7f633aaaebdd google::LogMessage::Fail()
> @ 0x7f633aab6281 google::RawLog__()
> @ 0x7f6339821262 __cxa_pure_virtual
> @ 0x55671cacc113
> testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith()
> @ 0x55671b532e78
> mesos::internal::tests::resource_provider::MockResourceProvider<>::disconnected()
> @ 0x7f633978f6b0 process::AsyncExecutorProcess::execute<>()
> @ 0x7f633979f218
> _ZN5cpp176invokeIZN7process8dispatchI7NothingNS1_20AsyncExecutorProcessERKSt8functionIFvvEES9_EENS1_6FutureIT_EERKNS1_3PIDIT0_EEMSE_FSB_T1_EOT2_EUlSt10unique_ptrINS1_7PromiseIS3_EESt14default_deleteISP_EEOS7_PNS1_11ProcessBaseEE_JSS_S7_SV_EEEDTclcl7forwardISB_Efp_Espcl7forwardIT0_Efp0_EEEOSB_DpOSX_
> @ 0x7f633a9f5d01 process::ProcessBase::consume()
> @ 0x7f633aa1a08a process::ProcessManager::resume()
> @ 0x7f633aa1db06
> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
> @ 0x7f633acc9f80 execute_native_thread_routine
> @ 0x7f6337142e25 start_thread
> @ 0x7f6336241bad __clone
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (MESOS-9933) Installed boost is preferred over bundled version.
Till Toenshoff created MESOS-9933:
-
Summary: Installed boost is preferred over bundled version.
Key: MESOS-9933
URL: https://issues.apache.org/jira/browse/MESOS-9933
Project: Mesos
Issue Type: Bug
Components: build
Affects Versions: 1.5.4
Reporter: Till Toenshoff
When building Mesos 1.5.x, I noticed my local build failing with;
{noformat}
../../../3rdparty/libprocess/../stout/include/stout/json.hpp:270:7: error: no
matching constructor for initialization of 'internal::Variant' (aka
'variant, boost::recursive_wrapper,
boost::recursive_wrapper, boost::recursive_wrapper,
boost::recursive_wrapper, boost::recursive_wrapper >')
: internal::Variant(value) {}
^ ~
{noformat}
It turned out that I had a more recent boost version installed on my system and
instead of using the bundled variant, the autotools build preferred my
installed version, leading to the above error. The installed version was in
{{/usr/local/include}}. After removing the installed variant, the build
continued, avoiding the error.
We should try to make sure bundled dependencies are generally preferred over
installed variants.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MESOS-9852) Slow memory growth in master due to deferred deletion of offer filters and timers.
[ https://issues.apache.org/jira/browse/MESOS-9852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904921#comment-16904921 ] longfei commented on MESOS-9852: I found that every terminated(no matter completed or unreachable) task would be put into slaves.unreachableTasks and would only be erased in _doRegistryGc. So only when an agent becomes unreachable or gone will its unreachableTasks(in master's memory) be released. If all agents are working properly, the master's memory will keep growing because of slaves.unreachableTasks. Am I right? > Slow memory growth in master due to deferred deletion of offer filters and > timers. > -- > > Key: MESOS-9852 > URL: https://issues.apache.org/jira/browse/MESOS-9852 > Project: Mesos > Issue Type: Bug > Components: allocation, master >Reporter: Benjamin Mahler >Assignee: Benjamin Mahler >Priority: Critical > Labels: resource-management > Fix For: 1.5.4, 1.6.3, 1.7.3, 1.8.1, 1.9.0 > > Attachments: _tmp_libprocess.Do1MrG_profile (1).dump, > _tmp_libprocess.Do1MrG_profile (1).svg, _tmp_libprocess.Do1MrG_profile > 24hours.dump, _tmp_libprocess.Do1MrG_profile 24hours.svg, screenshot-1.png, > statistics > > > The allocator does not keep a handle to the offer filter timer, which means > it cannot remove the timer overhead (in this case memory) when removing the > offer filter earlier (e.g. due to revive): > https://github.com/apache/mesos/blob/1.8.0/src/master/allocator/mesos/hierarchical.cpp#L1338-L1352 > In addition, the offer filter is allocated on the heap but not deleted until > the timer fires (which might take forever!): > https://github.com/apache/mesos/blob/1.8.0/src/master/allocator/mesos/hierarchical.cpp#L1321 > https://github.com/apache/mesos/blob/1.8.0/src/master/allocator/mesos/hierarchical.cpp#L1408-L1413 > https://github.com/apache/mesos/blob/1.8.0/src/master/allocator/mesos/hierarchical.cpp#L2249 > We'll need to try to backport this to all active release branches. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
