Re: Review Request 72239: Added stack trace logging to program crashes on windows.

2020-03-23 Thread Andrei Sekretenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72239/#review220059
---


Ship it!




Ship It!

- Andrei Sekretenko


On March 21, 2020, 12:22 a.m., Benjamin Mahler wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72239/
> ---
> 
> (Updated March 21, 2020, 12:22 a.m.)
> 
> 
> Review request for mesos, Andrei Sekretenko, Greg Mann, and Joseph Wu.
> 
> 
> Bugs: MESOS-7245
> https://issues.apache.org/jira/browse/MESOS-7245
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> For now, this starts with adding the necessary code to logging.cpp up
> in mesos. This gives us stack traces for any mesos binaries or
> libraries that initialize logging with `installFailureSignalHandler`
> set to true.
> 
> Ideally, we would either (1) move this into stout, or (2) upstream
> some changes to glog. Both approaches would allow us to also get
> stack traces from stout tests and libprocess tests.
> 
> For now however, this patch is just aiming to fix the urgent issue
> of a lack of stack traces for mesos crashes.
> 
> 
> Diffs
> -
> 
>   src/logging/logging.cpp baca51b3956dbaa282c03e73a2c261ff103f4821 
> 
> 
> Diff: https://reviews.apache.org/r/72239/diff/4/
> 
> 
> Testing
> ---
> 
> Ran through CI and locally, for the recent reservation update crash this 
> produces:
> 
> ```
> E 00:00:00.00 18216 logging.cpp:295] RAW: Received fatal exception 
> 0xc005 EXCEPTION_ACCESS_VIOLATION
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> google::protobuf::internal::RepeatedPtrIterator  const >::operator* [7FF75279306E+14] 
> (c:\users\bmahler\git\mesos\build\3rdparty\protobuf-3.5.0\src\protobuf-3.5.0\src\google\protobuf\repeated_field.h:2266)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::authorization::ActionObject::reserve [7FF753C94880+720] 
> (c:\users\bmahler\git\mesos\src\master\authorization.cpp:236)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::_reserve [7FF753D007CF+1423] 
> (c:\users\bmahler\git\mesos\src\master\http.cpp:1960)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::reserveResources 
> [7FF753D07FFC+380] (c:\users\bmahler\git\mesos\src\master\http.cpp:1984)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::api [7FF753CED99B+6571] 
> (c:\users\bmahler\git\mesos\src\master\http.cpp:352)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> ::operator() [7FF7527AA2E4+100] 
> (c:\users\bmahler\git\mesos\src\master\master.cpp:909)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_functor::_Call< 
> &,process::http::Request const 
> &,Option const &> 
> [7FF75262D61A+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::invoke< 
> &,process::http::Request const 
> &,Option const &> 
> [7FF7526E55AA+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_ret,0>::_Call<
>  &,process::http::Request const 
> &,Option const &> 
> [7FF75262D55A+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Func_impl_no_alloc<,process::Future,process::http::Request
>  const &,Option const &>::_Do_call 
> [7FF7527EDF2F+111] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Func_class,process::http::Request
>  const &,Option const 
> &>::operator() [7FF75472BAAD+157] (c:\program files (x86)\microsoft 
> visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> ::operator() [7FF75471911F+335] 
> (c:\users\bmahler\git\mesos\3rdparty\libprocess\src\process.cpp:3874)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_functor::_Call<,bool> 
> [7FF754638F1F+95] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::invoke<,bool> 
> [7FF75469AB2E+94] (c:\program files (x86)\microsoft visual 
>

Re: Review Request 72239: Added stack trace logging to program crashes on windows.

2020-03-20 Thread Mesos Reviewbot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72239/#review220052
---



Patch looks great!

Reviews applied: [72239]

Passed command: export OS='ubuntu:16.04' BUILDTOOL='autotools' COMPILER='gcc' 
CONFIGURATION='--verbose --disable-libtool-wrappers 
--disable-parallel-test-execution' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; 
./support/jenkins/buildbot.sh

- Mesos Reviewbot


On March 21, 2020, 12:22 a.m., Benjamin Mahler wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72239/
> ---
> 
> (Updated March 21, 2020, 12:22 a.m.)
> 
> 
> Review request for mesos, Andrei Sekretenko, Greg Mann, and Joseph Wu.
> 
> 
> Bugs: MESOS-7245
> https://issues.apache.org/jira/browse/MESOS-7245
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> For now, this starts with adding the necessary code to logging.cpp up
> in mesos. This gives us stack traces for any mesos binaries or
> libraries that initialize logging with `installFailureSignalHandler`
> set to true.
> 
> Ideally, we would either (1) move this into stout, or (2) upstream
> some changes to glog. Both approaches would allow us to also get
> stack traces from stout tests and libprocess tests.
> 
> For now however, this patch is just aiming to fix the urgent issue
> of a lack of stack traces for mesos crashes.
> 
> 
> Diffs
> -
> 
>   src/logging/logging.cpp baca51b3956dbaa282c03e73a2c261ff103f4821 
> 
> 
> Diff: https://reviews.apache.org/r/72239/diff/4/
> 
> 
> Testing
> ---
> 
> Ran through CI and locally, for the recent reservation update crash this 
> produces:
> 
> ```
> E 00:00:00.00 18216 logging.cpp:295] RAW: Received fatal exception 
> 0xc005 EXCEPTION_ACCESS_VIOLATION
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> google::protobuf::internal::RepeatedPtrIterator  const >::operator* [7FF75279306E+14] 
> (c:\users\bmahler\git\mesos\build\3rdparty\protobuf-3.5.0\src\protobuf-3.5.0\src\google\protobuf\repeated_field.h:2266)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::authorization::ActionObject::reserve [7FF753C94880+720] 
> (c:\users\bmahler\git\mesos\src\master\authorization.cpp:236)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::_reserve [7FF753D007CF+1423] 
> (c:\users\bmahler\git\mesos\src\master\http.cpp:1960)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::reserveResources 
> [7FF753D07FFC+380] (c:\users\bmahler\git\mesos\src\master\http.cpp:1984)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::api [7FF753CED99B+6571] 
> (c:\users\bmahler\git\mesos\src\master\http.cpp:352)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> ::operator() [7FF7527AA2E4+100] 
> (c:\users\bmahler\git\mesos\src\master\master.cpp:909)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_functor::_Call< 
> &,process::http::Request const 
> &,Option const &> 
> [7FF75262D61A+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::invoke< 
> &,process::http::Request const 
> &,Option const &> 
> [7FF7526E55AA+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_ret,0>::_Call<
>  &,process::http::Request const 
> &,Option const &> 
> [7FF75262D55A+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Func_impl_no_alloc<,process::Future,process::http::Request
>  const &,Option const &>::_Do_call 
> [7FF7527EDF2F+111] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Func_class,process::http::Request
>  const &,Option const 
> &>::operator() [7FF75472BAAD+157] (c:\program files (x86)\microsoft 
> visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> ::operator() [7FF75471911F+335] 
> (c:\users\bmahler\git\mesos\3rdparty\libprocess\src\process.cpp:3874)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_functor::_Call<,bool> 
> [7FF754638F1F+95] (c:\program files (x86)\microsoft

Re: Review Request 72239: Added stack trace logging to program crashes on windows.

2020-03-20 Thread Benjamin Mahler 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72239/
---

(Updated March 21, 2020, 12:22 a.m.)


Review request for mesos, Andrei Sekretenko, Greg Mann, and Joseph Wu.


Changes
---

- Added spin lock for dbghelp.h functions.
- Fixed a bad sizeof(trace) that was incorrectly getting the array size
- Added logging when the stack trace is clipped.
- Added documentation.
- Other minor changes.


Bugs: MESOS-7245
https://issues.apache.org/jira/browse/MESOS-7245


Repository: mesos


Description
---

For now, this starts with adding the necessary code to logging.cpp up
in mesos. This gives us stack traces for any mesos binaries or
libraries that initialize logging with `installFailureSignalHandler`
set to true.

Ideally, we would either (1) move this into stout, or (2) upstream
some changes to glog. Both approaches would allow us to also get
stack traces from stout tests and libprocess tests.

For now however, this patch is just aiming to fix the urgent issue
of a lack of stack traces for mesos crashes.


Diffs (updated)
-

  src/logging/logging.cpp baca51b3956dbaa282c03e73a2c261ff103f4821 


Diff: https://reviews.apache.org/r/72239/diff/4/

Changes: https://reviews.apache.org/r/72239/diff/3-4/


Testing
---

Ran through CI and locally, for the recent reservation update crash this 
produces:

```
E 00:00:00.00 18216 logging.cpp:295] RAW: Received fatal exception 
0xc005 EXCEPTION_ACCESS_VIOLATION
E 00:00:00.00 18216 logging.cpp:320] RAW:   
google::protobuf::internal::RepeatedPtrIterator::operator* [7FF75279306E+14] 
(c:\users\bmahler\git\mesos\build\3rdparty\protobuf-3.5.0\src\protobuf-3.5.0\src\google\protobuf\repeated_field.h:2266)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
mesos::authorization::ActionObject::reserve [7FF753C94880+720] 
(c:\users\bmahler\git\mesos\src\master\authorization.cpp:236)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
mesos::internal::master::Master::Http::_reserve [7FF753D007CF+1423] 
(c:\users\bmahler\git\mesos\src\master\http.cpp:1960)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
mesos::internal::master::Master::Http::reserveResources [7FF753D07FFC+380] 
(c:\users\bmahler\git\mesos\src\master\http.cpp:1984)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
mesos::internal::master::Master::Http::api [7FF753CED99B+6571] 
(c:\users\bmahler\git\mesos\src\master\http.cpp:352)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
::operator() [7FF7527AA2E4+100] 
(c:\users\bmahler\git\mesos\src\master\master.cpp:909)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Invoker_functor::_Call< 
&,process::http::Request const 
&,Option const &> 
[7FF75262D61A+122] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::invoke< &,process::http::Request 
const &,Option const &> 
[7FF7526E55AA+122] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Invoker_ret,0>::_Call<
 &,process::http::Request const 
&,Option const &> 
[7FF75262D55A+122] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Func_impl_no_alloc<,process::Future,process::http::Request
 const &,Option const &>::_Do_call 
[7FF7527EDF2F+111] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Func_class,process::http::Request
 const &,Option const &>::operator() 
[7FF75472BAAD+157] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
::operator() [7FF75471911F+335] 
(c:\users\bmahler\git\mesos\3rdparty\libprocess\src\process.cpp:3874)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Invoker_functor::_Call<,bool> 
[7FF754638F1F+95] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::invoke<,bool> 
[7FF75469AB2E+94] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
lambda::internal::Partial<,bool>::invoke_expand<,std::tuple,std::tuple<>,0>
 [7FF7546A2B9A+138]

Re: Review Request 72239: Added stack trace logging to program crashes on windows.

2020-03-20 Thread Benjamin Mahler 


> On March 20, 2020, 12:54 p.m., Andrei Sekretenko wrote:
> >

Great review, thanks for the suggestions Andrei!


> On March 20, 2020, 12:54 p.m., Andrei Sekretenko wrote:
> > src/logging/logging.cpp
> > Lines 115 (patched)
> > 
> >
> > Do I understand correctly that all the exception filter code tries to 
> > avoid dynamic allocation at all costs? 
> > 
> > Maybe  leave a NOTE? Or is it obvious?

Yes I tried to for the same reason as on POSIX (it's not async-signal-safe). I 
imagined some windows exceptions could arrive when a memory allocation lock is 
held (e.g. interrupt or user sent signal). I'll write this down.


> On March 20, 2020, 12:54 p.m., Andrei Sekretenko wrote:
> > src/logging/logging.cpp
> > Lines 117 (patched)
> > 
> >
> > Wrapped link (here and below) can be hard to copy into browser.
> > 
> > Not sure what our styleguide says; checked other parts of Mesos 
> > codebase - links like this are typically kept on a single line with `// 
> > NOLINT` at the end.

Yeah, I've personally found it cleaner to wrap them when I've added links, and 
I don't think it's a big burden to open them (especially since github and many 
IDEs don't seem to make them clickable), but I can use the NOLINT approach.


> On March 20, 2020, 12:54 p.m., Andrei Sekretenko wrote:
> > src/logging/logging.cpp
> > Lines 234 (patched)
> > 
> >
> > Do I understand correctly that each thread has its own unhandled 
> > exception filter (which we reset for all threads with one call of 
> > `SetUnhandledExceptionFilter`)
> > 
> > 
> > https://docs.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-setunhandledexceptionfilter
> > 
> > but the functions of `dbghelp.h` that we use are not thread-safe?
> > 
> > https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symgetlinefromaddr64
> > 
> > Doesn't this mean that we need to synchronize access to `dbghelp` or, 
> > at least, to make sure that concurrent run is dropped?
> > 
> > Given the nature of Mesos, I would not be surprised if we will 
> > eventually run into a pair of simultaneous exceptions (probably triggered 
> > by a common cause) in two threads. 
> > 
> > Having only one stack of two is probably tolerable, but we would want 
> > to avoid getting garbadge/triggering UB at all costs.

Good find!


> On March 20, 2020, 12:54 p.m., Andrei Sekretenko wrote:
> > src/logging/logging.cpp
> > Lines 308 (patched)
> > 
> >
> > In the not very likely case of a very deep stack, can we somehow 
> > explicitly inform the reader that the trace was clipped due to this 
> > limitation?

Good suggestion, thanks!


> On March 20, 2020, 12:54 p.m., Andrei Sekretenko wrote:
> > src/logging/logging.cpp
> > Lines 449 (patched)
> > 
> >
> > This function returns the previously installed exception filter, if 
> > there was one; do you have any idea whether we need to care of this or not, 
> > and why?
> > 
> > I would naively expect that there should be no handler initially. 
> > Probably, if the returned value is not NULL, this means that we don't 
> > know what is going on (i.e. what 3rd-party library, after we updated it, 
> > started to install its own filter before us) and should just crash here?

Since we don't have a notion of disabling the failure handling, I didn't add 
any preserve / restore code.

Tried adding a check, turns out it does not return NULL, not sure what is 
getting set up here (maybe a libc thing for mapping to signals?)


- Benjamin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72239/#review219968
---


On March 17, 2020, 12:09 a.m., Benjamin Mahler wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72239/
> ---
> 
> (Updated March 17, 2020, 12:09 a.m.)
> 
> 
> Review request for mesos, Andrei Sekretenko, Greg Mann, and Joseph Wu.
> 
> 
> Bugs: MESOS-7245
> https://issues.apache.org/jira/browse/MESOS-7245
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> For now, this starts with adding the necessary code to logging.cpp up
> in mesos. This gives us stack traces for any mesos binaries or
> libraries that initialize logging with `installFailureSignalHandler`
> set to true.
> 
> Ideally, we would either (1) move this into stout, or

Re: Review Request 72239: Added stack trace logging to program crashes on windows.

2020-03-20 Thread Benjamin Mahler 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72239/#review220051
---




src/logging/logging.cpp
Lines 311 (patched)


This isn't the right size to pass in! since each `void*` is more than 1 
byte.


- Benjamin Mahler


On March 17, 2020, 12:09 a.m., Benjamin Mahler wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72239/
> ---
> 
> (Updated March 17, 2020, 12:09 a.m.)
> 
> 
> Review request for mesos, Andrei Sekretenko, Greg Mann, and Joseph Wu.
> 
> 
> Bugs: MESOS-7245
> https://issues.apache.org/jira/browse/MESOS-7245
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> For now, this starts with adding the necessary code to logging.cpp up
> in mesos. This gives us stack traces for any mesos binaries or
> libraries that initialize logging with `installFailureSignalHandler`
> set to true.
> 
> Ideally, we would either (1) move this into stout, or (2) upstream
> some changes to glog. Both approaches would allow us to also get
> stack traces from stout tests and libprocess tests.
> 
> For now however, this patch is just aiming to fix the urgent issue
> of a lack of stack traces for mesos crashes.
> 
> 
> Diffs
> -
> 
>   src/logging/logging.cpp baca51b3956dbaa282c03e73a2c261ff103f4821 
> 
> 
> Diff: https://reviews.apache.org/r/72239/diff/3/
> 
> 
> Testing
> ---
> 
> Ran through CI and locally, for the recent reservation update crash this 
> produces:
> 
> ```
> E 00:00:00.00 18216 logging.cpp:295] RAW: Received fatal exception 
> 0xc005 EXCEPTION_ACCESS_VIOLATION
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> google::protobuf::internal::RepeatedPtrIterator  const >::operator* [7FF75279306E+14] 
> (c:\users\bmahler\git\mesos\build\3rdparty\protobuf-3.5.0\src\protobuf-3.5.0\src\google\protobuf\repeated_field.h:2266)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::authorization::ActionObject::reserve [7FF753C94880+720] 
> (c:\users\bmahler\git\mesos\src\master\authorization.cpp:236)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::_reserve [7FF753D007CF+1423] 
> (c:\users\bmahler\git\mesos\src\master\http.cpp:1960)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::reserveResources 
> [7FF753D07FFC+380] (c:\users\bmahler\git\mesos\src\master\http.cpp:1984)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::api [7FF753CED99B+6571] 
> (c:\users\bmahler\git\mesos\src\master\http.cpp:352)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> ::operator() [7FF7527AA2E4+100] 
> (c:\users\bmahler\git\mesos\src\master\master.cpp:909)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_functor::_Call< 
> &,process::http::Request const 
> &,Option const &> 
> [7FF75262D61A+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::invoke< 
> &,process::http::Request const 
> &,Option const &> 
> [7FF7526E55AA+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_ret,0>::_Call<
>  &,process::http::Request const 
> &,Option const &> 
> [7FF75262D55A+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Func_impl_no_alloc<,process::Future,process::http::Request
>  const &,Option const &>::_Do_call 
> [7FF7527EDF2F+111] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Func_class,process::http::Request
>  const &,Option const 
> &>::operator() [7FF75472BAAD+157] (c:\program files (x86)\microsoft 
> visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> ::operator() [7FF75471911F+335] 
> (c:\users\bmahler\git\mesos\3rdparty\libprocess\src\process.cpp:3874)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_functor::_Call<,bool> 
> [7FF754638F1F+95] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E

Re: Review Request 72239: Added stack trace logging to program crashes on windows.

2020-03-20 Thread Andrei Sekretenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72239/#review219968
---




src/logging/logging.cpp
Lines 115 (patched)


Do I understand correctly that all the exception filter code tries to avoid 
dynamic allocation at all costs? 

Maybe  leave a NOTE? Or is it obvious?



src/logging/logging.cpp
Lines 117 (patched)


Wrapped link (here and below) can be hard to copy into browser.

Not sure what our styleguide says; checked other parts of Mesos codebase - 
links like this are typically kept on a single line with `// NOLINT` at the end.



src/logging/logging.cpp
Lines 234 (patched)


Do I understand correctly that each thread has its own unhandled exception 
filter (which we reset for all threads with one call of 
`SetUnhandledExceptionFilter`)


https://docs.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-setunhandledexceptionfilter

but the functions of `dbghelp.h` that we use are not thread-safe?

https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symgetlinefromaddr64

Doesn't this mean that we need to synchronize access to `dbghelp` or, at 
least, to make sure that concurrent run is dropped?

Given the nature of Mesos, I would not be surprised if we will eventually 
run into a pair of simultaneous exceptions (probably triggered by a common 
cause) in two threads. 

Having only one stack of two is probably tolerable, but we would want to 
avoid getting garbadge/triggering UB at all costs.



src/logging/logging.cpp
Lines 308 (patched)


In the not very likely case of a very deep stack, can we somehow explicitly 
inform the reader that the trace was clipped due to this limitation?



src/logging/logging.cpp
Lines 449 (patched)


This function returns the previously installed exception filter, if there 
was one; do you have any idea whether we need to care of this or not, and why?

I would naively expect that there should be no handler initially. 
Probably, if the returned value is not NULL, this means that we don't know 
what is going on (i.e. what 3rd-party library, after we updated it, started to 
install its own filter before us) and should just crash here?


- Andrei Sekretenko


On March 17, 2020, 12:09 a.m., Benjamin Mahler wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72239/
> ---
> 
> (Updated March 17, 2020, 12:09 a.m.)
> 
> 
> Review request for mesos, Andrei Sekretenko, Greg Mann, and Joseph Wu.
> 
> 
> Bugs: MESOS-7245
> https://issues.apache.org/jira/browse/MESOS-7245
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> For now, this starts with adding the necessary code to logging.cpp up
> in mesos. This gives us stack traces for any mesos binaries or
> libraries that initialize logging with `installFailureSignalHandler`
> set to true.
> 
> Ideally, we would either (1) move this into stout, or (2) upstream
> some changes to glog. Both approaches would allow us to also get
> stack traces from stout tests and libprocess tests.
> 
> For now however, this patch is just aiming to fix the urgent issue
> of a lack of stack traces for mesos crashes.
> 
> 
> Diffs
> -
> 
>   src/logging/logging.cpp baca51b3956dbaa282c03e73a2c261ff103f4821 
> 
> 
> Diff: https://reviews.apache.org/r/72239/diff/3/
> 
> 
> Testing
> ---
> 
> Ran through CI and locally, for the recent reservation update crash this 
> produces:
> 
> ```
> E 00:00:00.00 18216 logging.cpp:295] RAW: Received fatal exception 
> 0xc005 EXCEPTION_ACCESS_VIOLATION
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> google::protobuf::internal::RepeatedPtrIterator  const >::operator* [7FF75279306E+14] 
> (c:\users\bmahler\git\mesos\build\3rdparty\protobuf-3.5.0\src\protobuf-3.5.0\src\google\protobuf\repeated_field.h:2266)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::authorization::ActionObject::reserve [7FF753C94880+720] 
> (c:\users\bmahler\git\mesos\src\master\authorization.cpp:236)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::_reserve [7FF753D007CF+1423] 
> (c:\users\bmahler\git\mesos\src\master\http.cpp:1960)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::reserveResources 
> [7FF753D07FFC+380] (c:\users\bmahler\git\mesos\src\master\http.cpp:1984)

Re: Review Request 72239: Added stack trace logging to program crashes on windows.

2020-03-16 Thread Mesos Reviewbot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72239/#review219967
---



Patch looks great!

Reviews applied: [72239]

Passed command: export OS='ubuntu:16.04' BUILDTOOL='autotools' COMPILER='gcc' 
CONFIGURATION='--verbose --disable-libtool-wrappers 
--disable-parallel-test-execution' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; 
./support/jenkins/buildbot.sh

- Mesos Reviewbot


On March 17, 2020, 12:09 a.m., Benjamin Mahler wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72239/
> ---
> 
> (Updated March 17, 2020, 12:09 a.m.)
> 
> 
> Review request for mesos, Andrei Sekretenko, Greg Mann, and Joseph Wu.
> 
> 
> Bugs: MESOS-7245
> https://issues.apache.org/jira/browse/MESOS-7245
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> For now, this starts with adding the necessary code to logging.cpp up
> in mesos. This gives us stack traces for any mesos binaries or
> libraries that initialize logging with `installFailureSignalHandler`
> set to true.
> 
> Ideally, we would either (1) move this into stout, or (2) upstream
> some changes to glog. Both approaches would allow us to also get
> stack traces from stout tests and libprocess tests.
> 
> For now however, this patch is just aiming to fix the urgent issue
> of a lack of stack traces for mesos crashes.
> 
> 
> Diffs
> -
> 
>   src/logging/logging.cpp baca51b3956dbaa282c03e73a2c261ff103f4821 
> 
> 
> Diff: https://reviews.apache.org/r/72239/diff/3/
> 
> 
> Testing
> ---
> 
> Ran through CI and locally, for the recent reservation update crash this 
> produces:
> 
> ```
> E 00:00:00.00 18216 logging.cpp:295] RAW: Received fatal exception 
> 0xc005 EXCEPTION_ACCESS_VIOLATION
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> google::protobuf::internal::RepeatedPtrIterator  const >::operator* [7FF75279306E+14] 
> (c:\users\bmahler\git\mesos\build\3rdparty\protobuf-3.5.0\src\protobuf-3.5.0\src\google\protobuf\repeated_field.h:2266)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::authorization::ActionObject::reserve [7FF753C94880+720] 
> (c:\users\bmahler\git\mesos\src\master\authorization.cpp:236)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::_reserve [7FF753D007CF+1423] 
> (c:\users\bmahler\git\mesos\src\master\http.cpp:1960)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::reserveResources 
> [7FF753D07FFC+380] (c:\users\bmahler\git\mesos\src\master\http.cpp:1984)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> mesos::internal::master::Master::Http::api [7FF753CED99B+6571] 
> (c:\users\bmahler\git\mesos\src\master\http.cpp:352)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> ::operator() [7FF7527AA2E4+100] 
> (c:\users\bmahler\git\mesos\src\master\master.cpp:909)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_functor::_Call< 
> &,process::http::Request const 
> &,Option const &> 
> [7FF75262D61A+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::invoke< 
> &,process::http::Request const 
> &,Option const &> 
> [7FF7526E55AA+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_ret,0>::_Call<
>  &,process::http::Request const 
> &,Option const &> 
> [7FF75262D55A+122] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Func_impl_no_alloc<,process::Future,process::http::Request
>  const &,Option const &>::_Do_call 
> [7FF7527EDF2F+111] (c:\program files (x86)\microsoft visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Func_class,process::http::Request
>  const &,Option const 
> &>::operator() [7FF75472BAAD+157] (c:\program files (x86)\microsoft 
> visual 
> studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> ::operator() [7FF75471911F+335] 
> (c:\users\bmahler\git\mesos\3rdparty\libprocess\src\process.cpp:3874)
> E 00:00:00.00 18216 logging.cpp:320] RAW:   
> std::_Invoker_functor::_Call<,bool> 
> [7FF754638F1F+95] (c:\program files (x86)\microsoft

Re: Review Request 72239: Added stack trace logging to program crashes on windows.

2020-03-16 Thread Mesos Reviewbot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72239/#review219962
---



Bad patch!

Reviews applied: [72239]

Failed command: ['bash', '-c', "set -o pipefail; export OS='ubuntu:16.04' 
BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose 
--disable-libtool-wrappers --disable-parallel-test-execution' 
ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/jenkins/buildbot.sh 2>&1 | 
tee build_72239"]

Error:
..
sr/include/apr-1.0 -pthread -Wall -Wsign-compare -Wformat-security 
-fstack-protector-strong -fPIC -g1 -O0 -Wno-unused-local-typedefs -std=c++11 -c 
../../../src/common/resources.cpp  -fPIC -DPIC -o 
common/.libs/libmesos_no_3rdparty_la-resources.o
/bin/bash ../libtool  --tag=CXX   --mode=compile g++ -DPACKAGE_NAME=\"mesos\" 
-DPACKAGE_TARNAME=\"mesos\" -DPACKAGE_VERSION=\"1.10.0\" 
-DPACKAGE_STRING=\"mesos\ 1.10.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" 
-DPACKAGE=\"mesos\" -DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 
-DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 
-DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 
-DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 
-DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 
-DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 -DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 
-DMESOS_HAS_JAVA=1 -DENABLE_NVML=1 -DHAVE_LIBSASL2=1 -DHAVE_SVN_VERSION_H=1 
-DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 
-DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_PYTHON=\"2.7\" -DMESOS_HAS_PYTHON=1 -I. 
-I../../../src   -Werror -DLIBDIR=\"/tmp/SRC/build/mesos-1.10.0/_inst/lib\" 
-DPKGLIBEXECDIR=\"/tmp/SRC/build/mesos-1.10.0/_inst/libexec/mes
 os\" -DPKGDATADIR=\"/tmp/SRC/build/mesos-1.10.0/_inst/share/mesos\" 
-DPKGMODULEDIR=\"/tmp/SRC/build/mesos-1.10.0/_inst/lib/mesos/modules\" 
-I../../../include -I../include -I../include/mesos -D__STDC_FORMAT_MACROS 
-I../3rdparty/boost-1.65.0 -I../3rdparty/concurrentqueue-7b69a8f 
-I../3rdparty/elfio-3.2 -I../3rdparty/glog-0.4.0/src 
-I../3rdparty/grpc-1.10.0/include -I../3rdparty/leveldb-1.19/include 
-I../3rdparty/libarchive-3.3.2/libarchive/ 
-I../../../3rdparty/libprocess/include  -I../3rdparty/nvml-352.79 
-I../3rdparty/picojson-1.3.0 -I../3rdparty/protobuf-3.5.0/src 
-I../3rdparty/rapidjson-1.1.0/include -I../../../3rdparty/stout/include 
-I../3rdparty/zookeeper-3.4.8/src/c/include 
-I../3rdparty/zookeeper-3.4.8/src/c/generated -I/usr/include/subversion-1 
-I/usr/include/apr-1 -I/usr/include/apr-1.0  -pthread -Wall 
-Wsign-compare -Wformat-security -fstack-protector-strong -fPIC -fPIE -g1 -O0 
-Wno-unused-local-typedefs -std=c++11 -c -o 
common/libmesos_no_3rdparty_la-resource_quantiti
 es.lo `test -f 'common/resource_quantities.cpp' || echo 
'../../../src/'`common/resource_quantities.cpp
/bin/bash ../libtool  --tag=CXX   --mode=compile g++ -DPACKAGE_NAME=\"mesos\" 
-DPACKAGE_TARNAME=\"mesos\" -DPACKAGE_VERSION=\"1.10.0\" 
-DPACKAGE_STRING=\"mesos\ 1.10.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" 
-DPACKAGE=\"mesos\" -DVERSION=\"1.10.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 
-DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 
-DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 
-DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 
-DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1 -DHAVE_OPENSSL_SSL_H=1 
-DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 -DHAVE_LIBAPR_1=1 -DHAVE_LIBCURL=1 
-DMESOS_HAS_JAVA=1 -DENABLE_NVML=1 -DHAVE_LIBSASL2=1 -DHAVE_SVN_VERSION_H=1 
-DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 
-DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_PYTHON=\"2.7\" -DMESOS_HAS_PYTHON=1 -I. 
-I../../../src   -Werror -DLIBDIR=\"/tmp/SRC/build/mesos-1.10.0/_inst/lib\" 
-DPKGLIBEXECDIR=\"/tmp/SRC/build/mesos-1.10.0/_inst/libexec/mes
 os\" -DPKGDATADIR=\"/tmp/SRC/build/mesos-1.10.0/_inst/share/mesos\" 
-DPKGMODULEDIR=\"/tmp/SRC/build/mesos-1.10.0/_inst/lib/mesos/modules\" 
-I../../../include -I../include -I../include/mesos -D__STDC_FORMAT_MACROS 
-I../3rdparty/boost-1.65.0 -I../3rdparty/concurrentqueue-7b69a8f 
-I../3rdparty/elfio-3.2 -I../3rdparty/glog-0.4.0/src 
-I../3rdparty/grpc-1.10.0/include -I../3rdparty/leveldb-1.19/include 
-I../3rdparty/libarchive-3.3.2/libarchive/ 
-I../../../3rdparty/libprocess/include  -I../3rdparty/nvml-352.79 
-I../3rdparty/picojson-1.3.0 -I../3rdparty/protobuf-3.5.0/src 
-I../3rdparty/rapidjson-1.1.0/include -I../../../3rdparty/stout/include 
-I../3rdparty/zookeeper-3.4.8/src/c/include 
-I../3rdparty/zookeeper-3.4.8/src/c/generated -I/usr/include/subversion-1 
-I/usr/include/apr-1 -I/usr/include/apr-1.0  -pthread -Wall 
-Wsign-compare -Wformat-security -fstack-protector-strong -fPIC -fPIE -g1 -O0 
-Wno-unused-local-typedefs -std=c++11 -c -o 
common/libmesos_no_3rdparty_la-resources_utils.l
 o

Review Request 72239: Added stack trace logging to program crashes on windows.

2020-03-16 Thread Benjamin Mahler 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72239/
---

Review request for mesos, Andrei Sekretenko, Greg Mann, and Joseph Wu.


Bugs: MESOS-7245
https://issues.apache.org/jira/browse/MESOS-7245


Repository: mesos


Description
---

For now, this starts with adding the necessary code to logging.cpp up
in mesos. This gives us stack traces for any mesos binaries or
libraries that initialize logging with `installFailureSignalHandler`
set to true.

Ideally, we would either (1) move this into stout, or (2) upstream
some changes to glog. Both approaches would allow us to also get
stack traces from stout tests and libprocess tests.

For now however, this patch is just aiming to fix the urgent issue
of a lack of stack traces for mesos crashes.


Diffs
-

  src/logging/logging.cpp baca51b3956dbaa282c03e73a2c261ff103f4821 


Diff: https://reviews.apache.org/r/72239/diff/1/


Testing
---

Ran through CI and locally, for the recent reservation update crash this 
produces:

```
E 00:00:00.00 18216 logging.cpp:295] RAW: Received fatal exception 
0xc005 EXCEPTION_ACCESS_VIOLATION
E 00:00:00.00 18216 logging.cpp:320] RAW:   
google::protobuf::internal::RepeatedPtrIterator::operator* [7FF75279306E+14] 
(c:\users\bmahler\git\mesos\build\3rdparty\protobuf-3.5.0\src\protobuf-3.5.0\src\google\protobuf\repeated_field.h:2266)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
mesos::authorization::ActionObject::reserve [7FF753C94880+720] 
(c:\users\bmahler\git\mesos\src\master\authorization.cpp:236)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
mesos::internal::master::Master::Http::_reserve [7FF753D007CF+1423] 
(c:\users\bmahler\git\mesos\src\master\http.cpp:1960)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
mesos::internal::master::Master::Http::reserveResources [7FF753D07FFC+380] 
(c:\users\bmahler\git\mesos\src\master\http.cpp:1984)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
mesos::internal::master::Master::Http::api [7FF753CED99B+6571] 
(c:\users\bmahler\git\mesos\src\master\http.cpp:352)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
::operator() [7FF7527AA2E4+100] 
(c:\users\bmahler\git\mesos\src\master\master.cpp:909)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Invoker_functor::_Call< 
&,process::http::Request const 
&,Option const &> 
[7FF75262D61A+122] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::invoke< &,process::http::Request 
const &,Option const &> 
[7FF7526E55AA+122] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Invoker_ret,0>::_Call<
 &,process::http::Request const 
&,Option const &> 
[7FF75262D55A+122] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Func_impl_no_alloc<,process::Future,process::http::Request
 const &,Option const &>::_Do_call 
[7FF7527EDF2F+111] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Func_class,process::http::Request
 const &,Option const &>::operator() 
[7FF75472BAAD+157] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\functional:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
::operator() [7FF75471911F+335] 
(c:\users\bmahler\git\mesos\3rdparty\libprocess\src\process.cpp:3874)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Invoker_functor::_Call<,bool> 
[7FF754638F1F+95] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::invoke<,bool> 
[7FF75469AB2E+94] (c:\program files (x86)\microsoft visual 
studio\2017\community\vc\tools\msvc\14.16.27023\include\type_traits:16707566)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
lambda::internal::Partial<,bool>::invoke_expand<,std::tuple,std::tuple<>,0>
 [7FF7546A2B9A+138] 
(c:\users\bmahler\git\mesos\3rdparty\stout\include\stout\lambda.hpp:292)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
lambda::internal::Partial<,bool>::operator()<>
 [7FF754620DCF+127] 
(c:\users\bmahler\git\mesos\3rdparty\stout\include\stout\lambda.hpp:331)
E 00:00:00.00 18216 logging.cpp:320] RAW:   
std::_Invoker_functor::_Call,bool>
 > [7FF75463D082+66]