[jira] [Comment Edited] (KUDU-1864) Thirdparty squeasel build fails on macOS 10.12
[
https://issues.apache.org/jira/browse/KUDU-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852630#comment-15852630
]
Alexey Serbin edited comment on KUDU-1864 at 2/4/17 6:20 AM:
-
I posted a patch for review:
https://gerrit.cloudera.org/#/c/5907/
At least it works for me on OS X 10.11.
A quick way to check whether it works on other OS X would be running the small
snippet below and see whether it outputs the expected '-I...' string.
{code:none}
#!/bin/sh
if openssl_cflags=$(pkg-config --cflags openssl) ; then
EXTRA_CFLAGS="$EXTRA_CFLAGS $openssl_cflags"
fi
echo "***"
echo $EXTRA_CFLAGS
echo "***"
{code}
was (Author: aserbin):
I posted a patch for review:
https://gerrit.cloudera.org/#/c/5907/
At least it works for me on OS X 10.11.
{code:none}
#!/bin/sh
if openssl_cflags=$(pkg-config --cflags openssl) ; then
EXTRA_CFLAGS="$EXTRA_CFLAGS $openssl_cflags"
fi
echo "***"
echo $EXTRA_CFLAGS
echo "***"
{code}
> Thirdparty squeasel build fails on macOS 10.12
> --
>
> Key: KUDU-1864
> URL: https://issues.apache.org/jira/browse/KUDU-1864
> Project: Kudu
> Issue Type: Bug
> Components: build
>Reporter: Dan Burkert
>
> Since recent changes to squeasel landed in 8993716d7, the thirdparty build of
> squeasel has been broken on macOS 10.12. The OpenSSL system headers in
> {{/usr/include/openssl}} are not present in 10.12, and squeasel now depends
> on them. The error:
> {code}
> <..>/kudu/thirdparty/src/squeasel-c304d3f3481b07bf153979155f02e0aab24d01de/squeasel.c:153:10:
> fatal error: 'openssl/ssl.h' file not found
> {code}
> An relatively easy workaround is to install OpenSSL via homebrew and set the
> environment variable {{EXTRA_CFLAGS="-I/usr/local/opt/openssl/include"}} when
> building thirdparty.
> We should come up with a long-term solution. [~tlipcon] suggested that we
> might try to use pkg-config.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (KUDU-1864) Thirdparty squeasel build fails on macOS 10.12
[
https://issues.apache.org/jira/browse/KUDU-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852630#comment-15852630
]
Alexey Serbin edited comment on KUDU-1864 at 2/4/17 6:18 AM:
-
I posted a patch for review:
https://gerrit.cloudera.org/#/c/5907/
At least it works for me on OS X 10.11.
{code:none}
#!/bin/sh
if openssl_cflags=$(pkg-config --cflags openssl) ; then
EXTRA_CFLAGS="$EXTRA_CFLAGS $openssl_cflags"
fi
echo "***"
echo $EXTRA_CFLAGS
echo "***"
{code}
was (Author: aserbin):
I posted a patch for review:
https://gerrit.cloudera.org/#/c/5907/
At least it works for me on OS X 10.11.
{code:none}
if openssl_cflags=$(pkg-config --cflags openssl) ; then
EXTRA_CFLAGS="$EXTRA_CFLAGS $openssl_cflags"
fi
echo "***"
echo $EXTRA_CFLAGS
echo "***"
{code}
> Thirdparty squeasel build fails on macOS 10.12
> --
>
> Key: KUDU-1864
> URL: https://issues.apache.org/jira/browse/KUDU-1864
> Project: Kudu
> Issue Type: Bug
> Components: build
>Reporter: Dan Burkert
>
> Since recent changes to squeasel landed in 8993716d7, the thirdparty build of
> squeasel has been broken on macOS 10.12. The OpenSSL system headers in
> {{/usr/include/openssl}} are not present in 10.12, and squeasel now depends
> on them. The error:
> {code}
> <..>/kudu/thirdparty/src/squeasel-c304d3f3481b07bf153979155f02e0aab24d01de/squeasel.c:153:10:
> fatal error: 'openssl/ssl.h' file not found
> {code}
> An relatively easy workaround is to install OpenSSL via homebrew and set the
> environment variable {{EXTRA_CFLAGS="-I/usr/local/opt/openssl/include"}} when
> building thirdparty.
> We should come up with a long-term solution. [~tlipcon] suggested that we
> might try to use pkg-config.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1864) Thirdparty squeasel build fails on macOS 10.12
[
https://issues.apache.org/jira/browse/KUDU-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852630#comment-15852630
]
Alexey Serbin commented on KUDU-1864:
-
I posted a patch for review:
https://gerrit.cloudera.org/#/c/5907/
At least it works for me on OS X 10.11.
{code:shell}
if openssl_cflags=$(pkg-config --cflags openssl) ; then
EXTRA_CFLAGS="$EXTRA_CFLAGS $openssl_cflags"
fi
echo "***"
echo $EXTRA_CFLAGS
echo "***"
{code}
> Thirdparty squeasel build fails on macOS 10.12
> --
>
> Key: KUDU-1864
> URL: https://issues.apache.org/jira/browse/KUDU-1864
> Project: Kudu
> Issue Type: Bug
> Components: build
>Reporter: Dan Burkert
>
> Since recent changes to squeasel landed in 8993716d7, the thirdparty build of
> squeasel has been broken on macOS 10.12. The OpenSSL system headers in
> {{/usr/include/openssl}} are not present in 10.12, and squeasel now depends
> on them. The error:
> {code}
> <..>/kudu/thirdparty/src/squeasel-c304d3f3481b07bf153979155f02e0aab24d01de/squeasel.c:153:10:
> fatal error: 'openssl/ssl.h' file not found
> {code}
> An relatively easy workaround is to install OpenSSL via homebrew and set the
> environment variable {{EXTRA_CFLAGS="-I/usr/local/opt/openssl/include"}} when
> building thirdparty.
> We should come up with a long-term solution. [~tlipcon] suggested that we
> might try to use pkg-config.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KUDU-1864) Thirdparty squeasel build fails on macOS 10.12
[
https://issues.apache.org/jira/browse/KUDU-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Serbin updated KUDU-1864:
Status: In Review (was: Open)
> Thirdparty squeasel build fails on macOS 10.12
> --
>
> Key: KUDU-1864
> URL: https://issues.apache.org/jira/browse/KUDU-1864
> Project: Kudu
> Issue Type: Bug
> Components: build
>Reporter: Dan Burkert
>
> Since recent changes to squeasel landed in 8993716d7, the thirdparty build of
> squeasel has been broken on macOS 10.12. The OpenSSL system headers in
> {{/usr/include/openssl}} are not present in 10.12, and squeasel now depends
> on them. The error:
> {code}
> <..>/kudu/thirdparty/src/squeasel-c304d3f3481b07bf153979155f02e0aab24d01de/squeasel.c:153:10:
> fatal error: 'openssl/ssl.h' file not found
> {code}
> An relatively easy workaround is to install OpenSSL via homebrew and set the
> environment variable {{EXTRA_CFLAGS="-I/usr/local/opt/openssl/include"}} when
> building thirdparty.
> We should come up with a long-term solution. [~tlipcon] suggested that we
> might try to use pkg-config.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1865) Create fast path for RespondSuccess() in KRPC
[
https://issues.apache.org/jira/browse/KUDU-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852617#comment-15852617
]
Todd Lipcon commented on KUDU-1865:
---
hacked on this for a few minutes tonight and managed to remove half of the
cross-thread allocations. Can you try http://gerrit.cloudera.org:8080/5905 and
see if it improves performance? We can also try with rpc-bench, but I think the
impact might be larger in a real application where it's also competing for
other allocation traffic.
> Create fast path for RespondSuccess() in KRPC
> -
>
> Key: KUDU-1865
> URL: https://issues.apache.org/jira/browse/KUDU-1865
> Project: Kudu
> Issue Type: Improvement
> Components: rpc
>Reporter: Sailesh Mukil
> Labels: perfomance, rpc
> Attachments: alloc-pattern.py, cross-thread.txt
>
>
> A lot of RPCs just respond with RespondSuccess() which returns the exact
> payload every time. This takes the same path as any other response by
> ultimately calling Connection::QueueResponseForCall() which has a few small
> allocations. These small allocations (and their corresponding deallocations)
> are called quite frequently (once for every IncomingCall) and end up taking
> quite some time in the kernel (traversing the free list, spin locks etc.)
> This was found when [~mmokhtar] ran some profiles on Impala over KRPC on a 20
> node cluster and found the following:
> The exact % of time spent is hard to quantify from the profiles, but these
> were the among the top 5 of the slowest stacks:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! __gnu_cxx::new_allocator::deallocate + 0x4 -
> new_allocator.h:110
> impalad ! std::_Vector_base std::allocator>::_M_deallocate + 0x5 - stl_vector.h:178
> impalad ! ~_Vector_base + 0x4 - stl_vector.h:160
> impalad ! ~vector - stl_vector.h:425 'slices' vector
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0xac -
> connection.cc:433
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ..
> {code}
> {code:java}
> impalad ! tcmalloc::CentralFreeList::FetchFromOneSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::RemoveRange + 0xc0 - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown
> source file]
> impalad ! operator new + 0x297 - [unknown source file]<--- Creating
> new 'OutboundTransferTask' object.
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0x76 -
> connection.cc:432
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ...
> {code}
> Even creating and deleting the 'RpcContext' takes a lot of time:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x16 -
> impala_internal_service.pb.cc:1221
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x8 -
> impala_internal_service.pb.cc:1222
> impalad ! kudu::DefaultDeleter::operator() + 0x5 -
> gscoped_ptr.h:145
> impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228
> impalad ! ~gscoped_ptr - gscoped_ptr.h:318
> impalad ! kudu::rpc::RpcContext::~RpcContext + 0x1e - rpc_context.cc:53
> <-
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1ff - rpc_context.cc:67
> {code}
> The above show that creating these small objects under moderately heavy load
> results in heavy contention in the kernel. We will benefit a lot if we create
> a fast path for 'RespondSuccess'.
> My suggestion is to create all these small objects at once along with
[jira] [Updated] (KUDU-1865) Create fast path for RespondSuccess() in KRPC
[
https://issues.apache.org/jira/browse/KUDU-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated KUDU-1865:
--
Attachment: cross-thread.txt
alloc-pattern.py
I just wrote a little program that takes the output of a tcmalloc trace and
counts allocations which originate on one thread and are freed on another. I
ran rpc-bench on this for 1 second (which only managed 480 RPCs given the debug
build and alloc tracing). Here are the stacks of per-RPC allocations which are
freed on a different thread than they are allocated.
> Create fast path for RespondSuccess() in KRPC
> -
>
> Key: KUDU-1865
> URL: https://issues.apache.org/jira/browse/KUDU-1865
> Project: Kudu
> Issue Type: Improvement
> Components: rpc
>Reporter: Sailesh Mukil
> Labels: perfomance, rpc
> Attachments: alloc-pattern.py, cross-thread.txt
>
>
> A lot of RPCs just respond with RespondSuccess() which returns the exact
> payload every time. This takes the same path as any other response by
> ultimately calling Connection::QueueResponseForCall() which has a few small
> allocations. These small allocations (and their corresponding deallocations)
> are called quite frequently (once for every IncomingCall) and end up taking
> quite some time in the kernel (traversing the free list, spin locks etc.)
> This was found when [~mmokhtar] ran some profiles on Impala over KRPC on a 20
> node cluster and found the following:
> The exact % of time spent is hard to quantify from the profiles, but these
> were the among the top 5 of the slowest stacks:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! __gnu_cxx::new_allocator::deallocate + 0x4 -
> new_allocator.h:110
> impalad ! std::_Vector_base std::allocator>::_M_deallocate + 0x5 - stl_vector.h:178
> impalad ! ~_Vector_base + 0x4 - stl_vector.h:160
> impalad ! ~vector - stl_vector.h:425 'slices' vector
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0xac -
> connection.cc:433
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ..
> {code}
> {code:java}
> impalad ! tcmalloc::CentralFreeList::FetchFromOneSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::RemoveRange + 0xc0 - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown
> source file]
> impalad ! operator new + 0x297 - [unknown source file]<--- Creating
> new 'OutboundTransferTask' object.
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0x76 -
> connection.cc:432
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ...
> {code}
> Even creating and deleting the 'RpcContext' takes a lot of time:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x16 -
> impala_internal_service.pb.cc:1221
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x8 -
> impala_internal_service.pb.cc:1222
> impalad ! kudu::DefaultDeleter::operator() + 0x5 -
> gscoped_ptr.h:145
> impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228
> impalad ! ~gscoped_ptr - gscoped_ptr.h:318
> impalad ! kudu::rpc::RpcContext::~RpcContext + 0x1e - rpc_context.cc:53
> <-
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1ff - rpc_context.cc:67
> {code}
> The above show that creating these small objects under moderately heavy load
> results in heavy contention in the kernel. We will benefit a lot if we create
> a fast path for 'RespondSuccess'.
> My suggestion is
[jira] [Commented] (KUDU-1865) Create fast path for RespondSuccess() in KRPC
[
https://issues.apache.org/jira/browse/KUDU-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852466#comment-15852466
]
Sailesh Mukil commented on KUDU-1865:
-
CC [~henryr]
> Create fast path for RespondSuccess() in KRPC
> -
>
> Key: KUDU-1865
> URL: https://issues.apache.org/jira/browse/KUDU-1865
> Project: Kudu
> Issue Type: Improvement
> Components: rpc
>Reporter: Sailesh Mukil
> Labels: perfomance, rpc
>
> A lot of RPCs just respond with RespondSuccess() which returns the exact
> payload every time. This takes the same path as any other response by
> ultimately calling Connection::QueueResponseForCall() which has a few small
> allocations. These small allocations (and their corresponding deallocations)
> are called quite frequently (once for every IncomingCall) and end up taking
> quite some time in the kernel (traversing the free list, spin locks etc.)
> This was found when [~mmokhtar] ran some profiles on Impala over KRPC on a 20
> node cluster and found the following:
> The exact % of time spent is hard to quantify from the profiles, but these
> were the among the top 5 of the slowest stacks:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! __gnu_cxx::new_allocator::deallocate + 0x4 -
> new_allocator.h:110
> impalad ! std::_Vector_base std::allocator>::_M_deallocate + 0x5 - stl_vector.h:178
> impalad ! ~_Vector_base + 0x4 - stl_vector.h:160
> impalad ! ~vector - stl_vector.h:425 'slices' vector
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0xac -
> connection.cc:433
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ..
> {code}
> {code:java}
> impalad ! tcmalloc::CentralFreeList::FetchFromOneSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::RemoveRange + 0xc0 - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown
> source file]
> impalad ! operator new + 0x297 - [unknown source file]<--- Creating
> new 'OutboundTransferTask' object.
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0x76 -
> connection.cc:432
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ...
> {code}
> Even creating and deleting the 'RpcContext' takes a lot of time:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x16 -
> impala_internal_service.pb.cc:1221
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x8 -
> impala_internal_service.pb.cc:1222
> impalad ! kudu::DefaultDeleter::operator() + 0x5 -
> gscoped_ptr.h:145
> impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228
> impalad ! ~gscoped_ptr - gscoped_ptr.h:318
> impalad ! kudu::rpc::RpcContext::~RpcContext + 0x1e - rpc_context.cc:53
> <-
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1ff - rpc_context.cc:67
> {code}
> The above show that creating these small objects under moderately heavy load
> results in heavy contention in the kernel. We will benefit a lot if we create
> a fast path for 'RespondSuccess'.
> My suggestion is to create all these small objects at once along with the
> 'InboundCall' object while it is being created, in a 'RespondSuccess'
> structure, and just use that structure when we want to send 'success' back to
> the sender. This would already contain the 'OutboundTransferTask', a 'Slice'
> with 'success', etc. We would expect that most RPCs respond with 'success' a
> majority of the time.
> How this would benefit us is that we
[jira] [Commented] (KUDU-1864) Thirdparty squeasel build fails on macOS 10.12
[
https://issues.apache.org/jira/browse/KUDU-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852440#comment-15852440
]
Dan Burkert commented on KUDU-1864:
---
I wouldn't be against that, but I think we have some users (notably [~aserbin])
who don't use homebrew.
> Thirdparty squeasel build fails on macOS 10.12
> --
>
> Key: KUDU-1864
> URL: https://issues.apache.org/jira/browse/KUDU-1864
> Project: Kudu
> Issue Type: Bug
> Components: build
>Reporter: Dan Burkert
>
> Since recent changes to squeasel landed in 8993716d7, the thirdparty build of
> squeasel has been broken on macOS 10.12. The OpenSSL system headers in
> {{/usr/include/openssl}} are not present in 10.12, and squeasel now depends
> on them. The error:
> {code}
> <..>/kudu/thirdparty/src/squeasel-c304d3f3481b07bf153979155f02e0aab24d01de/squeasel.c:153:10:
> fatal error: 'openssl/ssl.h' file not found
> {code}
> An relatively easy workaround is to install OpenSSL via homebrew and set the
> environment variable {{EXTRA_CFLAGS="-I/usr/local/opt/openssl/include"}} when
> building thirdparty.
> We should come up with a long-term solution. [~tlipcon] suggested that we
> might try to use pkg-config.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1864) Thirdparty squeasel build fails on macOS 10.12
[
https://issues.apache.org/jira/browse/KUDU-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852435#comment-15852435
]
David Alves commented on KUDU-1864:
---
should we just make this workaroud permanent (as in documented that we need to
install openssl and automatically add the flag in macos)?
> Thirdparty squeasel build fails on macOS 10.12
> --
>
> Key: KUDU-1864
> URL: https://issues.apache.org/jira/browse/KUDU-1864
> Project: Kudu
> Issue Type: Bug
> Components: build
>Reporter: Dan Burkert
>
> Since recent changes to squeasel landed in 8993716d7, the thirdparty build of
> squeasel has been broken on macOS 10.12. The OpenSSL system headers in
> {{/usr/include/openssl}} are not present in 10.12, and squeasel now depends
> on them. The error:
> {code}
> <..>/kudu/thirdparty/src/squeasel-c304d3f3481b07bf153979155f02e0aab24d01de/squeasel.c:153:10:
> fatal error: 'openssl/ssl.h' file not found
> {code}
> An relatively easy workaround is to install OpenSSL via homebrew and set the
> environment variable {{EXTRA_CFLAGS="-I/usr/local/opt/openssl/include"}} when
> building thirdparty.
> We should come up with a long-term solution. [~tlipcon] suggested that we
> might try to use pkg-config.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1864) Thirdparty squeasel build fails on macOS 10.12
[
https://issues.apache.org/jira/browse/KUDU-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852405#comment-15852405
]
Todd Lipcon commented on KUDU-1864:
---
I actually tried pkg-config on an el capitan box and it didn't output anything.
So, not so useful.
> Thirdparty squeasel build fails on macOS 10.12
> --
>
> Key: KUDU-1864
> URL: https://issues.apache.org/jira/browse/KUDU-1864
> Project: Kudu
> Issue Type: Bug
> Components: build
>Reporter: Dan Burkert
>
> Since recent changes to squeasel landed in 8993716d7, the thirdparty build of
> squeasel has been broken on macOS 10.12. The OpenSSL system headers in
> {{/usr/include/openssl}} are not present in 10.12, and squeasel now depends
> on them. The error:
> {code}
> <..>/kudu/thirdparty/src/squeasel-c304d3f3481b07bf153979155f02e0aab24d01de/squeasel.c:153:10:
> fatal error: 'openssl/ssl.h' file not found
> {code}
> An relatively easy workaround is to install OpenSSL via homebrew and set the
> environment variable {{EXTRA_CFLAGS="-I/usr/local/opt/openssl/include"}} when
> building thirdparty.
> We should come up with a long-term solution. [~tlipcon] suggested that we
> might try to use pkg-config.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (KUDU-1864) Thirdparty squeasel build fails on macOS 10.12
Dan Burkert created KUDU-1864:
-
Summary: Thirdparty squeasel build fails on macOS 10.12
Key: KUDU-1864
URL: https://issues.apache.org/jira/browse/KUDU-1864
Project: Kudu
Issue Type: Bug
Components: build
Reporter: Dan Burkert
Since recent changes to squeasel landed in 8993716d7, the thirdparty build of
squeasel has been broken on macOS 10.12. The OpenSSL system headers in
{{/usr/include/openssl}} are not present in 10.12, and squeasel now depends on
them. The error:
{code}
<..>/kudu/thirdparty/src/squeasel-c304d3f3481b07bf153979155f02e0aab24d01de/squeasel.c:153:10:
fatal error: 'openssl/ssl.h' file not found
{code}
An relatively easy workaround is to install OpenSSL via homebrew and set the
environment variable {{EXTRA_CFLAGS="-I/usr/local/opt/openssl/include"}} when
building thirdparty.
We should come up with a long-term solution. [~tlipcon] suggested that we might
try to use pkg-config.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1863) Deadlock while shutting down master
[ https://issues.apache.org/jira/browse/KUDU-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851765#comment-15851765 ] Todd Lipcon commented on KUDU-1863: --- Probably not urgent to fix this, since we don't Shutdown servers in real life. Fix might be something like avoiding holding the Messenger lock while clearing the rpc_services_ array (instead swap out under the lock, then clear outside it). But it doesn't repro often, so probably need a better repro first. > Deadlock while shutting down master > --- > > Key: KUDU-1863 > URL: https://issues.apache.org/jira/browse/KUDU-1863 > Project: Kudu > Issue Type: Bug > Components: master >Affects Versions: 1.3.0 >Reporter: Todd Lipcon > Fix For: Backlog > > > I saw ksck_remote-test fail because the master was hanging during its attempt > to shut down. Analysis follows. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KUDU-1863) Deadlock while shutting down master
[
https://issues.apache.org/jira/browse/KUDU-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851762#comment-15851762
]
Todd Lipcon commented on KUDU-1863:
---
{code}
thread holding lock, waiting for RPC worker thread T312 to exit
=
Thread 1 (Thread 0x7f8e75f51c40 (LWP 3239)):
...
#5 0x004c3fac in kudu::CountDownLatch::WaitFor (this=,
delta=...) at
/home/jenkins-slave/workspace/kudu-1/src/kudu/util/countdown_latch.h:91
#6 0x7f8e6c159d19 in kudu::ThreadJoiner::Join (this=) at
/home/jenkins-slave/workspace/kudu-1/src/kudu/util/thread.cc:466
#7 0x7f8e6e574fbe in kudu::rpc::ServicePool::Shutdown (this=) at /home/jenkins-slave/workspace/kudu-1/src/kudu/rpc/service_pool.cc:95
#8 0x7f8e6e574727 in kudu::rpc::ServicePool::~ServicePool
(this=0x7d38dac0) at
/home/jenkins-slave/workspace/kudu-1/src/kudu/rpc/service_pool.cc:74
#9 0x7f8e6e57480a in kudu::rpc::ServicePool::~ServicePool
(this=0x7d38dac0) at
/home/jenkins-slave/workspace/kudu-1/src/kudu/rpc/service_pool.cc:73
#10 0x7f8e732401a8 in kudu::RefCountedThreadSafe::DeleteInternal (x=0x7d38dac0) at /home/jenkins-slave/w
...
#19 0x7f8e6e53c8ce in clear (this=) at
/home/jenkins-slave/workspace/kudu-1/thirdparty/installed/tsan/include/c++/v1/unordered_map:1083
#20 kudu::rpc::Messenger::UnregisterAllServices (this=0x7d3a7700) at
/home/jenkins-slave/workspace/kudu-1/src/kudu/rpc/messenger.cc:227
#21 0x7f8e7323e3df in kudu::RpcServer::Shutdown (this=) at
/home/jenkins-slave/workspace/kudu-1/src/kudu/server/rpc_server.cc:176
RPC worker thread T312 not exiting because it is waiting on a write to finish
Thread 312 (rpc worker handling heartbeat):
- waiting on the SyncWrite callback
#6 0x7f8e74bd3b40 in kudu::master::SysCatalogTable::SyncWrite
(this=, req=0x7f8e3db2ed60, resp=0x7f8e3db2ed00) at
/home/jenkins-slave/workspace/kudu-1/src/kudu/master/sys_catalog.c
Reactor threads stuck waiting on messenger's lock (held by T1) , thus not
delivering write callback
Thread 316 (Thread 0x7f8e443cb700 (LWP 6333)):
...
#4 0x7f8e74ba99fa in kudu::rw_spinlock::lock_shared (this=0x7d4400073f88)
at /home/jenkins-slave/workspace/kudu-1/src/kudu/util/locks.h:94
#5 0x7f8e74b8d009 in kudu::shared_lock::shared_lock
(this=0x7f8e4438a028, m=...) at
/home/jenkins-slave/workspace/kudu-1/src/kudu/util/locks.h:245
#6 0x7f8e6e53cb68 in kudu::rpc::Messenger::QueueInboundCall
(this=0x7d3a7700, call=...) at
/home/jenkins-slave/workspace/kudu-1/src/kudu/rpc/messenger.cc:248
{code}
> Deadlock while shutting down master
> ---
>
> Key: KUDU-1863
> URL: https://issues.apache.org/jira/browse/KUDU-1863
> Project: Kudu
> Issue Type: Bug
> Components: master
>Affects Versions: 1.3.0
>Reporter: Todd Lipcon
> Fix For: Backlog
>
>
> I saw ksck_remote-test fail because the master was hanging during its attempt
> to shut down. Analysis follows.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (KUDU-1863) Deadlock while shutting down master
Todd Lipcon created KUDU-1863: - Summary: Deadlock while shutting down master Key: KUDU-1863 URL: https://issues.apache.org/jira/browse/KUDU-1863 Project: Kudu Issue Type: Bug Components: master Affects Versions: 1.3.0 Reporter: Todd Lipcon Fix For: Backlog I saw ksck_remote-test fail because the master was hanging during its attempt to shut down. Analysis follows. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
