Re: libthr and main thread stack size

2014-09-23 Thread Konstantin Belousov 
On Mon, Sep 22, 2014 at 09:19:30AM -0400, Daniel Eischen wrote:
 On Sun, 21 Sep 2014, Julian Elischer wrote:
 
  On 9/20/14, 3:27 AM, John Baldwin wrote:
  On Tuesday, September 16, 2014 11:13:24 AM Konstantin Belousov wrote:
  On Mon, Sep 15, 2014 at 03:47:41PM -0600, Justin T. Gibbs wrote:
  On Aug 8, 2014, at 5:22 AM, Konstantin Belousov kostik...@gmail.com
  wrote:
  
  ?
  
  Below is the patch which adds environment variable
  LIBPTHREAD_BIGSTACK_MAIN. Setting it to any value results in the
  main thread stack left as is, and other threads allocate stack
  below the area of RLIMIT_STACK. Try it. I do not want to set this
  behaviour as default.
  Is there a reason this should not be the default? Looking at the
  getrlimit() page on the OpenGroup?s site they say:
  
  RLIMIT_STACK This is the maximum size of the initial thread's stack,
  in bytes. The implementation does not automatically grow the stack
  beyond this limit. If this limit is exceeded, SIGSEGV shall be
  generated for the thread. If the thread is blocking SIGSEGV, or the
  process is ignoring or catching SIGSEGV and has not made arrangements
  to use an alternate stack, the disposition of SIGSEGV shall be set to
  SIG_DFL before it is generated.
  
  Does posix say something different?
  
  I ran into this issue when debugging a segfault on Postgres when
  running an (arguably quite bogus) query that should have fit within
  both the configured stack rlimit and Postgres? configured stack limit.
  The Postgres backend is really just single threaded, but happens
  to pull in libpthread due to the threading support in some of the
  libraries it uses. The segfault definitely violates POLA.
  
  ? Justin
  I am conservative to not disturb the address space layout in single go.
  If enough people test this setting, I can consider flipping the default
  to the reverse.
  
  I am still curious why the things were done in this way, but nobody
  replied.
  I suspect it was done out of reasons of being overly conservative in
  interpreting RLIMIT_STACK.  I think it is quite surprising behavior though 
  and
  would rather we make your option the default and implement what the Open 
  Group
  says above.
  
  that is my memory..
  The transition from a non threaded app to a threaded app with one thread is 
  sort of an undefined area.
  Feel free to change it if Dan agrees..
 
 I'm all for adopting what POSIX specifies as the default.  I
 would shy away from adding another knob (LIBPTHREAD_BIGSTACK_MAIN)
 if possible.

In the patch, default behaviour is to provide RLIMIT_STACK sized stack
for the main thread.  The knobs are there to restore the old AS layout
if my fears of the binary compatibility become real one day, and to
keep the interface compat with the stable/10, which already got a knob
merged.

That said, below the patch with libthr.7 man page merged to libthr.3,
and with the editing applied.

diff --git a/lib/libthr/libthr.3 b/lib/libthr/libthr.3
index bfbebec..aa4572c 100644
--- a/lib/libthr/libthr.3
+++ b/lib/libthr/libthr.3
@@ -1,6 +1,11 @@
 .\ Copyright (c) 2005 Robert N. M. Watson
+.\ Copyright (c) 2014 The FreeBSD Foundation, Inc.
 .\ All rights reserved.
 .\
+.\ Part of this documentation was written by
+.\ Konstantin Belousov k...@freebsd.org under sponsorship
+.\ from the FreeBSD Foundation.
+.\
 .\ Redistribution and use in source and binary forms, with or without
 .\ modification, are permitted provided that the following conditions
 .\ are met:
@@ -24,7 +29,7 @@
 .\
 .\ $FreeBSD$
 .\
-.Dd October 19, 2007
+.Dd September 20, 2014
 .Dt LIBTHR 3
 .Os
 .Sh NAME
@@ -45,8 +50,216 @@ has been optimized for use by applications expecting system 
scope thread
 semantics, and can provide significant performance improvements
 compared to
 .Lb libkse .
+.Pp
+The library is tightly integrated with the run-time link editor
+.Xr ld-elf.so.1 1
+and
+.Lb libc ;
+all three components must be built from the same source tree.
+Mixing
+.Li libc
+and
+.Nm
+libraries from different versions of
+.Fx
+is not supported.
+The run-time linker
+.Xr ld-elf.so.1 1
+has some code to ensure backward-compatibility with older versions of
+.Nm .
+.Pp
+The man page documents the quirks and tunables of the
+.Nm .
+When linking with
+.Li -lpthread ,
+the run-time dependency
+.Li libthr.so.3
+is recorded in the produced object.
+.Sh MUTEX ACQUISITION
+A locked mutex (see
+.Xr pthread_mutex_lock 3 )
+is represented by a volatile variable of type
+.Dv lwpid_t ,
+which records the global system identifier of the thread
+owning the lock.
+.Nm
+performs a contested mutex acquisition in three stages, each of which
+is more resource-consuming than the previous.
+.Pp
+First, a spin loop
+is performed, where the library attempts to acquire the lock by
+.Xr atomic 9
+operations.
+The loop count is controlled by the
+.Ev LIBPTHREAD_SPINLOOPS
+environment variable, with a default value of 2000.
+.Pp
+If the spin loop
+was unable to acquire the mutex, a yeild loop
+is executed,

Re: libthr and main thread stack size

2014-09-23 Thread Sergey Kandaurov 
On 23 September 2014 13:24, Konstantin Belousov kostik...@gmail.com wrote:
 In the patch, default behaviour is to provide RLIMIT_STACK sized stack
 for the main thread.  The knobs are there to restore the old AS layout
 if my fears of the binary compatibility become real one day, and to
 keep the interface compat with the stable/10, which already got a knob
 merged.

 That said, below the patch with libthr.7 man page merged to libthr.3,
 and with the editing applied.

 diff --git a/lib/libthr/libthr.3 b/lib/libthr/libthr.3
 index bfbebec..aa4572c 100644
 --- a/lib/libthr/libthr.3
 +++ b/lib/libthr/libthr.3
 @@ -1,6 +1,11 @@
  .\ Copyright (c) 2005 Robert N. M. Watson
 +.\ Copyright (c) 2014 The FreeBSD Foundation, Inc.
  .\ All rights reserved.
  .\
 +.\ Part of this documentation was written by
 +.\ Konstantin Belousov k...@freebsd.org under sponsorship
 +.\ from the FreeBSD Foundation.
 +.\
  .\ Redistribution and use in source and binary forms, with or without
  .\ modification, are permitted provided that the following conditions
  .\ are met:
 @@ -24,7 +29,7 @@
  .\
  .\ $FreeBSD$
  .\
 -.Dd October 19, 2007
 +.Dd September 20, 2014
  .Dt LIBTHR 3
  .Os
  .Sh NAME
 @@ -45,8 +50,216 @@ has been optimized for use by applications expecting 
 system scope thread
  semantics, and can provide significant performance improvements
  compared to
  .Lb libkse .
 +.Pp
 +The library is tightly integrated with the run-time link editor
 +.Xr ld-elf.so.1 1
 +and
 +.Lb libc ;
 +all three components must be built from the same source tree.
 +Mixing
 +.Li libc
 +and
 +.Nm
 +libraries from different versions of
 +.Fx
 +is not supported.
 +The run-time linker
 +.Xr ld-elf.so.1 1
 +has some code to ensure backward-compatibility with older versions of
 +.Nm .
 +.Pp
 +The man page documents the quirks and tunables of the
 +.Nm .
 +When linking with
 +.Li -lpthread ,
 +the run-time dependency
 +.Li libthr.so.3
 +is recorded in the produced object.
 +.Sh MUTEX ACQUISITION
 +A locked mutex (see
 +.Xr pthread_mutex_lock 3 )
 +is represented by a volatile variable of type
 +.Dv lwpid_t ,
 +which records the global system identifier of the thread
 +owning the lock.
 +.Nm
 +performs a contested mutex acquisition in three stages, each of which
 +is more resource-consuming than the previous.
 +.Pp
 +First, a spin loop
 +is performed, where the library attempts to acquire the lock by
 +.Xr atomic 9
 +operations.
 +The loop count is controlled by the
 +.Ev LIBPTHREAD_SPINLOOPS
 +environment variable, with a default value of 2000.
 +.Pp
 +If the spin loop
 +was unable to acquire the mutex, a yeild loop
typo: yield

[...]
  .Sh SEE ALSO
 -.Xr pthread 3
 +.Xr ktrace 1 ,
 +.Xr ld-elf.so.1 1 ,
 +.Xr getrlimit 2 ,
 +.Xr umtx 2 ,
 +.Xr dlclose 3 ,
 +.Xr dlopen 3 ,
 +.Xr errno 3 ,
 +.Xr getenv 3 ,
 +.Xr libc 3 ,
 +.Xr pthread_attr 3 ,
 +.Xr pthread_attr_setstacksize 3 ,
 +.Xr pthread_create 3 ,
 +.Xr signal 3 ,
 +.Xr atomic 9 .
no pediod there per mdoc

  .Sh AUTHORS
  .An -nosplit
  The
 diff --git a/lib/libthr/thread/thr_init.c b/lib/libthr/thread/thr_init.c
 index 9bf0e29..72a067a 100644
 --- a/lib/libthr/thread/thr_init.c
 +++ b/lib/libthr/thread/thr_init.c
 @@ -445,7 +445,7 @@ init_private(void)
 struct rlimit rlim;
 size_t len;
 int mib[2];
 -   char *env;
 +   char *env, *env_bigstack, *env_splitstack;

 _thr_umutex_init(_mutex_static_lock);
 _thr_umutex_init(_cond_static_lock);
 @@ -473,8 +473,9 @@ init_private(void)
 len = sizeof (_usrstack);
 if (sysctl(mib, 2, _usrstack, len, NULL, 0) == -1)
 PANIC(Cannot get kern.usrstack from sysctl);
 -   env = getenv(LIBPTHREAD_BIGSTACK_MAIN);
 -   if (env != NULL) {
 +   env_bigstack = getenv(LIBPTHREAD_BIGSTACK_MAIN);
 +   env_splitstack = getenv(LIBPTHREAD_SPLITSTACK_MAIN);
 +   if (bigstack != NULL || env_splitstack == NULL) {
looks like a typo: s/bigstack/env_bigstack/

 if (getrlimit(RLIMIT_STACK, rlim) == -1)
 PANIC(Cannot get stack rlimit);
 _thr_stack_initial = rlim.rlim_cur;

-- 
wbr,
pluknet
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: libthr and main thread stack size

2014-09-23 Thread Konstantin Belousov 
On Tue, Sep 23, 2014 at 02:02:01PM +0400, Sergey Kandaurov wrote:
  +   env_bigstack = getenv(LIBPTHREAD_BIGSTACK_MAIN);
  +   env_splitstack = getenv(LIBPTHREAD_SPLITSTACK_MAIN);
  +   if (bigstack != NULL || env_splitstack == NULL) {
 looks like a typo: s/bigstack/env_bigstack/

Indeed, thank you for noting.  This patch actually booted, and I verified
all 5 conditions.

diff --git a/lib/libthr/libthr.3 b/lib/libthr/libthr.3
index bfbebec..a5b75d4 100644
--- a/lib/libthr/libthr.3
+++ b/lib/libthr/libthr.3
@@ -1,6 +1,11 @@
 .\ Copyright (c) 2005 Robert N. M. Watson
+.\ Copyright (c) 2014 The FreeBSD Foundation, Inc.
 .\ All rights reserved.
 .\
+.\ Part of this documentation was written by
+.\ Konstantin Belousov k...@freebsd.org under sponsorship
+.\ from the FreeBSD Foundation.
+.\
 .\ Redistribution and use in source and binary forms, with or without
 .\ modification, are permitted provided that the following conditions
 .\ are met:
@@ -24,7 +29,7 @@
 .\
 .\ $FreeBSD$
 .\
-.Dd October 19, 2007
+.Dd September 20, 2014
 .Dt LIBTHR 3
 .Os
 .Sh NAME
@@ -45,8 +50,216 @@ has been optimized for use by applications expecting system 
scope thread
 semantics, and can provide significant performance improvements
 compared to
 .Lb libkse .
+.Pp
+The library is tightly integrated with the run-time link editor
+.Xr ld-elf.so.1 1
+and
+.Lb libc ;
+all three components must be built from the same source tree.
+Mixing
+.Li libc
+and
+.Nm
+libraries from different versions of
+.Fx
+is not supported.
+The run-time linker
+.Xr ld-elf.so.1 1
+has some code to ensure backward-compatibility with older versions of
+.Nm .
+.Pp
+The man page documents the quirks and tunables of the
+.Nm .
+When linking with
+.Li -lpthread ,
+the run-time dependency
+.Li libthr.so.3
+is recorded in the produced object.
+.Sh MUTEX ACQUISITION
+A locked mutex (see
+.Xr pthread_mutex_lock 3 )
+is represented by a volatile variable of type
+.Dv lwpid_t ,
+which records the global system identifier of the thread
+owning the lock.
+.Nm
+performs a contested mutex acquisition in three stages, each of which
+is more resource-consuming than the previous.
+.Pp
+First, a spin loop
+is performed, where the library attempts to acquire the lock by
+.Xr atomic 9
+operations.
+The loop count is controlled by the
+.Ev LIBPTHREAD_SPINLOOPS
+environment variable, with a default value of 2000.
+.Pp
+If the spin loop
+was unable to acquire the mutex, a yield loop
+is executed, performing the same
+.Xr atomic 9
+acquisition attempts as the spin loop,
+but each attempt is followed by a yield of the CPU time
+of the thread using the
+.Xr sched_yield 2
+syscall.
+By default, the yield loop
+is not executed.
+This is controlled by the
+.Ev LIBPTHREAD_YIELDLOOPS
+environment variable.
+.Pp
+If both the spin and yield loops
+failed to acquire the lock, the thread is taken off the CPU and
+put to sleep in the kernel with the
+.Xr umtx 2
+syscall.
+The kernel wakes up a thread and hands the ownership of the lock to
+the woken thread when the lock becomes available.
+.Sh THREAD STACKS
+Each thread is provided with a private user-mode stack area
+used by the C runtime.
+The size of the main (initial) thread stack is set by the kernel, and is
+controlled by the
+.Dv RLIMIT_STACK
+process resource limit (see
+.Xr getrlimit 2 ) .
+.Pp
+By default, the main thread's stack size is equal to the value of
+.Dv RLIMIT_STACK
+for the process.
+If the
+.Ev LIBPTHREAD_SPLITSTACK_MAIN
+environment variable is present in the process environment
+(its value does not matter),
+the main thread's stack is reduced to 4MB on 64bit architectures, and to
+2MB on 32bit architectures, when the threading library is initialized.
+The rest of the address space area which has been reserved by the
+kernel for the initial process stack is used for non-initial thread stacks
+in this case.
+The presence of the
+.Ev LIBPTHREAD_BIGSTACK_MAIN
+environment variable overrides
+.Ev LIBPTHREAD_SPLITSTACK_MAIN ;
+it is kept for backward-compatibility.
+.Pp
+The size of stacks for threads created by the process at run-time
+with the
+.Xr pthread_create 3
+call is controlled by thread attributes: see
+.Xr pthread_attr 3 ,
+in particular, the
+.Xr pthread_attr_setstacksize 3 ,
+.Xr pthread_attr_setguardsize 3
+and
+.Xr pthread_attr_setstackaddr 3
+functions.
+If no attributes for the thread stack size are specified, the default
+non-initial thread stack size is 2MB for 64bit architectures, and 1MB
+for 32bit architectures.
+.Sh RUN-TIME SETTINGS
+The following environment variables are recognized by
+.Nm
+and adjust the operation of the library at run-time:
+.Bl -tag -width LIBPTHREAD_SPLITSTACK_MAIN
+.It Ev LIBPTHREAD_BIGSTACK_MAIN
+Disables the reduction of the initial thread stack enabled by
+.Ev LIBPTHREAD_SPLITSTACK_MAIN .
+.It Ev LIBPTHREAD_SPLITSTACK_MAIN
+Causes a reduction of the initial thread stack, as described in the
+section
+.Sx THREAD STACKS .
+This was the default

Re: libthr and main thread stack size

2014-09-22 Thread Daniel Eischen 

On Sun, 21 Sep 2014, Julian Elischer wrote:


On 9/20/14, 3:27 AM, John Baldwin wrote:

On Tuesday, September 16, 2014 11:13:24 AM Konstantin Belousov wrote:

On Mon, Sep 15, 2014 at 03:47:41PM -0600, Justin T. Gibbs wrote:

On Aug 8, 2014, at 5:22 AM, Konstantin Belousov kostik...@gmail.com
wrote:

?


Below is the patch which adds environment variable
LIBPTHREAD_BIGSTACK_MAIN. Setting it to any value results in the
main thread stack left as is, and other threads allocate stack
below the area of RLIMIT_STACK. Try it. I do not want to set this
behaviour as default.

Is there a reason this should not be the default? Looking at the
getrlimit() page on the OpenGroup?s site they say:

RLIMIT_STACK This is the maximum size of the initial thread's stack,
in bytes. The implementation does not automatically grow the stack
beyond this limit. If this limit is exceeded, SIGSEGV shall be
generated for the thread. If the thread is blocking SIGSEGV, or the
process is ignoring or catching SIGSEGV and has not made arrangements
to use an alternate stack, the disposition of SIGSEGV shall be set to
SIG_DFL before it is generated.

Does posix say something different?

I ran into this issue when debugging a segfault on Postgres when
running an (arguably quite bogus) query that should have fit within
both the configured stack rlimit and Postgres? configured stack limit.
The Postgres backend is really just single threaded, but happens
to pull in libpthread due to the threading support in some of the
libraries it uses. The segfault definitely violates POLA.

? Justin

I am conservative to not disturb the address space layout in single go.
If enough people test this setting, I can consider flipping the default
to the reverse.

I am still curious why the things were done in this way, but nobody
replied.

I suspect it was done out of reasons of being overly conservative in
interpreting RLIMIT_STACK.  I think it is quite surprising behavior though 
and
would rather we make your option the default and implement what the Open 
Group

says above.


that is my memory..
The transition from a non threaded app to a threaded app with one thread is 
sort of an undefined area.

Feel free to change it if Dan agrees..


I'm all for adopting what POSIX specifies as the default.  I
would shy away from adding another knob (LIBPTHREAD_BIGSTACK_MAIN)
if possible.

--
DE
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: libthr and main thread stack size

2014-09-22 Thread John Baldwin 
On Sunday, September 21, 2014 09:36:25 PM Justin T. Gibbs wrote:
 On Sep 20, 2014, at 11:06 AM, Konstantin Belousov kostik...@gmail.com 
wrote:
  On Fri, Sep 19, 2014 at 03:27:25PM -0400, John Baldwin wrote:
  I suspect it was done out of reasons of being overly conservative in
  interpreting RLIMIT_STACK. I think it is quite surprising behavior
  though and would rather we make your option the default and implement
  what the Open Group says above.
  
  Ok, below is the patch.  I felt bad about adding yet another magic and
  undocumented tunable to our libthr.  Since there seems to be no
  alternative than a tunable to enforce old behaviour, I documented
  the quirks I am aware of.
 
 Why do we need to support the old behavior?  Any program that ran in the old
 model will run in the new.  In the unlikely event that someone was using
 the old scheme for administrative control, there are other mechanisms for
 this already available that we can point them to instead.

I agree with this.  In my experience the issue it has always been the opposite 
(people having issues with the main stack shrinking).

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: libthr and main thread stack size

2014-09-21 Thread Justin T. Gibbs 
On Sep 20, 2014, at 11:06 AM, Konstantin Belousov kostik...@gmail.com wrote:

 On Fri, Sep 19, 2014 at 03:27:25PM -0400, John Baldwin wrote:
 I suspect it was done out of reasons of being overly conservative in
 interpreting RLIMIT_STACK. I think it is quite surprising behavior
 though and would rather we make your option the default and implement
 what the Open Group says above.
 
 Ok, below is the patch.  I felt bad about adding yet another magic and
 undocumented tunable to our libthr.  Since there seems to be no
 alternative than a tunable to enforce old behaviour, I documented
 the quirks I am aware of.

Why do we need to support the old behavior?  Any program that ran in the old 
model will run in the new.  In the unlikely event that someone was using the 
old scheme for administrative control, there are other mechanisms for this 
already available that we can point them to instead.

—
Justin


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: libthr and main thread stack size

2014-09-20 Thread Konstantin Belousov 
On Fri, Sep 19, 2014 at 03:27:25PM -0400, John Baldwin wrote:
 I suspect it was done out of reasons of being overly conservative in
 interpreting RLIMIT_STACK. I think it is quite surprising behavior
 though and would rather we make your option the default and implement
 what the Open Group says above.

Ok, below is the patch.  I felt bad about adding yet another magic and
undocumented tunable to our libthr.  Since there seems to be no
alternative than a tunable to enforce old behaviour, I documented
the quirks I am aware of.

Doc people, please review the man page in the patch.

diff --git a/lib/libthr/thread/thr_init.c b/lib/libthr/thread/thr_init.c
index 9bf0e29..72a067a 100644
--- a/lib/libthr/thread/thr_init.c
+++ b/lib/libthr/thread/thr_init.c
@@ -445,7 +445,7 @@ init_private(void)
struct rlimit rlim;
size_t len;
int mib[2];
-   char *env;
+   char *env, *env_bigstack, *env_splitstack;
 
_thr_umutex_init(_mutex_static_lock);
_thr_umutex_init(_cond_static_lock);
@@ -473,8 +473,9 @@ init_private(void)
len = sizeof (_usrstack);
if (sysctl(mib, 2, _usrstack, len, NULL, 0) == -1)
PANIC(Cannot get kern.usrstack from sysctl);
-   env = getenv(LIBPTHREAD_BIGSTACK_MAIN);
-   if (env != NULL) {
+   env_bigstack = getenv(LIBPTHREAD_BIGSTACK_MAIN);
+   env_splitstack = getenv(LIBPTHREAD_SPLITSTACK_MAIN);
+   if (bigstack != NULL || env_splitstack == NULL) {
if (getrlimit(RLIMIT_STACK, rlim) == -1)
PANIC(Cannot get stack rlimit);
_thr_stack_initial = rlim.rlim_cur;
diff --git a/share/man/man7/libthr.7 b/share/man/man7/libthr.7
new file mode 100644
index 000..16d916f
--- /dev/null
+++ b/share/man/man7/libthr.7
@@ -0,0 +1,254 @@
+.\ Copyright (c) 2014 The FreeBSD Foundation, Inc.
+.\ All rights reserved.
+.\
+.\ This documentation was written by Konstantin Belousov k...@freebsd.org
+.\ under sponsorship from the FreeBSD Foundation.
+.\
+.\ Redistribution and use in source and binary forms, with or without
+.\ modification, are permitted provided that the following conditions
+.\ are met:
+.\ 1. Redistributions of source code must retain the above copyright
+.\notice, this list of conditions and the following disclaimer.
+.\ 2. Redistributions in binary form must reproduce the above copyright
+.\notice, this list of conditions and the following disclaimer in the
+.\documentation and/or other materials provided with the distribution.
+.\
+.\ THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
+.\ ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\ ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
+.\ FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\ OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\ HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\ LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\ OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\ SUCH DAMAGE.
+.\
+.\ $FreeBSD$
+.\
+.Dd September 20, 2014
+.Dt libthr
+.Os
+.Sh NAME
+.Nm libthr
+.Nd FreeBSD implementation of the Posix threading library
+.Sh LIBRARY
+.Lb libpthread
+.Sh DESCRIPTION
+The man page documents the quirks and tunables of the
+.Fx
+implementation for the
+.Lb libpthread .
+When linking with the
+.Li -lpthread ,
+the run-time dependency
+.Dv libthr.so.3
+library is recorded in the produced object.
+.Pp
+The library is tigthly integrated with the Run-time Link-editor
+.Xr ld-elf.so.1 1
+and
+.Lb libc ,
+all three components must be built from the same source tree.
+Mixing
+.Dv libc.so
+and
+.Nm
+libraries from different versions of
+.Fx
+is not supported.
+The run-time linker
+.Li ld-elf.so.1
+has some code to ensure backward-compatibility with older
+.Nm .
+.Sh MUTEX ACQUISITION
+The locked mutex (see
+.Xr pthread_mutex_lock 3 )
+is represented by a volatile variable of type
+.Dv lwpid_t ,
+which records the global system identifier of the thread
+owning the lock.
+The
+.Nm
+performs a congested mutex acquisition in three stages, each of which
+is more resource-consuming than the previous.
+.Pp
+First, the
+.Li spin loop
+is performed, where the library attempts to acquire the lock by
+.Xr atomic 9
+operations.
+The loop count is controlled by the
+.Ev LIBPTHREAD_SPINLOOPS
+environment variable.
+.Pp
+If the
+.Li spin loop
+was unable to acquire the mutex, the
+.Li yield loop
+is executed, performing the same
+.Xr atomic 9
+acquisition attempts as
+.Li spin loop ,
+but each attempt is followed by yield of the CPU time of the thread by
+.Xr

Re: libthr and main thread stack size

2014-09-20 Thread Julian Elischer 

On 9/20/14, 3:27 AM, John Baldwin wrote:

On Tuesday, September 16, 2014 11:13:24 AM Konstantin Belousov wrote:

On Mon, Sep 15, 2014 at 03:47:41PM -0600, Justin T. Gibbs wrote:

On Aug 8, 2014, at 5:22 AM, Konstantin Belousov kostik...@gmail.com
wrote:

?


Below is the patch which adds environment variable
LIBPTHREAD_BIGSTACK_MAIN. Setting it to any value results in the
main thread stack left as is, and other threads allocate stack
below the area of RLIMIT_STACK. Try it. I do not want to set this
behaviour as default.

Is there a reason this should not be the default? Looking at the
getrlimit() page on the OpenGroup?s site they say:

RLIMIT_STACK This is the maximum size of the initial thread's stack,
in bytes. The implementation does not automatically grow the stack
beyond this limit. If this limit is exceeded, SIGSEGV shall be
generated for the thread. If the thread is blocking SIGSEGV, or the
process is ignoring or catching SIGSEGV and has not made arrangements
to use an alternate stack, the disposition of SIGSEGV shall be set to
SIG_DFL before it is generated.

Does posix say something different?

I ran into this issue when debugging a segfault on Postgres when
running an (arguably quite bogus) query that should have fit within
both the configured stack rlimit and Postgres? configured stack limit.
The Postgres backend is really just single threaded, but happens
to pull in libpthread due to the threading support in some of the
libraries it uses. The segfault definitely violates POLA.

? Justin

I am conservative to not disturb the address space layout in single go.
If enough people test this setting, I can consider flipping the default
to the reverse.

I am still curious why the things were done in this way, but nobody
replied.

I suspect it was done out of reasons of being overly conservative in
interpreting RLIMIT_STACK.  I think it is quite surprising behavior though and
would rather we make your option the default and implement what the Open Group
says above.


that is my memory..
The transition from a non threaded app to a threaded app with one 
thread is sort of an undefined area.

Feel free to change it if Dan agrees..
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: libthr and main thread stack size

2014-09-19 Thread John Baldwin 
On Tuesday, September 16, 2014 11:13:24 AM Konstantin Belousov wrote:
 On Mon, Sep 15, 2014 at 03:47:41PM -0600, Justin T. Gibbs wrote:
  On Aug 8, 2014, at 5:22 AM, Konstantin Belousov kostik...@gmail.com
  wrote:
  
  ?
  
   Below is the patch which adds environment variable
   LIBPTHREAD_BIGSTACK_MAIN. Setting it to any value results in the
   main thread stack left as is, and other threads allocate stack
   below the area of RLIMIT_STACK. Try it. I do not want to set this
   behaviour as default.
  
  Is there a reason this should not be the default? Looking at the
  getrlimit() page on the OpenGroup?s site they say:
  
  RLIMIT_STACK This is the maximum size of the initial thread's stack,
  in bytes. The implementation does not automatically grow the stack
  beyond this limit. If this limit is exceeded, SIGSEGV shall be
  generated for the thread. If the thread is blocking SIGSEGV, or the
  process is ignoring or catching SIGSEGV and has not made arrangements
  to use an alternate stack, the disposition of SIGSEGV shall be set to
  SIG_DFL before it is generated.
  
  Does posix say something different?
  
  I ran into this issue when debugging a segfault on Postgres when
  running an (arguably quite bogus) query that should have fit within
  both the configured stack rlimit and Postgres? configured stack limit.
  The Postgres backend is really just single threaded, but happens
  to pull in libpthread due to the threading support in some of the
  libraries it uses. The segfault definitely violates POLA.
  
  ? Justin
 
 I am conservative to not disturb the address space layout in single go.
 If enough people test this setting, I can consider flipping the default
 to the reverse.
 
 I am still curious why the things were done in this way, but nobody
 replied.

I suspect it was done out of reasons of being overly conservative in 
interpreting RLIMIT_STACK.  I think it is quite surprising behavior though and 
would rather we make your option the default and implement what the Open Group 
says above.

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: libthr and main thread stack size

2014-09-16 Thread Konstantin Belousov 
On Mon, Sep 15, 2014 at 03:47:41PM -0600, Justin T. Gibbs wrote:
 On Aug 8, 2014, at 5:22 AM, Konstantin Belousov kostik...@gmail.com wrote:
 
 ?
 
  Below is the patch which adds environment variable
  LIBPTHREAD_BIGSTACK_MAIN. Setting it to any value results in the
  main thread stack left as is, and other threads allocate stack
  below the area of RLIMIT_STACK. Try it. I do not want to set this
  behaviour as default.

 Is there a reason this should not be the default? Looking at the
 getrlimit() page on the OpenGroup?s site they say:

 RLIMIT_STACK This is the maximum size of the initial thread's stack,
 in bytes. The implementation does not automatically grow the stack
 beyond this limit. If this limit is exceeded, SIGSEGV shall be
 generated for the thread. If the thread is blocking SIGSEGV, or the
 process is ignoring or catching SIGSEGV and has not made arrangements
 to use an alternate stack, the disposition of SIGSEGV shall be set to
 SIG_DFL before it is generated.

 Does posix say something different?

 I ran into this issue when debugging a segfault on Postgres when
 running an (arguably quite bogus) query that should have fit within
 both the configured stack rlimit and Postgres? configured stack limit.
 The Postgres backend is really just single threaded, but happens
 to pull in libpthread due to the threading support in some of the
 libraries it uses. The segfault definitely violates POLA.

 ? Justin

I am conservative to not disturb the address space layout in single go.
If enough people test this setting, I can consider flipping the default
to the reverse.

I am still curious why the things were done in this way, but nobody
replied.


pgpTZsjgorg00.pgp
Description: PGP signature

Re: libthr and main thread stack size

2014-09-15 Thread Justin T. Gibbs 
On Aug 8, 2014, at 5:22 AM, Konstantin Belousov kostik...@gmail.com wrote:

…

 Below is the patch which adds environment variable LIBPTHREAD_BIGSTACK_MAIN.
 Setting it to any value results in the main thread stack left as is, and
 other threads allocate stack below the area of RLIMIT_STACK.  Try it.
 I do not want to set this behaviour as default.

Is there a reason this should not be the default?  Looking at the getrlimit() 
page on the OpenGroup’s site they say:

RLIMIT_STACK
This is the maximum size of the initial thread's stack, in bytes. The 
implementation does not automatically grow the stack beyond this limit. If this 
limit is exceeded, SIGSEGV shall be generated for the thread. If the thread is 
blocking SIGSEGV, or the process is ignoring or catching SIGSEGV and has not 
made arrangements to use an alternate stack, the disposition of SIGSEGV shall 
be set to SIG_DFL before it is generated.

Does posix say something different?

I ran into this issue when debugging a segfault on Postgres when running an 
(arguably quite bogus) query that should have fit within both the configured 
stack rlimit and Postgres’ configured stack limit.  The Postgres backend is 
really just single threaded, but happens to pull in libpthread due to the 
threading support in some of the libraries it uses.  The segfault definitely 
violates POLA.

—
Justin


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: libthr and main thread stack size

2014-08-14 Thread Ivan A. Kosarev 

On 08/13/2014 01:40 AM, Dimitry Andric wrote:

On 08 Aug 2014, at 13:22, Konstantin Belousov kostik...@gmail.com wrote:

On Fri, Aug 08, 2014 at 12:32:56PM +0400, Ivan A. Kosarev wrote:

On 08/08/2014 09:28 AM, Konstantin Belousov wrote:

On Thu, Aug 07, 2014 at 04:18:12PM +0400, Ivan A. Kosarev wrote:

Hello,

According to libthr's thr_init.c (the 9.2 version) init_main_thread()
allocates s.c. red zone below the main stack in order to protect other
stacks. The size of the main stack is determined by the
_thr_stack_initial variable that is declared extern though it doesn't
seem it can be changed. The value of the variable is set to 4M on 64-bit
platforms which is obviously not sufficient for the most of real programs.

Can anyone please confirm that there is no way to increase the stack
size for the main thread and thus any program linked against libthr has
only a few megabytes of stack memory for its main thread--whatever the
system stack size (ulimit -s) is set to?

Yes, there is no way to change the main thread stack clamping.
Could you provide a reasonable use case for the 4MB stack ?

Traversing trees with recursive functions or on-stack grammar parsers?

I just ran into a similar issue while running one of clang 3.5's test
cases (see http://llvm.org/PR20635 ).  On i386, it used up approximately
2MB of stack, then ran into the guard page, and segfaulted.

I was quite amazed to find out that ulimit -s didn't help at all, until
I remembered this thread. :-)


Yes, that was the point: at least some of real programs consume more 
stack space than what we have with the hardcoded libthr's limits. Also, 
it looks a bit weird to me that adding thread support/linking against 
libthr affects how much stack space we can consume and whether it can be 
controlled with ulimit.


The original issue I've run into is one of the Asan's tests:

http://reviews.llvm.org/D4562

--

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: libthr and main thread stack size

2014-08-12 Thread Dimitry Andric 
On 08 Aug 2014, at 13:22, Konstantin Belousov kostik...@gmail.com wrote:
 On Fri, Aug 08, 2014 at 12:32:56PM +0400, Ivan A. Kosarev wrote:
 
 On 08/08/2014 09:28 AM, Konstantin Belousov wrote:
 On Thu, Aug 07, 2014 at 04:18:12PM +0400, Ivan A. Kosarev wrote:
 Hello,
 
 According to libthr's thr_init.c (the 9.2 version) init_main_thread()
 allocates s.c. red zone below the main stack in order to protect other
 stacks. The size of the main stack is determined by the
 _thr_stack_initial variable that is declared extern though it doesn't
 seem it can be changed. The value of the variable is set to 4M on 64-bit
 platforms which is obviously not sufficient for the most of real programs.
 
 Can anyone please confirm that there is no way to increase the stack
 size for the main thread and thus any program linked against libthr has
 only a few megabytes of stack memory for its main thread--whatever the
 system stack size (ulimit -s) is set to?
 Yes, there is no way to change the main thread stack clamping.
 Could you provide a reasonable use case for the 4MB stack ?
 
 Traversing trees with recursive functions or on-stack grammar parsers?

I just ran into a similar issue while running one of clang 3.5's test
cases (see http://llvm.org/PR20635 ).  On i386, it used up approximately
2MB of stack, then ran into the guard page, and segfaulted.

I was quite amazed to find out that ulimit -s didn't help at all, until
I remembered this thread. :-)


 Anyway, I somewhat sympathize to the idea to stop clamping the main
 thread stack, and to not reuse it for other threads stack carving.
 This also means that non-main threads stack allocator should stop
 tracking the explicit location for the stacks and rely on vm mmap(2)
 base selection instead.
 
 Yes, that would solve the problem.
 
 I do not know the motivations why the current scheme of stacks allocation
 was chosen.  The changes do not look too involved.
 In fact, I can resonably explain the current behaviour. The motivation
 seems to come from desire to interpret the RLIMIT_STACK as the global
 limit to the stack usage by all threads. From this PoV, it becomes clean
 why the other thread stacks are carved from the main stack.

Linux seems to interpret it as the default stack size for *each* new
thread (so I guess that also includes the main thread, if Linux has
such a concept):

''On Linux/x86-32, the default stack size for a new thread is 2
megabytes.  Under the NPTL threading implementation, if the RLIMIT_STACK
soft resource limit at the time the program started has any value other
than unlimited, then it determines the default stack size of new
threads.''


 Below is the patch which adds environment variable LIBPTHREAD_BIGSTACK_MAIN.
 Setting it to any value results in the main thread stack left as is, and
 other threads allocate stack below the area of RLIMIT_STACK.  Try it.
 I do not want to set this behaviour as default.

It works for my case, thanks.  I'm not sure if we should use Linux's
behavior of using RLIMIT_STACK for *all* threads, but I would definitely
expect that value for the main thread by default...

-Dimitry



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: libthr and main thread stack size

2014-08-08 Thread Ivan A. Kosarev 


On 08/08/2014 09:28 AM, Konstantin Belousov wrote:

On Thu, Aug 07, 2014 at 04:18:12PM +0400, Ivan A. Kosarev wrote:

Hello,

According to libthr's thr_init.c (the 9.2 version) init_main_thread()
allocates s.c. red zone below the main stack in order to protect other
stacks. The size of the main stack is determined by the
_thr_stack_initial variable that is declared extern though it doesn't
seem it can be changed. The value of the variable is set to 4M on 64-bit
platforms which is obviously not sufficient for the most of real programs.

Can anyone please confirm that there is no way to increase the stack
size for the main thread and thus any program linked against libthr has
only a few megabytes of stack memory for its main thread--whatever the
system stack size (ulimit -s) is set to?

Yes, there is no way to change the main thread stack clamping.
Could you provide a reasonable use case for the 4MB stack ?


Traversing trees with recursive functions or on-stack grammar parsers?



Anyway, I somewhat sympathize to the idea to stop clamping the main
thread stack, and to not reuse it for other threads stack carving.
This also means that non-main threads stack allocator should stop
tracking the explicit location for the stacks and rely on vm mmap(2)
base selection instead.


Yes, that would solve the problem.


I do not know the motivations why the current scheme of stacks allocation
was chosen.  The changes do not look too involved.


Thanks a lot.

--

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: libthr and main thread stack size

2014-08-08 Thread Konstantin Belousov 
On Fri, Aug 08, 2014 at 12:32:56PM +0400, Ivan A. Kosarev wrote:
 
 On 08/08/2014 09:28 AM, Konstantin Belousov wrote:
  On Thu, Aug 07, 2014 at 04:18:12PM +0400, Ivan A. Kosarev wrote:
  Hello,
 
  According to libthr's thr_init.c (the 9.2 version) init_main_thread()
  allocates s.c. red zone below the main stack in order to protect other
  stacks. The size of the main stack is determined by the
  _thr_stack_initial variable that is declared extern though it doesn't
  seem it can be changed. The value of the variable is set to 4M on 64-bit
  platforms which is obviously not sufficient for the most of real programs.
 
  Can anyone please confirm that there is no way to increase the stack
  size for the main thread and thus any program linked against libthr has
  only a few megabytes of stack memory for its main thread--whatever the
  system stack size (ulimit -s) is set to?
  Yes, there is no way to change the main thread stack clamping.
  Could you provide a reasonable use case for the 4MB stack ?
 
 Traversing trees with recursive functions or on-stack grammar parsers?
 
 
  Anyway, I somewhat sympathize to the idea to stop clamping the main
  thread stack, and to not reuse it for other threads stack carving.
  This also means that non-main threads stack allocator should stop
  tracking the explicit location for the stacks and rely on vm mmap(2)
  base selection instead.
 
 Yes, that would solve the problem.
 
  I do not know the motivations why the current scheme of stacks allocation
  was chosen.  The changes do not look too involved.
In fact, I can resonably explain the current behaviour. The motivation
seems to come from desire to interpret the RLIMIT_STACK as the global
limit to the stack usage by all threads. From this PoV, it becomes clean
why the other thread stacks are carved from the main stack.

Of course, it does not quite work this way, because there is no check
that we did not overflowed from the main stack area.

 
 Thanks a lot.
 

Below is the patch which adds environment variable LIBPTHREAD_BIGSTACK_MAIN.
Setting it to any value results in the main thread stack left as is, and
other threads allocate stack below the area of RLIMIT_STACK.  Try it.
I do not want to set this behaviour as default.

diff --git a/lib/libthr/thread/thr_init.c b/lib/libthr/thread/thr_init.c
index 937d83f..9bf0e29 100644
--- a/lib/libthr/thread/thr_init.c
+++ b/lib/libthr/thread/thr_init.c
@@ -37,6 +37,7 @@
 #include sys/types.h
 #include sys/signalvar.h
 #include sys/ioctl.h
+#include sys/resource.h
 #include sys/sysctl.h
 #include sys/ttycom.h
 #include sys/mman.h
@@ -441,6 +442,7 @@ init_main_thread(struct pthread *thread)
 static void
 init_private(void)
 {
+   struct rlimit rlim;
size_t len;
int mib[2];
char *env;
@@ -471,6 +473,12 @@ init_private(void)
len = sizeof (_usrstack);
if (sysctl(mib, 2, _usrstack, len, NULL, 0) == -1)
PANIC(Cannot get kern.usrstack from sysctl);
+   env = getenv(LIBPTHREAD_BIGSTACK_MAIN);
+   if (env != NULL) {
+   if (getrlimit(RLIMIT_STACK, rlim) == -1)
+   PANIC(Cannot get stack rlimit);
+   _thr_stack_initial = rlim.rlim_cur;
+   }
len = sizeof(_thr_is_smp);
sysctlbyname(kern.smp.cpus, _thr_is_smp, len, NULL, 0);
_thr_is_smp = (_thr_is_smp  1);
diff --git a/lib/libthr/thread/thr_stack.c b/lib/libthr/thread/thr_stack.c
index 15a9c82..e5d149e 100644
--- a/lib/libthr/thread/thr_stack.c
+++ b/lib/libthr/thread/thr_stack.c
@@ -246,7 +246,10 @@ _thr_stack_alloc(struct pthread_attr *attr)
THREAD_LIST_UNLOCK(curthread);
}
else {
-   /* Allocate a stack from usrstack. */
+   /*
+* Allocate a stack from or below usrstack, depending
+* on the LIBPTHREAD_BIGSTACK_MAIN env variable.
+*/
if (last_stack == NULL)
last_stack = _usrstack - _thr_stack_initial -
_thr_guard_default;
@@ -268,7 +271,7 @@ _thr_stack_alloc(struct pthread_attr *attr)
 
/* Map the stack and guard page together, and split guard
   page from allocated space: */
-   if ((stackaddr = mmap(stackaddr, stacksize+guardsize,
+   if ((stackaddr = mmap(stackaddr, stacksize + guardsize,
 _rtld_get_stack_prot(), MAP_STACK,
 -1, 0)) != MAP_FAILED 
(guardsize == 0 ||


pgpUGNSGxeqgN.pgp
Description: PGP signature

Re: libthr and main thread stack size

2014-08-07 Thread Konstantin Belousov 
On Thu, Aug 07, 2014 at 04:18:12PM +0400, Ivan A. Kosarev wrote:
 Hello,
 
 According to libthr's thr_init.c (the 9.2 version) init_main_thread() 
 allocates s.c. red zone below the main stack in order to protect other 
 stacks. The size of the main stack is determined by the 
 _thr_stack_initial variable that is declared extern though it doesn't 
 seem it can be changed. The value of the variable is set to 4M on 64-bit 
 platforms which is obviously not sufficient for the most of real programs.
 
 Can anyone please confirm that there is no way to increase the stack 
 size for the main thread and thus any program linked against libthr has 
 only a few megabytes of stack memory for its main thread--whatever the 
 system stack size (ulimit -s) is set to?

Yes, there is no way to change the main thread stack clamping.
Could you provide a reasonable use case for the 4MB stack ?

Anyway, I somewhat sympathize to the idea to stop clamping the main
thread stack, and to not reuse it for other threads stack carving.
This also means that non-main threads stack allocator should stop
tracking the explicit location for the stacks and rely on vm mmap(2)
base selection instead.

I do not know the motivations why the current scheme of stacks allocation
was chosen.  The changes do not look too involved.


pgpG3kDj2mpfu.pgp
Description: PGP signature