Re: [HACKERS] Threads in PostgreSQL

[Atri Sharma]

On Mon, Dec 21, 2015 at 11:51 AM, sri harsha 
wrote:

> Hi,
>
>Is it possible to use threads in Postgresql ?? I am using threads in my
> foreign data wrapper and i get the following error when i use the threads .
>
> *ERROR:  stack depth limit exceeded*
> *HINT:  Increase the configuration parameter "max_stack_depth" (currently
> 2048kB), after ensuring the platform's stack depth limit is adequate.*
>
>

PostgreSQL is a process-per-backend model.

Can you elaborate on what you are using the threads for?

Re: [HACKERS] Threads in PostgreSQL

[Chapman Flack]

On 12/21/15 01:24, Atri Sharma wrote:
> On Mon, Dec 21, 2015, sri harsha  wrote:
> 
>> I am using threads in my
>> foreign data wrapper and i get the following error when i use the threads .
>>
>> *ERROR:  stack depth limit exceeded*
>> *HINT:  Increase the configuration parameter "max_stack_depth" (currently
>> 2048kB), after ensuring the platform's stack depth limit is adequate.*

> PostgreSQL is a process-per-backend model.

To elaborate on that, it is a process per backend and most of the code
in that process has not been written for safe execution by multiple
threads. If your code starts other threads that only do other things,
but only one ever touches PostgreSQL backend code, that can be ok.
This happens in PL/Java, but it uses several interrelated precautions
to make sure no thread ever enters PostgreSQL backend code unless every
other thread is known to be out.

Even if you are taking that precaution, if the backend code is entered
by a different thread than last executed there, the stack depth tests
may be made by comparing the last thread's stack base to the current
thread's stack pointer, which will naturally give you a bogus result.
There is some API in miscadmin.h for manipulating the backend's idea
of the stack base, but there be dragons.

I am far short of the knowledgeable voices here, but in case you don't
hear from them right away, that is at least how I understand the matter.

Chapman Flack


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Threads in PostgreSQL

[Konstantin Knizhnik]

Hi,

PostgreSQL is not using threads but it is possible to spawn thread in 
your PostgreSQL extensions.

For example, I have used pool of threads in my IMCS extension.
But you need to build your  extension with -pthread:

CUSTOM_COPT = -pthread

Also, please take in account that many PostgreSQL functions (even in/out 
or comparison functions) are not reentrant: them are storing their state 
in global variables.
So you will get race conditions if you are calling  such functions from 
multiple threads.


Concerning stack overflow, I think that the most probable reason is 
trivial infinite recursion.

Did you inspect stack trace in debugger?






On 21.12.2015 09:21, sri harsha wrote:

Hi,

   Is it possible to use threads in Postgresql ?? I am using threads 
in my foreign data wrapper and i get the following error when i use 
the threads .


*ERROR:  stack depth limit exceeded*
*HINT:  Increase the configuration parameter "max_stack_depth" 
(currently 2048kB), after ensuring the platform's stack depth limit is 
adequate.*


No matter how much i increase the stack size , the error keeps 
occurring . How do i solve this problem ??



Thanks,
Harsha

Re: [HACKERS] threads stuff/UnixWare

[Bruce Momjian]

Tom Lane wrote:
 Bruce Momjian [EMAIL PROTECTED] writes:
  Larry Rosenman wrote:
  I agree.  the only issue is how to set up our makefiles to only do the
  -Kpthread/-pthreads(gcc) flags on the client code, and not do it for
  the backend itself.
 
  I think mixing a pgport that has thread flags with a backend that does
  not is more risky than just compiling everything with the same thread
  flags.
 
 Can we get this straight: is -Kpthread a compile flag or just a link flag?
 If the latter then it should not be needed in building the libpgport files.
 If the former then adding it to LDFLAGS is the wrong thing.

-Kpthread is needed for both compile and link, and the PTHREAD_CFLAGS is
used for both compile and link phases.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Thursday, May 13, 2004 10:05:22 -0400 Bruce Momjian 
[EMAIL PROTECTED] wrote:


Basically, as things set right now in CVS, Unixware is ready to go
because it thread for everything.  We don't have per-template thread
settings anymore because we test all of it in configure.
Was a change made to link initdb et al with -Kpthread?  It doesn't seem to
be in AnonCVS yet.


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Bruce Momjian]

Larry Rosenman wrote:
-- Start of PGP signed section.
 
 
 --On Thursday, May 13, 2004 10:05:22 -0400 Bruce Momjian 
 [EMAIL PROTECTED] wrote:
 
 
  Basically, as things set right now in CVS, Unixware is ready to go
  because it thread for everything.  We don't have per-template thread
  settings anymore because we test all of it in configure.
 
 Was a change made to link initdb et al with -Kpthread?  It doesn't seem to
 be in AnonCVS yet.

Really?  You are configuring with --enable-thread-safety?  I just
updated your template in CVS, and it is attached.  However, any old CVS
should work fine.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073
if test $GCC != yes; then
then
  # The -Kno_host is for a bug in the compiler.  See -hackers
  # discussion on 7-8/Aug/2003.
  cat conftest.c __EOF__
extern char *strcpy(char *, const char *);

static void f(char *p, int n){
strcpy(p+n,);
}
void g(void){
f(0, 0);
}
__EOF__

  if $CC -c -O -Kinline conftest.c conftest.err 21; then
CFLAGS=-O -Kinline
  else
CFLAGS=-O -Kinline,no_host
  fi
  rm -f conftest.*

  PTHREAD_CFLAGS=-Kpthread
fi

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Thursday, May 13, 2004 11:44:59 -0400 Bruce Momjian 
[EMAIL PROTECTED] wrote:

Larry Rosenman wrote:
-- Start of PGP signed section.


--On Thursday, May 13, 2004 10:05:22 -0400 Bruce Momjian
[EMAIL PROTECTED] wrote:
 Basically, as things set right now in CVS, Unixware is ready to go
 because it thread for everything.  We don't have per-template thread
 settings anymore because we test all of it in configure.
Was a change made to link initdb et al with -Kpthread?  It doesn't seem
to be in AnonCVS yet.
Really?  You are configuring with --enable-thread-safety?  I just
updated your template in CVS, and it is attached.  However, any old CVS
should work fine.
Nope, initdb is where we still die:

cc -O -Kinline initdb.o exec.o -L../../../src/interfaces/libpq -lpq 
-L../../../src/port -L/usr/local/lib -Wl,-R/usr/local/pgsql/lib -lz 
-lreadline -ltermcap -lresolv -lgen -lld -lsocket -lnsl -ldl -lm  -lpgport 
-o initdb
Undefined   first referenced
symbol  in file
pthread_mutex_unlocklibpq.so
pthread_getspecific libpq.so
pthread_mutex_lock  libpq.so
pthread_key_create  libpq.so
pthread_oncelibpq.so
pthread_setspecific libpq.so
UX:ld: ERROR: Symbol referencing errors. No output written to initdb
gmake[3]: *** [initdb] Error 1
gmake[3]: Leaving directory `/home/ler/pg-dev/pgsql-server/src/bin/initdb'
gmake[2]: *** [all] Error 2
gmake[2]: Leaving directory `/home/ler/pg-dev/pgsql-server/src/bin'
gmake[1]: *** [all] Error 2
gmake[1]: Leaving directory `/home/ler/pg-dev/pgsql-server/src'
gmake: *** [all] Error 2
$

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Bruce Momjian]

Larry Rosenman wrote:
  Really?  You are configuring with --enable-thread-safety?  I just
  updated your template in CVS, and it is attached.  However, any old CVS
  should work fine.
 Nope, initdb is where we still die:
 

OH!  I remember now.  What we have to do for this platform only is to
pass the thread flags into all compile/links.  I forgot that step.

Let me work on that and send you a patch.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] threads stuff/UnixWare

[Zeugswetter Andreas SB SD]

 I know, this sucks, but, I don't see any other way, other than linking
 *ALL* libpq-using programs (including initdb and friends) with -K
pthread.

How about making a libpq.so (without pthread) and a thread safe
libpq_r.so ?

Andreas

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Thursday, May 13, 2004 09:54:02 +0200 Zeugswetter Andreas SB SD 
[EMAIL PROTECTED] wrote:


I know, this sucks, but, I don't see any other way, other than linking
*ALL* libpq-using programs (including initdb and friends) with -K
pthread.

How about making a libpq.so (without pthread) and a thread safe
libpq_r.so ?
That's against the way this platform does things.  I.E. the only
library is libc.  All the other system supplied libs do the right thing
in the presence or absence of libthread in the image.
I did get a note from my SCO contacts that they are looking into how
to make it easier for stuff to be threads ready, but I don't expect
that to be ready for 7.5 release.
The -Kpthread on all libpq using programs is the easiest way FOR NOW.

Thanks,
LER
Andreas


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Tom Lane]

Larry Rosenman [EMAIL PROTECTED] writes:
 I did get a note from my SCO contacts that they are looking into how
 to make it easier for stuff to be threads ready, but I don't expect
 that to be ready for 7.5 release.

 The -Kpthread on all libpq using programs is the easiest way FOR NOW.

Hmm.  If there is work happening at the platform level to improve
matters, then I'd definitely vote for taking the simplest solution
(ie -Kpthread everywhere) for now.  We can always do all that other
work later if nothing gets fixed; but why invest a lot of work for
what might be only a short-term problem?

regards, tom lane

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Thursday, May 13, 2004 09:18:21 -0400 Tom Lane [EMAIL PROTECTED] 
wrote:

Larry Rosenman [EMAIL PROTECTED] writes:
I did get a note from my SCO contacts that they are looking into how
to make it easier for stuff to be threads ready, but I don't expect
that to be ready for 7.5 release.

The -Kpthread on all libpq using programs is the easiest way FOR NOW.
Hmm.  If there is work happening at the platform level to improve
matters, then I'd definitely vote for taking the simplest solution
(ie -Kpthread everywhere) for now.  We can always do all that other
work later if nothing gets fixed; but why invest a lot of work for
what might be only a short-term problem?
I agree.  the only issue is how to set up our makefiles to only do the
-Kpthread/-pthreads(gcc) flags on the client code, and not do it for
the backend itself.

			regards, tom lane


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Bruce Momjian]

Larry Rosenman wrote:
-- Start of PGP signed section.
 
 
 --On Thursday, May 13, 2004 09:18:21 -0400 Tom Lane [EMAIL PROTECTED] 
 wrote:
 
  Larry Rosenman [EMAIL PROTECTED] writes:
  I did get a note from my SCO contacts that they are looking into how
  to make it easier for stuff to be threads ready, but I don't expect
  that to be ready for 7.5 release.
 
  The -Kpthread on all libpq using programs is the easiest way FOR NOW.
 
  Hmm.  If there is work happening at the platform level to improve
  matters, then I'd definitely vote for taking the simplest solution
  (ie -Kpthread everywhere) for now.  We can always do all that other
  work later if nothing gets fixed; but why invest a lot of work for
  what might be only a short-term problem?

 [ added blank line here ] 
 I agree.  the only issue is how to set up our makefiles to only do the
 -Kpthread/-pthreads(gcc) flags on the client code, and not do it for
 the backend itself.

I think mixing a pgport that has thread flags with a backend that does
not is more risky than just compiling everything with the same thread
flags.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] threads stuff/UnixWare

[Tom Lane]

Bruce Momjian [EMAIL PROTECTED] writes:
 Larry Rosenman wrote:
 I agree.  the only issue is how to set up our makefiles to only do the
 -Kpthread/-pthreads(gcc) flags on the client code, and not do it for
 the backend itself.

 I think mixing a pgport that has thread flags with a backend that does
 not is more risky than just compiling everything with the same thread
 flags.

Can we get this straight: is -Kpthread a compile flag or just a link flag?
If the latter then it should not be needed in building the libpgport files.
If the former then adding it to LDFLAGS is the wrong thing.

regards, tom lane

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] threads stuff/UnixWare

[Marc G. Fournier]

On Wed, 12 May 2004, Larry Rosenman wrote:

 At the risk of getting my butt kicked again, is there any way we can
 talk about how to deal with threads on UnixWare and the libpq stuff?

 Has any other platform come up with a need to look for the real pthread_*
 calls from libpq?

 I would REALLY like us to support threaded programs in UnixWare, and would
 like to NOT antagonize the community, but would like to have a direction
 I can go to make this happen in 7.5 before we freeze.

Perfect time to start talking about it, at least :)

Do you have a proposed patch to look at for doing this?


Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 12:57:10 -0300 Marc G. Fournier 
[EMAIL PROTECTED] wrote:

On Wed, 12 May 2004, Larry Rosenman wrote:

At the risk of getting my butt kicked again, is there any way we can
talk about how to deal with threads on UnixWare and the libpq stuff?
Has any other platform come up with a need to look for the real pthread_*
calls from libpq?
I would REALLY like us to support threaded programs in UnixWare, and
would like to NOT antagonize the community, but would like to have a
direction I can go to make this happen in 7.5 before we freeze.
Perfect time to start talking about it, at least :)

Do you have a proposed patch to look at for doing this?
No, as I believe the way I want to do it was frowned upon.

I'd LIKE to be able to have PG wrappers for those functions, and
have the first invocation of them look via dlsym() for the real ones, and 
if they are NOT there, use fake functions that assume we are NOT threaded.

If they ARE present, indirect to the real functions.

I am looking for direction before expending a lot of effort on it, if it 
will ultimately be rejected.



Marc G. Fournier   Hub.Org Networking Services
(http://www.hub.org) Email: [EMAIL PROTECTED]   Yahoo!: yscrappy
ICQ: 7615664


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
 joining column's datatypes do not match

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 12:58:58 -0400 Tom Lane [EMAIL PROTECTED] 
wrote:

Larry Rosenman [EMAIL PROTECTED] writes:
At the risk of getting my butt kicked again, is there any way we can
talk about how to deal with threads on UnixWare and the libpq stuff?
In what way does the current thread stuff not work for you?

			regards, tom lane
In the initdb compile:

Undefined   first referenced
symbol  in file
pthread_mutex_unlocklibpq.so
pthread_getspecific libpq.so
pthread_mutex_lock  libpq.so
pthread_key_create  libpq.so
pthread_oncelibpq.so
pthread_setspecific libpq.so
UX:ld: ERROR: Symbol referencing errors. No output written to initdb
gmake[3]: *** [initdb] Error 1
gmake[3]: Leaving directory `/home/ler/pg-dev/pgsql-server/src/bin/initdb'
gmake[2]: *** [all] Error 2
gmake[2]: Leaving directory `/home/ler/pg-dev/pgsql-server/src/bin'
gmake[1]: *** [all] Error 2
gmake[1]: Leaving directory `/home/ler/pg-dev/pgsql-server/src'
gmake: *** [all] Error 2
$
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] threads stuff/UnixWare

[Tom Lane]

Larry Rosenman [EMAIL PROTECTED] writes:
 At the risk of getting my butt kicked again, is there any way we can
 talk about how to deal with threads on UnixWare and the libpq stuff?

In what way does the current thread stuff not work for you?

regards, tom lane

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] threads stuff/UnixWare

[Tom Lane]

Larry Rosenman [EMAIL PROTECTED] writes:
 --On Wednesday, May 12, 2004 12:58:58 -0400 Tom Lane [EMAIL PROTECTED] 
 In what way does the current thread stuff not work for you?

 In the initdb compile:

 Undefined   first referenced
 symbol  in file
 pthread_mutex_unlocklibpq.so
 pthread_getspecific libpq.so
 pthread_mutex_lock  libpq.so
 pthread_key_create  libpq.so
 pthread_oncelibpq.so
 pthread_setspecific libpq.so

Hmm.  And that means what?  Does Unixware not have these functions?
Are we just failing to suck in the needed library?

regards, tom lane

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 14:14:30 -0300 Marc G. Fournier 
[EMAIL PROTECTED] wrote:

On Wed, 12 May 2004, Larry Rosenman wrote:

I'd LIKE to be able to have PG wrappers for those functions, and have
the first invocation of them look via dlsym() for the real ones, and if
they are NOT there, use fake functions that assume we are NOT threaded.
Wouldn't it be easier to have a #define?

Correct me if I'm wrong here, but the problem is that we are expecting
thread functinos to be called x_, while Unixware defines them as pthread_,
right? so make Unixware specific #defines that map x_ to pthread_ ...
No, we are expecting them to have pthread_*.

On UnixWare, to get the pthread_* functions, you need to invoke a compiler
option (-Kpthread), to cause libpthread to be linked in.
Since libpq.so now REQUIRES the function, we need to either:

1) force ANY program that uses libpq to be compiled/linked with -Kpthread
or
2) dynamically determine if libpthread is loaded and use the real pthread_*
functions if they are there (libpthread is in the image), or use fake
singlethread versions of the pthread_* calls if libpthread is NOT in the 
image.





Marc G. Fournier   Hub.Org Networking Services
(http://www.hub.org) Email: [EMAIL PROTECTED]   Yahoo!: yscrappy
ICQ: 7615664


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] threads stuff/UnixWare

[Marc G. Fournier]

On Wed, 12 May 2004, Larry Rosenman wrote:

 I'd LIKE to be able to have PG wrappers for those functions, and have
 the first invocation of them look via dlsym() for the real ones, and if
 they are NOT there, use fake functions that assume we are NOT threaded.

Wouldn't it be easier to have a #define?

Correct me if I'm wrong here, but the problem is that we are expecting
thread functinos to be called x_, while Unixware defines them as pthread_,
right? so make Unixware specific #defines that map x_ to pthread_ ...




Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 13:18:35 -0400 Tom Lane [EMAIL PROTECTED] 
wrote:

Larry Rosenman [EMAIL PROTECTED] writes:
--On Wednesday, May 12, 2004 12:58:58 -0400 Tom Lane [EMAIL PROTECTED]
In what way does the current thread stuff not work for you?

In the initdb compile:

Undefined   first referenced
symbol  in file
pthread_mutex_unlocklibpq.so
pthread_getspecific libpq.so
pthread_mutex_lock  libpq.so
pthread_key_create  libpq.so
pthread_oncelibpq.so
pthread_setspecific libpq.so
Hmm.  And that means what?  Does Unixware not have these functions?
Are we just failing to suck in the needed library?
			regards, tom lane
No, we don't pass the -Kpthread (thread CFLAGS) to the C links.

This is the whole discussion we had back in January/February about forcing
-Kpthread for *ALL* libpq using programs, or dynamically determining
if the image already is linked -Kpthread, or not supporting threads
at all on UW.
#3 is unacceptable to me.

LER



--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
 joining column's datatypes do not match

Re: [HACKERS] threads stuff/UnixWare

[Marc G. Fournier]

On Wed, 12 May 2004, Larry Rosenman wrote:



 --On Wednesday, May 12, 2004 14:14:30 -0300 Marc G. Fournier
 [EMAIL PROTECTED] wrote:

  On Wed, 12 May 2004, Larry Rosenman wrote:
 
  I'd LIKE to be able to have PG wrappers for those functions, and have
  the first invocation of them look via dlsym() for the real ones, and if
  they are NOT there, use fake functions that assume we are NOT threaded.
 
  Wouldn't it be easier to have a #define?
 
  Correct me if I'm wrong here, but the problem is that we are expecting
  thread functinos to be called x_, while Unixware defines them as pthread_,
  right? so make Unixware specific #defines that map x_ to pthread_ ...
 No, we are expecting them to have pthread_*.

 On UnixWare, to get the pthread_* functions, you need to invoke a compiler
 option (-Kpthread), to cause libpthread to be linked in.

 Since libpq.so now REQUIRES the function, we need to either:

 1) force ANY program that uses libpq to be compiled/linked with -Kpthread

Ummm, shouldn't that be added to the port specific Makefile?


Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 15:02:30 -0300 Marc G. Fournier 
[EMAIL PROTECTED] wrote:

On Wed, 12 May 2004, Larry Rosenman wrote:



--On Wednesday, May 12, 2004 14:14:30 -0300 Marc G. Fournier
[EMAIL PROTECTED] wrote:
 On Wed, 12 May 2004, Larry Rosenman wrote:

 I'd LIKE to be able to have PG wrappers for those functions, and have
 the first invocation of them look via dlsym() for the real ones, and
 if they are NOT there, use fake functions that assume we are NOT
 threaded.

 Wouldn't it be easier to have a #define?

 Correct me if I'm wrong here, but the problem is that we are expecting
 thread functinos to be called x_, while Unixware defines them as
 pthread_, right? so make Unixware specific #defines that map x_ to
 pthread_ ...
No, we are expecting them to have pthread_*.
On UnixWare, to get the pthread_* functions, you need to invoke a
compiler option (-Kpthread), to cause libpthread to be linked in.
Since libpq.so now REQUIRES the function, we need to either:

1) force ANY program that uses libpq to be compiled/linked with -Kpthread
Ummm, shouldn't that be added to the port specific Makefile?
See my reply to Tom.  It forces ALL libpq using programs to be
linked with -Kpthread, which was deemed unacceptable.


Marc G. Fournier   Hub.Org Networking Services
(http://www.hub.org) Email: [EMAIL PROTECTED]   Yahoo!: yscrappy
ICQ: 7615664


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 15:39:34 -0300 Marc G. Fournier 
[EMAIL PROTECTED] wrote:

On Wed, 12 May 2004, Larry Rosenman wrote:



--On Wednesday, May 12, 2004 15:02:30 -0300 Marc G. Fournier
[EMAIL PROTECTED] wrote:
 On Wed, 12 May 2004, Larry Rosenman wrote:



 --On Wednesday, May 12, 2004 14:14:30 -0300 Marc G. Fournier
 [EMAIL PROTECTED] wrote:

  On Wed, 12 May 2004, Larry Rosenman wrote:
 
  I'd LIKE to be able to have PG wrappers for those functions, and
  have the first invocation of them look via dlsym() for the real
  ones, and if they are NOT there, use fake functions that assume we
  are NOT threaded.
 
  Wouldn't it be easier to have a #define?
 
  Correct me if I'm wrong here, but the problem is that we are
  expecting thread functinos to be called x_, while Unixware defines
  them as pthread_, right? so make Unixware specific #defines that
  map x_ to pthread_ ...
 No, we are expecting them to have pthread_*.

 On UnixWare, to get the pthread_* functions, you need to invoke a
 compiler option (-Kpthread), to cause libpthread to be linked in.

 Since libpq.so now REQUIRES the function, we need to either:

 1) force ANY program that uses libpq to be compiled/linked with
 -Kpthread

 Ummm, shouldn't that be added to the port specific Makefile?
See my reply to Tom.  It forces ALL libpq using programs to be
linked with -Kpthread, which was deemed unacceptable.
deemed unacceptable by whom?  Sounds to me alot simpler of a solution then
making wrappers for the pthread_* functions just to accommodate one OS ...
I could see it if this was a wide-spread problem, but it doesn't appear to
be ...
Bruce didn't want to muck ip all the .c programs in the distribution to
do that.
As well as any users using libpq.

I'm all ears if the COMMUNITY wants to do it.

LER


Marc G. Fournier   Hub.Org Networking Services
(http://www.hub.org) Email: [EMAIL PROTECTED]   Yahoo!: yscrappy
ICQ: 7615664


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Marc G. Fournier]

On Wed, 12 May 2004, Larry Rosenman wrote:



 --On Wednesday, May 12, 2004 15:02:30 -0300 Marc G. Fournier
 [EMAIL PROTECTED] wrote:

  On Wed, 12 May 2004, Larry Rosenman wrote:
 
 
 
  --On Wednesday, May 12, 2004 14:14:30 -0300 Marc G. Fournier
  [EMAIL PROTECTED] wrote:
 
   On Wed, 12 May 2004, Larry Rosenman wrote:
  
   I'd LIKE to be able to have PG wrappers for those functions, and have
   the first invocation of them look via dlsym() for the real ones, and
   if they are NOT there, use fake functions that assume we are NOT
   threaded.
  
   Wouldn't it be easier to have a #define?
  
   Correct me if I'm wrong here, but the problem is that we are expecting
   thread functinos to be called x_, while Unixware defines them as
   pthread_, right? so make Unixware specific #defines that map x_ to
   pthread_ ...
  No, we are expecting them to have pthread_*.
 
  On UnixWare, to get the pthread_* functions, you need to invoke a
  compiler option (-Kpthread), to cause libpthread to be linked in.
 
  Since libpq.so now REQUIRES the function, we need to either:
 
  1) force ANY program that uses libpq to be compiled/linked with -Kpthread
 
  Ummm, shouldn't that be added to the port specific Makefile?
 See my reply to Tom.  It forces ALL libpq using programs to be
 linked with -Kpthread, which was deemed unacceptable.

deemed unacceptable by whom?  Sounds to me alot simpler of a solution then
making wrappers for the pthread_* functions just to accommodate one OS ...
I could see it if this was a wide-spread problem, but it doesn't appear to
be ...


Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 15:59:19 -0300 Marc G. Fournier 
[EMAIL PROTECTED] wrote:

On Wed, 12 May 2004, Larry Rosenman wrote:

  Ummm, shouldn't that be added to the port specific Makefile?
 See my reply to Tom.  It forces ALL libpq using programs to be
 linked with -Kpthread, which was deemed unacceptable.

 deemed unacceptable by whom?  Sounds to me alot simpler of a solution
 then making wrappers for the pthread_* functions just to accommodate
 one OS ... I could see it if this was a wide-spread problem, but it
 doesn't appear to be ...
Bruce didn't want to muck ip all the .c programs in the distribution to
do that.
I'm confused now ... what .c programs would have to be mucked up if you
add -Kpthread to the Unixware specific Makefile?
initdb, et al.

See these comments in src/template/unixware:
# Disabled because flags are required for all apps using libpq.
# Waiting to see if other platforms need this too.  2004-03-22
THREAD_SUPPORT=yes
# verified UnixWare 7.1.4 2004-03-18
STRERROR_THREADSAFE=yes
GETPWUID_THREADSAFE=yes
GETHOSTBYNAME_THREADSAFE=yes
$
I turned it back on to see if the new stuff had fixed it.




Marc G. Fournier   Hub.Org Networking Services
(http://www.hub.org) Email: [EMAIL PROTECTED]   Yahoo!: yscrappy
ICQ: 7615664


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Marc G. Fournier]

On Wed, 12 May 2004, Larry Rosenman wrote:

   Ummm, shouldn't that be added to the port specific Makefile?
  See my reply to Tom.  It forces ALL libpq using programs to be
  linked with -Kpthread, which was deemed unacceptable.
 
  deemed unacceptable by whom?  Sounds to me alot simpler of a solution then
  making wrappers for the pthread_* functions just to accommodate one OS ...
  I could see it if this was a wide-spread problem, but it doesn't appear to
  be ...
 Bruce didn't want to muck ip all the .c programs in the distribution to
 do that.

I'm confused now ... what .c programs would have to be mucked up if you
add -Kpthread to the Unixware specific Makefile?


Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 15:39:54 -0400 Tom Lane [EMAIL PROTECTED] 
wrote:

Larry Rosenman [EMAIL PROTECTED] writes:
This is the whole discussion we had back in January/February about
forcing -Kpthread for *ALL* libpq using programs, or dynamically
determining if the image already is linked -Kpthread, or not supporting
threads at all on UW.
Oh, that business :-(.  AFAIR we did not much care for any of the
proposed alternatives.  Have you thought of any new options?
At this point I'd settle for saying that --enable-thread-safety on
Unixware will generate a library that requires -Kpthread.  This is
kinda grungy but it seems that any more-pleasant solution would
require a disproportionate amount of work.
If I did the work for the dlsym() stuff would you and the rest of core@
accept it?

			regards, tom lane


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Tom Lane]

Larry Rosenman [EMAIL PROTECTED] writes:
 This is the whole discussion we had back in January/February about forcing
 -Kpthread for *ALL* libpq using programs, or dynamically determining
 if the image already is linked -Kpthread, or not supporting threads
 at all on UW.

Oh, that business :-(.  AFAIR we did not much care for any of the
proposed alternatives.  Have you thought of any new options?

At this point I'd settle for saying that --enable-thread-safety on
Unixware will generate a library that requires -Kpthread.  This is
kinda grungy but it seems that any more-pleasant solution would
require a disproportionate amount of work.

regards, tom lane

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] threads stuff/UnixWare

[Marc G. Fournier]

On Wed, 12 May 2004, Larry Rosenman wrote:



 --On Wednesday, May 12, 2004 15:59:19 -0300 Marc G. Fournier
 [EMAIL PROTECTED] wrote:

  On Wed, 12 May 2004, Larry Rosenman wrote:
 
Ummm, shouldn't that be added to the port specific Makefile?
   See my reply to Tom.  It forces ALL libpq using programs to be
   linked with -Kpthread, which was deemed unacceptable.
  
   deemed unacceptable by whom?  Sounds to me alot simpler of a solution
   then making wrappers for the pthread_* functions just to accommodate
   one OS ... I could see it if this was a wide-spread problem, but it
   doesn't appear to be ...
  Bruce didn't want to muck ip all the .c programs in the distribution to
  do that.
 
  I'm confused now ... what .c programs would have to be mucked up if you
  add -Kpthread to the Unixware specific Makefile?
 initdb, et al.

 See these comments in src/template/unixware:
 # Disabled because flags are required for all apps using libpq.
 # Waiting to see if other platforms need this too.  2004-03-22
 THREAD_SUPPORT=yes
 # verified UnixWare 7.1.4 2004-03-18
 STRERROR_THREADSAFE=yes
 GETPWUID_THREADSAFE=yes
 GETHOSTBYNAME_THREADSAFE=yes
 $

 I turned it back on to see if the new stuff had fixed it.

k, I just looked at the template/unixware file myself, and it looks to me
like -Kpthread is already defined, and included as part of THREAD_CPPFLAGS
... is THREAD_CPPFLAGS not honor'd in all of the various Makefile's (ie.
initdb?) ...

Please correct me if I'm wrong here, but so far, the issue seems to be
that now that libpq uses pthread_* by default, Unixware requires -Kpthread
to be used when building all the various support programs that use libpq
... and -Kpthread is defined in template/unixware ... so I'm *really*
confused as to what is actually broken, except maybe the Makefile's
themselves ...




Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] threads stuff/UnixWare

[Tom Lane]

Larry Rosenman [EMAIL PROTECTED] writes:
 --On Wednesday, May 12, 2004 15:39:54 -0400 Tom Lane [EMAIL PROTECTED]=20
 wrote:
 At this point I'd settle for saying that --enable-thread-safety on
 Unixware will generate a library that requires -Kpthread.  This is
 kinda grungy but it seems that any more-pleasant solution would
 require a disproportionate amount of work.

 If I did the work for the dlsym() stuff would you and the rest of core@
 accept it?

How invasive a change are we talking about?  I'd be inclined to reject a
patch that makes libpq materially less readable ...

regards, tom lane

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 16:00:48 -0400 Tom Lane [EMAIL PROTECTED] 
wrote:

Larry Rosenman [EMAIL PROTECTED] writes:
--On Wednesday, May 12, 2004 15:39:54 -0400 Tom Lane
[EMAIL PROTECTED]=20 wrote:
At this point I'd settle for saying that --enable-thread-safety on
Unixware will generate a library that requires -Kpthread.  This is
kinda grungy but it seems that any more-pleasant solution would
require a disproportionate amount of work.

If I did the work for the dlsym() stuff would you and the rest of core@
accept it?
How invasive a change are we talking about?  I'd be inclined to reject a
patch that makes libpq materially less readable ...
			regards, tom lane
I was thinking of pq_pthread_* calls, and that function would
set a static flag for calling either the real pthread_* function
or a statically named version in libpgport.a that is a single thread
wrapper.
I know, this sucks, but, I don't see any other way, other than linking
*ALL* libpq-using programs (including initdb and friends) with -K pthread.


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Tom Lane]

Larry Rosenman [EMAIL PROTECTED] writes:
 I was thinking of pq_pthread_* calls, and that function would
 set a static flag for calling either the real pthread_* function
 or a statically named version in libpgport.a that is a single thread
 wrapper.

And how will you avoid having a link-time dependency on the real pthread
function?  You muttered about dlsym but how much code will that take,
and what kind of runtime penalty will we incur?  (IIRC the pthread
functions are performance critical.)

Even more to the point, can you make it work at all?  I seem to recall
from the prior discussion that -Kpthread actually changes some code
generation details on that platform.  Are -Kpthread and non -Kpthread
libraries interoperable at all?

 I know, this sucks, but, I don't see any other way, other than linking
 *ALL* libpq-using programs (including initdb and friends) with -K pthread.

-Kpthread doesn't sound that bad to me, as long as it's documented.

regards, tom lane

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 16:22:58 -0400 Tom Lane [EMAIL PROTECTED] 
wrote:

Larry Rosenman [EMAIL PROTECTED] writes:
I was thinking of pq_pthread_* calls, and that function would
set a static flag for calling either the real pthread_* function
or a statically named version in libpgport.a that is a single thread
wrapper.
And how will you avoid having a link-time dependency on the real pthread
function?  You muttered about dlsym but how much code will that take,
and what kind of runtime penalty will we incur?  (IIRC the pthread
functions are performance critical.)
The first call to ANY of the pthread_* functions would set the flag, and
would cache the dlsym() info.

Even more to the point, can you make it work at all?  I seem to recall
from the prior discussion that -Kpthread actually changes some code
generation details on that platform.  Are -Kpthread and non -Kpthread
libraries interoperable at all?
Yes, this is how libc deals with it.

We wouldn't have a hard symbol for the pthread_* calls.

I can ask the compiler guys at SCO if you want.



I know, this sucks, but, I don't see any other way, other than linking
*ALL* libpq-using programs (including initdb and friends) with -K
pthread.
-Kpthread doesn't sound that bad to me, as long as it's documented.

			regards, tom lane


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 17:29:30 -0300 Marc G. Fournier 
[EMAIL PROTECTED] wrote:

On Wed, 12 May 2004, Larry Rosenman wrote:



--On Wednesday, May 12, 2004 16:00:48 -0400 Tom Lane [EMAIL PROTECTED]
wrote:
 Larry Rosenman [EMAIL PROTECTED] writes:
 --On Wednesday, May 12, 2004 15:39:54 -0400 Tom Lane
 [EMAIL PROTECTED]=20 wrote:
 At this point I'd settle for saying that --enable-thread-safety on
 Unixware will generate a library that requires -Kpthread.  This is
 kinda grungy but it seems that any more-pleasant solution would
 require a disproportionate amount of work.

 If I did the work for the dlsym() stuff would you and the rest of
 core@ accept it?

 How invasive a change are we talking about?  I'd be inclined to reject
 a patch that makes libpq materially less readable ...

regards, tom lane
I was thinking of pq_pthread_* calls, and that function would
set a static flag for calling either the real pthread_* function
or a statically named version in libpgport.a that is a single thread
wrapper.
I know, this sucks, but, I don't see any other way, other than linking
*ALL* libpq-using programs (including initdb and friends) with -K
pthread.
k, a change that 'sucks', vs linking against -Kpthread ... I'm for the
-Kpthread route myself, which still sounds the 'clean' solution ...
that was rejected back in Jan-Mar.

BUT, I agree it would work.

I tried to submit the patch, and it was killed.

LER


Marc G. Fournier   Hub.Org Networking Services
(http://www.hub.org) Email: [EMAIL PROTECTED]   Yahoo!: yscrappy
ICQ: 7615664


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Marc G. Fournier]

On Wed, 12 May 2004, Larry Rosenman wrote:



 --On Wednesday, May 12, 2004 16:00:48 -0400 Tom Lane [EMAIL PROTECTED]
 wrote:

  Larry Rosenman [EMAIL PROTECTED] writes:
  --On Wednesday, May 12, 2004 15:39:54 -0400 Tom Lane
  [EMAIL PROTECTED]=20 wrote:
  At this point I'd settle for saying that --enable-thread-safety on
  Unixware will generate a library that requires -Kpthread.  This is
  kinda grungy but it seems that any more-pleasant solution would
  require a disproportionate amount of work.
 
  If I did the work for the dlsym() stuff would you and the rest of core@
  accept it?
 
  How invasive a change are we talking about?  I'd be inclined to reject a
  patch that makes libpq materially less readable ...
 
  regards, tom lane
 I was thinking of pq_pthread_* calls, and that function would
 set a static flag for calling either the real pthread_* function
 or a statically named version in libpgport.a that is a single thread
 wrapper.

 I know, this sucks, but, I don't see any other way, other than linking
 *ALL* libpq-using programs (including initdb and friends) with -K pthread.

k, a change that 'sucks', vs linking against -Kpthread ... I'm for the
-Kpthread route myself, which still sounds the 'clean' solution ...


Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] threads stuff/UnixWare

[Marc G. Fournier]

On Wed, 12 May 2004, Larry Rosenman wrote:

  k, a change that 'sucks', vs linking against -Kpthread ... I'm for the
  -Kpthread route myself, which still sounds the 'clean' solution ...
 that was rejected back in Jan-Mar.

 BUT, I agree it would work.

 I tried to submit the patch, and it was killed.

Please re-submit since I don't recall the patch ...


Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] threads stuff/UnixWare

[Tom Lane]

Larry Rosenman [EMAIL PROTECTED] writes:
 Please save us all time searching by providing a URL ...

 I can't find my posts on archives.postgresql.org, but can find it in
 MY archives.

Well, then we won't be able to find 'em either, so please repost.

 This is heading down the same path I was back on 3/22.

I think at the time we were hoping for a cleaner solution.  Since none
has emerged, it would be good of you to refresh our memories of where
things stood.

regards, tom lane

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] threads stuff/UnixWare

[Bruce Momjian]

Tom Lane wrote:
 Larry Rosenman [EMAIL PROTECTED] writes:
  Please save us all time searching by providing a URL ...
 
  I can't find my posts on archives.postgresql.org, but can find it in
  MY archives.
 
 Well, then we won't be able to find 'em either, so please repost.
 
  This is heading down the same path I was back on 3/22.
 
 I think at the time we were hoping for a cleaner solution.  Since none
 has emerged, it would be good of you to refresh our memories of where
 things stood.

[ Sorry I have been away from email today. ]

Larry, now that I have put the thread testing into configure, I am ready
to deal with Unixware.  In fact I posted to the list asking you about it
but was too lazy to look up your email address.

Anyway, I think the only solution is going to be to use the -K flag on
all compiles on that platform if you want threads, so anything that uses
libpq will need it.  We don't have a client-level flags option at this
point, but we can add it, or just the flag for the backend compile too.

I think that is our best option.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 21:08:25 -0400 Bruce Momjian 
[EMAIL PROTECTED] wrote:

Tom Lane wrote:
Larry Rosenman [EMAIL PROTECTED] writes:
 Please save us all time searching by providing a URL ...
 I can't find my posts on archives.postgresql.org, but can find it in
 MY archives.
Well, then we won't be able to find 'em either, so please repost.

 This is heading down the same path I was back on 3/22.

I think at the time we were hoping for a cleaner solution.  Since none
has emerged, it would be good of you to refresh our memories of where
things stood.
[ Sorry I have been away from email today. ]

Larry, now that I have put the thread testing into configure, I am ready
to deal with Unixware.  In fact I posted to the list asking you about it
but was too lazy to look up your email address.
Anyway, I think the only solution is going to be to use the -K flag on
all compiles on that platform if you want threads, so anything that uses
libpq will need it.  We don't have a client-level flags option at this
point, but we can add it, or just the flag for the backend compile too.
I think a client-level flag would be best, that way the backend doesn't get
the wrapped functions from libthread.
Can you do the patch?


I think that is our best option.


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Bruce Momjian]

Larry Rosenman wrote:
  [ Sorry I have been away from email today. ]
 
  Larry, now that I have put the thread testing into configure, I am ready
  to deal with Unixware.  In fact I posted to the list asking you about it
  but was too lazy to look up your email address.
 
  Anyway, I think the only solution is going to be to use the -K flag on
  all compiles on that platform if you want threads, so anything that uses
  libpq will need it.  We don't have a client-level flags option at this
  point, but we can add it, or just the flag for the backend compile too.
 I think a client-level flag would be best, that way the backend doesn't get
 the wrapped functions from libthread.
 
 Can you do the patch?

OK.  Can someone think of a clean way to add a flag to everything but
the backend build?  Hold, what about /port.  Larry, if we compile
anything with threads does everything that links with that object file
require the flag, even if there are no thread flags in the object file? 
I don't think so, but I am just checking.  Actually, my bet is that it
might do that because if it didn't, there would be no problem with using
that flag on the backend because there are no threading calls in there.

I am afraid of libpgport.  Those object files get used by the backend,
and by the client stuff.  Larry can -K and non-K object files be mixed
in the same binary?

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 21:55:40 -0400 Bruce Momjian 
[EMAIL PROTECTED] wrote:

Larry Rosenman wrote:
 [ Sorry I have been away from email today. ]

 Larry, now that I have put the thread testing into configure, I am
 ready to deal with Unixware.  In fact I posted to the list asking you
 about it but was too lazy to look up your email address.

 Anyway, I think the only solution is going to be to use the -K flag on
 all compiles on that platform if you want threads, so anything that
 uses libpq will need it.  We don't have a client-level flags option at
 this point, but we can add it, or just the flag for the backend
 compile too.
I think a client-level flag would be best, that way the backend doesn't
get the wrapped functions from libthread.
Can you do the patch?
OK.  Can someone think of a clean way to add a flag to everything but
the backend build?  Hold, what about /port.  Larry, if we compile
anything with threads does everything that links with that object file
require the flag, even if there are no thread flags in the object file?
I don't think so, but I am just checking.  Actually, my bet is that it
might do that because if it didn't, there would be no problem with using
that flag on the backend because there are no threading calls in there.
Yes, there would still be the overhead, because the functions that 
libthread wraps would go through that overhead since libthread does it's 
magic at _ini time.

Y'all were concerned with overhead in previous discussions.

If you want to link the backend with -Kpthread, you can, but basically the
client code (initdb, et al) should be -Kpthread, and the backend should NOT
since there are no threads calls in the backend.
I am afraid of libpgport.  Those object files get used by the backend,
and by the client stuff.  Larry can -K and non-K object files be mixed
in the same binary?
Yes, they can.

But, you MUST link ANY pthread_* using code with -K pthread.

See above discussion.



--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] threads stuff/UnixWare

[Bruce Momjian]

Larry Rosenman wrote:
 Yes, there would still be the overhead, because the functions that 
 libthread wraps would go through that overhead since libthread does it's 
 magic at _ini time.
 
 Y'all were concerned with overhead in previous discussions.
 
 If you want to link the backend with -Kpthread, you can, but basically the
 client code (initdb, et al) should be -Kpthread, and the backend should NOT
 since there are no threads calls in the backend.
 
 
  I am afraid of libpgport.  Those object files get used by the backend,
  and by the client stuff.  Larry can -K and non-K object files be mixed
  in the same binary?
 Yes, they can.
 
 But, you MUST link ANY pthread_* using code with -K pthread.

OK, can someone come up with a cute way to add flags only to things that
use libpq, meaning /interfaces and /bin?

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] threads stuff/UnixWare

[Larry Rosenman]

--On Wednesday, May 12, 2004 22:26:03 -0400 Bruce Momjian 
[EMAIL PROTECTED] wrote:

Larry Rosenman wrote:
Yes, there would still be the overhead, because the functions that
libthread wraps would go through that overhead since libthread does it's
magic at _ini time.
Y'all were concerned with overhead in previous discussions.

If you want to link the backend with -Kpthread, you can, but basically
the client code (initdb, et al) should be -Kpthread, and the backend
should NOT since there are no threads calls in the backend.

 I am afraid of libpgport.  Those object files get used by the backend,
 and by the client stuff.  Larry can -K and non-K object files be mixed
 in the same binary?
Yes, they can.
But, you MUST link ANY pthread_* using code with -K pthread.
OK, can someone come up with a cute way to add flags only to things that
use libpq, meaning /interfaces and /bin?
What about where -lpq is specified?

(I'm NOT a GNU Make guru).

LER



--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] Threads vs Processes

[Zeugswetter Andreas SB SD]

 When the address-of operator is applied to a thread-local variable, it 
 is evaluated at run-time and returns the address of the current thread's 
 instance of that variable. An address so obtained may be used by any 
 thread. When a thread terminates, any pointers to thread-local variables 
 in that thread become invalid.

Bummer, I would have thought one advantage of using TLS must surely be memory 
protection ? So the only for pg useful usage for TLS seems to be __declspec(thread) 
and __declspec(thread) static (both for stuff that do not need runtime 
preinitialization).

Maybe the techniques of electric fence could be used for protecting the shmem
at least a little bit.

Andreas

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] Threads vs Processes

[Shridhar Daithankar]

Tom Lane wrote:

Shridhar Daithankar [EMAIL PROTECTED] writes:

One thing that can be done is to arrange all globals/statics in a
structure and make that structure thread local.


That's about as far from non-invasive as I can imagine :-(
I know.

I have following half baked solution. Obviously it won't be as good and 
automatic as __declspec(thread). But still... Just for starters

1. Put all static/global variables in .h file. No more in .c files
2. Encapsulate them in a #define.
3. Create a new header files which includes lot of macro invocations. That way 
we expand a macro definition, the structure definition changes on recompile.

Like in a.h
a.h
---
#define module1 int a;\
int b;
---
b.h
---
typedef struct
{
 module1
 module2
} globalDataStructure
---

Whoever includes a.h earlier, needs to include b.h now. That way things will 
flow smoothly. This could be optional but I would rather not include a.h anymore 
as a good practice.

4. Add a macro definition for initialization function. The global initialization 
function will again invoke macro implementations which are free to redirect it 
to any function of choice.

That way we can maintain variables in modules that they are but still put them 
in a single compile unit when they get compiled. Only one file b.h, need to be 
maintained for addition of new files. The variables would be taken care of in 
the respective modules. ( This could nicely be done in C++ using classes but 
thats a
altogether different subject.)

The changes to actual C code obviously need to follow.

I agree that's messy and non transparent. but it still retains distributed 
nature of variable declarations and maintenance thereof..

But I feel we are missing the point. Last time threads for unix builds were 
discussed, one of the good suggestion was to use them to distribute load across 
CPU's rather than using thread per connection. Personally I don't think 
postgresql is missing lot of performance by using process/connection rather than 
thread/connection(Except for slowaris and windows perhaps).

Even on windows, the time taken for a new connection is unlikely to be an issue, 
I feel.

We really don't need threads to replace existing functionality. That would be 
dog work.

If we are writing new functionality, we can do it clean and portable way rather 
than fiddling with existing code. Changing working code is always a messy 
business irrespective of technicalities involved. As a support engineer, I can 
attest that.

 Shridhar

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
   (send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] Threads vs Processes

[Tom Lane]

Shridhar Daithankar [EMAIL PROTECTED] writes:
 We really don't need threads to replace existing functionality. That
 would be dog work.

No, that's not the point at all.  The problem we are facing at the
moment with the Windows port is lack of fork(), which means there's
no way for separate-subprocess backends to inherit variable values
from the postmaster.  Bruce has been trying to fix that by having the
subprocesses somehow reload or re-deduce all those variables; which
is messy, bug-prone, and probably race-condition-prone too.  In a
threaded implementation it would maybe be relatively easy to initialize
a new thread's TLS by copying the postmaster thread's TLS, in which case
a whole pile of as-yet-unwritten Windows-only code won't be needed.

regards, tom lane

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Threads vs Processes

[Shridhar Daithankar]

On Friday 26 September 2003 20:19, Tom Lane wrote:
 Shridhar Daithankar [EMAIL PROTECTED] writes:
  We really don't need threads to replace existing functionality. That
  would be dog work.

 No, that's not the point at all.  The problem we are facing at the
 moment with the Windows port is lack of fork(), which means there's
 no way for separate-subprocess backends to inherit variable values
 from the postmaster.  Bruce has been trying to fix that by having the
 subprocesses somehow reload or re-deduce all those variables; which
 is messy, bug-prone, and probably race-condition-prone too.  In a
 threaded implementation it would maybe be relatively easy to initialize
 a new thread's TLS by copying the postmaster thread's TLS, in which case
 a whole pile of as-yet-unwritten Windows-only code won't be needed.

Umm.. I understand child process created by createProcess does not inherit 
variable values from parent process. That's where problem originates..

We can simply create a registry key that would contain shared memory id from 
where a child process should get the variable values.

And that would need initialization function I talked about earlier.  And since 
anyways TLS-TLS copy is still needed anyways, I think this approach can 
still save us dealing with threads. 

God.. it doesn't get any less messy..I hope this is of some value..

 Shridhar


---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Threads vs Processes

[Bruce Momjian]

Tom Lane wrote:
 Shridhar Daithankar [EMAIL PROTECTED] writes:
  We really don't need threads to replace existing functionality. That
  would be dog work.
 
 No, that's not the point at all.  The problem we are facing at the
 moment with the Windows port is lack of fork(), which means there's
 no way for separate-subprocess backends to inherit variable values
 from the postmaster.  Bruce has been trying to fix that by having the
 subprocesses somehow reload or re-deduce all those variables; which
 is messy, bug-prone, and probably race-condition-prone too.  In a
 threaded implementation it would maybe be relatively easy to initialize
 a new thread's TLS by copying the postmaster thread's TLS, in which case
 a whole pile of as-yet-unwritten Windows-only code won't be needed.

I haven't said much in this thread yet because I wasn't sure what to
say.

I worked 2-3 weeks on Win32 back in March/April, and a few days last
month getting it to compile again, and for the interfaces/clients to run
under Win32.

I haven't had any time to work on the fork/exec, though I now have a few
weeks to spend on it, so it isn't that I have been trying to get
fork/exec working, and failing, it is that I haven't even tried lately.

My plan is to pass a few values to the child via the command line:

#ifdef EXEC_BACKEND
Assert(UsedShmemSegID != 0  UsedShmemSegAddr != NULL);
/* database name at the end because it might contain commas */
snprintf(pbuf, NAMEDATALEN + 256, %d,%d,%d,%p,%s, port-sock, 
canAcceptConnections(),
 UsedShmemSegID, UsedShmemSegAddr, port-database_name);
av[ac++] = pbuf;
#else

and pass the GUC settings via a special binary file.  (Those are already
in main CVS.)  The other values I plan to regenerate in the child the
same way the postmaster does it at initialization time.  The easy part
of that is that I only have to worry about postmaster variables.  All my
current fork/exec work is marked by #ifdef EXEC_BACKEND in current CVS,
so it can be easily ripped out.

One solution is for me to continue with this in the Win32 CVS version
until I have fork/exec() working on Unix, then test on Win32.  I think
that could be done in a few weeks, if not less.

Another solution, already mentioned, is to use threads and TLS.  This is
what SRA's code uses.  I know SRA wants to contribute that code back to
the community, so I can ask them to see if they are ready to release it.
That would show us all the changes needed to do threading.  Their code
is based on 7.3.X, rather than PeerDirect's which is based on 7.2.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Threads vs Processes

[Tom Lane]

Bruce Momjian [EMAIL PROTECTED] writes:
 One solution is for me to continue with this in the Win32 CVS version
 until I have fork/exec() working on Unix, then test on Win32.  I think
 that could be done in a few weeks, if not less.

 Another solution, already mentioned, is to use threads and TLS.  This is
 what SRA's code uses.  I know SRA wants to contribute that code back to
 the community, so I can ask them to see if they are ready to release it.

If you are willing to expend the effort, I think it would be worth the
time to pursue both approaches.  We don't yet have enough info to decide
which one will be cleaner, so we need to push forward on both until we
can make a realistic comparison.

regards, tom lane

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Threads vs Processes

[Bruce Momjian]

Tom Lane wrote:
 Bruce Momjian [EMAIL PROTECTED] writes:
  One solution is for me to continue with this in the Win32 CVS version
  until I have fork/exec() working on Unix, then test on Win32.  I think
  that could be done in a few weeks, if not less.
 
  Another solution, already mentioned, is to use threads and TLS.  This is
  what SRA's code uses.  I know SRA wants to contribute that code back to
  the community, so I can ask them to see if they are ready to release it.
 
 If you are willing to expend the effort, I think it would be worth the
 time to pursue both approaches.  We don't yet have enough info to decide
 which one will be cleaner, so we need to push forward on both until we
 can make a realistic comparison.

I think I know enough to get the fork/exec working, particularly because
most of the work can be tested under Unix.  I don't know enough to get
the threads working, and I have had no offers of help since I started.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Threads vs Processes

[Dann Corbit]

 -Original Message-
 From: Tom Lane [mailto:[EMAIL PROTECTED] 
 Sent: Friday, September 26, 2003 9:27 AM
 To: Bruce Momjian
 Cc: Shridhar Daithankar; [EMAIL PROTECTED]; 
 [EMAIL PROTECTED]
 Subject: Re: [HACKERS] Threads vs Processes 
 
 
 Bruce Momjian [EMAIL PROTECTED] writes:
  One solution is for me to continue with this in the Win32 
 CVS version 
  until I have fork/exec() working on Unix, then test on 
 Win32.  I think 
  that could be done in a few weeks, if not less.
 
  Another solution, already mentioned, is to use threads and 
 TLS.  This 
  is what SRA's code uses.  I know SRA wants to contribute that code 
  back to the community, so I can ask them to see if they are 
 ready to 
  release it.
 
 If you are willing to expend the effort, I think it would be 
 worth the time to pursue both approaches.  We don't yet have 
 enough info to decide which one will be cleaner, so we need 
 to push forward on both until we can make a realistic comparison.

I think the ideal situation would be a server that both threads and
forks.

Sometimes, we want the server to take on a new personality (with the
rights of the attaching user).  In such a case, threading is not a
sufficient answer.  Also, a forked process is a bit safer (though you
can put a try/catch around threads).

For performance with multiple queries from a single user, threads are
going to be faster than forking.  I think that the best model will be a
server that does both.

I know that there is some aversion to using C++, but the ACE framework
offers a consistent threading model that works for just about every
computer under the sun.
http://www.cs.wustl.edu/~schmidt/ACE.html

The license is basically BSD (it is called ACE, but it works exactly
like a BSD license).  With ACE, you program to a single API, and the
code works the same on every platform with a simple recompile.  It might
even be worthwhile to use the ACE higher level components to create a
server that supports multiple models of connection and threading.

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Threads vs Processes

[Christopher Kings-Lynne]

  We really don't need threads to replace existing functionality. That
  would be dog work.

 No, that's not the point at all.  The problem we are facing at the
 moment with the Windows port is lack of fork(), which means there's
 no way for separate-subprocess backends to inherit variable values
 from the postmaster.  Bruce has been trying to fix that by having the
 subprocesses somehow reload or re-deduce all those variables; which
 is messy, bug-prone, and probably race-condition-prone too.  In a
 threaded implementation it would maybe be relatively easy to initialize
 a new thread's TLS by copying the postmaster thread's TLS, in which case
 a whole pile of as-yet-unwritten Windows-only code won't be needed.

Kepp in mind though all the cool things that could be done if we had
thread capabilities.  eg. evaluating different subexpressings on
fdifferent cpuis for the one query, etc.

Chris



---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Threads vs Processes (was: NuSphere and PostgreSQL for windows)

[Tom Lane]

Claudio Natoli [EMAIL PROTECTED] writes:
 FWIW, I've got a threaded version of the WIN32_DEV branch more or less
 running (it is a terrible hack job, so NO, no patches... yet :-), as a
 proof of concept. Still a work in progress (ok, I've qualified it enough),
 but it is showing enough promise to convince me that threading is the way to
 go for the Win32 port.

How are you dealing with the issue of wanting some static variables to
be per-thread and others not?

regards, tom lane

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Threads vs Processes (was: NuSphere and PostgreSQL for window

[Claudio Natoli]

 
 Claudio Natoli [EMAIL PROTECTED] writes:
  FWIW, I've got a threaded version of the WIN32_DEV branch more or less
  running (it is a terrible hack job, so NO, no patches... yet :-), as a
  proof of concept. Still a work in progress (ok, I've qualified it
enough),
  but it is showing enough promise to convince me that threading is the
way to
  go for the Win32 port.
 
 How are you dealing with the issue of wanting some static variables to
 be per-thread and others not?
 
   regards, tom lane

To be perfectly honest, I'm still trying to familiarize myself with the code
sufficiently well so that I can tell which variables need to be per-thread
and which are shared (and, in turn, which of these need to be protected from
concurrent access). So, in short, I'm not dealing with the issue (and,
hence, it is only running in the very loosest sense of the word).

Unfortunately, I'm not yet even in a position to propose a reasonable model,
let alone one that'd play well with the existing code base. Perhaps some
time soon, hopefully... (I really want to get involved in this.)

Cheers,
Claudio

--- 
WE HAVE MOVED - PLEASE NOTE OUR NEW CONTACT DETAILS: 
THE BASEMENT, 33 EWELL STREET, BALMAIN NSW 2041 
TEL: +61 2 9555 1544 FAX: +61 2 9555 6911 
Certain disclaimers and policies apply to all email sent from Memetrics.
For the full text of these disclaimers and policies see 
http://www.memetrics.com/emailpolicy.html

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Threads vs Processes (was: NuSphere and PostgreSQL for window s)

[Tom Lane]

Claudio Natoli [EMAIL PROTECTED] writes:
 How are you dealing with the issue of wanting some static variables to
 be per-thread and others not?

 To be perfectly honest, I'm still trying to familiarize myself with the code
 sufficiently well so that I can tell which variables need to be per-thread
 and which are shared (and, in turn, which of these need to be protected from
 concurrent access).

Well, the first-order approximation would be to duplicate the current
fork semantics: *all* static variables are per-thread, and should be
copied from the parent thread at thread creation.  If there is some
reasonably non-invasive way to do that, we'd have a long leg up on the
problem.

regards, tom lane

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Threads vs Processes

[Shridhar Daithankar]

Tom Lane wrote:

Claudio Natoli [EMAIL PROTECTED] writes:

How are you dealing with the issue of wanting some static variables to
be per-thread and others not?


To be perfectly honest, I'm still trying to familiarize myself with the code
sufficiently well so that I can tell which variables need to be per-thread
and which are shared (and, in turn, which of these need to be protected from
concurrent access).


Well, the first-order approximation would be to duplicate the current
fork semantics: *all* static variables are per-thread, and should be
copied from the parent thread at thread creation.  If there is some
reasonably non-invasive way to do that, we'd have a long leg up on the
problem.
Hmm.. I was looking for some fast tutorials on thread local storage. I found 
this one..
http://publib16.boulder.ibm.com/pseries/en_US/aixprggd/genprogc/thread_specific_data.htm

Basically, in a process we are free to declare as many globals as we can. 
Converting them to thread local is not an easy job because each variable would 
need it's own key and there is limit on how many keys can be allocated.

One thing that can be done is to arrange all globals/statics in a structure and 
make that structure thread local. We need to change all invocations of any of 
those variables to use a pointer. We just need only one global variable. And 
some macro trickery possibly so that we can extend that structure easily and 
automatically.

Upshot is duplicating environment is easy. We need to do a huge memcpy and any 
specific depp copy of strings on thread creation. Besides even in process model, 
 this kind of initialization will allow to put all variables on heap instead of 
stack. But then we need to add initialization code explicitly.

Something like int a=10; can not be added just like that.

If globals are less than 100 in numbers, I think it should be reasonably blind 
job of converting them to a structure type stuff. Don't know really though. My 
estimations are always 10% of what it takes..:-)

I hope I got it correct..

 Shridhar

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
 subscribe-nomail command to [EMAIL PROTECTED] so that your
 message can get through to the mailing list cleanly

Re: [HACKERS] Threads vs Processes (was: NuSphere and PostgreSQL

[Manfred Spraul]

Tom Lane wrote:

Claudio Natoli [EMAIL PROTECTED] writes:
 

How are you dealing with the issue of wanting some static variables to
be per-thread and others not?
 

 

To be perfectly honest, I'm still trying to familiarize myself with the code
sufficiently well so that I can tell which variables need to be per-thread
and which are shared (and, in turn, which of these need to be protected from
concurrent access).
No. Not protected from concurrent access. Each thread must have it's own 
copy.

   

Well, the first-order approximation would be to duplicate the current
fork semantics: *all* static variables are per-thread, and should be
copied from the parent thread at thread creation.  If there is some
reasonably non-invasive way to do that, we'd have a long leg up on the
problem.
There is a declspec(thread) that makes a global variable per-thread. 
AFAIK it uses linker magic to replace the actual memory accesses with 
calls to TlsAlloc() etc. Note that declspec(thread) doesn't work from 
within dynamic link libraries, but that shouldn't be a big problem.

--
   Manfred


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Threads vs Processes (was: NuSphere and PostgreSQL for windows)

[Keith Bottner]

Typically variables that you want to be per-thread are stored in what
Microsoft calls Thread Local Storage (TLS). Variables that you want shared
you can just treat as globals and statics with the appropriate threading
synchronization primitives. With Windows 2000 and later you have up to 1088
TLS locations that you can use, of course these can be pointers to memory
which can store whatever you want. 

Keith

-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Tom Lane
Sent: Thursday, September 25, 2003 9:17 AM
To: Claudio Natoli
Cc: Robert Treat; Bruce Momjian; [EMAIL PROTECTED];
[EMAIL PROTECTED]
Subject: Re: [HACKERS] Threads vs Processes (was: NuSphere and PostgreSQL
for windows) 


Claudio Natoli [EMAIL PROTECTED] writes:
 FWIW, I've got a threaded version of the WIN32_DEV branch more or less 
 running (it is a terrible hack job, so NO, no patches... yet :-), as 
 a proof of concept. Still a work in progress (ok, I've qualified it 
 enough), but it is showing enough promise to convince me that 
 threading is the way to go for the Win32 port.

How are you dealing with the issue of wanting some static variables to be
per-thread and others not?

regards, tom lane

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Threads vs Processes (was: NuSphere and PostgreSQL for

[Bruce Momjian]

Keith Bottner wrote:
 Typically variables that you want to be per-thread are stored in what
 Microsoft calls Thread Local Storage (TLS). Variables that you want shared
 you can just treat as globals and statics with the appropriate threading
 synchronization primitives. With Windows 2000 and later you have up to 1088
 TLS locations that you can use, of course these can be pointers to memory
 which can store whatever you want. 

Goes GCC on Windows support TLS, or only Microsoft compilers?

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Threads vs Processes

[Tom Lane]

Shridhar Daithankar [EMAIL PROTECTED] writes:
 One thing that can be done is to arrange all globals/statics in a
 structure and make that structure thread local.

That's about as far from non-invasive as I can imagine :-(

I really, really want to avoid doing anything like the above, because it
would force us to expose to the whole backend many data structures and
state variables that are currently local to individual .c files.  That
complicates understanding and debugging tremendously, not to mention
slowing the edit/compile/debug cycle when you are changing such
structures.

regards, tom lane

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Threads vs Processes

[Merlin Moncure]

Both Microsoft and windows compilers support thread local storage.  *If*
you guys go with the threading model and *if* it does not introduce any
serious portability issues with gcc (big ifs, and I'm not familiar with
gcc), than IMO TLS is really the way to go.  I don't think any
reorganization of postgres's static variables is necessary.  TLS is
implemented in the win32 API, not the C Libs, so by giving up the syntax
sugar you can make direct calls and keep compiler independence in win32.

Microsoft syntax is __desclspec(thread) and Borland syntax is simply
__thread.  All TLS variables *must* be static (or implicitly static
through extern, i.e. no 'auto' variables) and their addresses can not be
assumed to be constant.  

Taking addresses of TLS variables should be considered illegal, as well
as pointers to TLS variables.  Another gotcha is that DLLs that have
__thread variables will GPF if loaded with LoadLibrary (they should be
static linked).  Of course, pg does not use dlls, but it's worth noting.

Merlin

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Threads vs Processes

[Keith Bottner]

Actually you can use a DLL with LoadLibrary as long as you do the following.

When a process uses load-time linking with this DLL, the entry-point
function is sufficient to manage the thread local storage. Problems can
occur with a process that uses run-time linking because the entry-point
function is not called for threads that exist before the LoadLibrary
function is called, so TLS memory is not allocated for these threads. The
following example solves this problem by checking the value returned by the
TlsGetValue function and allocating memory if the value indicates that the
TLS slot for this thread is not set.

LPVOID lpvData; 
 
// Retrieve a data pointer for the current thread.
 
lpvData = TlsGetValue(dwTlsIndex); 
 
// If NULL, allocate memory for this thread.
 
if (lpvData == NULL) 
{ 
lpvData = (LPVOID) LocalAlloc(LPTR, 256); 
if (lpvData != NULL) 
TlsSetValue(dwTlsIndex, lpvData); 
}

Unless gcc has extension as Borland and Microsoft do you will have to
utilize the API and not the compiler/linker customizations that make access
variables declared as __declspec(thread) or __thread under Borland.

Keith

-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Merlin Moncure
Sent: Thursday, September 25, 2003 2:49 PM
To: Tom Lane
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; Bruce
Momjian; Shridhar Daithankar; Claudio Natoli
Subject: Re: [HACKERS] Threads vs Processes 


Both Microsoft and windows compilers support thread local storage.  *If* you
guys go with the threading model and *if* it does not introduce any serious
portability issues with gcc (big ifs, and I'm not familiar with gcc), than
IMO TLS is really the way to go.  I don't think any reorganization of
postgres's static variables is necessary.  TLS is implemented in the win32
API, not the C Libs, so by giving up the syntax sugar you can make direct
calls and keep compiler independence in win32.

Microsoft syntax is __desclspec(thread) and Borland syntax is simply
__thread.  All TLS variables *must* be static (or implicitly static through
extern, i.e. no 'auto' variables) and their addresses can not be assumed to
be constant.  

Taking addresses of TLS variables should be considered illegal, as well as
pointers to TLS variables.  Another gotcha is that DLLs that have __thread
variables will GPF if loaded with LoadLibrary (they should be static
linked).  Of course, pg does not use dlls, but it's worth noting.

Merlin

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html


---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Threads vs Processes

[Myron Scott]

On Thursday, September 25, 2003, at 10:03 AM, Tom Lane wrote:

Shridhar Daithankar [EMAIL PROTECTED] writes:
One thing that can be done is to arrange all globals/statics in a
structure and make that structure thread local.
That's about as far from non-invasive as I can imagine :-(

I really, really want to avoid doing anything like the above, because 
it
would force us to expose to the whole backend many data structures and
state variables that are currently local to individual .c files.  That
complicates understanding and debugging tremendously, not to mention
slowing the edit/compile/debug cycle when you are changing such
structures.


Another option would be to create thread local hashtable or other lookup
structure to which you would register a structure for a particular .c 
file
or group of files.

You could then define the structures you need locally without affecting
 other parts of the codebase.


Myron Scott

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Threads vs Processes

[Tom Lane]

Merlin Moncure [EMAIL PROTECTED] writes:
 All TLS variables *must* be static (or implicitly static
 through extern, i.e. no 'auto' variables)

I assume you mean static as in not-auto, rather than static as in
not-global.  Otherwise we have a problem here.

 and their addresses can not be
 assumed to be constant.

Surely the addresses can be assumed constant within a thread.  Otherwise
we have a problem here too.

 Taking addresses of TLS variables should be considered illegal,

Sorry, no can accept that restriction.

regards, tom lane

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Threads vs Processes

[Kurt at DBC]

Tom Lane wrote:
I assume you mean static as in not-auto, rather than static as in
not-global.  Otherwise we have a problem here.
[...]
Surely the addresses can be assumed constant within a thread.  Otherwise
we have a problem here too.
[...]
Taking addresses of TLS variables should be considered illegal,
Sorry, no can accept that restriction.

I think you are okay on all 3 fronts,
from http://gcc.gnu.org/onlinedocs/gcc/Thread-Local.html#Thread-Local :
The __thread specifier may be used alone, with the extern or static 
specifiers, but with no other storage class specifier. When used with 
extern or static, __thread must appear immediately after the other 
storage class specifier.

and

When the address-of operator is applied to a thread-local variable, it 
is evaluated at run-time and returns the address of the current thread's 
instance of that variable. An address so obtained may be used by any 
thread. When a thread terminates, any pointers to thread-local variables 
in that thread become invalid.

Also see ISO/IEC 9899:1999 Edits for Thread-Local Storage :
http://gcc.gnu.org/onlinedocs/gcc/C99-Thread-Local-Edits.html#C99%20Thread-Local%20Edits
and ELF Handling For Thread-Local Storage,
http://people.redhat.com/drepper/tls.pdf may be of interest.
Cheers,
Kurt.


---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Threads vs Processes

[Claudio Natoli]

 
 Another option would be to create thread local hashtable or other lookup
 structure to which you would register a structure for a particular .c 
 file or group of files.
 
 You could then define the structures you need locally without 
 affecting other parts of the codebase.
 
 
 
 Myron Scott

A slight variant on this idea would be to mask the variables themselves,
using macrology on a hash keyed on the threadID.

To illustrate:
MyProc-errType

would, on a threaded system, become something like:
((PGProc*)ThreadLocalHash(MyProc,GetCurrentThreadID()))-errType

I'm imagining that an approach like this would fit in very nicely with the
existing code. Not the most performance friendly solution however (and I'm
guessing that there might be a place or two where this might be important
:-).

Just a thought,
Claudio

--- 
WE HAVE MOVED - PLEASE NOTE OUR NEW CONTACT DETAILS: 
THE BASEMENT, 33 EWELL STREET, BALMAIN NSW 2041 
TEL: +61 2 9555 1544 FAX: +61 2 9555 6911 
Certain disclaimers and policies apply to all email sent from Memetrics.
For the full text of these disclaimers and policies see 
http://www.memetrics.com/emailpolicy.html

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] threads/UnixWare

[Bruce Momjian]

Actually, your getpwuid_r is the old, pre-POSIX format.  The attached
email has the configure tests.  I was hoping we wouldn't need them, but
it seems we may.

---

Larry Rosenman wrote:
 In src/port, we have in threads.c:
 
 /*
  * Wrapper around getpwuid() or getpwuid_r() to mimic POSIX getpwuid_r()
  * behaviour, if it is not available.
  */
 int
 pqGetpwuid(uid_t uid, struct passwd * resultbuf, char *buffer,
size_t buflen, struct passwd ** result)
 {
 #if defined(USE_THREADS)  defined(HAVE_GETPWUID_R)
 
 /*
  * broken (well early POSIX draft) getpwuid_r() which returns 
 'struct
  * passwd *'
  */
 *result = getpwuid_r(uid, resultbuf, buffer, buflen);
 #else
 /* no getpwuid_r() available, just use getpwuid() */
 *result = getpwuid(uid);
 #endif
 return (*result == NULL) ? -1 : 0;
 }
 
 
 Which BREAKS if you have the correct getpwuid_r() like UnixWare does.
 
 Can someone help me with the configure checks/macros I need?
 $ grep getpwuid_r /usr/include/pwd.h
 int getpwuid_r(uid_t, struct passwd *, char *, size_t, struct passwd **);
 $
 
 
 
 
 -- 
 Larry Rosenman http://www.lerctr.org/~ler
 Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
 US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
 
 
 ---(end of broadcast)---
 TIP 8: explain analyze is your friend
 

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073
From [EMAIL PROTECTED] Tue Aug  5 06:13:10 2003
Return-path: [EMAIL PROTECTED]
Received: from internet.csl.co.uk (internet.csl.co.uk [194.130.52.3])
by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id h75AD8r29374
for [EMAIL PROTECTED]; Tue, 5 Aug 2003 06:13:09 -0400 (EDT)
Received: from euphrates.csl.co.uk (host-194-67.csl.co.uk [194.130.52.67])
by internet.csl.co.uk (8.12.8/8.12.8) with ESMTP id h75AD284032695;
Tue, 5 Aug 2003 11:13:02 +0100
Received: from kelvin.csl.co.uk by euphrates.csl.co.uk (8.9.3/ConceptI 2.4)
id LAA21628; Tue, 5 Aug 2003 11:13:00 +0100 (BST)
Received: from kelvin.csl.co.uk (localhost.localdomain [127.0.0.1])
by kelvin.csl.co.uk (8.12.8/8.12.8) with ESMTP id h75ACxU1028659;
Tue, 5 Aug 2003 11:12:59 +0100
Received: (from [EMAIL PROTECTED])
by kelvin.csl.co.uk (8.12.8/8.12.8/Submit) id h75ACsBW028655;
Tue, 5 Aug 2003 11:12:54 +0100
From: Lee Kindness [EMAIL PROTECTED]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: [EMAIL PROTECTED]
Date: Tue, 5 Aug 2003 11:12:54 +0100
To: Bruce Momjian [EMAIL PROTECTED]
Subject: --enable-thread-safety broken + patch regressions
X-Mailer: VM 7.07 under 21.4 (patch 12) Portable Code XEmacs Lucid
cc: Lee Kindness [EMAIL PROTECTED], [EMAIL PROTECTED]
Status: OR

Bruce, the changes you made yesterday to configure for
--enable-thread-safety have broken the build, at least for Linux on
Redhat 9.

Also, I took the opportunity to look at port/threads.c. It is missing
important functionality compaired to the patch I originally
submitted. For getpwuid_r, gethostbyname_r and strerror_r there are
three possible scenarios:

1. The OS doesn't have it (but the non _r function can still be thread
safe (i.e. HPUX 11)).

2. The OS has it, but the implmentation doesn't match the POSIX spec.

3. The OS has it, and the implmentation matches the POSIX spec.

Case 3 is not being considered. In my original patch this was handled
by the pqGetpwuid etc functions simply being defined to getpwuid_r
(except for pqStrerror).

I remember discussing with you that the implementation of pqStrerror
didn't really need the distinction between the two _r
versions. However I think the others do, and the native/correct _r
calls should be #defined in if they match the POSIX spec.

It's also worth considering that when the _r function is available AND
the normal function is also thread-safe then the _r version should
still be used since it has a clean API which removes unneeded locking
within the old function.

I've still got the latest (and earlier with some configure work)
patches I submitted up at:

 http://services.csl.co.uk/postgresql/

Thanks, Lee.


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] threads/UnixWare

[Larry Rosenman]

--On Friday, August 08, 2003 02:10:25 -0400 Bruce Momjian 
[EMAIL PROTECTED] wrote:

Actually, your getpwuid_r is the old, pre-POSIX format.  The attached
email has the configure tests.  I was hoping we wouldn't need them, but
it seems we may.
Err, SCO claims SUSv2, the Single Unix Specification Version 2.  V3 
**JUST** came
out.

I'll look at Lee's stuff.





--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
 subscribe-nomail command to [EMAIL PROTECTED] so that your
 message can get through to the mailing list cleanly

Re: [HACKERS] threads/UnixWare

[Lee Kindness]

Larry, haven't managed to look at that patch... But stuffed for time
just now - just about to head off for the weekend. I'm hoping to spend
a bit of time on this on Tuesday! So, i'll see how things have
progressed then.

L.

Larry Rosenman writes:
  --On Friday, August 08, 2003 11:53:34 +0100 Lee Kindness 
   I've not been keeping up with the thread re who has what version of
   getpwuid_r... But just to clarify things the right version is:
  
int getpwuid_r(uid_t uid, struct passwd *pwd, char *buffer,
   size_t bufsize, struct passwd **result);
  
   documented at:
  
http://www.opengroup.org/onlinepubs/007904975/functions/getpwuid.html
  
   My email to Bruce which he tacked onto his reply is a good summary of
   the current problems with the thread stuff.
  Ok.  Lee: Did you see the patch I posted?   Can you help on the configure 
  test(s)
  needed to set HAVE_POSIX_GETPWUID_R?
  
  I **THINK** I did the patch right to handle it if it's set.  I forced it 
  for UnixWare
  and it seems to work.
   Larry Rosenman writes:
 --On Friday, August 08, 2003 02:10:25 -0400 Bruce Momjian
 [EMAIL PROTECTED] wrote:
  Actually, your getpwuid_r is the old, pre-POSIX format.  The attached
  email has the configure tests.  I was hoping we wouldn't need them,
   butit seems we may.
 Err, SCO claims SUSv2, the Single Unix Specification Version 2.  V3
 **JUST** came
 out.

 I'll look at Lee's stuff.

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] threads/UnixWare

[Bruce Momjian]

Of course, I was wrong.  In fact, the patch below actually said it was
the pre-POSIX version of getpwuid_r().

---

Bruce Momjian wrote:
 
 Actually, your getpwuid_r is the old, pre-POSIX format.  The attached
 email has the configure tests.  I was hoping we wouldn't need them, but
 it seems we may.
 
 ---
 
 Larry Rosenman wrote:
  In src/port, we have in threads.c:
  
  /*
   * Wrapper around getpwuid() or getpwuid_r() to mimic POSIX getpwuid_r()
   * behaviour, if it is not available.
   */
  int
  pqGetpwuid(uid_t uid, struct passwd * resultbuf, char *buffer,
 size_t buflen, struct passwd ** result)
  {
  #if defined(USE_THREADS)  defined(HAVE_GETPWUID_R)
  
  /*
   * broken (well early POSIX draft) getpwuid_r() which returns 
  'struct
   * passwd *'
   */
  *result = getpwuid_r(uid, resultbuf, buffer, buflen);
  #else
  /* no getpwuid_r() available, just use getpwuid() */
  *result = getpwuid(uid);
  #endif
  return (*result == NULL) ? -1 : 0;
  }
  
  
  Which BREAKS if you have the correct getpwuid_r() like UnixWare does.
  
  Can someone help me with the configure checks/macros I need?
  $ grep getpwuid_r /usr/include/pwd.h
  int getpwuid_r(uid_t, struct passwd *, char *, size_t, struct passwd **);
  $
  
  
  
  
  -- 
  Larry Rosenman http://www.lerctr.org/~ler
  Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
  US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
  
  
  ---(end of broadcast)---
  TIP 8: explain analyze is your friend
  
 
 -- 
   Bruce Momjian|  http://candle.pha.pa.us
   [EMAIL PROTECTED]   |  (610) 359-1001
   +  If your life is a hard drive, |  13 Roberts Road
   +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

 From [EMAIL PROTECTED] Tue Aug  5 06:13:10 2003
 Return-path: [EMAIL PROTECTED]
 Received: from internet.csl.co.uk (internet.csl.co.uk [194.130.52.3])
   by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id h75AD8r29374
   for [EMAIL PROTECTED]; Tue, 5 Aug 2003 06:13:09 -0400 (EDT)
 Received: from euphrates.csl.co.uk (host-194-67.csl.co.uk [194.130.52.67])
   by internet.csl.co.uk (8.12.8/8.12.8) with ESMTP id h75AD284032695;
   Tue, 5 Aug 2003 11:13:02 +0100
 Received: from kelvin.csl.co.uk by euphrates.csl.co.uk (8.9.3/ConceptI 2.4)
   id LAA21628; Tue, 5 Aug 2003 11:13:00 +0100 (BST)
 Received: from kelvin.csl.co.uk (localhost.localdomain [127.0.0.1])
   by kelvin.csl.co.uk (8.12.8/8.12.8) with ESMTP id h75ACxU1028659;
   Tue, 5 Aug 2003 11:12:59 +0100
 Received: (from [EMAIL PROTECTED])
   by kelvin.csl.co.uk (8.12.8/8.12.8/Submit) id h75ACsBW028655;
   Tue, 5 Aug 2003 11:12:54 +0100
 From: Lee Kindness [EMAIL PROTECTED]
 MIME-Version: 1.0
 Content-Type: text/plain; charset=us-ascii
 Content-Transfer-Encoding: 7bit
 Message-ID: [EMAIL PROTECTED]
 Date: Tue, 5 Aug 2003 11:12:54 +0100
 To: Bruce Momjian [EMAIL PROTECTED]
 Subject: --enable-thread-safety broken + patch regressions
 X-Mailer: VM 7.07 under 21.4 (patch 12) Portable Code XEmacs Lucid
 cc: Lee Kindness [EMAIL PROTECTED], [EMAIL PROTECTED]
 Status: OR
 
 Bruce, the changes you made yesterday to configure for
 --enable-thread-safety have broken the build, at least for Linux on
 Redhat 9.
 
 Also, I took the opportunity to look at port/threads.c. It is missing
 important functionality compaired to the patch I originally
 submitted. For getpwuid_r, gethostbyname_r and strerror_r there are
 three possible scenarios:
 
 1. The OS doesn't have it (but the non _r function can still be thread
 safe (i.e. HPUX 11)).
 
 2. The OS has it, but the implmentation doesn't match the POSIX spec.
 
 3. The OS has it, and the implmentation matches the POSIX spec.
 
 Case 3 is not being considered. In my original patch this was handled
 by the pqGetpwuid etc functions simply being defined to getpwuid_r
 (except for pqStrerror).
 
 I remember discussing with you that the implementation of pqStrerror
 didn't really need the distinction between the two _r
 versions. However I think the others do, and the native/correct _r
 calls should be #defined in if they match the POSIX spec.
 
 It's also worth considering that when the _r function is available AND
 the normal function is also thread-safe then the _r version should
 still be used since it has a clean API which removes unneeded locking
 within the old function.
 
 I've still got the latest (and earlier with some configure work)
 patches I submitted up at:
 
  http://services.csl.co.uk/postgresql/
 
 Thanks, Lee.
 

 
 ---(end of broadcast)---
 TIP 7: don't forget to increase your free space map settings

-- 
  

Re: [HACKERS] threads/UnixWare

[Lee Kindness]

I've not been keeping up with the thread re who has what version of
getpwuid_r... But just to clarify things the right version is:

 int getpwuid_r(uid_t uid, struct passwd *pwd, char *buffer,
size_t bufsize, struct passwd **result); 

documented at:

 http://www.opengroup.org/onlinepubs/007904975/functions/getpwuid.html

My email to Bruce which he tacked onto his reply is a good summary of
the current problems with the thread stuff.

L.

Larry Rosenman writes:
  
  
  --On Friday, August 08, 2003 02:10:25 -0400 Bruce Momjian 
  [EMAIL PROTECTED] wrote:
  
  
   Actually, your getpwuid_r is the old, pre-POSIX format.  The attached
   email has the configure tests.  I was hoping we wouldn't need them, but
   it seems we may.
  Err, SCO claims SUSv2, the Single Unix Specification Version 2.  V3 
  **JUST** came
  out.
  
  I'll look at Lee's stuff.
  
  
  
  
  
  -- 
  Larry Rosenman http://www.lerctr.org/~ler
  Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
  US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
  

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] threads/UnixWare

[Peter Eisentraut]

Bruce Momjian writes:

 Actually, your getpwuid_r is the old, pre-POSIX format.

No, his is the right POSIX/SUS variant.

-- 
Peter Eisentraut   [EMAIL PROTECTED]

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] threads/UnixWare

[Larry Rosenman]

--On Friday, August 08, 2003 11:53:34 +0100 Lee Kindness 
[EMAIL PROTECTED] wrote:

I've not been keeping up with the thread re who has what version of
getpwuid_r... But just to clarify things the right version is:
 int getpwuid_r(uid_t uid, struct passwd *pwd, char *buffer,
size_t bufsize, struct passwd **result);
documented at:

 http://www.opengroup.org/onlinepubs/007904975/functions/getpwuid.html

My email to Bruce which he tacked onto his reply is a good summary of
the current problems with the thread stuff.
Ok.  Lee: Did you see the patch I posted?   Can you help on the configure 
test(s)
needed to set HAVE_POSIX_GETPWUID_R?

I **THINK** I did the patch right to handle it if it's set.  I forced it 
for UnixWare
and it seems to work.

LER

L.

Larry Rosenman writes:
 
 
  --On Friday, August 08, 2003 02:10:25 -0400 Bruce Momjian
  [EMAIL PROTECTED] wrote:
 
  
   Actually, your getpwuid_r is the old, pre-POSIX format.  The attached
   email has the configure tests.  I was hoping we wouldn't need them,
butit seems we may.
  Err, SCO claims SUSv2, the Single Unix Specification Version 2.  V3
  **JUST** came
  out.
 
  I'll look at Lee's stuff.
 
 
 
 
 
  --
  Larry Rosenman http://www.lerctr.org/~ler
  Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
  US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
 


--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Threads

[Curt Sampson]

On Sat, 4 Jan 2003, Christopher Kings-Lynne wrote:

 Also remember that in even well developed OS's like FreeBSD, all a
 process's threads will execute only on one CPU.

I would say that that's not terribly well developed. Solaris will split
a single processes' threads over multiple CPUs, and I expect most other
major vendors Unixes will as well. In the world of free software, the
next release of NetBSD will do the same. (The scheduler activations
system, which support m userland to n kernel threads mapping, was
recently merged from its branch into NetBSD-current.)

From my experience, threaded sorts would be a big win. I managed to
shave index generation time for a large table from about 12 hours to
about 8 hours by generating two indices in parallel after I'd added a
primary key to the table. It would have been much more of a win to be
able to generate the primary key followed by other indexes with parallel
sorts rather than having to generate the primary key on one CPU (while
the other remains idle), wait while that completes, generate two more
indices, and then generate the last one .

cjs
-- 
Curt Sampson  [EMAIL PROTECTED]   +81 90 7737 2974   http://www.netbsd.org
Don't you know, in this new Dark Age, we're all light.  --XTC

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

Re: [HACKERS] Threads

[Steve Wampler]

On Sat, 4 Jan 2003, Christopher Kings-Lynne wrote:
 
 Also remember that in even well developed OS's like FreeBSD, all a
 process's threads will execute only on one CPU.

I doubt that - it certainly isn't the case on Linux and Solaris.
A thread may *start* execution on the same CPU as it's parent, but
native threads are not likely to be constrained to a specific CPU
with an SMP OS.

-- 
Steve Wampler [EMAIL PROTECTED]
National Solar Observatory

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

Re: [HACKERS] Threads

[Shridhar Daithankar]

On Thursday 23 January 2003 08:42 pm, you wrote:
 On Sat, 4 Jan 2003, Christopher Kings-Lynne wrote:
  Also remember that in even well developed OS's like FreeBSD, all a
  process's threads will execute only on one CPU.

 I doubt that - it certainly isn't the case on Linux and Solaris.
 A thread may *start* execution on the same CPU as it's parent, but
 native threads are not likely to be constrained to a specific CPU
 with an SMP OS.

I am told that linuxthreads port available on freeBSD uses rfork and is 
capable of using multiple CPUs within a single process.

Native freeBSD threads can not do that. Need to check that with freeBSD5.0.

 Shridhar



---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

Re: [HACKERS] Threads

[Greg Copeland]

On Thu, 2003-01-23 at 09:12, Steve Wampler wrote:
 On Sat, 4 Jan 2003, Christopher Kings-Lynne wrote:
  
  Also remember that in even well developed OS's like FreeBSD, all a
  process's threads will execute only on one CPU.
 
 I doubt that - it certainly isn't the case on Linux and Solaris.
 A thread may *start* execution on the same CPU as it's parent, but
 native threads are not likely to be constrained to a specific CPU
 with an SMP OS.

You are correct.  When spawning additional threads, should an idle CPU
be available, it's very doubtful that the new thread will show any bias
toward the original thread's CPU.  Most modern OS's do run each thread
within a process spread across n-CPUs.  Those that don't are probably
attempting to modernize as we speak.

-- 
Greg Copeland [EMAIL PROTECTED]
Copeland Computer Consulting


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Threads

[mlw]

Greg Copeland wrote:

  On Thu, 2003-01-23 at 09:12, Steve Wampler wrote:
  
  
On Sat, 4 Jan 2003, Christopher Kings-Lynne wrote:


  Also remember that in even well developed OS's like FreeBSD, all a
process's threads will execute only on one CPU.
  

I doubt that - it certainly isn't the case on Linux and Solaris.
A thread may *start* execution on the same CPU as it's parent, but
native threads are not likely to be constrained to a specific CPU
with an SMP OS.

  
  
You are correct.  When spawning additional threads, should an idle CPU
be available, it's very doubtful that the new thread will show any bias
toward the original thread's CPU.  Most modern OS's do run each thread
within a process spread across n-CPUs.  Those that don't are probably
attempting to modernize as we speak

AFAIK, FreeBSD is one of the OSes that are trying to modernize. Last I looked
it did not have kernel threads.

  
  

Re: [HACKERS] Threads

[Shridhar Daithankar]

On 6 Jan 2003 at 6:48, Greg Copeland wrote:
  1) Get I/O time used fuitfully
 AIO may address this without the need for integrated threading. 
 Arguably, from the long thread that last appeared on the topic of AIO,
 some hold that AIO doesn't even offer anything beyond the current
 implementation.  As such, it's highly doubtful that integrated threading
 is going to offer anything beyond what a sound AIO implementation can
 achieve.

Either way, a complete aio or threading implementation is not available on 
major platforms that postgresql runs. Linux definitely does not have one, last 
I checked.

If postgresql is not using aio or threading, we should start using one of them, 
is what I feel. What do you say?

  2) Use multiple CPU better.
 Multiple processes tend to universally support multiple CPUs better than
 does threading.  On some platforms, the level of threading support is
 currently only user mode implementations which means no additional CPU
 use.  Furthermore, some platforms where user-mode threads are defacto,
 they don't even allow for scheduling bias resulting is less work being
 accomplished within the same time interval (work slice must be divided
 between n-threads within the process, all of which run on a single CPU).

The frame-work I have posted, threading is optional at build and should be a 
configuration option if it gets integrated. So for the platforms that can not 
spread threads across multiple CPUs, it can simply be turned off..

 Speaking for my self, I probably would of been more excited if the
 offered framework had addressed several issues.  The short list is:
 
 o Code needs to be more robust.  It shouldn't be calling exit directly
 as, I believe, it should be allowing for PostgreSQL to clean up some. 
 Correct me as needed.  I would of also expected the code of adopted
 PostgreSQL's semantics and mechanisms as needed (error reporting, etc). 
 I do understand it was an initial attempt to simply get something in
 front of some eyes and have something to talk about.  Just the same, I
 was expecting something that we could actually pull the trigger with.

That could be done.

 
 o Code isn't very portable.  Looked fairly okay for pthread platforms,
 however, there is new emphasis on the Win32 platform.  I think it would
 be a mistake to introduce something as significant as threading without
 addressing Win32 from the get-go.

If you search for pthread in thread.c, there are not many instances. Same 
goes for thread.h. From what I understand windows threading, it would be less 
than 10 minutes job to #ifdef the pthread related part on either file.

It is just that I have not played with windows threading and nor I am inclined 
to...;-)

 
 o I would desire a more highly abstracted/portable interface which
 allows for different threading and synchronization primitives to be
 used.  Current implementation is tightly coupled to pthreads. 
 Furthermore, on platforms such as Solaris, I would hope it would easily
 allow for plugging in its native threading primitives which are touted
 to be much more efficient than pthreads on said platform.

Same as above. If there can be two cases separated with #ifdef, there can be 
more.. But what is important is to have a thread that can be woken up as and 
when required with any function desired. That is the basic idea.

 o Code is not commented.  I would hope that adding new code for
 something as important as threading would be commented.

Agreed. 
 
 o Code is fairly trivial and does not address other primitives
 (semaphores, mutexs, conditions, TSS, etc) portably which would be
 required for anything but the most trivial of threaded work.  This is
 especially true in such an application where data IS the application. 
 As such, you must reasonably assume that threads need some form of
 portable serialization primitives, not to mention mechanisms for
 non-trivial communication.

I don't get this. Probably I should post a working example. It is not threads 
responsibility to make a function thread safe which is changed on the fly. The 
function has to make sure that it is thread safe. That is altogether different 
effort..
 
 o Does not address issues such as thread signaling or status reporting.

From what I learnt from pthreads on linux, I would not mix threads and signals. 
One can easily add code in runner function that disables any signals for thread 
while the thread starts running. This would leave original signal handling 
mechanism in place.

As far as status reporting is concerned, the thread sould be initiated while 
back-end starts and terminated with backend termination. What is about status 
reporting?
 
 o Pool interface is rather simplistic.  Does not currently support
 concepts such as wake pool, stop pool, pool status, assigning a pool to
 work, etc.  In fact, it's not altogether obvious what the capabilities
 intent is of the current pool implementation.

Could you please elaborate? I am using same interface in c++ for 

Re: [HACKERS] Threads

[Greg Copeland]

On Tue, 2003-01-07 at 02:00, Shridhar Daithankar wrote:
 On 6 Jan 2003 at 6:48, Greg Copeland wrote:
   1) Get I/O time used fuitfully
  AIO may address this without the need for integrated threading. 
  Arguably, from the long thread that last appeared on the topic of AIO,
  some hold that AIO doesn't even offer anything beyond the current
  implementation.  As such, it's highly doubtful that integrated threading
  is going to offer anything beyond what a sound AIO implementation can
  achieve.
 
 Either way, a complete aio or threading implementation is not available on 
 major platforms that postgresql runs. Linux definitely does not have one, last 
 I checked.
 

There are two or three significant AIO implementation efforts currently
underway for Linux.  One such implementation is available from the Red
Hat Server Edition (IIRC) and has been available for some time now.  I
believe Oracle is using it.  SGI also has an effort and I forget where
the other one comes from.  Nonetheless, I believe it's going to be a
hard fought battle to get AIO implemented simply because I don't think
anyone, yet, can truly argue a case on the gain vs effort.

 If postgresql is not using aio or threading, we should start using one of them, 
 is what I feel. What do you say?
 

I did originally say that I'd like to see an AIO implementation.  Then
again, I don't current have a position to stand other than simply saying
it *might* perform better.  ;)  Not exactly a position that's going to
win the masses over.  

  was expecting something that we could actually pull the trigger with.
 
 That could be done.
 

I'm sure it can, but that's probably the easiest item to address.

  
  o Code isn't very portable.  Looked fairly okay for pthread platforms,
  however, there is new emphasis on the Win32 platform.  I think it would
  be a mistake to introduce something as significant as threading without
  addressing Win32 from the get-go.
 
 If you search for pthread in thread.c, there are not many instances. Same 
 goes for thread.h. From what I understand windows threading, it would be less 
 than 10 minutes job to #ifdef the pthread related part on either file.
 
 It is just that I have not played with windows threading and nor I am inclined 
 to...;-)
 

Well, the method above is going to create a semi-ugly mess.  I've
written thread abstraction layers which cover OS/2, NT, and pthreads. 
Each have subtle distinction.  What really needs to be done is the
creation of another abstraction layer which your current code would sit
on top of.  That way, everything contained within is clear and easy to
read.  The big bonus is that as additional threading implementations
need to be added, only the low-level abstraction stuff needs to
modified.  Done properly, each thread implementation would be it's own
module requiring little #if clutter.

As you can see, that's a fair amount of work and far from where the code
currently is.

  
  o I would desire a more highly abstracted/portable interface which
  allows for different threading and synchronization primitives to be
  used.  Current implementation is tightly coupled to pthreads. 
  Furthermore, on platforms such as Solaris, I would hope it would easily
  allow for plugging in its native threading primitives which are touted
  to be much more efficient than pthreads on said platform.
 
 Same as above. If there can be two cases separated with #ifdef, there can be 
 more.. But what is important is to have a thread that can be woken up as and 
 when required with any function desired. That is the basic idea.
 

Again, there's a lot of work in creating a well formed abstraction layer
for all of the mechanics that are required.  Furthermore, different
thread implementations have slightly different semantics which further
complicates things.  Worse, some types of primitives are simply not
available with some thread implementations.  That means those platforms
require it to be written from the primitives that are available on the
platform.  Yet more work.


  o Code is fairly trivial and does not address other primitives
  (semaphores, mutexs, conditions, TSS, etc) portably which would be
  required for anything but the most trivial of threaded work.  This is
  especially true in such an application where data IS the application. 
  As such, you must reasonably assume that threads need some form of
  portable serialization primitives, not to mention mechanisms for
  non-trivial communication.
 
 I don't get this. Probably I should post a working example. It is not threads 
 responsibility to make a function thread safe which is changed on the fly. The 
 function has to make sure that it is thread safe. That is altogether different 
 effort..


You're right, it's not the thread's responsibility, however, it is the
threading toolkit's.  In this case, you're offering to be the toolkit
which functions across two platforms, just for starters.  Reasonably,
you should expect a third to quickly follow.

 

Re: [HACKERS] Threads

[Greg Stark]

Greg Copeland [EMAIL PROTECTED] writes:

 That's the power of using the process model that is currently in use. Should
 it do something naughty, we bitch and complain politely, throw our hands in
 the air and exit. We no longer have to worry about the state and validity of
 that backend.

You missed the point of his post. If one process in your database does
something nasty you damn well should worry about the state of and validity of
the entire database, not just that one backend.

Are you really sure you caught the problem before it screwed up the data in
shared memory? On disk?


This whole topic is in need of some serious FUD-dispelling and careful
analysis. Here's a more calm explanation of the situation on this particular
point. Perhaps I'll follow up with something on IO concurrency later.

The point in consideration here is really memory isolation. Threads by default
have zero isolation between threads. They can all access each other's memory
even including their stack. Most of that memory is in fact only needed by a
single thread. 

Processes by default have complete memory isolation. However postgres actually
weakens that by doing a lot of work in a shared memory pool. That memory gets
exactly the same protection as it would get in a threaded model, which is to
say none.

So the reality is that if you have a bug most likely you've only corrupted the
local data which can be easily cleaned up either way. In the thread model
there's also the unlikely but scary risk that you've damaged other threads'
memory. And in either case there's the possibility that you've damaged the
shared pool which is unrecoverable.

In theory minimising the one case of corrupting other threads' local data
shouldn't make a big difference to the risk in the case of an assertion
failure. I'm not sure in practice if that's true though. Processes probably
reduce the temptation to do work in the shared area too.

--
greg


---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] Threads

[Greg Copeland]

On Tue, 2003-01-07 at 12:21, Greg Stark wrote:
 Greg Copeland [EMAIL PROTECTED] writes:
 
  That's the power of using the process model that is currently in use. Should
  it do something naughty, we bitch and complain politely, throw our hands in
  the air and exit. We no longer have to worry about the state and validity of
  that backend.
 
 You missed the point of his post. If one process in your database does
 something nasty you damn well should worry about the state of and validity of
 the entire database, not just that one backend.
 

I can assure you I did not miss the point.  No idea why you're
continuing to spell it out.  In this case, it appears the quotation is
being taken out of context or it was originally stated in an improper
context.

 Are you really sure you caught the problem before it screwed up the data in
 shared memory? On disk?
 
 
 This whole topic is in need of some serious FUD-dispelling and careful
 analysis. Here's a more calm explanation of the situation on this particular
 point. Perhaps I'll follow up with something on IO concurrency later.
 


Hmmm.  Not sure what needs to be dispelled since I've not seen any FUD.


 The point in consideration here is really memory isolation. Threads by default
 have zero isolation between threads. They can all access each other's memory
 even including their stack. Most of that memory is in fact only needed by a
 single thread. 
 

Again, this has been covered already.


 Processes by default have complete memory isolation. However postgres actually
 weakens that by doing a lot of work in a shared memory pool. That memory gets
 exactly the same protection as it would get in a threaded model, which is to
 say none.
 

Again, this has all been covered, more or less.  You're comments seem to
imply that you did not fully read what has been said on the topic thus
far or that you misunderstood something that was said.  Of course, it's
also possible that I may of said something out of it's proper context
which may be confusing you.

I think it's safe to say I don't have any further comment unless
something new is being brought to the table.  Should there be something
new to cover, I'm happy to talk about it.  At this point, however, it
appears that it's been beat to death already.


-- 
Greg Copeland [EMAIL PROTECTED]
Copeland Computer Consulting


---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly

Re: [HACKERS] Threads

[Tom Lane]

Greg Stark [EMAIL PROTECTED] writes:
 You missed the point of his post. If one process in your database does
 something nasty you damn well should worry about the state of and validity of
 the entire database, not just that one backend.

Right.  And in fact we do blow away all the processes when any one of
them crashes or panics.  Nonetheless, memory isolation between processes
is a Good Thing, because it reduces the chances that a process gone
wrong will cause damage via other processes before they can be shut
down.

Here is a simple example of a scenario where that isolation buys us
something: suppose that we have a bug that tromps on memory starting at
some point X until it falls off the sbrk boundary and dumps core.
(There are plenty of ways to make that happen, such as miscalculating
the length of a memcpy or memset operation as -1.)  Such a bug causes
no serious damage in isolation, because the process suffering the
failure will be in a tight data-copying or data-zeroing loop until it
gets the SIGSEGV exception.  It won't do anything bad based on all the
data structures it has clobbered during its march to the end of memory.

However, put that same bug in a multithreading context, and it becomes
entirely possible that some other thread will be dispatched and will
try to make use of already-clobbered data structures before the ultimate
SIGSEGV exception happens.  Now you have the potential for unlimited
trouble.

In general, isolation buys you some safety anytime there is a delay
between the occurrence of a failure and its detection.

 Processes by default have complete memory isolation. However postgres
 actually weakens that by doing a lot of work in a shared memory
 pool. That memory gets exactly the same protection as it would get in
 a threaded model, which is to say none.

Yes.  We try to minimize the risk by keeping the shared memory pool
relatively small and not doing more than we have to in it.  (For
example, this was one of the arguments against creating a shared plan
cache.)  It's also very helpful that in most platforms, shared memory
is not address-wise contiguous to normal memory; thus for example a
process caught in a memset death march will hit a SIGSEGV before it
gets to the shared memory block.

It's interesting to note that this can be made into an argument for
not making shared_buffers very large: the larger the fraction of your
address space that the shared buffers occupy, the larger the chance
that a wild store will overwrite something you'd wish it didn't.
I can't recall anyone having made that point during our many discussions
of appropriate shared_buffer sizing.

 So the reality is that if you have a bug most likely you've only corrupted the
 local data which can be easily cleaned up either way. In the thread model
 there's also the unlikely but scary risk that you've damaged other threads'
 memory. And in either case there's the possibility that you've damaged the
 shared pool which is unrecoverable.

In a thread model, *most* of the accessible memory space would be shared
with other threads, at least potentially.  So I think you're wrong to
categorize the second case as unlikely.

regards, tom lane

---(end of broadcast)---
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Re: [HACKERS] Threads

[Ulrich Neumann]

Hello all,

it's very interesting to see the discussion of threads again.

I've portet PostgreSQL to a thread-per-connection model based on
pthreads
and it is functional. Most of the work was finding all the static
globals in the sourcefiles
and swapping them between threads and freeing memory if a thread
terminates.
(PostgreSQL isn't written very clean in the aspects of memory
handling).

My version of the thread-based PostgreSQL is not very efficient at the
moment because
I haven't done any optimisation of the code to better support threads
and I'm using just a
simple semaphore to control switching of data but this could be a
starting point for
others who want to see this code. If this direction will be taken
seriously I'm very willing
to help.

If someone is interested in the code I can send a zip file to everyone
who wants.

Ulrich
--
  This e-mail is virus scanned
  Diese e-mail ist virusgeprueft


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

Re: [HACKERS] Threads

[Shridhar Daithankar]

On 6 Jan 2003 at 12:22, Ulrich Neumann wrote:

 Hello all,
 If someone is interested in the code I can send a zip file to everyone
 who wants.

I suggest you preserver your work. The reason I suggested thread are mainly two 
folds.

1) Get I/O time used fuitfully
2) Use multiple CPU better.

It will not require as much code cleaning as your efforts might had. However 
your work will be very useful if somebody decides to use thread in any fashion 
in core postgresql.

I was hoping for bit more optimistic response given that what I suggested was 
totally optional at any point of time but very important from performance 
point. Besides the change would have been gradual as required..

Anyway..

Bye
 Shridhar

--
Robot, n.:  University administrator.


---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Threads

[Greg Copeland]

On Mon, 2003-01-06 at 05:36, Shridhar Daithankar wrote:
 On 6 Jan 2003 at 12:22, Ulrich Neumann wrote:
 
  Hello all,
  If someone is interested in the code I can send a zip file to everyone
  who wants.
 
 I suggest you preserver your work. The reason I suggested thread are mainly two 
 folds.
 
 1) Get I/O time used fuitfully


AIO may address this without the need for integrated threading. 
Arguably, from the long thread that last appeared on the topic of AIO,
some hold that AIO doesn't even offer anything beyond the current
implementation.  As such, it's highly doubtful that integrated threading
is going to offer anything beyond what a sound AIO implementation can
achieve.


 2) Use multiple CPU better.
 


Multiple processes tend to universally support multiple CPUs better than
does threading.  On some platforms, the level of threading support is
currently only user mode implementations which means no additional CPU
use.  Furthermore, some platforms where user-mode threads are defacto,
they don't even allow for scheduling bias resulting is less work being
accomplished within the same time interval (work slice must be divided
between n-threads within the process, all of which run on a single CPU).


 It will not require as much code cleaning as your efforts might had. However 
 your work will be very useful if somebody decides to use thread in any fashion 
 in core postgresql.
 
 I was hoping for bit more optimistic response given that what I suggested was 
 totally optional at any point of time but very important from performance 
 point. Besides the change would have been gradual as required..
 


Speaking for my self, I probably would of been more excited if the
offered framework had addressed several issues.  The short list is:

o Code needs to be more robust.  It shouldn't be calling exit directly
as, I believe, it should be allowing for PostgreSQL to clean up some. 
Correct me as needed.  I would of also expected the code of adopted
PostgreSQL's semantics and mechanisms as needed (error reporting, etc). 
I do understand it was an initial attempt to simply get something in
front of some eyes and have something to talk about.  Just the same, I
was expecting something that we could actually pull the trigger with.

o Code isn't very portable.  Looked fairly okay for pthread platforms,
however, there is new emphasis on the Win32 platform.  I think it would
be a mistake to introduce something as significant as threading without
addressing Win32 from the get-go.

o I would desire a more highly abstracted/portable interface which
allows for different threading and synchronization primitives to be
used.  Current implementation is tightly coupled to pthreads. 
Furthermore, on platforms such as Solaris, I would hope it would easily
allow for plugging in its native threading primitives which are touted
to be much more efficient than pthreads on said platform.

o Code is not commented.  I would hope that adding new code for
something as important as threading would be commented.

o Code is fairly trivial and does not address other primitives
(semaphores, mutexs, conditions, TSS, etc) portably which would be
required for anything but the most trivial of threaded work.  This is
especially true in such an application where data IS the application. 
As such, you must reasonably assume that threads need some form of
portable serialization primitives, not to mention mechanisms for
non-trivial communication.

o Does not address issues such as thread signaling or status reporting.

o Pool interface is rather simplistic.  Does not currently support
concepts such as wake pool, stop pool, pool status, assigning a pool to
work, etc.  In fact, it's not altogether obvious what the capabilities
intent is of the current pool implementation.

o Doesn't seem to address any form of thread communication facilities
(mailboxes, queues, etc).


There are probably other things that I can find if I spend more than
just a couple of minutes looking at the code.  Honestly, I love threads
but I can see that the current code offering is not much more than a
token in its current form.  No offense meant.

After it's all said and done, I'd have to see a lot more meat before I'd
be convinced that threading is ready for PostgreSQL; from both a social
and technological perspective.


Regards,

-- 
Greg Copeland [EMAIL PROTECTED]
Copeland Computer Consulting


---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] Threads

[Shridhar Daithankar]

On Saturday 04 January 2003 03:20 am, you wrote:
 I am sure, many of you would like to delete this message before reading,
  hold on. :-)

 I'm afraid most posters did not read the message.  Those who replied

 Why bother? did not address your challenge:

Our challenges may be..;-) 

Anyway you are absolutely right. Looks like evrybody thought it as an attempt 
to convert postgresql to a thread per connection model.

 I think threads are useful in difference situations namely parallelising
 blocking conditions and using multiple CPUs.

 This is indeed one of the few good reasons for threads.  Indeed,
 large/robust systems use a mix.

 The consensus of the group is that those who do the work are not ready
 for threads.  Which is fine.  Looking into my crystal ball, I see that
 it will happen, though it appears so far away.

I hope it happens and I will be able to contribute to it if I can.

 Shridhar

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly

Re: [HACKERS] Threads

[Sailesh Krishnamurthy]

 Shridhar == Shridhar Daithankar [EMAIL PROTECTED] writes:

Shridhar On Saturday 04 January 2003 03:20 am, you wrote:
 I am sure, many of you would like to delete this message
 before reading,  hold on. :-)
 
 I'm afraid most posters did not read the message.  Those who
 replied
 
 Why bother? did not address your challenge:

Shridhar Our challenges may be..;-)

Not having threading does reduce some of the freedom we've been having
in our work. But then we have ripped the process model a fair bit and
we have the freedom of an entirely new process to deal with data
streams entering the system and we're experimenting with threading for
asynchronous I/O there.

However, in general I agree with the spirit of the previous messages
in this thread that threading isn't the main issue for PG.

One thing that I missed so far in the threading thread. Context
switches are (IMHO) far cheaper between threads, because you save TLB
flushes. Whether this makes a real difference in a data intensive
application, I don't know. I wonder how easy it is to measure the x86
counters to see TLB flushes/misses.

In a database system, even if one process dies, I'd be very chary of
trusting it. So I am not too swayed by the fact that a
process-per-connection gets you better isolation. 

BTW, many commercial database systems also use per-process models on
Unix. However they are very aggressive with connection sharing and
reuse - even to the point of reusing the same process for multiple
active connections .. maybe at transaction boundaries. Good when a
connection is maintained for a long duaration with short-lived
transactions separated by fair amouns of time. 

Moreover, in db2 for instance, the same code base is used for both
per-thread and per-process models - in other words, the entire code is
MT-safe, and the scheduling mechanism is treated as a policy (Win32 is
MT, and some Unices MP). AFAICT though, postgres code, such as perhaps
the memory contexts is not MT-safe (of course the bufferpool/shmem
accesses are safe). 

-- 
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh

---(end of broadcast)---
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Re: [HACKERS] Threads

[Kaare Rasmussen]

 Umm.  No.  User or system level threads, the statement is true.  If a
 thread kills over, the process goes with it.  Furthermore, on Win32

Hm. This is a database system. If one of the backend processes dies 
unexpectedly, I'm not sure I would trust the consistency and state of the 
others.

Or maybe I'm just being chicken.

-- 
Kaare Rasmussen--Linux, spil,--Tlf:3816 2582
Kaki Datatshirts, merchandize  Fax:3816 2501
Howitzvej 75   Åben 12.00-18.00Email: [EMAIL PROTECTED]
2000 FrederiksbergLørdag 12.00-16.00   Web:  www.suse.dk

---(end of broadcast)---
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Re: [HACKERS] Threads

[Greg Copeland]

On Sat, 2003-01-04 at 06:59, Kaare Rasmussen wrote:
  Umm.  No.  User or system level threads, the statement is true.  If a
  thread kills over, the process goes with it.  Furthermore, on Win32
 
 Hm. This is a database system. If one of the backend processes dies 
 unexpectedly, I'm not sure I would trust the consistency and state of the 
 others.
 
 Or maybe I'm just being chicken.

I'd call that being wise.  That's the problem with using threads. 
Should a thread do something naughty, the state of the entire process is
in question.  This is true regardless if it is a user mode, kernel mode,
or hybrid thread implementation.  That's the power of using the process
model that is currently in use.  Should it do something naughty, we
bitch and complain politely, throw our hands in the air and exit.  We no
longer have to worry about the state and validity of that backend.  This
creates a huge systemic reliability surplus.

This is also why the concept of a hybrid thread/process implementation
keeps coming to the surface on the list.  If you maintain the process
model and only use threads for things that ONLY relate to the single
process (single session/connection), should a thread cause a problem,
you can still throw you hands in the air and exit just as is done now
without causing problems for, or questioning the validity of, other
backends.

The cool thing about such a concept is that it still opens the door for
things like parallel sorts and queries as it relates to a single
backend.


-- 
Greg Copeland [EMAIL PROTECTED]
Copeland Computer Consulting


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Threads

[mlw]

Please no threading threads!!!

Has anyone calculated the interval and period of PostgreSQL needs 
threads posts?

The *ONLY* advantage threading has over multiple processes is the time 
and resources used in creating new processes.

That being said, I admit that creating a threaded program is easier than 
one with multiple processes, but PostgreSQL is already there and working.

Drawbacks to a threaded model:

(1) One thread screws up, the whole process dies. In a multiple process 
application this is not too much of an issue.

(2) Heap fragmentation. In a long uptime application, such as a 
database, heap fragmentation is an important consideration. With 
multiple processes, each process manages its own heap and what ever 
fragmentation that exists goes away when the connection is closed.  A 
threaded server is far more vulnerable because the heap has to manage 
many threads and the heap has to stay active and unfragmented in 
perpetuity. This is why Windows applications usually end up using 2G of 
memory after 3 months of use. (Well, this AND memory leaks)

(3) Stack space. In a threaded application they are more limits to stack 
usage. I'm not sure, but I bet PostgreSQL would have a problem with a 
fixed size stack, I know the old ODBC driver did.

(4) Lock Contention. The various single points of access in a process 
have to be serialized for multiple threads. heap allocation, 
deallocation, etc all have to be managed. In a multple process model, 
these resources would be separated by process contexts.

(5) Lastly, why bother? Seriously? Process creation time is an issue 
true, but its an issue with threads as well, just not as bad. Anyone who 
is looking for performance should be using a connection pooling 
mechanism as is done in things like PHP.

I have done both threaded and process servers. The threaded servers are 
easier to write. The process based severs are more robust. From an 
operational point of view, a select foo from bar where x  y will take 
he same amount of time.




---(end of broadcast)---
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Re: [HACKERS] Threads

[Dann Corbit]

 -Original Message-
 From: mlw [mailto:[EMAIL PROTECTED]] 
 Sent: Friday, January 03, 2003 12:47 PM
 To: Shridhar Daithankar
 Cc: PGHackers
 Subject: Re: [HACKERS] Threads
 
 
 Please no threading threads!!!
 
 Has anyone calculated the interval and period of PostgreSQL needs 
 threads posts?
 
 The *ONLY* advantage threading has over multiple processes is 
 the time 
 and resources used in creating new processes.

Threading is absurdly easier to do portably than fork().

Will you fork() successfully on MVS, VMS, OS/2, Win32?

On some operating systems, thread creation is absurdly faster than
process creation (many orders of magnitude).
 
 That being said, I admit that creating a threaded program is 
 easier than 
 one with multiple processes, but PostgreSQL is already there 
 and working.
 
 Drawbacks to a threaded model:
 
 (1) One thread screws up, the whole process dies. In a 
 multiple process 
 application this is not too much of an issue.

If you use C++ you can try/catch and nothing bad happens to anything but
the naughty thread.
 
 (2) Heap fragmentation. In a long uptime application, such as a 
 database, heap fragmentation is an important consideration. With 
 multiple processes, each process manages its own heap and what ever 
 fragmentation that exists goes away when the connection is closed.  A 
 threaded server is far more vulnerable because the heap has to manage 
 many threads and the heap has to stay active and unfragmented in 
 perpetuity. This is why Windows applications usually end up 
 using 2G of 
 memory after 3 months of use. (Well, this AND memory leaks)

Poorly written applications leak memory.  Fragmentation is a legitimate
concern.
 
 (3) Stack space. In a threaded application they are more 
 limits to stack 
 usage. I'm not sure, but I bet PostgreSQL would have a problem with a 
 fixed size stack, I know the old ODBC driver did.

A single server with 20 threads will consume less total free store
memory and automatic memory than 20 servers.  You have to decide how
much stack to give a thread, that's true.
 
 (4) Lock Contention. The various single points of access in a process 
 have to be serialized for multiple threads. heap allocation, 
 deallocation, etc all have to be managed. In a multple process model, 
 these resources would be separated by process contexts.

Semaphores are more complicated than critical sections.  If anything, a
shared memory approach is more problematic and fragile, especially when
porting to multiple operating systems.
 
 (5) Lastly, why bother? Seriously? Process creation time is an issue 
 true, but its an issue with threads as well, just not as bad. 
 Anyone who 
 is looking for performance should be using a connection pooling 
 mechanism as is done in things like PHP.
 
 I have done both threaded and process servers. The threaded 
 servers are 
 easier to write. The process based severs are more robust. From an 
 operational point of view, a select foo from bar where x  
 y will take 
 he same amount of time.

Probably true.  I think a better solution is a server that can start
threads or processes or both.  But that's neither here nor there and I'm
certainly not volunteering to write it.

Here is a solution to the dilemma.  Make the one who suggests the
feature be the first volunteer on the team that writes it.

Is it a FAQ?  If not, it ought to be.

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly

Re: [HACKERS] Threads

[Greg Copeland]

On Fri, 2003-01-03 at 14:47, mlw wrote:
 Please no threading threads!!!
 

Ya, I'm very pro threads but I've long since been sold on no threads for
PostgreSQL.  AIO on the other hand... ;)

Your summary so accurately addresses the issue it should be a whole FAQ
entry on threads and PostgreSQL.  :)


 Drawbacks to a threaded model:
 
 (1) One thread screws up, the whole process dies. In a multiple process 
 application this is not too much of an issue.
 
 (2) Heap fragmentation. In a long uptime application, such as a 
 database, heap fragmentation is an important consideration. With 
 multiple processes, each process manages its own heap and what ever 
 fragmentation that exists goes away when the connection is closed.  A 
 threaded server is far more vulnerable because the heap has to manage 
 many threads and the heap has to stay active and unfragmented in 
 perpetuity. This is why Windows applications usually end up using 2G of 
 memory after 3 months of use. (Well, this AND memory leaks)


These are things that can't be stressed enough.  IMO, these are some of
the many reasons why applications running on MS platforms tend to have
much lower application and system up times (that and resources leaks
which are inherent to the platform).

BTW, if you do much in the way of threaded coding, there is libHorde
which is a heap library for heavily threaded, memory hungry
applications.  It excels in performance, reduces heap lock contention
(maintains multiple heaps in a very thread smart manner), and goes a
long way toward reducing heap fragmentation which is common for heavily
memory based, threaded applications.


 (3) Stack space. In a threaded application they are more limits to stack 
 usage. I'm not sure, but I bet PostgreSQL would have a problem with a 
 fixed size stack, I know the old ODBC driver did.
 

Most modern thread implementations use a page guard on the stack to
determine if it needs to grow or not.  Generally speaking, for most
modern platforms which support threading, stack considerations rarely
become an issue.


 (5) Lastly, why bother? Seriously? Process creation time is an issue 
 true, but its an issue with threads as well, just not as bad. Anyone who 
 is looking for performance should be using a connection pooling 
 mechanism as is done in things like PHP.
 
 I have done both threaded and process servers. The threaded servers are 
 easier to write. The process based severs are more robust. From an 
 operational point of view, a select foo from bar where x  y will take 
 he same amount of time.
 

I agree with this, however, using threads does open the door for things
like splitting queries and sorts across multiple CPUs.  Something the
current process model, which was previously agreed on, would not be able
to address because of cost.

Example: select foo from bar where x  y order by foo ;, could be run
on multiple CPUs if the sort were large enough to justify.

After it's all said and done, I do agree that threading just doesn't
seem like a good fit for PostgreSQL.

-- 
Greg Copeland [EMAIL PROTECTED]
Copeland Computer Consulting


---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Threads

[Greg Copeland]

On Fri, 2003-01-03 at 14:52, Dann Corbit wrote:
  -Original Message-
  (1) One thread screws up, the whole process dies. In a 
  multiple process 
  application this is not too much of an issue.
 
 If you use C++ you can try/catch and nothing bad happens to anything but
 the naughty thread.

That doesn't protect against the type of issues he's talking about. 
Invalid pointer reference is a very common snafu which really hoses
threaded applications.  Not to mention resource leaks AND LOCKED
resources which are inherently an issue on Win32.

Besides, it's doubtful that PostgreSQL is going to be rewritten in C++
so bringing up try/catch is pretty much an invalid argument.

  
  (2) Heap fragmentation. In a long uptime application, such as a 
  database, heap fragmentation is an important consideration. With 
  multiple processes, each process manages its own heap and what ever 
  fragmentation that exists goes away when the connection is closed.  A 
  threaded server is far more vulnerable because the heap has to manage 
  many threads and the heap has to stay active and unfragmented in 
  perpetuity. This is why Windows applications usually end up 
  using 2G of 
  memory after 3 months of use. (Well, this AND memory leaks)
 
 Poorly written applications leak memory.  Fragmentation is a legitimate
 concern.

And well written applications which attempt to safely handle segfaults,
etc., often leak memory and lock resources like crazy.  On Win32,
depending on the nature of the resources, once this happens, even
process termination will not free/unlock the resources.

  (4) Lock Contention. The various single points of access in a process 
  have to be serialized for multiple threads. heap allocation, 
  deallocation, etc all have to be managed. In a multple process model, 
  these resources would be separated by process contexts.
 
 Semaphores are more complicated than critical sections.  If anything, a
 shared memory approach is more problematic and fragile, especially when
 porting to multiple operating systems.

And critical sections lead to low performance on SMP systems for Win32
platforms.  No task can switch on ANY CPU for the duration of the
critical section.  It's highly recommend by MS as the majority of Win32
applications expect uniprocessor systems and they are VERY fast.  As
soon as multiple processors come into the mix, critical sections become
a HORRIBLE idea if any soft of scalability is desired.


 Is it a FAQ?  If not, it ought to be.

I agree.  I think mlw's list of reasons should be added to a faq.  It
terse yet says it all!


-- 
Greg Copeland [EMAIL PROTECTED]
Copeland Computer Consulting


---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] Threads

[[EMAIL PROTECTED]]

I am sure, many of you would like to delete this message before reading, hold 
on. :-)


I'm afraid most posters did not read the message.  Those who replied 
Why bother? did not address your challenge:

I think threads are useful in difference situations namely parallelising 
blocking conditions and using multiple CPUs.
 

This is indeed one of the few good reasons for threads.  Indeed, 
large/robust systems use a mix.

The consensus of the group is that those who do the work are not ready 
for threads.  Which is fine.  Looking into my crystal ball, I see that 
it will happen, though it appears so far away.

bbaker



---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Threads

[Tom Lane]

Serguei Mokhov [EMAIL PROTECTED] writes:
 (1) One thread screws up, the whole process dies. In a 
 multiple process application this is not too much of an issue.

 (1) is an issue only for user-level threads.

Uh, what other kind of thread have you got in mind here?

I suppose the lack-of-cross-thread-protection issue would go away if
our objective was only to use threads for internal parallelism in each
backend instance (ie, you still have one process per connection, but
internally it would use multiple threads to process subqueries in
parallel).

Of course that gives up the hope of faster connection startup that has
always been touted as a major reason to want Postgres to be threaded...

regards, tom lane

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

Re: [HACKERS] Threads

[Greg Copeland]

On Fri, 2003-01-03 at 19:34, Tom Lane wrote:
 Serguei Mokhov [EMAIL PROTECTED] writes:
  (1) One thread screws up, the whole process dies. In a 
  multiple process application this is not too much of an issue.
 
  (1) is an issue only for user-level threads.
 


Umm.  No.  User or system level threads, the statement is true.  If a
thread kills over, the process goes with it.  Furthermore, on Win32
platforms, it opens a whole can of worms no matter how you care to
address it.

 Uh, what other kind of thread have you got in mind here?
 
 I suppose the lack-of-cross-thread-protection issue would go away if
 our objective was only to use threads for internal parallelism in each
 backend instance (ie, you still have one process per connection, but
 internally it would use multiple threads to process subqueries in
 parallel).
 

Several have previously spoken about a hybrid approach (ala Apache). 
IIRC, it was never ruled out but it was simply stated that no one had
the energy to put into such a concept.

 Of course that gives up the hope of faster connection startup that has
 always been touted as a major reason to want Postgres to be threaded...
 
   regards, tom lane

Faster startup, should never be the primary reason as there are many
ways to address that issue already.  Connection pooling and caching are
by far, the most common way to address this issue.  Not only that, but
by definition, it's almost an oxymoron.  If you really need high
performance, you shouldn't be using transient connections, no matter how
fast they are.  This, in turn, brings you back to persistent connections
or connection pools/caches.


-- 
Greg Copeland [EMAIL PROTECTED]
Copeland Computer Consulting


---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Threads

[mlw]

Greg Copeland wrote:

  
  
  
Of course that gives up the hope of faster connection startup that has
always been touted as a major reason to want Postgres to be threaded...

			regards, tom lane

  
  
Faster startup, should never be the primary reason as there are many
ways to address that issue already.  Connection pooling and caching are
by far, the most common way to address this issue.  Not only that, but
by definition, it's almost an oxymoron.  If you really need high
performance, you shouldn't be using transient connections, no matter how
fast they are.  This, in turn, brings you back to persistent connections
or connection pools/caches.

Connection time should *never* be in the critical path. There, I've said
it!! People who complain about connection time are barking up the wrong tree.
Regardless of the methodology, EVERY OS has issues with thread creation,
process creation, the memory allocation, and system manipulation required
to manage it. Under load this is ALWAYS slower. 

I think that if there is ever a choice, "do I make startup time faster?"
or "Do I make PostgreSQL not need a dump/restore for upgrade" the upgrade
problem has a much higher impact to real PostgreSQL sites.