Re: Bug in log.c
Stipe Tolj wrote:
>
> ok, I just commited this:
>
> 2003-06-19 Stipe Tolj <[EMAIL PROTECTED]>
> * gwlib/gwthread.h, gwlib/gwthread-pthread.c: added function
> gwthread_table_slot() to provide the slot integer of threadtable
> the thread is using.
> * gwlib/log.c: fixed Michael's reported bug for the logging
> functions.
>
> please update your trees and check.
>
> It implements the module access to the thread[] mapping table inside
> the logging module to ensure we don't access over the array limit.
>
> Because gwthread_self() is unique and assigned to a free slot when the
> thread is registered, also (gwthread_self() % THREADTABLE_SIZE) is
> unique and provides us with the correct slot the thread belongs to.
bounced back the revision of the gwthread files. In order to have them
not changed I simply #define'ed things inside log.c directly:
/*
* Ensure we use the real threadtable slot number to map the thread id
* instead of the thread id reported by gwthread_self()
*/
#define thread_slot() (gwthread_self() % THREADTABLE_SIZE)
and hence:
void error(int e, const char *fmt, ...)
{
if ((e = thread_to[thread_slot()])) {
FUNCTION_GUTS_EXCL(GW_ERROR, "");
} else {
FUNCTION_GUTS(GW_ERROR, "");
}
}
that's it.
Stipe
[EMAIL PROTECTED]
---
Wapme Systems AG
Vogelsanger Weg 80
40470 Düsseldorf
Tel: +49-211-74845-0
Fax: +49-211-74845-299
E-Mail: [EMAIL PROTECTED]
Internet: http://www.wapme-systems.de
---
wapme.net - wherever you are
Re: Bug in log.c
ok, I just commited this: 2003-06-19 Stipe Tolj <[EMAIL PROTECTED]> * gwlib/gwthread.h, gwlib/gwthread-pthread.c: added function gwthread_table_slot() to provide the slot integer of threadtable the thread is using. * gwlib/log.c: fixed Michael's reported bug for the logging functions. please update your trees and check. It implements the module access to the thread[] mapping table inside the logging module to ensure we don't access over the array limit. Because gwthread_self() is unique and assigned to a free slot when the thread is registered, also (gwthread_self() % THREADTABLE_SIZE) is unique and provides us with the correct slot the thread belongs to. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: Bug in log.c
At 05:54 PM 6/19/03 +0200, Stipe Tolj wrote: Konstantin Vayner wrote > > imho there is no point logging each _thread_ into separate file... > maybe instead logging thread _types_ into separate files? > then , mapping can be made on thread name instead of id... it's more this way: smsc modules have IO threads, those get registered via log_to_thread() to a specific log-file and inside the smsc module code you don't have to change anything in the logging functions, which was my most concern about it. which was what made it so easy to roll into our smpp server. The problem is that the "thread_id" is used as an index into the logfiles array. The thread id is essentially incremented for each thread created, so under certain circumstances this can easily be higher than 1024. Either through server apps, or lots of use of the stop/start smsc admin functions etc. The thread id the logging functions use come from the gwlib thread functions that maintain a thread table. Each new thread increments the thread id, and the entry into the table is calculated by "threadnumber % THREADTABLE_SIZE", The new thread code checks for free slots. This is way to slow for logging purposes. But surely then we could in log_thread_to use the same idea thread_to[thread_id % THREADTABLE_SIZE] = idx Given that the thread creation code has allready checked that the slot "threadnumber % THREADTABLE_SIZE" is free in the thread table and the threadnumber is the thread_id. Nisan Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: Bug in log.c
Michael Mulcahy schrieb:
>
> > so what if I have a thread 10 which is a EMI/UCP process and
> > suddendly a new process gets added with 1034? we have a new conflict...
>
> No, that what I thought as well, but the gwthread-pthread.c takes
> care of that problem.
>
> see gwthread-pthread.c
>
> /* Find a free table entry and claim it. */
> first_try = next_threadnumber;
> do {
> ti->number = next_threadnumber++;
> /* Check if we looped all the way around the thread table. */
> if (ti->number == first_try + THREADTABLE_SIZE) {
> panic(0, "Cannot have more than %d active threads",
> THREADTABLE_SIZE);
> }
> } while (THREAD(ti->number) != NULL);
> THREAD(ti->number) = ti;
>
> active_threads++;
>
> where THREAD(t) is:
> #define THREAD(t) (threadtable[(t) % THREADTABLE_SIZE])
>
> The thread number is incremented and the mod 1024 position in
> array is checked to see if an entry exists. If an entry exists then
> the thread number is incremented again.
ok, once again I'll chance my opinion ;))
Andreas, do you agree that a (gwthread_self() % THREADTABLE_SIZE)
gives us a runtime unque ID of the thread?
Stipe
[EMAIL PROTECTED]
---
Wapme Systems AG
Vogelsanger Weg 80
40470 Düsseldorf
Tel: +49-211-74845-0
Fax: +49-211-74845-299
E-Mail: [EMAIL PROTECTED]
Internet: http://www.wapme-systems.de
---
wapme.net - wherever you are
Re: Bug in log.c
Andreas Fink schrieb: > > > so what if I have a thread 10 which is a EMI/UCP process and suddendly a new process > gets added with 1034? > we have a new conflict... ahhh, Andreas *is* right here. We don't take into account that the threadtable handling does a linear scan in the threadtable to find a free slot. So the gwthread_self() id can't be used to determine a unique thread id with the module operator, right?! Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: Bug in log.c
Andreas Fink wrote: > > so what if I have a thread 10 which is a EMI/UCP process and suddendly a new process > gets added with 1034? > we have a new conflict... no, the second thread (1034) can not be in the same threadtable slot as the first one. The slots are filled sequentially. So at a specific time only one therad can be assigned to one threadtable slot, right?! Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: Bug in log.c
Alexander Malysh wrote:
>
> oops, that was wrong :(
>
> here is the possible solution:
> static void delete_threadinfo(void)
> {
> struct threadinfo *threadinfo;
>
> threadinfo = getthreadinfo();
> list_destroy(threadinfo->joiners, NULL);
> close(threadinfo->wakefd_recv);
> close(threadinfo->wakefd_send);
> THREAD(threadinfo->number) = NULL;
> active_threads--;
> if (threadinfo->number < next_threadnumber)
> next_threadnumber = threadinfo->number;
> gw_assert(threadinfo != &mainthread);
> gw_free(threadinfo);
> }
>
> next_threadnumber is always the min. free slot number.
> worst case while creating new thread:
> next_threadnumber is used then max MAX_THREATABLE_SIZE-next_threadnumber
> loop.
to be honest I'm a bit concerned in too drastically change things
inside our thread handling.
I'll rather pick the easier and secure way here.
Stipe
[EMAIL PROTECTED]
---
Wapme Systems AG
Vogelsanger Weg 80
40470 Düsseldorf
Tel: +49-211-74845-0
Fax: +49-211-74845-299
E-Mail: [EMAIL PROTECTED]
Internet: http://www.wapme-systems.de
---
wapme.net - wherever you are
RE: Bug in log.c
> so what if I have a thread 10 which is a EMI/UCP process and
> suddendly a new process gets added with 1034? we have a new conflict...
No, that what I thought as well, but the gwthread-pthread.c takes
care of that problem.
see gwthread-pthread.c
/* Find a free table entry and claim it. */
first_try = next_threadnumber;
do {
ti->number = next_threadnumber++;
/* Check if we looped all the way around the thread table. */
if (ti->number == first_try + THREADTABLE_SIZE) {
panic(0, "Cannot have more than %d active threads",
THREADTABLE_SIZE);
}
} while (THREAD(ti->number) != NULL);
THREAD(ti->number) = ti;
active_threads++;
where THREAD(t) is:
#define THREAD(t) (threadtable[(t) % THREADTABLE_SIZE])
The thread number is incremented and the mod 1024 position in
array is checked to see if an entry exists. If an entry exists then
the thread number is incremented again.
Just to clarify this in my own mind I wrote a program
essentially the following:
/* create 100 threads that sleep indefinitely */
for (i; i < 100; i++)
gwthread_create(sleepy_thread, &val);
/* create 100 threads that sleep for small bit */
for (i = 0; i < 2000; i++)
{
gwthread_create(sleepy_thread, &val2);
gwthread_sleep(0.1);
}
the output shows how the thread number works:
2003-06-19 18:17:18 [0] DEBUG: Started thread 1021
(C:\dev\wirelesswindow\test\test_threads.c:sleepy_thread)
2003-06-19 18:17:18 [0] DEBUG: Started thread 1022
(C:\dev\wirelesswindow\test\test_threads.c:sleepy_thread)
2003-06-19 18:17:19 [0] DEBUG: Started thread 1023
(C:\dev\wirelesswindow\test\test_threads.c:sleepy_thread)
2003-06-19 18:17:19 [0] DEBUG: Started thread 1125
(C:\dev\wirelesswindow\test\test_threads.c:sleepy_thread)
2003-06-19 18:17:19 [0] DEBUG: Started thread 1126
(C:\dev\wirelesswindow\test\test_threads.c:sleepy_thread)
2003-06-19 18:17:19 [0] DEBUG: Started thread 1127
(C:\dev\wirelesswindow\test\test_threads.c:sleepy_thread)
2003-06-19 18:17:19 [0] DEBUG: Started thread 1128
(C:\dev\wirelesswindow\test\test_threads.c:sleepy_thread)
2003-06-19 18:17:19 [0] DEBUG: Started thread 1129
(C:\dev\wirelesswindow\test\test_threads.c:sleepy_thread)
2003-06-19 18:17:19 [0] DEBUG: Started thread 1130
(C:\dev\wirelesswindow\test\test_threads.c:sleepy_thread)
2003-06-19 18:17:19 [0] DEBUG: Started thread 1131
(C:\dev\wirelesswindow\test\test_threads.c:sleepy_thread)
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of
Andreas Fink
Sent: 19 June 2003 17:55
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: Bug in log.c
On Jeudi, juin 19, 2003, at 06:26 Uhr, Michael Mulcahy wrote:
Hi All,
Found a solution that I think should be ok.
In the gwthread-module the table is size 1024.
While the thread number may increase byond it the
id is a modulo of 1024 and so the modulo can be
used in the log.c to prevent the prior mentioned
problem.
suggested fix would be to use following:
if ((e = thread_to[(gwthread_self()%1024)])) {
Have tested this and appears to work.
Sorry for not spotting this earlier.
Kindest Regards,
Michael.
so what if I have a thread 10 which is a EMI/UCP process and suddendly a new
process gets added with 1034?
we have a new conflict...
Andreas Fink
Global Networks Switzerland AG
--
Tel: +41-61-333 Fax: +41-61-334 Mobile: +41-79-2457333
Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland
Web: http://www.global-networks.ch/ [EMAIL PROTECTED]
--
Re: Bug in log.c
oops, that was wrong :(
here is the possible solution:
static void delete_threadinfo(void)
{
struct threadinfo *threadinfo;
threadinfo = getthreadinfo();
list_destroy(threadinfo->joiners, NULL);
close(threadinfo->wakefd_recv);
close(threadinfo->wakefd_send);
THREAD(threadinfo->number) = NULL;
active_threads--;
if (threadinfo->number < next_threadnumber)
next_threadnumber = threadinfo->number;
gw_assert(threadinfo != &mainthread);
gw_free(threadinfo);
}
next_threadnumber is always the min. free slot number.
worst case while creating new thread:
next_threadnumber is used then max MAX_THREATABLE_SIZE-next_threadnumber
loop.
Am Donnerstag, 19. Juni 2003 18:52 schrieb Alexander Malysh:
> Hi,
>
> Am Donnerstag, 19. Juni 2003 18:36 schrieb Stipe Tolj:
> > Michael Mulcahy schrieb:
> > > Hi All,
> > >
> > > Found a solution that I think should be ok.
> > >
> > > In the gwthread-module the table is size 1024.
> > > While the thread number may increase byond it the
> > > id is a modulo of 1024 and so the modulo can be
> > > used in the log.c to prevent the prior mentioned
> > > problem.
> > >
> > > suggested fix would be to use following:
> > >
> > > if ((e = thread_to[(gwthread_self()%1024)])) {
> > >
> > > Have tested this and appears to work.
> >
> > how about having this inside gwthread-pthread.c as function:
> >
> > /* Return the threadtable sloot this thread is using. */
> > long gwthread_table_slot(void)
> > {
> > return (gwthread_self() % THREADTABLE_SIZE);
> > }
> >
> > and use this call instead of gwthread_self() for the logging
> > functions?!
>
> how about just reusing of freed thread number for next tread ?
>
> static void delete_threadinfo(void)
> {
> struct threadinfo *threadinfo;
>
> threadinfo = getthreadinfo();
> list_destroy(threadinfo->joiners, NULL);
> close(threadinfo->wakefd_recv);
> close(threadinfo->wakefd_send);
> THREAD(threadinfo->number) = NULL;
> active_threads--;
> next_threadnumber = threadinfo->number; /* reuse tread numer */
> gw_assert(threadinfo != &mainthread);
> gw_free(threadinfo);
> }
>
> > Stipe
> >
> > [EMAIL PROTECTED]
> > ---
> > Wapme Systems AG
> >
> > Vogelsanger Weg 80
> > 40470 Düsseldorf
> >
> > Tel: +49-211-74845-0
> > Fax: +49-211-74845-299
> >
> > E-Mail: [EMAIL PROTECTED]
> > Internet: http://www.wapme-systems.de
> > ---
> > wapme.net - wherever you are
--
Best regards / Mit besten Grüßen aus Köln
Dipl.-Ing.
Alexander Malysh
___
Centrium GmbH
Ehrenstrasse 2
50672 Köln
Fon: +49 (0221) 277 49 240
Fax: +49 (0221) 277 49 109
email: a.malysh at centrium.de
web: http://www.centrium.de
msn: olek2002 at hotmail.com
___
Please avoid sending me Word or PowerPoint attachments.
See http://www.fsf.org/philosophy/no-word-attachments.html
Re: Bug in log.c
On Jeudi, juin 19, 2003, at 06:03 Uhr, Stipe Tolj wrote: Nisan Bloch schrieb: Hi We run into this all the time with out SMPP server. Threads are spawned for new ESME connections and terminated on disconnect. Some of our users connect and disconnect very regularily and the thread-id often goes over 1024. This however means quite a few changes to the way the log stuff currently works. We may have to use a list instead of an array. hmmm, list are *slow* compared to a fixed array. You have to keep in mind that the logging functions are caller *really* a lot of times anywhere in the code, so the mapping has to be as fast as possible. there's an easier way. you make two array. one with the tread ID and one with the logfile stuff. the code has to go through array1 to find the correct thread ID to then find the thing it needs in array2. Worst case scenario: a loop from 1-1024 in a sequencial array of integers. Every CPU can do that blasting fast and cache helps. improvement could be done with hashing function. or by using thread%1024 as a default place and if its not there go searching... Andreas Fink Global Networks Switzerland AG -- Tel: +41-61-333 Fax: +41-61-334 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] --
Re: Bug in log.c
On Jeudi, juin 19, 2003, at 06:26 Uhr, Michael Mulcahy wrote:
Hi All,
Found a solution that I think should be ok.
In the gwthread-module the table is size 1024.
While the thread number may increase byond it the
id is a modulo of 1024 and so the modulo can be
used in the log.c to prevent the prior mentioned
problem.
suggested fix would be to use following:
if ((e = thread_to[(gwthread_self()%1024)])) {
Have tested this and appears to work.
Sorry for not spotting this earlier.
Kindest Regards,
Michael.
so what if I have a thread 10 which is a EMI/UCP process and suddendly a new process gets added with 1034?
we have a new conflict...
Andreas Fink
Global Networks Switzerland AG
--
Tel: +41-61-333 Fax: +41-61-334 Mobile: +41-79-2457333
Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland
Web: http://www.global-networks.ch/ [EMAIL PROTECTED]
--
Re: Bug in log.c
Hi,
Am Donnerstag, 19. Juni 2003 18:36 schrieb Stipe Tolj:
> Michael Mulcahy schrieb:
> > Hi All,
> >
> > Found a solution that I think should be ok.
> >
> > In the gwthread-module the table is size 1024.
> > While the thread number may increase byond it the
> > id is a modulo of 1024 and so the modulo can be
> > used in the log.c to prevent the prior mentioned
> > problem.
> >
> > suggested fix would be to use following:
> >
> > if ((e = thread_to[(gwthread_self()%1024)])) {
> >
> > Have tested this and appears to work.
>
> how about having this inside gwthread-pthread.c as function:
>
> /* Return the threadtable sloot this thread is using. */
> long gwthread_table_slot(void)
> {
> return (gwthread_self() % THREADTABLE_SIZE);
> }
>
> and use this call instead of gwthread_self() for the logging
> functions?!
how about just reusing of freed thread number for next tread ?
static void delete_threadinfo(void)
{
struct threadinfo *threadinfo;
threadinfo = getthreadinfo();
list_destroy(threadinfo->joiners, NULL);
close(threadinfo->wakefd_recv);
close(threadinfo->wakefd_send);
THREAD(threadinfo->number) = NULL;
active_threads--;
next_threadnumber = threadinfo->number; /* reuse tread numer */
gw_assert(threadinfo != &mainthread);
gw_free(threadinfo);
}
>
> Stipe
>
> [EMAIL PROTECTED]
> ---
> Wapme Systems AG
>
> Vogelsanger Weg 80
> 40470 Düsseldorf
>
> Tel: +49-211-74845-0
> Fax: +49-211-74845-299
>
> E-Mail: [EMAIL PROTECTED]
> Internet: http://www.wapme-systems.de
> ---
> wapme.net - wherever you are
--
Best regards / Mit besten Grüßen aus Köln
Dipl.-Ing.
Alexander Malysh
___
Centrium GmbH
Ehrenstrasse 2
50672 Köln
Fon: +49 (0221) 277 49 240
Fax: +49 (0221) 277 49 109
email: a.malysh at centrium.de
web: http://www.centrium.de
msn: olek2002 at hotmail.com
___
Please avoid sending me Word or PowerPoint attachments.
See http://www.fsf.org/philosophy/no-word-attachments.html
Re: Bug in log.c
> how about having this inside gwthread-pthread.c as function:
>
> /* Return the threadtable sloot this thread is using. */
> long gwthread_table_slot(void)
> {
> return (gwthread_self() % THREADTABLE_SIZE);
> }
>
> and use this call instead of gwthread_self() for the logging
> functions?!
how much performance impact would a function call have this way
instead of doing it directly inside the logging functions?!
Anyone having C performance experience here?
Stipe
[EMAIL PROTECTED]
---
Wapme Systems AG
Vogelsanger Weg 80
40470 Düsseldorf
Tel: +49-211-74845-0
Fax: +49-211-74845-299
E-Mail: [EMAIL PROTECTED]
Internet: http://www.wapme-systems.de
---
wapme.net - wherever you are
Re: Bug in log.c
Michael Mulcahy schrieb:
>
> Hi All,
>
> Found a solution that I think should be ok.
>
> In the gwthread-module the table is size 1024.
> While the thread number may increase byond it the
> id is a modulo of 1024 and so the modulo can be
> used in the log.c to prevent the prior mentioned
> problem.
>
> suggested fix would be to use following:
>
> if ((e = thread_to[(gwthread_self()%1024)])) {
>
> Have tested this and appears to work.
how about having this inside gwthread-pthread.c as function:
/* Return the threadtable sloot this thread is using. */
long gwthread_table_slot(void)
{
return (gwthread_self() % THREADTABLE_SIZE);
}
and use this call instead of gwthread_self() for the logging
functions?!
Stipe
[EMAIL PROTECTED]
---
Wapme Systems AG
Vogelsanger Weg 80
40470 Düsseldorf
Tel: +49-211-74845-0
Fax: +49-211-74845-299
E-Mail: [EMAIL PROTECTED]
Internet: http://www.wapme-systems.de
---
wapme.net - wherever you are
Re: Bug in log.c
At 05:54 PM 6/19/03 +0200, Stipe Tolj wrote: Konstantin Vayner wrote > > imho there is no point logging each _thread_ into separate file... > maybe instead logging thread _types_ into separate files? > then , mapping can be made on thread name instead of id... it's more this way: smsc modules have IO threads, those get registered via log_to_thread() to a specific log-file and inside the smsc module code you don't have to change anything in the logging functions, which was my most concern about it. which was what made it so easy to roll into our smpp server. The problem is that the "thread_id" is used as an index into the logfiles array. The thread id is essentially incremented for each thread created, so under certain circumstances this can easily be higher than 1024. Either through server apps, or lots of use of the stop/start smsc admin functions etc. The thread id the logging functions use come from the gwlib thread functions that maintain a thread table. Each new thread increments the thread id, and the entry into the table is calculated by "threadnumber % THREADTABLE_SIZE", The new thread code checks for free slots. This is way to slow for logging purposes. But surely then we could in log_thread_to use the same idea thread_to[thread_id % THREADTABLE_SIZE] = idx Given that the thread creation code has allready checked that the slot "threadnumber % THREADTABLE_SIZE" is free in the thread table and the threadnumber is the thread_id. Nisan Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
RE: Bug in log.c
Hi All,
Found a solution that I think should be ok.
In the gwthread-module the table is size 1024.
While the thread number may increase byond it the
id is a modulo of 1024 and so the modulo can be
used in the log.c to prevent the prior mentioned
problem.
suggested fix would be to use following:
if ((e = thread_to[(gwthread_self()%1024)])) {
Have tested this and appears to work.
Sorry for not spotting this earlier.
Kindest Regards,
Michael.
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] Behalf Of
> Stipe Tolj
> Sent: 19 June 2003 17:04
> To: Nisan Bloch
> Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> Subject: Re: Bug in log.c
>
>
> Nisan Bloch schrieb:
> >
> > Hi
> >
> > We run into this all the time with out SMPP server. Threads
> are spawned for
> > new ESME connections and terminated on disconnect. Some of our users
> > connect and disconnect very regularily and the thread-id
> often goes over
> > 1024. This however means quite a few changes to the way the
> log stuff
> > currently works. We may have to use a list instead of an array.
>
> hmmm, list are *slow* compared to a fixed array.
>
> You have to keep in mind that the logging functions are caller
> *really* a lot of times anywhere in the code, so the mapping has to be
> as fast as possible.
>
> Is there any concern in changing gwthread-pthread.c code in cycling
> between 1-1024 for the thread_ids instead of incrementing?!
>
> Stipe
>
> [EMAIL PROTECTED]
> ---
> Wapme Systems AG
>
> Vogelsanger Weg 80
> 40470 Düsseldorf
>
> Tel: +49-211-74845-0
> Fax: +49-211-74845-299
>
> E-Mail: [EMAIL PROTECTED]
> Internet: http://www.wapme-systems.de
> ---
> wapme.net - wherever you are
>
Re: Bug in log.c
Nisan Bloch schrieb: > > Hi > > We run into this all the time with out SMPP server. Threads are spawned for > new ESME connections and terminated on disconnect. Some of our users > connect and disconnect very regularily and the thread-id often goes over > 1024. This however means quite a few changes to the way the log stuff > currently works. We may have to use a list instead of an array. hmmm, list are *slow* compared to a fixed array. You have to keep in mind that the logging functions are caller *really* a lot of times anywhere in the code, so the mapping has to be as fast as possible. Is there any concern in changing gwthread-pthread.c code in cycling between 1-1024 for the thread_ids instead of incrementing?! Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: Bug in log.c
a hard limit to reach in normal life - but can still become an issue if someone is trying to crash the machine... for example, opening 2000 concurrent connections to the machine from the outside will pass the 1024 barrier without any problem... Stipe Tolj wrote: a) change gwthread code to use the freed slot ID number instead of incrementing. So we will never have a thread id above 1024 (which is the hard thread limit). b) thing how we can map the thread to the exlusive log file to be honest I'd like to pick a), because b) should be pretty fast because the logging functions are called a lot on high-load systems, so this has to be very fast and array lookups are pretty fast. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: Bug in log.c
Konstantin Vayner wrote > > imho there is no point logging each _thread_ into separate file... > maybe instead logging thread _types_ into separate files? > then , mapping can be made on thread name instead of id... it's more this way: smsc modules have IO threads, those get registered via log_to_thread() to a specific log-file and inside the smsc module code you don't have to change anything in the logging functions, which was my most concern about it. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
RE: Bug in log.c
Hi
We run into this all the time with out SMPP server. Threads are spawned for
new ESME connections and terminated on disconnect. Some of our users
connect and disconnect very regularily and the thread-id often goes over
1024. This however means quite a few changes to the way the log stuff
currently works. We may have to use a list instead of an array.
Nisan
At 03:51 PM 6/19/03 +0100, Michael Mulcahy wrote:
Hi Stipe,
There does not need to be more 1024 concurrent threads to
reproduce this problem. The problem is associated with
the internal thread number assigned to each thread. the
log output displays this number in brackets.
The new logging of threads to a different log file
expects this number to be between 0 and 1024 which
may not be the case.
The thread implementation contains an integer value that
is assigned to a new thread and is then incremented.
If you create a thread and destroy it many times then the
number can get larger than 1024.
For example:
Start the bearerbox
Write a script that connects to smsbox port and then disconnects.
put this is a loop for 2000 times.
2 threads will get created and then destroyed for each connection
establishment and for each disconnection. As the threads get created the
number grows.
2003-06-19 15:46:55 [6] DEBUG: Started thread 23 (gw\bb_boxc.c:function)
2003-06-19 15:46:55 [23] INFO: Client connected from <192.168.0.97>
2003-06-19 15:46:55 [23] DEBUG: Started thread 24 (gw\bb_boxc.c:boxc_sender)
2003-06-19 15:46:55 [23] INFO: Connection closed by the box <192.168.0.97>
2003-06-19 15:46:55 [24] DEBUG: Thread 24 (gw\bb_boxc.c:boxc_sender)
terminates.
2003-06-19 15:46:55 [23] DEBUG: Thread 23 (gw\bb_boxc.c:function)
terminates.
2003-06-19 15:46:56 [6] DEBUG: Started thread 25 (gw\bb_boxc.c:function)
2003-06-19 15:46:56 [25] INFO: Client connected from <192.168.0.97>
2003-06-19 15:46:56 [25] DEBUG: Started thread 26 (gw\bb_boxc.c:boxc_sender)
2003-06-19 15:46:56 [25] INFO: Connection closed by the box <192.168.0.97>
2003-06-19 15:46:56 [26] DEBUG: Thread 26 (gw\bb_boxc.c:boxc_sender)
terminates.
2003-06-19 15:46:56 [25] DEBUG: Thread 25 (gw\bb_boxc.c:function)
terminates.
2003-06-19 15:46:57 [6] DEBUG: Started thread 27 (gw\bb_boxc.c:function)
2003-06-19 15:46:57 [27] INFO: Client connected from <192.168.0.97>
2003-06-19 15:46:57 [27] DEBUG: Started thread 28 (gw\bb_boxc.c:boxc_sender)
2003-06-19 15:46:57 [27] INFO: Connection closed by the box <192.168.0.97>
2003-06-19 15:46:57 [28] DEBUG: Thread 28 (gw\bb_boxc.c:boxc_sender)
terminates.
here is a ruby script I wrote to reproduce this:
require 'socket'
count = 0
while (count < 3000)
t = TCPSocket.new('127.0.0.1', 13001)
t.close
count += 1
end
Hope that helps explain it a bit better,
Warmest Regards,
Michael.
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] Behalf Of
> Stipe Tolj
> Sent: 19 June 2003 14:58
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: Bug in log.c
>
>
> Hi Michael,
>
> Michael Mulcahy wrote:
> >
> > Hi All,
> >
> > There is a bug in the logging functions debug, info,
> warning, error and
> > panic
> > which cause a crash when called from a thread that has a
> thread id larger
> > than 1024, i.e. gwthread_self returns value > 1024
> >
> > In the log module there is an array for mapping threads to logfiles:
> >
> > #define THREADTABLE_SIZE 1024
> > static unsigned int thread_to [ (long) THREADTABLE_SIZE ];
> >
> > The log functions use this to identify which log file to
> write the data to,
> > as in following:
> >
> > void error(int e, const char *fmt, ...)
> > {
> > if ((e = thread_to[gwthread_self()])) { /
> Relevent line ***/
> > FUNCTION_GUTS_EXCL(GW_ERROR, "");
> > } else {
> > FUNCTION_GUTS(GW_ERROR, "");
> > }
> > }
> >
> > Unfortunately if gwthread_self returns a value greater than
> 1024 then
> > problems
> > occur.
> >
> > Comments or suggestions appreciated!
>
> you are right. But how to reproduce this?
>
> Have you ceriously running bearerbox with more than 1024 threads
> inside?
>
> Stipe
>
> [EMAIL PROTECTED]
> ---
> Wapme Systems AG
>
> Vogelsanger Weg 80
> 40470 Düsseldorf
>
> Tel: +49-211-74845-0
> Fax: +49-211-74845-299
>
> E-Mail: [EMAIL PROTECTED]
> Internet: http://www.wapme-systems.de
> ---
> wapme.net - wherever you are
>
Re: Bug in log.c
imho there is no point logging each _thread_ into separate file... maybe instead logging thread _types_ into separate files? then , mapping can be made on thread name instead of id... Stipe Tolj wrote: ok, so we have 2 alternative here: a) change gwthread code to use the freed slot ID number instead of incrementing. So we will never have a thread id above 1024 (which is the hard thread limit). b) thing how we can map the thread to the exlusive log file any ideas?! Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: Bug in log.c
> a) change gwthread code to use the freed slot ID number instead of > incrementing. So we will never have a thread id above 1024 (which is > the hard thread limit). > b) thing how we can map the thread to the exlusive log file to be honest I'd like to pick a), because b) should be pretty fast because the logging functions are called a lot on high-load systems, so this has to be very fast and array lookups are pretty fast. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: Bug in log.c
ok, so we have 2 alternative here: a) change gwthread code to use the freed slot ID number instead of incrementing. So we will never have a thread id above 1024 (which is the hard thread limit). b) thing how we can map the thread to the exlusive log file any ideas?! Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: Bug in log.c
Michael Mulcahy wrote: > > Hi Stipe, > > There does not need to be more 1024 concurrent threads to > reproduce this problem. The problem is associated with > the internal thread number assigned to each thread. the > log output displays this number in brackets. > > The new logging of threads to a different log file > expects this number to be between 0 and 1024 which > may not be the case. > > The thread implementation contains an integer value that > is assigned to a new thread and is then incremented. > > If you create a thread and destroy it many times then the > number can get larger than 1024. ok, got it now. It's an incremental counter. > Hope that helps explain it a bit better, definetly did! thanks a lot. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
RE: Bug in log.c
Hi Stipe,
There does not need to be more 1024 concurrent threads to
reproduce this problem. The problem is associated with
the internal thread number assigned to each thread. the
log output displays this number in brackets.
The new logging of threads to a different log file
expects this number to be between 0 and 1024 which
may not be the case.
The thread implementation contains an integer value that
is assigned to a new thread and is then incremented.
If you create a thread and destroy it many times then the
number can get larger than 1024.
For example:
Start the bearerbox
Write a script that connects to smsbox port and then disconnects.
put this is a loop for 2000 times.
2 threads will get created and then destroyed for each connection
establishment and for each disconnection. As the threads get created the
number grows.
2003-06-19 15:46:55 [6] DEBUG: Started thread 23 (gw\bb_boxc.c:function)
2003-06-19 15:46:55 [23] INFO: Client connected from <192.168.0.97>
2003-06-19 15:46:55 [23] DEBUG: Started thread 24 (gw\bb_boxc.c:boxc_sender)
2003-06-19 15:46:55 [23] INFO: Connection closed by the box <192.168.0.97>
2003-06-19 15:46:55 [24] DEBUG: Thread 24 (gw\bb_boxc.c:boxc_sender)
terminates.
2003-06-19 15:46:55 [23] DEBUG: Thread 23 (gw\bb_boxc.c:function)
terminates.
2003-06-19 15:46:56 [6] DEBUG: Started thread 25 (gw\bb_boxc.c:function)
2003-06-19 15:46:56 [25] INFO: Client connected from <192.168.0.97>
2003-06-19 15:46:56 [25] DEBUG: Started thread 26 (gw\bb_boxc.c:boxc_sender)
2003-06-19 15:46:56 [25] INFO: Connection closed by the box <192.168.0.97>
2003-06-19 15:46:56 [26] DEBUG: Thread 26 (gw\bb_boxc.c:boxc_sender)
terminates.
2003-06-19 15:46:56 [25] DEBUG: Thread 25 (gw\bb_boxc.c:function)
terminates.
2003-06-19 15:46:57 [6] DEBUG: Started thread 27 (gw\bb_boxc.c:function)
2003-06-19 15:46:57 [27] INFO: Client connected from <192.168.0.97>
2003-06-19 15:46:57 [27] DEBUG: Started thread 28 (gw\bb_boxc.c:boxc_sender)
2003-06-19 15:46:57 [27] INFO: Connection closed by the box <192.168.0.97>
2003-06-19 15:46:57 [28] DEBUG: Thread 28 (gw\bb_boxc.c:boxc_sender)
terminates.
here is a ruby script I wrote to reproduce this:
require 'socket'
count = 0
while (count < 3000)
t = TCPSocket.new('127.0.0.1', 13001)
t.close
count += 1
end
Hope that helps explain it a bit better,
Warmest Regards,
Michael.
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] Behalf Of
> Stipe Tolj
> Sent: 19 June 2003 14:58
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: Bug in log.c
>
>
> Hi Michael,
>
> Michael Mulcahy wrote:
> >
> > Hi All,
> >
> > There is a bug in the logging functions debug, info,
> warning, error and
> > panic
> > which cause a crash when called from a thread that has a
> thread id larger
> > than 1024, i.e. gwthread_self returns value > 1024
> >
> > In the log module there is an array for mapping threads to logfiles:
> >
> > #define THREADTABLE_SIZE 1024
> > static unsigned int thread_to [ (long) THREADTABLE_SIZE ];
> >
> > The log functions use this to identify which log file to
> write the data to,
> > as in following:
> >
> > void error(int e, const char *fmt, ...)
> > {
> > if ((e = thread_to[gwthread_self()])) { /
> Relevent line ***/
> > FUNCTION_GUTS_EXCL(GW_ERROR, "");
> > } else {
> > FUNCTION_GUTS(GW_ERROR, "");
> > }
> > }
> >
> > Unfortunately if gwthread_self returns a value greater than
> 1024 then
> > problems
> > occur.
> >
> > Comments or suggestions appreciated!
>
> you are right. But how to reproduce this?
>
> Have you ceriously running bearerbox with more than 1024 threads
> inside?
>
> Stipe
>
> [EMAIL PROTECTED]
> ---
> Wapme Systems AG
>
> Vogelsanger Weg 80
> 40470 Düsseldorf
>
> Tel: +49-211-74845-0
> Fax: +49-211-74845-299
>
> E-Mail: [EMAIL PROTECTED]
> Internet: http://www.wapme-systems.de
> ---
> wapme.net - wherever you are
>
Re: Bug in log.c
Hi Michael,
Michael Mulcahy wrote:
>
> Hi All,
>
> There is a bug in the logging functions debug, info, warning, error and
> panic
> which cause a crash when called from a thread that has a thread id larger
> than 1024, i.e. gwthread_self returns value > 1024
>
> In the log module there is an array for mapping threads to logfiles:
>
> #define THREADTABLE_SIZE 1024
> static unsigned int thread_to [ (long) THREADTABLE_SIZE ];
>
> The log functions use this to identify which log file to write the data to,
> as in following:
>
> void error(int e, const char *fmt, ...)
> {
> if ((e = thread_to[gwthread_self()])) { / Relevent line ***/
> FUNCTION_GUTS_EXCL(GW_ERROR, "");
> } else {
> FUNCTION_GUTS(GW_ERROR, "");
> }
> }
>
> Unfortunately if gwthread_self returns a value greater than 1024 then
> problems
> occur.
>
> Comments or suggestions appreciated!
you are right. But how to reproduce this?
Have you ceriously running bearerbox with more than 1024 threads
inside?
Stipe
[EMAIL PROTECTED]
---
Wapme Systems AG
Vogelsanger Weg 80
40470 Düsseldorf
Tel: +49-211-74845-0
Fax: +49-211-74845-299
E-Mail: [EMAIL PROTECTED]
Internet: http://www.wapme-systems.de
---
wapme.net - wherever you are
