Re: [AOLSERVER] What does 'exiting: exceeded max connections per thread' mean?
Hi, Is there a way of killing the threads once max connection has been reached or does aolserver automatically take care of this? Regards -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] What does 'exiting: exceeded max connections per thread' mean?
From my own experience, it seems that I get a ton of that like all the threads (or a majority of them) have exceeded max connections and it seems that it does take a while before they start serving again. I'd like to know if this is normal or should the turn around time be a lot quicker... Thanks! -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] AOLserver and Oracle unable to allocate db handle
Hi, I'm running on AOLserver 4.5.1 and everything for the most part seems fine. On occasion, I get this error message when the server tries to get a db handle: could not allocate 1 handle(s) from pool default From what I know, it should mean that AOLserver ran out of database pools to use. I tried upping the pools on the 'default' group but it didn't seem to help. I'm wondering if anyone can shed some light on the nature of this problem and possible things I should look into. Any help is appreciated. Thanks! -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsmemcache driver issue with data over 2kb
I have run some tests and Majid's changes fixed the bug, so I don't think my patch is relevant anymore. Cheers. On Sep 4, 11:12 am, Sep Ng thejackschm...@gmail.com wrote: Yes sir! As soon as possible, I will look at it and determine whether it's still relevant. I'll post a new diff if needed. On Sep 4, 10:55 am, Dossy Shiobara do...@panoptic.com wrote: After you test the latest code from Majid, determine if your patch is still relevant. If it is, I'll merge it in. On 8/31/10 7:17 PM, Sep Ng wrote: Okay, I guess I'll simply post the diff patch here for anyone who needs it. -- Dossy Shiobara | do...@panoptic.com |http://dossy.org/ Panoptic Computer Network |http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Oracle driver update and SF question
I have tested your code updates Majid and they work splendidly! On Sep 4, 10:54 am, Dossy Shiobara do...@panoptic.com wrote: I have committed Majid's code to GitHub that he has sent to me and have tagged it version 1.1. You can download the tarball for it using this URL: http://github.com/aolserver/nsmemcache/tarball/1.1 On 9/3/10 5:03 PM, Majid Khan wrote: Yes just remove what's there because there are multiple/redundant copies of the same file so its very confusing which file is the correct version of memcache and about the history, ever since I have imported no one added any patch so there is nothing that we will lose. The code I have sent you is the improved bug free version of nsmemcache which I have never commit to CVS (my bad) nor its documented. Sep I would want you to please do your test against the one which Dossy will commit. Dossy I haven't prepared any test case, I shall prepare the one and will let you know. -- Dossy Shiobara | do...@panoptic.com |http://dossy.org/ Panoptic Computer Network |http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Oracle driver update and SF question
Majid, I will be more than happy to do it. I just probably need to figure how and where to git pull. I don't think I have an account however. Regards. On Sep 4, 5:03 am, Majid Khan majidkha...@gmail.com wrote: Yes just remove what's there because there are multiple/redundant copies of the same file so its very confusing which file is the correct version of memcache and about the history, ever since I have imported no one added any patch so there is nothing that we will lose. The code I have sent you is the improved bug free version of nsmemcache which I have never commit to CVS (my bad) nor its documented. Sep I would want you to please do your test against the one which Dossy will commit. Dossy I haven't prepared any test case, I shall prepare the one and will let you know. Regards, Majid. On Fri, Sep 3, 2010 at 7:11 PM, Dossy Shiobara do...@panoptic.com wrote: Is it necessary to remove what's there (and lose the change history) or just commit your changes? Send me what you have and I can commit it, or you can fork and send a pull request through GitHub. Are your changes documented? How can they be tested? Will they conflict with Sep's BUFSIZE patch? Majid Khan majidkha...@gmail.com wrote: Hi Dossy, Is it possible to remove the one which you just added on github because it's not the clean version which I wanted to add at the time when I ported for aolserver and also I have fixed a type casting bug in the code later on. So I wanted to import a clean and a stable version of nsmemcache to github. I am not sure whether I would be able to import to github but if you want I can give you a tar ball of nsmecache and you can import that one. -- Dossy Shiobara -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsmemcache driver issue with data over 2kb
Yes sir! As soon as possible, I will look at it and determine whether it's still relevant. I'll post a new diff if needed. On Sep 4, 10:55 am, Dossy Shiobara do...@panoptic.com wrote: After you test the latest code from Majid, determine if your patch is still relevant. If it is, I'll merge it in. On 8/31/10 7:17 PM, Sep Ng wrote: Okay, I guess I'll simply post the diff patch here for anyone who needs it. -- Dossy Shiobara | do...@panoptic.com |http://dossy.org/ Panoptic Computer Network |http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsmemcache driver issue with data over 2kb
Okay, I guess I'll simply post the diff patch here for anyone who needs it. --- nsmemcache.c.4.5.1 2010-08-30 06:56:06.0 +0800 +++ nsmemcache.c2010-08-30 11:19:20.0 +0800 @@ -50,6 +50,7 @@ #define MC_SERVER_DELETED 2 /* Server is not used anymore */ int Ns_ModuleVersion = 1; +int bufsize = BUFSIZE; typedef struct mc_conn_t { @@ -184,6 +185,18 @@ key = Ns_SetValue(set, i); mc_server_add(mc, mc_server_create(key, 0, 0)); Ns_Log(Notice, nsmemcache: added %s, key); +} else if (!strcasecmp(key, bufsize)) { +key = Ns_SetValue(set, i); + +if (key != NULL) { + +if(sscanf(key,%d,bufsize) == -1) { +//Reset the value to default buffer size. +bufsize = BUFSIZE; +} else { +Ns_Log(Notice,nsmemcache: setting buffer size to %d, bufsize); +} +} } } // For AOL 4.0.10 @@ -664,7 +677,7 @@ return -1; } -rc = mc_conn_read(conn, BUFSIZE, 1, line); +rc = mc_conn_read(conn, bufsize, 1, line); if (rc = 0) { mc_conn_free(conn); mc_server_dead(mc, ms); @@ -725,7 +738,7 @@ /* VALUE key flags bytes [cas unique]\r\ndata block\r\nEND \r\n */ -rc = mc_conn_read(conn, BUFSIZE, 1, line); +rc = mc_conn_read(conn, bufsize, 1, line); if (rc = 0 || line == NULL) { mc_conn_free(conn); mc_server_dead(mc, ms); @@ -817,7 +830,7 @@ return -1; } -rc = mc_conn_read(conn, BUFSIZE, 1, line); +rc = mc_conn_read(conn, bufsize, 1, line); if (rc = 0) { mc_conn_free(conn); mc_server_dead(mc, ms); @@ -876,7 +889,7 @@ /* value\r\n */ -rc = mc_conn_read(conn, BUFSIZE, 1, line); +rc = mc_conn_read(conn, bufsize, 1, line); if (rc = 0) { mc_conn_free(conn); mc_server_dead(mc, ms); @@ -929,7 +942,7 @@ /* VERSION version\r\n */ -rc = mc_conn_read(conn, BUFSIZE, 1, line); +rc = mc_conn_read(conn, bufsize, 1, line); if (rc = 0) { mc_conn_free(conn); mc_server_dead(mc, ms); @@ -994,7 +1007,7 @@ return -1; } -rc = mc_conn_read(conn, BUFSIZE, 1, line); +rc = mc_conn_read(conn, bufsize, 1, line); if (rc = 0) { mc_conn_free(conn); mc_server_dead(mc, ms); @@ -1048,10 +1061,10 @@ // Read trailing \r\n and END\r\n if (strncmp(conn-ds.string, \r\n, 2)) { -rc = mc_conn_read(conn, BUFSIZE, 0, line); +rc = mc_conn_read(conn, bufsize, 0, line); } if (strstr(conn-ds.string, END\r\n) == NULL) { - rc = mc_conn_read(conn, BUFSIZE, 0, line); + rc = mc_conn_read(conn, bufsize, 0, line); } mc_conn_put(conn); return 1; On Aug 31, 5:43 am, Sep Ng thejackschm...@gmail.com wrote: Hi, I've been trying to use nsmemcache and integrate it with aolserver but I noticed that there's this bug where if you retrieve data over 2kb, the data gets truncated. After inspection, it looks like it's because the buffer size allocated when reading the contents via Ns_SockRecv is not adequate. I wonder if anyone has come across this partcular issue. I modified nsmemcache to accept a parameter setting from aolserver to define a custom buffer size. I wasn't entirely sure where the code submissions were. I'd be more than happy to send the changes upstream, hope someone can point me the way because the contributing patches section on the aolserver website was kind of confusing. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] nsmemcache driver issue with data over 2kb
Hi, I've been trying to use nsmemcache and integrate it with aolserver but I noticed that there's this bug where if you retrieve data over 2kb, the data gets truncated. After inspection, it looks like it's because the buffer size allocated when reading the contents via Ns_SockRecv is not adequate. I wonder if anyone has come across this partcular issue. I modified nsmemcache to accept a parameter setting from aolserver to define a custom buffer size. I wasn't entirely sure where the code submissions were. I'd be more than happy to send the changes upstream, hope someone can point me the way because the contributing patches section on the aolserver website was kind of confusing. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Oracle driver update and SF question
Hi Andrew, It seems that the changes you did improved the stability of aolserver. I will observe more of this, but the patch is looking good. On Aug 27, 6:13 am, Sep Ng thejackschm...@gmail.com wrote: Thanks for letting me know, Andrew. I will try to get around to testing the latest code base from CVS. I remember that the crashes happen way too often so I'm confident that if the crash was not already fixed, it's going to happen again. On Aug 27, 12:41 am, Andrew Steets ste...@gmail.com wrote: There was an uniititialized int * being passed into OCI in the code that handles 'ns_ora exec_plsql'. 2.7 is nearly 6 years old now. If you can post the backtrace of a crash w/ the most recent nsoracle from CVS I'm happy to take a look at it. Are you able to reliably reproduce the crash? -Andrew On Wed, Aug 25, 2010 at 6:15 PM, Sep Ng thejackschm...@gmail.com wrote: Hi, Can I ask what nature of the crash this fixes? I remember trying nsoracle 2.8 fork and it was extremely unstable and had to rollback to 2.7. Is this on the CVS tree now? Thanks! On Aug 26, 4:56 am, Andrew Steets ste...@gmail.com wrote: Hello, I just checked in a patch for the Oracle driver that fixes a crash bug we were seeing on some of our servers. Anyone running a relatively recent (last two years) version of the Oracle driver may want to switch. I saw some e-mail a while ago about switching to GitHub, but I don't see any of the modules on GitHub. I have some other nsoracle patches (eg. log warning and query text for queries running X seconds) that would probably be better suited to a private fork or branch. Any plans to migrate the modules to GitHub? -Andrew -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Oracle driver update and SF question
Thanks for letting me know, Andrew. I will try to get around to testing the latest code base from CVS. I remember that the crashes happen way too often so I'm confident that if the crash was not already fixed, it's going to happen again. On Aug 27, 12:41 am, Andrew Steets ste...@gmail.com wrote: There was an uniititialized int * being passed into OCI in the code that handles 'ns_ora exec_plsql'. 2.7 is nearly 6 years old now. If you can post the backtrace of a crash w/ the most recent nsoracle from CVS I'm happy to take a look at it. Are you able to reliably reproduce the crash? -Andrew On Wed, Aug 25, 2010 at 6:15 PM, Sep Ng thejackschm...@gmail.com wrote: Hi, Can I ask what nature of the crash this fixes? I remember trying nsoracle 2.8 fork and it was extremely unstable and had to rollback to 2.7. Is this on the CVS tree now? Thanks! On Aug 26, 4:56 am, Andrew Steets ste...@gmail.com wrote: Hello, I just checked in a patch for the Oracle driver that fixes a crash bug we were seeing on some of our servers. Anyone running a relatively recent (last two years) version of the Oracle driver may want to switch. I saw some e-mail a while ago about switching to GitHub, but I don't see any of the modules on GitHub. I have some other nsoracle patches (eg. log warning and query text for queries running X seconds) that would probably be better suited to a private fork or branch. Any plans to migrate the modules to GitHub? -Andrew -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Oracle driver update and SF question
Hi, Can I ask what nature of the crash this fixes? I remember trying nsoracle 2.8 fork and it was extremely unstable and had to rollback to 2.7. Is this on the CVS tree now? Thanks! On Aug 26, 4:56 am, Andrew Steets ste...@gmail.com wrote: Hello, I just checked in a patch for the Oracle driver that fixes a crash bug we were seeing on some of our servers. Anyone running a relatively recent (last two years) version of the Oracle driver may want to switch. I saw some e-mail a while ago about switching to GitHub, but I don't see any of the modules on GitHub. I have some other nsoracle patches (eg. log warning and query text for queries running X seconds) that would probably be better suited to a private fork or branch. Any plans to migrate the modules to GitHub? -Andrew -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Handling threads the right way
I've been poking around with the aolserver gdb traces I've done in the past and it looks like there's a high turnaround of threads turning into zombie threads. Personally, I think this is much inline with either bad tcl code or a configuration issue. I'm going to poke around and I have yet to go through the links Andrew provided but thought I'd throw this update out there. On Jul 1, 11:13 am, Sep Ng thejackschm...@gmail.com wrote: On Jul 1, 10:42 am, Andrew Piskorski a...@piskorski.com wrote: On Wed, Jun 30, 2010 at 05:21:40PM -0400, Andrew Piskorski wrote: If you manage to find a list somewhere of what MS Windows library calls are or are not thread-safe, then you could use various tools to find ALL the calls in your AOLserver binaries, and compare the two lists to see if AOLserver seems to be calling anything unsafe. Hm, I thought you were running AOLserver on MS Windows (which is possible but certainly unusual), but later you mention using ulimit, so in hindsight my assumption was almost certainly incorrect. Are you using some flavor of Linux like most people? I'm handling quite a few differnet flavours and it's kind of maddening. I've got a Lenny, an Etch and a few old RHEL ones as well. I brought up Windows because I read herehttp://aolserver.am.net/docs/tuning.adpx that there were instability issues with Windows and AOLserver with certain settings. And I was wondering if the same problems extend to other OSes like Linux. As for lists of thread-unsafe functions for various OSs, it seems some progress has been made since I last looked into it c. 2002 or 3. Some brief googling suggests: http://blog.josefsson.org/2009/06/23/thread-safe-functions/ http://etbe.coker.com.au/2009/06/14/finding-thread-unsafe-code/ http://developers.sun.com/solaris/articles/multithreaded.html http://www.devx.com/cplus/Article/4/1763/page/3 http://valgrind.org/info/tools.html#helgrind Those three guys' various lists of functions are of course unlikely to be conclusive. But it's a lot better than nothing. That's a great bunch of links and I will pour over them for sure! Personally, I think it's pretty crazy that shared libraries shipped with OSs don't provide some sort of simple list noting the thread-safety status of every single public function they provide. That sounds like common sense to me. Would have been nice. -- Andrew Piskorski a...@piskorski.comhttp://www.piskorski.com/ -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Handling threads the right way
On Jul 1, 4:59 am, Andrew Piskorski a...@piskorski.com wrote: On Tue, Jun 29, 2010 at 03:23:38PM -0700, Sep Ng wrote: Basically we had aolservers running and while serving pages, it's also doing some heavy load processing from a ton of scheduled custom written procedures. Scheduled using AOLserver built-in scheduler, ns_schedule_proc, ns_schedule_daily, or the like? Yeah. A lot of ns_schedule_* calls. We also have a thread that retrieves jobs from the database and spawns more threads to execute those jobs. Aolserver crashes and segmentation faults are fairly frequent and the logs at the time pointed to these running threads as a probable cause. Then the first place to look is in your custom code, it's the most likely place for the bug. Is your scheduled code purely Tcl or does it use any C code? If you turn off your scheduled procs, does the crashing go away? It's purely TCL code. When we did turn off the scheduled procs, aolserver would definitely not crash but with the scheduled tasks not running, aolserver does not seem to be in any burden at all and serving pages are pretty fast. This is a debugging problem, you need to find the bug before you decide how to fix it. After the crash look at the core file's stack trace in a debugger and see if that gives you any clues. Can you reproduce the problem by hitting your development AOLserver with a particular load-testing script? If the problem is non-obvious, you'll probably need that to track it down. I suppose this is as you describe it. I've been meaning to set the servers to create the core dump flie, but it never seemed to. That's a separate issue altogether and that even though ulimit reports the core file size as unlimited, it doesn't seem to be creating it. Your focus on AOLserver's thread creation and scheduling mechanisms seems misplaced. You're speculating about ways to fix some imagined problem, but you don't know yet whether your actual problem has any similarity at all to your speculations. So basically, what I'm currently beating my head over is to build a much cleaner and better way of handling all the load It's not clear that building any such thing will help you. If the crash-inducing bugs are in your custom scheduled code, it's fairly likely that they're still going to crash no matter what thread you run them in or how you go about scheduling those threads. That's a good point. If after lots of looking you REALLY can't find the crash-causing bug(s), THEN I'd start thinking about ways to live with and ameliorate the problem. The simplest one of course, which you've probably already done, is to just let your AOLserver crash and make sure that it's always able to come back up quickly and pick up as close to where it left off as possible. Yeah, we've been surviving on that for a while now. Better, is to isolate your custom scheduled code in an entirely separate process, with communication between your AOLserver and that helper process. AOLserver 4.5 definitely includes a mechanism for doing that, but I forget what it's called. That way, your code may well still crash, but it will only take down the helper process rather than your entire AOLserver. -- Andrew Piskorski a...@piskorski.comhttp://www.piskorski.com/ -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Handling threads the right way
On Jul 1, 5:21 am, Andrew Piskorski a...@piskorski.com wrote: On Tue, Jun 29, 2010 at 06:19:06PM -0700, Sep Ng wrote: How can I tell which ones are thread safe? This sounds like something I will need to look into before I start writing code. *All* AOLserver modules must be thread safe. If they have any parts that are not, then that's clearly a bug which their developer overlooked, and needs to be fixed. Keep in mind however, that it's entirely possible for a module to call some external C library which happens to be completely thread-safe on one operating system, but horribly unsafe another. All part of the fun of cross-platform programming... This likely means that whatever issues I'm hitting will still happen even with a rewrite. Alright, I will try and take a closer look at all of this. If you manage to find a list somewhere of what MS Windows library calls are or are not thread-safe, then you could use various tools to find ALL the calls in your AOLserver binaries, and compare the two lists to see if AOLserver seems to be calling anything unsafe. Unfortunately very few operating systems provide any such clear, consolidated documentation of what function calls are thread-safe vs. not. You're lucky if the docs on each individual function call even tell you, and of course as Gustaf mentioned, occasionally those docs are wrong. My general impression though, is that historically MS Windows has tended to have a lot FEWER non-thread-safe library calls in use than Unix. This is probably because Win32 was first written in an era when threads were very popular, while most versions of Unix have roots stretching back well before then. (Supposedly that's also why the multi-process support in Win32 has always been said to be lousy, but the success of multi-process Google Chrome on Win32 suggest that those problems have either been fixed, or can be effectively worked around.) I prefer libthread, since all such threads run in an event loop. I don't think I've ever heard of this on Aolserver... I always thought Aolserver's threads would eventually end up using the tcl+libthread but it seems that there's a real difference in this. The Tcl Threads Extension's libthread was written AFTER AOLserver; in fact AOLserver is what inspired Zoran to write libthread in the first place. Generally speaking, libthread is backwards compatible with AOLserver, but also includes some newer stuff that AOLserver does not. libthread is designed so that you can easily use it from inside AOLserver, including as a drop-in replacement for AOLserver's older nsv_* code. It should be technically feasible to change AOLserver to use libthread directly, but no one has done the work. (And anyway, none of that has much of anything to do with your debugging problem.) So typically the config file has no connection whatsoever to the threads of aolserver and that it only pertains to the connection threads, or am I confusing this even further? Your the threads of aolserver terminology above is certainly confused. A connection thread is one of the various flavors of threads used in AOLserver. Right. Thanks for that clarification. You guys have been tremendously helpful in giving me such immensely informative posts so this is all well appreciated. -- Andrew Piskorski a...@piskorski.comhttp://www.piskorski.com/ -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Handling threads the right way
On Jul 1, 10:42 am, Andrew Piskorski a...@piskorski.com wrote: On Wed, Jun 30, 2010 at 05:21:40PM -0400, Andrew Piskorski wrote: If you manage to find a list somewhere of what MS Windows library calls are or are not thread-safe, then you could use various tools to find ALL the calls in your AOLserver binaries, and compare the two lists to see if AOLserver seems to be calling anything unsafe. Hm, I thought you were running AOLserver on MS Windows (which is possible but certainly unusual), but later you mention using ulimit, so in hindsight my assumption was almost certainly incorrect. Are you using some flavor of Linux like most people? I'm handling quite a few differnet flavours and it's kind of maddening. I've got a Lenny, an Etch and a few old RHEL ones as well. I brought up Windows because I read here http://aolserver.am.net/docs/tuning.adpx that there were instability issues with Windows and AOLserver with certain settings. And I was wondering if the same problems extend to other OSes like Linux. As for lists of thread-unsafe functions for various OSs, it seems some progress has been made since I last looked into it c. 2002 or 3. Some brief googling suggests: http://blog.josefsson.org/2009/06/23/thread-safe-functions/ http://etbe.coker.com.au/2009/06/14/finding-thread-unsafe-code/ http://developers.sun.com/solaris/articles/multithreaded.html http://www.devx.com/cplus/Article/4/1763/page/3 http://valgrind.org/info/tools.html#helgrind Those three guys' various lists of functions are of course unlikely to be conclusive. But it's a lot better than nothing. That's a great bunch of links and I will pour over them for sure! Personally, I think it's pretty crazy that shared libraries shipped with OSs don't provide some sort of simple list noting the thread-safety status of every single public function they provide. That sounds like common sense to me. Would have been nice. -- Andrew Piskorski a...@piskorski.comhttp://www.piskorski.com/ -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] Handling threads the right way
Hi, I've been looking into building some thread intensive applications on top of aolserver (both on 4.0.10 and 4.5.1) and from experience, it seems that this maybe one of the easier points to get wrong and crash aolserver. I've wanted looked for documentation before but there does not seem to be any clear answers so I'm hoping someone can let me know of a couple of things. 1. Is it advisable for aolserver to run a detached ns thread that is supposed to run for the duration of the server? I was wondering about memory consumption and whether it's better to let the thread perform the task and die after which the server spawns another thread for a separate execution. 2. I read that in Windows, thread destruction can cause instability and possible memory leaks. Does this extend to other OS platforms? 3. I understand the general idea behind spawning threads with aolserver, but ideally, I'd like to avoid the taboos on them, so any idea about this is well-appreciated. Thanks for reading this and hope to hear back. Regards. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Handling threads the right way
Thanks for the input. Basically we had aolservers running and while serving pages, it's also doing some heavy load processing from a ton of scheduled custom written procedures. Aolserver crashes and segmentation faults are fairly frequent and the logs at the time pointed to these running threads as a probable cause. Possibly a configuration issue, but I remember we tried fiddling with those numbers and it never really helped. So basically, what I'm currently beating my head over is to build a much cleaner and better way of handling all the load but in so doing, I'm not entirely sure whether or not to spawn a lot of threads for the jobs, or basically keep it to a minimum. Judging from Andrew's post, it would likely be better to reuse threads but I'm not entirely sure how that happens. I mean, everytime you'd invoke ns_thread begin/begindetached, you are creating a new thread already, no? How do you reuse them? This is probably a stupid question, but is there a distinction between the threads and connection threads in aolserver? I know the connection threads are probably the connections to the database (I think). Thank you all for your insights and hope to hear more! On Jun 30, 12:59 am, Jeff Hobbs je...@activestate.com wrote: On 28/06/2010 11:25 PM, Sep Ng wrote: 2. I read that in Windows, thread destruction can cause instability and possible memory leaks. Does this extend to other OS platforms? Just to highlight this point - this is partially true. For some versions of msvcrt, the stock, documented thread calls actually would end up leaking memory. This is why Tcl does not use those, resorting to lower-level calls. This odd quirk of Windows msvcrt has become common knowledge over time. If you are not using Tcl threads (which layer over native threads), make sure you read up on _beginthreadex/_endthreadex. Jeff -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Handling threads the right way
On Jun 30, 8:27 am, Gustaf Neumann neum...@wu-wien.ac.at wrote: Am 30.06.10 00:23, schrieb Sep Ng: Basically we had aolservers running and while serving pages, it's also doing some heavy load processing from a ton of scheduled custom written procedures. Aolserver crashes and segmentation faults are fairly frequent and the logs at the time pointed to these running threads as a probable cause. Possibly a configuration issue, but I remember we tried fiddling with those numbers and it never really helped. From my experience, the two most common causes for crashes in aolserver are a) c-based extensions (tcl extensions and/or aolserver modules) which are not thread safe (e.g. using non thread-safe libraries) or which have bugs (e.g. double frees of memory), or b) running out of memory. The aolserver threads are typically heavy-weight, the zippy memory allocator over-allocates memory and does not free it. All threads use the same memory block, which is limited on most 32bit machines to 2GB. When you hit the limit, the server crashes. A solution for (b) is to compile everything (tcl, aolserver, modules) with 64 bit. That's a good point, but I'm not entirely sure that aolserver ever really hit the 2GB limit. I guess I couldn't really speculate on it. If you are using just Tcl + Aolserver commands, the server should never crash with recent versions of aolserver and tcl (e.g. tcl 8.4.19, 8.5.8). What C extensions or aolserver modules are you using? I think we tend to use a lot of aolserver modules. And I would suppose a little haphazardly too. These are the extra ones that I remember. nsoracle nscache nsclamav nsaspell (not sure) nssha1 (not sure) nszlib nsopenssl (not sure) and nsreturnz on 4.0.10 - basically this was a port I found sometime back but I know we're removing it when we fully move to 4.5.1. How can I tell which ones are thread safe? This sounds like something I will need to look into before I start writing code. The following heuristic might help in the search of the problem. If your crashes are random and independent of the load (e.g. happen with already a little load) then bugs in the C code of the extensions are likely. If crashes happens during thread end, bugs in the memory management of the c-extensions are likely (tcl deletes an interpreter with all associated memory). If the crashes happen under heavy load, thread-safety of some component is likely to be the problem (e.g. we had a long time problems with our aolserver, which ran stable up to about 600 concurrent users, then it started to crash. One of the problem was the kerberos library, which was not reentrant, although it claimed to be so). So basically, what I'm currently beating my head over is to build a much cleaner and better way of handling all the load but in so doing, I'm not entirely sure whether or not to spawn a lot of threads for the jobs, or basically keep it to a minimum. Judging from Andrew's post, it would likely be better to reuse threads but I'm not entirely sure how that happens. I mean, everytime you'd invoke ns_thread begin/begindetached, you are creating a new thread already, no? How do you reuse them? In general, you have two options: you can use aolserver-threads and threads created by the tcl thread library (libthread), which can be loaded as a module into the aolserver (when compiled with the appropriate flags). I prefer libthread, since all such threads run in an event loop. This makes it very easy to send to these threads commands which are implemented via event-loops. One can as well to implement this way easy communication between the threads. It is essentially the same programming model that can be used by using tcl+libthread outside of the aolserver. I don't think I've ever heard of this on Aolserver... I always thought Aolserver's threads would eventually end up using the tcl+libthread but it seems that there's a real difference in this. This is probably a stupid question, but is there a distinction between the threads and connection threads in aolserver? I know the connection threads are probably the connections to the database (I think). no, the connection threads handle incoming HTTP requests, they have a connection to the web-client. These are the threads controlled in the config file via e.g, maxthreads. Typically, a connection thread is created on demand (incoming request) and lives, until either it times out (it received no requests for a certain time) or it has served a maximum number of requests (all configurable in the config file). When it reaches the end of its live-cycle, it is terminated (and maybe recreated afterwards by new demand). The threads of aolserver are typically heavyweight, since every thread contains a full tcl interpreter. The weight depends on the size of the blueprint, which is determined essentially by all tcl-procs available
Re: [AOLSERVER] Has anyone had experience on 64-bit AOLserver for possible memory leaks?
That's very interesting to know, Gustaf. I've got 4.5.1 built though performance seems to be roughly the same. I'm using nsoracle and at this point, I think it might be the cause of some segmentation fault crashes. I'm using 2.8a1 which seems to be less stable than 2.7 though I'm also half expecting application level issues with aolserver. All seem to point to aolserver being capable of 64-bit, so I should suppose this is an application level problem. It's just really hard to figure out how to trace all the threads that are running. I hope you might give me some hint on debugging techniques on aolserver. Thanks very much for your insight. On Jun 15, 3:09 pm, Gustaf Neumann neum...@wu-wien.ac.at wrote: Am 14.06.10 09:24, schrieb Sep Ng: Thanks for the swift reply Gustaf. I have Debian Etch on 64-bit as the OS platform and I'm thinking that OS choice should not particularly affect performance, right? i doubt that there will be a perceivable difference between Linux versions, but this depends, how fine you measure :). openacs.org runs aolserver 4.5.1 with Ubuntu 8.04.3 LTS since more than a year without any problems. $ file /usr/local/aolserver/bin/nsd /usr/local/aolserver/bin/nsd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), for GNU/Linux 2.6.8, dynamically linked (uses shared libs), not stripped -gustaf -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] Has anyone had experience on 64-bit AOLserver for possible memory leaks?
Hi, I was looking over the discussion here... http://groups.google.com/group/aolserver/browse_thread/thread/7efbe7faff45419a?pli=1 And I've been experiencing something similar with 64-bit compiled Aolserver 4.0 release 10 wherein the cpu processing leaps to 100% or more, and memory consumption just keeps getting larger and larger even though I'm the only one accessing the system. I'm looking into upgrading to 4.5.1 but I'm not sure if there's anything I should look into. The thread posted had a suggestion that tcl shouldn't be compiled with --enable-64-bit but I wanted to find out if anyone had any up-to-date information on going 64-bit with AOLserver with both versions (4.0.10 and 4.5.1)? Looking forward to some ideas. Thanks! -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Is ns_info threads safe to use?
Hello everyone, I would like to say first of all I appreciate all the thought and ideas you've brought to the table. What I hope to achieve is to be able produce a rudimentary load management. I will take a look at ns_pools and see if I can lift anything from it. Tom, you are right that AOLserver puts the thread id on the logs and maybe it is worth a look. On Jun 3, 7:37 am, Tom Jackson t...@rmadilo.com wrote: This is an interesting topic, but I can't think of anything to be gained from a list of threads. Who cares that a thread exists? But if you are just concerned with threads, you can use ns_pools. All worker threads in AOLserver are in some named thread pool. If you don't use threadpools, all requests, no matter how many virtual servers you use are handled by the default threadpool. A query using ns_pools can give you a current thread count. Also you should know that every ns_log (error log) line has information about the threadpool and the thread id and every error log has the process id dot thread id just after the timestamp. If you wrote a script to examine the last part of the log file, you could discover which threads were active. Personally I would abandon the use of a list of living threads as a measure of anything. When AOLserver goes dark threads usually don't go away. tom jackson On Tue, 2009-06-02 at 12:12 +0200, Gustaf Neumann wrote: It certainly depends on what your application needs. There is no principal problem obtaining the thread id (eg. ns_thread id, ::thread::id). One could either use the sketched approach and simply record whatever the application needs, or ffone can to extend the xotcl-request monitor to track this information as well. just to get the information about running connection threads from the xotcl-request-monitor, use throttle running. best regards -gustaf neumann Sep Ng schrieb: Hi Gustav! Thanks for the info. I'm afraid xotcl-request-monitor may not be good enough if I do not have the thread ids, although I guess I could rewrite it to work with what xotcl-request-monitor provides. I did not know of one such monitoring page on OpenACS. I will take a look and see what I can find. On Jun 2, 3:42 pm, Gustaf Neumann neum...@wu.ac.at wrote: Sep Ng schrieb: Is there a way in AOLserver to do live monitoring on threads? We're sort of hoping to get info on what thread ids are running as of the moment on aolserver. The xotcl-request-monitor watches running requests, essentially via defining filters/traces for requsts and using a monitor thread for keeping track of the starts and ends of requests. If there are starts recorded without ends, it knows these requests are still running in some threads. This approach does not depened on ns_info, we use it on all our production sites. Originally i had one version for pure aolserver and one for OpenACS; since a while i just work on the OpenACS version (which is available via the public cvs repository of OpenACS). best regards -gustaf neumann -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Is ns_info threads safe to use?
Hi Jim! Thanks for the reply, Jim. I sort of felt that way too about ns_info threads. Is there a way in AOLserver to do live monitoring on threads? We're sort of hoping to get info on what thread ids are running as of the moment on aolserver. On Jun 2, 12:03 am, Jim Davidson jgdavid...@mac.com wrote: Hi, The code for ns_info threads looks about the same in 4.0 and 4.5 and it doesn't look super safe. It does have a lock around walking the list of threads but there are place where it copies data from strings which may be changing. And, like ns_server active, it's really designed for live diag work, not so much as a background health check. -Jim On May 31, 2009, at 7:53 PM, Sep Ng wrote: I'm handling aolserver 4.0.10 and yeah, I know it's kind of old. If I'm not mistaken, ns_info threads was not totally thread safe in that version and it caused quite a few crashes here. I think I've managed to get that number of crashing down a bit with mutex and a bit of aolserver 4.5 code, but it still does happen. I'm wondering though if ns_info threads was meant to be written as a proc for diagnosing AOLserver or if it's safe to use to periodically check AOLserver for running threads. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Question on two AOLserver tickets
Hi Dossy, To be frank, the only reason why I keep using the Trac ticket tracker is that it's the one I found. Had I known that the sourceforge ticket tracker was the active one, I would have used that one instead. Begs to question though... why two ticket trackers? Anyway, thanks everyone for all the valuable information! On May 21, 1:22 am, Dossy Shiobara do...@panoptic.com wrote: On 5/20/09 11:11 AM, Jim Davidson wrote: Yup -- agreed. I was talking with Dossy who says the Trac stuff is generally out of date anyway, the definitive bug list is still on Sourceforge (definitive in it's the place, can't say how accurate the bug reports are). I would love to hear that the AOLserver community prefers Trac for ticket tracking over SourceForge's tracker and that everyone would be happy if I refreshed the Trac tickets with a current snapshot of SourceForge data, and we could shut down the SourceForge tracker once and for all ... But, I don't want to kick that hornet's nest again, so I'll let someone else do it this time. :-) -- Dossy Shiobara | do...@panoptic.com |http://dossy.org/ Panoptic Computer Network |http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Question on two AOLserver tickets
I also checked on the other ticket I mentioned on the commit logs and it seems that the ticket may have been addressed with aolserver 4.5 too... so I think it's looking like everything is resolved. On May 20, 11:11 pm, Jim Davidson jgdavid...@mac.com wrote: Yup -- agreed. I was talking with Dossy who says the Trac stuff is generally out of date anyway, the definitive bug list is still on Sourceforge (definitive in it's the place, can't say how accurate the bug reports are). -Jim On May 15, 2009, at 6:35 PM, Jack Schmidt wrote: I guess if it's thread safe now, the ticket should be closed with a reference to aolserver 4.5. 2009/5/16 Jim Davidson jgdavid...@mac.com Hi, I'm looking at the head code and it appears it's now safe -- the connection list is walked with a pool lock held and the request data (method, url) seem to be copied with a global reqlock mutex held. Jade: What version of AOLserver are you using? -Jim On May 15, 2009, at 10:06 AM, Tom Jackson wrote: One thing which would be nice is to put the Tcl API into a module which as to be loaded, and keep the C API available in the core. It is obviously more work, but the admin has to make a choice to load the Tcl API, which is more open to misuse. I did something like this for a few additional ns_conn options. Gustaf's idea is easier to pull off at least initially, but it shouldn't be tied to an unrelated option like debugging. I leave on debugging for low traffic servers and I can also turn it on/off on a per-namespace basis. tom jackson On Thu, 2009-05-14 at 15:48 -0700, Sep Ng wrote: How about having the proc enable only if debug settings are turned on on AOLserver? -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- A scrum a day keeps the pigs at bay -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] Question on two AOLserver tickets
Hi, I'm trying to debug an AOLserver crash and the point of crash seems to be AppendConn in NS_GetProcInfo... I will post the stack trace after just for reference. Looking through the ticket tracker on AOLserver, I found two tickets of particular interest: http://dev.aolserver.com/trac/ticket/325 -- My question with this ticket is was it ever resolved? and the second ticket: http://dev.aolserver.com/trac/ticket/152 -- This problem should only happen if the command ns_server was called in a multi-threaded environment, right? Here is the call stack trace I'm working with. I'm more interested in Ticket #325 as it may be related to my problem. - Call Stack Trace - calling call entryargument values in hex location type point(? means dubious value) kpedbg_dmp_stack()+ call B5B81884 B45FFB74 ? 0 ? 219 kpeDbgCrash()+72 call B5B75E14 0 ? 5 ? 0 ? 80BD810 ? B45FFC08 ? B45FFBF0 ? kpeDbgSignalHandler call B5B867B4 0 ? 5 ? B72A331C ? 2 ? 4 ? ()+107 5F ? 4 ? B4600C5D ? skgesig_sigactionHa call B45FFC54 ? B739FFE0 ? ndler()+214 gsignal()+71 signal 6 ? B460110C ? B460118C ? abort()+265 call gsignal()6 ? B460152C ? 0 ? B7FC1E84 ? B4601550 ? B4601564 ? NsBlockSignals() call B7F749F0 3 ? B7FB9ED5 ? B ? 30 ? 46 ? B7F565F0 ? B7FC2420 call B ? 33 ? 0 ? 7B ? 7B ? C ? AppendConn()+117 call B7F74E20 B4601AE8 ? C ? 51C5 ? 0 ? 1 ? B7E46FF4 ? NsConnArgProc()+61 call AppendConn() B4601AE8 ? 80B0C1C ? B7FB51A2 ? ? 228E24D8 ? 0 ? Ns_GetProcInfo()+16 call B4601AE8 ? CD298C0 ? 1 B4601A28 ? B7F33C33 ? B4DF4EA1 ? B7E46BA0 ? ThreadArgProc()+43 call B7F74410 B4601AE8 ? B7F8E9B6 ? CD298C0 ? B7F6337C ? CCF7A20 ? Ns_ThreadList()+207 call B4601AE8 ? B7F8E9B6 ? CD298C0 ? 0 ? 4A0935D9 ? B7FBB174 ? NsTclInfoObjCmd()+5 call B7F73B30 B4601AE8 ? B7F8917B ? 46 B7FBC080 ? B7FB34D3 ? 0 ? B4601AE4 ? TclEvalObjvInternal call EF0B1C0 ? CE907D0 ? 2 ? ()+819 EC701D8 ? B304D010 ? A7DBAE50 ? TclExecuteByteCode( call _init()+184 CE907D0 ? 2 ? EC701D8 ? 0 ? )+107130 ? 0 ? TclCompEvalObj()+15 call TclExecuteByteCode( CE907D0 ? 0 ? 0 ? 0 ? 2 )B4602924 ? 34ECE ? TclObjInterpProc()+ call B7EBE8E0 CE907D0 ? ABF19440 ? 645120C4660 ? 1 ? B7F565F0 ? 18 ? TclEvalObjvInternal call ABF78CE8 ? CE907D0 ? 1 ? ()+819 EC701D4 ? B7F565F0 ? A7DBB540 ? TclExecuteByteCode( call _init()+184 CE907D0 ? 1 ? EC701D4 ? 0 ? )+107130 ? 0 ? TclCompEvalObj()+15 call TclExecuteByteCode( CE907D0 ? 3 ? 3 ? B7F565F0 ? 2 )B4602924 ? 34EC2 ? TclObjInterpProc()+ call B7EBE8E0 CE907D0 ? ABF19320 ? 645120C4260 ? 1 ? 100 ? 100 ? TclEvalObjvInternal call ABF76E28 ? CE907D0 ? 2 ? ()+819 EC701CC ? B7F565F0 ? A7DBAE50 ? TclExecuteByteCode( call _init()+184 CE907D0 ? 2 ? EC701CC ? 0 ? )+107130 ? 0 ? TclCompEvalObj()+15 call TclExecuteByteCode( CE907D0 ? 0 ? B7F2F0AB ? 2 )B7F565F0 ? A7DB7010 ? 87D6 ? TclObjInterpProc()+ call B7EBE8E0 CE907D0 ? ABF19158 ? 645120C4260 ?
Re: [AOLSERVER] Question on two AOLserver tickets
How about having the proc enable only if debug settings are turned on on AOLserver? On May 15, 6:08 am, Jim Davidson jgdavid...@mac.com wrote: Good idea. Maybe it would make sense to disable it by default with some config flag to enable? Jim Sent from my iPhone On May 14, 2009, at 4:49 PM, Tom Jackson t...@rmadilo.com wrote: Maybe calling the API should result in a ns_log Warning to indicate a potential crash. tom jackson On Thu, 2009-05-14 at 13:26 -0700, Jade Rubick wrote: I'm just happy we figured it out. We were using this call: set connections [ns_server active] But it wasn't in a scheduled proc, so I just moved it behind a password protection section, and put a warning around it. We seldom (never) used that page anyway. I think a bot may have found it or something. Jade Jade Rubick Director of Development TRUiST 120 Wall Street, 4th Floor New York, NY 10005 USA jrub...@truist.com +1 503 285 4963 +1 707 671 1333 fax www.truist.com The information contained in this email/document is confidential and may be legally privileged. Access to this email/document by anyone other than the intended recipient(s) is unauthorized. If you are not an intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance to it, is prohibited. On May 14, 2009, at 12:33 PM, Jim Davidson wrote: Yup -- should really have been documented better -- sorry about that. Anyway, what is the monitoring attempting to dig up? There may some other safe ways to get the same. -Jim On May 14, 2009, at 2:04 PM, Jade Rubick wrote: Ironically, we have some monitoring code that does use that functionality. So our monitoring is killing our servers. Nice! I'm removing that code now. Jade Rubick Director of Development TRUiST 120 Wall Street, 4th Floor New York, NY USA jrub...@truist.com +1 503 285 4963 +1 707 671 1333 fax www.truist.com The information contained in this email/document is confidential and may be legally privileged. Access to this mail/document by anyone other than the intended recipient(s) is unauthorized. If you are not an intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance to it, is prohibited. On Thu, May 14, 2009 at 10:19 AM, Jim Davidson jgdavid...@mac.com wrote: Hi, Do you have some sort of background job that calls ns_server active (or similar) regularly? That could lead to random crashes. The description in http://dev.aolserver.com/trac/ticket/152is accurate: The code, by design, is not strictly safe as it's assumed to only be used interactively and occasionally as part of debugging and performance monitoring/optimization. To make it safe would require adding mutex locks around areas that are assumed read-only and/or single-threaded which could possibly lead to lock contention. I can't say it those assumptions have ever been proven true or false but that was my thinking when the code was first written. -Jim On May 14, 2009, at 4:16 AM, Sep Ng wrote: Hi, I'm trying to debug an AOLserver crash and the point of crash seems to be AppendConn in NS_GetProcInfo... I will post the stack trace after just for reference. Looking through the ticket tracker on AOLserver, I found two tickets of particular interest: http://dev.aolserver.com/trac/ticket/325 -- My question with this ticket is was it ever resolved? and the second ticket: http://dev.aolserver.com/trac/ticket/152 -- This problem should only happen if the command ns_server was called in a multi-threaded environment, right? Here is the call stack trace I'm working with. I'm more interested in Ticket #325 as it may be related to my problem. - Call Stack Trace - calling call entry argument values in hex location type point (? means dubious value) kpedbg_dmp_stack()+ call B5B81884 B45FFB74 ? 0 ? 219 kpeDbgCrash()+72 call B5B75E14 0 ? 5 ? 0 ? 80BD810 ? B45FFC08 ? B45FFBF0 ? kpeDbgSignalHandler call B5B867B4
Re: [AOLSERVER] TLS 1.6 and Aolserver
Hi Jeff, Thank you for the link to the bug tracker ticket. Perhaps I can suggest to Jade to go with disabling DH. Thank you so much for your time. On May 8, 6:25 am, Jeff Hobbs je...@activestate.com wrote: If Diffie Helmann is the only real issue here, maybe this should also be considered and DH removed by default configure. https://sourceforge.net/tracker/?func=detailaid=1811445group_id=132... Jeff On 06/05/2009 6:45 PM, Sep Ng wrote: Hi Jeff, I'm going to review options on how to progress with this problem with Jade. I've traced and stepped into TlsInit, and CtxInit functions and as far as I can see, the mutex functions we wrote seems to be working. I wonder if there is some influence by aolserver or what not. I don't know. It also seems that allow_customize in CRYPTO_set_mem_functions is getting set to zero for some reason. I'm not totally sure why that is happening. At this moment, I don't know what to do. On May 7, 7:14 am, Jack Schmidt thejackschm...@gmail.com wrote: Hi, Sep here. I just tried by disabling nsopenssl and it crashes at the same point. I suppose this is definitely more related to using aolserver with tls. I've included a backtrace and it shows the same point of failure. We use aolserver 4.0.10, though I'm not sure how relevant it is to the discussion. I'll try to check the startup routine of aolserver and see if I can find anything. 2009/5/7 Jeff Hobbs je...@activestate.com Is it possible that both nsopenssl and tls are in use, and that they both might be initializing openssl in the same process? I'm not sure if this would be a support case if so. On 05/05/2009 6:16 PM, Sep Ng wrote: Hi Jeff, I took a closer look at the patch you posted. It seems that the CRYPTO_set_mem_functions is not succeeding. The default memory functions that CRYPTO uses are malloc, realloc, and free but from the back trace, it looks like ns_malloc, ns_realloc and ns_free are the ones being used for some reason. I think I'm running out of ideas here. It's unclear why CRYPTO_set_mem_function would return 0 instead of 1, unless it's some bug in my OpenSSL package in Ubuntu. On May 6, 8:42 am, Jack Schmidt thejackschm...@gmail.com wrote: I've just yanked the debug. This includes the backtrace and memory frame info and the local info for most of the frames up until #11 CTX_Init. As before, the crash happens when DH_free is called. 2009/5/6 Jeff Hobbs je...@activestate.com Of the presented patches, I didn't find one that seemed to actually work, so I wrote one based on those presented. It is attached. Please test it in your environments. I have tested that it passes the basic tls test suite against a threaded Tcl 8.5.7 core build on Linux-x64 (and verified that OPENSSL_THREADS was true for this install). This patch is against tls 1.6 head. Jeff On 05/05/2009 3:42 PM, Sep Ng wrote: I'll try it. I didn't give it much thought at first but looking at it again, I think it might prevent the long string of ns_free and other calls to free memory after DH_free. On May 6, 3:43 am, Jeff Hobbs je...@activestate.com wrote: Just starting to look at this, but from the nsopenssl.c I saw another interesting function not used by TLS: if (CRYPTO_set_mem_functions(ns_malloc, ns_realloc, ns_free) == 0) ... We could do the same and point to Tcl_Alloc, Tcl_Realloc and Tcl_Free. I'm not sure they are necessary, and CRYPTO_set_mem_debug_functions isn't used, but it might help debug. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- A scrum a day keeps the pigs at bay -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. bt-without-nsopenssl 17KViewDownload -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: [AOLSERVER] TLS 1.6 and Aolserver
I think I understand where you're going with that. But, I don't think OpenSSL checks if it has been set already. int CRYPTO_set_mem_functions(void *(*m)(size_t), void *(*r)(void *, size_t), void (*f)(void *)) { if (!allow_customize) return 0; if ((m == 0) || (r == 0) || (f == 0)) return 0; malloc_func=m; malloc_ex_func=default_malloc_ex; realloc_func=r; realloc_ex_func=default_realloc_ex; free_func=f; malloc_locked_func=m; malloc_locked_ex_func=default_malloc_locked_ex; free_locked_func=f; return 1; } I know that allow_customize is always 1 and that the only other check that happens is if one of the three functions in the CRYPTO_set_mem_functions parameters was left at zero (NULL?). I do suppose it is worth a try to test aolserver without nsopenssl, if the results differ. It's equally strange that setting -DNO_DH=1 would allow the entire setup (aolserver+nsopenssl+tls) to function without crashing. I will try and test aolserver without ssl and post back my results. On May 7, 12:13 am, Jeff Hobbs je...@activestate.com wrote: Is it possible that both nsopenssl and tls are in use, and that they both might be initializing openssl in the same process? I'm not sure if this would be a support case if so. On 05/05/2009 6:16 PM, Sep Ng wrote: Hi Jeff, I took a closer look at the patch you posted. It seems that the CRYPTO_set_mem_functions is not succeeding. The default memory functions that CRYPTO uses are malloc, realloc, and free but from the back trace, it looks like ns_malloc, ns_realloc and ns_free are the ones being used for some reason. I think I'm running out of ideas here. It's unclear why CRYPTO_set_mem_function would return 0 instead of 1, unless it's some bug in my OpenSSL package in Ubuntu. On May 6, 8:42 am, Jack Schmidt thejackschm...@gmail.com wrote: I've just yanked the debug. This includes the backtrace and memory frame info and the local info for most of the frames up until #11 CTX_Init. As before, the crash happens when DH_free is called. 2009/5/6 Jeff Hobbs je...@activestate.com Of the presented patches, I didn't find one that seemed to actually work, so I wrote one based on those presented. It is attached. Please test it in your environments. I have tested that it passes the basic tls test suite against a threaded Tcl 8.5.7 core build on Linux-x64 (and verified that OPENSSL_THREADS was true for this install). This patch is against tls 1.6 head. Jeff On 05/05/2009 3:42 PM, Sep Ng wrote: I'll try it. I didn't give it much thought at first but looking at it again, I think it might prevent the long string of ns_free and other calls to free memory after DH_free. On May 6, 3:43 am, Jeff Hobbs je...@activestate.com wrote: Just starting to look at this, but from the nsopenssl.c I saw another interesting function not used by TLS: if (CRYPTO_set_mem_functions(ns_malloc, ns_realloc, ns_free) == 0) ... We could do the same and point to Tcl_Alloc, Tcl_Realloc and Tcl_Free. I'm not sure they are necessary, and CRYPTO_set_mem_debug_functions isn't used, but it might help debug. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: [AOLSERVER] TLS 1.6 and Aolserver
Hi Jeff, I'm going to review options on how to progress with this problem with Jade. I've traced and stepped into TlsInit, and CtxInit functions and as far as I can see, the mutex functions we wrote seems to be working. I wonder if there is some influence by aolserver or what not. I don't know. It also seems that allow_customize in CRYPTO_set_mem_functions is getting set to zero for some reason. I'm not totally sure why that is happening. At this moment, I don't know what to do. On May 7, 7:14 am, Jack Schmidt thejackschm...@gmail.com wrote: Hi, Sep here. I just tried by disabling nsopenssl and it crashes at the same point. I suppose this is definitely more related to using aolserver with tls. I've included a backtrace and it shows the same point of failure. We use aolserver 4.0.10, though I'm not sure how relevant it is to the discussion. I'll try to check the startup routine of aolserver and see if I can find anything. 2009/5/7 Jeff Hobbs je...@activestate.com Is it possible that both nsopenssl and tls are in use, and that they both might be initializing openssl in the same process? I'm not sure if this would be a support case if so. On 05/05/2009 6:16 PM, Sep Ng wrote: Hi Jeff, I took a closer look at the patch you posted. It seems that the CRYPTO_set_mem_functions is not succeeding. The default memory functions that CRYPTO uses are malloc, realloc, and free but from the back trace, it looks like ns_malloc, ns_realloc and ns_free are the ones being used for some reason. I think I'm running out of ideas here. It's unclear why CRYPTO_set_mem_function would return 0 instead of 1, unless it's some bug in my OpenSSL package in Ubuntu. On May 6, 8:42 am, Jack Schmidt thejackschm...@gmail.com wrote: I've just yanked the debug. This includes the backtrace and memory frame info and the local info for most of the frames up until #11 CTX_Init. As before, the crash happens when DH_free is called. 2009/5/6 Jeff Hobbs je...@activestate.com Of the presented patches, I didn't find one that seemed to actually work, so I wrote one based on those presented. It is attached. Please test it in your environments. I have tested that it passes the basic tls test suite against a threaded Tcl 8.5.7 core build on Linux-x64 (and verified that OPENSSL_THREADS was true for this install). This patch is against tls 1.6 head. Jeff On 05/05/2009 3:42 PM, Sep Ng wrote: I'll try it. I didn't give it much thought at first but looking at it again, I think it might prevent the long string of ns_free and other calls to free memory after DH_free. On May 6, 3:43 am, Jeff Hobbs je...@activestate.com wrote: Just starting to look at this, but from the nsopenssl.c I saw another interesting function not used by TLS: if (CRYPTO_set_mem_functions(ns_malloc, ns_realloc, ns_free) == 0) ... We could do the same and point to Tcl_Alloc, Tcl_Realloc and Tcl_Free. I'm not sure they are necessary, and CRYPTO_set_mem_debug_functions isn't used, but it might help debug. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- A scrum a day keeps the pigs at bay -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. bt-without-nsopenssl 17KViewDownload -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: [AOLSERVER] TLS 1.6 and Aolserver
As a quick update. I'm using TCL threads to do this. The CRYPTO_* functions are, unsurprisingly used as defaults for openssl. What was happening in my code was that if I set CRYPTO_lock to be the lock callback function, it would end up in an infinite loop. The problem so far has been that as far as I understand, OpenSSL specifies that the functions must support n mutex locks where n is extracted from CRYPTO_num_locks(). I'll see if it is possible to do this with TCL_DECLARE_MUTEX. I appreciate all the info thus far. :) On May 5, 1:06 pm, Sep Ng thejackschm...@gmail.com wrote: Thanks for the link Jeff. I'm still trying to figure out if it's possible to use CRYPTO_* for the mutex though I agree 100% with you that if it had been the case, they wouldn't have a need for explicitly defining the function. As of now, using the CRYPTO_lock functions seem to yield some sort of deadlock where all the threads stop at some point until the SIGSEGV signal is emitted. The backtrace looks funny so I'll look at TCL threads after I exhaust this option. On May 5, 11:15 am, Jeff Hobbs je...@activestate.com wrote: Taking a quick look, that does appear to be perfectly matched to the callback that they want. Of course, if that is the case I wonder why they say this must be set, rather than making it optional. Otherwise you have Tcl_MutexLock and the related functions mentioned at: http://www.tcl.tk/man/tcl8.5/TclLib/Thread.htm For CRYPTO_set_id_callback, it looks like Tcl_GetCurrentThread is the right callback. On 4-May-09, at 7:43 PM, Sep Ng wrote: I'm working on this on behalf of Jade and I'd like to get some info on adding thread safe code on TLS. As noted by Andrew (huge thanks!), TLS does not set the CRYPTO_set_locking_callback and CRYPTO_set_id_callback callback functions which would be required to have TLS thread safe code. I was working on possibly using pthreads for mutex, but came across that OpenSSL does indeed come with Thread support. I would suppose it would be as straight forward as using the mutex designed by OpenSSL (e.g. CRYPTO_lock, etc.). I'd like to know if I'm heading to the right direction with this. I've begun reading up on OpenSSL's threads API but would appreciate any help. Thank you very much! On May 5, 6:45 am, Jade Rubick jrub...@truist.com wrote: Thank you, Andrew. We'll look into that. J Jade Rubick Director of Development TRUiST 120 Wall Street, 4th Floor New York, NY USA jrub...@truist.com +1 503 285 4963 +1 707 671 1333 fax www.truist.com The information contained in this email/document is confidential and may be legally privileged. Access to this mail/document by anyone other than the intended recipient(s) is unauthorized. If you are not an intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance to it, is prohibited. On Fri, May 1, 2009 at 12:42 PM, Andrew Steets ste...@gmail.com wrote: It's not a matter of compiling OpenSSL to be thread safe. Someone needs to update the TLS C code to call the right OpenSSL API functions on module initialization. In it's current state I don't see how the TLS module can safely call OpenSSL from a threaded context. From the Openssl docs: http://openssl.org/docs/crypto/threads.html#DESCRIPTION OpenSSL can safely be used in multi-threaded applications provided that at least two callback functions are set, locking_function and threadid_func. The TLS C code doesn't setup either one of those callbacks, so that's a problem. I'm not sure if that is your problem specifically but it would be a good place to start. -Andrew On Fri, May 1, 2009 at 12:59 PM, Jade Rubick jrub...@truist.com wrote: Jade Rubick Director of Development TRUiST 120 Wall Street, 4th Floor New York, NY USA jrub...@truist.com +1 503 285 4963 +1 707 671 1333 fax www.truist.com The information contained in this email/document is confidential and may be legally privileged. Access to this mail/document by anyone other than the intended recipient(s) is unauthorized. If you are not an intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance to it, is prohibited. -- Forwarded message -- From: Jack Schmidt thejackschm...@gmail.com Date: Thu, Apr 30, 2009 at 4:03 PM Subject: Re: [AOLSERVER] TLS 1.6 and Aolserver To: Jade Rubick jrub...@truist.com Cc: tech t...@volunteersolutions.org I just tried it by recompiling openssl with threads as compiler option and it produces the same problem. Maybe there's another way of making openssl thread safe. Not sure as of the moment. 2009/5/1 Jack Schmidt thejackschm...@gmail.com It's certainly a possibility. Since I'm
Re: [AOLSERVER] Fwd: [AOLSERVER] TLS 1.6 and Aolserver
I certainly hope I'm not spamming but this is likely to be the last update of the day (at least on my end). I wrote a pretty sketchy (and lots of bad programming) but I think I *might* have gotten the mutex initialization done. Unfortunately, on tests, it falls on a segmentation fault on exactly the same spot (DH_free, etc.). Here is the diff right now: --- tls.c.back 2009-05-05 10:06:59.0 +0800 +++ tls.c 2009-05-05 15:41:16.0 +0800 @@ -130,6 +130,58 @@ #define sk_SSL_CIPHER_value( sk, index)(SSL_CIPHER*)sk_value((sk), (index)) #endif +/* + * Thread-Safe TLS Code + */ + +#define OPENSSL_THREAD_DEFINES +#include openssl/opensslconf.h + +#if defined(OPENSSL_THREADS) + +#include openssl/crypto.h + +/* + * This is likely to be nasty coding practices + * Based from /crypto/cryptlib.c of OpenSSL + * Also based on NSOpenSSL + * + */ + +Tcl_Mutex locks[CRYPTO_NUM_LOCKS]; +size_t num_locks; + +static void CryptoThreadLockCallback(int mode, int n, const char *file, int line); +static unsigned long CryptoThreadIdCallback(void); + +static void +CryptoThreadLockCallback(int mode, int n, const char *file, int line) +{ +if (n = num_locks) +{ +n = num_locks - 1; +} + +if (mode CRYPTO_LOCK) +{ + Tcl_MutexLock(locks[n]); + //Tcl_MutexLock(locks); +} +else +{ + Tcl_MutexUnlock(locks[n]); + //Tcl_MutexUnlock(locks); +} +} + +static unsigned long +CryptoThreadIdCallback(void) +{ +return (unsigned long) Tcl_GetCurrentThread(); +} + +#endif + /* *--- @@ -1500,6 +1552,22 @@ channelTypeVersion = TLS_CHANNEL_VERSION_1; } +#if defined(OPENSSL_THREADS) + +num_locks = CRYPTO_num_locks(); +int lock = 0; +for (lock = 0; lock num_locks; lock++) +{ + TCL_DECLARE_MUTEX(mutex); + locks[lock]=mutex; +} + +CRYPTO_set_locking_callback(CryptoThreadLockCallback); +CRYPTO_set_id_callback(CryptoThreadIdCallback); + +#endif + + if (SSL_library_init() != 1) { Tcl_AppendResult(interp, could not initialize SSL library, NULL); return TCL_ERROR; We cannot use CRYPTO_lock because CRYPTO_lock calls the function we set with CRYPTO_set_locking_callback, so this is completely out of the picture. I agree with Jeff that TCL threads and mutex is the right way to handle this but I'm wondering if the code I wrote has some incorrect implementation, which is leading to still the same crash happening. Minor point is that I have yet to find a place to run Tcl_Finalize_mutex so we can unload the mutex. Should help prevent memory leaks, I think. I will e-mail Jade and Jeff the backtrace as I think it will only muck up the discussion. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: [AOLSERVER] TLS 1.6 and Aolserver
I'll try it. I didn't give it much thought at first but looking at it again, I think it might prevent the long string of ns_free and other calls to free memory after DH_free. On May 6, 3:43 am, Jeff Hobbs je...@activestate.com wrote: Just starting to look at this, but from the nsopenssl.c I saw another interesting function not used by TLS: if (CRYPTO_set_mem_functions(ns_malloc, ns_realloc, ns_free) == 0) ... We could do the same and point to Tcl_Alloc, Tcl_Realloc and Tcl_Free. I'm not sure they are necessary, and CRYPTO_set_mem_debug_functions isn't used, but it might help debug. On 05/05/2009 12:55 AM, Sep Ng wrote: I certainly hope I'm not spamming but this is likely to be the last update of the day (at least on my end). I wrote a pretty sketchy (and lots of bad programming) but I think I *might* have gotten the mutex initialization done. Unfortunately, on tests, it falls on a segmentation fault on exactly the same spot (DH_free, etc.). Here is the diff right now: --- tls.c.back 2009-05-05 10:06:59.0 +0800 +++ tls.c 2009-05-05 15:41:16.0 +0800 @@ -130,6 +130,58 @@ #define sk_SSL_CIPHER_value( sk, index) (SSL_CIPHER*)sk_value((sk), (index)) #endif +/* + * Thread-Safe TLS Code + */ + +#define OPENSSL_THREAD_DEFINES +#include openssl/opensslconf.h + +#if defined(OPENSSL_THREADS) + +#include openssl/crypto.h + +/* + * This is likely to be nasty coding practices + * Based from /crypto/cryptlib.c of OpenSSL + * Also based on NSOpenSSL + * + */ + +Tcl_Mutex locks[CRYPTO_NUM_LOCKS]; +size_t num_locks; + +static void CryptoThreadLockCallback(int mode, int n, const char *file, int line); +static unsigned long CryptoThreadIdCallback(void); + +static void +CryptoThreadLockCallback(int mode, int n, const char *file, int line) +{ +if (n = num_locks) +{ +n = num_locks - 1; +} + +if (mode CRYPTO_LOCK) +{ + Tcl_MutexLock(locks[n]); + //Tcl_MutexLock(locks); +} +else +{ + Tcl_MutexUnlock(locks[n]); + //Tcl_MutexUnlock(locks); +} +} + +static unsigned long +CryptoThreadIdCallback(void) +{ +return (unsigned long) Tcl_GetCurrentThread(); +} + +#endif + /* *--- @@ -1500,6 +1552,22 @@ channelTypeVersion = TLS_CHANNEL_VERSION_1; } +#if defined(OPENSSL_THREADS) + +num_locks = CRYPTO_num_locks(); +int lock = 0; +for (lock = 0; lock num_locks; lock++) +{ + TCL_DECLARE_MUTEX(mutex); + locks[lock]=mutex; +} + +CRYPTO_set_locking_callback(CryptoThreadLockCallback); +CRYPTO_set_id_callback(CryptoThreadIdCallback); + +#endif + + if (SSL_library_init() != 1) { Tcl_AppendResult(interp, could not initialize SSL library, NULL); return TCL_ERROR; We cannot use CRYPTO_lock because CRYPTO_lock calls the function we set with CRYPTO_set_locking_callback, so this is completely out of the picture. I agree with Jeff that TCL threads and mutex is the right way to handle this but I'm wondering if the code I wrote has some incorrect implementation, which is leading to still the same crash happening. Minor point is that I have yet to find a place to run Tcl_Finalize_mutex so we can unload the mutex. Should help prevent memory leaks, I think. I will e-mail Jade and Jeff the backtrace as I think it will only muck up the discussion. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: [AOLSERVER] TLS 1.6 and Aolserver
It seems that Tcl's Tcl_Alloc, Tcl_Realloc, and Tcl_Free are defined differently than the ones expected by CRYPTO_set_mem_functions (the functions are expected to be of void*). I tried to put a wrapper around it but I haven't come across much success in that. I will see if there are other avenues to this, maybe even using the standard malloc might do. On May 6, 6:42 am, Sep Ng thejackschm...@gmail.com wrote: I'll try it. I didn't give it much thought at first but looking at it again, I think it might prevent the long string of ns_free and other calls to free memory after DH_free. On May 6, 3:43 am, Jeff Hobbs je...@activestate.com wrote: Just starting to look at this, but from the nsopenssl.c I saw another interesting function not used by TLS: if (CRYPTO_set_mem_functions(ns_malloc, ns_realloc, ns_free) == 0) ... We could do the same and point to Tcl_Alloc, Tcl_Realloc and Tcl_Free. I'm not sure they are necessary, and CRYPTO_set_mem_debug_functions isn't used, but it might help debug. On 05/05/2009 12:55 AM, Sep Ng wrote: I certainly hope I'm not spamming but this is likely to be the last update of the day (at least on my end). I wrote a pretty sketchy (and lots of bad programming) but I think I *might* have gotten the mutex initialization done. Unfortunately, on tests, it falls on a segmentation fault on exactly the same spot (DH_free, etc.). Here is the diff right now: --- tls.c.back 2009-05-05 10:06:59.0 +0800 +++ tls.c 2009-05-05 15:41:16.0 +0800 @@ -130,6 +130,58 @@ #define sk_SSL_CIPHER_value( sk, index) (SSL_CIPHER*)sk_value((sk), (index)) #endif +/* + * Thread-Safe TLS Code + */ + +#define OPENSSL_THREAD_DEFINES +#include openssl/opensslconf.h + +#if defined(OPENSSL_THREADS) + +#include openssl/crypto.h + +/* + * This is likely to be nasty coding practices + * Based from /crypto/cryptlib.c of OpenSSL + * Also based on NSOpenSSL + * + */ + +Tcl_Mutex locks[CRYPTO_NUM_LOCKS]; +size_t num_locks; + +static void CryptoThreadLockCallback(int mode, int n, const char *file, int line); +static unsigned long CryptoThreadIdCallback(void); + +static void +CryptoThreadLockCallback(int mode, int n, const char *file, int line) +{ +if (n = num_locks) +{ +n = num_locks - 1; +} + +if (mode CRYPTO_LOCK) +{ + Tcl_MutexLock(locks[n]); + //Tcl_MutexLock(locks); +} +else +{ + Tcl_MutexUnlock(locks[n]); + //Tcl_MutexUnlock(locks); +} +} + +static unsigned long +CryptoThreadIdCallback(void) +{ +return (unsigned long) Tcl_GetCurrentThread(); +} + +#endif + /* *--- @@ -1500,6 +1552,22 @@ channelTypeVersion = TLS_CHANNEL_VERSION_1; } +#if defined(OPENSSL_THREADS) + +num_locks = CRYPTO_num_locks(); +int lock = 0; +for (lock = 0; lock num_locks; lock++) +{ + TCL_DECLARE_MUTEX(mutex); + locks[lock]=mutex; +} + +CRYPTO_set_locking_callback(CryptoThreadLockCallback); +CRYPTO_set_id_callback(CryptoThreadIdCallback); + +#endif + + if (SSL_library_init() != 1) { Tcl_AppendResult(interp, could not initialize SSL library, NULL); return TCL_ERROR; We cannot use CRYPTO_lock because CRYPTO_lock calls the function we set with CRYPTO_set_locking_callback, so this is completely out of the picture. I agree with Jeff that TCL threads and mutex is the right way to handle this but I'm wondering if the code I wrote has some incorrect implementation, which is leading to still the same crash happening. Minor point is that I have yet to find a place to run Tcl_Finalize_mutex so we can unload the mutex. Should help prevent memory leaks, I think. I will e-mail Jade and Jeff the backtrace as I think it will only muck up the discussion. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body
Re: [AOLSERVER] Fwd: [AOLSERVER] TLS 1.6 and Aolserver
I just tried your patch, Jeff. It still crashes. I'll try to get a backtrace right now. On May 6, 7:53 am, Jeff Hobbs je...@activestate.com wrote: Of the presented patches, I didn't find one that seemed to actually work, so I wrote one based on those presented. It is attached. Please test it in your environments. I have tested that it passes the basic tls test suite against a threaded Tcl 8.5.7 core build on Linux-x64 (and verified that OPENSSL_THREADS was true for this install). This patch is against tls 1.6 head. Jeff On 05/05/2009 3:42 PM, Sep Ng wrote: I'll try it. I didn't give it much thought at first but looking at it again, I think it might prevent the long string of ns_free and other calls to free memory after DH_free. On May 6, 3:43 am, Jeff Hobbs je...@activestate.com wrote: Just starting to look at this, but from the nsopenssl.c I saw another interesting function not used by TLS: if (CRYPTO_set_mem_functions(ns_malloc, ns_realloc, ns_free) == 0) ... We could do the same and point to Tcl_Alloc, Tcl_Realloc and Tcl_Free. I'm not sure they are necessary, and CRYPTO_set_mem_debug_functions isn't used, but it might help debug. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. [tls-thread.diff3K ]? announce.txt ? tls-thread.diff ? tls-verify.diff Index: Makefile.in === RCS file: /cvsroot/tls/tls/Makefile.in,v retrieving revision 1.27 diff -u -r1.27 Makefile.in --- Makefile.in 19 Mar 2008 22:57:03 - 1.27 +++ Makefile.in 5 May 2009 23:52:38 - @@ -205,7 +205,7 @@ file copy [file join $(srcdir) tls.tcl] tls.tcl \ } ;\ source [file join $(srcdir) tls.tcl]; \ - set argv $(TESTFLAGS); \ + set argv {$(TESTFLAGS)}; \ source [file join $(srcdir) tests all.tcl] | $(TCLSH) shell: binaries libraries Index: tls.c === RCS file: /cvsroot/tls/tls/tls.c,v retrieving revision 1.30 diff -u -r1.30 tls.c --- tls.c 19 Mar 2008 22:06:13 - 1.30 +++ tls.c 5 May 2009 23:52:38 - @@ -130,6 +130,46 @@ #define sk_SSL_CIPHER_value( sk, index)(SSL_CIPHER*)sk_value((sk), (index)) #endif +/* + * Thread-Safe TLS Code + */ + +#ifdef TCL_THREADS +#define OPENSSL_THREAD_DEFINES +#include openssl/opensslconf.h + +#ifdef OPENSSL_THREADS +#include openssl/crypto.h + +/* + * Threaded operation requires locking callbacks + * Based from /crypto/cryptlib.c of OpenSSL and NSOpenSSL. + */ + +static Tcl_Mutex locks[CRYPTO_NUM_LOCKS]; + +static void CryptoThreadLockCallback(int mode, int n, + const char *file, int line); +static unsigned long CryptoThreadIdCallback(void); + +static void +CryptoThreadLockCallback(int mode, int n, const char *file, int line) +{ +if (mode CRYPTO_LOCK) { + Tcl_MutexLock(locks[n]); +} else { + Tcl_MutexUnlock(locks[n]); +} +} + +static unsigned long +CryptoThreadIdCallback(void) +{ +return (unsigned long) Tcl_GetCurrentThread(); +} +#endif /* OPENSSL_THREADS */ +#endif /* TCL_THREADS */ + /* *--- @@ -1468,6 +1508,9 @@ { int major, minor, patchlevel, release, i; char rnd_seed[16] = GrzSlplKqUdnnzP!; /* 16 bytes */ +#if defined(OPENSSL_THREADS) defined(TCL_THREADS) +size_t num_locks; +#endif /* * The original 8.2.0 stacked channel implementation (and the patch @@ -1500,6 +1543,24 @@ channelTypeVersion = TLS_CHANNEL_VERSION_1; } +if (CRYPTO_set_mem_functions((void *(*)(size_t))Tcl_Alloc, + (void *(*)(void *, size_t))Tcl_Realloc, + (void(*)(void *))Tcl_Free) == 0) { + /* Not using Tcl's mem functions ... not critical */ +} + +#if defined(OPENSSL_THREADS) defined(TCL_THREADS) +/* should we consider allocating mutexes? */ +num_locks = CRYPTO_num_locks(); +if (num_locks CRYPTO_NUM_LOCKS) { + Tcl_AppendResult(interp, crypto num locks size error, NULL); + return TCL_ERROR; +} + +CRYPTO_set_locking_callback(CryptoThreadLockCallback); +CRYPTO_set_id_callback(CryptoThreadIdCallback); +#endif + if (SSL_library_init() != 1) { Tcl_AppendResult(interp, could not initialize SSL library, NULL); return TCL_ERROR; -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself
Re: [AOLSERVER] Fwd: [AOLSERVER] TLS 1.6 and Aolserver
Hi Jeff, I took a closer look at the patch you posted. It seems that the CRYPTO_set_mem_functions is not succeeding. The default memory functions that CRYPTO uses are malloc, realloc, and free but from the back trace, it looks like ns_malloc, ns_realloc and ns_free are the ones being used for some reason. I think I'm running out of ideas here. It's unclear why CRYPTO_set_mem_function would return 0 instead of 1, unless it's some bug in my OpenSSL package in Ubuntu. On May 6, 8:42 am, Jack Schmidt thejackschm...@gmail.com wrote: I've just yanked the debug. This includes the backtrace and memory frame info and the local info for most of the frames up until #11 CTX_Init. As before, the crash happens when DH_free is called. 2009/5/6 Jeff Hobbs je...@activestate.com Of the presented patches, I didn't find one that seemed to actually work, so I wrote one based on those presented. It is attached. Please test it in your environments. I have tested that it passes the basic tls test suite against a threaded Tcl 8.5.7 core build on Linux-x64 (and verified that OPENSSL_THREADS was true for this install). This patch is against tls 1.6 head. Jeff On 05/05/2009 3:42 PM, Sep Ng wrote: I'll try it. I didn't give it much thought at first but looking at it again, I think it might prevent the long string of ns_free and other calls to free memory after DH_free. On May 6, 3:43 am, Jeff Hobbs je...@activestate.com wrote: Just starting to look at this, but from the nsopenssl.c I saw another interesting function not used by TLS: if (CRYPTO_set_mem_functions(ns_malloc, ns_realloc, ns_free) == 0) ... We could do the same and point to Tcl_Alloc, Tcl_Realloc and Tcl_Free. I'm not sure they are necessary, and CRYPTO_set_mem_debug_functions isn't used, but it might help debug. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. ? announce.txt ? tls-thread.diff ? tls-verify.diff Index: Makefile.in === RCS file: /cvsroot/tls/tls/Makefile.in,v retrieving revision 1.27 diff -u -r1.27 Makefile.in --- Makefile.in 19 Mar 2008 22:57:03 - 1.27 +++ Makefile.in 5 May 2009 23:52:38 - @@ -205,7 +205,7 @@ file copy [file join $(srcdir) tls.tcl] tls.tcl \ } ;\ source [file join $(srcdir) tls.tcl]; \ - set argv $(TESTFLAGS); \ + set argv {$(TESTFLAGS)}; \ source [file join $(srcdir) tests all.tcl] | $(TCLSH) shell: binaries libraries Index: tls.c === RCS file: /cvsroot/tls/tls/tls.c,v retrieving revision 1.30 diff -u -r1.30 tls.c --- tls.c 19 Mar 2008 22:06:13 - 1.30 +++ tls.c 5 May 2009 23:52:38 - @@ -130,6 +130,46 @@ #define sk_SSL_CIPHER_value( sk, index)(SSL_CIPHER*)sk_value((sk), (index)) #endif +/* + * Thread-Safe TLS Code + */ + +#ifdef TCL_THREADS +#define OPENSSL_THREAD_DEFINES +#include openssl/opensslconf.h + +#ifdef OPENSSL_THREADS +#include openssl/crypto.h + +/* + * Threaded operation requires locking callbacks + * Based from /crypto/cryptlib.c of OpenSSL and NSOpenSSL. + */ + +static Tcl_Mutex locks[CRYPTO_NUM_LOCKS]; + +static void CryptoThreadLockCallback(int mode, int n, + const char *file, int line); +static unsigned long CryptoThreadIdCallback(void); + +static void +CryptoThreadLockCallback(int mode, int n, const char *file, int line) +{ +if (mode CRYPTO_LOCK) { + Tcl_MutexLock(locks[n]); +} else { + Tcl_MutexUnlock(locks[n]); +} +} + +static unsigned long +CryptoThreadIdCallback(void) +{ +return (unsigned long) Tcl_GetCurrentThread(); +} +#endif /* OPENSSL_THREADS */ +#endif /* TCL_THREADS */ + /* *--- @@ -1468,6 +1508,9 @@ { int major, minor, patchlevel, release, i; char rnd_seed[16] = GrzSlplKqUdnnzP!;/* 16 bytes */ +#if defined(OPENSSL_THREADS) defined(TCL_THREADS) +size_t num_locks; +#endif /* * The original 8.2.0 stacked channel implementation (and the patch @@ -1500,6 +1543,24 @@ channelTypeVersion = TLS_CHANNEL_VERSION_1; } +if (CRYPTO_set_mem_functions((void *(*)(size_t))Tcl_Alloc, + (void *(*)(void *, size_t))Tcl_Realloc, + (void(*)(void *))Tcl_Free) == 0) { + /* Not using Tcl's mem functions ... not critical */ +} + +#if defined(OPENSSL_THREADS) defined(TCL_THREADS) +/* should we consider allocating mutexes? */ +num_locks = CRYPTO_num_locks
Re: [AOLSERVER] Fwd: [AOLSERVER] TLS 1.6 and Aolserver
I'm working on this on behalf of Jade and I'd like to get some info on adding thread safe code on TLS. As noted by Andrew (huge thanks!), TLS does not set the CRYPTO_set_locking_callback and CRYPTO_set_id_callback callback functions which would be required to have TLS thread safe code. I was working on possibly using pthreads for mutex, but came across that OpenSSL does indeed come with Thread support. I would suppose it would be as straight forward as using the mutex designed by OpenSSL (e.g. CRYPTO_lock, etc.). I'd like to know if I'm heading to the right direction with this. I've begun reading up on OpenSSL's threads API but would appreciate any help. Thank you very much! On May 5, 6:45 am, Jade Rubick jrub...@truist.com wrote: Thank you, Andrew. We'll look into that. J Jade Rubick Director of Development TRUiST 120 Wall Street, 4th Floor New York, NY USA jrub...@truist.com +1 503 285 4963 +1 707 671 1333 fax www.truist.com The information contained in this email/document is confidential and may be legally privileged. Access to this mail/document by anyone other than the intended recipient(s) is unauthorized. If you are not an intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance to it, is prohibited. On Fri, May 1, 2009 at 12:42 PM, Andrew Steets ste...@gmail.com wrote: It's not a matter of compiling OpenSSL to be thread safe. Someone needs to update the TLS C code to call the right OpenSSL API functions on module initialization. In it's current state I don't see how the TLS module can safely call OpenSSL from a threaded context. From the Openssl docs: http://openssl.org/docs/crypto/threads.html#DESCRIPTION OpenSSL can safely be used in multi-threaded applications provided that at least two callback functions are set, locking_function and threadid_func. The TLS C code doesn't setup either one of those callbacks, so that's a problem. I'm not sure if that is your problem specifically but it would be a good place to start. -Andrew On Fri, May 1, 2009 at 12:59 PM, Jade Rubick jrub...@truist.com wrote: Jade Rubick Director of Development TRUiST 120 Wall Street, 4th Floor New York, NY USA jrub...@truist.com +1 503 285 4963 +1 707 671 1333 fax www.truist.com The information contained in this email/document is confidential and may be legally privileged. Access to this mail/document by anyone other than the intended recipient(s) is unauthorized. If you are not an intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance to it, is prohibited. -- Forwarded message -- From: Jack Schmidt thejackschm...@gmail.com Date: Thu, Apr 30, 2009 at 4:03 PM Subject: Re: [AOLSERVER] TLS 1.6 and Aolserver To: Jade Rubick jrub...@truist.com Cc: tech t...@volunteersolutions.org I just tried it by recompiling openssl with threads as compiler option and it produces the same problem. Maybe there's another way of making openssl thread safe. Not sure as of the moment. 2009/5/1 Jack Schmidt thejackschm...@gmail.com It's certainly a possibility. Since I'm also trying to debianize openssl from Gutsy with debug symbols, we can easily slip in a threaded build. 2009/5/1 Jade Rubick jrub...@truist.com Maybe we didn't compile openssl to be threadsafe? J Jade Rubick Director of Development TRUiST 120 Wall Street, 4th Floor New York, NY 10005 USA jrub...@truist.com +1 503 285 4963 +1 707 671 1333 fax www.truist.com The information contained in this email/document is confidential and may be legally privileged. Access to this email/document by anyone other than the intended recipient(s) is unauthorized. If you are not an intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance to it, is prohibited. Begin forwarded message: From: Andrew Steets ste...@gmail.com Date: April 29, 2009 6:16:14 PM PDT To: aolser...@listserv.aol.com Subject: Re: [AOLSERVER] TLS 1.6 and Aolserver Reply-To: AOLserver Discussion aolser...@listserv.aol.com Hello, We don't use this TLS package at Wayport, but I have seen similar errors with OpenSSL before in other applications. I pulled the TLS code and glanced through it. It doesn't look like you have registered the locking callbacks for openssl, which means any openssl calls are not thread safe. That's going to be a problem inside aolserver :-) Check out InitOpenSSL() nsopenssl.c (in the nsopenssl module). It does all the basic stuff you need to get OpenSSL running in a thread-safe manor. Also: http://openssl.org/docs/crypto/threads.html If you 'info threads' and see other threads inside openssl crypto functions this is almost certainly your
Re: [AOLSERVER] Fwd: [AOLSERVER] TLS 1.6 and Aolserver
Thanks for the link Jeff. I'm still trying to figure out if it's possible to use CRYPTO_* for the mutex though I agree 100% with you that if it had been the case, they wouldn't have a need for explicitly defining the function. As of now, using the CRYPTO_lock functions seem to yield some sort of deadlock where all the threads stop at some point until the SIGSEGV signal is emitted. The backtrace looks funny so I'll look at TCL threads after I exhaust this option. On May 5, 11:15 am, Jeff Hobbs je...@activestate.com wrote: Taking a quick look, that does appear to be perfectly matched to the callback that they want. Of course, if that is the case I wonder why they say this must be set, rather than making it optional. Otherwise you have Tcl_MutexLock and the related functions mentioned at: http://www.tcl.tk/man/tcl8.5/TclLib/Thread.htm For CRYPTO_set_id_callback, it looks like Tcl_GetCurrentThread is the right callback. On 4-May-09, at 7:43 PM, Sep Ng wrote: I'm working on this on behalf of Jade and I'd like to get some info on adding thread safe code on TLS. As noted by Andrew (huge thanks!), TLS does not set the CRYPTO_set_locking_callback and CRYPTO_set_id_callback callback functions which would be required to have TLS thread safe code. I was working on possibly using pthreads for mutex, but came across that OpenSSL does indeed come with Thread support. I would suppose it would be as straight forward as using the mutex designed by OpenSSL (e.g. CRYPTO_lock, etc.). I'd like to know if I'm heading to the right direction with this. I've begun reading up on OpenSSL's threads API but would appreciate any help. Thank you very much! On May 5, 6:45 am, Jade Rubick jrub...@truist.com wrote: Thank you, Andrew. We'll look into that. J Jade Rubick Director of Development TRUiST 120 Wall Street, 4th Floor New York, NY USA jrub...@truist.com +1 503 285 4963 +1 707 671 1333 fax www.truist.com The information contained in this email/document is confidential and may be legally privileged. Access to this mail/document by anyone other than the intended recipient(s) is unauthorized. If you are not an intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance to it, is prohibited. On Fri, May 1, 2009 at 12:42 PM, Andrew Steets ste...@gmail.com wrote: It's not a matter of compiling OpenSSL to be thread safe. Someone needs to update the TLS C code to call the right OpenSSL API functions on module initialization. In it's current state I don't see how the TLS module can safely call OpenSSL from a threaded context. From the Openssl docs: http://openssl.org/docs/crypto/threads.html#DESCRIPTION OpenSSL can safely be used in multi-threaded applications provided that at least two callback functions are set, locking_function and threadid_func. The TLS C code doesn't setup either one of those callbacks, so that's a problem. I'm not sure if that is your problem specifically but it would be a good place to start. -Andrew On Fri, May 1, 2009 at 12:59 PM, Jade Rubick jrub...@truist.com wrote: Jade Rubick Director of Development TRUiST 120 Wall Street, 4th Floor New York, NY USA jrub...@truist.com +1 503 285 4963 +1 707 671 1333 fax www.truist.com The information contained in this email/document is confidential and may be legally privileged. Access to this mail/document by anyone other than the intended recipient(s) is unauthorized. If you are not an intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance to it, is prohibited. -- Forwarded message -- From: Jack Schmidt thejackschm...@gmail.com Date: Thu, Apr 30, 2009 at 4:03 PM Subject: Re: [AOLSERVER] TLS 1.6 and Aolserver To: Jade Rubick jrub...@truist.com Cc: tech t...@volunteersolutions.org I just tried it by recompiling openssl with threads as compiler option and it produces the same problem. Maybe there's another way of making openssl thread safe. Not sure as of the moment. 2009/5/1 Jack Schmidt thejackschm...@gmail.com It's certainly a possibility. Since I'm also trying to debianize openssl from Gutsy with debug symbols, we can easily slip in a threaded build. 2009/5/1 Jade Rubick jrub...@truist.com Maybe we didn't compile openssl to be threadsafe? J Jade Rubick Director of Development TRUiST 120 Wall Street, 4th Floor New York, NY 10005 USA jrub...@truist.com +1 503 285 4963 +1 707 671 1333 fax www.truist.com The information contained in this email/document is confidential and may be legally privileged. Access to this email/document by anyone other than the intended recipient(s) is unauthorized. If you are not an intended recipient, any disclosure, copying, distribution, or any
Re: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages?
Hi, Just an update on this issue. I've managed to get gzip compression running, but instead of using what is built-in aolserver 4, I found ns_returnz that was modified for aolserver 4 and got that loading. The bad news is that gzip compression detection is totally on the application level and not handled by aolserver. That's a bit weird, but I think it'll do for now. Thanks to everyone for helping out here. On May 1, 8:31 am, Sep Ng thejackschm...@gmail.com wrote: Quick update: Just recompiled AOLserver 4.0.10 with the gzip path and still no go. I suppose I will scout the bug tracker on sourceforge to see if I find anything interesting. I wonder what problems did OpenACS introduce into this gzip issue... On May 1, 5:25 am, Sep Ng thejackschm...@gmail.com wrote: I really appreciate the help all of you have given me. I will look into the gzip compression patch and try other approaches if that does not work. Thank you very much for all the help! I post an update on how it has turned out asap. On May 1, 12:20 am, Fenton, Brian brian.fen...@quest.ie wrote: Hi Sep as I said, I know nothing about it myself, but the link I gave you does mention usage of ns_adp_compress, so I thought it may be of some assistance. I don't know how much these things changed from 4.0.10. Malte Sussdorf wrote a how-to for OpenACS and nginxhttp://cognovis.de/developer/en/nginx-loadbalancing cheers Brian From: AOLserver Discussion [aolser...@listserv.aol.com] On Behalf Of Sep Ng [thejackschm...@gmail.com] Sent: 30 April 2009 17:04 To: aolser...@listserv.aol.com Subject: Re: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages? Yes. I'm using it with OpenACS. It's been driving me crazy for days now. May I ask how you setup nginx and aolserver? That would help immensely. I also suppose I could achieve the same by setting up apache before aolserver. Would be nice though to understand what's going wrong with OpenACS and AOLserver. Brian, the link you provided, I think deals with adp gzip compression on AOLserver 4.5, so I'm not sure if it's even applicable on 4.0.10. On Apr 30, 8:33 pm, aT atif@gmail.com wrote: Fenton, Brian wrote: Hi, I personally don't know anything about it, but I do remember this discussion from last year http://www.mail-archive.com/aolser...@listserv.aol.com/msg11654.html hope this helps somewhat Brian From: AOLserver Discussion [aolser...@listserv.aol.com] On Behalf Of Sep Ng [thejackschm...@gmail.com] Sent: 30 April 2009 02:28 To: aolser...@listserv.aol.com Subject: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages? Hi, I've been spending a load of time reading through the aolserver source code and fiddling with my configurations. I put this on my aolserver parameters: ns_section ns/server/testsite/adp/compress ns_param enabled true ns_param level 4 ns_param minsize 1024 I then put ns_adp_compress 1 on my adp pages and it does not get compressed. Is there a patch out there I may have missed or am I using this wrong? I'd appreciate it if someone could point me to the right direction. Thanks! -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. Are you using openacs ? If so , there seems to be an issue with the way headers are handled in acs code and this compression to work . I could never make it work with Openacs , however vanilla aoslerver worked fine . We ended up using nginx before aolserver and are enjoying the gzip comrpession with no effort at all from aolserver side . -- Syed Atif Ali D. +971 4 3911914 F. +971 4 3911915 ___ Put your Nose to the Grindstone! -- Amalgamated Plastic Surgeons and Toolmakers, Ltd. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists
Re: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages?
Yes. I'm using it with OpenACS. It's been driving me crazy for days now. May I ask how you setup nginx and aolserver? That would help immensely. I also suppose I could achieve the same by setting up apache before aolserver. Would be nice though to understand what's going wrong with OpenACS and AOLserver. Brian, the link you provided, I think deals with adp gzip compression on AOLserver 4.5, so I'm not sure if it's even applicable on 4.0.10. On Apr 30, 8:33 pm, aT atif@gmail.com wrote: Fenton, Brian wrote: Hi, I personally don't know anything about it, but I do remember this discussion from last year http://www.mail-archive.com/aolser...@listserv.aol.com/msg11654.html hope this helps somewhat Brian From: AOLserver Discussion [aolser...@listserv.aol.com] On Behalf Of Sep Ng [thejackschm...@gmail.com] Sent: 30 April 2009 02:28 To: aolser...@listserv.aol.com Subject: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages? Hi, I've been spending a load of time reading through the aolserver source code and fiddling with my configurations. I put this on my aolserver parameters: ns_section ns/server/testsite/adp/compress ns_param enabled true ns_param level 4 ns_param minsize 1024 I then put ns_adp_compress 1 on my adp pages and it does not get compressed. Is there a patch out there I may have missed or am I using this wrong? I'd appreciate it if someone could point me to the right direction. Thanks! -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. Are you using openacs ? If so , there seems to be an issue with the way headers are handled in acs code and this compression to work . I could never make it work with Openacs , however vanilla aoslerver worked fine . We ended up using nginx before aolserver and are enjoying the gzip comrpession with no effort at all from aolserver side . -- Syed Atif Ali D. +971 4 3911914 F. +971 4 3911915 ___ Put your Nose to the Grindstone! -- Amalgamated Plastic Surgeons and Toolmakers, Ltd. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages?
I really appreciate the help all of you have given me. I will look into the gzip compression patch and try other approaches if that does not work. Thank you very much for all the help! I post an update on how it has turned out asap. On May 1, 12:20 am, Fenton, Brian brian.fen...@quest.ie wrote: Hi Sep as I said, I know nothing about it myself, but the link I gave you does mention usage of ns_adp_compress, so I thought it may be of some assistance. I don't know how much these things changed from 4.0.10. Malte Sussdorf wrote a how-to for OpenACS and nginxhttp://cognovis.de/developer/en/nginx-loadbalancing cheers Brian From: AOLserver Discussion [aolser...@listserv.aol.com] On Behalf Of Sep Ng [thejackschm...@gmail.com] Sent: 30 April 2009 17:04 To: aolser...@listserv.aol.com Subject: Re: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages? Yes. I'm using it with OpenACS. It's been driving me crazy for days now. May I ask how you setup nginx and aolserver? That would help immensely. I also suppose I could achieve the same by setting up apache before aolserver. Would be nice though to understand what's going wrong with OpenACS and AOLserver. Brian, the link you provided, I think deals with adp gzip compression on AOLserver 4.5, so I'm not sure if it's even applicable on 4.0.10. On Apr 30, 8:33 pm, aT atif@gmail.com wrote: Fenton, Brian wrote: Hi, I personally don't know anything about it, but I do remember this discussion from last year http://www.mail-archive.com/aolser...@listserv.aol.com/msg11654.html hope this helps somewhat Brian From: AOLserver Discussion [aolser...@listserv.aol.com] On Behalf Of Sep Ng [thejackschm...@gmail.com] Sent: 30 April 2009 02:28 To: aolser...@listserv.aol.com Subject: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages? Hi, I've been spending a load of time reading through the aolserver source code and fiddling with my configurations. I put this on my aolserver parameters: ns_section ns/server/testsite/adp/compress ns_param enabled true ns_param level 4 ns_param minsize 1024 I then put ns_adp_compress 1 on my adp pages and it does not get compressed. Is there a patch out there I may have missed or am I using this wrong? I'd appreciate it if someone could point me to the right direction. Thanks! -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. Are you using openacs ? If so , there seems to be an issue with the way headers are handled in acs code and this compression to work . I could never make it work with Openacs , however vanilla aoslerver worked fine . We ended up using nginx before aolserver and are enjoying the gzip comrpession with no effort at all from aolserver side . -- Syed Atif Ali D. +971 4 3911914 F. +971 4 3911915 ___ Put your Nose to the Grindstone! -- Amalgamated Plastic Surgeons and Toolmakers, Ltd. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages?
Quick update: Just recompiled AOLserver 4.0.10 with the gzip path and still no go. I suppose I will scout the bug tracker on sourceforge to see if I find anything interesting. I wonder what problems did OpenACS introduce into this gzip issue... On May 1, 5:25 am, Sep Ng thejackschm...@gmail.com wrote: I really appreciate the help all of you have given me. I will look into the gzip compression patch and try other approaches if that does not work. Thank you very much for all the help! I post an update on how it has turned out asap. On May 1, 12:20 am, Fenton, Brian brian.fen...@quest.ie wrote: Hi Sep as I said, I know nothing about it myself, but the link I gave you does mention usage of ns_adp_compress, so I thought it may be of some assistance. I don't know how much these things changed from 4.0.10. Malte Sussdorf wrote a how-to for OpenACS and nginxhttp://cognovis.de/developer/en/nginx-loadbalancing cheers Brian From: AOLserver Discussion [aolser...@listserv.aol.com] On Behalf Of Sep Ng [thejackschm...@gmail.com] Sent: 30 April 2009 17:04 To: aolser...@listserv.aol.com Subject: Re: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages? Yes. I'm using it with OpenACS. It's been driving me crazy for days now. May I ask how you setup nginx and aolserver? That would help immensely. I also suppose I could achieve the same by setting up apache before aolserver. Would be nice though to understand what's going wrong with OpenACS and AOLserver. Brian, the link you provided, I think deals with adp gzip compression on AOLserver 4.5, so I'm not sure if it's even applicable on 4.0.10. On Apr 30, 8:33 pm, aT atif@gmail.com wrote: Fenton, Brian wrote: Hi, I personally don't know anything about it, but I do remember this discussion from last year http://www.mail-archive.com/aolser...@listserv.aol.com/msg11654.html hope this helps somewhat Brian From: AOLserver Discussion [aolser...@listserv.aol.com] On Behalf Of Sep Ng [thejackschm...@gmail.com] Sent: 30 April 2009 02:28 To: aolser...@listserv.aol.com Subject: [AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages? Hi, I've been spending a load of time reading through the aolserver source code and fiddling with my configurations. I put this on my aolserver parameters: ns_section ns/server/testsite/adp/compress ns_param enabled true ns_param level 4 ns_param minsize 1024 I then put ns_adp_compress 1 on my adp pages and it does not get compressed. Is there a patch out there I may have missed or am I using this wrong? I'd appreciate it if someone could point me to the right direction. Thanks! -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. Are you using openacs ? If so , there seems to be an issue with the way headers are handled in acs code and this compression to work . I could never make it work with Openacs , however vanilla aoslerver worked fine . We ended up using nginx before aolserver and are enjoying the gzip comrpession with no effort at all from aolserver side . -- Syed Atif Ali D. +971 4 3911914 F. +971 4 3911915 ___ Put your Nose to the Grindstone! -- Amalgamated Plastic Surgeons and Toolmakers, Ltd. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can
[AOLSERVER] Does the gzip component work on AOLSERVER 4.0.10 on adp pages?
Hi, I've been spending a load of time reading through the aolserver source code and fiddling with my configurations. I put this on my aolserver parameters: ns_section ns/server/testsite/adp/compress ns_param enabled true ns_param level 4 ns_param minsize 1024 I then put ns_adp_compress 1 on my adp pages and it does not get compressed. Is there a patch out there I may have missed or am I using this wrong? I'd appreciate it if someone could point me to the right direction. Thanks! -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem running Aolserver with Valgrind
Yeah. I know it's supposed to work. Oracle just refuses to cooperate it seems. I'm not sure why nor how valgrind could result in a TNS permission denied error. I've seen a few cases similar to mine on the net, and none of them have a solution. On Sep 2, 1:53 pm, Mark Aufflick [EMAIL PROTECTED] wrote: Not that this helps you, but I successfully run aolserver under valgrind without any database module loaded. On Tue, Sep 2, 2008 at 11:14 AM, Sep Ng [EMAIL PROTECTED] wrote: Hi, I've been trying to get aolserver to run on Valgrind for a week now and it does work to a certain extent. My Aolserver interfaces with Oracle databases and my problem is that when I do run aolserver on valgrind and nsoracle driver attempts to connect to Oracle, I get a OCIServerAttach ()': ORA-12546: TNS: Permission Denied error. I'm not sure why it's doing this. It works fine without valgrind. I've even set my permissions bits on oracle binary as rwsrwsrwx. It's still not working. I need help, please. :( I don't know where to look. Thanks. Sep -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- Mark Aufflick contact info athttp://mark.aufflick.com/about/contact -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] Problem running Aolserver with Valgrind
Hi, I've been trying to get aolserver to run on Valgrind for a week now and it does work to a certain extent. My Aolserver interfaces with Oracle databases and my problem is that when I do run aolserver on valgrind and nsoracle driver attempts to connect to Oracle, I get a OCIServerAttach ()': ORA-12546: TNS: Permission Denied error. I'm not sure why it's doing this. It works fine without valgrind. I've even set my permissions bits on oracle binary as rwsrwsrwx. It's still not working. I need help, please. :( I don't know where to look. Thanks. Sep -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.