[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[
http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12322710
]
Graham Dumpleton commented on MODPYTHON-77:
---
I worked out what was going on a few days ago, but first time with Internet
access since. :-(
I agree that what is being proposed is reasonable provided that correct changes
which will work for any version of 2.3 or later versions of Python.
Some further clarification on the issue.
1. The new API does not really break the concept of multiple interpreters. What
it means is that if someone is lazy enough to use the new API they limit their
extension to only being usable with the first created interpreter. Someone can
still use the existing APIs and write the extension to work in a multi
interpreter context.
2. Even with the patches, an extension will still not work properly. In order
to work it is important that PythonImport and
PythonInterpreter be used as appropriate and that main_interpreter be
explicitly specified. This means that one can't have multiple interpreters for
different virtual hosts or applications which is a big shortcoming.
In short, extensions writers should be discouraged from using the new API if
want to work with mod_python.
Out ot time, gotta go.
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c,
mod_python.c.diff, mod_python.h.diff, src.zip
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
Py_DECREF(p); /*p-refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
Re: Getting ready for 3.2 beta 2
I've been away this weekend - just got back, but I'm too busy to try to read all the multiple-interpreter related comments. I guess my question is - can someone provide a quick summary of how far we are from 3.2.1b test tarbal? Thanks! Grisha On Thu, 1 Sep 2005, Graham Dumpleton wrote: Nicolas Lehuen wrote .. Well I for one am happy woth MODPYTHON-73, I've integrated Graham's patch and made unit test to check if everything was OK. Graham should be happy too :). As troublesome as I am, even I am happy at this point. :-) Unfortunately, probably will not be able to do any last build checks on MacOSX. I think I'll get killed tonight if I start working on the computer tonight since I fly off quite early tomorrow morning. If I am lucky I'll get just enough time to sync from the svn repository and then I'll play with it on the plane. I'll then be off the Internet for about 4-5 days so if there are any problems you will not hear about them in time anyway. Graham
Re: Getting ready for 3.2 beta 2
Well, if I've understood Jim's mail, apart from the new MODPYTHON-77, we're all set. Regards, Nicolas2005/9/6, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: I've been away this weekend - just got back, but I'm too busy to try toread all the multiple-interpreter related comments. I guess my question is- can someone provide a quick summary of how far we are from 3.2.1b testtarbal?Thanks!GrishaOn Thu, 1 Sep 2005, Graham Dumpleton wrote: Nicolas Lehuen wrote .. Well I for one am happy woth MODPYTHON-73, I've integrated Graham's patch and made unit test to check if everything was OK. Graham should be happy too :). As troublesome as I am, even I am happy at this point. :-) Unfortunately, probably will not be able to do any last build checks on MacOSX. I think I'll get killed tonight if I start working on the computer tonight since I fly off quite early tomorrow morning. If I am lucky I'll get just enough time to sync from the svn repository and then I'll play with it on the plane. I'll then be off the Internet for about 4-5 days so if there are any problems you will not hear about them in time anyway. Graham
Re: Getting ready for 3.2 beta 2
Gregory (Grisha) Trubetskoy wrote: I've been away this weekend - just got back, but I'm too busy to try to read all the multiple-interpreter related comments. I guess my question is - can someone provide a quick summary of how far we are from 3.2.1b test tarbal? I've also been away for the weekend. The patch for MODPYTHON-77 from last Thursday was causing apache to segfault and I have not had a chance to try the lastest changes from Boyan. As Graham stated on the weekend, the use of thread states can be very tricky. I think we should proceed with the 3.2.1b without the fix. That way we can take the time to make sure we understand the issues and fix it in 3.3. If that seems reasonable, I'll make the tarball today. Jim Thanks! Grisha On Thu, 1 Sep 2005, Graham Dumpleton wrote: Nicolas Lehuen wrote .. Well I for one am happy woth MODPYTHON-73, I've integrated Graham's patch and made unit test to check if everything was OK. Graham should be happy too :). As troublesome as I am, even I am happy at this point. :-) Unfortunately, probably will not be able to do any last build checks on MacOSX. I think I'll get killed tonight if I start working on the computer tonight since I fly off quite early tomorrow morning. If I am lucky I'll get just enough time to sync from the svn repository and then I'll play with it on the plane. I'll then be off the Internet for about 4-5 days so if there are any problems you will not hear about them in time anyway. Graham
Re: Getting ready for 3.2 beta 2
On Tue, 6 Sep 2005, Jim Gallacher wrote: As Graham stated on the weekend, the use of thread states can be very tricky. I think we should proceed with the 3.2.1b without the fix. That way we can take the time to make sure we understand the issues and fix it in 3.3. If that seems reasonable, I'll make the tarball today. +1 Grisha
Problem with $req-upload()
Hello,
when I try to use
my $req = APR::Request::Apache2-handle($r);
my $f = $req-upload('file');
I get the following error in the error log:
[Tue Sep 06 19:09:10 2005] [error] Can't locate
auto/APR/Request/Apache2/upload.al in @INC (@INC contains: d:/sites/lib
d:/sites/cgi-bin D:/Sites/Perl/lib D:/Sites/Perl/site/lib .
D:/Sites/Servers/Apache2) at d:/sites/lib/DWCore/Req.pm line 314\n
Re: Problem with $req-upload()
Joe Schaefer [EMAIL PROTECTED] writes:
Nikolay Ananiev [EMAIL PROTECTED] writes:
Hello,
when I try to use
my $req = APR::Request::Apache2-handle($r);
my $f = $req-upload('file');
Hmm, so far there's no upload() method in APR::Request.
Ah, oops. It is available after all: APR::Request::upload
is currently provided by the APR::Request::Param module.
Go figure.
--
Joe Schaefer
Re: Proposed connection state diagram
On Tuesday 06 September 2005 06:10, Brian Pane wrote: http://www.brianp.net/work/opensource/apache/async.html Shouldn't there be a transition from HANDLER to CHECK_REQUEST_LINE_READABLE in case the full response is sent and keep-alive is on? Torsten pgpGwArqwD7oI.pgp Description: PGP signature
Re: LoadModule questions on WIN32
Jim Jagielski wrote: Mladen Turk wrote: This means that both create_server_config and create_dir_config function *are always* called on LoadModule, before module's pre_config hook is called. Is this a correct behavior? I need to relook at the code, but it's my recall that what that function does is call ap_set_module_config, which stores the create_*_config func pointer, not call it. Well, seems that those two functions are called before pre_config even from Apache 1.3, so seems that functions are called, not stored. This implicates that your patch for balancer *should* move external hook loading in the pre_config, and that the hook in balancer is declared in register_hooks table. BTW, when you are testing those patches on nix, do you use static or dynamic modules? Regards, Mladen.
Re: LoadModule questions on WIN32
Mladen Turk wrote: Well, seems that those two functions are called before pre_config even from Apache 1.3, so seems that functions are called, not stored. This implicates that your patch for balancer *should* move external hook loading in the pre_config, and that the hook in balancer is declared in register_hooks table. Also, as mentioned, register_hooks must be assumed to be primary, since it is there that hooks such as pre_config are defined. For some reason, it looks like in your setup, the ordering is all wrong, or, at least, the normal and expected behavior due to the double config read isn't cleared out as required. BTW, when you are testing those patches on nix, do you use static or dynamic modules? Both. Plus, as I said, Bill Rowe also tested under Win32 and got no errors at all. -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ Sith happens - Yoda
Re: LoadModule questions on WIN32
Jim Jagielski wrote: Mladen Turk wrote: BTW, when you are testing those patches on nix, do you use static or dynamic modules? Both. Plus, as I said, Bill Rowe also tested under Win32 and got no errors at all. Can you ask him to send me the binaries. I'm feeling quite puzzled. I would really like to see how somebody can use the function from the .dll/.so that has not been loaded in the process address space when acquired. Regards, Mladen.
Re: LoadModule questions on WIN32
Mladen Turk wrote: Jim Jagielski wrote: Mladen Turk wrote: BTW, when you are testing those patches on nix, do you use static or dynamic modules? Both. Plus, as I said, Bill Rowe also tested under Win32 and got no errors at all. Can you ask him to send me the binaries. I'm feeling quite puzzled. I would really like to see how somebody can use the function from the .dll/.so that has not been loaded in the process address space when acquired. Will do... -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ Sith happens - Yoda
Re: LoadModule questions on WIN32
Mladen Turk wrote:
I was wondering, if you add:
static const char *const aszSucc[] = { mod_proxy.c, NULL};
to ap_proxy_balancer_register_hook(), and then adjust the
following line to look like this:
proxy_hook_load_lbmethods(add_lbmethods, aszSucc, NULL, APR_HOOK_FIRST);
Does it change the behavior you see?
--
===
Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/
Sith happens - Yoda
Re: LoadModule questions on WIN32
Jim Jagielski wrote:
I was wondering, if you add:
static const char *const aszSucc[] = { mod_proxy.c, NULL};
to ap_proxy_balancer_register_hook(), and then adjust the
following line to look like this:
proxy_hook_load_lbmethods(add_lbmethods, aszSucc, NULL, APR_HOOK_FIRST);
Does it change the behavior you see?
Sorry but not.
Same thing; 'add_lbmethods' is never called because it is called from
create_server_config that OTOH is called before mod_proxy_balancer is
loaded by the LoadModule.
Regards,
Mladen.
Re: LoadModule questions on WIN32
Mladen Turk wrote:
Jim Jagielski wrote:
I was wondering, if you add:
static const char *const aszSucc[] = { mod_proxy.c, NULL};
to ap_proxy_balancer_register_hook(), and then adjust the
following line to look like this:
proxy_hook_load_lbmethods(add_lbmethods, aszSucc, NULL, APR_HOOK_FIRST);
Does it change the behavior you see?
Sorry but not.
Same thing; 'add_lbmethods' is never called because it is called from
create_server_config that OTOH is called before mod_proxy_balancer is
loaded by the LoadModule.
Strange... Because c_s_c is actually called *multiple* times during the
actual load and start up, such as ap_read_config(). Note that ap_top_module
is populated.
--
===
Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/
Sith happens - Yoda
Re: LoadModule questions on WIN32
Jim Jagielski wrote: Sorry but not. Same thing; 'add_lbmethods' is never called because it is called from create_server_config that OTOH is called before mod_proxy_balancer is loaded by the LoadModule. Strange... Because c_s_c is actually called *multiple* times during the actual load and start up, such as ap_read_config(). Note that ap_top_module is populated. Look Jim, not sure how can I explain that to you, but here is what happens: 1. LoadModule mod_proxy create_server_config tries to load the function from proxy_load_balancer 2. LoadModule mod_proxy_balancer So, c_s_c from mod_proxy is referencing the function that is loaded *after* that call. Since the module (mod_proxy_balancer) has not been loaded yet at the time mod_proxy is referencing that function it is stored as NULL, and of course, later it causes the core dump. Also, I've tried the linux build (Redhat9) with: #! /bin/sh # # Created by configure ./configure \ --prefix=/home/mturk/builds/apache/head \ --with-port=8000 \ --with-mpm=worker \ --enable-so \ --enable-mods-shared=all \ --enable-proxy=shared \ --enable-proxy-balancer=shared \ --enable-proxy-http=shared \ --enable-proxy-ajp=shared \ $@ make make install running httpd and pinging localhost:8000/servlet-examples/ for the config: Proxy balancer://cluster BalancerMember ajp://localhost:8009 /Proxy ProxyPass /servlet-examples/ balancer://cluster//servlet-examples/ Gives the following in the error_log: [Tue Sep 06 20:03:05 2005] [debug] mod_proxy_balancer.c(41): proxy: BALANCER: canonicalising URL //cluster//servlet-examples/ [Tue Sep 06 20:03:06 2005] [notice] child pid 3455 exit signal Segmentation fault (11) So, this is 2.1.7-beta from pquerna. I wonder, I'm am stupid, or what ? Regards, Mladen.
Re: LoadModule questions on WIN32
Mladen Turk wrote: Look Jim, not sure how can I explain that to you, but here is what happens: 1. LoadModule mod_proxy create_server_config tries to load the function from proxy_load_balancer 2. LoadModule mod_proxy_balancer So, c_s_c from mod_proxy is referencing the function that is loaded *after* that call. Since the module (mod_proxy_balancer) has not been loaded yet at the time mod_proxy is referencing that function it is stored as NULL, and of course, later it causes the core dump. You forget that the server_conf is created twice... But I think I see what your theory is. [Tue Sep 06 20:03:05 2005] [debug] mod_proxy_balancer.c(41): proxy: BALANCER: canonicalising URL //cluster//servlet-examples/ [Tue Sep 06 20:03:06 2005] [notice] child pid 3455 exit signal Segmentation fault (11) So, this is 2.1.7-beta from pquerna. Weird that no one else sees this... can you try the perl framework test? -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ Sith happens - Yoda
Re: LoadModule questions on WIN32
Jim Jagielski wrote: [Tue Sep 06 20:03:05 2005] [debug] mod_proxy_balancer.c(41): proxy: BALANCER: canonicalising URL //cluster//servlet-examples/ [Tue Sep 06 20:03:06 2005] [notice] child pid 3455 exit signal Segmentation fault (11) So, this is 2.1.7-beta from pquerna. Weird that no one else sees this... can you try the perl framework test? Sorry, it makes no sense. I've spend too much on that already. Perhaps tomorrow. Regards, Mladen.
Re: LoadModule questions on WIN32
Mladen Turk wrote: Jim Jagielski wrote: [Tue Sep 06 20:03:05 2005] [debug] mod_proxy_balancer.c(41): proxy: BALANCER: canonicalising URL //cluster//servlet-examples/ [Tue Sep 06 20:03:06 2005] [notice] child pid 3455 exit signal Segmentation fault (11) So, this is 2.1.7-beta from pquerna. Weird that no one else sees this... can you try the perl framework test? Sorry, it makes no sense. I've spend too much on that already. Perhaps tomorrow. Sure thing... If your theory is correct, then having the LoadModule of the balancer before that of the proxy would fix the problem. -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ Sith happens - Yoda
proxy balancer hook (Was: Re: LoadModule questions on WIN32)
Can anyone else recreate the core dumps that Mladen is seeing with the balancer in 2.1.7-beta? The httpd-test perl framework has a simple test for it. -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/
Re: proxy balancer hook (Was: Re: LoadModule questions on WIN32)
Jim Jagielski wrote: Can anyone else recreate the core dumps that Mladen is seeing with the balancer in 2.1.7-beta? The httpd-test perl framework has a simple test for it. I can confirm Mladens core dumps. I used the latest revision from svn and did the following: ./configure --prefix=/usr/src/apache/apache_trunk \ --with-mpm=worker \ --enable-so \ --enable-mods-shared=all \ --enable-proxy=shared \ --enable-proxy-balancer=shared \ --enable-proxy-http=shared \ --enable-proxy-ajp=shared make make install I added the following lines to the default httpd.conf LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_connect_module modules/mod_proxy_connect.so LoadModule proxy_ftp_module modules/mod_proxy_ftp.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so Proxy balancer://cluster BalancerMember ajp://localhost:8009 /Proxy ProxyPass /servlet-examples/ balancer://cluster//servlet-examples/ Calling http://127.0.0.1/servlet-examples resulted in: [Tue Sep 06 22:11:53 2005] [notice] child pid 30700 exit signal Segmentation fault (11) Stacktrace: #0 0x in ?? () #1 0x403bca71 in find_best_worker (balancer=0x4083b94c, r=0x4083b950) at mod_proxy_balancer.c:379 #2 0x403bccee in proxy_balancer_pre_request (worker=0x4083b94c, balancer=0x4083b950, r=0x8171758, conf=0x80f2ed0, url=0x4083b954) at mod_proxy_balancer.c:511 #3 0x4039a2f9 in proxy_run_pre_request (worker=0x4083b94c, balancer=0x4083b950, r=0x8171758, conf=0x80f2ed0, url=0x4083b954) at mod_proxy.c:1947 #4 0x4039cafe in ap_proxy_pre_request (worker=0x4083b94c, balancer=0x4083b950, r=0x8171758, conf=0x80f2ed0, url=0x4083b954) at proxy_util.c:1325 System: glibc-2.2.5 Linux euler 2.4.21-20050503-ruediger #2 Tue May 3 12:40:08 CEST 2005 i686 unknown Putting LoadModule proxy_balancer_module modules/mod_proxy_balancer.so before LoadModule proxy_module modules/mod_proxy.so in httpd.conf results in httpd: Syntax error on line 101 of /usr/src/apache/apache_trunk/conf/httpd.conf: Cannot load /usr/src/apache/apache_trunk/modules/mod_proxy_balancer.so into server: /usr/src/apache/apache_trunk/modules/mod_proxy_balancer.so: undefined symbol: proxy_module So this does not fix it either. Regards Rüdiger
Re: proxy balancer hook (Was: Re: LoadModule questions on WIN32)
Hmmm My tests have all been http: Proxy balancer://foo BalancerMember http://www.google.com:80 loadfactor=1 ProxySet lbmethod=bytraffic /Proxy ProxyPass /bat/ balancer://foo/ Weird... Thanks for the feedback! [EMAIL PROTECTED] wrote: Jim Jagielski wrote: Can anyone else recreate the core dumps that Mladen is seeing with the balancer in 2.1.7-beta? The httpd-test perl framework has a simple test for it. I can confirm Mladens core dumps. I used the latest revision from svn and did the following: ./configure --prefix=/usr/src/apache/apache_trunk \ --with-mpm=worker \ --enable-so \ --enable-mods-shared=all \ --enable-proxy=shared \ --enable-proxy-balancer=shared \ --enable-proxy-http=shared \ --enable-proxy-ajp=shared make make install I added the following lines to the default httpd.conf LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_connect_module modules/mod_proxy_connect.so LoadModule proxy_ftp_module modules/mod_proxy_ftp.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so Proxy balancer://cluster BalancerMember ajp://localhost:8009 /Proxy ProxyPass /servlet-examples/ balancer://cluster//servlet-examples/ Calling http://127.0.0.1/servlet-examples resulted in: [Tue Sep 06 22:11:53 2005] [notice] child pid 30700 exit signal Segmentation fault (11) Stacktrace: #0 0x in ?? () #1 0x403bca71 in find_best_worker (balancer=0x4083b94c, r=0x4083b950) at mod_proxy_balancer.c:379 #2 0x403bccee in proxy_balancer_pre_request (worker=0x4083b94c, balancer=0x4083b950, r=0x8171758, conf=0x80f2ed0, url=0x4083b954) at mod_proxy_balancer.c:511 #3 0x4039a2f9 in proxy_run_pre_request (worker=0x4083b94c, balancer=0x4083b950, r=0x8171758, conf=0x80f2ed0, url=0x4083b954) at mod_proxy.c:1947 #4 0x4039cafe in ap_proxy_pre_request (worker=0x4083b94c, balancer=0x4083b950, r=0x8171758, conf=0x80f2ed0, url=0x4083b954) at proxy_util.c:1325 System: glibc-2.2.5 Linux euler 2.4.21-20050503-ruediger #2 Tue May 3 12:40:08 CEST 2005 i686 unknown Putting LoadModule proxy_balancer_module modules/mod_proxy_balancer.so before LoadModule proxy_module modules/mod_proxy.so in httpd.conf results in httpd: Syntax error on line 101 of /usr/src/apache/apache_trunk/conf/httpd.conf: Cannot load /usr/src/apache/apache_trunk/modules/mod_proxy_balancer.so into server: /usr/src/apache/apache_trunk/modules/mod_proxy_balancer.so: undefined symbol: proxy_module So this does not fix it either. Regards Rüdiger -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ Sith happens - Yoda
Re: ApacheCon BOF about Module Repository
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Folks, I have been meaning to follow up on this ever since that BOF, and am deeply annoyed that it has taken me this long. As a matter of fact, my memory of that BOF session has now faded to a considerable extent and I don't feel comfortable even giving a list of attendees because I would leave people out. I have pinned to my office wall the flip-over sheet with notes I took during the session and will now transcribe those. If anyone present at the event notices I'm leaving something out, please speak up. Building on my original message below, we discussed what should be implemented and how. One aspect of TCAHMAN that I hadn't covered in my original discussion is how to add modules from the repository to an existing Apache install. This would require a program, installed with the server, that can fetch the module code and run the build/install. We have tentatively named this program apxs++ since it's a logical extension of what apxs does today. For maximum compatibility, this tool would have to be written in C. Currently, apxs is a Perl program but you can't always count on the availability of Perl on the system, especially on Windows. The apxs++ tool would be available only when mod_so is available. If you're linking modules statically, you're compiling your own httpd and should be able to fetch the source code for the desired module(s) before you start compiling. I have no notes about the scenarios described below, fetching the module code from configure or pointing configure at a module source tarball, but I seem to recall the group was not overly enthusiastic about configure downloading and injecting source code into the build. All this goodness, if and when it happens, would be run from a newly created httpd-modules subproject. We discussed the proposed nature and structure of this subproject (which itself has not been proposed yet) and the general idea seemed to be that we creat a flat sandbox where module developers can commit to everything. Every httpd committer automatically gets httpd-modules, and the subproject could be a breeding ground for new httpd committers. If and when a module develops its own community, it can get its own subproject (example: mod_python) or even go top level (example: mod_perl, mod_tcl). The httpd-modules subproject would also own the repository code. The TCAHMAN system would be targeted at: a) builders (who build their own Apache) b) enhancers (what did I mean by this? Perhaps folks who want to hang additional modules into an existing Apache?) c) packagers (TCAHMAN could register installed modules with the various package registries out there, giving httpd packagers a powerful way to manage the installed core and modules) d) Testers (perl-framework) (Not sure what I meant by this) We would initially populate the repository with modules that were formerly in the core, and eventually open it up to third-party module developers. Having easy access to modules through TCAHMAN will allow us (httpd) to lighten the distribution Once we open the repository up to third-party developers, we may have to do a 'click through' (or key through) acknowledgement that we (ASF) are not responsible for code that is not ours. IANAL, so I don't know what is required/comfy. The TCAHMAN repository would utilize our existing mirror infrastructure, and would be a great service to offer third-party developers. We discussed CPAN, from which a lot of people blindly and trustingly download module upon module, as root. How did this get so trusted? Who is responsible for the code? We hear that nobody owns CPAN, and there is no identifiable target for any legal action anyone might want to bring. This obviously wouldn't fly for the ASF. The designated front for TCAHMAN would be modules.apache.org, which is currently run by Covalent. We would run TCAHMAN on our own infrastructure, so we'd need to get the vhost back from them. While this is technically really easy (we own the DNS for apache.org, after all), it would be a good thing to arrange a smooth transition. Every module uploaded to the network would come with metadata, including (but not limited to): * License * Versioning (compatible with (1.3, 2.0, 2.1, ...), not before, not after (MMN?)) * Documentation URL * Author info * Build options Tasks: * Write apxs++ * Define module metadata * Write the backend * Take back modules.apache.org That's all I have right now. Remarks? Comments? Additions? Flames? Please discuss: we need to find out whether there is community support for the idea. Thanks, Sander On Jul 21, 2005, at 3:14 AM, Sander Temme wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All who are at ApacheCon or are otherwise interested, I snatched a BOF slot tonight (Thursday the 21st) at 20:30 to
Re: proxy balancer hook (Was: Re: LoadModule questions on WIN32)
Hold on a tic... it seems to be related to whether or not it's within a Vhost...
Re: proxy balancer hook (Was: Re: LoadModule questions on WIN32)
Jim Jagielski wrote: Hold on a tic... it seems to be related to whether or not it's within a Vhost... I can confirm this Jim. Once I use VirtualHost 127.0.0.1:80 Proxy balancer://cluster BalancerMember ajp://localhost:8009 /Proxy ProxyPass /servlet-examples/ balancer://cluster//servlet-examples/ /VirtualHost instead of Proxy balancer://cluster BalancerMember ajp://localhost:8009 /Proxy ProxyPass /servlet-examples/ balancer://cluster//servlet-examples/ it does not Segfault any more. Regards Rüdiger
Re: ApacheCon BOF about Module Repository
Sander Temme wrote: For maximum compatibility, this tool would have to be written in C. Currently, apxs is a Perl program but you can't always count on the availability of Perl on the system, especially on Windows. The apxs++ tool would be available only when mod_so is available. Sorry for jumping in... been roughly following. As far as I can see, apxs is only written in perl because of the REGEX to find wether were are in a Location or other container before adding the LoadModule line. Could we not use PCRE and APR to rewrite this in C as well? Maybe 2.3 ? And this comming from a mod_perl commiter :) -- END What doesn't kill us can only make us stronger. Nothing is impossible. Philip M. Gollucci ([EMAIL PROTECTED]) 301.254.5198 Consultant / http://p6m7g8.net/Resume/ Senior Developer / Liquidity Services, Inc. http://www.liquidityservicesinc.com http://www.liquidation.com http://www.uksurplus.com http://www.govliquidation.com http://www.gowholesale.com
Re: proxy balancer hook (Was: Re: LoadModule questions on WIN32)
[EMAIL PROTECTED] wrote: Jim Jagielski wrote: Hold on a tic... it seems to be related to whether or not it's within a Vhost... I think this is the solution as for each virtual host the create_server_config is run for this virtual host again *after* all modules have been loaded. Thus it works inside a virtual host, but not outside. I was wondering why similar things work with mod_cache / mod_disk_cache and after a quick look to the code I found that (I guess no surprise to most here) they use ap_register_provider / ap_lookup_provider. mod_cache does check if the provider name specified in CacheEnable is valid *after* configuration during the request processing. Stupid question: Would that be a possible way here too? Regards Rüdiger
Re: proxy balancer hook (Was: Re: LoadModule questions on WIN32)
[EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: Jim Jagielski wrote: Hold on a tic... it seems to be related to whether or not it's within a Vhost... I think this is the solution as for each virtual host the create_server_config is run for this virtual host again *after* all modules have been loaded. Thus it works inside a virtual host, but not outside. I was wondering why similar things work with mod_cache / mod_disk_cache and after a quick look to the code I found that (I guess no surprise to most here) they use ap_register_provider / ap_lookup_provider. mod_cache does check if the provider name specified in CacheEnable is valid *after* configuration during the request processing. Stupid question: Would that be a possible way here too? Yeah, I'm looking into that, but I think that it might be a deeper bug; the hook should also be available for the 2nd main server config creation; the first run through should register it, and then the 2nd should allow it to be used. But even so, I will likely just make the load_lb call a direct call, and not a hook, while we work on it... -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ Sith happens - Yoda
Re: [PATCH] mod_authnz_ldap and satisfy all
On Sep 5, 2005, at 3:15 AM, Graham Leggett wrote:
Ryan Morgan wrote:
require ldap-group cn=Engineering,ou=Groups,o=SomeCompany,c=US
require ldap-group cn=QA,ou=Groups,o=SomeCompany,c=US
satisfy all
Could someone provide feedback on whether this is a feature that
could be
added to the ldap module?
Definitely a +1 in concept. Do the other authz modules handle
satisfy all in the same way?
Great.. Thanks for taking a look Graham.
Other than mod_access, none of the other authz modules handle the
satisfy
directive. mod_access uses it to specify how to handle authorization
when
both the require and allow directives are used.
This patch builds on that concept, but handles the case where
multiple require
lines are present. I figured using satisfy made more sense than
adding another
directive to the ldap module. It's entirely possible that satisfy
was not meant
to be used this way, but it seems to fit in nicely.
A quick eyeball of the patch shows up some C++ comments - can you
convert them to C comments?
Sorry about that, attached is an updated patch.
-Ryan
Index: modules/aaa/mod_authnz_ldap.c
===
--- modules/aaa/mod_authnz_ldap.c (revision 278618)
+++ modules/aaa/mod_authnz_ldap.c (working copy)
@@ -477,6 +477,9 @@
const apr_array_header_t *reqs_arr = ap_requires(r);
require_line *reqs = reqs_arr ? (require_line *)reqs_arr-elts : NULL;
+int satisfy = ap_satisfies(r);
+int require_failed = 0; /* flag for satisfy all */
+int req_user_success, req_group_success, req_attribute_success;
register int x;
const char *t;
@@ -602,9 +605,13 @@
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
[% APR_PID_T_FMT ] auth_ldap authorise:
require user: authorisation successful,
getpid());
-return OK;
+if (satisfy != SATISFY_ALL) {
+return OK;
+}
+break;
}
default: {
+require_failed = 1;
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
[% APR_PID_T_FMT ] auth_ldap authorise:
require user:
authorisation failed [%s][%s], getpid(),
@@ -614,24 +621,31 @@
/*
* Now break apart the line and compare each word on it
*/
+req_user_success = 0;
while (t[0]) {
w = ap_getword_conf(r-pool, t);
result = util_ldap_cache_compare(r, ldc, sec-url, req-dn,
sec-attribute, w);
-switch(result) {
-case LDAP_COMPARE_TRUE: {
-ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
- [% APR_PID_T_FMT ] auth_ldap
authorise:
- require user: authorisation
successful, getpid());
+
+if (result == LDAP_COMPARE_TRUE) {
+ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
+ [% APR_PID_T_FMT ] auth_ldap authorise:
+ require user: authorisation successful,
getpid());
+if (satisfy != SATISFY_ALL) {
return OK;
}
-default: {
-ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
- [% APR_PID_T_FMT ] auth_ldap
authorise:
- require user: authorisation failed
[%s][%s],
- getpid(), ldc-reason,
ldap_err2string(result));
-}
+req_user_success = 1;
+break;
+} else {
+ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
+ [% APR_PID_T_FMT ] auth_ldap authorise:
+ require user: authorisation failed
[%s][%s],
+ getpid(), ldc-reason,
ldap_err2string(result));
}
}
+if (!req_user_success) {
+/* all user checks failed for this require line */
+require_failed = 1;
+}
}
else if (strcmp(w, ldap-dn) == 0) {
if (req-dn == NULL || strlen(req-dn) == 0) {
@@ -648,9 +662,13 @@
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
[% APR_PID_T_FMT ] auth_ldap authorise:
require dn: authorisation successful,
getpid());
-return OK;
+if (satisfy != SATISFY_ALL) {
+return OK;
+}
+
Re: ApacheCon BOF about Module Repository
Sander Temme wrote: One aspect of TCAHMAN that I hadn't covered in my original discussion is how to add modules from the repository to an existing Apache install. This would require a program, installed with the server, that can fetch the module code and run the build/install. We have tentatively named this program apxs++ since it's a logical extension of what apxs does today. For maximum compatibility, this tool would have to be written in C. Currently, apxs is a Perl program but you can't always count on the availability of Perl on the system, especially on Windows. The apxs++ tool would be available only when mod_so is available. I think we did discuss this a little, though my memory is faded too. I don't see the problem with apxs being perl. Sure, it is not quite universally available, but then neither is a C compiler, which is another prerequisite for anything APXS-like. That implies at least the option of the tool downloading a binary (and its dependencies where applicable), and it no longer really looks like apxs. So why not preserve apxs as-is - perhaps enhanced to use the new archive for those who do have the prerequisites - and make a clean new start with the new tool? If you're linking modules statically, you're compiling your own httpd and should be able to fetch the source code for the desired module(s) before you start compiling. I don't think we should worry too much about supporting static compiles. Rather concentrate on making sure everyone has a dynamic build unless they consciously override that, for their own reasons. All this goodness, if and when it happens, would be run from a newly created httpd-modules subproject. We discussed the proposed nature and structure of this subproject (which itself has not been proposed yet) and the general idea seemed to be that we creat a flat sandbox where module developers can commit to everything. Every httpd committer automatically gets httpd-modules, and the subproject could be a breeding ground for new httpd committers. If and when a module develops its own community, it can get its own subproject (example: mod_python) or even go top level (example: mod_perl, mod_tcl). The httpd-modules subproject would also own the repository code. This blurs the distinction between bundled and addon modules. That may be no bad thing, but needs to be thought through. Do we, for example, officially accept bug reports for third-party modules in our bugzilla? 'Cos if not, that's a potentially nasty limbo state for the end-users. The TCAHMAN system would be targeted at: a) builders (who build their own Apache) b) enhancers (what did I mean by this? Perhaps folks who want to hang additional modules into an existing Apache?) c) packagers (TCAHMAN could register installed modules with the various package registries out there, giving httpd packagers a powerful way to manage the installed core and modules) d) Testers (perl-framework) (Not sure what I meant by this) e) Any server admin We would initially populate the repository with modules that were formerly in the core, and eventually open it up to third-party module developers. Having easy access to modules through TCAHMAN will allow us (httpd) to lighten the distribution Are you thinking of even the very-core modules? I think that could make sense, provided the default install always includes them unless the builder makes a conscious decision and overrides a warning about some core functions may not work. Once we open the repository up to third-party developers, we may have to do a 'click through' (or key through) acknowledgement that we (ASF) are not responsible for code that is not ours. IANAL, so I don't know what is required/comfy. Indeed. The TCAHMAN repository would utilize our existing mirror infrastructure, and would be a great service to offer third-party developers. We discussed CPAN, from which a lot of people blindly and trustingly download module upon module, as root. How did this get so trusted? Who is responsible for the code? We hear that nobody owns CPAN, and there is no identifiable target for any legal action anyone might want to bring. This obviously wouldn't fly for the ASF. The designated front for TCAHMAN would be modules.apache.org, which is currently run by Covalent. We would run TCAHMAN on our own infrastructure, so we'd need to get the vhost back from them. While this is technically really easy (we own the DNS for apache.org, after all), it would be a good thing to arrange a smooth transition. Every module uploaded to the network would come with metadata, including (but not limited to): * License * Versioning (compatible with (1.3, 2.0, 2.1, ...), not before, not after (MMN?)) * Documentation URL * Author info * Build options * Dependencies (e.g. external libs) * Exports (e.g. modules that provide or consume an API) *
Re: ApacheCon BOF about Module Repository
On Tue, 6 Sep 2005, Sander Temme wrote: Folks, I have been meaning to follow up on this ever since that BOF, and am deeply annoyed that it has taken me this long. As a matter of fact, my memory of that BOF session has now faded to a considerable extent and I don't feel comfortable even giving a list of attendees because I would leave people out. I have pinned to my office wall the flip-over sheet with notes I took during the session and will now transcribe those. If anyone present at the event notices I'm leaving something out, please speak up. Building on my original message below, we discussed what should be implemented and how. [ ... ] We discussed CPAN, from which a lot of people blindly and trustingly download module upon module, as root. How did this get so trusted? Who is responsible for the code? We hear that nobody owns CPAN, and there is no identifiable target for any legal action anyone might want to bring. This obviously wouldn't fly for the ASF. Jarkko Hietaniemi, the CPAN Master Librarian, wrote up some thoughts on CPAN-like archives at http://www.cpan.org/misc/ZCAN.html Most (but not all) activity in CPAN is connected with Perl modules; for this, PAUSE (http://pause.perl.org/) is the means by which authors can upload and manage submissions. The code for PAUSE is available: ftp://pause.perl.org/pub/PAUSE/PAUSE-code/ Every module uploaded to the network would come with metadata, including (but not limited to): [ ... ] In the Perl world, metadata now comes in the form of a META.yml file: http://module-build.sourceforge.net/META-spec.html which is based on YAML: http://www.yaml.org/ This is one area that the CPAN/PAUSE maintainers emphasize as being crucial in being able to index and search effectively. -- best regards, randy kobes
