Re: apr_dbd: Support for multiple database connections from the same virtual host
Sorry for delay. Hope I will manage to find some time till monday. On Mon, Mar 8, 2010 at 5:45 PM, Kevac Marko ma...@kevac.org wrote: If someone is still interested, I can update bug with newer patch in a few days. On Mon, Mar 8, 2010 at 5:44 PM, Kevac Marko ma...@kevac.org wrote: Hello. Probably you are talking about my bug and my patch - https://issues.apache.org/bugzilla/show_bug.cgi?id=45456 As far as I know this patch was not been reviewed nor included to trunk. We are using mod_dbd with this patch (probably slightly improved over time) in production. On Mon, Mar 8, 2010 at 12:13 AM, Graham Leggett minf...@sharp.fm wrote: Hi all, I am currently looking through the apr_dbd module to find whether it is possible to support multiple database connections per virtual host. I am finding references on the mailing list[1] that this capability was present on trunk, but looking at trunk, I don't see see any evidence of this kind of capability. Is this documented anywhere? [1] http://www.mail-archive.com/us...@httpd.apache.org/msg38390.html Regards, Graham -- -- Marko Kevac -- Marko Kevac -- Marko Kevac
Building httpd trunk. Problem with apr-util.
Hello. After cloning httpd git repository (mirror of svn one), I have tried to build trunk, but I don't know hot to tell build sripts what apr-util library to use. $ ./configure --with-apr=/usr/bin/apr-1-config --with-apr-util=/usr/bin/apu-1-config --prefix=/home/marko/micex/opt/httpd/trunk/ 21 | tee configure.log checking for chosen layout... Apache checking for working mkdir -p... yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu Configuring Apache Portable Runtime library ... checking for APR... yes setting CC to x86_64-linux-gnu-gcc setting CPP to x86_64-linux-gnu-gcc -E setting CFLAGS to -pthread setting CPPFLAGS to -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE setting LDFLAGS to Configuring Apache Portable Runtime Utility library... configure: error: APR-util not found. Please read the documentation There is no --with-apr-util in ./configure --help, so it is just guessing. ma...@marko-desktop:~/micex/distr/git/httpd$ which apr-1-config /usr/bin/apr-1-config ma...@marko-desktop:~/micex/distr/git/httpd$ which apu-1-config /usr/bin/apu-1-config By the way, what version of apr and apr-util httpd-trunk requires? -- Marko Kevac
Re: Building httpd trunk. Problem with apr-util.
Built httpd-trunk with apr-trunk and apr-util-trunk. But still, some questions remained: 1) Where information about what version of apr and apr-util is needed for each verson of httpd could be found? 2) Where is --with-apr-util or --with-apu configuration option? On Sun, Mar 14, 2010 at 11:46 AM, Kevac Marko ma...@kevac.org wrote: Hello. After cloning httpd git repository (mirror of svn one), I have tried to build trunk, but I don't know hot to tell build sripts what apr-util library to use. $ ./configure --with-apr=/usr/bin/apr-1-config --with-apr-util=/usr/bin/apu-1-config --prefix=/home/marko/micex/opt/httpd/trunk/ 21 | tee configure.log checking for chosen layout... Apache checking for working mkdir -p... yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu Configuring Apache Portable Runtime library ... checking for APR... yes setting CC to x86_64-linux-gnu-gcc setting CPP to x86_64-linux-gnu-gcc -E setting CFLAGS to -pthread setting CPPFLAGS to -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE setting LDFLAGS to Configuring Apache Portable Runtime Utility library... configure: error: APR-util not found. Please read the documentation There is no --with-apr-util in ./configure --help, so it is just guessing. ma...@marko-desktop:~/micex/distr/git/httpd$ which apr-1-config /usr/bin/apr-1-config ma...@marko-desktop:~/micex/distr/git/httpd$ which apu-1-config /usr/bin/apu-1-config By the way, what version of apr and apr-util httpd-trunk requires? -- Marko Kevac -- Marko Kevac
r-connection-aborted in mod_disk_cache and mod_mem_cache
Hello. Why this code is presented in disk cache, but not in mem cache?
if (r-connection-aborted || r-no_cache) {
ap_log_error(APLOG_MARK, APLOG_INFO, 0, r-server,
disk_cache: Discarding body for URL %s
because connection has been aborted.,
h-cache_obj-key);
/* Remove the intermediate cache file and return non-APR_SUCCESS */
file_cache_errorcleanup(dobj, r);
return APR_EGENERAL;
}
--
Marko Kevac
Re: apr_dbd: Support for multiple database connections from the same virtual host
Hello. Probably you are talking about my bug and my patch - https://issues.apache.org/bugzilla/show_bug.cgi?id=45456 As far as I know this patch was not been reviewed nor included to trunk. We are using mod_dbd with this patch (probably slightly improved over time) in production. On Mon, Mar 8, 2010 at 12:13 AM, Graham Leggett minf...@sharp.fm wrote: Hi all, I am currently looking through the apr_dbd module to find whether it is possible to support multiple database connections per virtual host. I am finding references on the mailing list[1] that this capability was present on trunk, but looking at trunk, I don't see see any evidence of this kind of capability. Is this documented anywhere? [1] http://www.mail-archive.com/us...@httpd.apache.org/msg38390.html Regards, Graham -- -- Marko Kevac
Re: apr_dbd: Support for multiple database connections from the same virtual host
If someone is still interested, I can update bug with newer patch in a few days. On Mon, Mar 8, 2010 at 5:44 PM, Kevac Marko ma...@kevac.org wrote: Hello. Probably you are talking about my bug and my patch - https://issues.apache.org/bugzilla/show_bug.cgi?id=45456 As far as I know this patch was not been reviewed nor included to trunk. We are using mod_dbd with this patch (probably slightly improved over time) in production. On Mon, Mar 8, 2010 at 12:13 AM, Graham Leggett minf...@sharp.fm wrote: Hi all, I am currently looking through the apr_dbd module to find whether it is possible to support multiple database connections per virtual host. I am finding references on the mailing list[1] that this capability was present on trunk, but looking at trunk, I don't see see any evidence of this kind of capability. Is this documented anywhere? [1] http://www.mail-archive.com/us...@httpd.apache.org/msg38390.html Regards, Graham -- -- Marko Kevac -- Marko Kevac
Re: Finding memory leaks in httpd and httpd modules
On Tue, Feb 16, 2010 at 2:10 PM, harish kulkarni wasinap...@gmail.com wrote: http://harishvk27.wordpress.com/2008/11/05/fixing-memory-leak-in-apache-mod_proxy/ Thank you. But I am surprised that no one uses valgrind. -- Marko Kevac Sent from Moscow, Mow, Russia
Re: Finding memory leaks in httpd and httpd modules
On Wed, Feb 17, 2010 at 12:49 AM, Sam Carleton scarle...@gmail.com wrote: I am in agreement with mcqueenorama. I have not done a ton of module programming, but what I have done is HEAVILY used by my app and core to the whole thing. My best advise is don't use the pools other then the request pool unless you REALLY need to, even when the module is initializing there is a temp pool which Apache clears after initialization. Thank you. Yes, I really need to use pools that live longer than request and sometimes I make mistakes which are causing leaks. Using --enable-pool-debug=all is probably best advice by now. I will try that shortly. I did run into a memory leak once that I found by simply checking all the pool usages, it turned out I was using a server pool rather then a request pool. Something that automatizes that checking is what I am looking for :-) I NEVER use anything other then a pool for memory allocation. The only exception is 3rd party lib's that allocate memory, such as my usage of SQLite, which I am sure allocates lots of memory, but I rely on it to clean up that memory, assuming I manage the resource correctly. To date, all runs well! I am using such libraries too. libmemcached for example. Registering free() function to exec when some pool will be cleaned is very useful apr lib feature. -- Marko Kevac Sent from Moscow, Mow, Russia
Re: unsupported compression issue seen in 2.3.5-alpha
I don't know if this is connected, but we experience some problems with gzip compressing. Open http://www.micex.ru via Chrome and and hit shift + refresh button. Sometimes tables will not load (a little rotating circle will appear). These tables are loaded via AJAX and when they are not loaded, Chrome shows some garbage after JSON result. This garbage happens very often in Chrome. Some debugging showed that web server answer with garbage and without differs in 20 bytes at the end. But only Chrome and some Delphi libraries return error when extracting these archives. Firefox, gzip console utility, they open it without problems. On Sat, Feb 13, 2010 at 10:30 PM, Paul Querna p...@querna.org wrote: Hi, For www.apache.org (and all TLP websites), we have received increased reports about invalid compression issues. Previously, we have heard about sporadic issues with Firefox only, and these were always resolved by users by clearing their cache, this issue seems to be a good description of it: https://bugzilla.mozilla.org/show_bug.cgi?id=366023 However, the newest reports have been about multiple browsers, Firefox, Chrome, Safari, and IE8, all reporting the issue at the same time. This makes be believe that the problem is something on the server side. If you look at the history of mod_deflate: http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/filters/mod_deflate.c?view=log r821471 is the only change in the last year -- and it changes how buckets are used -- and I view it as suspect in relation to the volume of reports we have received from users, but we don't have any firm evidence. Any ideas? Thanks, Paul -- Marko Kevac
Re: Using mod_rewrite in my authorization module. Need advice.
On Thu, Feb 4, 2010 at 10:52 AM, Sorin Manolache sor...@gmail.com wrote:
Try to set an apache request note in the authentication instead of
dynamically changing the configuration of mod_rewrite.c.
Thus, you'll have something like
RewriteRule /url %{ENV:destination}
The configuration would be always the same, just the value of the
destination request note changes dynamically, depending on the
authentication.
H... Interesting. Are you sure that this works?
Can I do something like RewriteRule %{ENV:source} %{ENV:destination} ?
--
Marko Kevac
Sent from Moscow, Mow, Russia
Re: Using mod_rewrite in my authorization module. Need advice.
On Thu, Feb 4, 2010 at 2:05 PM, Sorin Manolache sor...@gmail.com wrote: No, you cannot. The expansion does not work in the pattern (the second argument). But the second argument can be a regular expression. Hopefully you can write regexps for all your cases. Unfortunately I cannot. These should be in DB also, not in httpd.conf. Right now I am rewriting basic rewrite functionality with some mod_rewrite features like [QCA]. I hope this will be enough. -- Marko Kevac Sent from Moscow, Mow, Russia
Using mod_rewrite in my authorization module. Need advice.
Hello. I am porting our authorization module to mpm_worker from mpm_prefork. This authorization module was using some ugly, but nevertheless working hack which messes with mod_rewrite configuration. This authorization module check user permissions for current url A and rewrites url to AA or AB according to permission check result. Because I didn't want to reimplement mod_rewrite and because mod_rewrite don't have any API for url rewriting, this ugly hack was made. I am changing mod_rewrite configuration on the fly, putting A-AA or A-AB RewriteRule to it. I hope you understood that. Ok. This works. But only in mpm_prefork, where only one execution thread exists in apache process. Now I need same functionality, but in mpm_worker. And I don't have idea how to do it easily. Only two ideas came into my mind: 1) Reimplement needed mod_rewrite functionality in out authorization module. 2) Patch or fork mod_rewrite module. Implement some API (like rewrite(from, to);) which could be used for url rewriting with full mod_rewrite power. 3) Patch mod_rewrite and enclose RewriteRules configuration with locks. Sick! Ugly! What do you think? I am looking forward for comments and advices. Thanks. -- Marko Kevac Sent from Moscow, Mow, Russia
Re: Using mod_rewrite in my authorization module. Need advice.
Thanks a lot for your comments.
I'll try to describe requirements for our authorization module and
current implementation a little bit more.
Requirements:
1. Authorization must happen before mod_cache (also a little bit fixed
for our needs), so it must be executed in 'quick_handler'. This is
because in our architecture, almost every request is cached. Normal
requests and authorized requests. All of them.
2. Authorization configuration should be taken from DB, because it is
changed a lot and restarting Apache every time is stupid thing to do
in this case.
3. Authorization rules should consist of regular expression, which
matches some url and two substitution strings. One for 'granted' and
one for 'denied'. This is because sometimes, when user don't have
appropriate rights, we should send him same data, but maybe slightly
different. For example real time data and 20 minutes delayed data.
Sometimes we should send him to 401, sometimes to 403. It depends on
which url user is accessing.
Current implementation:
1. We keep authorization rules (consisting of url regexp and two
substitution strings) in DB.
2. In apache.conf there are stub RewriteRule, which is dynamically
substituted with new one. New one, which consist of url regexp A and
substitute string AA or AB according to authorization result.
3. User Request - Our Authorization module (authorization check +
mod_rewrite configuration dynamically changed) - mod_cache -
mod_rewrite substitution with our dynamic rule - mod_proxy or
anything else
This is working great in mpm_prefork where only one thread at a time
reads mod_rewrite configuration, but will not work in mpm_worker,
where many threads read mod_rewrite configuration at the same time.
I hope it's more clear now. Our architecture is NOT standard. So many
things can be unusual. Let's not dig too deeply here :-)
Most non standardiness is in the way we are caching things. We are
caching mostly everything, including authorized requests. So
authorization must happen before mod_cache.
Now i'll try to answer on each comment.
On Thu, Feb 4, 2010 at 12:44 AM, Sorin Manolache sor...@gmail.com wrote:
I think it would be clearer and we could help you better if you pasted
some snippets of your code and hack.
I hope it is clearer now, but i'll paste snippets if anything is not
enough clear.
I didn't really understand what you wanted to do. And I don't really
understand why mono-threading would break your hack. Normally a
request is processed by a single thread anyway.
Mono threading (mpm_prefork) will not break it. Multi threading
(mpm_worker) will, because more than one threat at a time reads
mod_rewrite configuration, which I am messing up with.
But look what you could do in your authentication module: You could
set an apache request note (or an environment variable) depending on
the authentication. And then call ap_run_translate_name that
re-invokes the rewrite rules.
Something like
determine if the user is priviledged or not
apr_table_set(r-notes, auth_status, priviledged_user ? yes : no);
ap_run_translate_name(r);
return OK;
and then in the configuration file you put
RewriteCond %{ENV:auth_status} yes
RewriteRule url /destination_priviledged
RewriteCond %{ENV:auth_status} no
RewriteRule url /destination_ordinary
The first time the translate_name is called, the auth_status note is
not defined. Thus no rewrite rule is applied and the URL is unchanged.
Next apache executes the request authentication hook from where you
invoke again the translate_name callback.
I could do that if our rules are kept in apache.conf, but they are
not. They are kept in database.
I cant keep them in apache.conf, because each change will require
apache restart.
By the way. The order in which the callbacks are called is
translate_name, then authentication/authorisation and last fixups.
Our authorization module is not standard authorization module, so our
order of execution is another.
- authorization - cache - translate_name - fixups
--
Marko Kevac
Generating a lot of data in content handler
Hello. I am doing a lot of ap_rprintf(r, ...) in my content handler, generating more than tens of megabytes of data per request. I suppose all this data is kept in r-pool, which is freed after request. So if one user consume 20 MiB, hundred will consume 2 GiB. How can I generate a lot of data without keeping all that data in r-pool till the end of request? -- Marko Kevac
Re: Probably OT. Difficulty implementing content handler Apache2
Check for segmentation faults in global error-log. -- Marko Kevac Gilda Radner - Adopted kids are such a pain - you have to teach them how to look like you. - http://www.brainyquote.com/quotes/authors/g/gilda_radner.html
Memory and pools
Hello Is memory (RSS) returned to OS after cleaning pool (for example r-pool)? After destroying pool? My module uses a lot of memory (~100 MiB). This memory is taken from r-pool. But after request is finished, this memory is not returned to OS - RSS is 100 MiB. Linux 2.6 - httpd-2.2.10 -- Marko Kevac
Re: build mod_proxy by source
apxs -c -o mod_proxy.so mod_proxy.c proxy_util.c On Mon, Jun 22, 2009 at 11:00 PM, h iroshan iroshanm...@gmail.com wrote: Hi All, I need to build mod_proxy by source rather than enable in the configuration. I dont know how to build it by apxs as it has two dependent files (proxy_util.c and mod_proxy.c) .Please help me to over come this problem. Best Regards, Iroshan Under Graduate-UCSC Sri Lanka -- Marko Kevac
Re: build mod_proxy by source
Look: $ ~/micex/opt/httpd-worker/bin/apxs -c -o mod_proxy.so mod_proxy.c proxy_util.c /home/marko/micex/opt/httpd-worker/build/libtool --silent --mode=compile gcc -prefer-pic -g -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -pthread -I/home/marko/micex/opt/httpd-worker/include -I/home/marko/micex/opt/httpd-worker/include -I/home/marko/micex/opt/httpd-worker/include -c -o mod_proxy.lo mod_proxy.c touch mod_proxy.slo /home/marko/micex/opt/httpd-worker/build/libtool --silent --mode=compile gcc -prefer-pic -g -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -pthread -I/home/marko/micex/opt/httpd-worker/include -I/home/marko/micex/opt/httpd-worker/include -I/home/marko/micex/opt/httpd-worker/include -c -o proxy_util.lo proxy_util.c touch proxy_util.slo /home/marko/micex/opt/httpd-worker/build/libtool --silent --mode=link gcc -o mod_proxy.la -rpath /home/marko/micex/opt/httpd-worker/modules -module -avoid-versionproxy_util.lo mod_proxy.lo $ ls -la .libs/mod_proxy.so -rwxr-xr-x 1 marko marko 177683 2009-06-23 01:30 .libs/mod_proxy.so On Tue, Jun 23, 2009 at 12:14 AM, h iroshan iroshanm...@gmail.com wrote: hi Kevac Marko, apxs -c -o mod_proxy.so mod_proxy.c proxy_util.c above command not generate mod_proxy.so . Please help me Regards Iroshan -- Marko Kevac
Re: mod_proxy_balancer dynamic change properties
On Wed, Jun 17, 2009 at 12:39 PM, h iroshaniroshanm...@gmail.com wrote: Hi kevac Marko, Thank you very much .I just want to do the same thing here. My workers are Tomcat and I am going to change lbfactor according to there Memory usage.Can we use normal socket connection to retrieve information from the workers or is there any simple way to communicate with worker node?. If you are using Java stuff, than you should look at mod_cluster. We read load information from shared memory. Load information is written there via separate program. -- Marko Kevac
Re: mod_proxy_balancer dynamic change properties
Yes, you can do that. We wrote module that changes loadfactor in accordance to machine load. On Wed, Jun 17, 2009 at 12:12 AM, h iroshaniroshanm...@gmail.com wrote: Hi Rüdiger, thanks lot , I already followed the source code of the mod_cluster. Its uses another shared memory implementation called slotmem rather than using scorebord. Are you sure is it possible to change this scorebord lbfactor property dynamically and it will affect the request distribution according to that dynamically updated lbfactor value.Please help me. Regards Iroshan 4th year under graduate in computer science University Of Colombo Sri Lanka -- Marko Kevac
Re: mod_proxy. mod_proxy_balancer and failovers
So, do we have any conclusion? Should I make modifications and post patch? -- Marko Kevac
Re: mod_proxy. mod_proxy_balancer and failovers
On Thu, Jun 4, 2009 at 4:32 PM, Jim Jagielski j...@jagunet.com wrote: Sure... no guarantees that they'll be applied, but it would be useful to see what you have in mind. Huh. I thought it would be easy as fixing just mod_proxy.c, but... It occurred that mod_proxy_http thinks that, for example, HTTP_INTERNAL_SERVER_ERROR is not an error if body was returned. So for mod_proxy_http only clean errors (without body) are true errors. Did i understood code right? Was that done on purpose? Why? -- Marko Kevac
Re: Making HTTP requests
I am using CURL. On Mon, Jun 1, 2009 at 9:01 PM, Michael Spiegle m...@nauticaltech.com wrote: Is there a proper/defined way to use mod_proxy for this, or should I just move forward with cURL? -- Marko Kevac
mod_proxy. mod_proxy_balancer and failovers
Hello. I am using mod_proxy and mod_proxy_balancer for proxying requests to group of workers. Recentry I have stumbled upon issues with failover retries when worker returns error. For example I have balancer with 10 equal workers. One of them breaks and for 1 of 10 requests return HTTP_NOT_FOUND. I am not able to tell or configure either mod_proxy or mod_proxy_balancer to make request to another worker if previous one returns HTTP_NOT_FOUND. I think that this should be pretty demanding feature. What do you think? I am willing to fix mod_proxy and to implement this feature. By the way nginx has it. -- Marko Kevac
Re: mod_proxy. mod_proxy_balancer and failovers
On Tue, Jun 2, 2009 at 3:49 PM, Plüm, Rüdiger, VF-Group ruediger.pl...@vodafone.com wrote: A 404 is no error regarding balancing and a proper HTTP response by the backend. You need to send back a HTTP_INTERNAL_ERROR or a HTTP_SERVICE_UNAVAILABLE in this case but without any error page. Who said that? :-) Is it in RFC or something? Provide mechanism, not policy. (c) -- Marko Kevac
Re: mod_proxy. mod_proxy_balancer and failovers
On Tue, Jun 2, 2009 at 3:49 PM, Plüm, Rüdiger, VF-Group ruediger.pl...@vodafone.com wrote: A 404 is no error regarding balancing and a proper HTTP response by the backend. You need to send back a HTTP_INTERNAL_ERROR or a HTTP_SERVICE_UNAVAILABLE in this case but without any error page. Maybe I should describe our position more deeply. Imagine one have 1000 workers and something breaks in one of them. This broken worker somethimes returns BLA_BLA_HTTP_ERROR. Yes, ideally this worker should be found and it should be fixed, but it does not mean that user should see BLA_BLA_HTTP_ERROR, when there are 999 workers in good shape. -- Marko Kevac
Re: mod_proxy. mod_proxy_balancer and failovers
On Tue, Jun 2, 2009 at 4:15 PM, Nick Kew n...@webthing.com wrote: How is the balancer supposed to know that one valid HTTP response is an error while another is correct? A 404 or other error response doesn't mean there's a problem with the server! These HTTP responses can be configured in httpd.conf. Something like: PROXY_BALANCER_RETRY 404 PROXY_BALANCER_RETRY 405 PROXY_BALANCER_RETRY 406 Yes, you are right, 404 does not mean that there is an error, but it could. I'd rather try asking few more workers (attempt limit) before returning error, than return right now error. Feature that I am talking about gives freedom. It should be very useful when creating high availability systems. Right now mod_proxy declares policy that only theeese errors are true errors and only if getting these we should retry. I think it is not right. -- Marko Kevac
Re: mod_proxy. mod_proxy_balancer and failovers
On Tue, Jun 2, 2009 at 4:15 PM, Nick Kew n...@webthing.com wrote: How is the balancer supposed to know that one valid HTTP response is an error while another is correct? A 404 or other error response doesn't mean there's a problem with the server! Sorry, didn't fully understood your question. List of errors should be configured in httpd.conf. If request returns these errors, we should try making requests to another workers $retry_limit$ times. $retry_limit$ already exists in mod_proxy, but we can introduce separate limit for each configured http error. -- Marko Kevac
Re: mod_proxy. mod_proxy_balancer and failovers
On Tue, Jun 2, 2009 at 6:08 PM, Rainer Jung rainer.j...@kippdata.de wrote: You have chosen an especially problematic example. If a not found triggers error status for the worker, then you will have continuous denial of service. Nearly every productive web server I know of, has every now and then (most of them more often) a not found in the logs. Examples are people don't have a favourites icon, have broken links, broken proxies in between, the usual attacker URLs e.g. for mysqladmin etc. Any random URL I send to your web server will trigger a not found. You said make it configurable. I doubt, that you will find many cases, that are really worth making them configurable, i.e. they happen in real world with a broken backend, but still can be used as a stable indicator for a broken backend. Don't stuck on 404. It can be any other HTTP error. 403, 401, BAD_REQUEST, anything... I know that situation that I am talking about don't happen very often, but it can happen. Architectors of high availability systems can experience such problem. In that case they can patch mod_proxy (which we will probably do if proposition can't find place in apache httpd) of swich to another proxy server. Feature I am proposing will allow users to change behaviour of proxy balancer and at the same time it will not ruin anything that exists by now. -- Marko Kevac
Re: [PATCH] mod_dbd: execute SQL statements after DB connection has been established
Anything new? -- Marko Kevac
Re: [PATCH] mod_dbd: execute SQL statements after DB connection has been established
On Mon, Apr 27, 2009 at 11:50 AM, Nick Kew n...@webthing.com wrote: What, since http://svn.apache.org/viewvc?view=revrevision=766938 ? Wow. I didn't know about that. Thanks. Than i will go with mod_dbd pool patches soon. -- Marko Kevac
Re: [PATCH] mod_dbd: execute SQL statements after DB connection has been established
Is this now looking good, Nick? -- Marko Kevac
How to parse POST arguments?
Hello 1. How to parse POST arguments? 2. Why there is no functions to parse GET and POST arguments in Apache? Thanks. -- Marko Kevac
Re: [PATCH] mod_dbd: execute SQL statements after DB connection has been established
On Tue, Apr 14, 2009 at 1:16 AM, Nick Kew n...@webthing.com wrote: On 12 Apr 2009, at 16:36, Kevac Marko wrote: https://issues.apache.org/bugzilla/show_bug.cgi?id=46827 This patch adds ability to the mod_dbd Apache module to execute any SQL statements after DB connection have been established. Thanks for that. I think it's looking good. Couple of points: 1. You've exposed dbd_cfg_t in mod_dbd.h - why? Yes. Because of line APR_DECLARE_EXTERNAL_HOOK(dbd, AP, apr_status_t, post_connect, (apr_pool_t *, dbd_cfg_t *, ap_dbd_t *)); in mod_dbd.h dbd_dfg_t was defined in mod_dbd.c thus invisible from mod_dbd.h 2. +DBD_DECLARE_NONSTD(void) ap_dbd_sql_init(server_rec *s, const char *query) Fixed. It was oversight. https://issues.apache.org/bugzilla/show_bug.cgi?id=46827 -- Marko Kevac
[PATCH] mod_dbd: execute SQL statements after DB connection has been established
https://issues.apache.org/bugzilla/show_bug.cgi?id=46827
This patch adds ability to the mod_dbd Apache module to execute any
SQL statements after DB connection have been established.
Patch is against httpd trunk.
Configuration example:
DBDriver mysql
DBDParams host=localhost,user=root,reconnect=0
DBDPersist On
DBDInitSQL USE passport
DBDInitSQL SET NAMES utf8
DBDMin 1
Order of the DBDInitSQL statements is preserved (statements will be
executed in same order they were in httpd.conf file).
Also this patch introduces optional post_connection hook, which
enables any other module to hook after connection were established.
--
Marko Kevac
--- modules/database/mod_dbd.h 2009-03-07 15:52:57.0 +0300
+++ /home/knight/micex/git/dbd_sql_init/mod_dbd.h 2009-04-12 18:57:00.0 +0400
@@ -55,6 +55,23 @@
#include httpd.h
#include apr_optional.h
#include apr_hash.h
+#include apr_hooks.h
+
+typedef struct {
+server_rec *server;
+const char *name;
+const char *params;
+int persist;
+#if APR_HAS_THREADS
+int nmin;
+int nkeep;
+int nmax;
+int exptime;
+int set;
+#endif
+apr_hash_t *queries;
+apr_array_header_t *init_queries;
+} dbd_cfg_t;
typedef struct {
apr_dbd_t *handle;
@@ -98,6 +115,9 @@
APR_DECLARE_OPTIONAL_FN(ap_dbd_t*, ap_dbd_cacquire, (conn_rec*));
APR_DECLARE_OPTIONAL_FN(void, ap_dbd_prepare, (server_rec*, const char*, const char*));
+APR_DECLARE_EXTERNAL_HOOK(dbd, AP, apr_status_t, post_connect,
+ (apr_pool_t *, dbd_cfg_t *, ap_dbd_t *));
+
#endif
/** @} */
--- modules/database/mod_dbd.c 2009-03-07 15:52:57.0 +0300
+++ /home/knight/micex/git/dbd_sql_init/mod_dbd.c 2009-04-12 18:58:17.0 +0400
@@ -39,6 +39,11 @@
extern module AP_MODULE_DECLARE_DATA dbd_module;
+APR_IMPLEMENT_OPTIONAL_HOOK_RUN_ALL(dbd, AP, apr_status_t, post_connect,
+(apr_pool_t *pool, dbd_cfg_t *cfg,
+ap_dbd_t *dbd),
+(pool, cfg, dbd), OK, DECLINED)
+
/ svr cfg: manage db connection pool /
#define NMIN_SET 0x1
@@ -46,21 +51,6 @@
#define NMAX_SET 0x4
#define EXPTIME_SET 0x8
-typedef struct {
-server_rec *server;
-const char *name;
-const char *params;
-int persist;
-#if APR_HAS_THREADS
-int nmin;
-int nkeep;
-int nmax;
-int exptime;
-int set;
-#endif
-apr_hash_t *queries;
-} dbd_cfg_t;
-
typedef struct dbd_group_t dbd_group_t;
struct dbd_group_t {
@@ -99,6 +89,8 @@
#define DEFAULT_NMAX 10
#define DEFAULT_EXPTIME 300
+#define DEFAULT_SQL_INIT_ARRAY_SIZE 5
+
static void *create_dbd_config(apr_pool_t *pool, server_rec *s)
{
svr_cfg *svr = apr_pcalloc(pool, sizeof(svr_cfg));
@@ -115,6 +107,8 @@
cfg-exptime = DEFAULT_EXPTIME;
#endif
cfg-queries = apr_hash_make(pool);
+cfg-init_queries = apr_array_make(pool, DEFAULT_SQL_INIT_ARRAY_SIZE,
+ sizeof(const char *));
return svr;
}
@@ -137,10 +131,33 @@
new-exptime = (add-setEXPTIME_SET) ? add-exptime : base-exptime;
#endif
new-queries = apr_hash_overlay(pool, add-queries, base-queries);
+new-init_queries = apr_array_append(pool, add-init_queries,
+ base-init_queries);
return svr;
}
+DBD_DECLARE_NONSTD(void) ap_dbd_sql_init(server_rec *s, const char *query)
+{
+svr_cfg *svr;
+const char **arr_item;
+
+svr = ap_get_module_config(s-module_config, dbd_module);
+if (!svr) {
+ /* some modules may call from within config directive handlers, and
+ * if these are called in a server context that contains no mod_dbd
+ * config directives, then we have to create our own server config
+ */
+ svr = create_dbd_config(config_pool, s);
+ ap_set_module_config(s-module_config, dbd_module, svr);
+}
+
+if (query) {
+arr_item = apr_array_push(svr-cfg-init_queries);
+*arr_item = query;
+}
+}
+
static const char *dbd_param(cmd_parms *cmd, void *dconf, const char *val)
{
const apr_dbd_driver_t *driver = NULL;
@@ -244,6 +261,17 @@
return NULL;
}
+static const char *dbd_init_sql(cmd_parms *cmd, void *dconf, const char *query)
+{
+if (!query || *query == '\n') {
+return You should specify SQL statement;
+}
+
+ap_dbd_sql_init(cmd-server, query);
+
+return NULL;
+}
+
static const command_rec dbd_cmds[] = {
AP_INIT_TAKE1(DBDriver, dbd_param, (void*)cmd_name, RSRC_CONF,
SQL Driver),
@@ -254,6 +282,8 @@
AP_INIT_TAKE12(DBDPrepareSQL, dbd_prepare, NULL, RSRC_CONF,
SQL statement to prepare (or nothing, to override
statement inherited from main server) and label),
+AP_INIT_TAKE1(DBDInitSQL, dbd_init_sql, NULL, RSRC_CONF,
+
Re: AP_FTYPE_PROTOCOL before AP_FTYPE_CONTENT_SET sometimes in 2.2.10
Fixed. It's because i didn't called ap_remove_output_filter(f). Why should I call this from within filter itself? On Wed, Apr 1, 2009 at 5:48 PM, Kevac Marko ma...@kevac.org wrote: Hello. ap_http_header_filter (AP_FTYPE_PROTOCOL) sometimes executed before my AP_FTYPE_CONTENT_SET filter. Any clue how that can happen? As a result, my output header in not added. 2.2.10 -- Marko Kevac -- Marko Kevac
AP_FTYPE_PROTOCOL before AP_FTYPE_CONTENT_SET sometimes in 2.2.10
Hello. ap_http_header_filter (AP_FTYPE_PROTOCOL) sometimes executed before my AP_FTYPE_CONTENT_SET filter. Any clue how that can happen? As a result, my output header in not added. 2.2.10 -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
Nick, any news? On Fri, Mar 13, 2009 at 3:54 PM, Nick Kew n...@webthing.com wrote: Thanks. I'll test-drive today. -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
On Tue, Mar 24, 2009 at 10:03 PM, Ronald Park rkpar...@gmail.com wrote: A user might presume that the order of commands listed in the config file would be preserved when the commands are run. By using a hash table, you'll likely get a random order causing problems. For example, perhaps the user needs to first load one table then choose a subset from that first table to file another; if you run them in the reserve order, the second table will likely be empty. :( Thank you. I will implement this. -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
On Fri, Mar 13, 2009 at 2:54 PM, Nick Kew n...@webthing.com wrote: Thanks. I'll test-drive today. Succeeded? -- Marko Kevac
mod_dbd and lost of connection to server
Hello. If our server connection is persisted, than connection is created right away and kept under the hood. If something happens to this connection (server timeout for example) than acquire will return NULL first time. Second time it will recreate connection. Maybe it is better to recreate this connection right away in mod_dbd? Thoughts? -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
On Tue, Mar 10, 2009 at 10:45 PM, Nick Kew n...@webthing.com wrote: Can I throw an alternative suggestion into the ring. Instead of running dbd_init_sql_init at the end of dbd_construct, run a hook there. Your function then runs on that hook, but it enables other modules to do their own thing too. I think it's a mistake in the current mod_dbd that dbd_prepared_init isn't hooked, and having the hook could've helped with some of the bugs we've seen. While you're hacking it would seem like a good opportunity to rectify all that! https://issues.apache.org/bugzilla/show_bug.cgi?id=46827 Implemented. Patch added. Comments? -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
Ok, here is sql init statement only patch against trunk: https://issues.apache.org/bugzilla/show_bug.cgi?id=46827 P.S. Thanks http://jukka.zitting.name/git/ for git mirrors. I am happy :-) P.P.S. Probably I should wait for sql init statement only patch inclusion before posting everything else? -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
On Fri, Mar 6, 2009 at 7:58 PM, Chris Darroch chr...@pearsoncmg.com wrote: One thought I had overnight is that you might, if you like, want to tackle the init SQL patch first, which seems unrelated to the multiple pools idea and significantly simpler, easier to review, and probably easier to backport as well. Done. Can you provide an example of how such initialization SQL queries would be used? And, as a question, might it be necessary for such queries to run before any other SQL queries are prepared? IIRC in the patch they are run after the other queries are prepared, but if they were truly required for connection initialization, they'd need to be absolutely first, no? Prepared statements are not executed, just parsed (simplified), so no, it is not important for initialization statement to be before prepared statements. -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
On Thu, Mar 5, 2009 at 10:05 PM, Chris Darroch chr...@pearsoncmg.com wrote: One difference, I think -- again, after only a quick review -- is that what I was trying to solve with the notion of a DBDGroup container and then a DBDGroup directive was to minimize the number of distinct configuration groups and make configuration as simple as possible. You could put, say, two DBDGroup sets of directives into the main server configuration. Then in a virtual host, you'd just need to say DBDGroup foo to specify which one you wanted to use there. It's great idea. But are you sure that it is good idea to change both main httpd.conf and VirtualHost for new DB connection to be added? These are separate things... It will introduce problems if, for example, httpd.conf is root writable and virtualhost-blabla.conf is user writable. Your DBDPool changes seem targeted at the issue of supporting multiple connections within, say, a single virtual host. I'm not 100% sure if that's the idea, so please correct me if I'm wrong. Then callers who want a particular connection pass the DBD pool name to ap_dbd_open_pool(), etc. Yes. We configure separate connection pools to separate DBses. MySQL, PostgreSQL, SQLite for example on different hosts and give them names. And user can chose connection from which connection pool he wants. This seems like a good enhancement to me, certainly, and we've seen some requests for it on the mailing list. If we're adding such configuration containers, though, I personally would want to ensure that we had a way to minimize the number of distinct connection pools involved. It looks to me as though the logic in dbd_post_config() which aims to do this, i.e., minimize the number of distinct DBD configurations (when using the existing set of directives) is removed in the patch. I may be missing something, but I think that's likely to be necessary in the future as well, to support existing installations and not require administrators to rewrite their configurations after upgrading. I have tried to keep everything concerning group minimization from original mod_dbd. If not - it is bug. I don't really like the way that minimization was implemented, but i have tried to minimize changes to mod_dbd source. Suppose you had many virtual hosts and each one needed one or both of two types of DB connection. You'd want to be able to specify the two types of connection in just two DBDPool or DBDGroup containers at the main server level, it seems to me. Then each virtual host could refer back to the ones it required, if, for instance, it was adding vhost-specific DBD authentication directives. But what if owner of blahblah.host.org needs third connection pool and he can't change httpd.conf? -- Marko Kevac
Re: mod_dbd analogue for memcached
On Thu, Mar 5, 2009 at 6:53 PM, Brian Akins br...@akins.org wrote: Is this based on the existing mod_memcache? After some observation of apr_memcache I have realized that mod_dbd pools are not needed here, because apr_memcache have own pools to each memcache server. So it can be based on existing mod_memcache. For our needs, mod_memcache (http://code.google.com/p/modmemcache/) will need one major change: memcache groups. MemcacheGroup group01 MemcacheServer localhost:1234 MemcacheServer localhost:1235 /MemcacheGroup MemcacheGroup group01 MemcacheServer localhost:1234 MemcacheServer localhost:1235 /MemcacheGroup And memcache API: ap_memcache_client(server_rec *s) ap_memcache_client_group(server_rec *s, char *group_name) As i have already mentioned, we need dynamic memcache servers within groups. That can be implemented as separated module which can take memcache structure from mod_memcache and enable/disable/add/remove needed servers according to some rules. -- Marko Kevac
Re: mod_dbd analogue for memcached
On Tue, Mar 10, 2009 at 8:49 PM, josh rotenberg joshrotenb...@gmail.com wrote: I'm just catching up, but I'd be happy to add some of this functionality to mod_memcache. Great. I think I'll send patch on this week. mod_memcache is not in httpd, so, probably, we should continue our talk somewhere else? Do you have plans to push mod_memcache to httpd-trunk? -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
There are some bugs in non-threaded build. I am fixing them right now. -- Marko Kevac
mod_dbd analogue for memcached
Hello I've created mod_dbd analogue for memcached (named connection pools to memcached servers, see also http://www.mail-archive.com/d...@httpd.apache.org/msg42922.html). It's now based on libmemcached (http://tangent.org/552/libmemcached.html). The question I want to ask is should I base my module on apr_memcache or not? Is apr_memcache mature enough? Whether is is used by someone? Thanks. -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
On Thu, Feb 12, 2009 at 12:56 AM, Nick Kew n...@webthing.com wrote:
Sounds OK in principle for trunk. If you want to post a patch
against trunk, I'll try and find the time to review it.
Here I am again.
The patch works well for us.
Is there something else that I can do for now?
--
Marko Kevac
diff -u trunk/mod_dbd.c mine/mod_dbd.c
--- trunk/mod_dbd.c 2009-03-05 15:19:18.0 +0300
+++ mine/mod_dbd.c 2009-03-05 15:19:44.0 +0300
@@ -20,6 +20,8 @@
* http://apache.webthing.com/database/
*/
+#define CORE_PRIVATE
+
#include apr_reslist.h
#include apr_strings.h
#include apr_hash.h
@@ -46,8 +48,13 @@
#define NMAX_SET 0x4
#define EXPTIME_SET 0x8
+#define DEFAULT_POOL_COUNT 2
+
+#define DBD_DEFAULT_POOL_NAME dbd_default_pool
+
typedef struct {
server_rec *server;
+const char *pool_name;
const char *name;
const char *params;
int persist;
@@ -59,6 +66,7 @@
int set;
#endif
apr_hash_t *queries;
+apr_hash_t *init_queries;
} dbd_cfg_t;
typedef struct dbd_group_t dbd_group_t;
@@ -77,8 +85,8 @@
};
typedef struct {
-dbd_cfg_t *cfg;
-dbd_group_t *group;
+apr_hash_t *cfgs;
+apr_hash_t *groups;
} svr_cfg;
typedef enum { cmd_name, cmd_params, cmd_persist,
@@ -102,79 +110,145 @@
static void *create_dbd_config(apr_pool_t *pool, server_rec *s)
{
svr_cfg *svr = apr_pcalloc(pool, sizeof(svr_cfg));
-dbd_cfg_t *cfg = svr-cfg = apr_pcalloc(pool, sizeof(dbd_cfg_t));
-cfg-server = s;
-cfg-name = no_dbdriver; /* to generate meaningful error messages */
-cfg-params = ; /* don't risk segfault on misconfiguration */
-cfg-persist = -1;
-#if APR_HAS_THREADS
-cfg-nmin = DEFAULT_NMIN;
-cfg-nkeep = DEFAULT_NKEEP;
-cfg-nmax = DEFAULT_NMAX;
-cfg-exptime = DEFAULT_EXPTIME;
-#endif
-cfg-queries = apr_hash_make(pool);
+svr-cfgs = apr_hash_make(pool);
+svr-groups = apr_hash_make(pool);
return svr;
}
-static void *merge_dbd_config(apr_pool_t *pool, void *basev, void *addv)
+DBD_DECLARE_NONSTD(void) ap_dbd_sql_init(server_rec *s, const char *pool_name,
+const char *query, const char *label)
{
-dbd_cfg_t *base = ((svr_cfg*) basev)-cfg;
-dbd_cfg_t *add = ((svr_cfg*) addv)-cfg;
-svr_cfg *svr = apr_pcalloc(pool, sizeof(svr_cfg));
-dbd_cfg_t *new = svr-cfg = apr_pcalloc(pool, sizeof(dbd_cfg_t));
+svr_cfg *svr;
+dbd_cfg_t *cfg = NULL;
-new-server = add-server;
-new-name = (add-name != no_dbdriver) ? add-name : base-name;
-new-params = strcmp(add-params, ) ? add-params : base-params;
-new-persist = (add-persist != -1) ? add-persist : base-persist;
-#if APR_HAS_THREADS
-new-nmin = (add-setNMIN_SET) ? add-nmin : base-nmin;
-new-nkeep = (add-setNKEEP_SET) ? add-nkeep : base-nkeep;
-new-nmax = (add-setNMAX_SET) ? add-nmax : base-nmax;
-new-exptime = (add-setEXPTIME_SET) ? add-exptime : base-exptime;
-#endif
-new-queries = apr_hash_overlay(pool, add-queries, base-queries);
+svr = ap_get_module_config(s-module_config, dbd_module);
+if (!svr) {
+ /* some modules may call from within config directive handlers, and
+ * if these are called in a server context that contains no mod_dbd
+ * config directives, then we have to create our own server config
+ */
+ svr = create_dbd_config(config_pool, s);
+ ap_set_module_config(s-module_config, dbd_module, svr);
+}
-return svr;
+cfg = apr_hash_get(svr-cfgs, pool_name, APR_HASH_KEY_STRING);
+if (NULL == cfg) {
+ap_log_error(APLOG_MARK, APLOG_CRIT, 0, s,
+could not found dbd configuration for pool %s, pool_name);
+return;
+}
+
+if (apr_hash_get(cfg-init_queries, label, APR_HASH_KEY_STRING)
+ strcmp(query, )) {
+ap_log_error(APLOG_MARK, APLOG_WARNING, 0, s,
+conflicting SQL statements with label %s, label);
+}
+
+apr_hash_set(cfg-init_queries, label, APR_HASH_KEY_STRING, query);
+}
+
+static const char *dbd_pool(cmd_parms *cmd, void *mconfig, const char *arg)
+{
+const char *errmsg = NULL;
+const char *endp = ap_strrchr_c(arg, '');
+
+dbd_cfg_t *conf;
+
+ap_conf_vector_t *new_dir_conf = ap_create_per_dir_config(cmd-pool);
+
+const char *err = ap_check_cmd_context(cmd,
+ NOT_IN_DIR_LOC_FILE|NOT_IN_LIMIT);
+
+if (err != NULL) {
+return err;
+}
+
+if (endp == NULL) {
+return apr_pstrcat(cmd-pool, cmd-cmd-name,
+directive missing closing '', NULL);
+}
+
+arg = apr_pstrndup(cmd-pool, arg, endp - arg);
+
+/* initialize our config and fetch it */
+conf = ap_set_config_vectors(cmd-server, new_dir_conf, arg,
+ dbd_module, cmd-pool);
+
+errmsg = ap_walk_config(cmd-directive-first_child, cmd, new_dir_conf);
+if (errmsg != NULL)
+return errmsg;
+
+return NULL;
}
mod_dbd analogue for memcached
Hello I've created mod_dbd analogue for memcached (named connection pools to memcached servers, see also http://www.mail-archive.com/dev@httpd.apache.org/msg42922.html). It's now based on libmemcached (http://tangent.org/552/libmemcached.html). The question I want to ask is should I base my module on apr_memcache or not? Is apr_memcache mature enough? Whether is is used by someone? Thanks. -- Marko Kevac
Re: mod_dbd analogue for memcached
On Thu, Mar 5, 2009 at 6:53 PM, Brian Akins br...@akins.org wrote: Is this based on the existing mod_memcache? No. It is based on mod_dbd. New one (http://www.mail-archive.com/dev@httpd.apache.org/msg42922.html) to be precise. mod_memcache, if we are talking about http://code.google.com/p/modmemcache/, is too simple. I need multiple name pools to multiple servers. Something like MemcachePool name1 Server 192.168.0.1:11211 Server 192.168.0.2:11211 Server 192.168.0.3:11211 Server 192.168.0.4:11211 Server 192.168.0.5:11211 Other options... /MemcachePool MemcachePool name2 Server 192.168.0.6:11211 Server 192.168.0.7:11211 Server 192.168.0.8:11211 Server 192.168.0.9:11211 Server 192.168.0.10:11211 Other options... /MemcachePool Also I will implement hook in module. Hooked function can filter servers list. For example (in python pseudo code): def dead_or_alive(servers_list): new_list = [] for server in servers_list: if server is in some list in shared memory, it is alive and it is not overloaded then: new_list.add(server) return new_list If returned list differs from saved one, than we recreate connections (maybe something smarter). This feature enables us to create dynamic, not static servers list. For high availability clusters for example. Alright, i will ty to base it on apr_memcache then. -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
Chris, thank you. I am impressed. I'm not experienced in commiting patches to open source projects, so I am very thankful for your comments. I'll try to fix things up very soon. 80 columns is a little bit ancient requirement in the world of 22 LCDs, but ok, i'll fix that too. -- Marko Kevac
Re: mod_dbd analogue for memcached
On Fri, Mar 6, 2009 at 2:02 AM, M. Brian Akins br...@akins.org wrote: Will you implement a default pool like mod_dbd does? Yes This feature enables us to create dynamic, not static servers list. For high availability clusters for example. Sounds interesting, I suppose. I just generate my configs from templates and they generate the correct server list based on whatever And after that you have to restart apache? It's not enough dynamic for us. We are dynamically changing memcached servers list without Apache restart. Based on information provided by small network daemon via shared memory (you can use SNMP here as well). It allows us to dynamically and on the fly add new cluster nodes with memcached instances or remove ones that we don't need. -- Marko Kevac
Re: [PATCH] mod_dbd with more than one pool
Now for every exported function we have pair of functions that accepts
pool_name and old one, which is just wrapper:
DBD_DECLARE_NONSTD(ap_dbd_t*) ap_dbd_open_pool(apr_pool_t *pool, server_rec *s,
const char *pool_name);
DBD_DECLARE_NONSTD(ap_dbd_t*) ap_dbd_open(apr_pool_t *pool, server_rec *s);
APR_DECLARE_OPTIONAL_FN(ap_dbd_t*, ap_dbd_open_pool, (apr_pool_t*,
server_rec*, const char*));
APR_DECLARE_OPTIONAL_FN(ap_dbd_t*, ap_dbd_open, (apr_pool_t*, server_rec*));
DBD_DECLARE_NONSTD(ap_dbd_t*) ap_dbd_open(apr_pool_t *pool, server_rec *s)
{
return ap_dbd_open_pool(pool, s, DBD_DEFAULT_POOL_NAME);
}
In httpd.conf you can create named pool inside DBDPool pool_name or
without. In second case pool_name is DBD_DEFAULT_POOL_NAME.
Thus old functions and old configuration is preserved.
What so you think?
Patch is ready, but it needs some testing before posting.
Re: [PATCH] mod_dbd with more than one pool
On Thu, Feb 12, 2009 at 12:43 AM, Brian Akins br...@akins.org wrote: I was looking to do the same thing to mod_memcache (which should be imported into trunk, IMO...) kni...@juffin:~/micex/git/apache$ tree modules/memcache/ modules/memcache/ |-- SConscript |-- mod_memcache.c `-- mod_memcache.h Same as mod_dbd :-) -- Marko Kevac
Re: Profiling Apache
On Wed, Jan 28, 2009 at 9:25 AM, Paras Fadte plf...@gmail.com wrote: Which tool is the most suitable for profiling Apache ? gprof ? Oprofile ? Used valgrind and gpt. -- Marko Kevac
Regular expression matching for multi line strings...
Hello.
ap_regexec() do not match multi line strings without AP_REG_NEWLINE
flag, but it should.
For example i have string:
SELECT\r\nstate.SECNAME,\r\nstate.SHORTNAME,\r\n
state.LATNAME,\r\nstate.DECIMALS,\r\nstate.PREVPRICE,\r\n
realtime.BOARDID as BOARDID,\r\nrealtime.SECID as SECID,\r\n
realtime.STATUS as STATUS,\r\nrealtime.BID as BID,\r\n
realtime.BIDDEPTH as BIDDEPTH,\r\nrealtime.BIDDEPTHT as
BIDDEPTHT,\r\nrealtime.NUMBIDS as NUMBIDS,\r\nrealtime.OFFER
as OFFER,\r\nrealtime.OFFERDEPTH as OFFERDEPTH,\r\n
realtime.OFFERDEPTHT as OFFERDEPTHT,\r\nrealtime.NUMOFFERS as
NUMOFFERS,\r\nrealtime.OPEN as OPEN,\r\nrealtime.HIGH as
HIGH,\r\nrealtime.LOW as LOW,\r\nrealtime.LAST as LAST,\r\n
realtime.`CHANGE` as `CHANGE`,\r\n
ROUND(realtime.LASTCHANGEPRCNT,state.DECIMALS) AS LASTCHANGEPRCNT,
\r\nrealtime.LASTCNGTOLASTWAPRICE as LASTCNGTOLASTWAPRICE,\r\n
realtime.QTY as QTY,\r\nrealtime.TIME as TIME,\r\n
realtime.VOLTODAY as VOLTODAY,\r\nrealtime.VALTODAY as
VALTODAY,\r\nrealtime.VALTODAY_USD as VALTODAY_USD,\r\n
realtime.TRADINGSTATUS as TRADINGSTATUS,\r\nrealtime.VALUE as
VALUE,\r\nrealtime.VALUE_USD as VALUE_USD,\r\nrealtime.WAPRICE
as WAPRICE,\r\nrealtime.WAPTOPREVWAPRICEPRCNT as
WAPTOPREVWAPRICEPRCNT,\r\nrealtime.WAPTOPREVWAPRICE as
WAPTOPREVWAPRICE,\r\nrealtime.HIGHBID as HIGHBID,\r\n
realtime.LOWOFFER as LOWOFFER,\r\nrealtime.NUMTRADES as
NUMTRADES,\r\nrealtime.PRICEMINUSPREVWAPRICE as
PRICEMINUSPREVWAPRICE,\r\nrealtime.CLOSEPRICE as CLOSEPRICE,\r\n
realtime.LASTBID as LASTBID,\r\nrealtime.LASTOFFER as
LASTOFFER,\r\nrealtime.MARKETPRICE as MARKETPRICE,\r\n
realtime.MARKETPRICETODAY as MARKETPRICETODAY,\r\n
realtime.LOPENPRICE as LOPENPRICE,\r\nrealtime.LCURRENTPRICE as
LCURRENTPRICE,\r\nrealtime.LCLOSEPRICE as LCLOSEPRICE,\r\n
realtime.MARKETPRICE2 as MARKETPRICE2,\r\nrealtime.ADMITTEDQUOTE
as ADMITTEDQUOTE,\r\nrealtime.OPENPERIODPRICE as
OPENPERIODPRICE,\r\nrealtime.UPDATETIME as UPDATETIME,\r\n
realtime.SEQNUM as SEQNUM,\r\nrealtime.SPREAD as SPREAD,\r\n
realtime.LASTCHANGE as LASTCHANGE,\r\nrealtime.LASTTOPREVPRICE as
LASTTOPREVPRICE,\r\nrealtime.SYSTIME as SYSTIME,\r\n
'%table_type%' as TABLE_TYPE\r\n\r\nFROM \r\n\r\n
online.stock_shares_stat_info state, \r\n
online.stock_shares_%table_type% realtime, \r\n
root_db_dev.securities securities,\r\n
root_db_dev.securities_to_security_collections sectocollection,\r\n
root_db_dev.security_collections collections,\r\n root_db_dev.boards
boards\r\n\r\nWHERE (\r\n state.BOARDID=realtime.BOARDID\r\n AND
\r\n state.SECID=realtime.SECID \r\n AND \r\n
securities.SECID=state.SECID \r\n AND \r\n
sectocollection.SECURITY_ID=securities.ID \r\n AND \r\n
sectocollection.SECURITY_COLLECTION_ID=collections.ID \r\n AND\r\n
boards.BOARDID=state.BOARDID\r\n AND \r\n
boards.board_group_id=?\r\n AND\r\n collections.ID=?\r\n AND\r\n
realtime.SEQNUM?\r\n AND\r\n realtime.NUMTRADES=?\r\n)
\r\n\r\norder by %sort_order% %sort_order_desc%\r\n\r\nLIMIT ?,?
And i want to change every %% to something else.
So i wrote small function:
char *utilSubstringSubstitute(apr_pool_t *pool, char *string, char
delimiter, char *substitute, char *value)
{
ap_regex_t *regex_compiled;
ap_regmatch_t regex_match[AP_MAX_REG_MATCH];
int rv;
char *regex = apr_psprintf(pool, (.*)%c%s%c(.*), delimiter,
substitute, delimiter);
regex_compiled = ap_pregcomp(pool, regex, 0);
if (!regex_compiled) {
return NULL;
}
rv = ap_regexec(regex_compiled, string, AP_MAX_REG_MATCH, regex_match, 0);
if (0 != rv) {
return NULL;
}
return ap_pregsub(pool, apr_psprintf(pool, $1%s$2, value),
string, AP_MAX_REG_MATCH, regex_match);
}
But it gives me
''marko' as TABLE_TYPE\r'
(marko is value argument)
What i am doing wrong?
Thanks.
--
С уважением, Кевац Марко
Sincerely yours, Kevac Marko
Unit testing
Hello. What is the best way to write unit tests for C apache modules? Maybe some frameworks? -- С уважением, Кевац Марко Sincerely yours, Kevac Marko
Re: new modules in trunk
On Mon, Dec 1, 2008 at 11:17 AM, Paul Querna [EMAIL PROTECTED] wrote: mod_heartmonitor: Collects these Multicast heartbeats for other modules to use. Interesting. Is there other modules that create separate thread(s) for bypassing request-response thing? -- С уважением, Кевац Марко Sincerely yours, Kevac Marko
Re: Howto implement a new algorithm scheduler in mod_proxy_balancer
What's monitor hook? On Wed, Oct 22, 2008 at 11:58 PM, Jim Jagielski [EMAIL PROTECTED] wrote: Sounds like 2 questions there: (1) is there anyway to create some sort of scheduler that does periodic functions and (2) how to obtain external server info (like CPU, etc...). Re #1: Under 2.2, there is a monitor hook, but it is pretty limited. Best thing is to create an external process which runs separately and adjusts shared memory data. Re #2: It ain't trivial. On Oct 22, 2008, at 2:38 PM, ricardo13 wrote: hi, I saw the link. So one more question. Can I development a scheduler that receive datas from cluster. For example: If machine1 == overloaded CPU sent requisition to machine2 How to do scheduler (via programming) obtain information of CPU from others machines ?? Thank You Ricardo Rainer Jung-3 wrote: ricardo13 schrieb: I don't find that examples. I find in the folder httpd.2.2.9/modules/proxy/ .there isn't folder example Where can I find ??? Is there any link ??? It's not yet in the stable branch 2.2, but you can find it in the trunk: http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/proxy/examples/ to watch it with a browser, or http://svn.apache.org/repos/asf/httpd/httpd/trunk/modules/proxy/examples/ as a URL to check it out via svn. Regards, Rainer Jim Jagielski wrote: Not sure what the question is... is it how to test your algo or how to implement it? There is a quick-and-dirty example submodule in trunk (under modules/ proxy/examples) that give some basic overview. On Oct 20, 2008, at 10:07 PM, ricardo13 wrote: Hi, I wanted implement a new algorithm scheduler to mod_proxy_balancer. I built a new algorithm and want test in apache2. Thank you Ricardo -- -- View this message in context: http://www.nabble.com/Howto-implement-a-new-algorithm-scheduler-in-mod_proxy_balancer-tp20082207p20117149.html Sent from the Apache HTTP Server - Module Writers mailing list archive at Nabble.com. -- С уважением, Кевац Марко Sincerely yours, Kevac Marko
Re: Access configuration variable in other modules
Though, it seems to imply that I have to export my own module in a header file and include it in mod_cgi. In other words, I have to add my module to the httpd build system. So I copied my module to the modules/generator directory. But I am not sure what the good way is to incorporate my module in the build httpd system; just typing make does not compile my module. Which build-files should I modify to accomplish this? Or you can remove mod_cgi from httpd build system and maintain it with apxs. I do not have answer to your question about how to insert your module to the httpd build system, sorry. I still mantain my fork of mod_dbd separarely. -- С уважением, Кевац Марко Sincerely yours, Kevac Marko
Re: Access configuration variable in other modules
Catch the example:
cache_module = ap_find_linked_module(mod_cache.c);
if (!cache_module) {
ap_log_error(APLOG_MARK, APLOG_ERR, 0, s, ERRTAG %s:
Can't find mod_cache module., __func__);
return;
}
conf_cache =
(cache_server_conf *)ap_get_module_config(
s-module_config, cache_module);
if (!conf_cache) {
ap_log_error(APLOG_MARK, APLOG_ERR, 0, s, ERRTAG %s:
Can't acquire mod_cache server configuration structure.,
__func__);
return;
}
On 9/13/08, Andrej van der Zee [EMAIL PROTECTED] wrote:
Hi,
I am writing my first Apache module for monitoring resource usage of
embedded interpreters and CGI applications on Linux (I am also adapting
mod_cgi and the apr-library). The problem is that my own module has some
server-level configuration options that I can initialize and use in my own
module, but I don't know how to access them in *another* module (i.e.
mod_cgi). What is the way to do this? Any examples of this in the modules
that come with the sources?
Cheers,
Andrej
--
С уважением, Кевац Марко
Sincerely yours, Kevac Marko
What should i do to bring attention to the bug?
Hello. What can i possibly do and what i should do to bring some attention to opened bug? Patch attached, so i need some review and testing, something... I would really like to see this feature in httpd. Bug i am talking about is here: https://issues.apache.org/bugzilla/show_bug.cgi?id=45456 -- С уважением, Кевац Марко Sincerely yours, Kevac Marko
Re: Child init process is not initialized with the data stored by master process
There are main server_rec structure for main server and one server_rec
for each Virtual Host.
So in child_init you should walk all of them:
while (s) {
do_smth();
s = s-next;
}
One of these should keep initialized structure.
On Thu, Jul 24, 2008 at 1:05 PM, lusob [EMAIL PROTECTED] wrote:
Which is the problem?
--
С уважением, Кевац Марко
Sincerely yours, Kevac Marko
Re: mod_dbd and more than one database server
On Thu, Jul 10, 2008 at 7:18 AM, Kevac Marko [EMAIL PROTECTED] wrote: Thank you, but i need pool of connections too. So i think i'll fix mod_dbd to allow more than one connection pool and slightly change api to allow choosing pool of connections. I am planning to implement something like this: DBDPool pool_name_1 DBDriver mysql DBDParam ... ... /DBDPool DBDPool pool_name_1 DBDriver mysql DBDParam ... ... /DBDPool API will require pool name, something like this: ap_dbd_acquire(request_rec*, const char *pool_name); ap_dbd_open(apr_pool_t*, server_rec*, const char *pool_name); Now what about groups... User will be available to use same pool, by giving only pool name and prepare statements: DBDPool name_that_already_exists DBDPrepare ... /DBDPool And if user set some variable, that is different from previous pool declaration: DBDPool name_that_already_exists DBDriver different_from_that_one_in_name_that_already_exists /DBDPool Than user will get something like Your name_that_already_exists definition contradicts with another definition. Please rename your pool, or remove contradicting parameters. What do you think about all that? -- С уважением, Кевац Марко Sincerely yours, Kevac Marko
Re: mod_dbd and more than one database server
Thank you, but i need pool of connections too. So i think i'll fix mod_dbd to allow more than one connection pool and slightly change api to allow choosing pool of connections. On Thu, Jul 10, 2008 at 5:34 AM, Tom Donovan [EMAIL PROTECTED] wrote: If you are writing your own module, you could always open ad-hoc database connections directly using the apr-util functions apr_dbd_get_driver(), apr_dbd_open_ex(), apr_dbd_close(), etc. instead of using the pool of connections provided by mod_dbd. -- С уважением, Кевац Марко Sincerely yours, Kevac Marko
Re: [可能是垃圾邮件] Re: how to catch segment fault
Yes, but it allows you to debug your module with gdb (with or without core file) with ease. Run gdb bin/httpd, print 'r -X'. It will start Apache in single mode (with one process). Than load server somehow and wait for gdb to catch sigfault. It will show you precise line of code where sigfault occurred. On Wed, Jun 25, 2008 at 9:56 AM, wangyz [EMAIL PROTECTED] wrote: thanks your answer. I had set CoreDumpDirector and run command ulimit -c unlimited .then startup server using this command apachect1 start but no any core file still. you said, maybe i will change the startup script. can you tell me how to change startup script -- С уважением, Кевац Марко Sincerely yours, Kevac Marko
Re: how to catch segment fault
Try the same you were doing, but build Apache with -g. CCFLAGS=-g ./configure make make install On Wed, Jun 25, 2008 at 5:44 AM, langwnwk [EMAIL PROTECTED] wrote: Eric Covener wrote: On Tue, Jun 24, 2008 at 4:39 AM, langwnwk [EMAIL PROTECTED] wrote: We had developed a clustering and loadbalancing module on apache. when we made benchmark test. I often find segments fault in error_log,such this [notice] child pid 25600 exit signal Segmentation fault (11). but this module run well. Set CoreDumpDirectory in httpd.conf and check ulimit -c; you may need to change your startup script for the latter. -- Eric Covener [EMAIL PROTECTED] thanks your answer. I had set CoreDumpDirector and run command ulimit -c unlimited .then startup server using this command apachect1 start but no any core file still. you said, maybe i will change the startup script. can you tell me how to change startup script -- С уважением, Кевац Марко Sincerely yours, Kevac Marko
ap_auth_type(r) problem
Hello. I have modified mod_cache slightly and wrote auth module. mod_mycache and mod_myauth for example. I want to be able to check permissions when using cache, so i have added hook in mod_mycache and implemented it in mod_myauth. Problem is that this hook should be executed only if AuthType mine is specified in httpd.conf Location. So i wanted to check ap_auth_type(r) in hook as i do it in my auth module main function. But it occured that in hook ap_cache_auth is NULL, and in mod_myauth main function it is mine. Why? And what should i do? Maybe it is somehow connected with fact that mod_mycache is executed from quick handler? -- С уважением, Кевац Марко Sincerely yours, Kevac Marko
