Re: Apache log modules
Hi Sorin,
Thanks for your reply.
request_rec-connection-id is a long int that is unique. It is built
from the process_id and thread_id of the apache thread that serves the
request.
Will this be unique for MPM worker across control processes / worker threads?
However, a client may open several connections to the server
during the same transaction, so I guess this does not help you much.
My assumption is that both client and server have KeepAlive enabled.
In that case, should there generally not be just one connection
only?
There's a module called unique_id. It creates a string that is stored
in the req-subprocess_env and can be logged with %{UNIQUE_ID}e. It
encodes the request timestamp, the connection-id, the _server_ IP and
a random number. It does not encode the client IP or port. However, if
you combine it with the client-IP and client port that you can log as
well in the same log line, you could, probably, extract what you want
after some log-postprocessing.
So I can decode the unique ID and extract the connection ID from it? I
guess so...
Different clients behind a NAT router will use different ports.
However, based solely on ports, you won't be able to distinguish
between two different clients on one hand or one client that makes
several connections on the other hand. A method that is used by some
of my colleagues in order to distinguish between different clients
(but I don't know much about it, so I can't tell you more) is to
analyse the TCP header of the packet in order to extract the TCP
sequence numbers.
That sounds interesting but as far as I can see this information is
not available in the application layer. Hard to imagine, but are there
any tricks to get this information in Apache modules? Or are your
colleagues using sniffers such as tcpdump?
Thank you,
Andrej
Apache log modules
Hi, I am looking for a way to deduct the concept of a transaction from the Apache log. What I mean is that I want to group HTTP requests that are sent by one particular client, for example when a user clicks a link in the browser. Then I want to be able to group all the HTTP requests that are the result of that one click (only taking account for requests that are going to our servers). Assuming that both client and server have the KeepAlive enabled, I though that maybe a custom Apache log-module could write the connection ID (if such a thing exists) to the log in order to distinguish different clients. Moreover, assuming that users wait for at least 1 second between clicks, I should be able to deduct transactions by grouping them on timestamps that fall within a second. The scheme with the connection ID, can it work, or am I misjudging something completely? Are there alternatives? Thank you, Andrej
Re: Apache log modules
Hi Ted, This is much better done at the application level. I am aware of that, but that is not an option unfortunately. Cheers, Andrej
Re: Apache log modules
Hi,
My condolences. I have felt your pain.
Thanks ;)
I just read the documentation of mod_log_config 2.3 and some
logging-options have been added, to my pleasant surprise! I was
thinking I could use:
%a = client's IP address
%{remote}p = client's port
Would this pair uniquely identify one client connection, also for
clients behind one NAT router?
Then, in combination with the option %k this *might* be a solution to
follow users (of course, with some restrictions):
%k = Number of keepalive requests handled on this connection.
Interesting if KeepAlive is being used, so that, for example, a '1'
means the first keepalive request after the initial one, '2' the
second, etc...; otherwise this is always 0 (indicating the initial
request).
Would this be feasible? Any comments?
Thank you,
Andrej
Re: *** glibc detected *** double free or corruption (!prev) in cleanup function
Hi Bronto, http://lmgtfy.com/?q=apache+httpd+valgrind Thank you for the tip :) You are funny! Cheers, Andrej
*** glibc detected *** double free or corruption (!prev) in cleanup function
Hi,
I have a question about apr_pool_cleanup_register for a child's pool.
I register a cleanup function that is called when the pool is
destroyed. In the cleanup function, I join a background thread that
first writes some log to a database:
static void mbrace_child_init(apr_pool_t *pool, server_rec *s)
{
apr_pool_cleanup_register(pool, 0, mbrace_bgthread_cleanup,
apr_pool_cleanup_null);
}
static apr_status_t mbrace_bgthread_cleanup(void*)
{
req_log_mngr-stop()
}
But, when it connects to the database in the background thread, it
sometimes ends up in the error below. I do not fully understand why,
but are there any restrictions to what I can do in the cleanup
function?
When running gdb on the generated core-dump, you can see that it
happens on line 134.
125 APR_DECLARE(void) apr_allocator_destroy(apr_allocator_t *allocator)
126 {
127 apr_uint32_t index;
128 apr_memnode_t *node, **ref;
129
130 for (index = 0; index MAX_INDEX; index++) {
131 ref = allocator-free[index];
132 while ((node = *ref) != NULL) {
133 *ref = node-next;
134 free(node); here it happens!!
135 }
136 }
137
138 free(allocator);
139 }
Hopefully somebody can push me in the right direction.
Thank you,
Andrej
*** glibc detected *** /usr/local/httpd-2.2.15/bin/httpd: double free
or corruption (!prev): 0x09461298 ***
=== Backtrace: =
/lib/tls/i686/cmov/libc.so.6[0xb7d5d454]
/lib/tls/i686/cmov/libc.so.6(cfree+0x96)[0xb7d5f4b6]
/usr/local/httpd-2.2.15/lib/libapr-1.so.0(apr_allocator_destroy+0x2e)[0xb7f3d2ce]
/usr/local/httpd-2.2.15/lib/libapr-1.so.0(apr_pool_destroy+0x235)[0xb7f3dfa5]
/usr/local/httpd-2.2.15/bin/httpd[0x80aa4c4]
/usr/local/httpd-2.2.15/bin/httpd[0x80aa986]
/usr/local/httpd-2.2.15/bin/httpd[0x80aabe3]
/usr/local/httpd-2.2.15/bin/httpd(ap_mpm_run+0x5fa)[0x80ab26a]
/usr/local/httpd-2.2.15/bin/httpd(main+0xac8)[0x80696d8]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe5)[0xb7d04685]
/usr/local/httpd-2.2.15/bin/httpd[0x8068601]
=== Memory map:
08048000-080c7000 r-xp fe:00 1171716
/usr/local/httpd-2.2.15/bin/httpd
080c7000-080c8000 r--p 0007e000 fe:00 1171716
/usr/local/httpd-2.2.15/bin/httpd
080c8000-080cb000 rw-p 0007f000 fe:00 1171716
/usr/local/httpd-2.2.15/bin/httpd
080cb000-080ce000 rw-p 080cb000 00:00 0
09254000-09479000 rw-p 09254000 00:00 0 [heap]
b6465000-b6466000 ---p b6465000 00:00 0
mod_proxy_balancer and HTTP response header
Hi, I am looking for way to examine the HTTP response generated by BalanceMember in my module that runs on the load balancer. I would like to read a custom HTTP response header that is added by the application, but I would like to read it on the load-balancer. I tried to read it in the ap_hook_log_transaction, but the header is not there. I am not sure about the internals of mod_proxy_balancer, but I guess the hook does not contain the request_rec of the response from the actual application running on BalanceMember? I was hoping somebody could clarify this? Thank you, Andrej
Re: LD_PRELOAD for modules
Hi William, LoadFile Saving on ink aren't you ;) Thanks for the advise, I will give it a go soon. Just one question you might be able to answer. From the documentation it looks like LoadFile is used to add additional code. In my case, I need to override existing functionality. For example, if mod_php is already linked in with the MySQL client library, I want my own modified client library to be used instead, like LD_PRELOAD does for binaries. From the documentation it is not clear, but can I do this with LoadFie? Any hints/tips? Thanks alot! Andrej
Re: LD_PRELOAD for modules
Hi, When mod_php is loaded into httpd, it's mysql dependencies/symbols will get resolved against the mysql client you have already LoadFile'd into httpd. Thanks that is could news, so the libraries in LoadFile directives are loaded first. Thanks, Andrej
Re: LD_PRELOAD for modules
Hi, Maybe a bit off-topic for this mailing list, but I need to do the same preload-trick for CGI applications written in C++. I was thinking about a wrapper bash-script that usese LD_PRELOAD in addition to a rewrite rule in Apache directive. I have some problems with the wrapper script: #!/bin/bash LD_PRELOAD=/path/to/modified/mysql/libmysql_client.so /mbrace/htbin/cpp-appl.cgi Obviously this does not work because I need to redirect stdin to cpp-appl.cgi and return whatever cpp-appl.cgi through stdout. Don't know how to do this. I am sure I am not the first one to do this, but can't find any useful examples. I was hoping you could help me out? Thank you, Andrej
Re: LD_PRELOAD for modules
Hi Sorin, In the Debian package of apache there's /etc/apache2/envvars. I'm not sure if this file appears in an out-of-the-box compilation of apache2. Anyway, /usr/sbin/apache2ctl looks for it and loads it before launching the server. The server in launched in the environment that is set in envvars. So you could try writing export LD_PRELOAD=... in envvars. I do not know if this has an effect on how apache2 spawns the CGI applications but you might give it a try. Thank you for putting me back on track! I found the solution I think. From the Apache documentation: SetEnv directive Syntax: SetEnv env-variable value Context: server config, virtual host, directory, .htaccess Override: FileInfo Status: Base Module: mod_env Compatibility: SetEnv is only available in Apache 1.1 and later. Directory and .htaccess context is available in Apache 1.3.7 and later. Sets an environment variable, which is then passed on to CGI scripts and SSI pages. Example: SetEnv SPECIAL_PATH /foo/bin
LD_PRELOAD for modules
Hi, I want to override a library that is linked with an Apache module, i.e. I want to use my modified version of the MySQL client library for example for mod_php. I would like to do this without relinking or even modifying the mod_php lib. I was hoping that I could use something like LD_PRELOAD to force my modified MySQL client library. Is there a way to do this? Thank you, Andrej
detect disconnect in CGI
Hi, I am using mod_cgi using Cgicc with C++ on Linux. I need a way to detect disconnects from the client. I think I have a way to do this, but I wanted to make sure if there is no easier way. I am writing a space back to the client every second as part if a custom HTTP response header. When the connection drops, this results in a sigpipe. I am using non-parsed-headers to make sure the spaces are really flushed to the client and not buffered by Apache. Is there a better way to do this? Thank you, Andrej
ap_hook_child_init
Hi, I have an Apache module for mpm-prefork that runs a background thread. The thread is initialized in ap_hook_child_init() which works just fine. But, I have to finish the thread neatly when the process exits. How can I force this? I could not find any hook-method that is called before the process is exited. How can I force the thread to be cleanup up neatly, preferably without breaking into the Apache core? Thank you, Andrej
Re: ap_hook_child_init
Hi, Register a cleanup on the pool passed to your child_init function. Yes of course, thank you. Cheers, Andrej
Re: segmentation fault in worker.c
Hi, I guess you are hit by this: https://issues.apache.org/bugzilla/show_bug.cgi?id=46467 Indeed so it was! Cheers, Andrej
segmentation fault in worker.c
Hi,
I compiled httpd-2.2.11 with ./configure --with-included-apr
--enable-ssl --disable-cgi --disable-cgid --with-mpm=prefork
--enable-status. HTTP requests seem to be processed fine from a users
point of view, but I get many segfaults in my apache log when I
seriously increase the workload. Here a trace from gdb:
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.
[New process 9935]
#0 apr_pollset_add (pollset=0x0, descriptor=0xbf8225dc) at
poll/unix/epoll.c:150
150 if (pollset-flags APR_POLLSET_NOCOPY) {
(gdb) print pollset
$1 = (apr_pollset_t *) 0x0
(gdb) bt
#0 apr_pollset_add (pollset=0x0, descriptor=0xbf8225dc) at
poll/unix/epoll.c:150
#1 0x080c2c41 in child_main (child_num_arg=value optimized out) at
prefork.c:532
#2 0x080c30f3 in make_child (s=0x9c849a8, slot=138) at prefork.c:746
#3 0x080c3ef8 in ap_mpm_run (_pconf=0x9c7d0a8, plog=0x9cbb1a0,
s=0x9c849a8) at prefork.c:881
#4 0x0806e808 in main (argc=164081968, argv=0xbf822904) at main.c:740
(gdb)
When I compiled with mpm-worker, I did not get into these problems.
These are my mpm-prefork settings:
IfModule mpm_prefork_module
ServerLimit 512
StartServers 100
MinSpareServers 25
MaxSpareServers 75
MaxClients 256
MaxRequestsPerChild 0
/IfModule
When I start about 4000 user sessions in a few seconds this happens,
not with lower values. Changing KeepAllive to On/Off does not change
anything.
I was hoping this rings a bell, otherwise I could provide you with
more information on your request, provided someone is kind enough to
pick this up ;)
Thank you,
Andrej
load mod_php.so for each child separately
Hi, If I am correct, on Linux all childs in MPM prefork share the same mod_php.so which is loaded in the parent process. Is there a way to load mod_php.so separately for each forked child? Can I force such a thing in httpd.conf or maybe in the child_init() hook in a module? Cheers, Andrej
Re: child_init for threads?
Hi, Note that there is a difference between the apr_thread_data_*() methods and the apr_threadkey_private_*() methods. The apr_thread_data_*() methods actually associate your data with the thread-pool. Since each new apr thread has it's own private thread-pool everything should work fine. The apr_threadkey_private_*() methods use the underlying thread libraries TLS routines. While apr_thread_data_* may work for you, true TLS is only supplied by apr_threadkey_private_* routines. Thanks, that's clear! In my case the apr-thread based suffices, I hope. But I realize now that it is difficult or impossible to get the apr_thread_t data from the current thread in an Apache module (for threaded MPMs). Is this possible to do this somehow? For example, does this work: apr_os_thread_t os_thd = apr_os_thread_current (); apr_thread_t *apr_thd; apr_status_t apr_os_thread_put(apr_thd, os_thd, r-pool); And what would this do in an MPM prefork? Or maybe there is some better technique? Thank you! Andrej
Re: child_init for threads?
Hi, apr_os_thread_t os_thd = apr_os_thread_current (); apr_thread_t *apr_thd; apr_status_t apr_os_thread_put(apr_thd, os_thd, r-pool); Okay, this clearly doesn't work. I misunderstood the documentation, but the sources don't lie. Anyway, how can I get the apr_thread_t for the current thread in a module for MPM worker? Cheers, Andrej
MaxRequestsPerChild for MPM worker threads?
Hi, Is there a way to let a MPM worker thread exit after serving X requests? I need to do some tests that requires checking if my custom hook ap_hook_thread_exit() is called properly. I found a directive to have one thread per child that gets me in the right direction (ThreadsPerChild 1). Now I need one to set the maximum number of requests served by a thread to get a fast thread_exit(). I tried MaxRequestsPerChild but it does not seem to do what I want. BTW, I start apache like /usr/local/apache2/bin/httpd -X . What option(s) could I use to force worker threads to exit after serving few requests? Thank you, Andrej
Re: child_init for threads?
Hi, Thanks. Yes. create_connection() is the first hook run as part of request processing, a delay in create_connection() will result in a delay in HTTP processing. That's too bad. I guess I have to hack this into the MPM then. One more question, what would be the way to store thread-specific data that is to be re-used across HTTP requests? Is there an example where this is done before? Thank you, Andrej
Re: child_init for threads?
Hi, Thanks, that helps! Since I am developing my modules in C++, I think I should be using this one: apr_status_t apr_thread_data_set( void * data, const char *key, apr_status_t(*)(void *) cleanup, apr_thread_t * thread ) If I understand correctly, I can pass my own cleanup function (calling delete to free memory) which is called automatically when the thread is destroyed by the APR framework. Would that be the way, or am I still misunderstanding something? Thank you, Andrej
child_init for threads?
Hi, I was wondering if there is an analogue hook like child_init() for threads that I can use for doing some time-consuming thread-initialisation that should not slow down an HTTP request handler. More specific, I want to initialise some per-thread variables that come from a database. Does the APR offer something like this, or is there a way to hack this into my module, without changing the APR? Thank you, Andrej
start httpd with one child for debugging
Hi, I believe I read some while ago that it is possible to start httpd with only one child (a special for debugging httpd). I am not able to find this option anymore. Does such an option really exist? Thank you, Andrej
Re: hook before receiving HTTP request
Hi, http://httpd.apache.org/docs/2.2/developer/modules.html ap_hook_pre_connection do any setup required just before processing, but after accepting ap_hook_create_request ?? (Assuming) ap_hook_create_request will be called when creating a request_rec * for each request on a connection (multiple request on a connection with KeepAlive enabled). Is that what you are looking for? Well, I saw those hooks too, but I think it is not exactly what I want. I need to hook into my module EVERY time before receiving a request. I need to do some general initialization before every HTTP request, but I do not want to disturb the request processing itself, in terms of performance loss. But I guess it is not there so I will start looking for another solution. Cheers, Andrej
hook before receiving HTTP request
Hi, I am looking for a hook function that I can call to initialize some structures before it accepts a connections. Similar like ap_log_transaction() is called after the HTTP reply is sent to the client, I need a hook that is called before a HTTP request is received. Is it there? Thank you, Andrej
post_config on reload
Hi,
I found this piece of code for dealing with the post_config issue (it is
called twice, while I need to initialise my stuff only once):
void *data;
const char *userdata_key = post_config_only_once_key;
apr_pool_userdata_get(data, userdata_key, s-process-pool);
if (!data) {
apr_pool_userdata_set((const void *)1, userdata_key,
apr_pool_cleanup_null, s-process-pool);
return OK;
} else {
// do initialize
}
I was wondering if there is a solution for detecting a post_config
invocation on behalf of apachctl restart instead of a clean start (in case
of a restart, I do not want to initialise my stuff again).
Thank you,
Andrej
Re: custom background thread and module sharing a data structure
Hi Saju, For the worker mpm, both cross thread and cross process protection will be needed. apr_proc_mutex.h family supplies cross-process protection, apr_thread_mutex.h provides cross thread protection. You locking method would be a wrapper method that first obtains a process level lock, and then inside that lock it obtains a thread level lock. In my specific case, do I really need thread-level locking in addition to process-level locking? Since I am running a background thread that is started in post_config(), worker-threads in the mpm worker model are running in a different process then my background thread anyway. So I guess process-level locking should suffice, or am I overlooking something? Thank you, Andrej
Re: post_config on reload
Hi, Do not do this - a restart should be a restart, not a half of a restart. You should be reinitializing whatever you do on a restart as well as a start. That's the whole point. I have one phrase that should illustrate why : memory leak. For example, if your extension creates another thread or spawns another process and that other thread/process doesn't clean up on a restart and contains a memory leak, what good is restart to you? Restart has suddenly become pointless. It is good etiquette to honour the concept of a restart. Okay, point taken. My extension indeed creates a thread. How can I know that a user issues a restart? In other words, where can I kill my thread? Cheers, Andrej
Re: custom background thread and module sharing a data structure
Hi, Thanks for your comments. See below... Worker mpm is a multithreaded, multiprocess mpm. Multiple child processes host multiple worker threads that run your module code. If your module services 2 concurrent requests in 2 different threads in the same process and both the threads need to access critical code at the same time, you will have to use thread locking to prevent race conditions. Yes, I understand the concept of mpm worker. My point was slightly different: When I start a background thread in the post_config()-hook, it is created in the address space of the server, and not of the processes that run worker threads for handling HTTP requests. Therefore, threads that handle HTTP requests always execute in a different process than the background thread, and thus thread-based locking is not necessary for sharing inter-process data. re: Using non APR locking mechanisms - sure they will work as long as you are using compatible mechanisms (like if apr_threads is based on pthreads, the alternate locking mechanism should work with pthreads). That's good news! Cheers, Andrej
Re: custom background thread and module sharing a data structure
Hi, My point is that within a single process, multiple threads can service requests that can end up firing your module code. If you only do process locking you can still have more than 1 thread executing your module code at the same time. Just a process level lock will *not* guarantee that only a single thread is executing your code within the process holding the lock. A inter thread lock is needed to synchronize multiple threads within that processes from stomping on a shared resource. Yeah you are right, two thread in the same process in the mpm worker model still execute the same code to access the shared inter-process resource. So I guess I need to add the thread-based mutexes within a APR_HAS_THREADS define, like in some other modules I have seen. Thanks for clearing this up! Cheers, Andrej
Re: custom background thread and module sharing a data structure
Hi, Thanks for the info. For the worker mpm, both cross thread and cross process protection will be needed. apr_proc_mutex.h family supplies cross-process protection, apr_thread_mutex.h provides cross thread protection. You locking method would be a wrapper method that first obtains a process level lock, and then inside that lock it obtains a thread level lock. Is there anything against using the interprocess- and thread-library from C++ boost and use those instead the ones provided by the APR-library? What I need to do is start a background thread and share some data with a custom module. Although C++ boost is cross-platform, for now my primary target is Linux only. Your opinion is appreciated! Cheers, Andrej
Re: custom background thread and module sharing a data structure
Hi, Yes, I did this. I will send you a commented source file sometimes today or early tomorrow (Central European Time), I don't really have time now. Great, I will be waiting for that! Thanks, Andrej
custom background thread and module sharing a data structure
Hi, I need to modify Apache and run one custom background thread. In addition, my custom modules have to be able to share a data structure with this background thread. Did anybody do this before? Is there an example I can use as a starting point? Thank you, Andrej -- Andrej van der Zee 2-40-19 Koenji-minami Suginami-ku, Tokyo 166-0003 JAPAN Mobile: +81-(0)80-65251092 Phone/Fax: +81-(0)3-3318-3155
process initialisation in prefork-MPM
Hi, I need to initialise some structures for each process (in prefork MPM) when it is created, and need to be able to access these structures in ap_hook_post_read_request and ap_hook_log_transaction handlers. I would like to initialise the structures before the first HTTP request is being served by the process. How should I do this? Thank you, Andrej -- Andrej van der Zee 2-40-19 Koenji-minami Suginami-ku, Tokyo 166-0003 JAPAN Mobile: +81-(0)80-65251092 Phone/Fax: +81-(0)3-3318-3155
Re: process initialisation in prefork-MPM
You hook child_init (ap_hook_child_init) and store your data in global variables that are accessed by post_read_request and log_transaction. Easy enough. Thanks, Andrej
Re: process initialisation in prefork-MPM
Hi, You hook child_init (ap_hook_child_init) and store your data in global variables that are accessed by post_read_request and log_transaction. BTW, in a worker-MPM, is child_init executed for every created worker-thread? Or is it per process? Thank you, Andrej
connection speed / data transfer time
Hi, I was wondering if it is possible to get the connection speed in an Apache module. Actually, I can get to the actual bytes transferred to/from the client, so the transfer time would also do. Thank you, Andrej
Re: connection speed / data transfer time
Hi Ray, Thanks for your reply. see mog_logio for the actual bytes transferred and mod_status or others for examples of the time required. I need the duration of network transfer per HTTP request and response, but I cannot find this information in mod_status. Are there any specific points in the source code that I can use for determining the transfer times (that is a beginning and end of request/response transfer)? Preferably in module hooks, but the APR is also fine. Thank you! Andrej
Re: mod_dbd and prepared statements (httpd-2.2.9)
Hi Tom, Thanks a lot for your help! Everything works as expected... Cheers, Andrej
Re: mod_dbd and prepared statements (httpd-2.2.9)
Hi,
I did not find a solution, I just stopped using prepared statements
altogether. But I tried to isolate the problem just now, and found somehow
that I cannot use FLOAT in prepared statement somehow (when I tried INT
columns it even segfaults). Below the source code of a mini-module to
illustrate this. This is the table I created in MySQL:
CREATE TABLE simple_table (duration FLOAT NOT NULL) ENGINE=INNODEDB;
And this is what appears in the MySQL log:
PrepareINSERT INTO simple_table (duration) VALUES (?)
ExecuteINSERT INTO simple_table (duration) VALUES ('')
If you want to reproduce it, dont forget to put this in httpd.conf:
LoadModule prep_stmt_module modules/mod_prep_stmt.so
PrepStmt on
- Complete mini-module -
#include httpd.h
#include http_config.h
#include http_core.h
#include http_log.h
#include http_protocol.h
#include http_connection.h
#include apr_file_info.h
#include apr_file_io.h
#include apr_dbd.h
#include mod_dbd.h
module AP_MODULE_DECLARE_DATA prep_stmt_module;
static int prep_stmt_write(request_rec *r);
typedef struct prep_stmt_config
{
int prep_stmt_on;
} prep_stmt_config;
static const char * prep_stmt_config_set_prep_stmt_on(cmd_parms *cmd, void
*dummy, int flag)
{
ap_dbd_prepare(cmd-server, INSERT INTO simple_table (duration) VALUES
(%f), insert_row);
prep_stmt_config *cfg = ap_get_module_config(cmd-server-module_config,
prep_stmt_module);
cfg-prep_stmt_on = flag;
return NULL;
}
static int prep_stmt_write(request_rec *r)
{
prep_stmt_config *cfg = ap_get_module_config(r-server-module_config,
prep_stmt_module);
if (!cfg-prep_stmt_on)
return DECLINED;
ap_dbd_t * dbd = ap_dbd_acquire(r);
apr_dbd_prepared_t *prepared = apr_hash_get(dbd-prepared, insert_row,
APR_HASH_KEY_STRING);
if (!prepared) {
ap_log_error(APLOG_MARK, APLOG_ERR, 0, r-server,
DBD Log: Failed to get prepared statement:
update_request);
return DECLINED;
}
int rv, nrows;
if (rv = apr_dbd_pvquery(dbd-driver, r-pool, dbd-handle, nrows,
prepared, 10.2, NULL)) {
const char *errmsg = apr_dbd_error(dbd-driver, dbd-handle, rv);
ap_log_error(APLOG_MARK, APLOG_ERR, 0, r-server,
DBD Log: Failed to execute prepared statement:
insert_row);
return DECLINED;
}
}
static const command_rec prep_stmt_cmds[] = {
AP_INIT_FLAG(PrepStmt, prep_stmt_config_set_prep_stmt_on,
NULL, RSRC_CONF, Enable DBD Log),
{ NULL }
};
static void prep_stmt_register_hooks(apr_pool_t *p)
{
ap_hook_log_transaction(prep_stmt_write, NULL, NULL, APR_HOOK_MIDDLE);
}
static void * prep_stmt_create_config(apr_pool_t *pool, server_rec *s)
{
prep_stmt_config *cfg = apr_pcalloc(pool, sizeof(prep_stmt_config));
return cfg;
}
module AP_MODULE_DECLARE_DATA prep_stmt_module =
{
STANDARD20_MODULE_STUFF,/* stuff that needs to be declared in every
2.0 mod */
NULL, /* create per-directory config structure*/
NULL, /* merge per-directory config
structures*/
prep_stmt_create_config, /* create per-server
config structure */
NULL, /* merge per-server config
structures */
prep_stmt_cmds, /* command
apr_table_t */
prep_stmt_register_hooks /* register
hooks */
};
mpm worker and mod_cgi
Hi, I am compiling mod_cgi with apxs outside the httpd-2.2.9 source tree. When I choose --with-mpm=prefork this works fine, but when I use --with-mpm=worker it seems not to use the implementation in the module at all (the cgi_handler hook method is never called, but CGI applications seem to work fine). In both cases I configure httpd with --disable-cgi. Am I doing something wrong, or is there something special about CGI for the worker mpm? Thank you, Andrej
APR_DECLARE_OPTIONAL_FN and APR_RETRIEVE_OPTIONAL_FN
Hi, I was wondering what the purpose is of the macros APR_DECLARE_OPTIONAL_FN and APR_RETRIEVE_OPTIONAL_FN. For example, in mod_authn_dbd APR_RETRIEVE_OPTIONAL_FN is used to get a hold on the functions ap_dbd_acquire and ap_dbd_prepare declared in in mod_dbd with APR_DECLARE_OPTIONAL_FN. I could also directly invoke these functions as they are declared in the header file mod_dbd.h and let the loading of the module fail with an unresolved symbol if mod_dbd is not loaded. What is the benefit for using those macros? Cheers, Andrej
Re: APR_DECLARE_OPTIONAL_FN and APR_RETRIEVE_OPTIONAL_FN
Hi, Without them, it became impossible to build plugable frameworks without linking one module to another. Yes of course, that would be necessary if you compile your module outside the httpd source tree with apxs I guess. Thanks a lot, Andrej
Re: number of bytes/packets sent/received
Thanks for your reply! Giving it a second thought, the problem looks complicated. Such filter-based counters would count the traffic of _one_ apache process. When you increment the counter, you have to protect it from concurrent access by other threads running in the same apache child process. Then, before the apache child process exits, you need to add the counter to a global counter shared among apache child processes (which has to be protected against race conditions as well). It's really hairy. Don't do it in apache. I forgot to mention, but I need the number of packets / bytes per HTTP request and log it to a database with other information. So I guess that would take case of all the hairy stuff. I think I will give it a go as you recommend with an input and output filter that only counts the bytes and just passes on the data, unless somebody comes with a better idea (what about the content-length for example, can I trust this?). I understood from your email that the headers are counted with it (I never wrote a filter before) and that's exactly what I need. Cheers, Andrej
Re: number of bytes/packets sent/received
Hi Dave, Would mod_logio be at all helpful? http://httpd.apache.org/docs/2.2/mod/mod_logio.html I haven't looked into it, but the source might give you some ideas. You may even have a direct solution if you use a module that logs directly to a database. I must admit that I don't know offhand if such a module exists. You really made it easy for me huh? Thanks alot. Cheers, Andrej
Re: number of bytes/packets sent/received
Hi, No, the Content-Length doesn't include the size of the headers. Yes of course, but you could add the length of all r-headers_in. I just wondered if the content-type reflects the actual size, for example if the request is compressed. While I was typing the message, Dave Ingram's message arrived. I think the solution he proposes is much better. Yes, thanks alot for your time anyway. Cheers, Andrej
Re: number of bytes/packets sent/received
Hi, A quick search reveals that there are some modules out there that do log to databases, but I don't know how well-maintained they are (mod_log_sql only mentions Apache 2.0, for example) or how much work they would be to use (an O'Reilly page http://www.onlamp.com/pub/a/apache/2005/02/10/database_logs.html?page=2 suggests that Apache's default mod_log_config needs to be rebuilt). Since Apache 2.1 there is mod_dbd that makes it pretty easy to log to a MySQL, sqlite2/3 and Postgres database. It also takes care of connection pooling. Cheers, Andrej
mod_dbd/apr_dbd and Commands out of sync
Hi, I am using mod_dbd for the first time and get into trouble when I execute the following in sequence: 1) do an INSERT 2) do a SELECT 3) do another INSERT On 3) I get the error Commands out of sync; you can't run this command now. This is what MySQL sais: If you get Commands out of sync; you can't run this command now in your client code, you are calling client functions in the wrong order. This can happen, for example, if you are using mysql_use_result() and try to execute a new query before you have called mysql_free_result(). It can also happen if you try to execute two queries that return data without calling mysql_use_result() or mysql_store_result() in between. I went through all the apr_dbd_XXX functions in apr-util/include/apr_dbd.h but could not find any way to free a result after 2). How should I deal with this issue? Cheers, Andrej
Re: mod_dbd/apr_dbd and Commands out of sync
Hi,
When I execute two SELECT statements after each other I do not get the
Comamnd out of sync error anymore if I iterate through all the rows
in the apr_dbd_results_t, but I do get all emty values and column
names for the 2nd SELECT.
Can you please post your code ?
Below the code. The second SELECT has 5 rows all with empty fields.
Thanks alot!
Andrej
apr_dbd_results_t *dbd_results = NULL;
int dbd_rv;
char *stmt = (char*) apr_psprintf(r-pool, SELECT request_type_id
FROM request_type WHERE name = '%s', req_cfg-tid);
if (dbd_rv = apr_dbd_select(dbd-driver, r-pool, dbd-handle,
dbd_results, stmt, 0)) {
const char *errmsg = apr_dbd_error(dbd-driver, dbd-handle, dbd_rv);
ap_log_error(APLOG_MARK, APLOG_ERR, 0, r-server, Perfmon: Failed
to execute statement '%s': %s, stmt, errmsg);
return DECLINED;
}
apr_dbd_row_t *dbd_row = NULL;
const char *request_type_id = NULL;
for (rv = apr_dbd_get_row(dbd-driver, r-pool, dbd_results, dbd_row, -1);
rv != -1;
rv = apr_dbd_get_row(dbd-driver, r-pool, dbd_results, dbd_row, -1)) {
request_type_id = apr_dbd_get_entry(dbd-driver, dbd_row, 0);
}
apr_dbd_results_t *dbd_results2 = NULL;
apr_dbd_row_t *dbd_row2 = NULL;
if (dbd_rv = apr_dbd_select(dbd-driver, r-pool, dbd-handle,
dbd_results2, SELECT * FROM request_type, 0)) {
const char *errmsg = apr_dbd_error(dbd-driver, dbd-handle, dbd_rv);
ap_log_error(APLOG_MARK, APLOG_ERR, 0, r-server, Perfmon: Failed
to execute statement '%s': %s, stmt, errmsg);
return DECLINED;
}
// Here I get all empty strings for the name!
ap_log_error(APLOG_MARK, APLOG_ERR, 0, r-server, Perfmon: name col:
, apr_dbd_get_name(dbd-driver, dbd_results2, 0));
for (rv = apr_dbd_get_row(dbd-driver, r-pool, dbd_results2, dbd_row2, -1);
rv != -1;
rv = apr_dbd_get_row(dbd-driver, r-pool, dbd_results2, dbd_row2, -1)) {
const char *request_type_id = apr_dbd_get_entry(dbd-driver, dbd_row2, 0);
// Here I get 5 rows with an empty request_type_id!
ap_log_error(APLOG_MARK, APLOG_ERR, 0, r-server, Perfmon: : ,
request_type_id);
}
Re: mod_dbd/apr_dbd and Commands out of sync
Hi Jerome, Thanks for your answer. I just went through the sources of apr_dbd and saw that I have to loop through all the rows in a apr_dbd_results_t before mysql_free_result() is called through apr_pool_cleanup_run(). In my code I am sure the result is exactly one, but unfortunately I have to write a loop anyway. Nevertheless, it works now... Cheers, Andrej
Re: mod_dbd/apr_dbd and Commands out of sync
Hi, I still have another open issue with mod_dbd I was hoping you could help me with. When I execute the query SELECT LAST_INSERT_ID() I always get an empty string as a result. I do receive one row, but nothing in it. Is this a mod_dbd issue you think, or a mysql issue? Cheers, Andrej
writing to database in Apache module
Hi, I wrote my first Apache module that gathers performance data of each request. Currently, I write the information to a log file in the log_transaction hook function. I would like to write this to a MySQL database instead. What is the recommended way to do this? Cheers, Andrej -- Andrej van der Zee 2-40-19 Koenji-minami Suginami-ku, Tokyo 166-0003 JAPAN Mobile: 0031-(0)80-65251092 Phone/Fax: 0031-(0)3-3318-3155
Access configuration variable in other modules
Hi, I am writing my first Apache module for monitoring resource usage of embedded interpreters and CGI applications on Linux (I am also adapting mod_cgi and the apr-library). The problem is that my own module has some server-level configuration options that I can initialize and use in my own module, but I don't know how to access them in *another* module (i.e. mod_cgi). What is the way to do this? Any examples of this in the modules that come with the sources? Cheers, Andrej
Access configuration variable in other modules
Hi, I am writing my first Apache module for monitoring resource usage of embedded interpreters and CGI applications on Linux (I am also adapting mod_cgi and the apr-library). The problem is that my own module has some server-level configuration options that I can initialize and use in my own module, but I don't know how to access them in *another* module (i.e. mod_cgi). What is the way to do this? Any examples of this in the modules that come with the sources? Cheers, Andrej
Re: Access configuration variable in other modules
Thanks, that seems satisfactory!
Though, it seems to imply that I have to export my own module in a header
file and include it in mod_cgi. In other words, I have to add my module to
the httpd build system. So I copied my module to the modules/generator
directory. But I am not sure what the good way is to incorporate my module
in the build httpd system; just typing make does not compile my module.
Which build-files should I modify to accomplish this?
Cheers,
Andrej
On Sat, Sep 13, 2008 at 8:25 PM, Kevac Marko [EMAIL PROTECTED] wrote:
Catch the example:
cache_module = ap_find_linked_module(mod_cache.c);
if (!cache_module) {
ap_log_error(APLOG_MARK, APLOG_ERR, 0, s, ERRTAG %s:
Can't find mod_cache module., __func__);
return;
}
conf_cache =
(cache_server_conf *)ap_get_module_config(
s-module_config, cache_module);
if (!conf_cache) {
ap_log_error(APLOG_MARK, APLOG_ERR, 0, s, ERRTAG %s:
Can't acquire mod_cache server configuration
structure.,
__func__);
return;
}
On 9/13/08, Andrej van der Zee [EMAIL PROTECTED] wrote:
Hi,
I am writing my first Apache module for monitoring resource usage of
embedded interpreters and CGI applications on Linux (I am also adapting
mod_cgi and the apr-library). The problem is that my own module has some
server-level configuration options that I can initialize and use in my
own
module, but I don't know how to access them in *another* module (i.e.
mod_cgi). What is the way to do this? Any examples of this in the
modules
that come with the sources?
Cheers,
Andrej
--
С уважением, Кевац Марко
Sincerely yours, Kevac Marko
--
Andrej van der Zee
2-40-19 Koenji-minami
Suginami-ku, Tokyo
166-0003 JAPAN
Mobile: 0031-(0)80-65251092
Phone/Fax: 0031-(0)3-3318-3155
Re: Apache modification questions
Hi, A post doesn't normally have anything in the QUERY_STRING. Rather a POSTed form has the stuff being sent in the request body, which is read from STDIN by a script. QUERY_STRING is still available for use. Consider this form: form method=post action=myscript.cgi input name=emailaddress The emailaddress input would NOT appear in QUERY_STRING, but rather would appear to the script as STDIN, or to another module as the request body. You can still do: form method=post action=myscript.cgi?id=reallyunique input name=emailaddress U yeah you are right, I am not thinking. Cheers, Andrej
apr_open_file and apr_write_file in post_read_request and log_transaction
Hi,
I am new to Apache modules and I am trying to open a file in the hook
post_read_request and write to the file in log_transaction. The file
is supposed to be only valid for the duration of the request and is
named with a unique identifier (created by mod_unique_id). The
apr_file_t is carried, with other data, in the configuration vector
(request_config) between the different hooks. In post_read_reqeust I
open it like this (req_cfg is the configuration vector
request_config):
req_cfg-log_file = NULL;
req_cfg-filepath = (char*) apr_pstrcat(r-pool, path_name,
unique_id, NULL);
apr_int32_t flags = APR_WRITE | APR_CREATE | APR_EXCL | APR_APPEND;
rv = apr_file_open(req_cfg-log_file, req_cfg-filepath, flags,
APR_OS_DEFAULT, r-pool);
if (rv != APR_SUCCESS || req_cfg-log_file == NULL) {
ap_log_error(APLOG_MARK, APLOG_ERR, 0, NULL, Failed to create
resource log file: %s,
req_cfg-filepath);
return DECLINED;
}
After this I am able to write to the file in this hook function. But,
when I try to write to the file in the hook log_transaction like this:
char *msg = (char*) apr_psprintf(r-pool, %s, some msg);
len = strlen(msg);
apr_file_write(req_cfg-log_file, msg, len);
Then I get a segfault because req_cfg-log_file is NULL. Here the
output from gdb:
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.
[New process 16759]
#0 apr_file_write (thefile=0x0, buf=0x81841d0, nbytes=0xbf8e5d94) at
file_io/unix/readwrite.c:151
151 if (thefile-buffered) {
(gdb) print thefile
$1 = (apr_file_t *) 0x0
(gdb) Quit
Because the file is opened with r-pool (with r request_rec) in
post_read_request I expect it to be open in the hook log_transaction.
So I guess I am wrong? Is it closed by the cleanup handlers? And if
so, what is the way to make sure it is still open?
Cheers,
Andrej
Re: Apache modification questions
Thanks for your comments. Until I get the book, can you tell if a module's hook function can execute in the same thread as the CGI application that serves the request? Also, I am unable to find the apache2 API for building modules. I found some documentation for developers on the apache2 website, but the link for Autogenerated Apache 2 code documentation is not working. Thank you, Andrej -- Andrej van der Zee 2-40-19 Koenji-minami Suginami-ku, Tokyo 166-0003 JAPAN Mobile: 0031-(0)80-65251092 Phone/Fax: 0031-(0)3-3318-3155
Re: Apache modification questions
Hi, Thanks for your comments. child_init is not the appropriate hook for your purpose. Use ap_hook_fixups for getting the ID and ap_hook_log_transaction for logging. In ap_hook_fixups, is it possible to get the thread/process ID of the CGI application serving the request? Moreover, is this thread/process already created? Or maybe the hook function is executed in the same thread/process as the CGI application? Every module has a register_hooks function. There, you call the two ap_hook functions above in order to hook your callbacks to the fixups and log_transaction events. Next you implement the two callbacks and that's it. That's clear. Cheers, Andrej -- Andrej van der Zee 2-40-19 Koenji-minami Suginami-ku, Tokyo 166-0003 JAPAN Mobile: 0031-(0)80-65251092 Phone/Fax: 0031-(0)3-3318-3155
Re: Apache modification questions
Hi, A CGI script is run by the code of a module, mod_perl, mod_php5, etc. In their register_hooks function they register their handler, which is a script interpreter basically. The handler callback is invoked in the same thread that did the rest of the request processing (URL parsing, authentication, fixups, etc). However, I don't know if the handler callback (of mod_perl, mod_php5, etc), which can be seen as a sort of third-party black box, spawns new processes/threads in which they parse the script, compile, etc. I guess they do not spawn new threads/processes but you have to read their docs or their sources in order to be sure. If they do not spawn new threads/processes, then the CGI is executed in the same thread as fixups and the rest of the request processing. Thanks that makes sense. If understood correctly, this means that I can add my own module to the chain of request processors that executes in the same thread as the hook function in mod_php5/mod_perl that executes CGI scripts. Though, if the module's hook function spawns a new process/thread for handling the CGI script is dependent on the module. Does anybody know if the hook functions of such modules usually spawning a new thread/process? My guess is that at least for compiled CGI application written in C/C++ a new process is forked in the hook function. Cheers, Andrej -- Andrej van der Zee 2-40-19 Koenji-minami Suginami-ku, Tokyo 166-0003 JAPAN Mobile: 0031-(0)80-65251092 Phone/Fax: 0031-(0)3-3318-3155
Apache modification questions
Hi, I am about to modify Apache with some custom logging for GET/POST requests (and more). It is for the purpose of research. If possible, I would like to get some guidance in how to implement my ideas. I will explain... Every GET/POST request to Apache will carry a request identifier. Adding the identifier to the request is the responsibility of the developer of the web page. In Apache (NOT the CGI application) I would like to extract the identifier from the request and write it to a log together with timestamp, request serve time and some specific information about the thread/process that handles the request. The CGI application serving the request should be untouched! I have two issues I would like to get some comments about, if possible: 1) What is the recommended way to carry the GET/POST request identifier (inserted by the developer of the web page) from the client to Apache? Add a custom HTTP header? Or should I do it in GET/POST variables? Any other alternatives? 2) I do need to attach to the thread/process handling the request to extract information just after starting and just before ending. Can I do this in an Apache module? I found the ap_hook_child_init() function but no similar exit()-function. Moreover, I need to access the request identifier and log to a file. Can all this be done in an Apache module? Hope you can help! Cheers, Andrej -- Andrej van der Zee 2-40-19 Koenji-minami Suginami-ku, Tokyo 166-0003 JAPAN Mobile: 0031-(0)80-65251092 Phone/Fax: 0031-(0)3-3318-3155
