Re: svn commit: r744281 - /httpd/apreq/trunk/glue/perl/xsbuilder/apreq_xs_postperl.h
- Original Message From: Fred Moyer f...@redhotpenguin.com To: apreq-dev@httpd.apache.org Sent: Friday, February 13, 2009 6:32:48 PM Subject: Re: svn commit: r744281 - /httpd/apreq/trunk/glue/perl/xsbuilder/apreq_xs_postperl.h On Fri, Feb 13, 2009 at 3:27 PM, wrote: Author: joes Date: Fri Feb 13 23:27:31 2009 New Revision: 744281 URL: http://svn.apache.org/viewvc?rev=744281view=rev Log: revert, yes mortal is correct here Could you give a lamens explanation of sv_2mortal for those of us aspiring hackers? I've seen it all over the place in the mod_perl and apreq codebase (and perl as well), but haven't ever been able to track down what it really does. Basically it's the C analog of lexically scoped variables. Mortal SV's are temporary variables, meant to be freed up by perl at the next convenient point (usually a FREETMPS call). If you create an SV in C, and unless you are constructing parts of an object, you probably want it to be mortal. Otherwise you wind up creating an internal perl scalar that has a refcount that never gets decremented and consequently freed.
Regarding Multithreading in apache modules
Hi all, Is it possible to use multithreading in apache modules written in C? As I searched, apr_thread_proc provides process thread management functions. Does it come as part of Apache 2.2? Thanks, Pooja * Meet us at Hall 2/Booth 2C111 at the Mobile World Congress, Barcelona, Feb 16-19, 2009. * Attend the qLabs webinar on Test Case Prioritization on Feb 20, 2009. Visit www.impetus.com for details. Impetus is the winner of the Economic Times Intel Smart Workplace Awards 2008 and the CNBC emerging India 2008. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Regarding Multithreading in apache modules
On Fri, Feb 13, 2009 at 11:33, Pooja Maheshwari pooja.maheshw...@impetus.co.in wrote: Hi all, Is it possible to use multithreading in apache modules written in C? Yes. As I searched, apr_thread_proc provides process thread management functions. Does it come as part of Apache 2.2? Yes. Apache itself uses threads to serve requests if it is configured to use the mpm_worker module. S -- A: Because it reverses the logical flow of conversation. Q: Why is top-posting frowned upon? A: Top-posting. Q: What is the most annoying thing in e-mail?
Re: Regarding Multithreading in apache modules
On Fri, Feb 13, 2009 at 11:59, Pooja Maheshwari pooja.maheshw...@impetus.co.in wrote: Hi Sorin, Thanks for the prompt response. So is it possible to create new threads and do my custom tasks in parallel? Do you have any useful pointers to working with apr_thread_proc? I have a doubt that using multithreading inside apache module, which is itself part of an apache main process thread could be much beneficial? The module I intend to write is basically for generating log files using the client side data sent as part of the request. This data is going to be of uniform size mostly. I do not really understand your problem and your need to use threads. Why not hook log_transaction for your purpose? S -Original Message- From: Sorin Manolache [mailto:sor...@gmail.com] Sent: Friday, February 13, 2009 4:21 PM To: modules-dev@httpd.apache.org Subject: Re: Regarding Multithreading in apache modules On Fri, Feb 13, 2009 at 11:33, Pooja Maheshwari pooja.maheshw...@impetus.co.in wrote: Hi all, Is it possible to use multithreading in apache modules written in C? Yes. As I searched, apr_thread_proc provides process thread management functions. Does it come as part of Apache 2.2? Yes. Apache itself uses threads to serve requests if it is configured to use the mpm_worker module. S -- A: Because it reverses the logical flow of conversation. Q: Why is top-posting frowned upon? A: Top-posting. Q: What is the most annoying thing in e-mail? * Meet us at Hall 2/Booth 2C111 at the Mobile World Congress, Barcelona, Feb 16-19, 2009. * Attend the qLabs webinar on Test Case Prioritization on Feb 20, 2009. Visit www.impetus.com for details. Impetus is the winner of the Economic Times Intel Smart Workplace Awards 2008 and the CNBC emerging India 2008. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference. -- A: Because it reverses the logical flow of conversation. Q: Why is top-posting frowned upon? A: Top-posting. Q: What is the most annoying thing in e-mail?
Re: Transparent proxy setup works fine, but want to confirm the settings
Pranav Desai wrote:
I am trying to setup Apache 2.2.9 as a transparent proxy. So that the
users don't have to configure their browsers. Now the URLs coming in
are relative for transparent proxy, so normally apache tries to look
it up on the filesystem and it obviously fails. So I added a
RewriteRule to convert the relative to absolute URLs.
RewriteEngine On
RewriteRule ^/(.*) http://%{HTTP_HOST}/$1 [P]
RewriteLog logs/rewrite_log
RewriteLogLevel 5
Now, it works perfectly for all traffic expect the one that is
destined for the server itself. E.g.
http://apache_proxy_ip:port/
Whenever I access the above link, the rewrite engine loops and the
server reaches the MaxClient. I have included the log below.
That would make perfect sense though, you are asking the server to send
you to the server prefixed with the host header, and when you use the
hostname of the proxy server itself, you create a loop by definition,
which means...
So, I added some conditions to not apply the RewriteRule for HOST
destined to the server.
RewriteCond %{HTTP_HOST} !10.1.0.206.*
RewriteRule ^/(.*) http://%{HTTP_HOST}/$1 [P]
...this is a sensible workaround.
I wanted to confirm if this is the right way to do transparent proxy
or is there a better way to make it more solid ?
In theory this will work as is, I am not sure whether there is an option
in the proxy to do this natively without the need for rewrite.
Regards,
Graham
--
smime.p7s
Description: S/MIME Cryptographic Signature
DTrace-compatible request_rec for Solaris and Leopard
See http://svn.apache.org/viewvc/httpd/httpd/trunk/ap.d?view=markuppathrev=723565, which has a mapping for some 64-bit build. That needs some help in portability and auto-generation. In case it helps anyone DTracing Apache, attached is a request_rec mapping which matches trunk that is ready to use with Solaris and Leopard on x86 (32-bit and 64-bit). (I have a slightly different one for the 2.2.x branch as well.) See the hints for dtrace command-line parameters near the top. Handling the alignment differences between C-on-ia32 and D (the DTrace language) are the worst part of the exercise. It is very possible that having the offsets and sizes (or name of compatible DTrace data types) available to DTrace scripts is all that is needed. Meanwhile, if the request_rec alignment/padding were cleaned up for 2.4 it should be straightforward for someone to use a script to generate offsets for request_rec fields which match their build of Apache. (The script would generate a C program via apxs which uses offsetof() to output definitions that a DTrace script could use. Is anybody out there DTracing Apache? Any thoughts on having field offsets instead of structure definitions available to scripts, in case the former can be provided in a more maintainable manner? -- Born in Roswell... married an alien... httpd_dtrace.h Description: Binary data
Re: Using gzip and CustomLog
Did anybody had a chance to look at the attached rotatelogs file? On Tue, Feb 10, 2009 at 2:32 PM, Paras Fadte plf...@gmail.com wrote: Hi Rainer, I have attached the modified rotatelogs.c file (originally taken from apache 2.0.55 ) . Can you please have a look at it and let me know its shortcomings and chances that it could seg fault ? Thanks in advance. -Paras On Tue, Feb 10, 2009 at 1:37 PM, Paras Fadte plf...@gmail.com wrote: Thanks Rainer. On Sun, Feb 8, 2009 at 8:50 PM, Rainer Jung rainer.j...@kippdata.de wrote: On 28.01.2009 06:50, Paras Fadte wrote: I have somewhat modified the rotatlogs utility to support compression . Although it creates files in compressed format (.gz) and rotates them properly the issue that i am facing is that when apache is restarted (graceful or stop/start way ) the last created compressed file doesn't seem to get closed . Is there a way to rectify this ? For compression I am using zlib . When httpd is restarted or stopped, then most rotatelogs processes get stopped via a signal. The signal could depend on the platform, but in my case it's SIGTERM. You can truss your rotatelogs to verify yourself, whether that's true for your Linux system too. truss will show you the signal the rotatelogs process received before terminating. Then you need to register a signal handler, e.g. apr_signal(SIG_TERM, my_signal_handler); which gets called automatically, whenever the process receives the respective signal. Your signal handler my_signal_handler() could then set an internal flag, indicating that you want to cleanup and exit rotatelogs. You can check this flag before and after the blocking read from the log pipe, and if it is set, close your gzip output cleanly and exit rotatelogs. You can temporarily deactivate or activate all signal handlers for SIG_TERM with apr_signal_unblock(SIG_TERM); and apr_signal_block(SIG_TERM); The ErrorLog for the global server behaves a little different, when restarting Apache it doesn't get a signal but instead it will get an EPIPE when trying to read from the log pipe. Regards, Rainer On Fri, Jan 23, 2009 at 1:41 PM, Paras Fadteplf...@gmail.com wrote: Thanks Rainer, yeah.. me not a pro at development . On Fri, Jan 23, 2009 at 1:30 PM, Rainer Jungrainer.j...@kippdata.de wrote: On 23.01.2009 08:45, Paras Fadte wrote: Can you please tell me in which file ? I assume you are building rotatelogs from within the httpd sources. There is a file support/Makefile, which contains a line $(LINK) $(rotatelogs_LTFLAGS) $(rotatelogs_OBJECTS) $(PROGRAM_LDADD) Simply add -lz at the end of the line: $(LINK) $(rotatelogs_LTFLAGS) $(rotatelogs_OBJECTS) $(PROGRAM_LDADD) -lz In case you don't know what a Makefile is and how it basically works, you need to read about how to do C software development. Regards, Rainer On Fri, Jan 23, 2009 at 1:09 PM, Rainer Jungrainer.j...@kippdata.de wrote: On 23.01.2009 07:55, Paras Fadte wrote: Hi, I get following error when I try to use compress function of zlib in rotatelogs.c . I have included zlib.h in rotatelogs.c . /home/paras/httpd-2.0.55/support/rotatelogs.c:294: undefined reference to `compress' collect2: ld returned 1 exit status Is it linking error ? where should I make the changes to eliminate this error? Add -lz to the linking flags.
Problems with EOS optimisation in ap_core_output_filter() and file buckets.
In ap_core_output_filter() there exists the code starting with:
/* Completed iterating over the brigade, now determine if we want
* to buffer the brigade or send the brigade out on the network.
*
* Save if we haven't accumulated enough bytes to send, the connection
* is not about to be closed, and:
*
* 1) we didn't see a file, we don't have more passes over the
* brigade to perform, AND we didn't stop at a FLUSH bucket.
* (IOW, we will save plain old bytes such as HTTP headers)
* or
* 2) we hit the EOS and have a keep-alive connection
* (IOW, this response is a bit more complex, but we save it
* with the hope of concatenating with another response)
*/
if (nbytes + flen AP_MIN_BYTES_TO_WRITE
!AP_BUCKET_IS_EOC(last_e)
((!fd !more !APR_BUCKET_IS_FLUSH(last_e))
|| (APR_BUCKET_IS_EOS(last_e)
c-keepalive == AP_CONN_KEEPALIVE))) {
This is some sort of optimisation which in the case of a keep alive
connection, will hold over sending data out on connection until later
if EOS is present and amount of data is less than nominal minimum
bytes to send.
Later in this section of code it has:
/* Do a read on each bucket to pull in the
* data from pipe and socket buckets, so
* that we don't leave their file descriptors
* open indefinitely. Do the same for file
* buckets, with one exception: allow the
* first file bucket in the brigade to remain
* a file bucket, so that we don't end up
* doing an mmap+memcpy every time a client
* requests a 8KB file over a keepalive
* connection.
*/
if (APR_BUCKET_IS_FILE(bucket) !file_bucket_saved) {
file_bucket_saved = 1;
}
else {
const char *buf;
apr_size_t len = 0;
rv = apr_bucket_read(bucket, buf, len,
APR_BLOCK_READ);
if (rv != APR_SUCCESS) {
ap_log_cerror(APLOG_MARK, APLOG_ERR, rv,
c, core_output_filter:
Error reading from bucket.);
return HTTP_INTERNAL_SERVER_ERROR;
}
}
What the end result of the code is, is that if you have a file bucket
getting this far where length of file is less than 8000 and an EOS
follows it, then the actual file bucket is held over rather than data
being read and buffered. This is as commented is to avoid doing an
mmap+memcpy. What it means though is that the file descriptor within
the file bucket must be maintained and cannot be closed as soon as
ap_pass_brigade() has been called.
For me this is an issue as the file descriptor has been supplied from
a special object returned by a higher level application and it would
be hard to maintain the file as open beyond the life of the request,
up till end of keep alive or a subsequent request over same
connection. Doing a dup on the file decriptor is also not necessarily
an option.
The end result then is that later when file bucket is processed, the
file descriptor has already been closed and one gets the error:
(9)Bad file descriptor: core_output_filter: writing data to the network
I know that I can circumvent the EOS optimisation by inserting a flush
bucket, but based on documentation it isn't gauranteed that a flush
bucket will always propagate down the filter chain and actually push
out data.
/**
* Create a flush bucket. This indicates that filters should flush their
* data. There is no guarantee that they will flush it, but this is the
* best we can do.
* @param list The freelist from which this bucket should be allocated
* @return The new bucket, or NULL if allocation failed
*/
APU_DECLARE(apr_bucket *) apr_bucket_flush_create(apr_bucket_alloc_t *list);
How can one gaurantee that the file bucket will actually be flushed
and not held over by a filter?
If it gets to the core output filter, another way to avoid EOS
optimisation is to forcibly set keepalive to AP_CONN_CLOSE, which for
the application concerned is probably reasonable, but is obviously a
bit of a hack.
Finally, my problem only arises because I insert an eos after file
bucket and before calling ap_pass_brigade(). If one uses ap_send_fd(),
it doesn't insert eos before calling ap_pass_brigade(), with something
else obviously later inserting eos. If ap_pass_brigade() is called
without eos first time and only later with eos, will that help in
ensuring it is flushed out, or can
Re: svn commit: r744281 - /httpd/apreq/trunk/glue/perl/xsbuilder/apreq_xs_postperl.h
On Fri, Feb 13, 2009 at 3:27 PM, j...@apache.org wrote: Author: joes Date: Fri Feb 13 23:27:31 2009 New Revision: 744281 URL: http://svn.apache.org/viewvc?rev=744281view=rev Log: revert, yes mortal is correct here Could you give a lamens explanation of sv_2mortal for those of us aspiring hackers? I've seen it all over the place in the mod_perl and apreq codebase (and perl as well), but haven't ever been able to track down what it really does. Modified: httpd/apreq/trunk/glue/perl/xsbuilder/apreq_xs_postperl.h Modified: httpd/apreq/trunk/glue/perl/xsbuilder/apreq_xs_postperl.h URL: http://svn.apache.org/viewvc/httpd/apreq/trunk/glue/perl/xsbuilder/apreq_xs_postperl.h?rev=744281r1=744280r2=744281view=diff == --- httpd/apreq/trunk/glue/perl/xsbuilder/apreq_xs_postperl.h (original) +++ httpd/apreq/trunk/glue/perl/xsbuilder/apreq_xs_postperl.h Fri Feb 13 23:27:31 2009 @@ -250,13 +250,13 @@ } if (obj != Nullsv) -sv_setsv(*hv_fetch(data, _r, 2, 1), newRV_inc(obj)); +sv_setsv(*hv_fetch(data, _r, 2, 1), sv_2mortal(newRV_inc(obj))); sv_setiv(*hv_fetch(data, rc, 2, 1), rc); sv_setpv(*hv_fetch(data, file, 4, 1), CopFILE(PL_curcop)); sv_setiv(*hv_fetch(data, line, 4, 1), CopLINE(PL_curcop)); sv_setpv(*hv_fetch(data, func, 4, 1), func); -sv_setsv(ERRSV, sv_bless(newRV_noinc((SV*)data), stash)); +sv_setsv(ERRSV, sv_2mortal(sv_bless(newRV_noinc((SV*)data), stash))); Perl_croak(aTHX_ Nullch); }
Re: Transparent proxy setup works fine, but want to confirm the settings
On Fri, Feb 13, 2009 at 1:26 AM, Graham Leggett minf...@sharp.fm wrote:
Pranav Desai wrote:
I am trying to setup Apache 2.2.9 as a transparent proxy. So that the
users don't have to configure their browsers. Now the URLs coming in
are relative for transparent proxy, so normally apache tries to look
it up on the filesystem and it obviously fails. So I added a
RewriteRule to convert the relative to absolute URLs.
RewriteEngine On
RewriteRule ^/(.*) http://%{HTTP_HOST}/$1 [P]
RewriteLog logs/rewrite_log
RewriteLogLevel 5
Now, it works perfectly for all traffic expect the one that is
destined for the server itself. E.g.
http://apache_proxy_ip:port/
Whenever I access the above link, the rewrite engine loops and the
server reaches the MaxClient. I have included the log below.
That would make perfect sense though, you are asking the server to send you
to the server prefixed with the host header, and when you use the hostname
of the proxy server itself, you create a loop by definition, which means...
So, I added some conditions to not apply the RewriteRule for HOST
destined to the server.
RewriteCond %{HTTP_HOST} !10.1.0.206.*
RewriteRule ^/(.*) http://%{HTTP_HOST}/$1 [P]
...this is a sensible workaround.
I wanted to confirm if this is the right way to do transparent proxy
or is there a better way to make it more solid ?
In theory this will work as is, I am not sure whether there is an option in
the proxy to do this natively without the need for rewrite.
I checked the proxy, and there isn't anything to specifically do this,
but maybe I could have used some ReverseProxy config to get the same
behavior, but I thought RewriteRule was a bit cleaner.
-- Pranav
Regards,
Graham
--
