Forbidden errors when files are moved over other files on NFS?
Hello All, I'm trying to get to the bottom of a problem my customers are experiencing while serving files that are checked out from SVN on one server on an NFS mount and served by Apache from NFS mounts on other hosts. The problem is not limited to SVN, it is also created by move commands issued on the server (see below). The setup is as following: * One shellserver has all homedirs mounted over NFS. * Clients distribute their sourcecode to several accounts through SVN update commands. * Webservers serve the checked out files from similar NFS mounts. When updating the checkout and thereby modifying a file named (say testfile), SVN creates a temporary file (say testfile.tmp) in which the new content is stored. To guarantee atomicity testfile.tmp is then moved over testfile. So far so good on the shellserver. This is where things go wrong on the webserver hosts, however: 1) The NFS client on the webserver thinks the original file is still in use. 2) Apache accesses the old inode. 3) Apache gets ESTALE and returns a FORBIDDEN status. The file is no longer served. Let us agree that this destroys any chance of using SVN over NFS :) This is easily recreated using an NFS server and two hosts that have an export mounted. Let's do things with mv only, just to show that the problem is not subversion's: 1) On one host (A), create an NFS export. Mount it somewhere on the same machine. 2) On another host (B), mount the same export. 3) Create a file on (A). 4) Access the file on (B) through the fileserver. 5) Create a new file, move it over testfile. 6) Access the file on (B) through the fileserver ESTALE :( [Fri May 23 11:44:40 2008] [error] [client 82.95.102.2] (116)Stale NFS file handle: access to /test/testfile failed Now I know that ESTALE is a critical error. However, something is done differently between Apache and for example cat. Now, moving files over other files is pretty common so I really need to fix the problem. * Is this known behaviour? * Is it even Apache's fault? * Why does stat work but does Apache fail to server the file? There are some additional links at the bottom of the mail for reference. Thanks for your time! Regards, Allard Hoeve Response by an NFS developer: http://sourceforge.net/mailarchive/message.php?msg_name=20080522214136.GB18822%40fieldses.org Another summary: http://linux-nfs.org/pipermail/nfsv4/2008-April/008452.html Same problem, other reporter: http://linux-nfs.org/pipermail/nfsv4/2008-April/008423.html
Resource protection, specified by a database
Hi Is there a way to enforce digest protection on a per resource basis through the use of database as opposed to an htaccess file? So that the database maintains which resources can be accessed by which groups and what resources require authorization. This is so that access protection can be dynamically set on a per resource basis without needing to edit htaccess files. Also, in the same scheme, is it possible to specify that the digest authentication should be optional? i.e. not send an unauthorized header, just a WWW-Authenticate header. I'm not referring to hacks such as remove a header I mean a proper module that takes in this option, preferably to work with the module that does the database specification of resource protection. I'm guessing it would involve setting a custom authentication module for the entire tree, which then consulted a database to see whether it actually needed to require an authorization header, and check the database that the authenticated user should be authorized for the given resource. Seen as your authentication framework has changed a bit, and I see there is a authz_dbd module in development, is there any plan for work in this regard? Does this sound like a feasible manner to accomplish this? Tim
Re: 2.2.9 status
Jim Jagielski wrote: Right now, we are waiting on an APR release before we can release 2.2.9... Does matter if we go with APR 1.3.0 or 1.2.x, we need a release (showstopper related to 1.2) I have noted problem in httpd-2.2.8 with mpm worker when using mod_proxy_ajp ping/pong it seems fixed in 2.2.x. +++ Fri May 23 17:22:00 2008] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong [Fri May 23 17:22:00 2008] [error] ajp_msg_append_uint8(): BufferOverflowException 4 4 [Fri May 23 17:22:00 2008] [error] ajp_handle_cping_cpong: ajp_marshal_into_msgb failed [Fri May 23 17:22:00 2008] [error] (120001)APR does not understand this error code: proxy: AJP: cping/cpong failed to (null) (localhost) [Fri May 23 17:22:00 2008] [debug] proxy_util.c(1873): proxy: AJP: has released connection for (localhost) +++ Any hints what fixes it? - Just to make sure I am not wrong when saying it is fixed now - Cheers Jean-Frederic
Re: mod_proxy race condition bug #37770
I think that the deflate filter might need to be fixed in order to work with this patch, I think the EOC bucket is causing the deflate filter to cause apache to return a 200 OK to the client with a blank page. I had to make our mod_perl application remove the deflate filter from the chain when we got an EOC in order to make this work. So perhaps the deflate filter needs to handle this itself. And perhaps the same problem will affect many other filters. On Thu, May 22, 2008 at 4:57 PM, Adam Woodworth [EMAIL PROTECTED] wrote: Ok my mistake, the setup I was using turned out to be the problem. I'm now able to see the patch working. Sorry about that. Thanks again Ruediger. Now I just have to modify our mod_perl application to behave correctly when mod_proxy sends EOC. Adam On Thu, May 22, 2008 at 4:28 AM, Ruediger Pluem [EMAIL PROTECTED] wrote: On 05/21/2008 10:34 PM, Adam Woodworth wrote: I think there is a problem with r657443 (and thus I assume r645813): I applied the r657443 patch to my copy of the 2.2.8 official release source and it doesn't work right. The problem is that the change to mod_proxy_http.c checks c-keepalives for a value, but c-keepalives is filled out by ap_http_header_filter(), which isn't called until later. So, c-keepalives is always 0 at this point. Which also means No, it is not always 0 at this point. It has a different value once the connection to the client is not used for the first time. that the number of keepalives message in the ap_log_rerror() msg isn't going to mean anything, but at this point it never reaches it. Also, I patched the patch to manually set c-keepalives to 1 for testing so that I could test out this r657443 patch. However, what happened was that Apache would kick back an error page to the client, instead of just closing the connection. So, no improvement. Has this patch been tested and verified anywhere? Yes, with the test code I posted here on list. Regards Rüdiger
Re: mod_proxy race condition bug #37770
And FWIW, when the EOC bucket is sent down, our mod_perl filter sees an ERROR bucket, i.e. the bucket type is ERROR not EOC, if that matters. On Fri, May 23, 2008 at 1:19 PM, Adam Woodworth [EMAIL PROTECTED] wrote: I think that the deflate filter might need to be fixed in order to work with this patch, I think the EOC bucket is causing the deflate filter to cause apache to return a 200 OK to the client with a blank page. I had to make our mod_perl application remove the deflate filter from the chain when we got an EOC in order to make this work. So perhaps the deflate filter needs to handle this itself. And perhaps the same problem will affect many other filters. On Thu, May 22, 2008 at 4:57 PM, Adam Woodworth [EMAIL PROTECTED] wrote: Ok my mistake, the setup I was using turned out to be the problem. I'm now able to see the patch working. Sorry about that. Thanks again Ruediger. Now I just have to modify our mod_perl application to behave correctly when mod_proxy sends EOC. Adam On Thu, May 22, 2008 at 4:28 AM, Ruediger Pluem [EMAIL PROTECTED] wrote: On 05/21/2008 10:34 PM, Adam Woodworth wrote: I think there is a problem with r657443 (and thus I assume r645813): I applied the r657443 patch to my copy of the 2.2.8 official release source and it doesn't work right. The problem is that the change to mod_proxy_http.c checks c-keepalives for a value, but c-keepalives is filled out by ap_http_header_filter(), which isn't called until later. So, c-keepalives is always 0 at this point. Which also means No, it is not always 0 at this point. It has a different value once the connection to the client is not used for the first time. that the number of keepalives message in the ap_log_rerror() msg isn't going to mean anything, but at this point it never reaches it. Also, I patched the patch to manually set c-keepalives to 1 for testing so that I could test out this r657443 patch. However, what happened was that Apache would kick back an error page to the client, instead of just closing the connection. So, no improvement. Has this patch been tested and verified anywhere? Yes, with the test code I posted here on list. Regards Rüdiger
Re: mod_proxy race condition bug #37770
Err, sorry, we do see the EOC bucket, it's just after the ERROR and EOS buckets (duh). :) On Fri, May 23, 2008 at 1:31 PM, Adam Woodworth [EMAIL PROTECTED] wrote: And FWIW, when the EOC bucket is sent down, our mod_perl filter sees an ERROR bucket, i.e. the bucket type is ERROR not EOC, if that matters. On Fri, May 23, 2008 at 1:19 PM, Adam Woodworth [EMAIL PROTECTED] wrote: I think that the deflate filter might need to be fixed in order to work with this patch, I think the EOC bucket is causing the deflate filter to cause apache to return a 200 OK to the client with a blank page. I had to make our mod_perl application remove the deflate filter from the chain when we got an EOC in order to make this work. So perhaps the deflate filter needs to handle this itself. And perhaps the same problem will affect many other filters. On Thu, May 22, 2008 at 4:57 PM, Adam Woodworth [EMAIL PROTECTED] wrote: Ok my mistake, the setup I was using turned out to be the problem. I'm now able to see the patch working. Sorry about that. Thanks again Ruediger. Now I just have to modify our mod_perl application to behave correctly when mod_proxy sends EOC. Adam On Thu, May 22, 2008 at 4:28 AM, Ruediger Pluem [EMAIL PROTECTED] wrote: On 05/21/2008 10:34 PM, Adam Woodworth wrote: I think there is a problem with r657443 (and thus I assume r645813): I applied the r657443 patch to my copy of the 2.2.8 official release source and it doesn't work right. The problem is that the change to mod_proxy_http.c checks c-keepalives for a value, but c-keepalives is filled out by ap_http_header_filter(), which isn't called until later. So, c-keepalives is always 0 at this point. Which also means No, it is not always 0 at this point. It has a different value once the connection to the client is not used for the first time. that the number of keepalives message in the ap_log_rerror() msg isn't going to mean anything, but at this point it never reaches it. Also, I patched the patch to manually set c-keepalives to 1 for testing so that I could test out this r657443 patch. However, what happened was that Apache would kick back an error page to the client, instead of just closing the connection. So, no improvement. Has this patch been tested and verified anywhere? Yes, with the test code I posted here on list. Regards Rüdiger
Re: Palm Treo access to OWA via Apache 2.2.x Proxy
On May 22, 2008, at 1:23 PM, Ray Van Dolson wrote: that of the above. I could open a bz bug for this and include tcpdumps (can the bugs be tagged private?) No, Bugzilla cannot do that. At least not how we have it set up. Don't submit anything you don't want the entire httpd development community (and by extension, the entire world) to see. S. -- Sander Temme [EMAIL PROTECTED] PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF smime.p7s Description: S/MIME cryptographic signature
Re: 2.2.9 status
On 05/23/2008 05:59 PM, jean-frederic clere wrote: Jim Jagielski wrote: Right now, we are waiting on an APR release before we can release 2.2.9... Does matter if we go with APR 1.3.0 or 1.2.x, we need a release (showstopper related to 1.2) I have noted problem in httpd-2.2.8 with mpm worker when using mod_proxy_ajp ping/pong it seems fixed in 2.2.x. +++ Fri May 23 17:22:00 2008] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong [Fri May 23 17:22:00 2008] [error] ajp_msg_append_uint8(): BufferOverflowException 4 4 [Fri May 23 17:22:00 2008] [error] ajp_handle_cping_cpong: ajp_marshal_into_msgb failed [Fri May 23 17:22:00 2008] [error] (120001)APR does not understand this error code: proxy: AJP: cping/cpong failed to (null) (localhost) [Fri May 23 17:22:00 2008] [debug] proxy_util.c(1873): proxy: AJP: has released connection for (localhost) +++ Any hints what fixes it? - Just to make sure I am not wrong when saying it is fixed now - I guess this was https://svn.apache.org/viewvc?view=revrevision=615931. Back it out and try if it breaks again :-). Regards Rüdiger
Re: mod_proxy race condition bug #37770
On 05/23/2008 07:55 PM, Adam Woodworth wrote: Err, sorry, we do see the EOC bucket, it's just after the ERROR and EOS buckets (duh). :) This is what I expect. I have checked and I think I know what's going wrong, but I have no solution yet how to fix it. The solution I am looking for should not involve a change to the mod_deflate filter or the need to remove it from the chain. Regards Rüdiger
Re: mod_proxy race condition bug #37770
Thanks Ruediger. Keep us posted. Is this mod_proxy patch going to get into 2.2.9, or wait until another 2.2.x release when the deflate issue can be resolved? Cheers, Adam On Fri, May 23, 2008 at 5:51 PM, Ruediger Pluem [EMAIL PROTECTED] wrote: On 05/23/2008 07:55 PM, Adam Woodworth wrote: Err, sorry, we do see the EOC bucket, it's just after the ERROR and EOS buckets (duh). :) This is what I expect. I have checked and I think I know what's going wrong, but I have no solution yet how to fix it. The solution I am looking for should not involve a change to the mod_deflate filter or the need to remove it from the chain. Regards Rüdiger
Re: mod_proxy race condition bug #37770
On 05/24/2008 12:10 AM, Adam Woodworth wrote: Thanks Ruediger. Keep us posted. Is this mod_proxy patch going to get into 2.2.9, or wait until another 2.2.x release when the deflate issue can be resolved? As it looks currently it will be in 2.2.9, which doesn't mean that I do not find a fix for this issue before 2.2.9 :-). Regards Rüdiger
