So we're likely dealing with mod_security overriding the response. I haven't tinkered with that module in some time to give you a definitive answer.
On Fri, Sep 15, 2023 at 11:49 AM Marc <m...@f1-outsourcing.eu> wrote: > > > > What is returning the 500 response here? Is php/python/perl involved? > > No, I think this mod_security is generating this > > > As for the scrapers, you are absolutely wasting your time customizing the > > response. I would just return a 403, actually. > > I think you might be right. I did not expect to waste so much time on > trying to just send an 'empty' body. > > > > > > > > See the ErrorDocument directive. > > > > It does not seem to work. It looks like this config is skipped and > > the error is loaded directly from the httpd binary. > > > > ErrorDocument 500 /406.html > > > > > Now, why is that response not suitable? And why would you respond > > with a > > > 200 for a blocked user agent? > > > > I think it is better to return to scrapers 200 and empty content, > > instead of notifying them so they can reconfigure their systems. > > > > > > > > > > > Where/how can I change this message? > > > > > > <p>The server encountered an internal error or > > > misconfiguration and was unable to complete > > > your request.</p> > > > <p>Please contact the server administrator at > > > xxx to inform them of the time this error occurred, > > > and the actions you performed just before this error.</p> > > > <p>More information about this error may be available > > > in the server error log.</p> > > > </body></html> > > > > > > or as a work-a-round, how can refuse access with > modsecurity > > and just > > > generate a 200 blank page response. > > > > > > SecRule REQUEST_HEADERS:User-Agent "blockthisua" > > > "id:'13006',phase:2,log,deny,status:200" > > > > > > > > >
