Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0
Also, sorry I didn't see the APR question below. httpd -V Server version: Apache/2.4.46 (Unix) Server built: Nov 24 2020 16:30:58 Server's Module Magic Number: 20120211:93 Server loaded: APR 1.7.0, APR-UTIL 1.6.1 Compiled using: APR 1.7.0, APR-UTIL 1.6.1 Architecture: 64-bit Server MPM: worker threaded: yes (fixed thread count) forked: yes (variable process count) Server compiled with -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_PROC_PTHREAD_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=256 -D HTTPD_ROOT="/home/rc/objs/ssws_ssws_rh6/linux/install" -D SUEXEC_BIN="/home/rc/objs/ssws_ssws_rh6/linux/install/bin/suexec" -D DEFAULT_PIDLOG="logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf" svc...@web70.lab1.ariba.com> On Mon, Dec 7, 2020, 5:41 PM Yann Ylavic wrote: > On Mon, Dec 7, 2020 at 6:33 PM Wendell Hatcher > wrote: > > > > Yann, quick question would it be ok to have a cronjob running daily > maybe at slow periods that runs a script to delete the sephmore segments > using a script while apache is running? It shouldn't cause issues with the > apache server processing data correct? > > I think that the error is about shared memories (SHM), not semaphores > (your httpd -V output shows "-D APR_USE_PROC_PTHREAD_SERIALIZE" which > indicates the use of pthread global mutexes, not backed by a system > descriptor/inode, so nothing to cleanup really). > > The SHMs usually do have a descriptor or inode to delete somewhere on > your system, but I don't think it's safe to do so while httpd is > running. > By the way, you didn't tell which SHM mechanism was used by the APR > library on your system (which system? httpd/APR compiled manually? > SysV, POSIX or other SHM mechanism? Where do your SHMs land on the > filesystem?). > It is not shown by httpd -V unfortunately (I added it to trunk lately, > but it's not in any 2.4 version yet). > > > > Second ndly, we want to run a script during our apache stop process that > will deleting the files from the shm folder we are hoping this will > reschedule le our hanging apache servers during stops and starts with a > slotmem error. > > I'm not sure to understand this, deleting the SHMs could avoid > conflicts temporarily but as soon as all the httpd instances are > started again, the conflicts will show up again if they exist in the > configuration files. > > One particular thing to look at are the URLs used in balancer://[id]> declarations, in all the instances. > Each balancer should have a unique [id], system wide, because a SHM > will be created (system wide) based on this [id]. This is usually not > an issue with a single httpd instance where uniqueness makes sense and > balancer1,..,n is simply/often used, but when this first instance gets > copied to create others then it breaks.. > You can use almost anything as [id], provided it's unique and the same > [id] is used in the associated ProxyPass. It can be a UUID for > example, a tool like `uuidgen` can generate one when a new balancer > needs to be added. > > Hth, > Yann. >
Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0
Awesome, thanks! This is what I figured as well that the issue will
continue if we did not update the balancer. Let me look into it further and
see if we can figure it out how to update the balancer names to unique IDs
I didn't think about this and thought about splitting our mod_proxy conf
files using multiple vhosts. This wouldn't work as we use the same fill url
call https://server one.com for all services.
Below is what we're thinking of doing but I feel it will not solve the
problem and am worried that deleting the sephmore segments while the server
is running can cause issues in production.
Our plan to patch the problem below.
Update kernel parameters: kernel.sem= 500 32000 200 256 (inside of the
/etc/sysctl.conf ) (Completed in devlab).
3. Use root crontab job to remove unused semaphores (Not Completed) :
for x in $(ipcs -m | awk 'NR>1 {print $2}'); do ipcrm -m $x; done
4. Update shutdown apache script to remove .shm files during stopsvc (Not
Completed)
On Mon, Dec 7, 2020, 5:41 PM Yann Ylavic wrote:
> On Mon, Dec 7, 2020 at 6:33 PM Wendell Hatcher
> wrote:
> >
> > Yann, quick question would it be ok to have a cronjob running daily
> maybe at slow periods that runs a script to delete the sephmore segments
> using a script while apache is running? It shouldn't cause issues with the
> apache server processing data correct?
>
> I think that the error is about shared memories (SHM), not semaphores
> (your httpd -V output shows "-D APR_USE_PROC_PTHREAD_SERIALIZE" which
> indicates the use of pthread global mutexes, not backed by a system
> descriptor/inode, so nothing to cleanup really).
>
> The SHMs usually do have a descriptor or inode to delete somewhere on
> your system, but I don't think it's safe to do so while httpd is
> running.
> By the way, you didn't tell which SHM mechanism was used by the APR
> library on your system (which system? httpd/APR compiled manually?
> SysV, POSIX or other SHM mechanism? Where do your SHMs land on the
> filesystem?).
> It is not shown by httpd -V unfortunately (I added it to trunk lately,
> but it's not in any 2.4 version yet).
>
>
> > Second ndly, we want to run a script during our apache stop process that
> will deleting the files from the shm folder we are hoping this will
> reschedule le our hanging apache servers during stops and starts with a
> slotmem error.
>
> I'm not sure to understand this, deleting the SHMs could avoid
> conflicts temporarily but as soon as all the httpd instances are
> started again, the conflicts will show up again if they exist in the
> configuration files.
>
> One particular thing to look at are the URLs used in balancer://[id]> declarations, in all the instances.
> Each balancer should have a unique [id], system wide, because a SHM
> will be created (system wide) based on this [id]. This is usually not
> an issue with a single httpd instance where uniqueness makes sense and
> balancer1,..,n is simply/often used, but when this first instance gets
> copied to create others then it breaks..
> You can use almost anything as [id], provided it's unique and the same
> [id] is used in the associated ProxyPass. It can be a UUID for
> example, a tool like `uuidgen` can generate one when a new balancer
> needs to be added.
>
> Hth,
> Yann.
>
Re: [users@httpd] Re: GET working but POST failing with error 502
On Mon, Dec 7, 2020 at 5:55 PM Anton Shepelev wrote: > > All POST requests fail: when I either repeat the same > requiest over and over, or try different requrest. All GET > requests work. This sounds like a crash or something which causes Redmine to close the connection upon receipt of the POST request (without any response). > > > If some requests succeed, it may be a problem of > > connection reuse between Redmine (running on 127.0.0.1) > > and the httpd proxy (configured to forward the requests to > > localhost). > > I should still like to sound that possibilty. Can you please > explain to the Apache dummy that I am how to reconfigure > either httpd or Redmine hosting to avoid that collition? I don't know what a "Bitnami Redmine stack" httpd configuration looks like unfortunately. There should be a line like: ProxyPass ... http://127.0.0.1:3002 ... or: ... ProxyPass http://127.0.0.1:3002 ... ... somewhere, and adding disablereuse=on to the end of this ProxyPass line would disable connection reuse. If that changes nothing, something is happening in the Redmine application which httpd really can't know about.. If Redmine crashed (which could explain why there are no logs on its side), you should see something like "child pid exit signal..." in one of the log files. Possibly you could ask on Bitnami forums how to get more traces from Redmine. Hth, Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0
On Mon, Dec 7, 2020 at 6:33 PM Wendell Hatcher wrote: > > Yann, quick question would it be ok to have a cronjob running daily maybe at > slow periods that runs a script to delete the sephmore segments using a > script while apache is running? It shouldn't cause issues with the apache > server processing data correct? I think that the error is about shared memories (SHM), not semaphores (your httpd -V output shows "-D APR_USE_PROC_PTHREAD_SERIALIZE" which indicates the use of pthread global mutexes, not backed by a system descriptor/inode, so nothing to cleanup really). The SHMs usually do have a descriptor or inode to delete somewhere on your system, but I don't think it's safe to do so while httpd is running. By the way, you didn't tell which SHM mechanism was used by the APR library on your system (which system? httpd/APR compiled manually? SysV, POSIX or other SHM mechanism? Where do your SHMs land on the filesystem?). It is not shown by httpd -V unfortunately (I added it to trunk lately, but it's not in any 2.4 version yet). > Second ndly, we want to run a script during our apache stop process that will > deleting the files from the shm folder we are hoping this will reschedule le > our hanging apache servers during stops and starts with a slotmem error. I'm not sure to understand this, deleting the SHMs could avoid conflicts temporarily but as soon as all the httpd instances are started again, the conflicts will show up again if they exist in the configuration files. One particular thing to look at are the URLs used in declarations, in all the instances. Each balancer should have a unique [id], system wide, because a SHM will be created (system wide) based on this [id]. This is usually not an issue with a single httpd instance where uniqueness makes sense and balancer1,..,n is simply/often used, but when this first instance gets copied to create others then it breaks.. You can use almost anything as [id], provided it's unique and the same [id] is used in the associated ProxyPass. It can be a UUID for example, a tool like `uuidgen` can generate one when a new balancer needs to be added. Hth, Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0
Yann, quick question would it be ok to have a cronjob running daily maybe at slow periods that runs a script to delete the sephmore segments using a script while apache is running? It shouldn't cause issues with the apache server processing data correct? Second ndly, we want to run a script during our apache stop process that will deleting the files from the shm folder we are hoping this will reschedule le our hanging apache servers during stops and starts with a slotmem error. -Wendell On Sun, Dec 6, 2020, 6:45 PM Wendell Hatcher wrote: > Will do! I appreciate the help with this as well. > > On Sun, Dec 6, 2020, 6:26 PM Yann Ylavic wrote: > >> The logs from mod_slotmem_shm are the most interesting, so "LogLevel >> slotmem_shm:debug" should be enough (to limit debug logging to this >> module). >> This should show which shared memories paths (i.e. >> .../slotmem-shm-*.shm) are created/reused/attached by which instance, >> and figure out what kind of duplicates there may be.. >> >> On Sun, Dec 6, 2020 at 8:21 PM Wendell Hatcher >> wrote: >> > >> > Let me gather this information from one of our servers next week. I >> will provide a little more background information tomorrow as well. >> > >> > On Sat, Dec 5, 2020, 7:08 PM Yann Ylavic wrote: >> >> >> >> On Fri, Dec 4, 2020 at 7:45 PM Wendell Hatcher >> >> wrote: >> >> > >> >> > This has nothing to do with underlying SHM stuff and everything to >> do with how multiple balancers, global and outside of virtual hosts, are >> initialized and sometimes persisted. This was changed greatly in 2.4.29.You >> should find the minimal set of balancers that reproduces the error and It >> is likely related to duplicate virtual hosts or duplicate proxy "workers". >> >> >> >> Could you please provide the full error_log (with LogLevel debug) of >> >> the httpd startup failure? >> >> If there are multiple instances (and potential duplicates), the >> >> error_log of all the instances may help too (with LogLevel debug, >> >> still). >> >> >> >> Regards; >> >> Yann. >> >> >> >> - >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> >> For additional commands, e-mail: users-h...@httpd.apache.org >> >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> For additional commands, e-mail: users-h...@httpd.apache.org >> >>
[users@httpd] Re: GET working but POST failing with error 502
Yann Ylavic: > Do all the POST requests fail or only some of them? All POST requests fail: when I either repeat the same requiest over and over, or try different requrest. All GET requests work. > If some requests succeed, it may be a problem of > connection reuse between Redmine (running on 127.0.0.1) > and the httpd proxy (configured to forward the requests to > localhost). I should still like to sound that possibilty. Can you please explain to the Apache dummy that I am how to reconfigure either httpd or Redmine hosting to avoid that collition? - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
[users@httpd] Updating apache from 2.4.6 to 2.4.46
Hi All, I am building RPM for apace 2.4.46 as want o upgrade 2.4.6 to 2.4.46. Before building httpd rpm, built and installed apr and apr-util. While building httpd rpm I am getting below error, Executed command : rpmbuild -tb httpd-2.4.46.tar.bz2 Error encountered : ab.o: In function `main': /root/rpmbuild/BUILD/httpd-2.4.46/support/ab.c:2305: undefined reference to `TLS_client_method' /root/rpmbuild/BUILD/httpd-2.4.46/support/ab.c:2560: undefined reference to `TLS_client_method' /root/rpmbuild/BUILD/httpd-2.4.46/support/ab.c:2637: undefined reference to `OPENSSL_init_ssl' /root/rpmbuild/BUILD/httpd-2.4.46/support/ab.c:2638: undefined reference to `OPENSSL_init_ssl' /root/rpmbuild/BUILD/httpd-2.4.46/support/ab.c:2647: undefined reference to `SSL_CTX_set_options' ab.o: In function `test': /root/rpmbuild/BUILD/httpd-2.4.46/support/ab.c:1990: undefined reference to `SSL_in_init' collect2: error: ld returned 1 exit status make[2]: *** [ab] Error 1 make[2]: Leaving directory `/root/rpmbuild/BUILD/httpd-2.4.46/support' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/root/rpmbuild/BUILD/httpd-2.4.46/support' make: *** [all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.hmVImT (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.hmVImT (%build) Can you please help me out ? Thanks, Kapil This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.
Re: [users@httpd] Re: GET working but POST failing with error 502
On Mon, Dec 7, 2020 at 1:39 PM Anton Shepelev wrote: > > I wrote: > > > The problem is that Redmine's REST API works for reading > > but not for writing, that is the GET command works whereas > > the POST command fails with error 502 > > Even with the most detailed logging, the failed POST > requests do not appear in the Redmine log, whereas > successful GET request are all there. Does that mean POST > requests are lost before they even reach Redmine? If so, it > may be a problem on the level of Apache or network. We > access our test Remine locally, via 127.0.0.1 . Do all the POST requests fail or only some of them? If some requests succeed, it may be a problem of connection reuse between Redmine (running on 127.0.0.1) and the httpd proxy (configured to forward the requests to localhost). Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
[users@httpd] Re: GET working but POST failing with error 502
I wrote: > The problem is that Redmine's REST API works for reading > but not for writing, that is the GET command works whereas > the POST command fails with error 502 Even with the most detailed logging, the failed POST requests do not appear in the Redmine log, whereas successful GET request are all there. Does that mean POST requests are lost before they even reach Redmine? If so, it may be a problem on the level of Apache or network. We access our test Remine locally, via 127.0.0.1 . - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
