On Wed, 1 Sept 2021 at 00:46, Dino Ciuffetti wrote:
> Reading the source code:
>
> From mod_slotmem_shm:
> ...
> 401 apr_shm_remove(fname, pool);
> 402 rv = apr_shm_create(&shm, size, fname, gpool);
> ...
> 408 ap_log_error(APLOG_MARK, rv == APR_SUCCESS ? APLOG_DEBUG :
> APLOG_ERR,
> 409 rv, ap_server_conf, APLOGNO(02611)
> 410 "create: apr_shm_%s(%s) %s",
> 411 fbased && is_child_process() ? "attach" :
> "create",
> 412 fname, rv == APR_SUCCESS ? "succeeded" :
> "failed");
>
>
> Autoconf defines APR_USE_SHMEM_SHMGET for SHM namebased memory allocation
> ...
> decision on anonymous shared memory allocation method... 4.4BSD-style
> mmap() via MAP_ANON
> decision on namebased memory allocation method... SysV IPC shmget()
>
>
> in APR, shm.c, apr_shm_create():
> 380 if ((new_m->shmid = shmget(new_m->shmkey, new_m->realsize,
> 381SHM_R | SHM_W | IPC_CREAT |
> IPC_EXCL)) < 0) {
> 382 apr_file_close(file);
> 383 return errno;
> 384 }
>
>
> From shmget() manual page, possible errors in errno:
>ENOSPC: All possible shared memory IDs have been taken (SHMMNI), or
> allocating a segment of the requested size would cause the system to
> exceed the system-wide limit on shared memory (SHMALL).
>
> So your "No space left on device" is not on your filesystem but on your
> shared memory sysv.
>
> Since SHMALL on a 64bit linux system is very big (please try this: cat
> /proc/sys/kernel/shmall), I would bet on a low SHMMNI value on your system
> (pls: cat /proc/sys/kernel/shmmni)
>
> SHMMNI is the global maximum number of shared memory segments on your
> system.
> Default on Ubuntu should be 4096. Please try to increase the value (echo
> 8192 > /proc/sys/kernel/shmmni) or more.
>
>
> HTH.
> Ciao, Dino.
>
>
>
> 27 agosto 2021 11:43, "Spil Oss" wrote:
>
> > Hi,
> >
> > I've been experiencing a failed apache2 service on Ubuntu 21.04 when
> > performing a reload using the `systemctl reload apache2` command. The
> > command does not always fail, but seems to be failing more often as
> > the number of vhosts increases (currently ca 120). My
> >
> > The `systemctl reload apache2` command exits without error, but the
> > service ends up in a failed state. Running `systemctl start apache2`
> > after this failure starts the service without issues.
> > I had taken to do running `systemctl reload apache2; systemctl status
> > apache2` to validate that I have a running service, but this would
> > report "success" even when the service is "failed".
> >
> > Expecting some timing issue, I increased the "RestartSec" systemd
> > parameter to 500ms using
> > `/etc/systemd/system/apache2.service.d/override.conf`
> > [Service]
> > RestartSec=500ms
> >
> > This has not fixed the issue either.
> >
> > Testing the reload using `apachectl -k graceful` can also trigger the
> > "failed" state of the process.
> >
> > The consistent error is with the persistence of shared memory
> > segments. The indicated error is incorrect, there's plenty of space on
> > the filesystem. The configuration has been kept as close as possible
> > to the default Ubuntu config.
> >
> > My gut feeling is some weird interaction between graceful and systemd
> > as seen in the logs. The RestartSec change not solving the problem
> > kind of goes against that.
> >
> > Any help appreciated! Thanks, Bernard Spil.
> >
> > $ df -h /var/run/apache2/
> > Filesystem Size Used Avail Use% Mounted on
> > tmpfs 1.6G 6.6M 1.6G 1% /run
> >
> > $ ls mods-enabled/*.load | sed 's/mods-enabled\///;s/\.load//'
> > access_compat
> > alias
> > auth_mellon
> > authn_core
> > authn_file
> > authz_core
> > authz_host
> > authz_user
> > brotli
> > deflate
> > dir
> > env
> > filter
> > headers
> > http2
> > lbmethod_byrequests
> > mime
> > mpm_event
> > negotiation
> > proxy
> > proxy_balancer
> > proxy_http
> > proxy_http2
> > proxy_wstunnel
> > remoteip
> > reqtimeout
> > rewrite
> > setenvif
> > slotmem_shm
> > socache_shmcb
> > ssl
> > status
> >
> > /var/log/apache2/error.log:
> > [Fri Aug 27 00:00:18.881934 2021] [mpm_event:notice] [pid 138928:tid
> > 140168396681856] AH00493: SIGUSR1 received. Doing graceful restart
> > [Fri Aug 27 00:00:19.155640 2021] [slotmem_shm:error] [pid 138928:tid
> > 140168396681856] (28)No space left on device: AH02611: create:
> >
> apr_shm_create(/var/run/apache2/slotmem-shm-pd38fd8d0_acc_example_org_6.shm)
> > failed
> > [Fri Aug 27 00:00:19.155679 2021] [:emerg] [pid 138928:tid
> > 140168396681856] AH00020: Configuration Failed, exiting
> > [Fri Aug 27 00:05:01.645184 2021] [core:warn] [pid 166984:tid
> > 140602886914688] AH00098: pid file /var/run/apache2/apache2.pid
> > overwritten -- Unclean shutdown of previous Apache run?
> > [Fri Aug 27 00:05:01.656692 2021] [mpm_event:notice] [pid 166984:tid
> > 140602886914688] AH00489: Apache/2.4.46 (Ubuntu) OpenSS
