Re: [GENERAL] Upgrade from 9.4 -> 9.5, FreeBSD 10.2-STABLE, fails on initdb

[Tom Lane]

I wrote:
> Karl Denninger  writes:
>> $ initdb -D data-default
>> ...
>> creating template1 database in data-default/base/1 ... FATAL:  could not
>> create semaphores: Invalid argument
>> DETAIL:  Failed system call was semget(2, 17, 03600).

> Hmm.  On my Linux box, "man semget" says EINVAL means

>EINVAL nsems  is less than 0 or greater than the limit on the number of
>   semaphores per semaphore set (SEMMSL), or a semaphore set corre-
>   sponding  to  key  already  exists, and nsems is larger than the
>   number of semaphores in that set.

> which agrees with the POSIX spec.  Is FreeBSD the same?

BTW, looking at the code, I see that during initdb we would have tried
semaphore key 1 before 2.  So presumably, on key 1 we got an error code
that we recognized as meaning "semaphore set already exists", but then on
key 2 we got EINVAL instead.  That makes this even more curious.  I'd
be interested to see what "ipcs -s" says, if you have that command.
(You might need to run it as root to be sure it will show all sempaphores.)

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Upgrade from 9.4 -> 9.5, FreeBSD 10.2-STABLE, fails on initdb

[Karl Denninger]

On 2/4/2016 12:47, Tom Lane wrote:
> I wrote:
>> Karl Denninger  writes:
>>> $ initdb -D data-default
>>> ...
>>> creating template1 database in data-default/base/1 ... FATAL:  could not
>>> create semaphores: Invalid argument
>>> DETAIL:  Failed system call was semget(2, 17, 03600).
>> Hmm.  On my Linux box, "man semget" says EINVAL means
>>EINVAL nsems  is less than 0 or greater than the limit on the number 
>> of
>>   semaphores per semaphore set (SEMMSL), or a semaphore set 
>> corre-
>>   sponding  to  key  already  exists, and nsems is larger than 
>> the
>>   number of semaphores in that set.
>> which agrees with the POSIX spec.  Is FreeBSD the same?
> BTW, looking at the code, I see that during initdb we would have tried
> semaphore key 1 before 2.  So presumably, on key 1 we got an error code
> that we recognized as meaning "semaphore set already exists", but then on
> key 2 we got EINVAL instead.  That makes this even more curious.  I'd
> be interested to see what "ipcs -s" says, if you have that command.
> (You might need to run it as root to be sure it will show all sempaphores.)
>
>   regards, tom lane
There was indeed a "2" key out by the web server process; I shut it down
and cleared it and the upgrade is now running

Also filed a kernel bug with the FreeBSD folks against 10.2-STABLE as
the man page says you should have gotten back EEXIST.


-- 
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [GENERAL] Upgrade from 9.4 -> 9.5, FreeBSD 10.2-STABLE, fails on initdb

[Adrian Klaver]

On 02/04/2016 10:02 AM, Karl Denninger wrote:

$ initdb -D data-default
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory data-default ... ok
creating subdirectories ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 400kB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in data-default/base/1 ... FATAL:  could not
create semaphores: Invalid argument
DETAIL:  Failed system call was semget(2, 17, 03600).
child process exited with exit code 1
initdb: removing contents of data directory "data-default"
$
$ sysctl -a|grep semm
kern.ipc.semmsl: 512
kern.ipc.semmnu: 256
kern.ipc.semmns: 512
kern.ipc.semmni: 256

The system is running 9.4 just fine and the kernel configuration
requirements shouldn't have changed for semaphores should they?



Where did the 9.5 version come from and was it the same source as the 
9.4 version?




--
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/



--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Upgrade from 9.4 -> 9.5, FreeBSD 10.2-STABLE, fails on initdb

[Tom Lane]

Karl Denninger  writes:
> $ initdb -D data-default
> ...
> creating template1 database in data-default/base/1 ... FATAL:  could not
> create semaphores: Invalid argument
> DETAIL:  Failed system call was semget(2, 17, 03600).

Hmm.  On my Linux box, "man semget" says EINVAL means

   EINVAL nsems  is less than 0 or greater than the limit on the number of
  semaphores per semaphore set (SEMMSL), or a semaphore set corre-
  sponding  to  key  already  exists, and nsems is larger than the
  number of semaphores in that set.

which agrees with the POSIX spec.  Is FreeBSD the same?

Proceeding on the assumption that it is ...

17 is the same nsems value we've been using for donkey's years, so the
SEMMSL aspect of this seems unlikely to apply; what presumably is
happening is a collision with an existing semaphore's key.  Our code is
prepared for that, but it expects a different error code in such cases,
either EEXIST or EACCES:

/*
 * Fail quietly if error indicates a collision with existing set. One
 * would expect EEXIST, given that we said IPC_EXCL, but perhaps we
 * could get a permission violation instead?  Also, EIDRM might occur
 * if an old set is slated for destruction but not gone yet.
 */

It sounds like your kernel is returning EINVAL in preference to any of
those codes, which would be pretty broken.  I do not want to make our code
treat EINVAL as meaning we should retry with a different key, because if
the problem is indeed the SEMMSL limit, we'd be in an infinite loop.

You can probably get past this for the moment if you can remove the
semaphore set with key 2, but I'd advise filing a FreeBSD kernel
bug about their choice of errno.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Upgrade from 9.4 -> 9.5, FreeBSD 10.2-STABLE, fails on initdb

[Adrian Klaver]

On 02/04/2016 10:02 AM, Karl Denninger wrote:

$ initdb -D data-default
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory data-default ... ok
creating subdirectories ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 400kB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in data-default/base/1 ... FATAL:  could not
create semaphores: Invalid argument
DETAIL:  Failed system call was semget(2, 17, 03600).
child process exited with exit code 1
initdb: removing contents of data directory "data-default"
$
$ sysctl -a|grep semm
kern.ipc.semmsl: 512
kern.ipc.semmnu: 256
kern.ipc.semmns: 512
kern.ipc.semmni: 256

The system is running 9.4 just fine and the kernel configuration
requirements shouldn't have changed for semaphores should they?


If it helps, the hint from the source code:

http://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/port/sysv_sema.c;h=f6f15169200a03e9da46ae348994f04297d22017;hb=HEAD

"This error does *not* mean that you have run out of disk space. It 
occurs when either the system limit for the maximum number of semaphore 
sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), 
would be exceeded.  You need to raise the respective kernel parameter. 
Alternatively, reduce PostgreSQL's consumption of semaphores by reducing 
its max_connections parameter.
The PostgreSQL documentation contains more information about configuring 
your system for PostgreSQL."





--
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/



--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Upgrade from 9.4 -> 9.5, FreeBSD 10.2-STABLE, fails on initdb

[Karl Denninger]

On 2/4/2016 12:28, Tom Lane wrote:
> Karl Denninger  writes:
>> $ initdb -D data-default
>> ...
>> creating template1 database in data-default/base/1 ... FATAL:  could not
>> create semaphores: Invalid argument
>> DETAIL:  Failed system call was semget(2, 17, 03600).
> Hmm.  On my Linux box, "man semget" says EINVAL means
>
>EINVAL nsems  is less than 0 or greater than the limit on the number of
>   semaphores per semaphore set (SEMMSL), or a semaphore set corre-
>   sponding  to  key  already  exists, and nsems is larger than the
>   number of semaphores in that set.
>
> which agrees with the POSIX spec.  Is FreeBSD the same?
>
> Proceeding on the assumption that it is ...
>
> 17 is the same nsems value we've been using for donkey's years, so the
> SEMMSL aspect of this seems unlikely to apply; what presumably is
> happening is a collision with an existing semaphore's key.  Our code is
> prepared for that, but it expects a different error code in such cases,
> either EEXIST or EACCES:
>
> /*
>  * Fail quietly if error indicates a collision with existing set. One
>  * would expect EEXIST, given that we said IPC_EXCL, but perhaps we
>  * could get a permission violation instead?  Also, EIDRM might occur
>  * if an old set is slated for destruction but not gone yet.
>  */
>
> It sounds like your kernel is returning EINVAL in preference to any of
> those codes, which would be pretty broken.  I do not want to make our code
> treat EINVAL as meaning we should retry with a different key, because if
> the problem is indeed the SEMMSL limit, we'd be in an infinite loop.
>
> You can probably get past this for the moment if you can remove the
> semaphore set with key 2, but I'd advise filing a FreeBSD kernel
> bug about their choice of errno.
>
>   regards, tom lane

That sounds like it well may be the problem

s655642 --rw-rw-rw- www  www  www 
www 3 12:29:14  7:56:04


Oh look, the web server process has a semaphore set out with a "2" key

I've filed a bug report.  Thanks.

-- 
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/


smime.p7s
Description: S/MIME Cryptographic Signature