Re: [HACKERS] Incorrect formula for SysV IPC parameters

[Kyotaro HORIGUCHI]

Thank you for pushing this.

At Tue, 16 Feb 2016 15:07:32 +0900, Fujii Masao  wrote 
in 

Re: [HACKERS] Incorrect formula for SysV IPC parameters

[Fujii Masao]

On Fri, Feb 12, 2016 at 11:19 PM, Fujii Masao  wrote:
> On Fri, Feb 5, 2016 at 2:17 PM, Kyotaro HORIGUCHI
>  wrote:
>> At Thu, 4 Feb 2016 21:43:04 +0900, Fujii Masao  wrote 
>> in 
>>> On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI
>>>  wrote:
>>> > Hello, I found that the formulas to calculate SEMMNI and SEMMNS
>>> > are incorrect in 9.2 and later.
>>> >
>>> > http://www.postgresql.org/docs/9.5/static/kernel-resources.html
>>> >
>>> > But actually the number of semaphores PostgreSQL needs is
>>> > calculated as following in 9.4 and later.
>> ...
>>> > So, the formula for SEMMNI should be
>>> >
>>> > ceil((max_connections + autovacuum_max_workers + max_worker_processes + 
>>> > 5) / 16)
>>> >
>>> > and SEMMNS should have the same fix.
>>> >
>>> >
>>> > In 9.3 and 9.2, the documentation says the same thing but
>> ...
>>> > ceil((max_connections + autovacuum_max_workers + 5) / 16)
>>> >
>>> > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct.
>>>
>>> Good catch!
>>
>> Thanks.
>>
>>> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS
>>> under the Table 17-1.
>>
>> Oops! Thank you for pointing it out.
>>
>> The original description doesn't mention the magic-number ('5' in
>> the above formulas, which previously was '4') so I haven't added
>> anything about it.
>>
>> Process of which the number is limited by max_worker_processes is
>> called 'background process' (not 'backend worker') in the
>> documentation so I used the name to mention it in the additional
>> description.
>>
>> The difference in the body text for 9.2, 9.3 is only a literal
>> '4' to '5' in the formula.
>
> Thanks for updating the patches!
>
> They look good to me except that the formulas don't include the number of
> background processes requesting shared memory access, i.e.,
> GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following
> formula in 9.3?
>
>   ceil((max_connections + autovacuum_max_workers + number of
> background proceses + 5) / 16)
>
> Attached patch uses the above formula for 9.3. I'm thinking to push your
> patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3.
> Comments?

Pushed. Thanks for the report and patches!

Regards,

-- 
Fujii Masao


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Incorrect formula for SysV IPC parameters

[Fujii Masao]

On Mon, Feb 15, 2016 at 2:31 PM, Kyotaro HORIGUCHI
 wrote:
> Thanks for looking at this.
>
> At Fri, 12 Feb 2016 23:19:45 +0900, Fujii Masao  wrote 
> in 
>> >> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS
>> >> under the Table 17-1.
>> >
>> > Oops! Thank you for pointing it out.
>> >
>> > The original description doesn't mention the magic-number ('5' in
>> > the above formulas, which previously was '4') so I haven't added
>> > anything about it.
>> >
>> > Process of which the number is limited by max_worker_processes is
>> > called 'background process' (not 'backend worker') in the
>> > documentation so I used the name to mention it in the additional
>> > description.
>> >
>> > The difference in the body text for 9.2, 9.3 is only a literal
>> > '4' to '5' in the formula.
>>
>> Thanks for updating the patches!
>>
>> They look good to me except that the formulas don't include the number of
>> background processes requesting shared memory access, i.e.,
>> GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following
>> formula in 9.3?
>>
>>   ceil((max_connections + autovacuum_max_workers + number of
>> background proceses + 5) / 16)
>>
>> Attached patch uses the above formula for 9.3. I'm thinking to push your
>> patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3.
>> Comments?
>
> One concern is that users don't have any simple way to know how
> many bg-workers a server runs in their current configuration.

Users need to read the document of the extensions they want to load,
to see the number of background worker processes which will be running.

> The formula donsn't make sense without it.

IMO, documenting "incorrect" formula can cause more troubles.

Regards,

-- 
Fujii Masao


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Incorrect formula for SysV IPC parameters

[Kyotaro HORIGUCHI]

Thanks for looking at this.

At Fri, 12 Feb 2016 23:19:45 +0900, Fujii Masao  wrote 
in 
> >> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS
> >> under the Table 17-1.
> >
> > Oops! Thank you for pointing it out.
> >
> > The original description doesn't mention the magic-number ('5' in
> > the above formulas, which previously was '4') so I haven't added
> > anything about it.
> >
> > Process of which the number is limited by max_worker_processes is
> > called 'background process' (not 'backend worker') in the
> > documentation so I used the name to mention it in the additional
> > description.
> >
> > The difference in the body text for 9.2, 9.3 is only a literal
> > '4' to '5' in the formula.
> 
> Thanks for updating the patches!
> 
> They look good to me except that the formulas don't include the number of
> background processes requesting shared memory access, i.e.,
> GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following
> formula in 9.3?
> 
>   ceil((max_connections + autovacuum_max_workers + number of
> background proceses + 5) / 16)
> 
> Attached patch uses the above formula for 9.3. I'm thinking to push your
> patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3.
> Comments?

One concern is that users don't have any simple way to know how
many bg-workers a server runs in their current configuration.
The formula donsn't make sense without it.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Incorrect formula for SysV IPC parameters

[Fujii Masao]

On Fri, Feb 5, 2016 at 2:17 PM, Kyotaro HORIGUCHI
 wrote:
> At Thu, 4 Feb 2016 21:43:04 +0900, Fujii Masao  wrote 
> in 
>> On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI
>>  wrote:
>> > Hello, I found that the formulas to calculate SEMMNI and SEMMNS
>> > are incorrect in 9.2 and later.
>> >
>> > http://www.postgresql.org/docs/9.5/static/kernel-resources.html
>> >
>> > But actually the number of semaphores PostgreSQL needs is
>> > calculated as following in 9.4 and later.
> ...
>> > So, the formula for SEMMNI should be
>> >
>> > ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) 
>> > / 16)
>> >
>> > and SEMMNS should have the same fix.
>> >
>> >
>> > In 9.3 and 9.2, the documentation says the same thing but
> ...
>> > ceil((max_connections + autovacuum_max_workers + 5) / 16)
>> >
>> > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct.
>>
>> Good catch!
>
> Thanks.
>
>> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS
>> under the Table 17-1.
>
> Oops! Thank you for pointing it out.
>
> The original description doesn't mention the magic-number ('5' in
> the above formulas, which previously was '4') so I haven't added
> anything about it.
>
> Process of which the number is limited by max_worker_processes is
> called 'background process' (not 'backend worker') in the
> documentation so I used the name to mention it in the additional
> description.
>
> The difference in the body text for 9.2, 9.3 is only a literal
> '4' to '5' in the formula.

Thanks for updating the patches!

They look good to me except that the formulas don't include the number of
background processes requesting shared memory access, i.e.,
GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following
formula in 9.3?

  ceil((max_connections + autovacuum_max_workers + number of
background proceses + 5) / 16)

Attached patch uses the above formula for 9.3. I'm thinking to push your
patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3.
Comments?

Regards,

-- 
Fujii Masao


fix_doc_sysvipc_93.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Incorrect formula for SysV IPC parameters

[Fujii Masao]

On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI
 wrote:
> Hello, I found that the formulas to calculate SEMMNI and SEMMNS
> are incorrect in 9.2 and later.
>
> http://www.postgresql.org/docs/9.5/static/kernel-resources.html
>
> All of them say that the same thing as following,
>
> | SEMMNI  Maximum number of semaphore identifiers (i.e., sets)
> |
> |   at least ceil((max_connections + autovacuum_max_workers + 4) / 16)
> |
> | SEMMNSMaximum number of semaphores system-wide
> |
> |  ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17
> |plus room for other applications
>
> But actually the number of semaphores PostgreSQL needs is
> calculated as following in 9.4 and later.
>
>   numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4)
>   MaxConnections = max_connections + autovacuum_max_workers + 1 +
>max_worker_processes
>
> So, the formula for SEMMNI should be
>
> ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 
> 16)
>
> and SEMMNS should have the same fix.
>
>
> In 9.3 and 9.2, the documentation says the same thing but
> actually it is calculated as following,
>
>   numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4)
>   MaxConnections = max_connections + autovacuum_max_workers + 1 +
>GetNumShmemAttachedBgworkers()
>
> Omitting GetNumShmemAttachedBgworkers, the actual formula is
>
> ceil((max_connections + autovacuum_max_workers + 5) / 16)
>
>
> In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct.
>
>
> I attached two patches for 9.2-9.3 and 9.4-9.6dev
> respectively. patch command complains a bit on applying it on
> 9.2.

Good catch!

ISTM that you also need to change the descriptions about SEMMNI and SEMMNS
under the Table 17-1.

Regards,

-- 
Fujii Masao


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Incorrect formula for SysV IPC parameters

[Kyotaro HORIGUCHI]

At Thu, 4 Feb 2016 21:43:04 +0900, Fujii Masao  wrote in 

> On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI
>  wrote:
> > Hello, I found that the formulas to calculate SEMMNI and SEMMNS
> > are incorrect in 9.2 and later.
> >
> > http://www.postgresql.org/docs/9.5/static/kernel-resources.html
> >
> > But actually the number of semaphores PostgreSQL needs is
> > calculated as following in 9.4 and later.
...
> > So, the formula for SEMMNI should be
> >
> > ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) 
> > / 16)
> >
> > and SEMMNS should have the same fix.
> >
> >
> > In 9.3 and 9.2, the documentation says the same thing but
...
> > ceil((max_connections + autovacuum_max_workers + 5) / 16)
> >
> > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct.
> 
> Good catch!

Thanks.

> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS
> under the Table 17-1.

Oops! Thank you for pointing it out.

The original description doesn't mention the magic-number ('5' in
the above formulas, which previously was '4') so I haven't added
anything about it.

Process of which the number is limited by max_worker_processes is
called 'background process' (not 'backend worker') in the
documentation so I used the name to mention it in the additional
description.

The difference in the body text for 9.2, 9.3 is only a literal
'4' to '5' in the formula.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center
>From 6ed2f296cc5899f75a2e817f470e5da07bcb0d2c Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi 
Date: Wed, 3 Feb 2016 11:35:43 +0900
Subject: [PATCH] Fix the formula to calculate SEMMNI and SEMMNS in
 documentation

---
 doc/src/sgml/runtime.sgml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml
index 0db3807..2bc0a91 100644
--- a/doc/src/sgml/runtime.sgml
+++ b/doc/src/sgml/runtime.sgml
@@ -645,13 +645,13 @@ psql: could not connect to server: No such file or directory

 SEMMNI
 Maximum number of semaphore identifiers (i.e., sets)
-at least ceil((max_connections + autovacuum_max_workers + 4) / 16)
+at least ceil((max_connections + autovacuum_max_workers + 5) / 16)

 

 SEMMNS
 Maximum number of semaphores system-wide
-ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 plus room for other applications
+ceil((max_connections + autovacuum_max_workers + 5) / 16) * 17 plus room for other applications

 

@@ -712,7 +712,7 @@ psql: could not connect to server: No such file or directory
 linkend="sysvipc-parameters">).  The parameter SEMMNI
 determines the limit on the number of semaphore sets that can
 exist on the system at one time.  Hence this parameter must be at
-least ceil((max_connections + autovacuum_max_workers + 4) / 16).
+least ceil((max_connections + autovacuum_max_workers + 5) / 16).
 Lowering the number
 of allowed connections is a temporary workaround for failures,
 which are usually confusingly worded No space
-- 
1.8.3.1

>From c196c82665e4081a6f90938ac86bb63e4e60e45c Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi 
Date: Wed, 3 Feb 2016 11:07:12 +0900
Subject: [PATCH] Fix the formula to calculate SEMMNI and SEMMNS in
 documentation

---
 doc/src/sgml/runtime.sgml | 14 --
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml
index cda05f5..dd42dd0 100644
--- a/doc/src/sgml/runtime.sgml
+++ b/doc/src/sgml/runtime.sgml
@@ -645,13 +645,13 @@ psql: could not connect to server: No such file or directory

 SEMMNI
 Maximum number of semaphore identifiers (i.e., sets)
-at least ceil((max_connections + autovacuum_max_workers + 4) / 16)
+at least ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16)

 

 SEMMNS
 Maximum number of semaphores system-wide
-ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 plus room for other applications
+ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) * 17 plus room for other applications

 

@@ -699,20 +699,22 @@ psql: could not connect to server: No such file or directory
 

 PostgreSQL uses one semaphore per allowed connection
-() and allowed autovacuum worker
-process (), in sets of 16.
+(), allowed autovacuum worker process
+() and allowed background
+process (), in sets of 16.
 Each such set will
 also contain a 17th semaphore which contains a magic
 number, to detect collision with semaphore sets used by
 other 

[HACKERS] Incorrect formula for SysV IPC parameters

[Kyotaro HORIGUCHI]

Hello, I found that the formulas to calculate SEMMNI and SEMMNS
are incorrect in 9.2 and later.

http://www.postgresql.org/docs/9.5/static/kernel-resources.html

All of them say that the same thing as following,

| SEMMNI  Maximum number of semaphore identifiers (i.e., sets)
| 
|   at least ceil((max_connections + autovacuum_max_workers + 4) / 16)
| 
| SEMMNSMaximum number of semaphores system-wide
| 
|  ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17
|plus room for other applications

But actually the number of semaphores PostgreSQL needs is
calculated as following in 9.4 and later.

  numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4)
  MaxConnections = max_connections + autovacuum_max_workers + 1 +
   max_worker_processes

So, the formula for SEMMNI should be

ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16)

and SEMMNS should have the same fix.


In 9.3 and 9.2, the documentation says the same thing but
actually it is calculated as following,

  numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4)
  MaxConnections = max_connections + autovacuum_max_workers + 1 +
   GetNumShmemAttachedBgworkers()

Omitting GetNumShmemAttachedBgworkers, the actual formula is

ceil((max_connections + autovacuum_max_workers + 5) / 16)


In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct.


I attached two patches for 9.2-9.3 and 9.4-9.6dev
respectively. patch command complains a bit on applying it on
9.2.

On the platforforms that doesn't have tas operation needs
additional 1024 + 64 semaphores but I understand it as out of
scope of the documentation.

One concern is that 'at least' and 'plus room for other
applications' are mixed in the table 17-1, especially for SEMMNI
and SEMMNS. It seems to me that they should be in the same
wording, but it is not an actual problem.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

>From 6da5ad413dff4724fee75f1ba09013b6033f76ca Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi 
Date: Wed, 3 Feb 2016 11:35:43 +0900
Subject: [PATCH] Fix the formula to calculate SEMMNI and SEMMNS in
 documentation

---
 doc/src/sgml/runtime.sgml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml
index 0db3807..45579da 100644
--- a/doc/src/sgml/runtime.sgml
+++ b/doc/src/sgml/runtime.sgml
@@ -645,13 +645,13 @@ psql: could not connect to server: No such file or directory

 SEMMNI
 Maximum number of semaphore identifiers (i.e., sets)
-at least ceil((max_connections + autovacuum_max_workers + 4) / 16)
+at least ceil((max_connections + autovacuum_max_workers + 5) / 16)

 

 SEMMNS
 Maximum number of semaphores system-wide
-ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 plus room for other applications
+ceil((max_connections + autovacuum_max_workers + 5) / 16) * 17 plus room for other applications

 

-- 
1.8.3.1

>From aaf51a50e942edfa35af20a8a1d8b8719155664b Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi 
Date: Wed, 3 Feb 2016 11:19:58 +0900
Subject: [PATCH] Fix the formula to calculate SEMMNI and SEMMNS in
 documentation

---
 doc/src/sgml/runtime.sgml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml
index 650d455..4b2e403 100644
--- a/doc/src/sgml/runtime.sgml
+++ b/doc/src/sgml/runtime.sgml
@@ -645,13 +645,13 @@ psql: could not connect to server: No such file or directory

 SEMMNI
 Maximum number of semaphore identifiers (i.e., sets)
-at least ceil((max_connections + autovacuum_max_workers + 4) / 16)
+at least ceil((max_connections + autovacuum_max_workers max_worker_processes + 5) / 16)

 

 SEMMNS
 Maximum number of semaphores system-wide
-ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 plus room for other applications
+ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) * 17 plus room for other applications

 

-- 
1.8.3.1


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers