Re: [HACKERS] Problems with huge_pages and IBM Power8

[Andres Freund]

On 2016-04-14 12:05:39 +0200, Bernd Helmle wrote:
> 
> 
> --On 12. April 2016 13:20:10 -0700 Andres Freund  wrote:
> 
> >> When working with huge_pages, we initially got this error.
> >> 
> >> munmap(0x3efbe400) failed: Invalid argument
> > 
> > *munmap*, not mmap failed? that's odd; because there the hugepagesize
> > shouldn't have much of an influence. If something fails it should be the
> > initial mmap.  Could you show a strace of a failed start with an
> > unmodified postgres?
> 
> Well, the manpage on the the buildfarm member chub says this:
> 
> 
> The address addr must be a multiple of the page size.  All pages containing
> a part of the indicated range are unmapped,  and
>subsequent  references to these pages will generate SIGSEGV.  It is
> not an error if the indicated range does not contain any
>mapped pages.
> 

Uh, but that's something different, no? The address we pass to munmap()
is what mmap() returned. The page size bit is relevant if you want to
unmap part of a mapped region.


> Though, we haven't encountered any problems on the production machines so
> far.

That's after applying some patch to pad to a larger size?

Greetings,

Andres Freund


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Problems with huge_pages and IBM Power8

[Bert]

hmpf; are you sure?
I just checked on our own rhel7 system, and RemoveIPC is set to 'no' by
default..

On Tue, Apr 12, 2016 at 10:26 PM, Tom Lane  wrote:

> Andres Freund  writes:
> > On 2016-04-12 21:58:14 +0200, reiner peterke wrote:
> >> Looking for some insight into this issue.  the error from the postgres
> >> log on ubuntu is below.  It apperas to be related to semephores.
>
> > I've a bit of a hard time believing that this is related to huge pages.
>
> I'm betting that's this:
>
>
> http://www.postgresql.org/message-id/cak7teys9-o4bterbs3xuk2bffnnd55u2sm9j5r2fi7v6bhj...@mail.gmail.com
>
> regards, tom lane
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>



-- 
Bert Desmet
0477/305361

Re: [HACKERS] Problems with huge_pages and IBM Power8

[Bernd Helmle]

--On 12. April 2016 13:20:10 -0700 Andres Freund  wrote:

>> When working with huge_pages, we initially got this error.
>> 
>> munmap(0x3efbe400) failed: Invalid argument
> 
> *munmap*, not mmap failed? that's odd; because there the hugepagesize
> shouldn't have much of an influence. If something fails it should be the
> initial mmap.  Could you show a strace of a failed start with an
> unmodified postgres?

Well, the manpage on the the buildfarm member chub says this:


The address addr must be a multiple of the page size.  All pages containing
a part of the indicated range are unmapped,  and
   subsequent  references to these pages will generate SIGSEGV.  It is
not an error if the indicated range does not contain any
   mapped pages.


There are other production machines running on this POWER8 machine, all
have huge pages activated and i've seen this message several times, but
afair only during initdb and when stopping the server. No crashes. 

Indeed, I had the exactly same patch than Reiner, when investigating this,
but i couldn't find an easy way to make it somehow automatic for other page
sizes (e.g. on large Intel machines you can have even 1G Pages).

Though, we haven't encountered any problems on the production machines so
far.

-- 
Thanks

Bernd


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Problems with huge_pages and IBM Power8

[reiner peterke]

> On Apr 12, 2016, at 10:26 PM, Tom Lane  wrote:
> 
> Andres Freund  writes:
>> On 2016-04-12 21:58:14 +0200, reiner peterke wrote:
>>> Looking for some insight into this issue.  the error from the postgres
>>> log on ubuntu is below.  It apperas to be related to semephores.
> 
>> I've a bit of a hard time believing that this is related to huge pages.
> 
> I'm betting that's this:
> 
> http://www.postgresql.org/message-id/cak7teys9-o4bterbs3xuk2bffnnd55u2sm9j5r2fi7v6bhj...@mail.gmail.com
> 
>   regards, tom lane

Hi Tom,

You appear to have been correct.  :-)
The being led to believe it was connected to huge_pages turns out to have been 
a coincidence since we had been working with huge_pages.

Postgres happened to crash close enough to trying the huge_pages and did not 
appear to crash when they were disabled led us to make that conclusion.

After a bit more careful testing we found out that postgres did indeed crash 
even without huge_pages.  The setting in the link appears to have resolved this 
issue.

Thanks.

reiner



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Problems with huge_pages and IBM Power8

[Tom Lane]

reiner peterke  writes:
>> On Apr 12, 2016, at 10:20 PM, Andres Freund  wrote:
>> I've a bit of a hard time believing that this is related to huge pages.

> Well all i have at the moment is that when we disabled huge pages on the 
> kernel level and then restarted postgres there were no additional crashes.

That log excerpt shows pretty conclusively that both your semaphores and
your shared memory were yanked out from under you.  That's not a Postgres
bug, and it seems unlikely to be related to huge pages either.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Problems with huge_pages and IBM Power8

[reiner peterke]

> On Apr 12, 2016, at 10:20 PM, Andres Freund  wrote:
> 
> On 2016-04-12 21:58:14 +0200, reiner peterke wrote:
>> Hi
>> 
>> We have been doing some testing with Postgres (9.5.2) compiled on a Power8 
>> running Centos 7
>> 
>> When working with huge_pages, we initially got this error.
>> 
>> munmap(0x3efbe400) failed: Invalid argument
> 
> *munmap*, not mmap failed? that's odd; because there the hugepagesize
> shouldn't have much of an influence. If something fails it should be the
> initial mmap.  
I’ll double check in the morning, but i did copy it from the log.

> Could you show a strace of a failed start with an
> unmodified postgres?

we didn’t have the error when not using huge_pages.

>> after a bit of investigation we noticed that hugepagesize is har coded
>> to 2MB
> 
> Note it's not actually hardcoded to some size. It's just about rounding
> the size to a multiple of 2MB due to an older kernel bug:
>   /*
>* Round up the request size to a suitable large value.
>*
>* Some Linux kernel versions are known to have a bug, which 
> causes
>* mmap() with MAP_HUGETLB to fail if the request size is not a
>* multiple of any supported huge page size. To work around 
> that, we
>* round up the request size to nearest 2MB. 2MB is the most 
> common
>* huge page page size on affected systems.
> 
> 
>> Going further, we tried testing hugepages also on Ubuntu 16.04, also on the 
>> power8.  On Ubuntu Postgres did not like the hugepages at all (set also to 
>> 16MB)  and consistently crashed.
> 
>> Looking for some insight into this issue.  the error from the postgres
>> log on ubuntu is below.  It apperas to be related to semephores.
> 
> I've a bit of a hard time believing that this is related to huge pages.

Well all i have at the moment is that when we disabled huge pages on the kernel 
level and then restarted postgres there were no additional crashes.
Unfortunately I cannot access the server now.  I will look further tomorrow.


> 
> 
> Greetings,
> 
> Andres Freund

Sincerely,

Reiner Peterke



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Problems with huge_pages and IBM Power8

[Tom Lane]

Andres Freund  writes:
> On 2016-04-12 21:58:14 +0200, reiner peterke wrote:
>> Looking for some insight into this issue.  the error from the postgres
>> log on ubuntu is below.  It apperas to be related to semephores.

> I've a bit of a hard time believing that this is related to huge pages.

I'm betting that's this:

http://www.postgresql.org/message-id/cak7teys9-o4bterbs3xuk2bffnnd55u2sm9j5r2fi7v6bhj...@mail.gmail.com

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Problems with huge_pages and IBM Power8

[Andres Freund]

On 2016-04-12 21:58:14 +0200, reiner peterke wrote:
> Hi
> 
> We have been doing some testing with Postgres (9.5.2) compiled on a Power8 
> running Centos 7
> 
> When working with huge_pages, we initially got this error.
> 
> munmap(0x3efbe400) failed: Invalid argument

*munmap*, not mmap failed? that's odd; because there the hugepagesize
shouldn't have much of an influence. If something fails it should be the
initial mmap.  Could you show a strace of a failed start with an
unmodified postgres?

> after a bit of investigation we noticed that hugepagesize is har coded
> to 2MB

Note it's not actually hardcoded to some size. It's just about rounding
the size to a multiple of 2MB due to an older kernel bug:
/*
 * Round up the request size to a suitable large value.
 *
 * Some Linux kernel versions are known to have a bug, which 
causes
 * mmap() with MAP_HUGETLB to fail if the request size is not a
 * multiple of any supported huge page size. To work around 
that, we
 * round up the request size to nearest 2MB. 2MB is the most 
common
 * huge page page size on affected systems.


> Going further, we tried testing hugepages also on Ubuntu 16.04, also on the 
> power8.  On Ubuntu Postgres did not like the hugepages at all (set also to 
> 16MB)  and consistently crashed.

> Looking for some insight into this issue.  the error from the postgres
> log on ubuntu is below.  It apperas to be related to semephores.

I've a bit of a hard time believing that this is related to huge pages.


Greetings,

Andres Freund


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Problems with huge_pages and IBM Power8

[reiner peterke]

Hi

We have been doing some testing with Postgres (9.5.2) compiled on a Power8 
running Centos 7

When working with huge_pages, we initially got this error.

munmap(0x3efbe400) failed: Invalid argument

after a bit of investigation we noticed that hugepagesize is har coded to 2MB

src/backend/port/sysv_shmem.c (ligne 360)
 
...
int hugepagesize = 2 * 1024 * 1024;

But on the power they were configured to 16MB.  Recompiling to 16MB (8 * 1024 * 
1024) and we had no problems with the tests.

My initial questions are.

1 what is the hugepagesize hard coded to 2MB?
2 are there any side effect in setting it to 16MB?
3 since on the poer hugepages can have different values, would it be possible 
to have this value configurable?

Going further, we tried testing hugepages also on Ubuntu 16.04, also on the 
power8.  On Ubuntu Postgres did not like the hugepages at all (set also to 
16MB)  and consistently crashed.

Looking for some insight into this issue.  the error from the postgres log on 
ubuntu is below.
It apperas to be related to semephores. 

I don't have the compile optiona at the moment, I can provide those are other 
detais as needed.

Reiner

2016-04-12 12:26:42 CEST : 0 FATAL:  semctl(7864340, 14, SETVAL, 0) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  server process (PID 13352) exited with exit 
code 1
2016-04-12 12:26:42 CEST : 0 LOG:  terminating any other active server processes
2016-04-12 12:26:42 CEST facturation:system_dba 0 10.32.32.200WARNING:  
terminating connection because of crash of another server process
2016-04-12 12:26:42 CEST facturation:system_dba 0 10.32.32.200DETAIL:  The 
postmaster has commanded this server process to roll back the current 
transaction and exit, because another server process exited abnormally and 
possibly corrupted shared memory.
2016-04-12 12:26:42 CEST facturation:system_dba 0 10.32.32.200HINT:  In a 
moment you should be able to reconnect to the database and repeat your command.
2016-04-12 12:26:42 CEST postgres:admin 0 10.32.16.3WARNING:  terminating 
connection because of crash of another server process
2016-04-12 12:26:42 CEST postgres:admin 0 10.32.16.3DETAIL:  The postmaster has 
commanded this server process to roll back the current transaction and exit, 
because another server process exited abnormally and possibly corrupted shared 
memory.
2016-04-12 12:26:42 CEST postgres:admin 0 10.32.16.3HINT:  In a moment you 
should be able to reconnect to the database and repeat your command.
2016-04-12 12:26:42 CEST postgres:perf_user 0 ::1WARNING:  terminating 
connection because of crash of another server process
2016-04-12 12:26:42 CEST postgres:perf_user 0 ::1DETAIL:  The postmaster has 
commanded this server process to roll back the current transaction and exit, 
because another server process exited abnormally and possibly corrupted shared 
memory.
2016-04-12 12:26:42 CEST postgres:perf_user 0 ::1HINT:  In a moment you should 
be able to reconnect to the database and repeat your command.
2016-04-12 12:26:42 CEST : 0 WARNING:  terminating connection because of crash 
of another server process
2016-04-12 12:26:42 CEST : 0 DETAIL:  The postmaster has commanded this server 
process to roll back the current transaction and exit, because another server 
process exited abnormally and possibly corrupted shared memory.
2016-04-12 12:26:42 CEST : 0 HINT:  In a moment you should be able to reconnect 
to the database and repeat your command.
2016-04-12 12:26:42 CEST : 0 LOG:  all server processes terminated; 
reinitializing
2016-04-12 12:26:42 CEST : 0 LOG:  could not remove shared memory segment 
"/PostgreSQL.1612071802": No such file or directory
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7274497, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7307267, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7340036, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7372805, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7405574, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7438343, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7471112, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7503881, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7536650, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7569419, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7602188, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7634957, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7667726, 0, IPC_RMID, ...) failed: 
Invalid argument
2016-04-12 12:26:42 CEST : 0 LOG:  semctl(7700495, 0, IPC_RMID, ...)