Re: [GENERAL] incorrect checksum detected on "global/pg_filenode.map" when VACUUM FULL is executed

[Michael Paquier]

On Mon, Aug 22, 2016 at 1:31 PM, John R Pierce  wrote:
> On 8/21/2016 9:13 PM, Tatsuki Kadomoto wrote:
>>
>> Can we point out a specific bug that can lead to this?
>
>
> 9.2.6 fixed several data corruption bugs,
> https://www.postgresql.org/docs/current/static/release-9-2-6.html
>
> 9.2.9 fixed a GiST index corruption problem...
> https://www.postgresql.org/docs/current/static/release-9-2-9.html
>
> I would upgrade to 9.2.18, the latest 9.2 version,
> https://www.postgresql.org/docs/current/static/release-9-2-18.html

Yep, this is mandatory. You are taking a lot of risks here by only using 9.2.4.
-- 
Michael


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] incorrect checksum detected on "global/pg_filenode.map" when VACUUM FULL is executed

[John R Pierce]

On 8/21/2016 9:13 PM, Tatsuki Kadomoto wrote:

Can we point out a specific bug that can lead to this?


9.2.6 fixed several data corruption bugs, 
https://www.postgresql.org/docs/current/static/release-9-2-6.html


9.2.9 fixed a GiST index corruption problem... 
https://www.postgresql.org/docs/current/static/release-9-2-9.html


I would upgrade to 9.2.18, the latest 9.2 version, 
https://www.postgresql.org/docs/current/static/release-9-2-18.html



--
john r pierce, recycling bits in santa cruz



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] incorrect checksum detected on "global/pg_filenode.map" when VACUUM FULL is executed

[Tatsuki Kadomoto]

John, Michael,


Thanks. The server is Dell PowerEdge R720.


The checksum error only was reported only once so I guess it was automatically 
fixed immediately.


The log was output only one time while the box was running very long time (> 1 
year.).

I have hundreds of the machines with same configuration running very long time 
as well,

but the incident was only once.


Can we point out a specific bug that can lead to this?

Or HW issue is the most probable cause?

I don't see any HW error logs.


Regards,

Tatsuki


From: pgsql-general-ow...@postgresql.org  
on behalf of John R Pierce 
Sent: Monday, August 22, 2016 12:42:41 PM
To: pgsql-general@postgresql.org
Subject: Re: [GENERAL] incorrect checksum detected on "global/pg_filenode.map" 
when VACUUM FULL is executed

On 8/21/2016 8:37 PM, Michael Paquier wrote:

PosgreSQL version is 9.2.4.


You are missing a couple of years worth of bug fixes here..

3+ years, to be more specific, 9.2.4 was released in april 2013.

indeed, and several of the bug fixes involved data corruption.   current is 
9.2.18


--
john r pierce, recycling bits in santa cruz

Re: [GENERAL] incorrect checksum detected on "global/pg_filenode.map" when VACUUM FULL is executed

[John R Pierce]

On 8/21/2016 8:37 PM, Michael Paquier wrote:

PosgreSQL version is 9.2.4.

You are missing a couple of years worth of bug fixes here..


3+ years, to be more specific, 9.2.4 was released in april 2013.

indeed, and several of the bug fixes involved data corruption. current 
is 9.2.18



--
john r pierce, recycling bits in santa cruz

Re: [GENERAL] incorrect checksum detected on "global/pg_filenode.map" when VACUUM FULL is executed

[Michael Paquier]

On Mon, Aug 22, 2016 at 12:27 PM, Tatsuki Kadomoto
 wrote:
> I see incorrect checksum detected on "global/pg_filenode.map" when "VACUUM
> FULL" is executed.
>
> The error message didn't repeat. It showed up only once.
>
> Is this expected? Can someone give me a plausible scenario why this
> happened?
> Aug 16 20:51:19 postgres[22329]: [2-1] FATAL:  relation mapping file
> "global/pg_filenode.map" contains incorrect checksum
> It looks like it's pretty rare and I have seen it only once at 2 different
> machines.

You are facing a corruption:
https://wiki.postgresql.org/wiki/Corruption
Be sure that you have a clean backup of your database first!

> PosgreSQL version is 9.2.4.

You are missing a couple of years worth of bug fixes here..
-- 
Michael


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] incorrect checksum detected on "global/pg_filenode.map" when VACUUM FULL is executed

[John R Pierce]

On 8/21/2016 8:27 PM, Tatsuki Kadomoto wrote:



I see incorrect checksum detected on "global/pg_filenode.map" when 
"VACUUM FULL" is executed.


The error message didn't repeat. It showed up only once.

Is this expected? Can someone give me a plausible scenario why this 
happened?



Aug 16 20:51:19 postgres[22329]: [2-1] FATAL:  relation mapping file 
"global/pg_filenode.map" contains incorrect checksum



It looks like it's pretty rare and I have seen it only once at 2 
different machines.



PosgreSQL version is 9.2.4.




that suggests unreliable hardware.  does this server not have ECC memory ?


--
john r pierce, recycling bits in santa cruz

[GENERAL] incorrect checksum detected on "global/pg_filenode.map" when VACUUM FULL is executed

[Tatsuki Kadomoto]

Hello,


I see incorrect checksum detected on "global/pg_filenode.map" when "VACUUM 
FULL" is executed.

The error message didn't repeat. It showed up only once.

Is this expected? Can someone give me a plausible scenario why this happened?


Aug 16 20:51:19 postgres[22329]: [2-1] FATAL:  relation mapping file 
"global/pg_filenode.map" contains incorrect checksum


It looks like it's pretty rare and I have seen it only once at 2 different 
machines.


PosgreSQL version is 9.2.4.


Regards,

Tatsuki

Re: [GENERAL] endash not a graphic character?

[Bruno Wolff III]

On Sun, Aug 21, 2016 at 14:24:16 -0400,
 Tom Lane  wrote:


Unfortunately, these particular characters are U+2013 and U+2014 so you
lose.


Thanks for saving me some time, as it would have taken me quite a while 
to figure that out.


I'll adjust the constraint so that good strings aren't rejected. Which 
was my immediate problem. I'm not that worried about bad strings getting 
added, since the data also gets checked before trying to add it to 
the database.



Obviously there's room for improvement here, but so far nobody's been
motivated to work on it.  Last discussion about it (AFAIR) was this
thread:


One thing I would suggest is documenting this limitation under: 
https://www.postgresql.org/docs/9.6/static/functions-matching.html#FUNCTIONS-POSIX-REGEXP


I might have missed it, but I did try reading that section to see if I was 
doing something wrong before asking on the list. In particular I would 
expect this limitation to be noted under:

9.7.3.6. Limits and Compatibility


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] endash not a graphic character?

[Tom Lane]

Bruno Wolff III  writes:
> However I am wondering about my use of [[:graph:]] to match characters 
> that have glyphs. I was not expecting there to be characters that have 
> glyphs to not be in the graph class. In the short term I might want to 
> change the way I am testing that.

[ looks into code... ]  The [[:foo:]] notations only work up to Unicode
code point U+7FF at the moment, per this comment in regc_pg_locale.c:

 * Decide how many character codes we ought to look through.  For C locale
 * there's no need to go further than 127.  Otherwise, if the encoding is
 * UTF8 go up to 0x7FF, which is a pretty arbitrary cutoff but we cannot
 * extend it as far as we'd like (say, 0x, the end of the Basic
 * Multilingual Plane) without creating significant performance issues due
 * to too many characters being fed through the colormap code.  This will
 * need redesign to fix reasonably, but at least for the moment we have
 * all common European languages covered.  Otherwise (not C, not UTF8) go
 * up to 255.  These limits are interrelated with restrictions discussed
 * at the head of this file.

Unfortunately, these particular characters are U+2013 and U+2014 so you
lose.

Obviously there's room for improvement here, but so far nobody's been
motivated to work on it.  Last discussion about it (AFAIR) was this
thread:

https://www.postgresql.org/message-id/flat/24241.1329347196%40sss.pgh.pa.us

I'm not sure if any of the subsequent work on the regex engine would
make it any easier to fix than it seemed at the time.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] endash not a graphic character?

[Bruno Wolff III]

On Sun, Aug 21, 2016 at 12:30:21 -0500,
 Bruno Wolff III  wrote:


I should also try the equivalent test in perl to see if it is more 
likely tied to the unicode implementation on my system or if it 
appears to be Postgres specific.


It looks like my locale may not be being set the way I expect. I tried 
testing in perl and initially I got results consistent with Postgres, 
but when I added code to make sure perl was working in utf-8 mode I 
started getting the expected results.


I would have expected manually adding a collation to the queries would 
have worked even if the default was not what I expected. So pointers 
to what I am missing would still be appreciated.



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] endash not a graphic character?

[Bruno Wolff III]

On Sun, Aug 21, 2016 at 08:12:23 +1000,
 rob stone  wrote:


You can't use  (emdash) or  (endash)?
Or their hex equivalents. See the Unicode chart.


By the way, those aren't the correct codes. That only works if your 
code treats iso-5589-1 code points as windows 1252 code points. That 
may happen to work in many cases, but isn't a good thing to bet on.

(Single byte utf8 codes match iso-8859-1, not windows 1252.)


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] endash not a graphic character?

[Bruno Wolff III]

On Sun, Aug 21, 2016 at 08:12:23 +1000,
 rob stone  wrote:


You can't use  (emdash) or  (endash)?
Or their hex equivalents. See the Unicode chart.


I am not the source of the data, but I can special case them one way 
or the other.


However I am wondering about my use of [[:graph:]] to match characters 
that have glyphs. I was not expecting there to be characters that have 
glyphs to not be in the graph class. In the short term I might want to 
change the way I am testing that.


I should also try the equivalent test in perl to see if it is more likely 
tied to the unicode implementation on my system or if it appears to be 
Postgres specific.



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general