Re: [GENERAL] memory leaking ?

2014-05-28 Thread Tom Lane
Zhemin Zhou  writes:
> We met a problem after running the website for one week. We used a 
> function to convert and save binary files into the database (as bytea). 
> This function worked well in the old version but sometimes makes the new 
> version of postgres crash. This random crash is not file specific. 

AFAICT, it's pure luck that it didn't crash the older system too.  You're
allocating the output buffer too small, at least for cases where "size"
isn't a multiple of 3:

>  bytea *result = (bytea *) 
> palloc(VARHDRSZ+sizeof(char)*(4*(size)/3+15));

For example, if size = 2, 4*2/3 is only 2, but the loop will write 4 bytes
of data.  So the function sometimes clobbers bytes beyond what it
allocated, which unsurprisingly corrupts malloc's data structures.
You need to round up not truncate in this division.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] memory leaking ?

2014-05-28 Thread Zhemin Zhou

Hi all

We have been using postgreSQL version 8.1.11 for 6 years and have to 
move our website to another server recently. We did not manage to get 
the version 8.1.11 thus have to use version 8.4.12 in  the new server. 
This version is still quite old but we cannot make the databases running 
in any of newer versions.


We met a problem after running the website for one week. We used a 
function to convert and save binary files into the database (as bytea). 
This function worked well in the old version but sometimes makes the new 
version of postgres crash. This random crash is not file specific. 
Sometimes, the function works well in all binary files but makes crashes 
again and again at other times.


The OS system in the new server is: Ubuntu 12.04.2.

The error messages are:
2(56608) SELECT: malloc.c:2451: sYSMALLOc: Assertion `(old_top == 
(((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - 
__builtin_offsetof (struct malloc_chunk, fd && old_size == 0) || 
((unsigned long) (old_size) >= (unsigned long)(
(((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * 
(sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && 
((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' 
failed.

LOG:  server process (PID 8866) was terminated by signal 6: Aborted
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back 
the current transaction and exit, because another server process exited 
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and 
repeat your command.

WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back 
the current transaction and exit, because another server process exited 
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and 
repeat your command.

WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back 
the current transaction and exit, because another server process exited 
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and 
repeat your command.

WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back 
the current transaction and exit, because another server process exited 
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and 
repeat your command.

WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back 
the current transaction and exit, because another server process exited 
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and 
repeat your command.

WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back 
the current transaction and exit, because another server process exited 
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and 
repeat your command.

WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back 
the current transaction and exit, because another server process exited 
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and 
repeat your command.

LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at 2014-05-26 
10:53:57 BST
LOG:  database system was not properly shut down; automatic recovery in 
progress

LOG:  redo starts at 1/D2BC56A0
LOG:  record with zero length at 1/D2BCC600
LOG:  redo done at 1/D2BCC5C8
LOG:  last completed transaction was at log time 2014-05-26 
11:31:07.172215+01

LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started



And here is the C function, which was written at about 10 years ago and 
was compiled by "gcc -O2 -shared -fPIC convert_bin_to_64.c -o 
convert_bin_to_64.so".


#include "postgres.h"
#include "fmgr.h"
#include 
PG_MODULE_MAGIC;

bytea *convert_bin_to_64(bytea *t)
/* unsingned char weil Sonderzeichen auch negativ sein k?nnen*/
{
int ps,v0,v1,v2,v3,vl,ps2;
char charset[70];
int size=VARSIZE(t)-VARHDRSZ; /* VARSIZE and VARHDRSZ are 
PostgreSql Macros*/
bytea *result = (bytea *) 
palloc(VARHDRSZ+sizeof(char)*(4*(size)/3+15));


strcpy(charset,"abcdefghijklmnopqrstuvwxyzABCDEFGH