Re: Lots of memory allocated when reassigning Large Objects

2021-12-01 Thread Guillaume Lelarge
Le mer. 1 déc. 2021 à 19:48, Tom Lane a écrit : > Justin Pryzby writes: > > @Guillaume: Even if memory use with the patch isn't constant, I imagine > it's > > enough to have avoided OOM. > > I think it's good enough in HEAD. In the back branches, the sinval > queue growth is bad enough that the

Re: Lots of memory allocated when reassigning Large Objects

2021-12-01 Thread Tom Lane
Justin Pryzby writes: > @Guillaume: Even if memory use with the patch isn't constant, I imagine it's > enough to have avoided OOM. I think it's good enough in HEAD. In the back branches, the sinval queue growth is bad enough that there's still an issue. Still, this is a useful improvement, so I

Re: Lots of memory allocated when reassigning Large Objects

2021-11-30 Thread Guillaume Lelarge
Le mar. 30 nov. 2021 à 00:25, Tom Lane a écrit : > Guillaume Lelarge writes: > > Le lun. 29 nov. 2021 à 22:27, Tom Lane a écrit : > >> I'm checking it in HEAD though; perhaps there's something else wrong > >> in the back branches? > > > That's also what I was thinking. I was only trying with v1

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Justin Pryzby
On Mon, Nov 29, 2021 at 01:40:31PM -0500, Tom Lane wrote: > DROP OWNED BY likely has similar issues. I tried a few more commands but found no significant issue. IMO if you have 100k tables, then you can afford 1GB RAM. SELECT format('CREATE TABLE t%s()', a) FROM generate_series(1,)a\gexec SET

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Tom Lane
Guillaume Lelarge writes: > Le lun. 29 nov. 2021 à 22:27, Tom Lane a écrit : >> I'm checking it in HEAD though; perhaps there's something else wrong >> in the back branches? > That's also what I was thinking. I was only trying with v14. I just checked > with v15devel, and your patch works alrigh

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Guillaume Lelarge
Le lun. 29 nov. 2021 à 22:47, Tom Lane a écrit : > Guillaume Lelarge writes: > > Le lun. 29 nov. 2021 à 22:27, Tom Lane a écrit : > >> I'm checking it in HEAD though; perhaps there's something else wrong > >> in the back branches? > > > That's also what I was thinking. I was only trying with v1

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Tom Lane
Guillaume Lelarge writes: > Le lun. 29 nov. 2021 à 22:27, Tom Lane a écrit : >> I'm checking it in HEAD though; perhaps there's something else wrong >> in the back branches? > That's also what I was thinking. I was only trying with v14. I just checked > with v15devel, and your patch works alrigh

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Guillaume Lelarge
Le lun. 29 nov. 2021 à 22:27, Tom Lane a écrit : > Guillaume Lelarge writes: > > Le lun. 29 nov. 2021 à 20:39, Tom Lane a écrit : > >> [ looks closer ... ] Ah, that patch is a bit buggy: it fails to do the > >> right thing in the cases where the loop does a "continue". The attached > >> revis

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Tom Lane
Guillaume Lelarge writes: > Le lun. 29 nov. 2021 à 20:39, Tom Lane a écrit : >> [ looks closer ... ] Ah, that patch is a bit buggy: it fails to do the >> right thing in the cases where the loop does a "continue". The attached >> revision seems to behave properly. > I've tried your patch with m

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Guillaume Lelarge
Le lun. 29 nov. 2021 à 20:39, Tom Lane a écrit : > Guillaume Lelarge writes: > > I've tried Justin's patch but it didn't help with my memory allocation > > issue. FWIW, I attach the patch I used in v14. > > [ looks closer ... ] Ah, that patch is a bit buggy: it fails to do the > right thing in

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Tom Lane
Guillaume Lelarge writes: > I've tried Justin's patch but it didn't help with my memory allocation > issue. FWIW, I attach the patch I used in v14. [ looks closer ... ] Ah, that patch is a bit buggy: it fails to do the right thing in the cases where the loop does a "continue". The attached revi

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Guillaume Lelarge
Le lun. 29 nov. 2021 à 19:40, Tom Lane a écrit : > Justin Pryzby writes: > > I reproduced the issue like this. > > > psql postgres -c 'CREATE ROLE two WITH login superuser' > > psql postgres two -c "SELECT lo_import('/dev/null') FROM > generate_series(1,22111)" >/dev/null > > psql postgres -c 'S

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Tom Lane
Justin Pryzby writes: > I reproduced the issue like this. > psql postgres -c 'CREATE ROLE two WITH login superuser' > psql postgres two -c "SELECT lo_import('/dev/null') FROM > generate_series(1,22111)" >/dev/null > psql postgres -c 'SET client_min_messages=debug; SET log_statement_stats=on;' >

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Justin Pryzby
On Mon, Nov 29, 2021 at 01:49:24PM +0100, Guillaume Lelarge wrote: > One of our customers had a very bad issue while trying to reassign objects > from user A to user B. He had a lot of them, and the backend got very > hungry for memory. It finally all went down when the linux kernel decided > to ki

Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Guillaume Lelarge
Hi, One of our customers had a very bad issue while trying to reassign objects from user A to user B. He had a lot of them, and the backend got very hungry for memory. It finally all went down when the linux kernel decided to kill the backend (-9 of course). I attach three shell scripts showing t