Re: [HACKERS] Logical replication ApplyContext bloat

2017-05-09 Thread Stas Kelvich
> On 9 May 2017, at 21:53, Peter Eisentraut > wrote: > > On 5/8/17 11:26, Peter Eisentraut wrote: >> I think it would make more sense to reset ApplyMessageContext in >> apply_dispatch(), so that it is done consistently after every message >> (as the name

Re: [HACKERS] Logical replication ApplyContext bloat

2017-05-09 Thread Peter Eisentraut
On 5/8/17 11:26, Peter Eisentraut wrote: > I think it would make more sense to reset ApplyMessageContext in > apply_dispatch(), so that it is done consistently after every message > (as the name implies), not only some messages. Committed that. > Also, perhaps ApplyMessageContext should be a

Re: [HACKERS] Logical replication ApplyContext bloat

2017-05-08 Thread Peter Eisentraut
On 5/5/17 03:09, Noah Misch wrote: > [Action required within three days. This is a generic notification.] > > The above-described topic is currently a PostgreSQL 10 open item. Peter, > since you committed the patch believed to have created it, you own this open > item. If some other commit is

Re: [HACKERS] Logical replication ApplyContext bloat

2017-05-08 Thread Peter Eisentraut
I'm not sure about some of the details. I think it would make more sense to reset ApplyMessageContext in apply_dispatch(), so that it is done consistently after every message (as the name implies), not only some messages. Also, perhaps ApplyMessageContext should be a child of

Re: [HACKERS] Logical replication ApplyContext bloat

2017-05-05 Thread Noah Misch
On Wed, May 03, 2017 at 12:02:41PM +0300, Stas Kelvich wrote: > > On 20 Apr 2017, at 17:01, Dilip Kumar wrote: > > On Thu, Apr 20, 2017 at 7:04 PM, Stas Kelvich > > wrote: > >> Thanks for noting. > >> > >> Added short description of ApplyContext

Re: [HACKERS] Logical replication ApplyContext bloat

2017-05-03 Thread Stas Kelvich
> On 20 Apr 2017, at 17:01, Dilip Kumar wrote: > > On Thu, Apr 20, 2017 at 7:04 PM, Stas Kelvich > wrote: >> Thanks for noting. >> >> Added short description of ApplyContext and ApplyMessageContext to README. > > Typo > > /analysys/analysis

Re: [HACKERS] Logical replication ApplyContext bloat

2017-04-20 Thread Dilip Kumar
On Thu, Apr 20, 2017 at 7:04 PM, Stas Kelvich wrote: > Thanks for noting. > > Added short description of ApplyContext and ApplyMessageContext to README. +ApplyContext --- permanent during whole lifetime of apply worker. It is +possible to use TopMemoryContext here as

Re: [HACKERS] Logical replication ApplyContext bloat

2017-04-20 Thread Stas Kelvich
> On 19 Apr 2017, at 16:07, Alvaro Herrera wrote: > > Stas Kelvich wrote: > >> With patch MemoryContextStats() shows following hierarchy during slot >> operations in >> apply worker: >> >> TopMemoryContext: 83824 total in 5 blocks; 9224 free (8 chunks); 74600 used

Re: [HACKERS] Logical replication ApplyContext bloat

2017-04-19 Thread Alvaro Herrera
Stas Kelvich wrote: > With patch MemoryContextStats() shows following hierarchy during slot > operations in > apply worker: > > TopMemoryContext: 83824 total in 5 blocks; 9224 free (8 chunks); 74600 used > ApplyContext: 8192 total in 1 blocks; 6520 free (4 chunks); 1672 used >

Re: [HACKERS] Logical replication ApplyContext bloat

2017-04-19 Thread Stas Kelvich
> On 19 Apr 2017, at 14:30, Petr Jelinek wrote: > > On 19/04/17 12:46, Stas Kelvich wrote: >> >> Right now ApplyContext cleaned after each transaction and by this patch I >> basically >> suggest to clean it after each command counter increment. >> >> So in

Re: [HACKERS] Logical replication ApplyContext bloat

2017-04-19 Thread Petr Jelinek
On 19/04/17 12:46, Stas Kelvich wrote: > >> On 19 Apr 2017, at 13:34, Simon Riggs wrote: >> >> On 19 April 2017 at 11:24, Petr Jelinek wrote: >> >>> I'd still like you to add comment to the ApplyContext saying that it's >>> cleaned after

Re: [HACKERS] Logical replication ApplyContext bloat

2017-04-19 Thread Stas Kelvich
> On 19 Apr 2017, at 13:34, Simon Riggs wrote: > > On 19 April 2017 at 11:24, Petr Jelinek wrote: > >> I'd still like you to add comment to the ApplyContext saying that it's >> cleaned after handling each message so nothing that needs to

Re: [HACKERS] Logical replication ApplyContext bloat

2017-04-19 Thread Simon Riggs
On 19 April 2017 at 11:24, Petr Jelinek wrote: > I'd still like you to add comment to the ApplyContext saying that it's > cleaned after handling each message so nothing that needs to survive for > example whole transaction can be allocated in it as that's not very

Re: [HACKERS] Logical replication ApplyContext bloat

2017-04-19 Thread Petr Jelinek
On 19/04/17 11:55, Stas Kelvich wrote: > >> On 19 Apr 2017, at 12:37, Petr Jelinek wrote: >> >> On 18/04/17 13:45, Stas Kelvich wrote: >>> Hi, >>> >>> currently logical replication worker uses ApplyContext to decode received >>> data >>> and that context is never

Re: [HACKERS] Logical replication ApplyContext bloat

2017-04-19 Thread Stas Kelvich
> On 19 Apr 2017, at 12:37, Petr Jelinek wrote: > > On 18/04/17 13:45, Stas Kelvich wrote: >> Hi, >> >> currently logical replication worker uses ApplyContext to decode received >> data >> and that context is never freed during transaction processing. Hence if

Re: [HACKERS] Logical replication ApplyContext bloat

2017-04-19 Thread Petr Jelinek
On 18/04/17 13:45, Stas Kelvich wrote: > Hi, > > currently logical replication worker uses ApplyContext to decode received data > and that context is never freed during transaction processing. Hence if > publication > side is performing something like 10M row inserts in single transaction, then

[HACKERS] Logical replication ApplyContext bloat

2017-04-18 Thread Stas Kelvich
Hi, currently logical replication worker uses ApplyContext to decode received data and that context is never freed during transaction processing. Hence if publication side is performing something like 10M row inserts in single transaction, then subscription worker will occupy more than 10G of