Hi Tim, On Tue, Jan 15, 2019 at 09:32:42PM +0100, Tim Düsterhus wrote: > Willy, > > Am 15.01.19 um 15:32 schrieb Willy Tarreau: > > Got it! I thought the problem was local to a process and that we > > replicated bad data, but in fact not, it's a distributed race. In > > this case there is no other short-term solution, and the drift has > > no reason to significantly accumulate over time. The only long-term > > solution I'd be seeing to work around this specific pattern would be > > to keep such values as differential pairs : > > - count and synchronize the number of ++ > > - count and synchronize the number of -- > > In this case the real value is the difference between the two. But > > it's a bit overkill and is still prone to other races when connections > > appear in parallel on the two peers. Then at this point better use an > > external aggregator. > > Ideally the peers would exchange their local values, only.
Yes and that's how this currently works. > These are > obviously correct (otherwise it would be broken even without peers). > Then the instances aggregate the values (e.g. sum up all the values for > the number of connections) themselves to get the correct value. That's exactly how HapTech's stick-table aggregator works :-) > It's a good example for a "long term" issue for the soon-to-come(TM) issue > tracker ;) > > > > > OK I'm merging Tim's patch now. > > > > It was not meant for actually applying (that's why it was missing out on > backport information as well), Bah, too late, misunderstood, sorry :-) > because fixing just one counter is fixing > the symptoms. I suspect the other stick table values are affected from a > possible underflow as well. I've checked the rest of the related code for variations and only this key works like a gauge, up and down. The other ones in the worst case will simply lose some counts but will not face such a problem. > But if you are fine with the patch it's fine, I guess. A real solution > definitely requires breaking compatibility with the current peer protocol. Yes and not only the protocol but the implementations as well. While I wanted to have multi-peers sync a long time ago, after starting to think about the algo I figured this would consume a huge amount of memory (N*peers per node) so I don't want to see that into haproxy anymore! This amount of memory is better used for network buffers. Also doing the aggregation on the nodes themselves cause an extra SPOF difficulty when one node goes down. Cheers, Willy
