Hello Bastien,
Thank you for your feedback. I hope you don't mind I let it percolate for a
while.
> Eclair doesn't do any rate-limiting. We wanted to "feel the pain" before
> adding
> anything, and to be honest we haven't really felt it yet.
I understand the “feel the pain first” approach, but attempting set
reconciliation has forced me to confront the issue a bit early.
My thoughts on sync were that set-reconciliation would only be used once a node
had fully synced gossip through traditional means (initial_routing_sync /
gossip_queries.) There should be many levers to pull in order to help maintain
sync after this. I'm going to have to experiment with them a bit before I can
claim they are sufficient, but I'm optimistic.
> One thing that may help here from an implementation's point of view is to
> avoid
> sending a disabled channel update every time a channel goes offline. What
> eclair does to avoid spamming is to only send a disabled channel update when
> someone actually tries to use that channel. Of course, if people choose this
> offline node in their route, you don't have a choice and will need to send a
> disabled channel update, but we've observed that many channels come back
> online before we actually need to use them, so we're saving two channel
> updates
> (one to disable the channel and one to re-enable it). I think all
> implementations
> should do this. Is that the case today?
> We could go even further, and when we receive an htlc that should be relayed
> to an offline node, wait a bit to give them an opportunity to come online
> instead
> of failing the htlc and sending a disabled channel update. Eclair currently
> doesn'tdo that, but it would be very easy to add.
Core-Lightning also delays sending disabled channel updates in an effort to
minimize unnecessary gossip. I hadn’t considered an additional delay before
failing an htlc on a disabled channel. That will be interesting to explore in
the context of transient disconnects of Tor v3 nodes.
I like the idea of a block_height in the channel update tlv. That would be
sufficient to enable a simple rate-limit heuristic for this application anyway.
Allowing leeway for the chain tip is no problem. I would also expect most
implementations to hold a couple updates in reserve, defaulting to predated
updates when available. This would allow a “burst” functionality similar to the
current LND/CLN rate-limit, but the responsibility is now placed on the
originating node to provide that allowance.
Cheers,
Alex
------- Original Message -------
On Friday, April 15th, 2022 at 2:15 AM, Bastien TEINTURIER <[email protected]>
wrote:
> Good morning Alex,
>
>> I’ve been investigating set reconciliation as a means to reduce bandwidth
>
>> and redundancy of gossip message propagation.
>
> Cool project, glad to see someone working on it! The main difficulty here will
> indeed be to ensure that the number of differences between sets is bounded.
> We will need to maintain a mechanism to sync the whole graph from scratch
> for new nodes, so the minisketch diff must be efficient enough otherwise nodes
> will just fall back to a full sync way too often (which would waste a lot of
> bandwidth).
>
>> Picking several offending channel ids, and digging further, the majority of
>> these
>
>> appear to be flapping due to Tor or otherwise intermittent connections.
>
> One thing that may help here from an implementation's point of view is to
> avoid
> sending a disabled channel update every time a channel goes offline. What
> eclair does to avoid spamming is to only send a disabled channel update when
> someone actually tries to use that channel. Of course, if people choose this
> offline node in their route, you don't have a choice and will need to send a
> disabled channel update, but we've observed that many channels come back
> online before we actually need to use them, so we're saving two channel
> updates
> (one to disable the channel and one to re-enable it). I think all
> implementations
> should do this. Is that the case today?
>
> We could go even further, and when we receive an htlc that should be relayed
> to an offline node, wait a bit to give them an opportunity to come online
> instead
> of failing the htlc and sending a disabled channel update. Eclair currently
> doesn't
> do that, but it would be very easy to add.
>
>> - A common listing of current default rate limits across lightning network
>> implementations.
>
> Eclair doesn't do any rate-limiting. We wanted to "feel the pain" before
> adding
> anything, and to be honest we haven't really felt it yet.
>
>> which will use a common, simple heuristic to accept or reject a gossip
>> message.
>
>> (Think one channel update per block, or perhaps one per block_height << 5.)
>
> I think it would be easy to come to agreement between implementations and
> restrict channel updates to at most one every N blocks. We simply need to add
> the `block_height` in a tlv in `channel_update` and then we'll be able to
> actually
> rate-limit based on it. Given how much time it takes to upgrade most of the
> network, it may be a good idea to add the `block_height` tlv now in the spec,
> and act on it later? Unless your work requires bigger changes in channel
> update
> in which case it will probably be a new message.
>
> Note that it will never be completely accurate though, as different nodes can
> have different blockchain tips. My nodes may be one or two blocks late
> compared
> to the node that emits the channel update. We need to allow a bit of leeway
> there.
>
> Cheers,
> Bastien
>
> Le jeu. 14 avr. 2022 à 23:06, Alex Myers <[email protected]> a écrit :
>
>> Hello lightning developers,
>>
>> I’ve been investigating set reconciliation as a means to reduce bandwidth
>> and redundancy of gossip message propagation. This builds on some earlier
>> work from Rusty using the minisketch library [1]. The idea is that each node
>> will build a sketch representing it’s own gossip set. Alice’s node will
>> encode and transmit this sketch to Bob’s node, where it will be merged with
>> his own sketch, and the differences produced. These differences should
>> ideally be exactly the latest missing gossip of both nodes. Due to size
>> constraints, the set differences will necessarily be encoded, but Bob’s node
>> will be able to identify which gossip Alice is missing, and may then
>> transmit exactly those messages.
>>
>> This process is relatively straightforward, with the caveat that the sets
>> must otherwise match very closely (each sketch has a maximum capacity for
>> differences.) The difficulty here is that each node and lightning
>> implementation may have its own rules for gossip acceptance and propagation.
>> Depending on their gossip partners, not all gossip may propagate to the
>> entire network.
>>
>> Core-lightning implements rate limiting for incoming channel updates and
>> node announcements. The default rate limit is 1 per day, with a burst of 4.
>> I analyzed my node’s gossip over a 14 day period, and found that, of all
>> publicly broadcasting half-channels, 18% of them fell afoul of our
>> spam-limiting rules at least once. [2]
>>
>> Picking several offending channel ids, and digging further, the majority of
>> these appear to be flapping due to Tor or otherwise intermittent
>> connections. Well connected nodes may be more susceptible to this due to
>> more frequent routing attempts, and failures resulting in a returned channel
>> update (which otherwise might not have been broadcast.)A slight relaxation
>> of the rate limit resolves the majority of these cases.
>>
>> A smaller subset of channels broadcast frequent channel updates with minor
>> adjustments to htlc_maximum_msat and fee_proportional_millionths parameters.
>> These nodes appear to be power users, with many channels and large balances.
>> I assume this is automated channel management at work.
>>
>> Core-Lightning has updated rate-limiting in the upcoming release to achieve
>> a higher acceptance of incoming gossip, however, it seems that a broader
>> discussion of rate limits may now be worthwhile. A few immediate ideas:
>>
>> - A common listing of current default rate limits across lightning network
>> implementations.
>>
>> - Internal checks of RPC input to limit or warn of network propagation
>> issues if certain rates are exceeded.
>>
>> - A commonly adopted rate-limit standard.
>>
>> My aim is a set reconciliation gossip type, which will use a common, simple
>> heuristic to accept or reject a gossip message. (Think one channel update
>> per block, or perhaps one per block_height << 5.) See my github for my
>> current draft. [3] This solution allows tighter consensus, yet suffers from
>> the same problem as original anti-spam measures – it remains somewhat
>> arbitrary. I would like to start a conversation regarding gossip
>> propagation, channel_update and node_announcement usage, and perhaps even
>> bandwidth goals for syncing gossip in the future (how about a million
>> channels?) This would aid in the development of gossip set reconciliation,
>> but could also benefit current node connection and routing reliability more
>> generally.
>>
>> Thanks,
>>
>> Alex
>>
>> [1] https://github.com/sipa/minisketch
>>
>> [2]
>> https://github.com/endothermicdev/lnspammityspam/blob/main/sampleoutput.txt
>>
>> [3]
>> https://github.com/endothermicdev/lightning-rfc/blob/gossip-minisketch/07-routing-gossip.md#set-reconciliation
>>
>> _______________________________________________
>> Lightning-dev mailing list
>> [email protected]
>> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
_______________________________________________
Lightning-dev mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
