Re: [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us"

[Antoine Riard]

Hi list,

As I received a lot of feedback on the full disclosure of the 16th week of
October and the following posts, some accurate, I'm taking time to address
a few of them.

I think one of the most recurring feedback is the fact that the replacement
cycling issue laid out in the initial full disclosure post could have been
better explained or presented. Here you have a typical disclosure dilemma
encountered by any security researcher, on one hand you wish to explain
them as best you can to enlighten the public audience, on the other hand if
you come up with a full exploitation rootkit, its manual and a paper with a
lot of graphic informing on all the way to exploit, you might expose the
public audience funds.

As a reminder, at the time of the full disclosure the 16th, some lightning
implementations were still adding up mitigations in their codebase and
corresponding release has only been tagged last week. Moreover, I think
when you're doing a full disclosure as a security researcher, it is wiser
to be conservative in the flow of information disclosed, as you might have
missed some observations or insights worsening the severity and
practicality of exploitation.

In my reasonable and modest experience, the amount and clarity of
information you're revealing in matters of security disclosure is an art
rhythmed by contingencies, not a science.

Additionally, there is one "unusual" security risk only affecting Bitcoin
second-layers, namely triggering winds of panic among lightning node
operators. This group of users started to manually force-close channels and
holistically congestion network mempools, opening the door to opportunistic
"flood & loot" exploitation, a risk known by lightning experts for years.

E.g bare option anchor commitment transactions are 900 WU (bolt3), max
block size is 4_000_000 WU, if you have half of the 80k of _public_
channels, you might have a hour and half of full blocks, which might offer
the opportunity of stealing payments from payee, based on historical
timelocks.

This last risk is based on experience from previous security coordination
work and I did inform the core security list of this concern in date of the
5 october: " if we see an abnormal rate of network mempools congestion from
LN node operators manually force-closing their channels with low-trusted
peers, this phenomena has already happened during past Lightning security
issues disclosures" as it could have provoked disruptions ecosystem-wide
beyond lightning.

There have been some voices speaking up on the sudden announcement from my
side to step down from lightning development. While there has been complete
distorsion of my statement by non-specialized journalists and I've been the
first one to complain, howev er it was very deliberate on my side to ring
the bell on the very poor state of lightning security and marked the score
than this new replacement cycling issue is a far more concerning than other
major security risks of lightning known so far. As a friendly reminder, we
start to pile up a very serious stack of security risks: pinnings, channel
jamming, replacement cycling, time dilation (all those ones could kill
lightning if exploited at scale - period) and more minor ones such as dust
HTLC exposure, fee or liquidity griefing and other denial-of-service.

Hard things about hard things, every new major security risk discovered
somehow constrains lightning developers to go back on the whiteboard and
check that any mitigation in development for each risk is not broken or the
security properties with. Most of the time, we might think in isolation to
ease the mitigation of research and development work. Astute adversaries
might not give us those flowers. At the end, technical reality advises that
lightning security issues are better solved at the base-layer, and this is
where expert time is cruelly missing to make lightning more robust on the
very long-term.

On a more ethical plan, there is no wish on my side to stay accountable on
the coordinated handling of security issues in the lightning ecosystem,
when I have the inner conviction than replacement cycling issues are
practical and critical, that they cannot be solved correctly on the
lightning-side, that any serious mitigation on the base-layer might take
numerous years and their integration back in the lightning-side might take
another couple of years. Once again, I warned the whole community of the
nest of issues at the mempool-level for lightning very consistently during
the past years.

With humility, if those replacement cycling attacks start to be played out
at scale by sophisticated attackers during the coming future, I'm not
competent to improvise an effective mitigation. And even less competent to
coordinate dissemination of such eventual patch of mitigations across an
ecosystem of nodes, without losing inter-compatibility between
implementations, wallets and releases versions (and here the lack of
standardized mechanism like dynamic upgrade bolt 

Re: [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us"

[Antoine Riard]

Hi,

As I've been shown offline Twitter posts misrepresenting my previous mail,
I think it's good to correct them. The security flaws are not "intentional
backdoor" or whatever misrepresentation that would question the competence
and know-how of the Bitcoin and Lightning development community.

The replacement cycling issue discovered has been known by a small circle
of Bitcoin developers since December 2022. As it appears to some experts
and it has been commented publicly, changes at the bitcoin base-layer might
be the most substantial fixes. Those changes take time and here this is
akin to how the linux kernel, bsds and OS vendors are working [0].

All I can say is that we had recently had internal discussion on how to
improve coordinated security fixes and patching processes for the coming
decades. This is an area of concern where I've always been at the forefront
as early as 2020 / 2021 [1].

In the meanwhile, lightning experts have already deployed mitigations which
are hardening the lightning ecosystem significantly in face of simple or
medium attacks. More advanced attacks can only be mounted if you have
sufficient p2p and mempool knowledge as was pointed out by other bitcoin
experts like Matt or Peter (which take years to acquire for average bitcoin
developers) and the months of preparation to attempt them.

If you're a journalist reporting on the information in mainstream crypto
publications, I'll suggest waiting to do so before expert reporters of
bitcoin circles who have more in-field knowledge can do so and qualify the
technical situation with more distance. As I've already been interviewed by
top financial publication years ago for my work on bitcoin, as a journalist
you're engaging your own reputation on the information you're reporting.
Thanks for being responsible here.

This is the nature of the electronic communication and contemporaneous
media that information is extremely fluid and there is no native anti-DoS
mechanism to slow down the propagation of sensitive information where
mitigations are still in deployment. A reason I'm not on social media of
any kind [2]. In the meanwhile, it's good to go to read senecca and marcus
aurelius take the situation with stoicism and with a zelt of meditation [3].

All my previous statements are mostly technically correct (even if some
could have been written with more clarity once again I'm not an english
native [4]). While I wish to wait the week of the 30th Oct o discuss
further what is best fix and what are the trade-offs as a community as a
wide (give time some laggard lightning implementations ship fixes), though
I'll comment further on the mailing list if the flow of information on
"social media" is DoSing the ability of the bitcoin community to work on
the long-term appropriate fixes in a responsible and constructive fashion.

[0] See meltdown class of vulnerability and how operating systems are
handling hardware-sourced vulnerabilities
https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability). Most of
the time they do their best on the software side and they go to see with
hardware vendors how to do the necessary updates.

[1]
https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-April/003002.html

[2] And for the wider analysis on contemporaneous culture of information
propagation and network effect, I can only recommend to read venkatesh
rao's ribbonfarm essays http://ribbonfarm.com

[3] There are very good reasons why some executives at top modern
technology companies are doing meditation daily, some even hours. "mind
illuminated" is a good read.

[4] While my former employer, Chaincode Labs, paid for my english lessons
back in 2020. Generally it was a good insight from them to train people on
how to communicate in a crisis.


Le ven. 20 oct. 2023 à 07:56, Antoine Riard  a
écrit :

> Hi,
>
> After writing the mail reply on the economics of sequential malicious
> replacement of honest HTLC-timeout, I did write one more test to verify the
> behavior on core mempool, and it works as expected.
>
>
> https://github.com/ariard/bitcoin/commit/30f5d5b270e4ff195e8dcb9ef6b7ddcc5f6a1bf2
>
> Responsible disclosure process has followed the lines of hardware issues
> affecting operating system, as documented for the Linux kernel, while
> adapted to the bitcoin ecosystem:
>
> https://docs.kernel.org/6.1/process/embargoed-hardware-issues.html
>
> Effective now, I'm halting my involvement with the development of the
> lightning network and its implementations, including coordinating the
> handling of security issues at the protocol level (I informed some senior
> lightning devs in that sense before).
>
> Closed the very old issue which was affected to me at this purpose on the
> bolt repository:
>
> https://github.com/lightning/bolts/pull/772
>
> I think this new class of replacement cycling attacks puts lightning in a
> very perilous position, where only a sustainable fix can happen at the
> base-layer, e.g adding a memory-intensive history 

Re: [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us"

[Antoine Riard]

Hi,

After writing the mail reply on the economics of sequential malicious
replacement of honest HTLC-timeout, I did write one more test to verify the
behavior on core mempool, and it works as expected.

https://github.com/ariard/bitcoin/commit/30f5d5b270e4ff195e8dcb9ef6b7ddcc5f6a1bf2

Responsible disclosure process has followed the lines of hardware issues
affecting operating system, as documented for the Linux kernel, while
adapted to the bitcoin ecosystem:

https://docs.kernel.org/6.1/process/embargoed-hardware-issues.html

Effective now, I'm halting my involvement with the development of the
lightning network and its implementations, including coordinating the
handling of security issues at the protocol level (I informed some senior
lightning devs in that sense before).

Closed the very old issue which was affected to me at this purpose on the
bolt repository:

https://github.com/lightning/bolts/pull/772

I think this new class of replacement cycling attacks puts lightning in a
very perilous position, where only a sustainable fix can happen at the
base-layer, e.g adding a memory-intensive history of all-seen transactions
or some consensus upgrade. Deployed mitigations are worth something in face
of simple attacks, though I don't think they're stopping advanced attackers
as said in the first full disclosure mail.

Those types of changes are the ones necessitating the utmost transparency
and buy-in of the community as a whole, as we're altering the full-nodes
processing requirements or the security architecture of the decentralized
bitcoin ecosystem in its integrality.

On the other hand fully explaining why such changes would be warranted for
the sake of lightning and for designing them well, we might need to lay out
in complete state practical and critical attacks on a ~5 355 public BTC
ecosystem.

Hard dilemma.

There might be a lesson in terms of bitcoin protocol deployment, we might
have to get them right at first try. Little second chance to fix them in
flight.

I'll be silent on those issues on public mailing lists until the week of
the 30 oct. Enough material has been published and other experts are
available. Then I'll be back focusing more on bitcoin core.

Best,
Antoine

Le lun. 16 oct. 2023 à 17:57, Antoine Riard  a
écrit :

> (cross-posting mempool issues identified are exposing lightning chan to
> loss of funds risks, other multi-party bitcoin apps might be affected)
>
> Hi,
>
> End of last year (December 2022), amid technical discussions on eltoo
> payment channels and incentives compatibility of the mempool anti-DoS
> rules, a new transaction-relay jamming attack affecting lightning channels
> was discovered.
>
> After careful analysis, it turns out this attack is practical and
> immediately exposed lightning routing hops carrying HTLC traffic to loss of
> funds security risks, both legacy and anchor output channels. A potential
> exploitation plausibly happening even without network mempools congestion.
>
> Mitigations have been designed, implemented and deployed by all major
> lightning implementations during the last months.
>
> Please find attached the release numbers, where the mitigations should be
> present:
> - LDK: v0.0.118 - CVE-2023 -40231
> - Eclair: v0.9.0 - CVE-2023-40232
> - LND: v.0.17.0-beta - CVE-2023-40233
> - Core-Lightning: v.23.08.01 - CVE-2023-40234
>
> While neither replacement cycling attacks have been observed or reported
> in the wild since the last ~10 months or experimented in real-world
> conditions on bitcoin mainet, functional test is available exercising the
> affected lightning channel against bitcoin core mempool (26.0 release
> cycle).
>
> It is understood that a simple replacement cycling attack does not demand
> privileged capabilities from an attacker (e.g no low-hashrate power) and
> only access to basic bitcoin and lightning software. Yet I still think
> executing such an attack successfully requests a fair amount of bitcoin
> technical know-how and decent preparation.
>
> From my understanding of those issues, it is yet to be determined if the
> mitigations deployed are robust enough in face of advanced replacement
> cycling attackers, especially ones able to combine different classes of
> transaction-relay jamming such as pinnings or vetted with more privileged
> capabilities.
>
> Please find a list of potential affected bitcoin applications in this full
> disclosure report using bitcoin script timelocks or multi-party
> transactions, albeit no immediate security risk exposure as severe as the
> ones affecting lightning has been identified. Only cursory review of
> non-lightning applications has been conducted so far.
>
> There is a paper published summarizing replacement cycling attacks on the
> lightning network:
>
> https://github.com/ariard/mempool-research/blob/2023-10-replacement-paper/replacement-cycling.pdf
>
>  ## Problem
>
> A lightning node allows HTLCs forwarding (in bolt3's parlance accepted
> HTLC on incoming link and offered HTLC 

Re: [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us"

[Antoine Riard]

The disclosure mails noted a 3rd mitigation beyond mempool scanning and
transaction re-signing / re-broadcasting, namely bumping CLTV delta.

Generally bumping CLTV delta is a basic line of mitigations for a lot of
lightning attacks, as it gives opportunity to node operators to intervene
and re-broadcast their time-sensitive transactions on other interfaces (e.g
a secondary full-node if the first one is eclipsed).

About the second mitigation transaction re-signing, if done correctly at
least sounds to put an economic cost (denominated in fees / feerates) on
the attack. This is unclear to me if the game-theory of this cost holds.

One thing which sounds to me making the attack harder is stratum v2
deployment, as you're increasing the number of miners which might do their
own block templates, and therefore the number of miners' mempools where an
attacker has to successfully continuously replace in cycles channels
counterparties transactions.

A replacement buffer or history of transactions at the mempool level might
be a mitigation to this attack. I believe this is yet to be seen if it can
be made robust enough.

I don't know if folks like tadge or rusty who have been involved in the
early design of lightning have more ideas of mitigations. Fees was noted as
a hard issue in the original paper.

Le mer. 18 oct. 2023 à 01:17, Matt Corallo  a
écrit :

> There appears to be some confusion about this issue and the mitigations.
> To be clear, the deployed
> mitigations are not expected to fix this issue, its arguable if they
> provide anything more than a PR
> statement.
>
> There are two discussed mitigations here - mempool scanning and
> transaction re-signing/re-broadcasting.
>
> Mempool scanning relies on regularly checking the mempool of a local node
> to see if we can catch the
> replacement cycle mid-cycle. It only works if wee see the first
> transaction before the second
> transaction replaces it.
>
> Today, a large majority of lightning nodes run on machines with a Bitcoin
> node on the same IP
> address, making it very clear what the "local node" of the lightning node
> is. An attacker can
> trivially use this information to connect to said local node and do the
> replacement quickly,
> preventing the victim from seeing the replacement.
>
> More generally, however, similar discoverability is true for mining pools.
> An attacker performing
> this attack is likely to do the replacement attack on a miner's node
> directly, potentially reducing
> the reach of the intermediate transaction to only miners, such that the
> victim can never discover it
> at all.
>
> The second mitigation is similarly pathetic. Re-signing and
> re-broadcasting the victim's transaction
> in an attempt to get it to miners even if its been removed may work, if
> the attacker is super lazy
> and didn't finish writing their attack system. If the attacker is
> connected to a large majority of
> hashrate (which has historically been fairly doable), they can simply do
> their replacement in a
> cycle aggressively and arbitrarily reduce the probability that the
> victim's transaction gets confirmed.
>
> Now, the above is all true in a spherical cow kinda world, and the P2P
> network has plenty of slow
> nodes and strange behavior. Its possible that these mitigations might, by
> some stroke of luck,
> happen to catch such an attack and prevent it, because something took
> longer than the attacker
> intended or whatever. But, that's a far cry from any kind of material
> "fix" for the issue.
>
> Ultimately the only fix for this issue will be when miners keep a history
> of transactions they've
> seen and try them again after they may be able to enter the mempool
> because of an attack like this.
>
> Matt
>
> On 10/16/23 12:57 PM, Antoine Riard wrote:
> > (cross-posting mempool issues identified are exposing lightning chan to
> loss of funds risks, other
> > multi-party bitcoin apps might be affected)
> >
> > Hi,
> >
> > End of last year (December 2022), amid technical discussions on eltoo
> payment channels and
> > incentives compatibility of the mempool anti-DoS rules, a new
> transaction-relay jamming attack
> > affecting lightning channels was discovered.
> >
> > After careful analysis, it turns out this attack is practical and
> immediately exposed lightning
> > routing hops carrying HTLC traffic to loss of funds security risks, both
> legacy and anchor output
> > channels. A potential exploitation plausibly happening even without
> network mempools congestion.
> >
> > Mitigations have been designed, implemented and deployed by all major
> lightning implementations
> > during the last months.
> >
> > Please find attached the release numbers, where the mitigations should
> be present:
> > - LDK: v0.0.118 - CVE-2023 -40231
> > - Eclair: v0.9.0 - CVE-2023-40232
> > - LND: v.0.17.0-beta - CVE-2023-40233
> > - Core-Lightning: v.23.08.01 - CVE-2023-40234
> >
> > While neither replacement cycling attacks have been observed or reported
> in 

Re: [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us"

[Antoine Riard]

> We have conducted one so far, multiple scenarios to look at.

_*none*_ so far. typo of mine - apologize english is not my native language.

We discussed conducting experiments pre-disclosure in an e-mail of the 11th
August 2023.

"If someone is down to setup a "black box" Lightning infra on mainet, I'm
game on to exercise the vulnerabilities and mitigations during the coming
months and revisit the disclosure date dependent on the learnings."

However as the number of Lightning worldwide experts who have level of
knowledge and understandings to take part to experiments is I think
restrained to people listed on the disclosure mails _and_ we had other
pendings non-disclosed security issues at the time like the ones revealed
"fake channel DoS vector" the 23th August 2023, we didn't conduct them.

Le mar. 17 oct. 2023 à 18:47, Antoine Riard  a
écrit :

> Hi Ziggie,
>
> > thanks for this detailed explanation. This class of pinning attacks
> sound not too unlikely especially if the attacker targets channels with
> high capacity and very loose channel policies (allowing the full htlc
> > amount to be the channel capacity). Could you add more details about the
> attack you observed on mainnet ? How did you monitor the chain, are the
> some tools available I can run in parallel to my
> > lightning software to record this kind of suspicious behaviour (which
> did you use)?
>
> Just to give a clarification no _such_ attack has been observed on
> mainnet, since I and other LN maintainers have been aware of this issue. If
> there is a confusion on the disclosure mail thanks to point to it, I'll
> correct it.
>
> We discussed privately to experiment a demo attack in restrained dev
> circles, like we have done in the past for some LN sec issues. We have
> conducted one so far, multiple scenarios to look at.
>
> I confirm the risk of exposure if an attacker targets channels with high
> capacity and loose channel policies. Note there is no way to configure the
> cap for the total value of outbound HTLC in-flight, which is the flow
> affected.
>
> If you would like to observe the existence of such an attack happening,
> look at your mempool logs and the amount of HTLC output being
> systematically conflicted out with the following sequence (HTLC-timeout -
> HTLC-preimage - HTLC-timeout - ...).
>
> As an observation note, this is not akin to a pinning attack, as there is
> no "honest" or "malicious" transaction pinned in network mempools. And it
> can happen without network mempools congestion.
>
> > What's also worth mentioning here is that you do not really have to
> control 2 neighbouring nodes to target your victim. If you can cycle the
> attack on the tail side and delay the confirmation of the htlc- timeout
> covenant the peer at the front (incoming link) of the victim will
> force-close the channel and claim his timeout-path in the same way
> (canceling back the initial htlc amount to the attackers initial node).
>
> I think this is a behavior worthy of testing.
>
> > Apart from that I think one can even introduce some kind of feebumping
> race between the victim and the attacker on the tail side of the attack
> making the attack even more costly. I think
> > currently when lightning nodes see the preimage in the mempool (during
> the time where they already can spend the same output with the
> timeout-covenant) we are honest and just extract
> > the preimage and don't try to race this tx output.
>
> Local-mempool preimage monitoring has been implemented by Eclair for years
> as a mitigation against old school pinning attacks on second-stage HTLC
> transactions.
>
> This mechanism has been implemented by LND in the last months, following
> the report of replacement cycling attacks. As of today this is not
> implemented by Core-Lightning or LDK.
>
> > So we maybe should start feebumping this output if we end up in this
> scenario? If we see the preimage and can also claim this output via the
> htlc-timeout path, we should aggressively fee-bump (racing this output) our
> htlc-output in addition to grabbing the preimage and claiming it on the
> incoming. This is only feasible with anchor channels where we can add fees
> to the htlc-covenant. This would make the attack more costly for a peer
> when he knows that we use fees up to 50% of the htlc value. When you cycle
> this 144 times you will be at a heavy loss trying to steal this htlc.
>
> This is the "defensive fee mitigation" proposed in the paper. Coming with
> some unknown.
>
> > I would add another mitigation to the list for node runners to restrict
> the amount and number of HTLCs  for big channels to unknown peers. It
> quickly comes with a loss when the HTLCs the attacker tries to steal are
> small.
>
> See the point above on the lack of way at the spec-level to negotiate cap
> on the total value of outbound HTLC in-flight.
>
> Le mar. 17 oct. 2023 à 08:21, ziggie1984  a
> écrit :
>
>> ## Deployed LN mitigations
>>
>> Aggressive rebroadcasting: As the 

Re: [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us"

[Antoine Riard]

Hi Zeeman,

> At block height 100, `B` notices the `B->C` HTLC timelock is expired
without `C` having claimed it, so `B` forces the `BC` channel onchain.
> However, onchain feerates have risen and the commitment transaction and
HTLC-timeout transaction do not confirm.

This is not that the HTLC-timeout does not confirm. It is replaced in
cycles by C's HTLC-preimage which is still valid after `B->C` HTLC timelock
has expired. And this HTLC-preimage is subsequently replaced itself.

See the test here:
https://github.com/ariard/bitcoin/commit/19d61fa8cf22a5050b51c4005603f43d72f1efcf

> At block height 144, `B` is still not able to claim the `A->B` HTLC, so
`A` drops the `AB` channel onchain.
> As the fees are up-to-date, this confirms immediately and `A` is able to
recover the HTLC funds.
> However, the feerates of the `BC` pre-signed transactions remain at
the old, uncompetitive feerates.

This is correct that A tries to recover the HTLC funds on the `A===B`
channel.

However, there is no need to consider the fee rates nor mempool congestion
as the exploit lays on the replacement mechanism itself (in simple
scenario).

> At this point, `C` broadcasts an HTLC-success transaction with high
feerates that CPFPs the commitment tx.
> However, it replaces the HTLC-timeout transaction, which is at the old,
low feerate.
> `C` is thus able to get the value of the HTLC, but `B` is now no longer
able to use the knowledge of the preimage, as its own incoming HTLC was
already confirmed as claimed by `A`.

This is correct that `C` broadcasts an HTLC-success transaction at block
height 144.

However `C` broadcasts this high feerate transaction at _every block_
between blocks 100 and 144 to replace B's HTLC-timeout transaction.

> Let me also explain to non-Lightning experts why HTLC-timeout is
presigned in this case and why `B` cannot feebump it.

Note `B` can feebump the HTLC-timeout for anchor output channels thanks to
sighash_single | anyonecanpay on C's signature.

Le mar. 17 oct. 2023 à 11:34, ZmnSCPxj  a écrit :

> Good morning Antoine et al.,
>
> Let me try to rephrase the core of the attack.
>
> There exists these nodes on the LN (letters `A`, `B`, and `C` are nodes,
> `==` are channels):
>
> A = B = C
>
> `A` routes `A->B->C`.
>
> The timelocks, for example, could be:
>
>A->B timeelock = 144
>B->C timelock = 100
>
> The above satisfies the LN BOLT requirements, as long as `B` has a
> `cltv_expiry_delta` of 44 or lower.
>
> After `B` forwards the HTLC `B->C`, C suddenly goes offline, and all the
> signed transactions --- commitment transaction and HTLC-timeout
> transactions --- are "stuck" at the feerate at the time.
>
> At block height 100, `B` notices the `B->C` HTLC timelock is expired
> without `C` having claimed it, so `B` forces the `BC` channel onchain.
> However, onchain feerates have risen and the commitment transaction and
> HTLC-timeout transaction do not confirm.
>
> In the mean time, `A` is still online with `B` and updates the onchain
> fees of the `AB` channel pre-signed transactions (commitment tx and
> HTLC-timeout tx) to the latest.
>
> At block height 144, `B` is still not able to claim the `A->B` HTLC, so
> `A` drops the `AB` channel onchain.
> As the fees are up-to-date, this confirms immediately and `A` is able to
> recover the HTLC funds.
> However, the feerates of the `BC` pre-signed transactions remain at
> the old, uncompetitive feerates.
>
> At this point, `C` broadcasts an HTLC-success transaction with high
> feerates that CPFPs the commitment tx.
> However, it replaces the HTLC-timeout transaction, which is at the old,
> low feerate.
> `C` is thus able to get the value of the HTLC, but `B` is now no longer
> able to use the knowledge of the preimage, as its own incoming HTLC was
> already confirmed as claimed by `A`.
>
> Is the above restatement accurate?
>
> 
>
> Let me also explain to non-Lightning experts why HTLC-timeout is presigned
> in this case and why `B` cannot feebump it.
>
> In the Poon-Dryja mechanism, the HTLCs are "infected" by the Poon-Dryja
> penalty case, and are not plain HTLCs.
>
> A plain HTLC offerred by `B` to `C` would look like this:
>
> (B && OP_CLTV) || (C && OP_HASH160)
>
> However, on the commitment transaction held by `B`, it would be infected
> by the penalty case in this way:
>
> (B && C && OP_CLTV) || (C && OP_HASH160) || (C && revocation)
>
> There are two changes:
>
> * The addition of a revocation branch `C && revocation`.
> * The branch claimable by `B` in the "plain" HTLC (`B && OP_CLTV`) also
> includes `C`.
>
> These are necessary in case `B` tries to cheat and this HTLC is on an old,
> revoked transaction.
> If the revoked transaction is *really* old, the `OP_CLTV` would already
> impose a timelock far in the past.
> This means that a plain `B && OP_CLTV` branch can be claimed by `B` if it
> retained this very old revoked transaction.
>
> To prevent that, `C` is added to the `B && 

Re: [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us"

[Matt Corallo]

There appears to be some confusion about this issue and the mitigations. To be clear, the deployed 
mitigations are not expected to fix this issue, its arguable if they provide anything more than a PR 
statement.


There are two discussed mitigations here - mempool scanning and transaction 
re-signing/re-broadcasting.

Mempool scanning relies on regularly checking the mempool of a local node to see if we can catch the 
replacement cycle mid-cycle. It only works if wee see the first transaction before the second 
transaction replaces it.


Today, a large majority of lightning nodes run on machines with a Bitcoin node on the same IP 
address, making it very clear what the "local node" of the lightning node is. An attacker can 
trivially use this information to connect to said local node and do the replacement quickly, 
preventing the victim from seeing the replacement.


More generally, however, similar discoverability is true for mining pools. An attacker performing 
this attack is likely to do the replacement attack on a miner's node directly, potentially reducing 
the reach of the intermediate transaction to only miners, such that the victim can never discover it 
at all.


The second mitigation is similarly pathetic. Re-signing and re-broadcasting the victim's transaction 
in an attempt to get it to miners even if its been removed may work, if the attacker is super lazy 
and didn't finish writing their attack system. If the attacker is connected to a large majority of 
hashrate (which has historically been fairly doable), they can simply do their replacement in a 
cycle aggressively and arbitrarily reduce the probability that the victim's transaction gets confirmed.


Now, the above is all true in a spherical cow kinda world, and the P2P network has plenty of slow 
nodes and strange behavior. Its possible that these mitigations might, by some stroke of luck, 
happen to catch such an attack and prevent it, because something took longer than the attacker 
intended or whatever. But, that's a far cry from any kind of material "fix" for the issue.


Ultimately the only fix for this issue will be when miners keep a history of transactions they've 
seen and try them again after they may be able to enter the mempool because of an attack like this.


Matt

On 10/16/23 12:57 PM, Antoine Riard wrote:
(cross-posting mempool issues identified are exposing lightning chan to loss of funds risks, other 
multi-party bitcoin apps might be affected)


Hi,

End of last year (December 2022), amid technical discussions on eltoo payment channels and 
incentives compatibility of the mempool anti-DoS rules, a new transaction-relay jamming attack 
affecting lightning channels was discovered.


After careful analysis, it turns out this attack is practical and immediately exposed lightning 
routing hops carrying HTLC traffic to loss of funds security risks, both legacy and anchor output 
channels. A potential exploitation plausibly happening even without network mempools congestion.


Mitigations have been designed, implemented and deployed by all major lightning implementations 
during the last months.


Please find attached the release numbers, where the mitigations should be 
present:
- LDK: v0.0.118 - CVE-2023 -40231
- Eclair: v0.9.0 - CVE-2023-40232
- LND: v.0.17.0-beta - CVE-2023-40233
- Core-Lightning: v.23.08.01 - CVE-2023-40234

While neither replacement cycling attacks have been observed or reported in the wild since the last 
~10 months or experimented in real-world conditions on bitcoin mainet, functional test is available 
exercising the affected lightning channel against bitcoin core mempool (26.0 release cycle).


It is understood that a simple replacement cycling attack does not demand privileged capabilities 
from an attacker (e.g no low-hashrate power) and only access to basic bitcoin and lightning 
software. Yet I still think executing such an attack successfully requests a fair amount of bitcoin 
technical know-how and decent preparation.


 From my understanding of those issues, it is yet to be determined if the mitigations deployed are 
robust enough in face of advanced replacement cycling attackers, especially ones able to combine 
different classes of transaction-relay jamming such as pinnings or vetted with more privileged 
capabilities.


Please find a list of potential affected bitcoin applications in this full disclosure report using 
bitcoin script timelocks or multi-party transactions, albeit no immediate security risk exposure as 
severe as the ones affecting lightning has been identified. Only cursory review of non-lightning 
applications has been conducted so far.


There is a paper published summarizing replacement cycling attacks on the 
lightning network:
https://github.com/ariard/mempool-research/blob/2023-10-replacement-paper/replacement-cycling.pdf 



  ## Problem

A lightning node allows HTLCs 

Re: [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us"

[Antoine Riard]

Hi Ziggie,

> thanks for this detailed explanation. This class of pinning attacks sound
not too unlikely especially if the attacker targets channels with high
capacity and very loose channel policies (allowing the full htlc
> amount to be the channel capacity). Could you add more details about the
attack you observed on mainnet ? How did you monitor the chain, are the
some tools available I can run in parallel to my
> lightning software to record this kind of suspicious behaviour (which did
you use)?

Just to give a clarification no _such_ attack has been observed on mainnet,
since I and other LN maintainers have been aware of this issue. If there is
a confusion on the disclosure mail thanks to point to it, I'll correct it.

We discussed privately to experiment a demo attack in restrained dev
circles, like we have done in the past for some LN sec issues. We have
conducted one so far, multiple scenarios to look at.

I confirm the risk of exposure if an attacker targets channels with high
capacity and loose channel policies. Note there is no way to configure the
cap for the total value of outbound HTLC in-flight, which is the flow
affected.

If you would like to observe the existence of such an attack happening,
look at your mempool logs and the amount of HTLC output being
systematically conflicted out with the following sequence (HTLC-timeout -
HTLC-preimage - HTLC-timeout - ...).

As an observation note, this is not akin to a pinning attack, as there is
no "honest" or "malicious" transaction pinned in network mempools. And it
can happen without network mempools congestion.

> What's also worth mentioning here is that you do not really have to
control 2 neighbouring nodes to target your victim. If you can cycle the
attack on the tail side and delay the confirmation of the htlc- timeout
covenant the peer at the front (incoming link) of the victim will
force-close the channel and claim his timeout-path in the same way
(canceling back the initial htlc amount to the attackers initial node).

I think this is a behavior worthy of testing.

> Apart from that I think one can even introduce some kind of feebumping
race between the victim and the attacker on the tail side of the attack
making the attack even more costly. I think
> currently when lightning nodes see the preimage in the mempool (during
the time where they already can spend the same output with the
timeout-covenant) we are honest and just extract
> the preimage and don't try to race this tx output.

Local-mempool preimage monitoring has been implemented by Eclair for years
as a mitigation against old school pinning attacks on second-stage HTLC
transactions.

This mechanism has been implemented by LND in the last months, following
the report of replacement cycling attacks. As of today this is not
implemented by Core-Lightning or LDK.

> So we maybe should start feebumping this output if we end up in this
scenario? If we see the preimage and can also claim this output via the
htlc-timeout path, we should aggressively fee-bump (racing this output) our
htlc-output in addition to grabbing the preimage and claiming it on the
incoming. This is only feasible with anchor channels where we can add fees
to the htlc-covenant. This would make the attack more costly for a peer
when he knows that we use fees up to 50% of the htlc value. When you cycle
this 144 times you will be at a heavy loss trying to steal this htlc.

This is the "defensive fee mitigation" proposed in the paper. Coming with
some unknown.

> I would add another mitigation to the list for node runners to restrict
the amount and number of HTLCs  for big channels to unknown peers. It
quickly comes with a loss when the HTLCs the attacker tries to steal are
small.

See the point above on the lack of way at the spec-level to negotiate cap
on the total value of outbound HTLC in-flight.

Le mar. 17 oct. 2023 à 08:21, ziggie1984  a
écrit :

> ## Deployed LN mitigations
>
> Aggressive rebroadcasting: As the replacement cycling attacker benefits
> from the HTLC-timeout being usually broadcast by lightning nodes only once
> every block, or less the replacement cycling malicious transactions paid
> only equal the sum of the absolute fees paid by the HTLC, adjusted with the
> replacement penalty. Rebroadcasting randomly and multiple times before the
> next block increases the absolute fee cost for the attacker.
>
> Implemented and deployed by Eclair, Core-Lightning, LND and LDK .
>
> Local-mempool preimage monitoring: As the replacement cycling attacker in
> a simple setup broadcast the HTLC-preimage to all the network mempools, the
> honest lightning node is able to catch on the flight the unconfirmed
> HTLC-preimage, before its subsequent mempool replacement. The preimage can
> be extracted from the second-stage HTLC-preimage and used to fetch the
> off-chain inbound HTLC with a cooperative message or go on-chain with it to
> claim the accepted HTLC output.
>
>
> Hi Antoine,
>
> thanks for this detailed 

Re: [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us"

[ZmnSCPxj via Lightning-dev]

Good morning Antoine et al.,

Let me try to rephrase the core of the attack.

There exists these nodes on the LN (letters `A`, `B`, and `C` are nodes, `==` 
are channels):

A = B = C

`A` routes `A->B->C`.

The timelocks, for example, could be:

   A->B timeelock = 144
   B->C timelock = 100

The above satisfies the LN BOLT requirements, as long as `B` has a 
`cltv_expiry_delta` of 44 or lower.

After `B` forwards the HTLC `B->C`, C suddenly goes offline, and all the signed 
transactions --- commitment transaction and HTLC-timeout transactions --- are 
"stuck" at the feerate at the time.

At block height 100, `B` notices the `B->C` HTLC timelock is expired without 
`C` having claimed it, so `B` forces the `BC` channel onchain.
However, onchain feerates have risen and the commitment transaction and 
HTLC-timeout transaction do not confirm.

In the mean time, `A` is still online with `B` and updates the onchain fees of 
the `AB` channel pre-signed transactions (commitment tx and HTLC-timeout 
tx) to the latest.

At block height 144, `B` is still not able to claim the `A->B` HTLC, so `A` 
drops the `AB` channel onchain.
As the fees are up-to-date, this confirms immediately and `A` is able to 
recover the HTLC funds.
However, the feerates of the `BC` pre-signed transactions remain at the 
old, uncompetitive feerates.

At this point, `C` broadcasts an HTLC-success transaction with high feerates 
that CPFPs the commitment tx.
However, it replaces the HTLC-timeout transaction, which is at the old, low 
feerate.
`C` is thus able to get the value of the HTLC, but `B` is now no longer able to 
use the knowledge of the preimage, as its own incoming HTLC was already 
confirmed as claimed by `A`.

Is the above restatement accurate?



Let me also explain to non-Lightning experts why HTLC-timeout is presigned in 
this case and why `B` cannot feebump it.

In the Poon-Dryja mechanism, the HTLCs are "infected" by the Poon-Dryja penalty 
case, and are not plain HTLCs.

A plain HTLC offerred by `B` to `C` would look like this:

(B && OP_CLTV) || (C && OP_HASH160)

However, on the commitment transaction held by `B`, it would be infected by the 
penalty case in this way:

(B && C && OP_CLTV) || (C && OP_HASH160) || (C && revocation)

There are two changes:

* The addition of a revocation branch `C && revocation`.
* The branch claimable by `B` in the "plain" HTLC (`B && OP_CLTV`) also 
includes `C`.

These are necessary in case `B` tries to cheat and this HTLC is on an old, 
revoked transaction.
If the revoked transaction is *really* old, the `OP_CLTV` would already impose 
a timelock far in the past.
This means that a plain `B && OP_CLTV` branch can be claimed by `B` if it 
retained this very old revoked transaction.

To prevent that, `C` is added to the `B && OP_CLTV` branch.
We also introduce an HTLC-timeout transaction, which spends the `B && C && 
OP_CLTV` branch, and outputs to:

(B && OP_CSV) || (C && revocation)

Thus, even if `B` held onto a very old revoked commitment transaction and 
attempts to spend the timelock branch (because the `OP_CLTV` is for an old 
blockheight), it still has to contend with a new output with a *relative* 
timelock.

Unfortunately, this means that the HTLC-timeout transaction is pre-signed, and 
has a specific feerate.
In order to change the feerate, both `B` and `C` have to agree to re-sign the 
HTLC-timeout transaction at the higher feerate.

However, the HTLC-success transaction in this case spends the plain `(C && 
OP_HASH160)` branch, which only involves `C`.
This allows `C` to feebump the HTLC-success transaction arbitrarily even if `B` 
does not cooperate.

While anchor outputs can be added to the HTLC-timeout transaction as well, `C` 
has a greater advantage here due to being able to RBF the HTLC-timeout out of 
the way (1 transaction), while `B` has to get both HTLC-timeout and a CPFP-RBF 
of the anchor output of the HTLC-timeout transaction (2 transactions).
`C` thus requires a smaller fee to achieve a particular feerate due to having 
to push a smaller number of bytes compared to `B`.

Regards,
ZmnSCPxj
___
Lightning-dev mailing list
Lightning-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev

Re: [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us"

[ziggie1984 via Lightning-dev]

> ## Deployed LN mitigations
>
> Aggressive rebroadcasting: As the replacement cycling attacker benefits from 
> the HTLC-timeout being usually broadcast by lightning nodes only once every 
> block, or less the replacement cycling malicious transactions paid only equal 
> the sum of the absolute fees paid by the HTLC, adjusted with the replacement 
> penalty. Rebroadcasting randomly and multiple times before the next block 
> increases the absolute fee cost for the attacker.
>
> Implemented and deployed by Eclair, Core-Lightning, LND and LDK .
>
> Local-mempool preimage monitoring: As the replacement cycling attacker in a 
> simple setup broadcast the HTLC-preimage to all the network mempools, the 
> honest lightning node is able to catch on the flight the unconfirmed 
> HTLC-preimage, before its subsequent mempool replacement. The preimage can be 
> extracted from the second-stage HTLC-preimage and used to fetch the off-chain 
> inbound HTLC with a cooperative message or go on-chain with it to claim the 
> accepted HTLC output.

Hi Antoine,

thanks for this detailed explanation. This class of pinning attacks sound not 
too unlikely especially if the attacker targets channels with high capacity and 
very loose channel policies (allowing the full htlc amount to be the channel 
capacity). Could you add more details about the attack you observed on mainnet 
? How did you monitor the chain, are the some tools available I can run in 
parallel to my lightning software to record this kind of suspicious behaviour 
(which did you use)?
What's also worth mentioning here is that you do not really have to control 2 
neighbouring nodes to target your victim. If you can cycle the attack on the 
tail side and delay the confirmation of the htlc-timeout covenant the peer at 
the front (incoming link) of the victim will force-close the channel and claim 
his timeout-path in the same way (canceling back the initial htlc amount to the 
attackers initial node).

Apart from that I think one can even introduce some kind of feebumping race 
between the victim and the attacker on the tail side of the attack making the 
attack even more costly. I think currently when lightning nodes see the 
preimage in the mempool (during the time where they already can spend the same 
output with the timeout-covenant) we are honest and just extract the preimage 
and don't try to race this tx output. So we maybe should start feebumping this 
output if we end up in this scenario? If we see the preimage and can also claim 
this output via the htlc-timeout path, we should aggressively fee-bump (racing 
this output) our htlc-output in addition to grabbing the preimage and claiming 
it on the incoming. This is only feasible with anchor channels where we can add 
fees to the htlc-covenant. This would make the attack more costly for a peer 
when he knows that we use fees up to 50% of the htlc value. When you cycle this 
144 times you will be at a heavy loss trying to steal this htlc.

I would add another mitigation to the list for node runners to restrict the 
amount and number of HTLCs for big channels to unknown peers. It quickly comes 
with a loss when the HTLCs the attacker tries to steal are small.

Kind regards,

ziggie

--- Original Message ---
On Monday, October 16th, 2023 at 18:57, Antoine Riard  
wrote:

> (cross-posting mempool issues identified are exposing lightning chan to loss 
> of funds risks, other multi-party bitcoin apps might be affected)
>
> Hi,
>
> End of last year (December 2022), amid technical discussions on eltoo payment 
> channels and incentives compatibility of the mempool anti-DoS rules, a new 
> transaction-relay jamming attack affecting lightning channels was discovered.
>
> After careful analysis, it turns out this attack is practical and immediately 
> exposed lightning routing hops carrying HTLC traffic to loss of funds 
> security risks, both legacy and anchor output channels. A potential 
> exploitation plausibly happening even without network mempools congestion.
>
> Mitigations have been designed, implemented and deployed by all major 
> lightning implementations during the last months.
>
> Please find attached the release numbers, where the mitigations should be 
> present:
> - LDK: v0.0.118 - CVE-2023 -40231
> - Eclair: v0.9.0 - CVE-2023-40232
> - LND: v.0.17.0-beta - CVE-2023-40233
> - Core-Lightning: v.23.08.01 - CVE-2023-40234
>
> While neither replacement cycling attacks have been observed or reported in 
> the wild since the last ~10 months or experimented in real-world conditions 
> on bitcoin mainet, functional test is available exercising the affected 
> lightning channel against bitcoin core mempool (26.0 release cycle).
>
> It is understood that a simple replacement cycling attack does not demand 
> privileged capabilities from an attacker (e.g no low-hashrate power) and only 
> access to basic bitcoin and lightning software. Yet I still think executing 
> such an attack successfully requests a