Of all wallets I've tried, by a huge margin, bitcoin core performs the best
with large wallets. I know several massive casinos like bustabit (which has a
lot more transactions than your wallet) have been using bitcoin core for large
wallets, so it can work.
The reason I don't recommend normal people use core, is because of the
initial-block-download and not defaulting to a pruned mode.
The thing that makes your wallet slow is the large amount of wallet
transactions. So the secret to running a large bitcoin core wallet is to just
call `RemovePrunedFunds` on old transactions. Its actually a little tricky
though, because you need to make sure the transaction is "safe to remove". The
first, most obvious, reason a transaction would not be safe to remove is if it
has a wallet utxo that you have not spent or have recently spent. The second,
less obvious reason is if transaction B spends from transaction A, and you
remove transaction B your wallet will think transaction A is unspent (but it's
not!). So you have to prune "Depth first".
-Ryan
------- Original Message -------
On Saturday, August 20th, 2022 at 7:16 AM, Felipe Micaroni Lalli via
bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> Hi dear devs,
>
> 1. THE ISSUE - DAEMON+CLI
> ========================
> I had a wallet in a server production since 2017 (5 years old) and when it
> reached about 273 MB, 2.079.337 transactions and 446.503 generated addresses,
> the performance started to degrade exponentially.
>
> Most of the commands, e.g. "getbalance", "walletpassphrase" and
> "getreceivedbyaddress" started to timeout (more than 15 minutes delay -
> default timeout). The CPU was 100% used (all 32 cores - with 150 load avg)
> and the machine became almost unusable breaking everything else, with the
> default config of 16 RPC threads and 15 min timeout and some attempt calls
> per mi
>
> Increasing the timeout and/or the RPC threads in the config file turns things
> even worse.
>
> Putting the wallet.dat in a very fast SSD disk and increasing the size of the
> cache (I tried with 8GB) have improved but I'm not sure if it is enough.
>
> 2. TEST ON BITCOIN QT
> ====================
> If you try to load the wallet in the "bitcoin-qt" everything gets stuck, even
> the system (OS/UI) doesn't respond anymore. You click on a button and receive
> the message "window doesn't respond, wait or terminate?" - if you wait it
> releases after a while but it is slow and hard to use the wallet anyway.
>
> 3. WHY IS THIS SO BAD?
> =====================
> This is bad because the standard client becomes almost useless for the wallet
> feature:
>
> 3.1) the wallet Qt already is not so popular among end users. It doesn't look
> modern, slow to first sync and hard to use. That's why people prefer to use
> Electrum or Wasabi - I personally don't care but it's the sad truth;
>
> 3.2) it becomes useless now also for servers in production, forcing them to
> use third party solutions for huge wallets. Even if you split in 10 wallets
> it will just delay 10 times more each to degrade, postponing the problem but
> not eliminating it. Not to mention the slow and daily degradation.
>
> 4. SHOULD WE GIVE UP THE WALLET FEATURE?
> ========================================
> Then, Bitcoin Core becomes just a reference implementation and blocks
> relayers, but as an application wallet itself turns into a really bad choice.
> --- It leads me to the following question: if we won't invest time on
> improving this, shouldn't we remove the wallet feature at all? Why keep a
> wallet feature that is not useful for the end user nor the production server?
> Is it useful for what then?
>
> 5. THE CURRENT "SOLUTION" IS BAD
> ===============================
> Currently, the only "solution" for huge wallets is shameful: create a new one
> and send the funds there from time to time. But when is the right time
> exactly? The performance degrades suddenly or gets worse slowly for each new
> address and/or tx?. And besides not being an elegant solution and "not in the
> handbook", it also can break a lot of things like monitoring old addresses
> and also can lead to privacy concerns unifying lots of inputs in a big and
> expensive tx.
>
> 6. OTHER USER CASES?
> ====================
> I think this could also become an issue if we have LN nodes that use the
> Bitcoin Core wallet infrastructure behind to open / close many channels for a
> long time.
>
> 7. FINAL THOUGHTS
> =================
> If moving the wallet from a HDD to a SSD improved a lot, maybe just caching
> the entire wallet in memory could improve even more, but I'm afraid some code
> optimization is also necessary.
>
> 8. SOME QUESTIONS
> ==================
> 8.1) Can we "optimize" a huge wallet without moving the funds to a new one?
> Like a "fsck" or eqv?
>
> 8.2) Can we improve the cache usage somehow? Putting the entire wallet in
> memory, for example?
>
> 8.3) Is it possible to reduce the wallet size (273 MB is too much for a HD
> wallet)?
>
> 8.4) Can we tell the CLI to ignore old addresses? What if I need to watch
> only the last 30 days?
>
> 8.5) How to improve the I/O treatment and/or CPU usage in the main thread on
> Bitcoin-Qt to avoid window freezing on big and huge wallets?
>
> 8.6) In the last case (if it was not possible to optimize the wallet or the
> CLI & Qt), can the CLI just warn the user like: "the wallet is becoming too
> big and slow, execute the command 'archive'". And then, the command "archive"
> could rename the current wallet to something like
> "wallet.dat.archive_until_20220818", create a new "wallet.dat" and move the
> funds automatically? Also, would it be nice to have an
> "autoarchivehugewallets=1" in the file config?
>
> 9. POSSIBLE RELATED AND TESTS
> =============================
>
> [1] https://github.com/bitcoin/bitcoin/issues/15015
> [2] https://github.com/bitcoin/bitcoin/issues/15148
> [3] https://github.com/bitcoin/bitcoin/issues/16874
> [4] https://github.com/bitcoin/bitcoin/pull/17135
> [5] https://github.com/bitcoin/bitcoin/pull/18160
> [6] https://github.com/bitpay/bitcore-node/issues/463
> [7] https://github.com/RavenProject/Ravencoin/issues/499
> [8] https://github.com/sugarchain-project/sugarchain/issues/106
> [9]
> https://bitcoin.stackexchange.com/questions/111844/loadwallet-takes-too-much-and-times-out
> [10] https://bitcoin.stackexchange.com/a/45713/1761
>
> ANOTHER POSSIBLE BUG
> ======================
>
> Even if my node is 100% sync:
> 2022-08-20T13:11:43Z UpdateTip: new
> best=00000000000000000005bba0593c2be0f1d322223501591d2b31b544e3af3d0b
> height=750300 version=0x2fffe000 log2_work=93.687081 tx=758181489
> date='2022-08-20T13:11:16Z' progress=1.000000 cache=4.6MiB(34964txo)
>
> After a "loadwallet" command I am getting an old / wrong balance. The wallet
> is already empty because I had moved the funds to a new one 3 or 4 days ago
> but it is still showing the old balance. I didn't receive any warning message
> saying the need for a rescan or something like that.
>
> I am trying the "rescanblockchain" command but it is running and it taking a
> looooooooooooooong time.
>
> Best regards,
>
> Felipe.
_______________________________________________
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
