Yes, wg is not a request/response protocol. But it does have some state. Can't wireguard remember the last local address that each peer sent traffic? It is just like the tracking already in use for peer ip address. If there is an "last address" it would be nice if we could hint the kernel to use that as the source address, with a fallback to the current behavior if the address is not available. It might solve a couple of problems. I just don't know if it is possible to hint the source address without enforcing it. It not, wg would have to deal with cases when the address is gone.
Regards, Luiz