I've done some further investigations and found that these extremely slow speeds seem to only apply to apt repositories (I tried multiple different ones and all of them have the same problem). The card is consistently hitting max speeds / near max speeds everywhere except on apt. Using apt-fast and a 5GHz network significantly helps out.
Still unsure if the wrong firmware selection is linked to this issue though.
