Are you sure that the key exchange is not a factor? Have you measured SSL setup times compared to post-setup transfer times?
4K RSA is computationally expensive. Are you sure that the rest of your system is secure enough to justify that instead of 2K RSA? /r$ -- Principal Security Engineer Akamai Technology Cambridge, MA