Bignum fixes backported in 7.8

[Jonas Walld�n @ Pike developers forum]

I've merged most of the bignum fixes from 8.0 into 7.8 since we need
to build with clang on OS X. I've excluded changes that added new API
or that removed the AUTO_BIGNUM define and everything passes
verbose_verify on my local machine. If you see any unintended
side-effects please let me know.