You can, but OSes usually give you randomized memory layout as a security measure.
What if the memory allocation scheme were something like: randomly pick memory locations below some threshold from the 32bit segment and then above the threshold pick from elsewhere?