Sorry I can't help you, although personally I think the spacing looks just right. Maybe with your line spacing settings it looks funny? I agree thought that LyX does not seem to be very flexible here.
I just wanted to add to your comments the following. When you start a "display formula", if you hit return before doing so it adds an extra space. If you start the equation but don't hit return before no space is added. The problem is that in LyX you can't see the difference, so you only notice the large gap when you compile the document. Bit annoying right? Chris.