Pulling Xorg dependencies on headless machine

[nazriel]

What are the rationales for pulling in Xorg dependencies on 
headless machines like web server?

I am installing DMD on Debian machine from D-APT but I guess 
installing from dlang.org dpkg would result in the same behaviour.

Packages like noveau, readeon, intel drivers and other Xorg 
utilis are not very useful in such scenario.

Any way to fix it?