Raspbian is primarily intended to be used as a desktop replacement and/or a network video/media player, so it needs a GUI to fulfil the purpose, and as far as I know there's no separate server (=headless) version because, as you noted, it's very easy to remove the GUI part using standard system methods.
Regarding "clean" versus "unclean" installation, you should not worry this much, it's not Windows. Any package, once removed, does not occupy any disk space and does not leave any processes running. You may delete all traces of packages, including unwanted configuration settings, using --purge
:
apt-get remove --purge PACKAGE
And you may even clean the apt cache with
sudo apt-get clean
to pretend it was never installed on your system.