I found out what the problem was with X. The iBookDV install guide on hadess.net turns out to be a little out of date. You should not put the "novideo" line in your yaboot.conf, and then aty128fb will work correctly, even under an old (2.2.x) kernel.
Thanks to hadess for patiently troubleshooting this problem. -Jeremy