As of now I couldn't find anything helpfull. But as the opensolaris aac driver seems to be derived from the freebsd aac driver, which had problems on machines with more (and equal) then 4GB of ram. The Symptomes seem to be exactly like described - first it seems all ok but after the first request the Drive wont answer anymore.
The freebsd bug seems to be fixed already, and according to the bug tracker it shouldn't be in Opensolaris, but for me it still is. Does anyone has any advice? This message posted from opensolaris.org
