I've been trying for a couple days now to get Nexenta installed on a second partition to dual boot with OS X. I thought I'd share my experience. I have not yet been successful.
As the original poster mentioned, I used BootCamp to resize the partition. This left me 12GB at the end of the disk for Solaris, 43GB in the middle with OS X, and ~200MB at the front for EFI. I also used fdisk to change the partition ID to 0xBF (191) for Solaris2. However, once inside 'format' in the Nexenta installer, it was obvious that Solaris was not properly interpreting the partitioning. First, when I would print the partition layout, I would get slice 0 beginning at cyl 0 and ending about 200MB later. I believe this maps to the EFI partition. Strangely, slice 2's first cylinder appeared to be at the beginning of the Solaris partition. I updated the root slice to use cylinder numbers that matched the second slice. It appeared to accept that, so I continued with my install. However the install bombed out quickly. Upon inspection, df showed the newly created filesystem was only 200MB. I'm not sure what the cause is for this yet. It's worth mentioning that I needed BootCamp to get the Nexenta (or even grub from an SXCR CD) to boot successfully into the installer. I think this is due to the BIOS emulation. When attempting to boot directly from the CD/DVD without using BootCamp (ie. holding down 'c' when booting instead of 'option') the boot process seems to work if you instruct grub what to do manually (as if it could not read its menu.lst), but the installer never appears. When booting from CD via BootCamp, I get the installer as expected. I'll post again if I make any more progress. This message posted from opensolaris.org
