EC2 doesn't mount SSDs automatically, so I actually reformat those as ex4, mount them and point hdfs to them. I haven't tweaked any FS options and not sure what cause the problem. I will follow Christopher's suggestion and see whether I can find any cause.
-chen On Fri, Jan 17, 2014 at 11:22 AM, Matei Zaharia <[email protected]> wrote: > What file system do you have? One thing we’ve observed is that ext3, which is > the default on ephemeral disks on EC2, scales very poorly to multicore > workloads. We recommend reformatting those as XFS (which is very fast to > format) or ext4 (which unfortunately takes a few hours to finalize). Maybe > there are also other FS options that affect SSDs. > > Matei > > On Jan 17, 2014, at 8:01 AM, Andrew Ash <[email protected]> wrote: > >> Are there different amounts of RAM on the SSD machines vs the Spinny disk >> machines? >> >> Sent from my mobile phone >> On Jan 17, 2014 5:22 AM, "Jay" <[email protected]> wrote: >> >>> OS memory cache?? >>> >>> Sent from my iPad. >>> >>>> 在 2014年1月16日，上午6:04，Chen Jin <[email protected]> 写道： >>>> >>>> Dear Spark developers: >>>> >>>> We are benchmarking spark operations such as filter, group, join on >>>> ssd instance i2.2xlarge on EC2. Most operations are similar or >>>> slightly better than ephemeral disks on EC2, however, the performance >>>> of group operation on SDD are much worse than regular disks, at least >>>> 2x to 3x worse. Could any of you shed some lights on this behavior? >>>> >>>> Thanks a lot, >>>> >>>> -chen >>> >
