Scott, This looks more like " bug#*6596237 Stop looking and start ganging <http://monaco.sfbay/detail.jsf?cr=6596237>". * What version of Solaris are the production servers running (S10 or Opensolaris) ?
Thanks and regards, Sanjeev. Scott wrote: > Hello, > > I have several ~12TB storage servers using Solaris with ZFS. Two of them > have recently developed performance issues where the majority of time in an > spa_sync() will be spent in the space_map_*() functions. During this time, > "zpool iostat" will show 0 writes to disk, while it does hundreds or > thousands of small (~3KB) reads each second, presumably reading space map > data from disk to find places to put the new blocks. The result is that it > can take several minutes for an spa_sync() to complete, even if I'm only > writing a single 128KB block. > > Using DTrace, I can see that space_map_alloc() frequently returns -1 for > 128KB blocks. From my understanding of the ZFS code, that means that one or > more metaslabs has no 128KB blocks available. Because of that, it seems to > be spending a lot of time going through different space maps which aren't > able to all be cached in RAM at the same time, thus causing bad performance > as it has to read from the disks. The on-disk space map size seems to be > about 500MB. > > I assume the simple solution is to leave enough free space available so that > the space map functions don't have to hunt around so much. This problem > starts happening when there's about 1TB free out of the 12TB. It seems like > such a shame to waste that much space, so if anyone has any suggestions, I'd > be glad to hear them. > > 1) Is there anything I can do to temporarily fix the servers that are having > this problem? They are production servers, and I have customers complaining, > so a temporary fix is needed. > > 2) Is there any sort of tuning I can do with future servers to prevent this > from becoming a problem? Perhaps a way to make sure all the space maps are > always in RAM? > > 3) I set recordsize=32K and turned off compression, thinking that should fix > the performance problem for now. However, using a DTrace script to watch > calls to space_map_alloc(), I see that it's still looking for 128KB blocks > (!!!) for reasons that are unclear to me, thus it hasn't helped the problem. > > Thanks, > Scott > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > -- Solaris Revenue Products Engineering, India Engineering Center, Sun Microsystems India Pvt Ltd. Tel: x27521 +91 80 669 27521 _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
