Re: ZFS performance question
On 08/20/10 12:30, Heinrich Rebehn wrote: > I am somewhat concerned about the numbers for per-char-output and > per-char-input. In fact, i have never before seen that low numbers in a > bonnie test. Using a single disk with UFS yields about 6 times as much. > > BTW: Running OpenSolaris on the same hardware yields 110306 for > per-char-write and 94698 for per-char-read. "per-char" stats are different between different operating systems because of how they are implemented. Apparently, bonnie++ forces full disk writes (fsyncs) for each byte written on BSDs, but Linux (and apparently Solaris) somehow manage to write-cache this (or at least - cache it much more). It only matters if you have software which depends on this caching and performs slowly otherwise. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: ZFS performance question
On 20.08.2010, at 14:52, Vladislav V. Prodan wrote: > 20.08.2010 15:12, Heinrich Rebehn wrote: >> r...@antsrv4 [/data/nocompression] # kldstat > > #kldstat -v Id Refs AddressSize Name 19 0x8010 d6aa98 kernel (/boot/kernel/kernel) Contains modules: Id Name 93 ataraid 340 if_lo 407 elf32 328 shell 327 elf64 313 pseudofs 280 uether 341 if_tun 339 if_gif 342 if_vlan 350 mld 349 igmp 337 if_faith 332 sysvmsg 333 sysvsem 335 sem 334 sysvshm 356 nfslockd 351 nfs_common 355 nfssvc 354 nfsserver 360 krpc 361 ufs 312 procfs 352 nfs 311 msdosfs 310 devfs 326 cd9660 6 cam 63 ata 16 sa 15 pass 13 ada 322 g_part_mbr 321 g_part_gpt 320 g_part_ebr 319 g_part_bsd 10 probe 12 ch 14 da 7 xpt 11 cd 8 aprobe 17 ses 9 pmp 221 pci/ppc 220 isa/ppc 219 acpi/ppc 218 ppbus/ppi 113 pci/dc 217 ppc/ppbus 216 ppbus/lpt 112 dc/miibus 215 ppbus/plip 214 pci/pcn 213 pcn/miibus 111 pccard/cs 212 pci/vgapci 211 pci/pcib 210 pcib/pci 110 isa/cs 53 alc/miibus 52 ahc 109 cpu/ichss 51 ahd 108 pci/ciss 50 pci/ahd 209 pci/isab 208 pci/ignore_pci 207 pci/hostb 206 pci/fixup_pci 205 pci/cbb 49 pci/ahc_pci 48 isa/ahc_isa 204 isa/cbb 47 pccard/aic 22 nexus/acpi 31 pcib/acpi_pci 107 cbb/cardbus 106 pci/bt 105 isa/bt 203 pcic/pccard 411 cpu/p4tcc 410 cpu/hwpstate 202 cbb/pccard 201 null 409 cpu/est 408 cpu/powernow 200 pci/nge 199 nge/miibus 104 pci/bge 103 bge/miibus 20 pci/aacch 406 isa/vga 405 vgapci/vgapm 198 pci/mskc 404 isa/sc 403 isa/atrtc 402 acpi/atrtc 401 scrndr-vga 400 scterm-scteken 197 mskc/msk 196 msk/miibus 102 pci/bfe 399 pci/nfe 398 nfe/miibus 101 bfe/miibus 397 pci/hptrr 396 pci/hptmv 395 pccard/fdc 394 isa/fdc 393 acpi/fdc 392 fdc/fd 391 io 100 pci/bce 390 isa/ed 193 pci/mpt 99 bce/miibus 389 atkbdc/psm 388 isa/psmcpnp 387 acpi/psmcpnp 46 pci/age 386 isa/atkbdc 385 acpi/atkbdc 384 atkbdc/atkbd 383 pci/arcmsr 382 hostb/agp_via 381 hostb/agp_intel 380 vgapci/agp_i810 379 hostb/agp_amd64 45 age/miibus 190 pci/mly 378 legacy/pcib 377 isa/pcibus_pnp 376 isa/atdma 375 acpi/atdma 374 legacy/isa 189 pci/mlx 188 mlx/mlxd 373 isa/attimer 372 acpi/attimer 187 miibus/xmphy 186 miibus/ukphy 185 miibus/truephy 371 root/nexus 370 nexus/ram 369 isa/sysresource 184 miibus/tlphy 183 miibus/tdkphy 182 miibus/smcphy 181 miibus/ruephy 180 miibus/rlphy 179 miibus/rgephy 178 miibus/qsphy 177 miibus/pnaphy 176 miibus/nsphyter 175 miibus/nsphy 368 nexus/legacy 367 legacy/cpu 174 miibu
Re: ZFS performance question
20.08.2010 15:12, Heinrich Rebehn wrote: > r...@antsrv4 [/data/nocompression] # kldstat #kldstat -v ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: ZFS performance question
On 20.08.2010, at 12:48, Olivier Smedts wrote: > 2010/8/20 Heinrich Rebehn : >> Hi all, >> >> After setting up our new server (Intel Q9550 CPU, 8GB RAM, 4 x ST31000340NS) >> i did a bonnie++ benchmark on the zfs raidz that i created on 4 partitions >> on the 4 disks. >> >> r...@antsrv4 [/data/nocompression] # bonnie++ -u root -d . >> Using uid:0, gid:0. >> Writing a byte at a time...done >> Writing intelligently...done >> Rewriting...done >> Reading a byte at a time...done >> Reading intelligently...done >> start 'em...done...done...done...done...done... >> Create files in sequential order...done. >> Stat files in sequential order...done. >> Delete files in sequential order...done. >> Create files in random order...done. >> Stat files in random order...done. >> Delete files in random order...done. >> Version 1.96 --Sequential Output-- --Sequential Input- >> --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec >> %CP >> antsrv4.ant.uni 16G94 81 151823 39 100780 27 303 99 239853 30 >> 148.3 4 >> Latency 4733ms6051ms8173ms 37504us 917ms1030ms >> Version 1.96 --Sequential Create-- Random >> Create >> antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec >> %CP >> 16 24183 97 + +++ 21715 98 17209 98 + +++ 21291 >> 97 >> Latency 19779us 128us 162us 39671us 38us 77us >> 1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282303232,16G,,94,81,151823,39,100780,27,303,99,239853,30,148.3,4,16,24183,97,+,+++,21715,98,17209,98,+,+++,21291,97, >> >> 733ms,6051ms,8173ms,37504us,917ms,1030ms,19779us,128us,162us,39671us,38us,77us >> >> >> I am somewhat concerned about the numbers for per-char-output and >> per-char-input. In fact, i have never before seen that low numbers in a >> bonnie test. Using a single disk with UFS yields about 6 times as much. >> >> >> I know that this is not crucial for a normal file server, but i want to rule >> out any configuration errors. >> >> Is this normal for ZFS?. Disks are accessed using ahci. I did not attempt >> any tuning yet. > > Can you try without AHCI ? > http://www.phoronix.com/scan.php?page=article&item=freebsd_zfs_cam&num=5 Hmm, that looks impressive, but: r...@antsrv4 [/data/nocompression] # bonnie++ -u root -d . Using uid:0, gid:0. [snip] Delete files in random order...done. Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP antsrv4.ant.uni 16G 107 99 152759 39 118765 33 304 99 227534 29 137.4 4 Latency 216ms6110ms6436ms 108ms 717ms 784ms Version 1.96 --Sequential Create-- Random Create antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 23264 94 + +++ 20417 96 24430 97 + +++ 20866 97 Latency 20142us 138us3506us 19454us 128us 77us 1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282317527,16G,,107,99,152759,39,118765,33,304,99,227534,29,137.4,4,16,23264,94,+,+++,20417,96,24430,97,+,+++,20866,97,216ms,6110ms,6436ms,108ms,717ms,784ms,20142us,138us,3506us,19454us,128us,77us Almost no difference. Only the create rate increased. r...@antsrv4 [/data/nocompression] # kldstat Id Refs AddressSize Name 19 0x8010 d6aa98 kernel 21 0x80e6b000 19eb18 zfs.ko 32 0x8100a000 3868 opensolaris.ko r...@antsrv4 [/data/nocompression] # ls /dev/ada* ls: /dev/ada*: No such file or directory -Heinrich > >> BTW: Running OpenSolaris on the same hardware yields 110306 for >> per-char-write and 94698 for per-char-read. >> > > > > -- > Olivier Smedts _ > ASCII ribbon campaign ( ) > e-mail: oliv...@gid0.org- against HTML email & vCards X > www: http://www.gid0.org- against proprietary attachments / \ > > "Il y a seulement 10 sortes de gens dans le monde : > ceux qui comprennent le binaire, > et ceux qui ne le comprennent pas." > ___ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" Heinrich Rebehn University of Bremen Physics / Electrical and Electronics Engineering - Department of Telecommunications - Phone : +49/421/218-6239
Re: ZFS performance question
2010/8/20 Heinrich Rebehn : > Hi all, > > After setting up our new server (Intel Q9550 CPU, 8GB RAM, 4 x ST31000340NS) > i did a bonnie++ benchmark on the zfs raidz that i created on 4 partitions on > the 4 disks. > > r...@antsrv4 [/data/nocompression] # bonnie++ -u root -d . > Using uid:0, gid:0. > Writing a byte at a time...done > Writing intelligently...done > Rewriting...done > Reading a byte at a time...done > Reading intelligently...done > start 'em...done...done...done...done...done... > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...done. > Create files in random order...done. > Stat files in random order...done. > Delete files in random order...done. > Version 1.96 --Sequential Output-- --Sequential Input- > --Random- > Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec > %CP > antsrv4.ant.uni 16G 94 81 151823 39 100780 27 303 99 239853 30 > 148.3 4 > Latency 4733ms 6051ms 8173ms 37504us 917ms 1030ms > Version 1.96 --Sequential Create-- Random > Create > antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 16 24183 97 + +++ 21715 98 17209 98 + +++ 21291 97 > Latency 19779us 128us 162us 39671us 38us 77us > 1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282303232,16G,,94,81,151823,39,100780,27,303,99,239853,30,148.3,4,16,24183,97,+,+++,21715,98,17209,98,+,+++,21291,97, > > 733ms,6051ms,8173ms,37504us,917ms,1030ms,19779us,128us,162us,39671us,38us,77us > > > I am somewhat concerned about the numbers for per-char-output and > per-char-input. In fact, i have never before seen that low numbers in a > bonnie test. Using a single disk with UFS yields about 6 times as much. > > > I know that this is not crucial for a normal file server, but i want to rule > out any configuration errors. > > Is this normal for ZFS?. Disks are accessed using ahci. I did not attempt any > tuning yet. Can you try without AHCI ? http://www.phoronix.com/scan.php?page=article&item=freebsd_zfs_cam&num=5 > BTW: Running OpenSolaris on the same hardware yields 110306 for > per-char-write and 94698 for per-char-read. > > --Heinrich > > > Heinrich Rebehn > > University of Bremen > Physics / Electrical and Electronics Engineering > - Department of Telecommunications - > > Phone : +49/421/218-62394 > Fax : -3341 > > > > > > ___ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" > -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email & vCards X www: http://www.gid0.org - against proprietary attachments / \ "Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas." ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
