Re: ZFS performance question
On 08/20/10 12:30, Heinrich Rebehn wrote: > I am somewhat concerned about the numbers for per-char-output and > per-char-input. In fact, i have never before seen that low numbers in a > bonnie test. Using a single disk with UFS yields about 6 times as much. > > BTW: Running OpenSolaris on the same hardware yields 110306 for > per-char-write and 94698 for per-char-read. "per-char" stats are different between different operating systems because of how they are implemented. Apparently, bonnie++ forces full disk writes (fsyncs) for each byte written on BSDs, but Linux (and apparently Solaris) somehow manage to write-cache this (or at least - cache it much more). It only matters if you have software which depends on this caching and performs slowly otherwise. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: ZFS performance question
On 20.08.2010, at 14:52, Vladislav V. Prodan wrote: > 20.08.2010 15:12, Heinrich Rebehn wrote: >> r...@antsrv4 [/data/nocompression] # kldstat > > #kldstat -v Id Refs AddressSize Name 19 0x8010 d6aa98 kernel (/boot/kernel/kernel) Contains modules: Id Name 93 ataraid 340 if_lo 407 elf32 328 shell 327 elf64 313 pseudofs 280 uether 341 if_tun 339 if_gif 342 if_vlan 350 mld 349 igmp 337 if_faith 332 sysvmsg 333 sysvsem 335 sem 334 sysvshm 356 nfslockd 351 nfs_common 355 nfssvc 354 nfsserver 360 krpc 361 ufs 312 procfs 352 nfs 311 msdosfs 310 devfs 326 cd9660 6 cam 63 ata 16 sa 15 pass 13 ada 322 g_part_mbr 321 g_part_gpt 320 g_part_ebr 319 g_part_bsd 10 probe 12 ch 14 da 7 xpt 11 cd 8 aprobe 17 ses 9 pmp 221 pci/ppc 220 isa/ppc 219 acpi/ppc 218 ppbus/ppi 113 pci/dc 217 ppc/ppbus 216 ppbus/lpt 112 dc/miibus 215 ppbus/plip 214 pci/pcn 213 pcn/miibus 111 pccard/cs 212 pci/vgapci 211 pci/pcib 210 pcib/pci 110 isa/cs 53 alc/miibus 52 ahc 109 cpu/ichss 51 ahd 108 pci/ciss 50 pci/ahd 209 pci/isab 208 pci/ignore_pci 207 pci/hostb 206 pci/fixup_pci 205 pci/cbb 49 pci/ahc_pci 48 isa/ahc_isa 204 isa/cbb 47 pccard/aic 22 nexus/acpi 31 pcib/acpi_pci 107 cbb/cardbus 106 pci/bt 105 isa/bt 203 pcic/pccard 411 cpu/p4tcc 410 cpu/hwpstate 202 cbb/pccard 201 null 409 cpu/est 408 cpu/powernow 200 pci/nge 199 nge/miibus 104 pci/bge 103 bge/miibus 20 pci/aacch 406 isa/vga 405 vgapci/vgapm 198 pci/mskc 404 isa/sc 403 isa/atrtc 402 acpi/atrtc 401 scrndr-vga 400 scterm-scteken 197 mskc/msk 196 msk/miibus 102 pci/bfe 399 pci/nfe 398 nfe/miibus 101 bfe/miibus 397 pci/hptrr 396 pci/hptmv 395 pccard/fdc 394 isa/fdc 393 acpi/fdc 392 fdc/fd 391 io 100 pci/bce 390 isa/ed 193 pci/mpt 99 bce/miibus 389 atkbdc/psm 388 isa/psmcpnp 387 acpi/psmcpnp 46 pci/age 386 isa/atkbdc 385 acpi/atkbdc 384 atkbdc/atkbd 383 pci/arcmsr 382 hostb/agp_via 381 hostb/agp_intel 380 vgapci/agp_i810 379 hostb/agp_amd64 45 age/miibus 190 pci/mly 378 legacy/pcib 377 isa/pcibus_pnp 376 isa/atdma 375 acpi/atdma 374 legacy/isa 189 pci/mlx 188 mlx/mlxd 373 isa/attimer 372 acpi/attimer 187 miibus/xmphy 186 miibus/ukphy 185 miibus/truephy 371 root/nexus 370 nexus/ram 369 isa/sysresource 184 miibus/tlphy 183 miibus/tdkphy 182 miibus/smcphy 181 miibus/ruephy 180 miibus/rlphy 179 miibus/rgephy 178 miibus/qsphy 177 miibus/pnaphy 176 miibus/nsphyter 175 miibus/nsphy 368 nexus/legacy 367 legacy/cpu 174 miibu
Re: ZFS performance question
20.08.2010 15:12, Heinrich Rebehn wrote: > r...@antsrv4 [/data/nocompression] # kldstat #kldstat -v ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: ZFS performance question
On 20.08.2010, at 12:48, Olivier Smedts wrote: > 2010/8/20 Heinrich Rebehn : >> Hi all, >> >> After setting up our new server (Intel Q9550 CPU, 8GB RAM, 4 x ST31000340NS) >> i did a bonnie++ benchmark on the zfs raidz that i created on 4 partitions >> on the 4 disks. >> >> r...@antsrv4 [/data/nocompression] # bonnie++ -u root -d . >> Using uid:0, gid:0. >> Writing a byte at a time...done >> Writing intelligently...done >> Rewriting...done >> Reading a byte at a time...done >> Reading intelligently...done >> start 'em...done...done...done...done...done... >> Create files in sequential order...done. >> Stat files in sequential order...done. >> Delete files in sequential order...done. >> Create files in random order...done. >> Stat files in random order...done. >> Delete files in random order...done. >> Version 1.96 --Sequential Output-- --Sequential Input- >> --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec >> %CP >> antsrv4.ant.uni 16G94 81 151823 39 100780 27 303 99 239853 30 >> 148.3 4 >> Latency 4733ms6051ms8173ms 37504us 917ms1030ms >> Version 1.96 --Sequential Create-- Random >> Create >> antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec >> %CP >> 16 24183 97 + +++ 21715 98 17209 98 + +++ 21291 >> 97 >> Latency 19779us 128us 162us 39671us 38us 77us >> 1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282303232,16G,,94,81,151823,39,100780,27,303,99,239853,30,148.3,4,16,24183,97,+,+++,21715,98,17209,98,+,+++,21291,97, >> >> 733ms,6051ms,8173ms,37504us,917ms,1030ms,19779us,128us,162us,39671us,38us,77us >> >> >> I am somewhat concerned about the numbers for per-char-output and >> per-char-input. In fact, i have never before seen that low numbers in a >> bonnie test. Using a single disk with UFS yields about 6 times as much. >> >> >> I know that this is not crucial for a normal file server, but i want to rule >> out any configuration errors. >> >> Is this normal for ZFS?. Disks are accessed using ahci. I did not attempt >> any tuning yet. > > Can you try without AHCI ? > http://www.phoronix.com/scan.php?page=article&item=freebsd_zfs_cam&num=5 Hmm, that looks impressive, but: r...@antsrv4 [/data/nocompression] # bonnie++ -u root -d . Using uid:0, gid:0. [snip] Delete files in random order...done. Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP antsrv4.ant.uni 16G 107 99 152759 39 118765 33 304 99 227534 29 137.4 4 Latency 216ms6110ms6436ms 108ms 717ms 784ms Version 1.96 --Sequential Create-- Random Create antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 23264 94 + +++ 20417 96 24430 97 + +++ 20866 97 Latency 20142us 138us3506us 19454us 128us 77us 1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282317527,16G,,107,99,152759,39,118765,33,304,99,227534,29,137.4,4,16,23264,94,+,+++,20417,96,24430,97,+,+++,20866,97,216ms,6110ms,6436ms,108ms,717ms,784ms,20142us,138us,3506us,19454us,128us,77us Almost no difference. Only the create rate increased. r...@antsrv4 [/data/nocompression] # kldstat Id Refs AddressSize Name 19 0x8010 d6aa98 kernel 21 0x80e6b000 19eb18 zfs.ko 32 0x8100a000 3868 opensolaris.ko r...@antsrv4 [/data/nocompression] # ls /dev/ada* ls: /dev/ada*: No such file or directory -Heinrich > >> BTW: Running OpenSolaris on the same hardware yields 110306 for >> per-char-write and 94698 for per-char-read. >> > > > > -- > Olivier Smedts _ > ASCII ribbon campaign ( ) > e-mail: oliv...@gid0.org- against HTML email & vCards X > www: http://www.gid0.org- against proprietary attachments / \ > > "Il y a seulement 10 sortes de gens dans le monde : > ceux qui comprennent le binaire, > et ceux qui ne le comprennent pas." > ___ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" Heinrich Rebehn University of Bremen Physics / Electrical and Electronics Engineering - Department of Telecommunications - Phone : +49/421/218-6239
Re: ZFS performance question
2010/8/20 Heinrich Rebehn : > Hi all, > > After setting up our new server (Intel Q9550 CPU, 8GB RAM, 4 x ST31000340NS) > i did a bonnie++ benchmark on the zfs raidz that i created on 4 partitions on > the 4 disks. > > r...@antsrv4 [/data/nocompression] # bonnie++ -u root -d . > Using uid:0, gid:0. > Writing a byte at a time...done > Writing intelligently...done > Rewriting...done > Reading a byte at a time...done > Reading intelligently...done > start 'em...done...done...done...done...done... > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...done. > Create files in random order...done. > Stat files in random order...done. > Delete files in random order...done. > Version 1.96 --Sequential Output-- --Sequential Input- > --Random- > Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec > %CP > antsrv4.ant.uni 16G 94 81 151823 39 100780 27 303 99 239853 30 > 148.3 4 > Latency 4733ms 6051ms 8173ms 37504us 917ms 1030ms > Version 1.96 --Sequential Create-- Random > Create > antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 16 24183 97 + +++ 21715 98 17209 98 + +++ 21291 97 > Latency 19779us 128us 162us 39671us 38us 77us > 1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282303232,16G,,94,81,151823,39,100780,27,303,99,239853,30,148.3,4,16,24183,97,+,+++,21715,98,17209,98,+,+++,21291,97, > > 733ms,6051ms,8173ms,37504us,917ms,1030ms,19779us,128us,162us,39671us,38us,77us > > > I am somewhat concerned about the numbers for per-char-output and > per-char-input. In fact, i have never before seen that low numbers in a > bonnie test. Using a single disk with UFS yields about 6 times as much. > > > I know that this is not crucial for a normal file server, but i want to rule > out any configuration errors. > > Is this normal for ZFS?. Disks are accessed using ahci. I did not attempt any > tuning yet. Can you try without AHCI ? http://www.phoronix.com/scan.php?page=article&item=freebsd_zfs_cam&num=5 > BTW: Running OpenSolaris on the same hardware yields 110306 for > per-char-write and 94698 for per-char-read. > > --Heinrich > > > Heinrich Rebehn > > University of Bremen > Physics / Electrical and Electronics Engineering > - Department of Telecommunications - > > Phone : +49/421/218-62394 > Fax : -3341 > > > > > > ___ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" > -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org - against HTML email & vCards X www: http://www.gid0.org - against proprietary attachments / \ "Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas." ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
ZFS performance question
Hi all, After setting up our new server (Intel Q9550 CPU, 8GB RAM, 4 x ST31000340NS) i did a bonnie++ benchmark on the zfs raidz that i created on 4 partitions on the 4 disks. r...@antsrv4 [/data/nocompression] # bonnie++ -u root -d . Using uid:0, gid:0. Writing a byte at a time...done Writing intelligently...done Rewriting...done Reading a byte at a time...done Reading intelligently...done start 'em...done...done...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP antsrv4.ant.uni 16G94 81 151823 39 100780 27 303 99 239853 30 148.3 4 Latency 4733ms6051ms8173ms 37504us 917ms1030ms Version 1.96 --Sequential Create-- Random Create antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 24183 97 + +++ 21715 98 17209 98 + +++ 21291 97 Latency 19779us 128us 162us 39671us 38us 77us 1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282303232,16G,,94,81,151823,39,100780,27,303,99,239853,30,148.3,4,16,24183,97,+,+++,21715,98,17209,98,+,+++,21291,97, 733ms,6051ms,8173ms,37504us,917ms,1030ms,19779us,128us,162us,39671us,38us,77us I am somewhat concerned about the numbers for per-char-output and per-char-input. In fact, i have never before seen that low numbers in a bonnie test. Using a single disk with UFS yields about 6 times as much. I know that this is not crucial for a normal file server, but i want to rule out any configuration errors. Is this normal for ZFS?. Disks are accessed using ahci. I did not attempt any tuning yet. BTW: Running OpenSolaris on the same hardware yields 110306 for per-char-write and 94698 for per-char-read. --Heinrich Heinrich Rebehn University of Bremen Physics / Electrical and Electronics Engineering - Department of Telecommunications - Phone : +49/421/218-62394 Fax :-3341 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
