Hello, I did simple tryout for GLFS 2.0rc8 on CentOS Linux with 2x Dual Core Xeon 4GB RAM. My benchmarks, it is high load single stream access on loopback glfs mount on a single server with high performance FC RAID. Target volume is XFS formatted. Local benchmark results are as follows.(benchmark command is XDD command)
Buf I/O READ = about 660MB/s WRITE = about 480MB/s Direct I/O 4MB block read = about 540MB/s 4MB block write = about 350MB/s The results for GLFS loopback mount volume are as follows. Buf I/O READ = about 460MB/s WRITE = about 330MB/s Direct I/O 4MB block read = about 160MB/s 4MB block write = about 100MB/s Buf I/O with GLFS is good results with small block size. But The large block size access is slows down. Direct I/O is poor performance without follow the block size. Attached please find a detailed information text. I want to use glfs with professional video applications on IB networks. Video applications are using storages with large uncompress image sequences and/or uncompress large movie files.(up to 2K/4K) The block size control and direct I/O performance are important for them. Please advise me about options/configurations for improve the performance, and theory for improve performance by block size on GLFS. My best regards, hideo
############################################################################################################# GLUSTERFS server config ############################################################################################################# file : /etc/glusterfs/glusterfsd.vol volume posix type storage/posix option directory /vol1 end-volume volume brick type performance/io-threads option thread-count 8 subvolumes posix end-volume ### Add network serving capability to above brick. volume server type protocol/server option transport-type tcp #option transport-type ib-sdp option client-volume-filename /etc/glusterfs/glusterfs-client.vol option auth.addr.brick.allow * # Allow access to "brick" volume subvolumes brick end-volume ############################################################################################################# GLUSTERFS client config ############################################################################################################# file : /etc/glusterfs/glusterfsd.vol volume client type protocol/client option transport-type tcp #option transport-type ib-sdp option remote-host 192.168.2.99 # IP address of the remote brick #option remote-host 10.10.11.99 # IP address of the remote brick option remote-subvolume brick # name of the remote volume end-volume volume iot type performance/io-threads option thread-count 4 subvolumes client end-volume #volume writebehind # type performance/write-behind # option aggregate-size 1MB # option window-size 4MB # subvolumes iot #end-volume #volume cache # type performance/io-cache # option cache-size 512MB # subvolumes writebehind #end-volume ############################################################################################################# GLUSTERFS Processes ############################################################################################################# root 14188 1 4 14:21 ? 00:05:39 /usr/sbin/glusterfsd -f /etc/glusterfs/glusterfsd.vol root 14234 1 4 14:21 ? 00:05:26 /usr/sbin/glusterfs --log-level=NORMAL --volfile=/etc/glusterfs/glusterfs-client.vol /mnt/gluster/ ############################################################################################################# XFS access ############################################################################################################# xdd - Version 4.2 - AHPCRC/UofMN Writing to target /vol1/tmp/BIGFILE1 alignment is on 4096 byte boundary Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time hi_time times errors 256 4096 16384 492.95 497.79 486.08 8.713 8.628 8.836 3 0 512 4096 8192 462.39 466.97 456.28 9.289 9.197 9.413 3 0 1024 4096 4096 474.29 484.45 469.01 9.056 8.866 9.158 3 0 2048 4096 2048 493.54 497.72 488.56 8.702 8.629 8.791 3 0 4096 4096 1024 478.62 491.81 468.66 8.974 8.733 9.164 3 0 8192 4096 512 495.77 498.05 491.55 8.663 8.624 8.738 3 0 xdd - Version 4.2 - AHPCRC/UofMN Reading from target /vol1/tmp/BIGFILE1 alignment is on 4096 byte boundary Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time hi_time times errors 256 4096 16384 657.26 661.85 648.28 6.535 6.489 6.625 3 0 512 4096 8192 665.44 667.95 662.94 6.454 6.430 6.479 3 0 1024 4096 4096 664.09 667.62 660.69 6.467 6.433 6.501 3 0 2048 4096 2048 660.24 663.74 658.39 6.505 6.471 6.523 3 0 4096 4096 1024 659.22 661.27 657.62 6.515 6.495 6.531 3 0 8192 4096 512 659.04 660.13 658.03 6.517 6.506 6.527 3 0 ------------------------------------------------------------------------------------------------------------- XFS dio access ------------------------------------------------------------------------------------------------------------- xdd - Version 4.2 - AHPCRC/UofMN Writing to target /vol1/tmp/BIGFILE1 alignment is on 4096 byte boundary Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time hi_time times errors 256 4096 16384 173.02 174.56 170.90 24.824 24.605 25.131 3 0 512 4096 8192 233.86 235.84 231.36 18.365 18.211 18.564 3 0 1024 4096 4096 281.98 293.42 276.26 15.231 14.637 15.547 3 0 2048 4096 2048 323.80 344.48 314.07 13.264 12.468 13.675 3 0 4096 4096 1024 350.19 378.46 335.33 12.265 11.349 12.808 3 0 8192 4096 512 372.96 416.52 351.59 11.516 10.311 12.216 3 0 xdd - Version 4.2 - AHPCRC/UofMN Reading from target /vol1/tmp/BIGFILE1 alignment is on 4096 byte boundary Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time hi_time times errors 256 4096 16384 265.35 267.20 263.36 16.186 16.074 16.308 3 0 512 4096 8192 389.13 397.20 380.43 11.037 10.813 11.290 3 0 1024 4096 4096 479.39 482.50 477.37 8.959 8.902 8.997 3 0 2048 4096 2048 510.98 519.00 506.37 8.405 8.275 8.482 3 0 4096 4096 1024 544.88 551.91 531.71 7.882 7.782 8.078 3 0 8192 4096 512 576.87 577.16 576.63 7.445 7.442 7.448 3 0 ############################################################################################################# GLFS access ############################################################################################################# xdd - Version 4.2 - AHPCRC/UofMN Writing to target /mnt/gluster/tmp/gBIGFILE1 alignment is on 4096 byte boundary Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time hi_time times errors 256 4096 16384 354.10 358.66 345.77 12.129 11.975 12.421 3 0 512 4096 8192 343.38 351.11 334.04 12.508 12.232 12.858 3 0 1024 4096 4096 339.03 340.54 336.34 12.668 12.612 12.770 3 0 2048 4096 2048 336.89 346.33 331.90 12.749 12.401 12.941 3 0 4096 4096 1024 322.51 330.84 318.21 13.317 12.982 13.497 3 0 8192 4096 512 326.85 332.55 321.83 13.140 12.915 13.345 3 0 xdd - Version 4.2 - AHPCRC/UofMN Reading from target /mnt/gluster/tmp/gBIGFILE1 alignment is on 4096 byte boundary Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time hi_time times errors 256 4096 16384 504.89 507.85 502.00 8.507 8.457 8.556 3 0 512 4096 8192 496.10 500.03 492.83 8.658 8.589 8.715 3 0 1024 4096 4096 474.26 475.46 472.14 9.056 9.033 9.097 3 0 2048 4096 2048 447.06 447.78 446.29 9.607 9.592 9.624 3 0 4096 4096 1024 433.91 437.49 431.88 9.898 9.817 9.945 3 0 8192 4096 512 423.96 432.86 415.48 10.131 9.922 10.337 3 0 ------------------------------------------------------------------------------------------------------------- GLFS dio access ------------------------------------------------------------------------------------------------------------- xdd - Version 4.2 - AHPCRC/UofMN Writing to target /mnt/gluster/tmp/gBIGFILE1 alignment is on 4096 byte boundary Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time hi_time times errors 256 4096 16384 98.43 99.40 97.13 43.637 43.207 44.217 3 0 512 4096 8192 101.81 103.86 99.76 42.187 41.353 43.052 3 0 1024 4096 4096 101.08 102.48 98.96 42.491 41.909 43.402 3 0 2048 4096 2048 96.96 97.77 95.65 44.295 43.928 44.902 3 0 4096 4096 1024 99.76 103.59 97.83 43.052 41.462 43.903 3 0 8192 4096 512 100.31 103.76 97.21 42.818 41.395 44.185 3 0 xdd - Version 4.2 - AHPCRC/UofMN Reading from target /mnt/gluster/tmp/gBIGFILE1 alignment is on 4096 byte boundary Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time hi_time times errors 256 4096 16384 163.79 170.76 153.86 26.222 25.152 27.915 3 0 512 4096 8192 169.70 170.21 169.25 25.309 25.233 25.377 3 0 1024 4096 4096 169.49 171.35 166.58 25.340 25.066 25.783 3 0 2048 4096 2048 156.99 162.27 154.36 27.358 26.467 27.824 3 0 4096 4096 1024 163.43 164.24 162.67 26.280 26.150 26.403 3 0 8192 4096 512 79.28 117.87 50.34 54.178 36.438 85.323 3 0
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users