Hello,
I did simple tryout for GLFS 2.0rc8 on CentOS Linux with 2x Dual Core
Xeon 4GB RAM.
My benchmarks, it is high load single stream access on loopback glfs
mount on a single server with high performance FC RAID.
Target volume is XFS formatted. Local benchmark results are as
follows.(benchmark command is XDD command)
Buf I/O
READ = about 660MB/s
WRITE = about 480MB/s
Direct I/O
4MB block read = about 540MB/s
4MB block write = about 350MB/s
The results for GLFS loopback mount volume are as follows.
Buf I/O
READ = about 460MB/s
WRITE = about 330MB/s
Direct I/O
4MB block read = about 160MB/s
4MB block write = about 100MB/s
Buf I/O with GLFS is good results with small block size.
But The large block size access is slows down.
Direct I/O is poor performance without follow the block size.
Attached please find a detailed information text.
I want to use glfs with professional video applications on IB networks.
Video applications are using storages with large uncompress image
sequences and/or uncompress large movie files.(up to 2K/4K)
The block size control and direct I/O performance are important for them.
Please advise me about options/configurations for improve the
performance, and theory for improve performance by block size on GLFS.
My best regards,
hideo
#############################################################################################################
GLUSTERFS server config
#############################################################################################################
file : /etc/glusterfs/glusterfsd.vol
volume posix
type storage/posix
option directory /vol1
end-volume
volume brick
type performance/io-threads
option thread-count 8
subvolumes posix
end-volume
### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp
#option transport-type ib-sdp
option client-volume-filename /etc/glusterfs/glusterfs-client.vol
option auth.addr.brick.allow * # Allow access to "brick" volume
subvolumes brick
end-volume
#############################################################################################################
GLUSTERFS client config
#############################################################################################################
file : /etc/glusterfs/glusterfsd.vol
volume client
type protocol/client
option transport-type tcp
#option transport-type ib-sdp
option remote-host 192.168.2.99 # IP address of the remote brick
#option remote-host 10.10.11.99 # IP address of the remote brick
option remote-subvolume brick # name of the remote volume
end-volume
volume iot
type performance/io-threads
option thread-count 4
subvolumes client
end-volume
#volume writebehind
# type performance/write-behind
# option aggregate-size 1MB
# option window-size 4MB
# subvolumes iot
#end-volume
#volume cache
# type performance/io-cache
# option cache-size 512MB
# subvolumes writebehind
#end-volume
#############################################################################################################
GLUSTERFS Processes
#############################################################################################################
root 14188 1 4 14:21 ? 00:05:39 /usr/sbin/glusterfsd -f
/etc/glusterfs/glusterfsd.vol
root 14234 1 4 14:21 ? 00:05:26 /usr/sbin/glusterfs
--log-level=NORMAL --volfile=/etc/glusterfs/glusterfs-client.vol /mnt/gluster/
#############################################################################################################
XFS access
#############################################################################################################
xdd - Version 4.2 - AHPCRC/UofMN
Writing to target /vol1/tmp/BIGFILE1 alignment is on 4096 byte boundary
Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time
hi_time times errors
256 4096 16384 492.95 497.79 486.08 8.713 8.628
8.836 3 0
512 4096 8192 462.39 466.97 456.28 9.289 9.197
9.413 3 0
1024 4096 4096 474.29 484.45 469.01 9.056 8.866
9.158 3 0
2048 4096 2048 493.54 497.72 488.56 8.702 8.629
8.791 3 0
4096 4096 1024 478.62 491.81 468.66 8.974 8.733
9.164 3 0
8192 4096 512 495.77 498.05 491.55 8.663 8.624
8.738 3 0
xdd - Version 4.2 - AHPCRC/UofMN
Reading from target /vol1/tmp/BIGFILE1 alignment is on 4096 byte boundary
Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time
hi_time times errors
256 4096 16384 657.26 661.85 648.28 6.535 6.489
6.625 3 0
512 4096 8192 665.44 667.95 662.94 6.454 6.430
6.479 3 0
1024 4096 4096 664.09 667.62 660.69 6.467 6.433
6.501 3 0
2048 4096 2048 660.24 663.74 658.39 6.505 6.471
6.523 3 0
4096 4096 1024 659.22 661.27 657.62 6.515 6.495
6.531 3 0
8192 4096 512 659.04 660.13 658.03 6.517 6.506
6.527 3 0
-------------------------------------------------------------------------------------------------------------
XFS dio access
-------------------------------------------------------------------------------------------------------------
xdd - Version 4.2 - AHPCRC/UofMN
Writing to target /vol1/tmp/BIGFILE1 alignment is on 4096 byte boundary
Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time
hi_time times errors
256 4096 16384 173.02 174.56 170.90 24.824 24.605
25.131 3 0
512 4096 8192 233.86 235.84 231.36 18.365 18.211
18.564 3 0
1024 4096 4096 281.98 293.42 276.26 15.231 14.637
15.547 3 0
2048 4096 2048 323.80 344.48 314.07 13.264 12.468
13.675 3 0
4096 4096 1024 350.19 378.46 335.33 12.265 11.349
12.808 3 0
8192 4096 512 372.96 416.52 351.59 11.516 10.311
12.216 3 0
xdd - Version 4.2 - AHPCRC/UofMN
Reading from target /vol1/tmp/BIGFILE1 alignment is on 4096 byte boundary
Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time
hi_time times errors
256 4096 16384 265.35 267.20 263.36 16.186 16.074
16.308 3 0
512 4096 8192 389.13 397.20 380.43 11.037 10.813
11.290 3 0
1024 4096 4096 479.39 482.50 477.37 8.959 8.902
8.997 3 0
2048 4096 2048 510.98 519.00 506.37 8.405 8.275
8.482 3 0
4096 4096 1024 544.88 551.91 531.71 7.882 7.782
8.078 3 0
8192 4096 512 576.87 577.16 576.63 7.445 7.442
7.448 3 0
#############################################################################################################
GLFS access
#############################################################################################################
xdd - Version 4.2 - AHPCRC/UofMN
Writing to target /mnt/gluster/tmp/gBIGFILE1 alignment is on 4096 byte boundary
Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time
hi_time times errors
256 4096 16384 354.10 358.66 345.77 12.129 11.975
12.421 3 0
512 4096 8192 343.38 351.11 334.04 12.508 12.232
12.858 3 0
1024 4096 4096 339.03 340.54 336.34 12.668 12.612
12.770 3 0
2048 4096 2048 336.89 346.33 331.90 12.749 12.401
12.941 3 0
4096 4096 1024 322.51 330.84 318.21 13.317 12.982
13.497 3 0
8192 4096 512 326.85 332.55 321.83 13.140 12.915
13.345 3 0
xdd - Version 4.2 - AHPCRC/UofMN
Reading from target /mnt/gluster/tmp/gBIGFILE1 alignment is on 4096 byte
boundary
Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time
hi_time times errors
256 4096 16384 504.89 507.85 502.00 8.507 8.457
8.556 3 0
512 4096 8192 496.10 500.03 492.83 8.658 8.589
8.715 3 0
1024 4096 4096 474.26 475.46 472.14 9.056 9.033
9.097 3 0
2048 4096 2048 447.06 447.78 446.29 9.607 9.592
9.624 3 0
4096 4096 1024 433.91 437.49 431.88 9.898 9.817
9.945 3 0
8192 4096 512 423.96 432.86 415.48 10.131 9.922
10.337 3 0
-------------------------------------------------------------------------------------------------------------
GLFS dio access
-------------------------------------------------------------------------------------------------------------
xdd - Version 4.2 - AHPCRC/UofMN
Writing to target /mnt/gluster/tmp/gBIGFILE1 alignment is on 4096 byte boundary
Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time
hi_time times errors
256 4096 16384 98.43 99.40 97.13 43.637 43.207
44.217 3 0
512 4096 8192 101.81 103.86 99.76 42.187 41.353
43.052 3 0
1024 4096 4096 101.08 102.48 98.96 42.491 41.909
43.402 3 0
2048 4096 2048 96.96 97.77 95.65 44.295 43.928
44.902 3 0
4096 4096 1024 99.76 103.59 97.83 43.052 41.462
43.903 3 0
8192 4096 512 100.31 103.76 97.21 42.818 41.395
44.185 3 0
xdd - Version 4.2 - AHPCRC/UofMN
Reading from target /mnt/gluster/tmp/gBIGFILE1 alignment is on 4096 byte
boundary
Reqsize mbytes #ops Avg_Rate Hi_rate Low_rate elapsed low_time
hi_time times errors
256 4096 16384 163.79 170.76 153.86 26.222 25.152
27.915 3 0
512 4096 8192 169.70 170.21 169.25 25.309 25.233
25.377 3 0
1024 4096 4096 169.49 171.35 166.58 25.340 25.066
25.783 3 0
2048 4096 2048 156.99 162.27 154.36 27.358 26.467
27.824 3 0
4096 4096 1024 163.43 164.24 162.67 26.280 26.150
26.403 3 0
8192 4096 512 79.28 117.87 50.34 54.178 36.438
85.323 3 0
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
