Just a quick update on where I got to regarding this.
Spent quite a few days & nights playing and was finally able to get
the following results (from 140 tests of bonnie++):
Block Write:
Min 93.35 MB/s
Avg 98.57 MB/s
Max 103.22 MB/s
Block Read:
Min 127.79 MB/s
Avg 136.45 MB/s
Max 142.27 MB/s
This is for the xen linux client reading and writing to a reiserfs
file-system on top of LVM on top of SW RAID1 across two iSCSI exported
drives that exist as an LV on top of SW RAID0 on the targets. One
target being on the Xen host machine the other over the real network.
After reading the AoE alignment paper referenced from this mailing
list - I was a little confused as to where the realignment took place.
I've assumed that you change these values on the exported drive and
not on the source drive. I did play with changing the heads/sectors
value on the exported drives. When I did this - if I created a
partition with fdisk I got significantly slower writes and reads. If
I changed the heads/sectors value but didn't create a partition and
just had SW RAID use the whole disk I received a 17% improvement in
write speeds as per the RAID rebuild speed provided by "/proc/mdstat".
The file-system tests by dd also showed a 13% improvement but the raw
LV read test suffered a 22% decrease.
The read and write values are the average from 10 runs of dd whereas
the RAID1 check value is the average value from /proc/mdstat during
one rebuild session. This is all from the point of the xen client
only.
Without Alignment
With Alignment on exported target
Read /dev/sda 143
145 MB/s 1.38 % Inc
Read /dev/sdb 103
107 MB/s 3.74 % Inc
Read /dev/md0 106
106 MB/s 0.00 % No Change
Read /dev/vg_md0/iscsi_test 145 113
MB/s 22.07 % Dec
Write /mnt/iscsi_test/iscsi_test.raw 106 106
MB/s 0.00 % No Change
Read /mnt/iscsi_test/iscsi_test.raw 92.9 107
MB/s 13.18 % Inc
RAID1 Check/Rebuild 63
76 MB/s 17.11 % Inc
Some observations:
* When running iSCSI across a xen virtual bridged network you can
almost get raw disk speed. (sda is exported from the xen host and
147MB/s is the raw speed for the test within the host on the raw
disk).
* Although some layers might be low (e.g. md0=106MB/s) the actual
workable file-system speed might be quite reasonable and faster than
expected (e.g. iscsi_test_raw =107MB/s)
* Therefore, it seems quite pointless pontificating over speeds of
various layers - only the file-system layer counts.
* By extension you only need to optimise a couple key layers to
improve the results - not all layers. For example - I did try using
blockdev --setra 65536 on all layers on both target and initiator and
the speeds dropped through the floor. Exactly which layers should
have blockdev optimised is yet to be determined. It appears at the
moment that on the target the raw disks need blockdev as well as the
iSCSI exported disks on the initiator. It also appears that blockdev
on md0 seems to be of some help but is a bit iffy - can either
drastically improve the speed or crash the speed through the floor
depending on value. The LVM auto read-ahead may also give better
results than a blanket blockdev in some cases.
* That changing the heads/sector values for the exported disk does
improve speed. Just don't use a partition as it crashes the speed
once the geometry change has been made - use the complete disk - at
least for SW RAID.
* On write - the network is 99% utilised (as per nettop) but on reads
even for the speeds above the network is only ~80% utilised. I'm not
sure why the big difference. Maybe I have to change the target from
IET to something else or maybe I have to tweek the network
optimisation settings. But why it is better in one direction than the
other given that most things are equal are a mystery. I'm now also
looking at what xen interactions there might be.
Areas for further effort:
* The raw disks are 512 sectors and are using either 4K or 128K SW
RAID0 chunks on the targets***. The xen client RAID1 of the exported
disks are using chunk sizes of 32K. It would be interesting to see
what aligning all these chunk values would do for performance. Even
aligning the disk chuck values against the network MTU.
* To use the LVM dm-mirror instead of the SW RAID1 code to see what
performance that might have on the xen client across the iSCSI
exported drives.
* Try to find out which blockdev values to use on the different layers
or which layers to optimise and which not to get the best file-system
speeds. Early tests have resulted in 110MB/s or better reads under
some situations - but the results are currently not consistent.
* Try to optimise the network further to try to get the full 117MB/s
for a /dev/sdb read. I can reach this speed using dd > netcat >
netcat > dd so it should be possible with iSCSI as the PCI bus
doesn't seem to be a limiting factor.
* Work out why the big difference in network utilisation between read
and write. Look into what affect xen might play with this.
Using RAID1 on exported iSCSI disks at least looks like a reasonable
solution for redundancy. I've already encountered a situation where
an eSATA cable was dislodged from the xen host and the client keep
running uninterrupted using the exported iSCSI across the real network
without any noticeable effects. In fact I didn't notice for a couple
of days. And re-adding was easy and trouble-free and almost as fast
as it would have been given a raw physical disk I might even look at
this as a solution for backups in the future (having RAID1 across 3 or
o more exported disks - to take a backup - break the mirror).
Thanks for people's advice and suggestions.
Adrian.
***Why the difference in SW RAID0 chunk sizes between targets?: I've
got a bug in my xen kernel where if I use SW RAID0 chuck sizes smaller
than 128K on the xen host - I get file-system corruption on the xen
clients. Not sure why as I have not found anyone willing to help me
drill down to find out why. Ext2/3 is worse than reiserfs. Reiserfs
has survived for ~6months whereas ext3 will kill everything in
seconds.***
On 11/22/08, Bart Van Assche <[EMAIL PROTECTED]> wrote:
> On Sat, Nov 22, 2008 at 10:58 AM, Tracy Reed <[EMAIL PROTECTED]> wrote:
>> On Sat, Nov 22, 2008 at 09:44:09AM +0100, Bart Van Assche spake thusly:
>>> Maybe not the advice you are looking for, but did you already have a
>>> look at the SCST iSCSI target implementation ? It's faster than IET
>>> and better maintained. There is also a section about how to tune
>>
>> Does it do MC/S or Error Recovery Level 2 or 3?
>
> Not that I know of. Would you like to see these features implemented in SCST
> ?
>
> (CC'd Vladislav Bolkhovitin, SCST maintainer)
>
> Bart.
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~----------~----~----~----~------~----~------~--~---
