Hi Rafiu Yes, I've just reproduced in on an Intel based system with a completely default 2.2 x86_64 with a conary updateall run. No bonding or additional drivers loaded.
Dave -----Original Message----- From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] Sent: Friday, 1 December 2006 1:33 p.m. To: Dave Watkins Cc: [email protected] Subject: Re: [OF-users] iSCSI bug? Dave Watkins wrote: > This _may_ have been a wild goose chase from the beginning. > Are you able to reproduce on a 64-bit system with different hardware? R. > > The OF box I was using had a temporary 3ware raid card (PCI-X) in it until the new 9650SE (PCI-E) cards became available. This has now happened and when this card arrived it seems there are PCI-E issues on the Tyan board we are using. When both PCI-E slots on the board are populated much worse things happen, such as lockups at POST or in the BIOS depending on which slots the cards are physically in. I'm beginning to suspect we were seeing this same issue when stressing the PCI-E NIC but not to the same degree since at the time PCI-E traffic was much less. Of note is that said lockups don't occur unless both cards are installed, the new 3ware or the Intel NIC by themsleves operate fine with the exception of seeing the error that started this thread so I don't think it's the new 3ware card causing the problem > > The only hole in the above theory is that even when I was using the on-board NIC's to test we were still seeing this and they are both connected to a PCI-X bus. > > In any case Tyan are aware of the issue now and I'm hoping to see a BIOS release to solve it in the near future > > ________________________________ > > From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] > Sent: Tue 11/28/2006 10:49 AM > To: Dave Watkins > Cc: [email protected] > Subject: Re: [OF-users] iSCSI bug? > > > > Dave Watkins wrote: > >> When you say "local box" do you mean the openfiler box >> > Yes please. > >> or the windows >> box? I have run local tests on the windows box without isse but haven't >> tried the openfiler box. I will try disabling jumbo frames, I have tried >> without NAPI and flow control with no success >> >> -----Original Message----- >> From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] >> Sent: Tuesday, 28 November 2006 10:27 a.m. >> To: Dave Watkins >> Cc: [email protected] >> Subject: Re: [OF-users] iSCSI bug? >> >> Dave Watkins wrote: >> >> >>> I'll start with a better description and get onto working with the new >>> packages >>> >>> I'm running iometer on the Windows Server 2003 x64 box against an >>> >>> >> iSCSI >> >> >>> volume using the MS iSCSI initiator (2.02). >>> >>> Using iometer and selecting any of the 0% read, 0% random access >>> specifications will return expected performance numbers, but stopping >>> that test, and changing the access specification to any 100% read, 0% >>> random test will generate the errors. Larger block sizes _seem_ to >>> >>> >> make >> >> >>> it happen more frequently so I have created a 256k block size test for >>> the above access specifications. I have been using 16 and 64 >>> >>> >> outstanding >> >> >>> I/O's but anything above zero seems to show the error. >>> >>> Networking on both ends is via Intel e1000 cards and jumbo frames are >>> enabled, and so is flow control, NAPI is also enabled on the Openfiler >>> box. All other network settings are default. >>> >>> >>> >> Try without all the tweaking and then enable them one by one. >> >> Also have you successfully run the benchmarks on the local box without >> going through iSCSI? >> >> >> R. >> >> >> >>> Dave >>> >>> -----Original Message----- >>> From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] >>> Sent: Tuesday, 28 November 2006 1:30 a.m. >>> To: Dave Watkins >>> Cc: [email protected] >>> Subject: Re: [OF-users] iSCSI bug? >>> >>> Dave Watkins wrote: >>> >>> >>> >>>> Still the same, both with bonding enabled and disabled unfortunatly >>>> >>>> >>>> >>>> >>> First: >>> >>> try adding "nosoftlockup" to the grub boot options and then run the >>> benchmarks again. >>> >>> Next: >>> >>> http://www.openfiler.com/download/PACKAGES/iscsi_trgt-kernel-r78.ccs >>> http://www.openfiler.com/download/PACKAGES/iscsi_trgt-r78.ccs >>> >>> (kernel and userland) >>> >>> same as before (--replace-files) >>> >>> Finally: >>> >>> Also a bit more detail about your test set-up (components, parameters, >>> >>> >> >> >>> triggers etc) would be great. >>> >>> R. >>> >>> >>> >>>> -----Original Message----- >>>> From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] >>>> Sent: Monday, 27 November 2006 2:52 p.m. >>>> To: Rafiu Fakunle >>>> Cc: Dave Watkins; [email protected] >>>> Subject: Re: [OF-users] iSCSI bug? >>>> >>>> Rafiu Fakunle wrote: >>>> >>>> >>>> >>>> >>>>> Dave Watkins wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Ok, UP is fine. To be sure it wasn't the e1000 driver I also tried >>>>>> >>>>>> >>>>>> >>>>>> >>>> using >>>> >>>> >>>> >>>> >>>>>> only the Broadcom NIC's as well. Under UP there is no error, under >>>>>> >>>>>> >>>>>> >>>>>> >>>> SMP >>>> >>>> >>>> >>>> >>>>>> the error reoccurs even with e1000 not loaded and no bonding. >>>>>> >>>>>> Hope this helps >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> Immensely. I'm just doing up a changeset for you. >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >> http://www.openfiler.com/download/PACKAGES/iscsi_trgt-kernel-0.4.14.ccs >> >> >>> >>> >>> >>>> conary update iscsi_trgt-kernel-0.4.14.ccs --replace-files >>>> >>>> Then test again with 2.6.17.14-0.3.smp.x86_64 (with and without >>>> >>>> >>>> >>> bonding) >>> >>> >>> >>>> Thx, >>>> >>>> R. >>>> >>>> >>>> >>>> >>>>> R. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Dave >>>>>> >>>>>> -----Original Message----- >>>>>> From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] Sent: Monday, 27 >>>>>> November 2006 1:15 p.m. >>>>>> To: Dave Watkins >>>>>> Cc: [email protected] >>>>>> Subject: Re: [OF-users] iSCSI bug? >>>>>> >>>>>> OK, and UP without trunking? >>>>>> >>>>>> R. >>>>>> >>>>>> Dave Watkins wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> With or without trunking seem to generate the same problem >>>>>>> >>>>>>> Without trunking I got >>>>>>> BUG: soft lockup detected on CPU#0! >>>>>>> >>>>>>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210} >>>>>>> <ffffffff80289151>{update_process_times+66} >>>>>>> <ffffffff802713fe>{smp_local_timer_interrupt+35} >>>>>>> <ffffffff80271463>{smp_apic_timer_interrupt+65} >>>>>>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI> >>>>>>> <ffffffff80224b87>{tcp_sendmsg+0} >>>>>>> <ffffffff80413bba>{inet_ioctl+0} >>>>>>> <ffffffff88141216>{:iscsi_trgt:is_data_available+62} >>>>>>> <ffffffff881419e7>{:iscsi_trgt:istd+1460} >>>>>>> <ffffffff80403ea6>{tcp_sendpage+0} >>>>>>> <ffffffff8027fef6>{__wake_up_common+67} >>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>> <ffffffff88141433>{:iscsi_trgt:istd+0} >>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>> <ffffffff80231a7d>{kthread+200} >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>> <ffffffff8025f8a2>{child_rip+8} >>>> >>>> >>>> >>>> >>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>>>>>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>>>>>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>>>>>> <ffffffff802319b5>{kthread+0} >>>>>>> >>>>>>> >>>>>>> >>> <ffffffff8025f89a>{child_rip+0} >>> >>> >>> >>>>>>> BUG: soft lockup detected on CPU#0! >>>>>>> >>>>>>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210} >>>>>>> <ffffffff80289151>{update_process_times+66} >>>>>>> <ffffffff802713fe>{smp_local_timer_interrupt+35} >>>>>>> <ffffffff80271463>{smp_apic_timer_interrupt+65} >>>>>>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI> >>>>>>> <ffffffff80224b87>{tcp_sendmsg+0} >>>>>>> <ffffffff881411c0>{:iscsi_trgt:nthread_wakeup+35} >>>>>>> <ffffffff881411b3>{:iscsi_trgt:nthread_wakeup+22} >>>>>>> <ffffffff8814219a>{:iscsi_trgt:istd+3431} >>>>>>> <ffffffff80403ea6>{tcp_sendpage+0} >>>>>>> <ffffffff8027fef6>{__wake_up_common+67} >>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>> <ffffffff88141433>{:iscsi_trgt:istd+0} >>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>> <ffffffff80231a7d>{kthread+200} >>>>>>> <ffffffff8025f8a2>{child_rip+8} >>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>>>>>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>>>>>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>>>>>> <ffffffff802319b5>{kthread+0} >>>>>>> <ffffffff8025f89a>{child_rip+0} >>>>>>> >>>>>>> Re-enabling trunking again and I get >>>>>>> BUG: soft lockup detected on CPU#0! >>>>>>> >>>>>>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210} >>>>>>> <ffffffff80289151>{update_process_times+66} >>>>>>> <ffffffff802713fe>{smp_local_timer_interrupt+35} >>>>>>> <ffffffff80271463>{smp_apic_timer_interrupt+65} >>>>>>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI> >>>>>>> <ffffffff80254356>{tcp_ioctl+0} >>>>>>> <ffffffff8020af50>{__might_sleep+30} >>>>>>> <ffffffff802326d7>{lock_sock+28} >>>>>>> <ffffffff80263257>{_spin_lock_bh+9} >>>>>>> <ffffffff8022fd23>{release_sock+15} >>>>>>> <ffffffff802543a2>{tcp_ioctl+76} >>>>>>> <ffffffff80413c44>{inet_ioctl+138} >>>>>>> <ffffffff88141216>{:iscsi_trgt:is_data_available+62} >>>>>>> <ffffffff8814125a>{:iscsi_trgt:do_recv+41} >>>>>>> <ffffffff8023081f>{qdisc_restart+24} >>>>>>> <ffffffff8022eaa6>{dev_queue_xmit+510} >>>>>>> <ffffffff8807c266>{:bonding:bond_dev_queue_xmit+489} >>>>>>> <ffffffff8023277e>{lock_sock+195} >>>>>>> <ffffffff8807fd96>{:bonding:bond_xmit_roundrobin+154} >>>>>>> <ffffffff80232136>{__tcp_push_pending_frames+1367} >>>>>>> <ffffffff8022fd23>{release_sock+15} >>>>>>> <ffffffff80225551>{tcp_sendmsg+2506} >>>>>>> <ffffffff80236f84>{do_sock_write+199} >>>>>>> <ffffffff803dbac1>{sock_writev+220} >>>>>>> <ffffffff8025db21>{cache_alloc_refill+237} >>>>>>> <ffffffff80220d80>{tcp_transmit_skb+1579} >>>>>>> <ffffffff80408067>{tcp_retransmit_skb+1352} >>>>>>> <ffffffff80254356>{tcp_ioctl+0} >>>>>>> <ffffffff8024f5a4>{finish_wait+52} >>>>>>> <ffffffff803e0d10>{sk_stream_wait_memory+458} >>>>>>> <ffffffff80291608>{autoremove_wake_function+0} >>>>>>> <ffffffff80291608>{autoremove_wake_function+0} >>>>>>> <ffffffff8022fd23>{release_sock+15} >>>>>>> <ffffffff80246a25>{try_to_wake_up+955} >>>>>>> <ffffffff88141609>{:iscsi_trgt:istd+470} >>>>>>> <ffffffff80403ea6>{tcp_sendpage+0} >>>>>>> <ffffffff8027fef6>{__wake_up_common+67} >>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>> <ffffffff88141433>{:iscsi_trgt:istd+0} >>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>> <ffffffff80231a7d>{kthread+200} >>>>>>> <ffffffff8025f8a2>{child_rip+8} >>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>>>>>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>>>>>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>>>>>> <ffffffff802319b5>{kthread+0} >>>>>>> <ffffffff8025f89a>{child_rip+0} >>>>>>> >>>>>>> Without trunking though the write performance after this doesn't >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>> seem >>>> >>>> >>>> >>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> to >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> be affected (still at about 80-90MB rather than down at less than >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> 10MB) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] Sent: Monday, 27 >>>>>>> November 2006 12:27 p.m. >>>>>>> To: Dave Watkins >>>>>>> Cc: [email protected] >>>>>>> Subject: Re: [OF-users] iSCSI bug? >>>>>>> >>>>>>> Dave Watkins wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Sorry about that, I remembered as soon as I sent it that I hadn't >>>>>>>> included version. It's x86_64 version 2.2 (did a conary updateall >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> from >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>> 2.1 beta. Uname -r gives 2.6.17.14-0.3.smp.x86_64. >>>>>>>> >>>>>>>> I'll try with a UP kernel although it will take some time as I >>>>>>>> >>>>>>>> >>>>>>>> >>> have >>> >>> >>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> to >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>> rebuild the e1000 module from the UP kernel sources. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Try without the network trunking anyway in the meantime. Would be >>>>>>> >>>>>>> >>>>>>> >>> an >>> >>> >>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>> >>>> >>>> >>>> >>>>>>> interesting test. >>>>>>> >>>>>>> R. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> I'll let you know >>>>>>>> if I can reproduce on the UP kernel. >>>>>>>> >>>>>>>> I don't think it's related to that ticket as they are all writes >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> anyway >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> and they only see the problem on large files. >>>>>>>> >>>>>>>> Dave >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] Sent: Monday, 27 >>>>>>>> >>>>>>>> >> >> >>>>>>>> November 2006 11:40 a.m. >>>>>>>> To: Dave Watkins >>>>>>>> Cc: [email protected] >>>>>>>> Subject: Re: [OF-users] iSCSI bug? >>>>>>>> >>>>>>>> Hi Dave, >>>>>>>> >>>>>>>> Excellent test and bug report. >>>>>>>> >>>>>>>> I wonder whether it may be related to this: >>>>>>>> >>>>>>>> https://project.openfiler.com/tracker/ticket/435 >>>>>>>> >>>>>>>> Can you try to reproduce with a UP kernel pls. >>>>>>>> >>>>>>>> Also I need the output of `uname -r` >>>>>>>> >>>>>>>> Thx, >>>>>>>> >>>>>>>> R. >>>>>>>> >>>>>>>> FTR: this is running r58 from IET svn >>>>>>>> >>>>>>>> >>>>>>>> Dave Watkins wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Hi All >>>>>>>>> >>>>>>>>> I think I've found a bug in the iscsi target software in my >>>>>>>>> benchmarking/testing. >>>>>>>>> >>>>>>>>> Some background on the hardware first in case it may be related. >>>>>>>>> Dual core/dual opteron with 2GB of ram >>>>>>>>> 3ware 8006 2 port raid card for OS drives >>>>>>>>> 3ware 9550SX card for data drives >>>>>>>>> Dual GB Broadcom on-board NIC's teamed into bond0 (management) >>>>>>>>> Quad port Intel PCI-E GB NIC with all 4 ports teamed into bond1 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>> (main >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>> iscsi data network) >>>>>>>>> 4 x 250GB WD SATA HDD's in RAID5 >>>>>>>>> >>>>>>>>> Of note here is that I have had to replace the e1000 driver with >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>> the >>>> >>>> >>>> >>>> >>>>>>>>> latest from Intel to support the quad port card >>>>>>>>> >>>>>>>>> I have made some volumes and mounted them on various windows >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>> servers >>>> >>>> >>>> >>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> and >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> have been using iobench to tune performance of the system. When >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>> using >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> a >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> read only test pattern I see this >>>>>>>>> >>>>>>>>> BUG: soft lockup detected on CPU#0! >>>>>>>>> >>>>>>>>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210} >>>>>>>>> <ffffffff80289151>{update_process_times+66} >>>>>>>>> <ffffffff802713fe>{smp_local_timer_interrupt+35} >>>>>>>>> <ffffffff80271463>{smp_apic_timer_interrupt+65} >>>>>>>>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI> >>>>>>>>> <ffffffff88141486>{:iscsi_trgt:istd+83} >>>>>>>>> <ffffffff88141476>{:iscsi_trgt:istd+67} >>>>>>>>> <ffffffff80403ea6>{tcp_sendpage+0} >>>>>>>>> <ffffffff8027fef6>{__wake_up_common+67} >>>>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>>>> <ffffffff88141433>{:iscsi_trgt:istd+0} >>>>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>>>> <ffffffff80231a7d>{kthread+200} >>>>>>>>> <ffffffff8025f8a2>{child_rip+8} >>>>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>>>> <ffffffff802319b5>{kthread+0} >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>> <ffffffff8025f89a>{child_rip+0} >>>> >>>> >>>> >>>> >>>>>>>>> BUG: soft lockup detected on CPU#0! >>>>>>>>> >>>>>>>>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210} >>>>>>>>> <ffffffff80289151>{update_process_times+66} >>>>>>>>> <ffffffff802713fe>{smp_local_timer_interrupt+35} >>>>>>>>> <ffffffff80271463>{smp_apic_timer_interrupt+65} >>>>>>>>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI> >>>>>>>>> <ffffffff802631ec>{_spin_unlock_irqrestore+8} >>>>>>>>> <ffffffff80246a25>{try_to_wake_up+955} >>>>>>>>> <ffffffff881411cc>{:iscsi_trgt:nthread_wakeup+47} >>>>>>>>> <ffffffff8814219a>{:iscsi_trgt:istd+3431} >>>>>>>>> <ffffffff80403ea6>{tcp_sendpage+0} >>>>>>>>> <ffffffff8027fef6>{__wake_up_common+67} >>>>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>>>> <ffffffff88141433>{:iscsi_trgt:istd+0} >>>>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>>>> <ffffffff80231a7d>{kthread+200} >>>>>>>>> <ffffffff8025f8a2>{child_rip+8} >>>>>>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>>>>>> <ffffffff802319b5>{kthread+0} >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>> <ffffffff8025f89a>{child_rip+0} >>>> >>>> >>>> >>>> >>>>>>>>> Doing write only based patterns this doesn't come up. After this >>>>>>>>> performance of the system dives (from about 110MB/sec of iscsi >>>>>>>>> performance to about 10MB/sec). >>>>>>>>> >>>>>>>>> This is fairly reproducible here so if you need anymore >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>> information >>>> >>>> >>>> >>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> just >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> ask. >>>>>>>>> >>>>>>>>> Dave >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >> ------------------------------------------------------------------------ >> >> >>> >>> >>> >>>> >>>> >>>> >>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Openfiler-users mailing list >>>>>>>>> [email protected] >>>>>>>>> https://lists.openfiler.com/mailman/listinfo/openfiler-users >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Openfiler-users mailing list >>>>> [email protected] >>>>> https://lists.openfiler.com/mailman/listinfo/openfiler-users >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> >> >> > > > > > _______________________________________________ Openfiler-users mailing list [email protected] https://lists.openfiler.com/mailman/listinfo/openfiler-users
