[PATCH RFC] dapltest: Add final send/recv sync for transaction tests.
The transaction tests need both sides to send a sync message after running
the test. This ensures that all remote operations are complete before
dapltest deregeisters memory and disconnects the endpoints.
Without this logic, we see async errors on iwarp devices because a read
or write arrives after the rmr has been destroyed. I believe this is
more likely to happen with iWARP than IB because iWARP completions only
indicate the local buffer can be reused. It doesn't imply that the
message has even arrived at the peer, let alone been placed in memory.
I've tested this on cxgb4 only so far.
Similar logic is probably needed for the performance tests as well.
Signed-off-by: Steve Wise sw...@opengridcomputing.com
---
test/dapltest/test/dapl_transaction_stats.c | 10 ++
test/dapltest/test/dapl_transaction_test.c | 139 +++
2 files changed, 149 insertions(+), 0 deletions(-)
diff --git a/test/dapltest/test/dapl_transaction_stats.c
b/test/dapltest/test/dapl_transaction_stats.c
index f9d6377..6422ee3 100644
--- a/test/dapltest/test/dapl_transaction_stats.c
+++ b/test/dapltest/test/dapl_transaction_stats.c
@@ -59,6 +59,16 @@ DT_transaction_stats_set_ready(DT_Tdep_Print_Head * phead,
DT_Mdep_Unlock(transaction_stats-lock);
}
+void
+DT_transaction_stats_reset_wait_count(DT_Tdep_Print_Head * phead,
+ Transaction_Stats_t * transaction_stats,
+ unsigned int num)
+{
+ DT_Mdep_Lock(transaction_stats-lock);
+ transaction_stats-wait_count = num;
+ DT_Mdep_Unlock(transaction_stats-lock);
+}
+
bool
DT_transaction_stats_wait_for_all(DT_Tdep_Print_Head * phead,
Transaction_Stats_t * transaction_stats)
diff --git a/test/dapltest/test/dapl_transaction_test.c
b/test/dapltest/test/dapl_transaction_test.c
index 779ea86..b94a2cc 100644
--- a/test/dapltest/test/dapl_transaction_test.c
+++ b/test/dapltest/test/dapl_transaction_test.c
@@ -1799,6 +1799,145 @@ DT_Transaction_Run(DT_Tdep_Print_Head * phead,
Transaction_Test_t * test_ptr)
} /* end loop for each op */
} /* end loop for iteration */
+ /*
+* Final sync up to ensure all previous remote operations have
+* finished.
+*
+* Without a final sync exchange, we see async errors on iwarp
+* devices because a read or write arrives after the rmr has
+* been destroyed. I believe this is more likely to happen with
+* iWARP than IB because iWARP completions only indicate the
+* local buffer can be reused. It doesn't imply that the
+* message has even arrived at the peer, let alone been placed in
memory.
+*/
+ if (test_ptr-is_server) {
+ /*
+* Server
+*/
+ DT_Tdep_PT_Debug(1,
+(phead,
+ Test[ F64x ]: Send Final Sync to Client\n,
+ test_ptr-base_port));
+ for (i = 0; i test_ptr-cmd-eps_per_thread; i++) {
+ if (!DT_post_recv_buffer(phead,
+
test_ptr-ep_context[i].ep_handle,
+test_ptr-ep_context[i].bp,
+SYNC_RECV_BUFFER_ID,
SYNC_BUFF_SIZE)) {
+ goto bail;
+ }
+ if (!DT_post_send_buffer(phead,
+test_ptr-ep_context[i].
+ep_handle,
+test_ptr-ep_context[i].bp,
+SYNC_SEND_BUFFER_ID,
+SYNC_BUFF_SIZE)) {
+ DT_Tdep_PT_Debug(1,
+(phead,
+ Test[ F64x
+ ]: Server final sync send
error\n,
+ test_ptr-base_port));
+ goto bail;
+ }
+ }
+ for (i = 0; i test_ptr-cmd-eps_per_thread; i++) {
+ DAT_DTO_COMPLETION_EVENT_DATA dto_stat;
+
+ if (!DT_dto_event_wait(phead,
+ test_ptr-ep_context[i].
+ reqt_evd_hdl, dto_stat)) {
+ DT_Tdep_PT_Debug(1,
+(phead,
+ Test[ F64x
+ ]: Server final sync send
error\n,
+
Re: Need help with strange mlx4_core error
2014-02-04 18:19 GMT+04:00 Vasiliy Tolstov v.tols...@selfip.ru: 2014-02-04 Vasiliy Tolstov v.tols...@selfip.ru: This is pretty much very old firmware, could you update it please with the latest GA from the mellanox site and see if things smile back? also send the lspci | grep -i Mellanox I'm update firmware to 2.7.200 (that is the latest from ftp supermicro). Now i'm try to test it. Sorry for bumping old thread. Now i again have this error: ibstat: CA 'mlx4_0' CA type: MT26428 Number of ports: 1 Firmware version: 2.7.5558 Hardware version: a0 Node GUID: 0x0030489dc9d8 System image GUID: 0x0030489dc9db Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 7 LMC: 0 SM lid: 6 Capability mask: 0x02510868 dmesg: [1067167.780266] mlx4_ib destroy_qp_common: modify QP 00110e to RESET failed. [1068708.243893] mlx4_ib destroy_qp_common: modify QP 001118 to RESET failed. [1174994.897394] mlx4_core :03:00.0: Internal error detected: [1174994.897484] mlx4_core :03:00.0: buf[00]: 001805a5 [1174994.897516] mlx4_core :03:00.0: buf[01]: [1174994.897544] mlx4_core :03:00.0: buf[02]: 200715b6 [1174994.897577] mlx4_core :03:00.0: buf[03]: [1174994.897607] mlx4_core :03:00.0: buf[04]: 0018050c [1174994.897637] mlx4_core :03:00.0: buf[05]: 0001 [1174994.897666] mlx4_core :03:00.0: buf[06]: 2fd8 [1174994.897697] mlx4_core :03:00.0: buf[07]: 0084 [1174994.897727] mlx4_core :03:00.0: buf[08]: db9f [1174994.897756] mlx4_core :03:00.0: buf[09]: 4000 [1174994.897786] mlx4_core :03:00.0: buf[0a]: [1174994.897815] mlx4_core :03:00.0: buf[0b]: [1174994.897845] mlx4_core :03:00.0: buf[0c]: [1174994.897874] mlx4_core :03:00.0: buf[0d]: [1174994.897904] mlx4_core :03:00.0: buf[0e]: [1174994.897934] mlx4_core :03:00.0: buf[0f]: [1174994.897963] mlx4_en :03:00.0: Internal error detected, restarting device [1175000.294830] mlx4_ib destroy_qp_common: modify QP 001eca to RESET failed. [1175000.352029] bonding: bond1: releasing active interface ib0 [1175000.352072] bonding: bond1: Warning: clearing HW address of bond1 while it still has VLANs. [1175000.352182] bonding: bond1: When re-adding slaves, make sure the bond's HW address matches its VLANs'. [1175000.352517] bonding: bond1: destroying bond bond1. [1175000.354034] bonding: bond1: released all slaves [1175000.354694] ib0: ib_cq_destroy (recv) failed [1175000.354721] ib0: ib_destroy_srq failed: -16 [1175000.355780] ib0: ib_dealloc_pd failed -- Vasiliy Tolstov, e-mail: v.tols...@selfip.ru jabber: v...@selfip.ru -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Need help with strange mlx4_core error
2014-02-28 22:24 GMT+04:00 Vasiliy Tolstov v.tols...@selfip.ru: This is pretty much very old firmware, could you update it please with the latest GA from the mellanox site and see if things smile back? also send the lspci | grep -i Mellanox And after looking dmesg i see sometime early: [271701.934646] device ib0 left promiscuous mode [292670.940271] [ cut here ] [292670.940309] WARNING: at /tmp/buildd/linux-3.10.11/net/sched/sch_generic.c:255 dev_watchdog+0xe3/0x153() [292670.940359] NETDEV WATCHDOG: ib0 (mlx4_core): transmit queue 0 timed out [292670.940388] Modules linked in: tcp_diag inet_diag ip_set_hash_net xt_set ip_set nfnetlink xt_tcpudp xt_pkttype ip6table_mangle ip6table_raw iptable_mangle iptable_raw ip6table_filter ip6_tables iptable_filter ip_tables 8021q garp stp mrp llc bonding ib_ucm ib_uverbs ib_addr ib_umad ib_ipoib ib_srp scsi_transport_srp ib_cm scsi_tgt mlx 4_ib ib_sa ib_mad ib_core mlx4_en ipmi_si ipmi_devintf ipmi_msghandler ipt_NETFLOW(O) x_tables dm_mod md_mod iTCO_wdt coretemp acpi_cpufreq i7core_edac snd_pcm mperf iTCO _vendor_support snd_page_alloc snd_timer edac_core snd soundcore psmouse kvm_intel i2c_i801 ioatdma pcspkr serio_raw kvm lpc_ich evdev joydev mfd_core crc32c_intel proces sor button thermal_sys squashfs loop aufs(C) hid_generic usbhid hid ata_generic ata_piix microcode uhci_hcd ehci_pci mpt2sas libata ehci_hcd raid_class scsi_transport_sas usbcore usb_common scsi_mod mlx4_core igb i2c_algo_bit i2c_core dca ptp pps_core [last unloaded: nf_conntrack] [292670.940991] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G CIO 3.10-3-amd64 #1 Debian 3.10.11-3+0~20131204153435.21+wheezy~1.gbpa177bd [292670.941045] Hardware name: Supermicro B8DT6/B8DT6, BIOS 080015 03/03/2010 [292670.941075] 8138d323 8103be31 8801bfc0dc00 [292670.941130] 8801bfc03e08 8801b8688000 0100 [292670.941185] 812e5e78 8801b8688348 8103bee1 8153628f [292670.941240] Call Trace: [292670.941262] IRQ [8138d323] ? dump_stack+0xd/0x17 [292670.941303] [8103be31] ? warn_slowpath_common+0x5f/0x77 [292670.941334] [812e5e78] ? netif_tx_lock+0x7a/0x7a [292670.941364] [8103bee1] ? warn_slowpath_fmt+0x45/0x4a [292670.941394] [812e5e65] ? netif_tx_lock+0x67/0x7a [292670.941426] [812e5f5b] ? dev_watchdog+0xe3/0x153 [292670.941458] [810471ee] ? call_timer_fn+0x4b/0xf6 [292670.941488] [812e5e78] ? netif_tx_lock+0x7a/0x7a [292670.941518] [810482fe] ? run_timer_softirq+0x18d/0x1d6 [292670.941548] [81042576] ? __do_softirq+0xec/0x209 [292670.941578] [8104275e] ? irq_exit+0x3f/0x83 [292670.941608] [8100e6f3] ? do_IRQ+0x81/0x97 [292670.941639] [81390a6d] ? common_interrupt+0x6d/0x6d [292670.941666] EOI [812a70ec] ? arch_local_irq_enable+0x4/0x8 [292670.941708] [812a74af] ? cpuidle_enter_state+0x46/0xb1 [292670.941739] [812a75f0] ? cpuidle_idle_call+0xd6/0x147 [292670.941770] [81013add] ? arch_cpu_idle+0x6/0x1a [292670.941802] [810734d3] ? cpu_startup_entry+0x114/0x191 [292670.941835] [816b3d51] ? start_kernel+0x3e8/0x3f3 [292670.941864] [816b377f] ? repair_env_string+0x57/0x57 [292670.941895] [816b359a] ? x86_64_start_kernel+0xf2/0xfd [292670.941924] ---[ end trace 119aa908393ed9b3 ]--- [292670.941950] ib0: transmit timeout: latency 1356 msecs [292670.941977] ib0: queue stopped 1, tx_head 595756, tx_tail 595756 -- Vasiliy Tolstov, e-mail: v.tols...@selfip.ru jabber: v...@selfip.ru -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH RFC] dapltest: Add final send/recv sync for transaction tests.
Hey Arlin,
This fix isn't quite correct: If one side finishes its transaction IO, then
posts the final sync SEND, but the peer hasn't finished with its transaction
IO, then the peer could post a READ request, for example, and then wait for the
READ completion. However the READ response would get sent behind the local
side's final sync SEND and wouldn't be placed because the peer's final sync
RECV hasn't been posted yet. Sigh. I'll need to think more on this, but I'd
like feedback if anybody has opinions/questions/comments...
Steve.
-Original Message-
From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-
ow...@vger.kernel.org] On Behalf Of Steve Wise
Sent: Friday, February 28, 2014 11:14 AM
To: arlin.r.da...@intel.com
Cc: linux-rdma@vger.kernel.org
Subject: [PATCH RFC] dapltest: Add final send/recv sync for transaction
tests.
The transaction tests need both sides to send a sync message after running
the test. This ensures that all remote operations are complete before
dapltest deregeisters memory and disconnects the endpoints.
Without this logic, we see async errors on iwarp devices because a read
or write arrives after the rmr has been destroyed. I believe this is
more likely to happen with iWARP than IB because iWARP completions only
indicate the local buffer can be reused. It doesn't imply that the
message has even arrived at the peer, let alone been placed in memory.
I've tested this on cxgb4 only so far.
Similar logic is probably needed for the performance tests as well.
Signed-off-by: Steve Wise sw...@opengridcomputing.com
---
test/dapltest/test/dapl_transaction_stats.c | 10 ++
test/dapltest/test/dapl_transaction_test.c | 139
+++
2 files changed, 149 insertions(+), 0 deletions(-)
diff --git a/test/dapltest/test/dapl_transaction_stats.c
b/test/dapltest/test/dapl_transaction_stats.c
index f9d6377..6422ee3 100644
--- a/test/dapltest/test/dapl_transaction_stats.c
+++ b/test/dapltest/test/dapl_transaction_stats.c
@@ -59,6 +59,16 @@
DT_transaction_stats_set_ready(DT_Tdep_Print_Head * phead,
DT_Mdep_Unlock(transaction_stats-lock);
}
+void
+DT_transaction_stats_reset_wait_count(DT_Tdep_Print_Head * phead,
+Transaction_Stats_t * transaction_stats,
+unsigned int num)
+{
+ DT_Mdep_Lock(transaction_stats-lock);
+ transaction_stats-wait_count = num;
+ DT_Mdep_Unlock(transaction_stats-lock);
+}
+
bool
DT_transaction_stats_wait_for_all(DT_Tdep_Print_Head * phead,
Transaction_Stats_t * transaction_stats)
diff --git a/test/dapltest/test/dapl_transaction_test.c
b/test/dapltest/test/dapl_transaction_test.c
index 779ea86..b94a2cc 100644
--- a/test/dapltest/test/dapl_transaction_test.c
+++ b/test/dapltest/test/dapl_transaction_test.c
@@ -1799,6 +1799,145 @@ DT_Transaction_Run(DT_Tdep_Print_Head *
phead, Transaction_Test_t * test_ptr)
} /* end loop for each op */
} /* end loop for iteration */
+ /*
+ * Final sync up to ensure all previous remote operations have
+ * finished.
+ *
+ * Without a final sync exchange, we see async errors on iwarp
+ * devices because a read or write arrives after the rmr has
+ * been destroyed. I believe this is more likely to happen with
+ * iWARP than IB because iWARP completions only indicate the
+ * local buffer can be reused. It doesn't imply that the
+ * message has even arrived at the peer, let alone been placed in
memory.
+ */
+ if (test_ptr-is_server) {
+ /*
+ * Server
+ */
+ DT_Tdep_PT_Debug(1,
+ (phead,
+ Test[ F64x ]: Send Final Sync to
Client\n,
+ test_ptr-base_port));
+ for (i = 0; i test_ptr-cmd-eps_per_thread; i++) {
+ if (!DT_post_recv_buffer(phead,
+ test_ptr-
ep_context[i].ep_handle,
+ test_ptr-
ep_context[i].bp,
+ SYNC_RECV_BUFFER_ID,
SYNC_BUFF_SIZE)) {
+ goto bail;
+ }
+ if (!DT_post_send_buffer(phead,
+ test_ptr-ep_context[i].
+ ep_handle,
+ test_ptr-
ep_context[i].bp,
+ SYNC_SEND_BUFFER_ID,
+ SYNC_BUFF_SIZE)) {
+ DT_Tdep_PT_Debug(1,
+ (phead,
+ Test[ F64x
+
Re: Proposal for simplifying NFS/RDMA client memory registration
On 2/26/2014 8:44 AM, Chuck Lever wrote: Hi- Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we’ve built and run functional tests to determine what is working and what is broken. One complication is the number of memory registration modes supported by the RPC/RDMA transport: there are seven. These were added over the years to support particular HCAs or as proof-of-concept. The transport chooses a registration mode at mount time based on what the link HCA supports. Not all HCAs support all memory registration modes, so our test matrix is quite large. I’d like to propose removing support for one or more of these memory registration modes in the name of making it easier to change this code and test it without breaking something that we can’t test. BOUNCEBUFFERS - All HCAs support this mode. Does not use RDMA READ and WRITE, and the client end copies data into place. RDMA is offloaded, by data copy is not. I’m told it was never intended for production use. REGISTER - Safe but relatively slow. Uses reg_phys_mr verb which is not supported in mlx4/mlx5, but all other HCAs/providers can use this mode. MEM_WINDOWS - Uses bind_mr verb. Safe, but supports only a narrow range of HCAs. MEM_WINDOWS_ASYNC - Not always safe, and only a narrow range of HCAs is supported. MTHCA_FMR - Uses alloc_fmr verb. Safe, reasonably fast, but only a narrow range of older HCAs is supported. The MTHCA FMR is not completely safe - it protects only on page boundaries, therefore the neighboring bytes are vulnerable to silent corruption (reads) and exposure (write). It is quite correct that they are supported on only a specific set of legacy Mellanox HCA. You should consider removing the code that looked for this PCI ID and attempted to alter the device's wire MTU, to overcome another of its limitations. FRMR - Safe, generally fast. Currently the preferred registration mode, but is not supported with some older HCAs/providers. This should be, by far, the preferred mode. Also, if I recall correctly, the server depends on this mode being available/supported. However, it may not be supported by Soft iWARP. Physical addressing is used. ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. Not safe is an understatement. It exposes all of client physical memory to the peer, for both read and write. A simple pointer error on the server will silently corrupt the client. This mode was intended only for testing, and in experimental deployments. Tom. I propose removing BOUNCEBUFFERS since it is not intended for production use. I propose removing ALLPHYSICAL and MEM_WINDOWS_ASYNC as they are not generally safe. RFC 5666 suggests that unsafe memory registration modes be avoided. I propose removing MEM_WINDOWS as it adds complexity without adding a lot of HCA compatibility. I propose removing MTHCA_FMR as I’m told it is hard to obtain HCAs we would need for testing this registration mode, and these are all old adapters anyway. This leaves NFS/RDMA client support for REGISTER and FRMR, which should cover all existing HCAs, and it is easy to test both of these memory registration modes with just one or two well-picked HCAs. We would contribute these changes to the client code base. The NFS/RDMA server code could use similar attention, but we are not volunteering to change it at this time. Thoughts/comments? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line unsubscribe linux-nfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Proposal for simplifying NFS/RDMA client memory registration
On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng s.wendy.ch...@gmail.com wrote: ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey t...@talpey.com wrote: On 2/26/2014 8:44 AM, Chuck Lever wrote: Hi- Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we've built and run functional tests to determine what is working and what is broken. [snip] ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. Not safe is an understatement. It exposes all of client physical memory to the peer, for both read and write. A simple pointer error on the server will silently corrupt the client. This mode was intended only for testing, and in experimental deployments. (sorry, resend .. previous reply bounced back due to gmail html format) Please keep ALLPHYSICAL for now - as our embedded system needs it. Thanks, Wendy -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
CONTACT KENYA COMMERCIAL BANK LTD IMMEDIATELY FOR YOUR PAYMENT (3MILLION UNITED STATE DOLLARS)
Attention: Please, Kindly contact KENYA COMMERCIAL BANK immediately for your compensation payment of (3MILLION UNITED STATE DOLLARS) via email: Email:(servic...@kenya.ncommercialbnk.com) quote your payment Ref No:KCB/00Y/2014 For purposes of immediate payment. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
contact me
contact me for details of $21.4m transfer. The information contained in this electronic communication is intended solely for the individual(s) or entity to which it is addressed. It may contain proprietary, confidential and/or legally privileged information. Any review, retransmission, dissemination, printing, copying or other use of, or taking any action in reliance on the contents of this information by person(s) or entities other than the intended recipient is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us by responding to this email or telephone and immediately and permanently delete all copies of this message and any attachments from your system(s). The contents of this message do not necessarily represent the views or policies of Aditya Birla Group. Computer viruses can be transmitted via email. Aditya Birla Group Companies attempts to sweep e-mails and attachments for viruses, it does not guarantee that either are virus free. The recipient should check this email and any attachments for the presence of viruses. Aditya Birla Group does not accept any liability for any damage sustained as a result of viruses. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Proposal for simplifying NFS/RDMA client memory registration
Hi Wendy- On Feb 28, 2014, at 5:26 PM, Wendy Cheng s.wendy.ch...@gmail.com wrote: On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng s.wendy.ch...@gmail.com wrote: ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey t...@talpey.com wrote: On 2/26/2014 8:44 AM, Chuck Lever wrote: Hi- Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we've built and run functional tests to determine what is working and what is broken. [snip] ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. Not safe is an understatement. It exposes all of client physical memory to the peer, for both read and write. A simple pointer error on the server will silently corrupt the client. This mode was intended only for testing, and in experimental deployments. (sorry, resend .. previous reply bounced back due to gmail html format) Please keep ALLPHYSICAL for now - as our embedded system needs it. This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
