Yeah I noticed that struct fc_frame just contains an skbuff. But I saw calls to fc_frame_free in certain cases and direct calls to kfree_skb in other places instead. I may play with adding a count within kfree_skb but I'll have to find a way to limit it to only the Niantic port and only the FCoE frames, if possible.
Yes, I see the difference in my alloc/free numbers between dd with and without direct I/O. This would lead me to think the issue is caching, as you suggested, but the cache numbers to not increase. I have tried dd with block sizes of 512, 4M, 16M. We did write tests with other storage mediums with 4M and 16M block sizes among others. Some attempts went through cleanly, others caused "Buffer I/O errors" and required that the fcoe service was restarted. I just did a couple runs with bs=4K count=10000. It looks like the freed number is half of the allocated number. As Vasu mentioned, I guess they are not supposed to be the same. Output below... ~# dd if=/dev/zero of=/dev/sde1 bs=4K count=10000 oflag=dire fcoe_start_io: Frame #10000 (rc = 0) fc_frame_alloc: allocated 10000 frames fcoe_start_io: Frame #20000 (rc = 0) fc_frame_alloc: allocated 20000 frames fc_frame_free: freed 10000 frames fcoe_start_io: Frame #30000 (rc = 0) fc_frame_alloc: allocated 30000 frames fcoe_start_io: Frame #40000 (rc = 0) fc_frame_alloc: allocated 40000 frames fc_frame_free: freed 20000 frames fcoe_start_io: Frame #50000 (rc = 0) fc_frame_alloc: allocated 50000 frames 10000+0 records in 10000+0 records out 40960000 bytes (41 MB) copied, 8.05651 seconds, 5.1 MB/s [email protected]:~# dd if=/dev/zero of=/dev/sde1 bs=4K count=10000 oflag=dire fcoe_start_io: Frame #60000 (rc = 0) fc_frame_alloc: allocated 60000 frames fcoe_start_io: Frame #70000 (rc = 0) fc_frame_alloc: allocated 70000 frames fc_frame_free: freed 30000 frames fcoe_start_io: Frame #80000 (rc = 0) fc_frame_alloc: allocated 80000 frames fcoe_start_io: Frame #90000 (rc = 0) fc_frame_alloc: allocated 90000 frames fc_frame_free: freed 40000 frames fcoe_start_io: Frame #100000 (rc = 0) fcoe_xmit: TxFrames = 100000 fc_frame_alloc: allocated 100000 frames 10000+0 records in 10000+0 records out 40960000 bytes (41 MB) copied, 6.08639 seconds, 6.7 MB/s Thanks, Tristan -----Original Message----- From: Joe Eykholt (jeykholt) Sent: Tuesday, July 27, 2010 5:49 PM To: Tristan Cheever (tcheever) Cc: [email protected]; Walter Song (wasong) Subject: Re: [Open-FCoE] Huge memory usage with mkfs.ext3 over FCoE On 7/27/10 5:28 PM, Tristan Cheever (tcheever) wrote: > One more note... > > I added printk to fc_frame_alloc and fc_frame_free to show the number of > frames allocated vs freed. > > Without Direct I/O: Number of allocated frames is much greater than > number freed. > With Direct I/O: They are identical > > When going through the code, it seemed as though not all frames are > freed via fc_frame_free. The fc_frames are actually sk_buffs, so when frames are sent out the ethernet they'll be freed by kfree_skb() or something like that. Frames received by Ethernet are usually converted into fc_frames and freed by fc_frame_free(). Maybe we should call them sk_buffs everywhere, but we don't. One could add a destructor function to the sk_buffs to count them being freed. Are you doing dd with and without iflag=direct and noticing that difference? I wonder if the difference might be the I/O size. With direct I/O are you doing 512-byte reads or writes? Maybe you could try 4K or larger and see if they're still different. Not sure what's going on here. Joe _______________________________________________ devel mailing list [email protected] http://www.open-fcoe.org/mailman/listinfo/devel
