Hi Tristan, Let's bottom-post: that's the convention on this alias. My comments are in-line.
On 7/27/10 6:26 PM, Tristan Cheever (tcheever) wrote: > Yeah I noticed that struct fc_frame just contains an skbuff. But I saw > calls to fc_frame_free in certain cases and direct calls to kfree_skb in > other places instead. I may play with adding a count within kfree_skb > but I'll have to find a way to limit it to only the Niantic port and > only the FCoE frames, if possible. You could set skb->destructor to a function to count the frames. I don't think skb->destructor is used by the incoming frames, but if it is non-NULL you shouldn't overwrite it. But I don't think you're seeing an skb leak. See below. > Yes, I see the difference in my alloc/free numbers between dd with and > without direct I/O. This would lead me to think the issue is caching, > as you suggested, but the cache numbers to not increase. I have tried > dd with block sizes of 512, 4M, 16M. We did write tests with other > storage mediums with 4M and 16M block sizes among others. Some attempts > went through cleanly, others caused "Buffer I/O errors" and required > that the fcoe service was restarted. > > I just did a couple runs with bs=4K count=10000. It looks like the > freed number is half of the allocated number. As Vasu mentioned, I > guess they are not supposed to be the same. Output below... > > ~# dd if=/dev/zero of=/dev/sde1 bs=4K count=10000 oflag=dire > > fcoe_start_io: Frame #10000 (rc = 0) > > fc_frame_alloc: allocated 10000 frames > > fcoe_start_io: Frame #20000 (rc = 0) > > fc_frame_alloc: allocated 20000 frames > > fc_frame_free: freed 10000 frames > > fcoe_start_io: Frame #30000 (rc = 0) > > fc_frame_alloc: allocated 30000 frames > > fcoe_start_io: Frame #40000 (rc = 0) > > fc_frame_alloc: allocated 40000 frames > > fc_frame_free: freed 20000 frames > > fcoe_start_io: Frame #50000 (rc = 0) > > fc_frame_alloc: allocated 50000 frames > 10000+0 records in > 10000+0 records out > 40960000 bytes (41 MB) copied, 8.05651 seconds, 5.1 MB/s > [email protected]:~# dd if=/dev/zero of=/dev/sde1 bs=4K count=10000 > oflag=dire Each write allocates a command frame, and a data frame. Since Niantec does large send offload (LSO) only one data frame is allocated. These two frames are freed by the ixgbe driver using kfree_skb() or similar. Then a response frame is received, that's the one that libfc frees with fc_frame_free(). So, that explains your counts. 2 frames allocated, 1 freed per request. One thing you could do is figure out how much is leaking (if any) per I/O. Look at /proc/slabinfo (assuming you're using the slab allocator ... if not you could reconfigure to do that). Copy /proc/slabinfo somewhere, run 1000 or so I/Os and then diff to see which slabs are changing by how much. You could also look at vmstats before and after the I/O. I used something like that to find a leak in another area recently. > fcoe_start_io: Frame #60000 (rc = 0) > > fc_frame_alloc: allocated 60000 frames > > fcoe_start_io: Frame #70000 (rc = 0) > > fc_frame_alloc: allocated 70000 frames > > fc_frame_free: freed 30000 frames > > fcoe_start_io: Frame #80000 (rc = 0) > > fc_frame_alloc: allocated 80000 frames > > fcoe_start_io: Frame #90000 (rc = 0) > > fc_frame_alloc: allocated 90000 frames > > fc_frame_free: freed 40000 frames > > fcoe_start_io: Frame #100000 (rc = 0) > fcoe_xmit: TxFrames = 100000 > > fc_frame_alloc: allocated 100000 frames > 10000+0 records in > 10000+0 records out > 40960000 bytes (41 MB) copied, 6.08639 seconds, 6.7 MB/s > > Thanks, > Tristan > > -----Original Message----- > From: Joe Eykholt (jeykholt) > Sent: Tuesday, July 27, 2010 5:49 PM > To: Tristan Cheever (tcheever) > Cc: [email protected]; Walter Song (wasong) > Subject: Re: [Open-FCoE] Huge memory usage with mkfs.ext3 over FCoE > > On 7/27/10 5:28 PM, Tristan Cheever (tcheever) wrote: >> One more note... >> >> I added printk to fc_frame_alloc and fc_frame_free to show the number > of >> frames allocated vs freed. >> >> Without Direct I/O: Number of allocated frames is much greater than >> number freed. >> With Direct I/O: They are identical >> >> When going through the code, it seemed as though not all frames are >> freed via fc_frame_free. > > The fc_frames are actually sk_buffs, so when frames are sent out > the ethernet they'll be freed by kfree_skb() or something like that. > Frames received by Ethernet are usually converted into fc_frames > and freed by fc_frame_free(). Maybe we should call them sk_buffs > everywhere, but we don't. > > One could add a destructor function to the sk_buffs to count them > being freed. > > Are you doing dd with and without iflag=direct and noticing that > difference? > > I wonder if the difference might be the I/O size. With direct I/O > are you doing 512-byte reads or writes? Maybe you could try 4K > or larger and see if they're still different. > > Not sure what's going on here. > > Joe > > _______________________________________________ devel mailing list [email protected] http://www.open-fcoe.org/mailman/listinfo/devel
