Re: mlx4en, timer irq @100%... (11.0 stuck on high network load ???)

2017-09-07 Thread Julien Charbon
Hi Ben, On 8/31/17 12:04 PM, Ben RUBSON wrote: >> On 28 Aug 2017, at 11:27, Julien Charbon wrote: >> >> On 8/28/17 10:25 AM, Ben RUBSON wrote: >>>> On 16 Aug 2017, at 11:02, Ben RUBSON wrote: >>>> >>>>> On 15 Aug 2017, at 23:33, Ju

Re: mlx4en, timer irq @100%... (11.0 stuck on high network load ???)

2017-08-28 Thread Julien Charbon
Hi Ben, On 8/28/17 10:25 AM, Ben RUBSON wrote: >> On 16 Aug 2017, at 11:02, Ben RUBSON wrote: >> >>> On 15 Aug 2017, at 23:33, Julien Charbon wrote: >>> >>> On 8/11/17 11:32 AM, Ben RUBSON wrote: >>>>> On 08 Aug 2017, at 13:33, Julie

Re: mlx4en, timer irq @100%... (11.0 stuck on high network load ???)

2017-08-15 Thread Julien Charbon
Hi Ben, On 8/11/17 11:32 AM, Ben RUBSON wrote: >> On 08 Aug 2017, at 13:33, Julien Charbon wrote: >> >> On 8/8/17 10:31 AM, Hans Petter Selasky wrote: >>> >>> Suggested fix attached. >> >> I agree we your conclusion. Just for the record, more

Re: mlx4en, timer irq @100%... (11.0 stuck on high network load ???)

2017-08-08 Thread Julien Charbon
Hi, On 8/8/17 10:31 AM, Hans Petter Selasky wrote: > On 08/08/17 10:06, Ben RUBSON wrote: >>> On 08 Aug 2017, at 10:02, Hans Petter Selasky wrote: >>> >>> On 08/08/17 10:00, Ben RUBSON wrote: kgdb) print *twq_2msl.tqh_first $2 = { tw_inpcb = 0xf8031c570740, >>> >>> print *

Re: 11.0 stuck on high network load

2016-10-14 Thread Julien Charbon
Hi, On 10/14/16 11:35 AM, Slawa Olhovchenkov wrote: > On Thu, Oct 13, 2016 at 06:14:29PM +0200, Julien Charbon wrote: >> On 10/13/16 5:17 PM, Slawa Olhovchenkov wrote: >>> On Thu, Oct 13, 2016 at 05:06:00PM +0200, Julien Charbon wrote: >>> >>>>>&

Re: 11.0 stuck on high network load

2016-10-13 Thread Julien Charbon
On 10/13/16 5:17 PM, Slawa Olhovchenkov wrote: > On Thu, Oct 13, 2016 at 05:06:00PM +0200, Julien Charbon wrote: > >>>> will give you that trace in the core, and without INVARIANT then it is >>>> better to use dtrace: >>>> >>>> $ cat tcp-twst

Re: 11.0 stuck on high network load

2016-10-13 Thread Julien Charbon
Hi Slawa, On 10/13/16 4:38 PM, Slawa Olhovchenkov wrote: > On Thu, Oct 13, 2016 at 01:56:21PM +0200, Julien Charbon wrote: >>>> Something like: >>> >>> Yes, thanks! >> >> Proposed changes added in the review: >> >> https://reviews.

Re: 11.0 stuck on high network load

2016-10-13 Thread Julien Charbon
Hi Slawa, On 10/12/16 5:42 PM, Slawa Olhovchenkov wrote: > On Wed, Oct 12, 2016 at 05:17:35PM +0200, Julien Charbon wrote: > >>>>>>>>>> I see, thus just for the context: The TCP stack in sys/dev/cxgb* >>>>>>>>>> is a >&g

Re: 11.0 stuck on high network load

2016-10-12 Thread Julien Charbon
Hi Slawa, On 10/12/16 3:01 PM, Slawa Olhovchenkov wrote: > On Wed, Oct 12, 2016 at 02:35:11PM +0200, Julien Charbon wrote: >> On 10/12/16 2:13 PM, Slawa Olhovchenkov wrote: >>> On Wed, Oct 12, 2016 at 02:06:59PM +0200, Julien Charbon wrote: >>>>>>>&

Re: 11.0 stuck on high network load

2016-10-12 Thread Julien Charbon
Hi Slawa, On 10/12/16 2:13 PM, Slawa Olhovchenkov wrote: > On Wed, Oct 12, 2016 at 02:06:59PM +0200, Julien Charbon wrote: >>>>>>> sofree() call tcp_usr_detach() and in tcp_usr_detach() we have >>>>>>> unexpected INP_TIMEWAIT. >>>>>&

Re: 11.0 stuck on high network load

2016-10-12 Thread Julien Charbon
Hi Slawa, On 10/12/16 11:52 AM, Slawa Olhovchenkov wrote: > On Wed, Oct 12, 2016 at 11:42:38AM +0200, Julien Charbon wrote: >> On 10/12/16 11:29 AM, Slawa Olhovchenkov wrote: >>> On Wed, Oct 12, 2016 at 11:19:48AM +0200, Julien Charbon wrote: >>> >>>>>

Re: 11.0 stuck on high network load

2016-10-12 Thread Julien Charbon
Hi Slawa, On 10/12/16 10:40 AM, Slawa Olhovchenkov wrote: > On Wed, Oct 12, 2016 at 10:18:18AM +0200, Julien Charbon wrote: >> On 10/11/16 2:11 PM, Slawa Olhovchenkov wrote: >>> On Tue, Oct 11, 2016 at 09:20:17AM +0200, Julien Charbon wrote: >>>> Then threads a

Re: 11.0 stuck on high network load

2016-10-12 Thread Julien Charbon
On 10/12/16 11:29 AM, Slawa Olhovchenkov wrote: > On Wed, Oct 12, 2016 at 11:19:48AM +0200, Julien Charbon wrote: > >>> if INP_WLOCK is like spinlock -- this is dead lock. >>> if INP_WLOCK is like mutex -- thread1 resheduled. >> >> Thanks, I understand you q

Re: 11.0 stuck on high network load

2016-10-12 Thread Julien Charbon
Hi Slawa, On 10/11/16 2:11 PM, Slawa Olhovchenkov wrote: > On Tue, Oct 11, 2016 at 09:20:17AM +0200, Julien Charbon wrote: >> Then threads are competing for the INP_WLOCK lock. For the example, >> let's say the thread A wants to run tcp_input()/in_pcblookup_mbuf() an

Re: 11.0 stuck on high network load

2016-10-11 Thread Julien Charbon
Hi Slawa, On 10/10/16 7:35 PM, Slawa Olhovchenkov wrote: > On Mon, Oct 10, 2016 at 05:44:21PM +0200, Julien Charbon wrote: >>>> can check the current other usages of goto findpcb in tcp_input(). The >>>> rational here being: >>>> >>>> -

Re: 11.0 stuck on high network load

2016-10-10 Thread Julien Charbon
Hi, On 10/10/16 4:29 PM, Slawa Olhovchenkov wrote: > On Mon, Oct 10, 2016 at 04:03:39PM +0200, Julien Charbon wrote: >> On 10/10/16 3:32 PM, Slawa Olhovchenkov wrote: >>> On Mon, Oct 10, 2016 at 01:26:12PM +0200, Julien Charbon wrote: >>>> On 10/6/16 1:10 PM, Slaw

Re: 11.0 stuck on high network load

2016-10-10 Thread Julien Charbon
Hi Slawa, On 10/10/16 3:32 PM, Slawa Olhovchenkov wrote: > On Mon, Oct 10, 2016 at 01:26:12PM +0200, Julien Charbon wrote: >> On 10/6/16 1:10 PM, Slawa Olhovchenkov wrote: >>> On Thu, Oct 06, 2016 at 09:28:06AM +0200, Julien Charbon wrote: >>> >>>> 2. thr

Re: 11.0 stuck on high network load

2016-10-10 Thread Julien Charbon
Hi, On 10/6/16 1:10 PM, Slawa Olhovchenkov wrote: > On Thu, Oct 06, 2016 at 09:28:06AM +0200, Julien Charbon wrote: > >> 2. thread1: In tcp_close() the inp is marked with INP_DROPPED flag, the >> process continues and calls INP_WUNLOCK() here: >> >> https://g

Re: 11.0 stuck on high network load

2016-10-06 Thread Julien Charbon
Hi, On 9/28/16 1:59 PM, Slawa Olhovchenkov wrote: > On Wed, Sep 28, 2016 at 12:06:47PM +0200, Julien Charbon wrote: >> >> I am still trying to reproduce your issue, without success so far. Thanks for Slawa effort and multiple debug report we start seeing the bottom of th

Re: 11.0 stuck on high network load

2016-10-06 Thread Julien Charbon
Hi Hiren, On 10/6/16 9:44 AM, hiren panchasara wrote: > On 10/06/16 at 09:28P, Julien Charbon wrote: >> On 9/28/16 1:59 PM, Slawa Olhovchenkov wrote: >>> On Wed, Sep 28, 2016 at 12:06:47PM +0200, Julien Charbon wrote: >>>> >>>> I am still trying to re

Re: 11.0 stuck on high network load

2016-09-28 Thread Julien Charbon
Hi Slawa, On 9/26/16 7:22 PM, Slawa Olhovchenkov wrote: > On Mon, Sep 26, 2016 at 11:33:12AM +0200, Julien Charbon wrote: >> On 9/25/16 2:46 PM, Slawa Olhovchenkov wrote: >>> On Fri, Sep 23, 2016 at 11:01:43PM +0300, Slawa Olhovchenkov wrote: >>>>> On 9/21/16

Re: 11.0 stuck on high network load

2016-09-26 Thread Julien Charbon
Hi Slawa, On 9/25/16 2:46 PM, Slawa Olhovchenkov wrote: > On Fri, Sep 23, 2016 at 11:01:43PM +0300, Slawa Olhovchenkov wrote: >> On Wed, Sep 21, 2016 at 11:25:18PM +0200, Julien Charbon wrote: >>> >>> On 9/21/16 9:51 PM, Slawa Olhovchenkov wrote: >>>> O

Re: 11.0 stuck on high network load

2016-09-26 Thread Julien Charbon
Hi Slawa, On 9/25/16 2:46 PM, Slawa Olhovchenkov wrote: > On Fri, Sep 23, 2016 at 11:01:43PM +0300, Slawa Olhovchenkov wrote: >>> On 9/21/16 9:51 PM, Slawa Olhovchenkov wrote: >>>> On Wed, Sep 21, 2016 at 09:11:24AM +0200, Julien Charbon wrote: >>>>>

Re: 11.0 stuck on high network load

2016-09-26 Thread Julien Charbon
On 9/25/16 8:58 PM, Slawa Olhovchenkov wrote: > On Fri, Sep 23, 2016 at 10:16:56PM +0300, Slawa Olhovchenkov wrote: > >> On Thu, Sep 22, 2016 at 01:20:45PM +0300, Slawa Olhovchenkov wrote: >> >>> On Thu, Sep 22, 2016 at 12:04:40PM +0200, Julien Charbon wrote: >>

Re: 11.0 stuck on high network load

2016-09-26 Thread Julien Charbon
Hi Slawa, On 9/23/16 9:16 PM, Slawa Olhovchenkov wrote: > On Thu, Sep 22, 2016 at 01:20:45PM +0300, Slawa Olhovchenkov wrote: > >> On Thu, Sep 22, 2016 at 12:04:40PM +0200, Julien Charbon wrote: >> >>>>> These paths can indeed compete for the same INP loc

Re: 11.0 stuck on high network load

2016-09-22 Thread Julien Charbon
Hi Slawa, On 9/22/16 11:53 AM, Slawa Olhovchenkov wrote: > On Wed, Sep 21, 2016 at 11:25:18PM +0200, Julien Charbon wrote: >> On 9/21/16 9:51 PM, Slawa Olhovchenkov wrote: >>> On Wed, Sep 21, 2016 at 09:11:24AM +0200, Julien Charbon wrote: >>>> You can also use D

Re: 11.0 stuck on high network load

2016-09-22 Thread Julien Charbon
Hi Slawa, On 9/21/16 10:31 AM, Slawa Olhovchenkov wrote: > On Wed, Sep 21, 2016 at 09:11:24AM +0200, Julien Charbon wrote: >> On 9/20/16 10:26 PM, Slawa Olhovchenkov wrote: >>> On Tue, Sep 20, 2016 at 10:00:25PM +0200, Julien Charbon wrote: >>>> On 9/19/16 10:4

Re: 11.0 stuck on high network load

2016-09-21 Thread Julien Charbon
Hi Slawa, On 9/21/16 9:51 PM, Slawa Olhovchenkov wrote: > On Wed, Sep 21, 2016 at 09:11:24AM +0200, Julien Charbon wrote: >> You can also use Dtrace and lockstat (especially with the lockstat -s >> option): >> >> https://wiki.freebsd.org/DTrace/One-L

Re: 11.0 stuck on high network load

2016-09-21 Thread Julien Charbon
Hi Slawa, On 9/20/16 10:26 PM, Slawa Olhovchenkov wrote: > On Tue, Sep 20, 2016 at 10:00:25PM +0200, Julien Charbon wrote: >> On 9/19/16 10:43 PM, Slawa Olhovchenkov wrote: >>> On Mon, Sep 19, 2016 at 10:32:13PM +0200, Julien Charbon wrote: >>>> >>>>

Re: 11.0 stuck on high network load

2016-09-20 Thread Julien Charbon
Hi Slawa, On 9/19/16 10:43 PM, Slawa Olhovchenkov wrote: > On Mon, Sep 19, 2016 at 10:32:13PM +0200, Julien Charbon wrote: >> >>> @ CPU_CLK_UNHALTED_CORE [4653445 samples] >>> >>> 51.86% [2413083] lock_delay @ /boot/kernel.VSTREAM/kernel >>> 10

Re: 11.0 stuck on high network load

2016-09-19 Thread Julien Charbon
Hi Slawa, On 9/16/16 9:03 PM, Slawa Olhovchenkov wrote: > On Fri, Sep 16, 2016 at 11:30:53AM -0700, hiren panchasara wrote: > >> On 09/16/16 at 09:18P, Slawa Olhovchenkov wrote: >>> On Thu, Sep 15, 2016 at 12:06:33PM +0300, Slawa Olhovchenkov wrote: >>> On Thu, Sep 15, 2016 at 11:59:38AM +