Hi Ben,

On 8/31/17 12:04 PM, Ben RUBSON wrote:
>> On 28 Aug 2017, at 11:27, Julien Charbon <j...@freebsd.org> wrote:
>>
>> On 8/28/17 10:25 AM, Ben RUBSON wrote:
>>>> On 16 Aug 2017, at 11:02, Ben RUBSON <ben.rub...@gmail.com> wrote:
>>>>
>>>>> On 15 Aug 2017, at 23:33, Julien Charbon <j...@freebsd.org> wrote:
>>>>>
>>>>> On 8/11/17 11:32 AM, Ben RUBSON wrote:
>>>>>>> On 08 Aug 2017, at 13:33, Julien Charbon <j...@freebsd.org> wrote:
>>>>>>>
>>>>>>> On 8/8/17 10:31 AM, Hans Petter Selasky wrote:
>>>>>>>>
>>>>>>>> Suggested fix attached.
>>>>>>>
>>>>>>> I agree we your conclusion.  Just for the record, more precisely this
>>>>>>> regression seems to have been introduced with:
>>>>>>> (...)
>>>>>>> Thus good catch, and your patch looks good.  I am going to just verify
>>>>>>> the other in_pcbrele_wlocked() calls in TCP stack.
>>>>>>
>>>>>> Julien, do you plan to make this fix reach 11.0-p12 ?
>>>>>
>>>>> I am checking if your issue is another flavor of the issue fixed by:
>>>>>
>>>>> https://svnweb.freebsd.org/base?view=revision&revision=307551
>>>>> https://reviews.freebsd.org/D8211
>>>>>
>>>>> This fix in not in 11.0 but in 11.1.  Currently I did not found how an
>>>>> inp in INP_TIMEWAIT state can have been INP_FREED without having its tw
>>>>> set to NULL already except the issue fixed by r307551.
>>>>>
>>>>> Thus could you try to apply this patch:
>>>>>
>>>>> https://github.com/freebsd/freebsd/commit/acb5bfda99b753d9ead3529d04f20087c5f7d0a0.patch
>>>>>
>>>>> and see if you can still reproduce this issue?
>>>>
>>>> Thank you for your answer Julien.
>>>> Unfortunately, I'm not sure at all how to reproduce the issue.
>>>> I have other servers which are 100% identical to this one, same workload,
>>>> same some-months uptime, but they did not trigger the bug yet.
>>>>
>>>> If other network stack experts (I'm not) agree with your analysis,
>>>> we could then certainly go further with D8211 / r307551.
>>>>
>>>> One thing that perhaps might help :
>>>> # netstat -an | grep TIME_WAIT$ | wc -l
>>>> 468
>>>>
>>>> Note that due to this running bug, sendmail has lots of difficulties to 
>>>> send outgoing mails.
>>>> As soon as I run the above netstat command, I receive a lot of stacked 
>>>> mails (more than 20 this time).
>>>> As if netstat was able to somehow help...
>>>>
>>>> Number of TIME_WAIT connections however does not decrease, but increases.
>>>>
>>>>> And in the spirit of r307551 fix and based on Hans patch I will also
>>>>> propose to add a kernel log describing the issue instead of starting an
>>>>> infinite loop when INVARIANT is not set.
>>>>
>>>> Which should then never be triggered :)
>>>> Good idea I think !
>>>
>>> What about :
>>> D8211/r307551
>>> + Hans' patch
>>> + Julien's idea of a kernel log (sort of "We should not be here but we are")
>>
>> I did this change and I am testing it
> 
> Good news !
> 
>> on your side did you try this patch applied on 11.0?
>>
>> https://github.com/freebsd/freebsd/commit/acb5bfda99b753d9ead3529d04f20087c5f7d0a0.patch
> 
> Yes, patch applied and running correctly,
> however hard to say whether or not it solves this issue,
> as there is no easy way to reproduce it.

 No problem, it is just a matter of not seeing the issue anymore during
a long enough period.

I created a review that includes Hans's patch and uses the same
log(LOG_ERR) logic than r307551:
https://reviews.freebsd.org/D12267

 On my side, TCP smoke tests are ok.  And I am going to launch our TCP
QA on it while receiving review comments.

> Mail sent to FreeBSD Security Team !
> 
> Many thanks, let's stay tuned !

 Thanks to you and Hans for reporting that issue.  And in summary:

 - Applying r307551 on top of 11.0 should prevent this case to happen
 - D12267 will prevent the tcp_tw_2msl_scan() infinite loop while
reporting the error, in case a regression defeating r307551 is introduced

 Thanks.

--
Julien
_______________________________________________
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Reply via email to