11.0-stable can not buildworld

[k simon]

Hi,lists,

  11.0-stable r306419 can not buildworld. The output as below:

clang  -O2 -pipe -fno-omit-frame-pointer -march=core2 
-I/usr/src/lib/libc/include -I/usr/src/lib/libc/../../include 
-I/usr/src/lib/libc/amd64 -DNLS  -D__DBINTERFACE_PRIVATE 
-I/usr/src/lib/libc/../../contrib/gdtoa 
-I/usr/src/lib/libc/../../contrib/libc-vis -DINET6 
-I/usr/obj/usr/src/lib/libc -I/usr/src/lib/libc/resolv -D_ACL_PRIVATE 
-DPOSIX_MISTAKE -I/usr/src/lib/libc/../libmd 
-I/usr/src/lib/libc/../../contrib/jemalloc/include -DMALLOC_PRODUCTION 
-I/usr/src/lib/libc/../../contrib/tzcode/stdtime 
-I/usr/src/lib/libc/stdtime -I/usr/src/lib/libc/locale -DBROKEN_DES 
-DPORTMAP -DDES_BUILTIN -I/usr/src/lib/libc/rpc -DYP -DNS_CACHING 
-DSYMBOL_VERSIONING -MD  -MF.depend.__vdso_gettimeofday.o 
-MT__vdso_gettimeofday.o -std=gnu99 -fstack-protector-strong 
-Wsystem-headers -Werror -Wall -Wno-format-y2k -Wno-uninitialized 
-Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int 
-Wno-unused-const-variable -Wno-tautological-compare -Wno-unused-value 
-Wno-parentheses-equality -Wno-unused-function -Wno-enum-conversion 
-Wno-unused-local-typedef -Wno-switch -Wno-switch-enum 
-Wno-knr-promoted-parameter  -Qunused-arguments  -I/usr/src/lib/libutil 
-I/usr/src/lib/msun/amd64 -I/usr/src/lib/msun/x86 
-I/usr/src/lib/msun/src -c /usr/src/lib/libc/sys/__vdso_gettimeofday.c 
-o __vdso_gettimeofday.o
/usr/src/lib/libc/sys/__vdso_gettimeofday.c:43:27: error: too many 
arguments to function call, expected single argument 'vdso_th', have 2 
arguments

error = __vdso_gettc(th, );
 ^~~
/usr/include/sys/vdso.h:65:1: note: '__vdso_gettc' declared here
u_int __vdso_gettc(const struct vdso_timehands *vdso_th);
^
1 error generated.
--- __error.o ---
--- __vdso_gettimeofday.o ---
*** [__vdso_gettimeofday.o] Error code 1

make[4]: stopped in /usr/src/lib/libc
--- __error.o ---




Simon
200922
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 11.0 stuck on high network load

[Julien Charbon]

 Hi Slawa,

On 9/21/16 9:51 PM, Slawa Olhovchenkov wrote:
> On Wed, Sep 21, 2016 at 09:11:24AM +0200, Julien Charbon wrote:
>>  You can also use Dtrace and lockstat (especially with the lockstat -s
>> option):
>>
>> https://wiki.freebsd.org/DTrace/One-Liners#Kernel_Locks
>> https://www.freebsd.org/cgi/man.cgi?query=lockstat=FreeBSD+11.0-RELEASE
>>
>>  But I am less familiar with Dtrace/lockstat tools.
> 
> I am still use old kernel and got lockdown again.
> Try using lockstat (I am save more output), interesting may be next:
> 
> R/W writer spin on writer: 190019 events in 1.070 seconds (177571 events/sec)
> 
> ---
> Count indv cuml rcnt nsec Lock   Caller  
> 140839  74%  74% 0.0024659 tcpinp tcp_tw_2msl_scan+0xc6   
> 
>   nsec -- Time Distribution -- count Stack   
>   4096 |   913   tcp_twstart+0xa3
>   8192 |   58191 tcp_do_segment+0x201f   
>  16384 |@@ 29594 tcp_input+0xe1c 
>  32768 |   23447 ip_input+0x15f  
>  65536 |@@@16197 
> 131072 |@  8674  
> 262144 |   3358  
> 524288 |   456   
>1048576 |   9 
> ---
> Count indv cuml rcnt nsec Lock   Caller  
> 49180  26% 100% 0.0015929 tcpinp tcp_tw_2msl_scan+0xc6   
> 
>   nsec -- Time Distribution -- count Stack   
>   4096 |   157   pfslowtimo+0x54 
>   8192 |@@@24796 softclock_call_cc+0x179 
>  16384 |@@ 11223 softclock+0x44  
>  32768 |   7426  
> intr_event_execute_handlers+0x95
>  65536 |@@ 3918  
> 131072 |   1363  
> 262144 |   278   
> 524288 |   19
> ---

 This is interesting, it seems that you have two call paths competing
for INP locks here:

 - pfslowtimo()/tcp_tw_2msl_scan(reuse=0) and

 - tcp_input()/tcp_twstart()/tcp_tw_2msl_scan(reuse=1)

 These paths can indeed compete for the same INP lock, as both
tcp_tw_2msl_scan() calls always start with the first inp found in
twq_2msl list.  But in both cases, this first inp should be quickly used
and its lock released anyway, thus that could explain your situation it
that the TCP stack is doing that all the time, for example:

 - Let say that you are running out completely and constantly of tcptw,
and then all connections transitioning to TIME_WAIT state are competing
with the TIME_WAIT timeout scan that tries to free all the expired
tcptw.  If the stack is doing that all the time, it can appear like
"live" locked.

 This is just an hypothesis and as usual might be a red herring.
Anyway, could you run:

$ vmstat -z | head -2; vmstat -z | grep -E 'tcp|sock'

 Ideally, once when everything is ok, and once when you have the issue
to see the differences (if any).

 If it appears your are quite low in tcptw, and if you have enough
memory, could you try increase the tcptw limit using sysctl
net.inet.tcp.maxtcptw?  And actually see if it improve (or not) your
performance.

 My 2 cents.

--
Julien



signature.asc
Description: OpenPGP digital signature

Jenkins build is back to stable : FreeBSD_stable_10 #404

[jenkins-admin]

See 

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 11.0 stuck on high network load

[Slawa Olhovchenkov]

On Wed, Sep 21, 2016 at 09:11:24AM +0200, Julien Charbon wrote:

> 
>  You can also use Dtrace and lockstat (especially with the lockstat -s
> option):
> 
> https://wiki.freebsd.org/DTrace/One-Liners#Kernel_Locks
> https://www.freebsd.org/cgi/man.cgi?query=lockstat=FreeBSD+11.0-RELEASE
> 
>  But I am less familiar with Dtrace/lockstat tools.

I am still use old kernel and got lockdown again.
Try using lockstat (I am save more output), interesting may be next:

R/W writer spin on writer: 190019 events in 1.070 seconds (177571 events/sec)

---
Count indv cuml rcnt nsec Lock   Caller  
140839  74%  74% 0.0024659 tcpinp tcp_tw_2msl_scan+0xc6   

  nsec -- Time Distribution -- count Stack   
  4096 |   913   tcp_twstart+0xa3
  8192 |   58191 tcp_do_segment+0x201f   
 16384 |@@ 29594 tcp_input+0xe1c 
 32768 |   23447 ip_input+0x15f  
 65536 |@@@16197 
131072 |@  8674  
262144 |   3358  
524288 |   456   
   1048576 |   9 
---
Count indv cuml rcnt nsec Lock   Caller  
49180  26% 100% 0.0015929 tcpinp tcp_tw_2msl_scan+0xc6   

  nsec -- Time Distribution -- count Stack   
  4096 |   157   pfslowtimo+0x54 
  8192 |@@@24796 softclock_call_cc+0x179 
 16384 |@@ 11223 softclock+0x44  
 32768 |   7426  
intr_event_execute_handlers+0x95
 65536 |@@ 3918  
131072 |   1363  
262144 |   278   
524288 |   19
---


> >>  #1. Try above kernel options at least once, and see what you can get.
> > 
> > OK, I am try this after some time.
> > 
> >>  #2. If #1 is a total failure try below patch:  It won't solve anything,
> >> it just makes tcp_tw_2msl_scan() less greedy when there is contention on
> >> the INP write lock.  If it makes the debugging more feasible, continue
> >> to #3.
> > 
> > OK, thanks.
> > What purpose to not skip locked tcptw in this loop?
> 
>  If I understand your question correctly:  According to your pmcstat
> result, tcp_tw_2msl_scan() currently struggles with a write lock
> (__rw_wlock_hard) and the only write lock used tcp_tw_2msl_scan() is
> INP_WLOCK.  No sign of contention on TW_RLOCK(V_tw_lock) currently.
> 
> 51.86%  [2413083]  lock_delay @ /boot/kernel.VSTREAM/kernel
>  100.0%  [2413083]   __rw_wlock_hard
>   100.0%  [2413083]tcp_tw_2msl_scan
> 
> --
> Julien
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Problem with nsswitch.conf

[Shawn Bakhtiar]

Oh Jesus!!! Thank you.. that worked.. obviously...

> On Sep 21, 2016, at 10:28 AM, Eric van Gyzen  wrote:
> 
> On 09/21/2016 12:21, Shawn Bakhtiar wrote:
>> Good morning All,
>> 
>> I'm trying to configure my server as an LDAP client. I installed the nslcd 
>> service and it's working great.
>> 
>> My problem is when I issue the command getent passwd it only returns the 
>> LDAP user not the local users. 
>> 
>> #
>> # nsswitch.conf(5) - name service switch configuration file
>> # $FreeBSD: releng/10.2/etc/nsswitch.conf 224765 2011-08-10 20:52:02Z dougb $
>> #
>> group: file ldap
>> group_compat: nis ldap
>> hosts: files dns
>> networks: files
>> passwd: file ldap
>> passwd_compat: nis ldap
>> shells: files 
>> services: files 
>> services_compat: nis
>> protocols: files 
>> rpc: files
>> 
>> 
>> When I change the above group and passwd setting back to compat (which was 
>> the default configuration) I get the local users but none of the ldap users 
>> show up. In fact nslcd is not even called (i've checked by running it in 
>> debug mode). So how do I configure nsswitch to use both the local 
>> /etc/passwd file and the ldap. I need this because without it services will 
>> not start. IE nslcd complains that nslcd is not a valid user when using the 
>> above configuration.
> 
> It should be "files", plural.
> 
> Eric

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Problem with nsswitch.conf

[Eric van Gyzen]

On 09/21/2016 12:21, Shawn Bakhtiar wrote:
> Good morning All,
>
> I'm trying to configure my server as an LDAP client. I installed the nslcd 
> service and it's working great.
>
> My problem is when I issue the command getent passwd it only returns the LDAP 
> user not the local users. 
>
> #
> # nsswitch.conf(5) - name service switch configuration file
> # $FreeBSD: releng/10.2/etc/nsswitch.conf 224765 2011-08-10 20:52:02Z dougb $
> #
> group: file ldap
> group_compat: nis ldap
> hosts: files dns
> networks: files
> passwd: file ldap
> passwd_compat: nis ldap
> shells: files 
> services: files 
> services_compat: nis
> protocols: files 
> rpc: files
>
>
> When I change the above group and passwd setting back to compat (which was 
> the default configuration) I get the local users but none of the ldap users 
> show up. In fact nslcd is not even called (i've checked by running it in 
> debug mode). So how do I configure nsswitch to use both the local /etc/passwd 
> file and the ldap. I need this because without it services will not start. IE 
> nslcd complains that nslcd is not a valid user when using the above 
> configuration.

It should be "files", plural.

Eric
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Problem with nsswitch.conf

[Shawn Bakhtiar]

Good morning All,

I'm trying to configure my server as an LDAP client. I installed the nslcd 
service and it's working great.

My problem is when I issue the command getent passwd it only returns the LDAP 
user not the local users. 

#
# nsswitch.conf(5) - name service switch configuration file
# $FreeBSD: releng/10.2/etc/nsswitch.conf 224765 2011-08-10 20:52:02Z dougb $
#
group: file ldap
group_compat: nis ldap
hosts: files dns
networks: files
passwd: file ldap
passwd_compat: nis ldap
shells: files 
services: files 
services_compat: nis
protocols: files 
rpc: files


When I change the above group and passwd setting back to compat (which was the 
default configuration) I get the local users but none of the ldap users show 
up. In fact nslcd is not even called (i've checked by running it in debug 
mode). So how do I configure nsswitch to use both the local /etc/passwd file 
and the ldap. I need this because without it services will not start. IE nslcd 
complains that nslcd is not a valid user when using the above configuration.

Any help would greatly be appreciated,
Shawn

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Jenkins build became unstable: FreeBSD_stable_10 #403

[jenkins-admin]

See 

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 11.0 stuck on high network load

[Slawa Olhovchenkov]

On Wed, Sep 21, 2016 at 09:11:24AM +0200, Julien Charbon wrote:

> 
>  Hi Slawa,
> 
> On 9/20/16 10:26 PM, Slawa Olhovchenkov wrote:
> > On Tue, Sep 20, 2016 at 10:00:25PM +0200, Julien Charbon wrote:
> >> On 9/19/16 10:43 PM, Slawa Olhovchenkov wrote:
> >>> On Mon, Sep 19, 2016 at 10:32:13PM +0200, Julien Charbon wrote:
> 
> > @ CPU_CLK_UNHALTED_CORE [4653445 samples]
> >
> > 51.86%  [2413083]  lock_delay @ /boot/kernel.VSTREAM/kernel
> >  100.0%  [2413083]   __rw_wlock_hard
> >   100.0%  [2413083]tcp_tw_2msl_scan
> >99.99%  [2412958] pfslowtimo
> > 100.0%  [2412958]  softclock_call_cc
> >  100.0%  [2412958]   softclock
> >   100.0%  [2412958]intr_event_execute_handlers
> >100.0%  [2412958] ithread_loop
> > 100.0%  [2412958]  fork_exit
> >00.01%  [125] tcp_twstart
> > 100.0%  [125]  tcp_do_segment
> >  100.0%  [125]   tcp_input
> >   100.0%  [125]ip_input
> >100.0%  [125] swi_net
> > 100.0%  [125]  intr_event_execute_handlers
> >  100.0%  [125]   ithread_loop
> >   100.0%  [125]fork_exit
> 
>   The only write lock tcp_tw_2msl_scan() tries to get is a
>  INP_WLOCK(inp).  Thus here, tcp_tw_2msl_scan() seems to be stuck
>  spinning on INP_WLOCK (or pfslowtimo() is going crazy and calls
>  tcp_tw_2msl_scan() at high rate but this will be quite unexpected).
> 
>   Thus my hypothesis is that something is holding the INP_WLOCK and not
>  releasing it, and tcp_tw_2msl_scan() is spinning on it.
> 
>   If you can, could you compile the kernel with below options:
> 
>  optionsDDB # Support DDB.
>  optionsDEADLKRES   # Enable the deadlock resolver
>  optionsINVARIANTS  # Enable calls of extra sanity
>  checking
>  optionsINVARIANT_SUPPORT   # Extra sanity checks of internal
>  structures, required by INVARIANTS
>  optionsWITNESS # Enable checks to detect
>  deadlocks and cycles
>  optionsWITNESS_SKIPSPIN# Don't run witness on spinlocks
>  for speed
> >>>
> >>> Currently this host run with 100% CPU load (on all cores), i.e.
> >>> enabling WITNESS will be significant drop performance.
> >>> Can I use only some subset of options?
> >>>
> >>> Also, I can some troubles to DDB enter in this case.
> >>> May be kgdb will be success (not tryed yet)?
> >>
> >>  If these kernel options will certainly slow down your kernel, they also
> >> might found the root cause of your issue before reaching the point where
> >> you have 100% cpu load on all cores (thanks to INVARIANTS).  I would
> >> suggest:
> > 
> > Hmmm, may be I am not clarified.
> > This host run at peak hours with 100% CPU load as normal operation,
> > this is for servering 2x10G, this is CPU load not result of lock
> > issuse, this is not us case. And this is because I am fear to enable
> > WITNESS -- I am fear drop performance.
> > 
> > This lock issuse happen irregulary and may be caused by other issuse
> > (nginx crashed). In this case about 1/3 cores have 100% cpu load,
> > perhaps by this lock -- I am can trace only from one core and need
> > more then hour for this (may be on other cores different trace, I
> > can't guaranted anything).
> 
>  I see, especially if you are running in production WITNESS might indeed
> be not practical for you.  In this case, I would suggest before doing
> WITNESS and still get more information to:
> 
>  #0: Do a lock profiling:
> 
> https://www.freebsd.org/cgi/man.cgi?query=LOCK_PROFILING
> 
> options LOCK_PROFILING
> 
>  Example of usage:
> 
> # Run
> $ sudo sysctl debug.lock.prof.enable=1
> $ sleep 10
> $ sudo sysctl debug.lock.prof.enable=0
> 
> # Get results
> $ sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats |
> sort -n -k 4 -r

OK, but in case of leak lock (why inp lock too long for
tcp_tw_2msl_scan?) I can't see cause of this lock running this
commands after stuck happen?

>  You can also use Dtrace and lockstat (especially with the lockstat -s
> option):
> 
> https://wiki.freebsd.org/DTrace/One-Liners#Kernel_Locks
> https://www.freebsd.org/cgi/man.cgi?query=lockstat=FreeBSD+11.0-RELEASE
> 
>  But I am less familiar with Dtrace/lockstat tools.

OK, interesting too. Thanks.

> >>  #1. Try above kernel options at least once, and see what you can get.
> > 
> > OK, I am try this after some time.
> > 
> >>  #2. If #1 is a total failure try below patch:  It won't solve anything,
> >> it just makes tcp_tw_2msl_scan() less greedy when there is contention on
> >> the INP write lock.  If it makes the debugging more feasible, continue
> >> to #3.
> > 
> > OK, thanks.
> > What purpose to not skip locked tcptw in this loop?
> 
>  If 

Re: 11.0 stuck on high network load

[Julien Charbon]

 Hi Slawa,

On 9/20/16 10:26 PM, Slawa Olhovchenkov wrote:
> On Tue, Sep 20, 2016 at 10:00:25PM +0200, Julien Charbon wrote:
>> On 9/19/16 10:43 PM, Slawa Olhovchenkov wrote:
>>> On Mon, Sep 19, 2016 at 10:32:13PM +0200, Julien Charbon wrote:

> @ CPU_CLK_UNHALTED_CORE [4653445 samples]
>
> 51.86%  [2413083]  lock_delay @ /boot/kernel.VSTREAM/kernel
>  100.0%  [2413083]   __rw_wlock_hard
>   100.0%  [2413083]tcp_tw_2msl_scan
>99.99%  [2412958] pfslowtimo
> 100.0%  [2412958]  softclock_call_cc
>  100.0%  [2412958]   softclock
>   100.0%  [2412958]intr_event_execute_handlers
>100.0%  [2412958] ithread_loop
> 100.0%  [2412958]  fork_exit
>00.01%  [125] tcp_twstart
> 100.0%  [125]  tcp_do_segment
>  100.0%  [125]   tcp_input
>   100.0%  [125]ip_input
>100.0%  [125] swi_net
> 100.0%  [125]  intr_event_execute_handlers
>  100.0%  [125]   ithread_loop
>   100.0%  [125]fork_exit

  The only write lock tcp_tw_2msl_scan() tries to get is a
 INP_WLOCK(inp).  Thus here, tcp_tw_2msl_scan() seems to be stuck
 spinning on INP_WLOCK (or pfslowtimo() is going crazy and calls
 tcp_tw_2msl_scan() at high rate but this will be quite unexpected).

  Thus my hypothesis is that something is holding the INP_WLOCK and not
 releasing it, and tcp_tw_2msl_scan() is spinning on it.

  If you can, could you compile the kernel with below options:

 optionsDDB # Support DDB.
 optionsDEADLKRES   # Enable the deadlock resolver
 optionsINVARIANTS  # Enable calls of extra sanity
 checking
 optionsINVARIANT_SUPPORT   # Extra sanity checks of internal
 structures, required by INVARIANTS
 optionsWITNESS # Enable checks to detect
 deadlocks and cycles
 optionsWITNESS_SKIPSPIN# Don't run witness on spinlocks
 for speed
>>>
>>> Currently this host run with 100% CPU load (on all cores), i.e.
>>> enabling WITNESS will be significant drop performance.
>>> Can I use only some subset of options?
>>>
>>> Also, I can some troubles to DDB enter in this case.
>>> May be kgdb will be success (not tryed yet)?
>>
>>  If these kernel options will certainly slow down your kernel, they also
>> might found the root cause of your issue before reaching the point where
>> you have 100% cpu load on all cores (thanks to INVARIANTS).  I would
>> suggest:
> 
> Hmmm, may be I am not clarified.
> This host run at peak hours with 100% CPU load as normal operation,
> this is for servering 2x10G, this is CPU load not result of lock
> issuse, this is not us case. And this is because I am fear to enable
> WITNESS -- I am fear drop performance.
> 
> This lock issuse happen irregulary and may be caused by other issuse
> (nginx crashed). In this case about 1/3 cores have 100% cpu load,
> perhaps by this lock -- I am can trace only from one core and need
> more then hour for this (may be on other cores different trace, I
> can't guaranted anything).

 I see, especially if you are running in production WITNESS might indeed
be not practical for you.  In this case, I would suggest before doing
WITNESS and still get more information to:

 #0: Do a lock profiling:

https://www.freebsd.org/cgi/man.cgi?query=LOCK_PROFILING

options LOCK_PROFILING

 Example of usage:

# Run
$ sudo sysctl debug.lock.prof.enable=1
$ sleep 10
$ sudo sysctl debug.lock.prof.enable=0

# Get results
$ sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats |
sort -n -k 4 -r

 You can also use Dtrace and lockstat (especially with the lockstat -s
option):

https://wiki.freebsd.org/DTrace/One-Liners#Kernel_Locks
https://www.freebsd.org/cgi/man.cgi?query=lockstat=FreeBSD+11.0-RELEASE

 But I am less familiar with Dtrace/lockstat tools.

>>  #1. Try above kernel options at least once, and see what you can get.
> 
> OK, I am try this after some time.
> 
>>  #2. If #1 is a total failure try below patch:  It won't solve anything,
>> it just makes tcp_tw_2msl_scan() less greedy when there is contention on
>> the INP write lock.  If it makes the debugging more feasible, continue
>> to #3.
> 
> OK, thanks.
> What purpose to not skip locked tcptw in this loop?

 If I understand your question correctly:  According to your pmcstat
result, tcp_tw_2msl_scan() currently struggles with a write lock
(__rw_wlock_hard) and the only write lock used tcp_tw_2msl_scan() is
INP_WLOCK.  No sign of contention on TW_RLOCK(V_tw_lock) currently.

51.86%  [2413083]  lock_delay @ /boot/kernel.VSTREAM/kernel
 100.0%  [2413083]   __rw_wlock_hard
  100.0%  [2413083]tcp_tw_2msl_scan

--
Julien
___