Re: amd64 -> armv6 [and powerpc64] -r302331 -> -r302412 re-cross-build (update): got "sh: ./make_keys: Exec format error" again for init_ketry.h in ncursesw

2016-07-10 Thread Mark Millard
On 2016-Jul-8, at 12:23 AM, Mark Millard  wrote --but with 
a few []'d notes added:

> [Before the below cross build/update attempt I updated my amd64 from -r302331 
> -> -r302412.]
> 
> Summary: It appears that WITHOUT_META_MODE= still needs to be forced for 
> cross compiles at least sometimes in order to avoid "Exec format error". man 
> src.conf only mentions WITHOUT_META_MODE= in one place:
> 
>> WITH_DIRDEPS_BUILD
> . . .
>> WITH_META_MODE (unless WITHOUT_META_MODE is set explicitly)
> . . .
>> This must be set in the environment, make command line, or
>> /etc/src-env.conf, not /etc/src.conf.
> 
> 
> In attempting to update my cross build (amd64 -> armv6 [or powerpc64]) from 
> -r302331 to -r302412 it failed with [armv6 example]:
> 
>> --- init_keytry.h ---
>> sh: ./make_keys: Exec format error
>> *** [init_keytry.h] Error code 126
>> 
>> make[4]: stopped in /usr/src/lib/ncurses/ncursesw
>> .ERROR_TARGET='init_keytry.h'
>> .ERROR_META_FILE='/usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncursesw/init_keytry.h.meta'
>> .MAKE.LEVEL='4'
>> MAKEFILE=''
>> .MAKE.MODE='meta missing-filemon=yes missing-meta=yes silent=yes verbose'
>> .CURDIR='/usr/src/lib/ncurses/ncursesw'
>> .MAKE='make'
>> .OBJDIR='/usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncursesw'
>> .TARGETS='all'
>> DESTDIR='/usr/obj/clang/arm.armv6/usr/src/tmp'
>> LD_LIBRARY_PATH=''
>> MACHINE='arm'
>> MACHINE_ARCH='armv6'
>> MAKEOBJDIRPREFIX='/usr/obj/clang/arm.armv6'
>> MAKESYSPATH='/usr/src/share/mk'
>> MAKE_VERSION='20160606'
>> PATH='/usr/obj/clang/arm.armv6/usr/src/tmp/legacy/usr/sbin:/usr/obj/clang/arm.armv6/usr/src/tmp/legacy/usr/bin:/usr/obj/clang/arm.armv6/usr/src/tmp/legacy/bin:/usr/obj/clang/arm.armv6/usr/src/tmp/usr/sbin:/usr/obj/clang/arm.armv6/usr/src/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin'
>> SRCTOP='/usr/src'
>> OBJTOP='/usr/obj/clang/arm.armv6/usr/src'
>> .MAKE.MAKEFILES='/usr/src/share/mk/sys.mk /usr/src/share/mk/local.sys.env.mk 
>> /usr/src/share/mk/src.sys.env.mk 
>> /root/src.configs/src.conf.rpi2-clang-bootstrap.amd64-host 
>> /usr/src/share/mk/bsd.mkopt.mk /root/src.configs/make.conf 
>> /usr/src/share/mk/local.sys.mk /usr/src/share/mk/src.sys.mk /etc/src.conf 
>> /usr/src/lib/ncurses/ncursesw/Makefile 
>> /usr/src/lib/ncurses/ncursesw/../ncurses/Makefile 
>> /usr/src/share/mk/src.opts.mk /usr/src/share/mk/bsd.own.mk 
>> /usr/src/share/mk/bsd.opts.mk /usr/src/share/mk/bsd.cpu.mk 
>> /usr/src/share/mk/bsd.compiler.mk /usr/src/share/mk/bsd.compiler.mk 
>> /usr/src/lib/ncurses/ncursesw/../config.mk /usr/src/share/mk/bsd.lib.mk 
>> /usr/src/share/mk/bsd.init.mk /usr/src/share/mk/local.init.mk 
>> /usr/src/share/mk/src.init.mk /usr/src/lib/ncurses/ncursesw/../Makefile.inc 
>> /usr/src/lib/ncurses/ncursesw/../../Makefile.inc 
>> /usr/src/share/mk/bsd.libnames.mk /usr/src/share/mk/src.libnames.mk 
>> /usr/src/share/mk/bsd.symver.mk /usr/src/share/mk/bsd.nls.mk 
>> /usr/src/share/mk/bsd.files.mk
  /usr/src/share/mk/bsd.incs.mk /usr/src/share/mk/bsd.confs.mk 
/usr/src/share/mk/bsd.links.mk /usr/src/share/mk/bsd.dep.mk 
/usr/src/share/mk/bsd.clang-analyze.mk /usr/src/share/mk/bsd.obj.mk 
/usr/src/share/mk/bsd.subdir.mk /usr/src/share/mk/bsd.sys.mk'
>> .PATH='. /usr/src/lib/ncurses/ncursesw 
>> /usr/src/lib/ncurses/ncursesw/../ncurses 
>> /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/include 
>> /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/ncurses/base 
>> /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/ncurses/tinfo 
>> /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/ncurses/tty 
>> /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/ncurses/widechar 
>> /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/ncurses/trace 
>> /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/man'
>> 1 error
> 
> again.
> 
> This was based on:
> 
>> # more 
>> ~/sys_build_scripts.amd64-host/make_rpi2_nodebug_clang_bootstrap-amd64-host.sh
>>  
>> kldload -n filemon && \
>> script 
>> ~/sys_typescripts/typescript_make_rpi2_nodebug_clang_bootstrap-amd64-host-$(date
>>  +%Y-%m-%d:%H:%M:%S) \
>> env __MAKE_CONF="/root/src.configs/make.conf" 
>> SRC_ENV_CONF="/root/src.configs/src.conf.rpi2-clang-bootstrap.amd64-host" \
>> WITH_META_MODE=yes \
>> MAKEOBJDIRPREFIX="/usr/obj/clang" \
>> make $*
> 
> and. . .
> 
>> # more ~/src.configs/src.conf.rpi2-clang-bootstrap.amd64-host 
>> TO_TYPE=armv6
>> #
>> KERNCONF=RPI2-NODBG
>> TARGET=arm
>> .if ${.MAKE.LEVEL} == 0
>> TARGET_ARCH=${TO_TYPE}
>> .export TARGET_ARCH
>> .endif
>> #
>> WITH_CROSS_COMPILER=
>> WITHOUT_SYSTEM_COMPILER=
>> #
>> #CPUTYPE=soft
>> WITH_LIBCPLUSPLUS=
>> WITH_BINUTILS_BOOTSTRAP=
>> WITH_CLANG_BOOTSTRAP=
>> WITH_CLANG=
>> WITH_CLANG_IS_CC=
>> WITH_CLANG_FULL=
>> WITH_CLANG_EXTRAS=
>> WITH_LLDB=
>> #
>> WITH_BOOT=
>> WITHOUT_LIB32=
>> WITHOUT_LIBSOFT=
>> #
>> WITHOUT_ELFTOOLCHAIN_BOOTSTRAP=
>> WITHOUT_GCC_BOOTSTRAP=
>> WITHOUT_GCC=
>> WITHOUT_GCC_IS_CC=
>> WITHOUT_GNUCXX=
>> #
>> NO_WERROR=
>> #WERROR=
>> MALLOC_P

Re: ath (AR9460) no longer works after going to 11-STABLE r302483

2016-07-10 Thread Adrian Chadd
Hi,

That message just means a receive interrupt was handled whilst the NIC
was in reset, which is mostly fine.

Since you've reverted the ath driver directories without success, I'm
mostly out of simple ideas. I think you need to bisect the whole
kernel version until you find the commit that broke things.

Thanks!


-a
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: ath (AR9460) no longer works after going to 11-STABLE r302483

2016-07-10 Thread Wolfgang Zenker
* Wolfgang Zenker  [160710 19:47]:
> * Adrian Chadd  [160709 21:33]:
>> There weren't any changes in net80211 between those times, and ath(4)
>> only has some changes for locationing; nothing that should cause that
>> particular issue.

>> I'll go update to the latest -head on something with that NIC and test
>> it out some more.

>> Try reverting sys/dev/ath/ and sys/contrib/dev/ath/ back to r302387
>> and test? (but leave the rest of the kernel as-is.)

> using a kernel with sys/dev/ath/ and sys/contrib/dev/ath/ from r302387
> and the rest at r302483 the nic does not work either, same symptoms.

> Any other things to test?

I noticed one more thing: after a while I get this message on console:

ath0: ath_edma_recv_tasklet: sc_inreset_cnt > 0; skipping

Wolfgang
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: ath (AR9460) no longer works after going to 11-STABLE r302483

2016-07-10 Thread Wolfgang Zenker
Hi,

* Adrian Chadd  [160709 21:33]:
> There weren't any changes in net80211 between those times, and ath(4)
> only has some changes for locationing; nothing that should cause that
> particular issue.

> I'll go update to the latest -head on something with that NIC and test
> it out some more.

> Try reverting sys/dev/ath/ and sys/contrib/dev/ath/ back to r302387
> and test? (but leave the rest of the kernel as-is.)

using a kernel with sys/dev/ath/ and sys/contrib/dev/ath/ from r302387
and the rest at r302483 the nic does not work either, same symptoms.

Any other things to test?

Wolfgang
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [Bug 210953] 11.0 -r302412 via powerpc64-xtoolchain-gcc fails to build: dev/ahci/ahci.c:288:22: error: unknown conversion type character 'b' in format; too many arguments for format

2016-07-10 Thread Ngie Cooper

> On Jul 9, 2016, at 23:03, Mark Millard  wrote:
> 
> On 2016-Jul-9, at 8:53 PM, Ngie Cooper  wrote:
> 
>>> On Jul 9, 2016, at 18:52, bugzilla-nore...@freebsd.org wrote:
>>> 
>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210953
>>> 
>>> Mark Millard  changed:
>> 
>> I accidentally committed this regression to kern.mk. I don't have bugzilla 
>> login right now. Please assign the bug to me and I'll mark it fixed for the 
>> rev I did it in with head and stable/10.
>> 
>> Thanks!
>> -Ngie
> 
> So far as I know I've no control over the Assignee field for any bug via 
> bugzilla. Someone that has such can do what Ngie requested and assign 210953 
> to him.
> 
> A rebuild based on -r302457 completed fine for the powerpc64-gcc use, 
> confirming Ngie's note, at least for 11.0-STABLE.
> 
> I've no stable/10 context and so have not tested there. Otherwise I might 
> have just classified the report as "Overcome By Events" myself.
> 
> I have added a comment to 210953 about my rebuild test and Ngie's material 
> above.

stable/10 was a typo. I meant stable/11.

No worries.. I'll take care of it when I get home soon.

Thanks!

> 
> ===
> Mark Millard
> markmi at dsl-only.net
> 
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: GOST in OPENSSL_BASE

2016-07-10 Thread Andrey Chernov
On 10.07.2016 18:28, Andrey Chernov wrote:
> On 10.07.2016 18:13, Andrey Chernov wrote:
>> On 10.07.2016 18:12, Andrey Chernov wrote:
>>> On 10.07.2016 18:01, Slawa Olhovchenkov wrote:
 On Sun, Jul 10, 2016 at 05:10:04PM +0300, Andrey Chernov wrote:

> On 10.07.2016 16:30, Slawa Olhovchenkov wrote:
>> I am surprised lack of support GOST in openssl-base.
>> Can be this enabled before 11.0 released?
>
> AFAIK openssl maintainers says something like they can't support this
> code and it will become rotten shortly with new changes, so they drop it.
>

 Upstream or FreeBSD maintainers?

>>>
>>> Openssl maintainers.
>>>
>> I.e. upstream.
>>
> They mean built-in one, dropped from openssl 1.1.0 and above. It is
> still available as 3rd party at:
> https://github.com/gost-engine/engine
> 

>From their Changelog:
*) The GOST engine was out of date and therefore it has been removed. An
up to date GOST engine is now being maintained in an external
repository. See: https://wiki.openssl.org/index.php/Binaries. Libssl
still retains support for GOST ciphersuites (these are only activated if
a GOST engine is present).
[Matt Caswell]


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: GOST in OPENSSL_BASE

2016-07-10 Thread Andrey Chernov
On 10.07.2016 18:13, Andrey Chernov wrote:
> On 10.07.2016 18:12, Andrey Chernov wrote:
>> On 10.07.2016 18:01, Slawa Olhovchenkov wrote:
>>> On Sun, Jul 10, 2016 at 05:10:04PM +0300, Andrey Chernov wrote:
>>>
 On 10.07.2016 16:30, Slawa Olhovchenkov wrote:
> I am surprised lack of support GOST in openssl-base.
> Can be this enabled before 11.0 released?

 AFAIK openssl maintainers says something like they can't support this
 code and it will become rotten shortly with new changes, so they drop it.

>>>
>>> Upstream or FreeBSD maintainers?
>>>
>>
>> Openssl maintainers.
>>
> I.e. upstream.
> 
They mean built-in one, dropped from openssl 1.1.0 and above. It is
still available as 3rd party at:
https://github.com/gost-engine/engine



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: GOST in OPENSSL_BASE

2016-07-10 Thread Andrey Chernov
On 10.07.2016 18:12, Andrey Chernov wrote:
> On 10.07.2016 18:01, Slawa Olhovchenkov wrote:
>> On Sun, Jul 10, 2016 at 05:10:04PM +0300, Andrey Chernov wrote:
>>
>>> On 10.07.2016 16:30, Slawa Olhovchenkov wrote:
 I am surprised lack of support GOST in openssl-base.
 Can be this enabled before 11.0 released?
>>>
>>> AFAIK openssl maintainers says something like they can't support this
>>> code and it will become rotten shortly with new changes, so they drop it.
>>>
>>
>> Upstream or FreeBSD maintainers?
>>
> 
> Openssl maintainers.
> 
I.e. upstream.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: GOST in OPENSSL_BASE

2016-07-10 Thread Andrey Chernov
On 10.07.2016 18:01, Slawa Olhovchenkov wrote:
> On Sun, Jul 10, 2016 at 05:10:04PM +0300, Andrey Chernov wrote:
> 
>> On 10.07.2016 16:30, Slawa Olhovchenkov wrote:
>>> I am surprised lack of support GOST in openssl-base.
>>> Can be this enabled before 11.0 released?
>>
>> AFAIK openssl maintainers says something like they can't support this
>> code and it will become rotten shortly with new changes, so they drop it.
>>
> 
> Upstream or FreeBSD maintainers?
> 

Openssl maintainers.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: GOST in OPENSSL_BASE

2016-07-10 Thread Slawa Olhovchenkov
On Sun, Jul 10, 2016 at 05:10:04PM +0300, Andrey Chernov wrote:

> On 10.07.2016 16:30, Slawa Olhovchenkov wrote:
> > I am surprised lack of support GOST in openssl-base.
> > Can be this enabled before 11.0 released?
> 
> AFAIK openssl maintainers says something like they can't support this
> code and it will become rotten shortly with new changes, so they drop it.
> 

Upstream or FreeBSD maintainers?
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


ipfw sched show output mangled (PR/bug 210882)

2016-07-10 Thread Guido Falsi
Hi,

I have filed the PR in the subject [1], I'd like to ask if someone could
have a look, it's a bug I noticed on head which is also going to affect
11.0 if not corrected in time.

Since r270424, the output of certain ipfw show commands is mangled, due
to buffers not being cleaned between uses. The PR contains an example of
mangled "ipfw sched show" command.

The PR has two patches which together restore the output as it used to
be before r270424.

Is there someone available to review and commit it to the tree?

Thanks in adance!


[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210882

-- 
Guido Falsi 
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] microoptimize locking primitives by introducing randomized delay between atomic ops

2016-07-10 Thread Ian Lepore
On Sun, 2016-07-10 at 13:13 +0200, Mateusz Guzik wrote:
> If the lock is contended, primitives like __mtx_lock_sleep will spin
> checking if the owner is running or the lock was freed. The problem
> is
> that once it is discovered that the lock is free, multiple CPUs are
> likely to try to do the atomic op which will make it more costly for
> everyone and throughput suffers.
> 
> The standard thing to do is to have some sort of a randomized delay
> so
> that this kind of behaviour is reduced.
> 
> As such, below is a trivial hack which takes cpu_ticks() into account
> and performs % 2048, which in my testing gives reasonbly good
> results.
> 
> Please note there is definitely way more room for improvement in
> general.
> 
> In terms of results, there was no statistically significant change in
> -j 40 buildworld nor buildkernel.
> 
> However, a 40-way find on a ports tree placed on tmpfs yielded the
> following:
> 
> x vanilla
> + patched
> +
> +
> > + x
> >  x x x  |
> > + ++++  +  + ++   +   + x   x 
> >  x   x x x|
> >|_MA__| 
> >  |AM__| |
> +
> +
> N   Min   MaxMedian   Avg   
>  Stddev
> x  2012.43115.95214.897   14.7444   
>  0.74241657
> +  20 8.10311.8639.0135   9.44565
>  1.0059484
> Difference at 95.0% confidence
>   -5.29875 +/- 0.565836
>   -35.9374% +/- 3.83764%
>   (Student's t, pooled s = 0.884057)
> 
> The patch:
[...]

What about platforms that don't have a useful implementation of
cpu_ticks()?

What about platforms that don't suffer the large expense for atomic ops
that x86 apparently does?

-- Ian

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: GOST in OPENSSL_BASE

2016-07-10 Thread Andrey Chernov
On 10.07.2016 16:30, Slawa Olhovchenkov wrote:
> I am surprised lack of support GOST in openssl-base.
> Can be this enabled before 11.0 released?

AFAIK openssl maintainers says something like they can't support this
code and it will become rotten shortly with new changes, so they drop it.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


GOST in OPENSSL_BASE

2016-07-10 Thread Slawa Olhovchenkov
I am surprised lack of support GOST in openssl-base.
Can be this enabled before 11.0 released?

Subject: svn commit: r412619 - in head/dns: bind9-devel bind910 bind99

Author: mat
Date: Wed Apr  6 13:53:09 2016
New Revision: 412619
URL: https://svnweb.freebsd.org/changeset/ports/412619

Log:
  Stop bringing in OpenSSL from ports, it builds fine with the base one on
  9, and WITH_OPENSSL_PORT does not belong in a port's Makefile anyway.
  
  Not bumping PORTREVISION because:
  - if you are building with poudriere, it will detect that a dependency
has changed and rebuild it.
  - if you are building from ports, you will have OpenSSL from ports
installed, and it will choose to use it.
  
  Sponsored by: Absolight

+.include 
+
+.if ( ${PORT_OPTIONS:MGOST} || ${PORT_OPTIONS:MGOST_ASN1} ) && 
defined(WITH_OPENSSL_BASE)
+BROKEN=OpenSSL from the base system does not support GOST, add \
+   WITH_OPENSSL_PORT=yes to your /etc/make.conf and rebuild everything \
+   that needs SSL.
+.endif
+
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] microoptimize locking primitives by introducing randomized delay between atomic ops

2016-07-10 Thread Mateusz Guzik
On Sun, Jul 10, 2016 at 03:22:47PM +0300, Konstantin Belousov wrote:
> On Sun, Jul 10, 2016 at 01:13:26PM +0200, Mateusz Guzik wrote:
> > If the lock is contended, primitives like __mtx_lock_sleep will spin
> > checking if the owner is running or the lock was freed. The problem is
> > that once it is discovered that the lock is free, multiple CPUs are
> > likely to try to do the atomic op which will make it more costly for
> > everyone and throughput suffers.
> > 
> > The standard thing to do is to have some sort of a randomized delay so
> > that this kind of behaviour is reduced.
> > 
> > As such, below is a trivial hack which takes cpu_ticks() into account
> > and performs % 2048, which in my testing gives reasonbly good results.
> > 
> > Please note there is definitely way more room for improvement in general.
> > 
> > In terms of results, there was no statistically significant change in
> > -j 40 buildworld nor buildkernel.
> > 
> > However, a 40-way find on a ports tree placed on tmpfs yielded the 
> > following:
> 
> I am curious why did you added randomizer to sx adaptive loop but not to
> lockmgr loop, and probably most important, to the spinlocks (unless I
> misread the patch).

This is a simple first step where I modified loops which I suspect
benefit the most from the trivial change.

lockmgr and other places do have loops with a configurable but the same
delay for everyone. So they are already somewhat randomized by the time
of the arrival in the primitive.

On the other hand loops modified in the patch all check for the
availability of the lock without constant delay and thus are
significantly more suspectible to trying to grab it at the same time.

That said, I do intent to take care of the "static" loops as well later.

Meanwhile, running:
time (find . -depth 2 -type dir | xargs -n 1 -P 40 -I DIR make -C DIR
build-depends-list > /dev/null 2> /dev/null)

drops from ~4:00 minutes to ~2:42 with the patch applied,

-- 
Mateusz Guzik 
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] microoptimize locking primitives by introducing randomized delay between atomic ops

2016-07-10 Thread Konstantin Belousov
On Sun, Jul 10, 2016 at 01:13:26PM +0200, Mateusz Guzik wrote:
> If the lock is contended, primitives like __mtx_lock_sleep will spin
> checking if the owner is running or the lock was freed. The problem is
> that once it is discovered that the lock is free, multiple CPUs are
> likely to try to do the atomic op which will make it more costly for
> everyone and throughput suffers.
> 
> The standard thing to do is to have some sort of a randomized delay so
> that this kind of behaviour is reduced.
> 
> As such, below is a trivial hack which takes cpu_ticks() into account
> and performs % 2048, which in my testing gives reasonbly good results.
> 
> Please note there is definitely way more room for improvement in general.
> 
> In terms of results, there was no statistically significant change in
> -j 40 buildworld nor buildkernel.
> 
> However, a 40-way find on a ports tree placed on tmpfs yielded the following:

I am curious why did you added randomizer to sx adaptive loop but not to
lockmgr loop, and probably most important, to the spinlocks (unless I
misread the patch).
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


[PATCH] microoptimize locking primitives by introducing randomized delay between atomic ops

2016-07-10 Thread Mateusz Guzik
If the lock is contended, primitives like __mtx_lock_sleep will spin
checking if the owner is running or the lock was freed. The problem is
that once it is discovered that the lock is free, multiple CPUs are
likely to try to do the atomic op which will make it more costly for
everyone and throughput suffers.

The standard thing to do is to have some sort of a randomized delay so
that this kind of behaviour is reduced.

As such, below is a trivial hack which takes cpu_ticks() into account
and performs % 2048, which in my testing gives reasonbly good results.

Please note there is definitely way more room for improvement in general.

In terms of results, there was no statistically significant change in
-j 40 buildworld nor buildkernel.

However, a 40-way find on a ports tree placed on tmpfs yielded the following:

x vanilla
+ patched
++
| + x x 
x x  |
|+ ++++  +  + ++   +   + x   x  x   
x x x|
||_MA__|  
|AM__| |
++
N   Min   MaxMedian   AvgStddev
x  2012.43115.95214.897   14.74440.74241657
+  20 8.10311.8639.0135   9.44565 1.0059484
Difference at 95.0% confidence
-5.29875 +/- 0.565836
-35.9374% +/- 3.83764%
(Student's t, pooled s = 0.884057)

The patch:

diff --git a/sys/kern/kern_mutex.c b/sys/kern/kern_mutex.c
index 012cf7c..db3b950 100644
--- a/sys/kern/kern_mutex.c
+++ b/sys/kern/kern_mutex.c
@@ -444,7 +444,7 @@ __mtx_lock_sleep(volatile uintptr_t *c, uintptr_t tid, int 
opts,
m->lock_object.lo_name);
while (mtx_owner(m) == owner &&
TD_IS_RUNNING(owner)) {
-   cpu_spinwait();
+   lock_delay();
 #ifdef KDTRACE_HOOKS
spin_cnt++;
 #endif
diff --git a/sys/kern/kern_rwlock.c b/sys/kern/kern_rwlock.c
index 6a904d2..3b512b5 100644
--- a/sys/kern/kern_rwlock.c
+++ b/sys/kern/kern_rwlock.c
@@ -438,7 +438,7 @@ __rw_rlock(volatile uintptr_t *c, const char *file, int 
line)
"lockname:\"%s\"", rw->lock_object.lo_name);
while ((struct thread*)RW_OWNER(rw->rw_lock) ==
owner && TD_IS_RUNNING(owner)) {
-   cpu_spinwait();
+   lock_delay();
 #ifdef KDTRACE_HOOKS
spin_cnt++;
 #endif
@@ -799,7 +799,7 @@ __rw_wlock_hard(volatile uintptr_t *c, uintptr_t tid, const 
char *file,
rw->lock_object.lo_name);
while ((struct thread*)RW_OWNER(rw->rw_lock) == owner &&
TD_IS_RUNNING(owner)) {
-   cpu_spinwait();
+   lock_delay();
 #ifdef KDTRACE_HOOKS
spin_cnt++;
 #endif
diff --git a/sys/kern/kern_sx.c b/sys/kern/kern_sx.c
index 2a81c04..1bfd46f 100644
--- a/sys/kern/kern_sx.c
+++ b/sys/kern/kern_sx.c
@@ -579,7 +579,7 @@ _sx_xlock_hard(struct sx *sx, uintptr_t tid, int opts, 
const char *file,
GIANT_SAVE();
while (SX_OWNER(sx->sx_lock) == x &&
TD_IS_RUNNING(owner)) {
-   cpu_spinwait();
+   lock_delay();
 #ifdef KDTRACE_HOOKS
spin_cnt++;
 #endif
@@ -889,10 +889,10 @@ _sx_slock_hard(struct sx *sx, int opts, const char *file, 
int line)
GIANT_SAVE();
while (SX_OWNER(sx->sx_lock) == x &&
TD_IS_RUNNING(owner)) {
+   lock_delay();
 #ifdef KDTRACE_HOOKS
spin_cnt++;
 #endif
-   cpu_spinwait();
}
KTR_STATE0(KTR_SCHED, "thread",
sched_tdname(curthread), "running");
diff --git a/sys/kern/subr_lock.c b/sys/kern/subr_lock.c
index e78d5a9..55a0fb7 100644
--- a/sys/kern/subr_lock.c
+++ b/sys/kern/subr_lock.c
@@ -103,6 +103,16 @@ lock_destroy(struct lock_object *lock)
lock->lo_flags &= ~LO_INITIALIZED;
 }
 
+void
+lock_delay(void)
+{
+   int i, delay;
+
+   delay = cpu_ticks() 

Re: ls eat high CPU time when LANG=zh_CN.UTF-8 and LC_ALL=zh_CN.UTF-8

2016-07-10 Thread Baptiste Daroussin
On Sun, Jul 10, 2016 at 04:23:03PM +0800, Huang Wen Hui wrote:
> I can reproduce it on a clean 11.0-BETA1 VM.
> 
> 
> 2016-07-09 9:03 GMT+08:00 Huang Wen Hui :
> 
> > For some reasons, r302324 seems not include in 11.0-ALPHA6?
> >
> > 2016-07-09 8:52 GMT+08:00 Huang Wen Hui :
> >
> >> Revert back r302324, Chinese locale problem is gone.
> >>
> >> Cheers
> >> Huang Wen Hui
> >>
Thanks, I can reproduce now, I'm looking into it.

Bapt


signature.asc
Description: PGP signature


Re: ls eat high CPU time when LANG=zh_CN.UTF-8 and LC_ALL=zh_CN.UTF-8

2016-07-10 Thread Huang Wen Hui
I can reproduce it on a clean 11.0-BETA1 VM.


2016-07-09 9:03 GMT+08:00 Huang Wen Hui :

> For some reasons, r302324 seems not include in 11.0-ALPHA6?
>
> 2016-07-09 8:52 GMT+08:00 Huang Wen Hui :
>
>> Revert back r302324, Chinese locale problem is gone.
>>
>> Cheers
>> Huang Wen Hui
>>
>> 2016-07-05 16:50 GMT+08:00 Baptiste Daroussin :
>>
>>> On Tue, Jul 05, 2016 at 12:16:42PM +0800, Huang Wen Hui wrote:
>>> > These 2 files can make ls suck:
>>> >
>>> > touch 火灾1
>>> > touch 火灾2
>>> >
>>> > 2 files start with 2 same Chinese chars.
>>> >
>>> I cannot reproduce on my head laptop, neither on a clean 11.0-ALPHA6
>>> jail.
>>>
>>> I'll try on a clean 11.0-ALPHA6 VM
>>>
>>> Best regards,
>>> Bapt
>>>
>>
>>
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"