Re: svn commit: r367577 - in head: share/mk sys/conf tools/build/options
On Tue, Nov 10, 2020 at 02:44:45PM -0500, Shawn Webb wrote: > On Tue, Nov 10, 2020 at 07:17:29PM +, Brooks Davis wrote: > > On Tue, Nov 10, 2020 at 07:15:14PM +, Brooks Davis wrote: > > > Author: brooks > > > Date: Tue Nov 10 19:15:13 2020 > > > New Revision: 367577 > > > URL: https://svnweb.freebsd.org/changeset/base/367577 > > > > > > Log: > > > Support initializing stack variables on function entry > > > > > > There are two options: > > >- WITH_INIT_ALL_ZERO: Zero all variables on the stack. > > >- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined > > > patterns. > > > > > > The exact pattern are a compiler implementation detail and vary by type. > > > They are somewhat documented in the LLVM commit message: > > > https://reviews.llvm.org/rL349442 > > > I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather > > > than naming them after the LLVM specific compiler flags. > > > > > > In a range of consumer products, options like these are used in > > > both debug and production builds with debugs builds using patterns > > > (intended to provoke crashes on use of uninitialized values) and > > > production using zeros (deemed more likely to lead to harmless > > > misbehavior or NULL-pointer dereferences). > > > > We've tested this extensively in CheriBSD on RISC-V, in the wild it's > > probably most tested on Arm64 and x86. > > > > Despite the silly compiler flag you'll spot in the code, the zeroing > > option isn't going away in practice as Apple, Google, and Microsoft all > > ship with this feature in some of their products. > > HardenedBSD's testing of this last year on amd64 have (privately) > shown the feature to really hinder performance on more complex > applications (like when applied to clang/lld). A build of base > without init all zero applied to clang/lld would take around 1.5 > hours on my system. A build with it applied to clang/lld took around > four hours, if my memory serves correctly. I would probably advise > against applying it system-wide. But YMMV. I agree a more nuanced approach is likely useful in practice, but this does work and is part of the configuration we shipped for DARPA's FETT bug bounty. Hopefully this provides a starting point for further exploration. -- Brooks signature.asc Description: PGP signature
Re: svn commit: r367577 - in head: share/mk sys/conf tools/build/options
Sorry about that. I've fixed it in r367579. -- Brooks On Tue, Nov 10, 2020 at 12:46:45PM -0800, Matthew Macy wrote: > These flags aren't defined by default when building external kernel modules: > > gmake[2]: Entering directory '/usr/home/matt/devel/ZoF/module' > env -u MAKEFLAGS make -C /home/matt/devel/ZoF/module -f Makefile.bsd -w > make[3]: Entering directory `/home/matt/devel/ZoF/module' > make[3]: "/usr/home/matt/devel/freebsd/sys/conf/kern.mk" line 233: > Malformed conditional (${MK_INIT_ALL_ZERO} == "yes") > make[3]: Fatal errors encountered -- cannot continue > make[3]: stopped in /home/matt/devel/ZoF/module > gmake[2]: *** [Makefile:53: modules-FreeBSD] Error 1 > gmake[2]: Leaving directory '/usr/home/matt/devel/ZoF/module' > > On Tue, Nov 10, 2020 at 11:15 AM Brooks Davis wrote: > > > > Author: brooks > > Date: Tue Nov 10 19:15:13 2020 > > New Revision: 367577 > > URL: https://svnweb.freebsd.org/changeset/base/367577 > > > > Log: > > Support initializing stack variables on function entry > > > > There are two options: > >- WITH_INIT_ALL_ZERO: Zero all variables on the stack. > >- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns. > > > > The exact pattern are a compiler implementation detail and vary by type. > > They are somewhat documented in the LLVM commit message: > > https://reviews.llvm.org/rL349442 > > I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather > > than naming them after the LLVM specific compiler flags. > > > > In a range of consumer products, options like these are used in > > both debug and production builds with debugs builds using patterns > > (intended to provoke crashes on use of uninitialized values) and > > production using zeros (deemed more likely to lead to harmless > > misbehavior or NULL-pointer dereferences). > > > > Reviewed by: emaste > > Obtained from:CheriBSD > > Sponsored by: DARPA > > Differential Revision:https://reviews.freebsd.org/D27131 > > > > Added: > > head/tools/build/options/WITH_INIT_ALL_PATTERN (contents, props changed) > > head/tools/build/options/WITH_INIT_ALL_ZERO (contents, props changed) > > Modified: > > head/share/mk/bsd.compiler.mk > > head/share/mk/bsd.lib.mk > > head/share/mk/bsd.opts.mk > > head/share/mk/bsd.prog.mk > > head/sys/conf/kern.mk > > > > Modified: head/share/mk/bsd.compiler.mk > > == > > --- head/share/mk/bsd.compiler.mk Tue Nov 10 19:09:35 2020 > > (r367576) > > +++ head/share/mk/bsd.compiler.mk Tue Nov 10 19:15:13 2020 > > (r367577) > > @@ -24,6 +24,7 @@ > > # - c++11: supports full (or nearly full) C++11 programming > > environment. > > # - retpoline: supports the retpoline speculative execution vulnerability > > # mitigation. > > +# - init-all: supports stack variable initialization. > > # > > # These variables with an X_ prefix will also be provided if XCC is set. > > # > > @@ -214,7 +215,7 @@ ${X_}COMPILER_FEATURES= c++11 c++14 > > ${X_}COMPILER_FEATURES+= c++17 > > .endif > > .if ${${X_}COMPILER_TYPE} == "clang" > > -${X_}COMPILER_FEATURES+= retpoline > > +${X_}COMPILER_FEATURES+= retpoline init-all > > .endif > > > > .else > > > > Modified: head/share/mk/bsd.lib.mk > > == > > --- head/share/mk/bsd.lib.mkTue Nov 10 19:09:35 2020(r367576) > > +++ head/share/mk/bsd.lib.mkTue Nov 10 19:15:13 2020(r367577) > > @@ -85,6 +85,25 @@ LDFLAGS+= -Wl,-zretpolineplt > > .endif > > .endif > > > > +# Initialize stack variables on function entry > > +.if ${MK_INIT_ALL_ZERO} == "yes" > > +.if ${COMPILER_FEATURES:Minit-all} > > +CFLAGS+= -ftrivial-auto-var-init=zero \ > > + > > -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang > > +CXXFLAGS+= -ftrivial-auto-var-init=zero \ > > + > > -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang > > +.else > > +.warning InitAll (zeros) requested but not support by compiler > > +.endif > > +.elif ${MK_INIT_ALL_PATTERN} == "yes" > > +.if ${COMPILER_FEATURES:Minit-all} > > +CFLAGS+= -ftrivial-auto-var-init=pattern > > +CXXFLAGS+= -ftrivial-auto-var-init=pattern > > +.else > > +.warning InitAll (pattern) requested but not support by compiler > > +.endif > > +.endif > > + > > .if ${MK_DEBUG_FILES} != "no" && empty(DEBUG_FLAGS:M-g) && \ > > empty(DEBUG_FLAGS:M-gdwarf*) > > CFLAGS+= ${DEBUG_FILES_CFLAGS} > > > > Modified: head/share/mk/bsd.opts.mk > > == > > --- head/share/mk/bsd.opts.mk Tue Nov 10 19:09:35 2020(r367576) > > +++ head/share/mk/bsd.opts.mk Tue Nov 10 19:15:13 2020(r367577) > > @@ -71,6 +71,8 @@ __DEFAULT_NO_OPTIONS = \ > > BIND_NOW \ > > CCACHE_BUIL
Re: svn commit: r367577 - in head: share/mk sys/conf tools/build/options
These flags aren't defined by default when building external kernel modules: gmake[2]: Entering directory '/usr/home/matt/devel/ZoF/module' env -u MAKEFLAGS make -C /home/matt/devel/ZoF/module -f Makefile.bsd -w make[3]: Entering directory `/home/matt/devel/ZoF/module' make[3]: "/usr/home/matt/devel/freebsd/sys/conf/kern.mk" line 233: Malformed conditional (${MK_INIT_ALL_ZERO} == "yes") make[3]: Fatal errors encountered -- cannot continue make[3]: stopped in /home/matt/devel/ZoF/module gmake[2]: *** [Makefile:53: modules-FreeBSD] Error 1 gmake[2]: Leaving directory '/usr/home/matt/devel/ZoF/module' On Tue, Nov 10, 2020 at 11:15 AM Brooks Davis wrote: > > Author: brooks > Date: Tue Nov 10 19:15:13 2020 > New Revision: 367577 > URL: https://svnweb.freebsd.org/changeset/base/367577 > > Log: > Support initializing stack variables on function entry > > There are two options: >- WITH_INIT_ALL_ZERO: Zero all variables on the stack. >- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns. > > The exact pattern are a compiler implementation detail and vary by type. > They are somewhat documented in the LLVM commit message: > https://reviews.llvm.org/rL349442 > I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather > than naming them after the LLVM specific compiler flags. > > In a range of consumer products, options like these are used in > both debug and production builds with debugs builds using patterns > (intended to provoke crashes on use of uninitialized values) and > production using zeros (deemed more likely to lead to harmless > misbehavior or NULL-pointer dereferences). > > Reviewed by: emaste > Obtained from:CheriBSD > Sponsored by: DARPA > Differential Revision:https://reviews.freebsd.org/D27131 > > Added: > head/tools/build/options/WITH_INIT_ALL_PATTERN (contents, props changed) > head/tools/build/options/WITH_INIT_ALL_ZERO (contents, props changed) > Modified: > head/share/mk/bsd.compiler.mk > head/share/mk/bsd.lib.mk > head/share/mk/bsd.opts.mk > head/share/mk/bsd.prog.mk > head/sys/conf/kern.mk > > Modified: head/share/mk/bsd.compiler.mk > == > --- head/share/mk/bsd.compiler.mk Tue Nov 10 19:09:35 2020 > (r367576) > +++ head/share/mk/bsd.compiler.mk Tue Nov 10 19:15:13 2020 > (r367577) > @@ -24,6 +24,7 @@ > # - c++11: supports full (or nearly full) C++11 programming environment. > # - retpoline: supports the retpoline speculative execution vulnerability > # mitigation. > +# - init-all: supports stack variable initialization. > # > # These variables with an X_ prefix will also be provided if XCC is set. > # > @@ -214,7 +215,7 @@ ${X_}COMPILER_FEATURES= c++11 c++14 > ${X_}COMPILER_FEATURES+= c++17 > .endif > .if ${${X_}COMPILER_TYPE} == "clang" > -${X_}COMPILER_FEATURES+= retpoline > +${X_}COMPILER_FEATURES+= retpoline init-all > .endif > > .else > > Modified: head/share/mk/bsd.lib.mk > == > --- head/share/mk/bsd.lib.mkTue Nov 10 19:09:35 2020(r367576) > +++ head/share/mk/bsd.lib.mkTue Nov 10 19:15:13 2020(r367577) > @@ -85,6 +85,25 @@ LDFLAGS+= -Wl,-zretpolineplt > .endif > .endif > > +# Initialize stack variables on function entry > +.if ${MK_INIT_ALL_ZERO} == "yes" > +.if ${COMPILER_FEATURES:Minit-all} > +CFLAGS+= -ftrivial-auto-var-init=zero \ > +-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang > +CXXFLAGS+= -ftrivial-auto-var-init=zero \ > +-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang > +.else > +.warning InitAll (zeros) requested but not support by compiler > +.endif > +.elif ${MK_INIT_ALL_PATTERN} == "yes" > +.if ${COMPILER_FEATURES:Minit-all} > +CFLAGS+= -ftrivial-auto-var-init=pattern > +CXXFLAGS+= -ftrivial-auto-var-init=pattern > +.else > +.warning InitAll (pattern) requested but not support by compiler > +.endif > +.endif > + > .if ${MK_DEBUG_FILES} != "no" && empty(DEBUG_FLAGS:M-g) && \ > empty(DEBUG_FLAGS:M-gdwarf*) > CFLAGS+= ${DEBUG_FILES_CFLAGS} > > Modified: head/share/mk/bsd.opts.mk > == > --- head/share/mk/bsd.opts.mk Tue Nov 10 19:09:35 2020(r367576) > +++ head/share/mk/bsd.opts.mk Tue Nov 10 19:15:13 2020(r367577) > @@ -71,6 +71,8 @@ __DEFAULT_NO_OPTIONS = \ > BIND_NOW \ > CCACHE_BUILD \ > CTF \ > +INIT_ALL_PATTERN \ > +INIT_ALL_ZERO \ > INSTALL_AS_USER \ > PIE \ > RETPOLINE \ > @@ -84,6 +86,10 @@ __DEFAULT_DEPENDENT_OPTIONS = \ > > > .include > + > +.if ${MK_INIT_ALL_PATTERN} == "yes" && ${MK_INIT_ALL_ZERO} == "yes" > +.error WITH_INIT_ALL_PATTERN and WITH_INIT_ALL_ZERO are mutually exclusive. > +.endif > > #
Re: svn commit: r367577 - in head: share/mk sys/conf tools/build/options
On Tue, Nov 10, 2020 at 07:17:29PM +, Brooks Davis wrote: > On Tue, Nov 10, 2020 at 07:15:14PM +, Brooks Davis wrote: > > Author: brooks > > Date: Tue Nov 10 19:15:13 2020 > > New Revision: 367577 > > URL: https://svnweb.freebsd.org/changeset/base/367577 > > > > Log: > > Support initializing stack variables on function entry > > > > There are two options: > >- WITH_INIT_ALL_ZERO: Zero all variables on the stack. > >- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns. > > > > The exact pattern are a compiler implementation detail and vary by type. > > They are somewhat documented in the LLVM commit message: > > https://reviews.llvm.org/rL349442 > > I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather > > than naming them after the LLVM specific compiler flags. > > > > In a range of consumer products, options like these are used in > > both debug and production builds with debugs builds using patterns > > (intended to provoke crashes on use of uninitialized values) and > > production using zeros (deemed more likely to lead to harmless > > misbehavior or NULL-pointer dereferences). > > We've tested this extensively in CheriBSD on RISC-V, in the wild it's > probably most tested on Arm64 and x86. > > Despite the silly compiler flag you'll spot in the code, the zeroing > option isn't going away in practice as Apple, Google, and Microsoft all > ship with this feature in some of their products. HardenedBSD's testing of this last year on amd64 have (privately) shown the feature to really hinder performance on more complex applications (like when applied to clang/lld). A build of base without init all zero applied to clang/lld would take around 1.5 hours on my system. A build with it applied to clang/lld took around four hours, if my memory serves correctly. I would probably advise against applying it system-wide. But YMMV. Thanks, -- Shawn Webb Cofounder / Security Engineer HardenedBSD GPG Key ID: 0xFF2E67A277F8E1FA GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9 3633 C85B 0AF8 AB23 0FB2 https://git-01.md.hardenedbsd.org/HardenedBSD/pubkeys/src/branch/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc signature.asc Description: PGP signature
Re: svn commit: r367577 - in head: share/mk sys/conf tools/build/options
On Tue, Nov 10, 2020 at 07:15:14PM +, Brooks Davis wrote: > Author: brooks > Date: Tue Nov 10 19:15:13 2020 > New Revision: 367577 > URL: https://svnweb.freebsd.org/changeset/base/367577 > > Log: > Support initializing stack variables on function entry > > There are two options: >- WITH_INIT_ALL_ZERO: Zero all variables on the stack. >- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns. > > The exact pattern are a compiler implementation detail and vary by type. > They are somewhat documented in the LLVM commit message: > https://reviews.llvm.org/rL349442 > I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather > than naming them after the LLVM specific compiler flags. > > In a range of consumer products, options like these are used in > both debug and production builds with debugs builds using patterns > (intended to provoke crashes on use of uninitialized values) and > production using zeros (deemed more likely to lead to harmless > misbehavior or NULL-pointer dereferences). We've tested this extensively in CheriBSD on RISC-V, in the wild it's probably most tested on Arm64 and x86. Despite the silly compiler flag you'll spot in the code, the zeroing option isn't going away in practice as Apple, Google, and Microsoft all ship with this feature in some of their products. -- Brooks signature.asc Description: PGP signature