Module Name: src Committed By: rillig Date: Tue Feb 23 14:17:21 UTC 2021
Modified Files: src/usr.bin/make/unit-tests: var-class-cmdline.exp var-class-cmdline.mk varmod-loop.exp varmod-loop.mk Log Message: make: demonstrate how to undefine variables during evaluation For a very long time now, I had thought that it would be impossible to undefine global variables during the evaluation of variable expressions. This is something that the memory management in Var_Parse relies upon, see the comment 'the value of the variable must not change'. After several unsuccessful attempts at referring to an already freed previous value of a variable, today I discovered how to unset a global variable while evaluating an expression, which has the same effect. To demonstrate that this use-after-free can reliably crash make, it would need a memory allocator with a debug mode that never re-allocates the same memory block after it has been used once. This is something that jemalloc cannot do at the moment. Valgrind would be another idea, but that has not been ported to NetBSD. Undefining a global variable while evaluating an expression is made possible by an implementation detail of the modifier ':@'. That modifier undefines the loop variable, without restoring its previous value, see ApplyModifier_Loop. By the very old conventions of ODE Make, these loop variables are named '.V.' and thus do not conflict with variables from other naming conventions. In NetBSD and pkgsrc, these loop variables are typically called 'var', sometimes '_var' with a leading underscore, which also doesn't conflict with the typical form 'VAR' of variables in the global namespace. Therefore, in practice these loop variables don't interfere with other variables. One case that can practically arise is when an outer variable has a modifier ':@word@${VAR.${word}}@' and one of the referenced variables uses the same variable name in the modifier, see varmod-loop.mk 1.10 line 91 for a detailed explanation. By using the ${:@VAR@@} modifier in a place that is evaluated with cmdline scope, it is not only possible to undefine global variables, it is possible to undefine cmdline variables as well. When evaluated in a specific make target, the expression ${:@\@@@} can even be used to undefine the variable '.TARGET', which will probably crash make with an assertion failure. To generate a diff of this commit: cvs rdiff -u -r1.2 -r1.3 src/usr.bin/make/unit-tests/var-class-cmdline.exp cvs rdiff -u -r1.3 -r1.4 src/usr.bin/make/unit-tests/var-class-cmdline.mk cvs rdiff -u -r1.5 -r1.6 src/usr.bin/make/unit-tests/varmod-loop.exp cvs rdiff -u -r1.9 -r1.10 src/usr.bin/make/unit-tests/varmod-loop.mk Please note that diffs are not public domain; they are subject to the copyright notices on the relevant files.
Modified files: Index: src/usr.bin/make/unit-tests/var-class-cmdline.exp diff -u src/usr.bin/make/unit-tests/var-class-cmdline.exp:1.2 src/usr.bin/make/unit-tests/var-class-cmdline.exp:1.3 --- src/usr.bin/make/unit-tests/var-class-cmdline.exp:1.2 Mon Feb 22 22:04:28 2021 +++ src/usr.bin/make/unit-tests/var-class-cmdline.exp Tue Feb 23 14:17:21 2021 @@ -1,4 +1,4 @@ make: "var-class-cmdline.mk" line 23: global -make: "var-class-cmdline.mk" line 30: makeflags +make: "var-class-cmdline.mk" line 32: makeflags makeflags exit status 0 Index: src/usr.bin/make/unit-tests/var-class-cmdline.mk diff -u src/usr.bin/make/unit-tests/var-class-cmdline.mk:1.3 src/usr.bin/make/unit-tests/var-class-cmdline.mk:1.4 --- src/usr.bin/make/unit-tests/var-class-cmdline.mk:1.3 Mon Feb 22 22:04:28 2021 +++ src/usr.bin/make/unit-tests/var-class-cmdline.mk Tue Feb 23 14:17:21 2021 @@ -1,4 +1,4 @@ -# $NetBSD: var-class-cmdline.mk,v 1.3 2021/02/22 22:04:28 rillig Exp $ +# $NetBSD: var-class-cmdline.mk,v 1.4 2021/02/23 14:17:21 rillig Exp $ # # Tests for variables specified on the command line. # @@ -23,9 +23,11 @@ VAR= global .info ${VAR} # The global variable is "overridden" by simply deleting it and then -# installing the cmdline variable instead. Since there is no way to +# installing the cmdline variable instead. Since there is no obvious way to # undefine a cmdline variable, there is no need to remember the old value # of the global variable could become visible again. +# +# See varmod-loop.mk for a non-obvious way to undefine a cmdline variable. .MAKEFLAGS: VAR=makeflags .info ${VAR} Index: src/usr.bin/make/unit-tests/varmod-loop.exp diff -u src/usr.bin/make/unit-tests/varmod-loop.exp:1.5 src/usr.bin/make/unit-tests/varmod-loop.exp:1.6 --- src/usr.bin/make/unit-tests/varmod-loop.exp:1.5 Thu Nov 12 00:40:55 2020 +++ src/usr.bin/make/unit-tests/varmod-loop.exp Tue Feb 23 14:17:21 2021 @@ -1,10 +1,10 @@ -ParseReadLine (117): 'USE_8_DOLLARS= ${:U1:@var@${8_DOLLARS}@} ${8_DOLLARS} $$$$$$$$' +ParseReadLine (132): 'USE_8_DOLLARS= ${:U1:@var@${8_DOLLARS}@} ${8_DOLLARS} $$$$$$$$' CondParser_Eval: ${USE_8_DOLLARS} != "\$\$\$\$ \$\$\$\$ \$\$\$\$" lhs = "$$$$ $$$$ $$$$", rhs = "$$$$ $$$$ $$$$", op = != -ParseReadLine (122): 'SUBST_CONTAINING_LOOP:= ${USE_8_DOLLARS}' +ParseReadLine (137): 'SUBST_CONTAINING_LOOP:= ${USE_8_DOLLARS}' CondParser_Eval: ${SUBST_CONTAINING_LOOP} != "\$\$ \$\$\$\$ \$\$\$\$" lhs = "$$ $$$$ $$$$", rhs = "$$ $$$$ $$$$", op = != -ParseReadLine (147): '.MAKEFLAGS: -d0' +ParseReadLine (162): '.MAKEFLAGS: -d0' ParseDoDependency(.MAKEFLAGS: -d0) :+one+ +two+ +three+: :x1y x2y x3y: Index: src/usr.bin/make/unit-tests/varmod-loop.mk diff -u src/usr.bin/make/unit-tests/varmod-loop.mk:1.9 src/usr.bin/make/unit-tests/varmod-loop.mk:1.10 --- src/usr.bin/make/unit-tests/varmod-loop.mk:1.9 Thu Feb 4 21:42:47 2021 +++ src/usr.bin/make/unit-tests/varmod-loop.mk Tue Feb 23 14:17:21 2021 @@ -1,4 +1,4 @@ -# $NetBSD: varmod-loop.mk,v 1.9 2021/02/04 21:42:47 rillig Exp $ +# $NetBSD: varmod-loop.mk,v 1.10 2021/02/23 14:17:21 rillig Exp $ # # Tests for the :@var@...${var}...@ variable modifier. @@ -34,13 +34,26 @@ mod-loop-varname: @echo :${:U1 2 3:@\\@x${${:Ux:S,x,\\,}}y@}: # The variable name can technically be empty, and in this situation - # the variable value cannot be accessed since the empty variable is - # protected to always return an empty string. + # the variable value cannot be accessed since the empty "variable" + # is protected to always return an empty string. @echo empty: :${:U1 2 3:@@x${}y@}: -# The :@ modifier resolves the variables a little more often than expected. -# In particular, it resolves _all_ variables from the scope, and not only -# the loop variable (in this case v). + +# The :@ modifier resolves the variables from the replacement text once more +# than expected. In particular, it resolves _all_ variables from the scope, +# and not only the loop variable (in this case v). +SRCS= source +CFLAGS.source= before +ALL_CFLAGS:= ${SRCS:@src@${CFLAGS.${src}}@} # note the ':=' +CFLAGS.source+= after +.if ${ALL_CFLAGS} != "before" +. error +.endif + + +# In the following example, the modifier ':@' expands the '$$' to '$'. This +# means that when the resulting expression is evaluated, these resulting '$' +# will be interpreted as starting a subexpression. # # The d means direct reference, the i means indirect reference. RESOLVE= ${RES1} $${RES1} @@ -48,9 +61,11 @@ RES1= 1d${RES2} 1i$${RES2} RES2= 2d${RES3} 2i$${RES3} RES3= 3 +# TODO: convert to '.if'. mod-loop-resolve: @echo $@:${RESOLVE:@v@w${v}w@:Q}: + # Until 2020-07-20, the variable name of the :@ modifier could end with one # or two dollar signs, which were silently ignored. # There's no point in allowing a dollar sign in that position. @@ -145,3 +160,87 @@ SUBST_CONTAINING_LOOP:= ${USE_8_DOLLARS} . error .endif .MAKEFLAGS: -d0 + +# After looping over the words of the expression, the loop variable gets +# undefined. The modifier ':@' uses an ordinary global variable for this, +# which is different from the '.for' loop, which replaces ${var} with +# ${:Uvalue} in the body of the loop. This choice of implementation detail +# can be used for a nasty side effect. The expression ${:U:@VAR@@} evaluates +# to an empty string, plus it undefines the variable 'VAR'. This is the only +# possibility to undefine a global variable during evaluation. +GLOBAL= before-global +RESULT:= ${:U${GLOBAL} ${:U:@GLOBAL@@} ${GLOBAL:Uundefined}} +.if ${RESULT} != "before-global undefined" +. error +.endif + +# The above side effect of undefining a variable from a certain scope can be +# further combined with the otherwise undocumented implementation detail that +# the argument of an '.if' directive is evaluated in cmdline scope. Putting +# these together makes it possible to undefine variables from the cmdline +# scope, something that is not possible in a straight-forward way. +.MAKEFLAGS: CMDLINE=cmdline +.if ${:U${CMDLINE}${:U:@CMDLINE@@}} != "cmdline" +. error +.endif +# Now the cmdline variable got undefined. +.if ${CMDLINE} != "cmdline" +. error +.endif +# At this point, it still looks as if the cmdline variable were defined, +# since the value of CMDLINE is still "cmdline". That impression is only +# superficial though, the cmdline variable is actually deleted. To +# demonstrate this, it is now possible to override its value using a global +# variable, something that was not possible before: +CMDLINE= global +.if ${CMDLINE} != "global" +. error +.endif +# Now undefine that global variable again, to get back to the original value. +.undef CMDLINE +.if ${CMDLINE} != "cmdline" +. error +.endif +# What actually happened is that when CMDLINE was set by the '.MAKEFLAGS' +# target in the cmdline scope, that same variable was exported to the +# environment, see Var_SetWithFlags. +.unexport CMDLINE +.if ${CMDLINE} != "cmdline" +. error +.endif +# The above '.unexport' has no effect since UnexportVar requires a global +# variable of the same name to be defined, otherwise nothing is unexported. +CMDLINE= global +.unexport CMDLINE +.undef CMDLINE +.if ${CMDLINE} != "cmdline" +. error +.endif +# This still didn't work since there must not only be a global variable, the +# variable must be marked as exported as well, which it wasn't before. +CMDLINE= global +.export CMDLINE +.unexport CMDLINE +.undef CMDLINE +.if ${CMDLINE:Uundefined} != "undefined" +. error +.endif +# Finally the variable 'CMDLINE' from the cmdline scope is gone, and all its +# traces from the environment are gone as well. To do that, a global variable +# had to be defined and exported, something that is far from obvious. To +# recap, here is the essence of the above story: +.MAKEFLAGS: CMDLINE=cmdline # have a cmdline + environment variable +.if ${:U:@CMDLINE@@}} # undefine cmdline, keep environment +.endif +CMDLINE= global # needed for deleting the environment +.export CMDLINE # needed for deleting the environment +.unexport CMDLINE # delete the environment +.undef CMDLINE # delete the global helper variable +.if ${CMDLINE:Uundefined} != "undefined" +. error # 'CMDLINE' is gone now from all scopes +.endif + + +# TODO: Actually trigger the undefined behavior (use after free) that was +# already suspected in Var_Parse, in the comment 'the value of the variable +# must not change'.