z/VM 5.3 has a Rexx STSI() function. TSO Rexx does not, even at z/OS 1.9. Running APF-authorized this code (or something like it):
MODESET MODE=SUP SLR 0,0 STSI 0(0) MODESET MODE=PROB should return a '30000000' in R0 if z/OS is running as a guest. I don't have a second-level z/OS to try it on but it does return a '20000000' for z/OS in an LPAR.