> My first try would be --with-build-sysroot. Does that fail in some way?
It's ignored without --with-sysroot, but if you use --with-sysroot, the cross-built native *also* expects to use a sysroot, which means binutils must also be built with a sysroot, even if its "/".