On discussion with chs@, I realize that: (a) you could just use uao_create(INT64_MAX - PAGE_SIZE), like tmpfs does, and not bother with defining a uao_resize (just .pgo_put the pages when you truncate and update the length under the lock);
but (b) this is kind of duplicating the existing functionality of tmpfs, and it's a shame to have to do that. That said, I suspect it'll be easier to: 1. add .fo_truncate (which I think is reasonable to do anyway), and 2. duplicate the tmpfs functionality anyway just for memfd_create, than to go through the effort of finding a way to incorporate the Linux-style O_TMPFILE and linkat(..., AT_EMPTY_PATH) into NetBSD VFS (although it might be worthwhile to do that in the long run). It would also be technically possible to use vn_open("/var/shm/...") with a randomly generated pathname and then unlink it immediately, but that would likely introduce an exploitable race unless there's a clever way I haven't thought of to avoid the race.