Currently the idea I'm most attached to is using groups as realms, while keeping auth data in a separate file (or even hardcoded inside config.h). I think it's a pretty elegant solution (and it would match how I already manage shell user access to my server files), but relying on filesystem metadata can be problematic as it can get lost relatively easily (e.g: archive formats used for backup which do not store user info).
Let me know how this ideas feels overall, I'm kinda biased as I love (ab)using the filesystem for my own benefit. Best regards. José Miguel