Maybe that's because x11-display take a few milliseconds to start, and that xmodmap actually starts before x11-display is actually done loading.
I've encountered this type of issue with emacs-server, where I used this solution on RDE, but it's emacs/service specific : https://lists.sr.ht/~abcdw/rde-devel/patches/48753 I can't help you more, I don't use x11 anymore, but if that's the issue, maybe go through the x11 documentation, find if you can start it with a pid file, and if that's the case, you can try using a background shepherd process waiting for the pid-file instead of a foreground one. -- Best regards, Nicolas Graves