Is there a reason org-loaddefs.el is not byte compiled? I see org-make-org-loaddefs in org-fixup.el puts in no-byte-compile, but I think byte compiling could give it dynamic docstrings and so keep the docstring text out of memory until actually viewed (which of course might be never).
If the loaddefs are included in the dumped emacs then perhaps byte compiling is not wanted in that case, but when using org-mode as a separate add-on letting it byte compile could be good.