Signing isn't the same as encryption. Don't confuse the two. You only need to
sign a message once; it's encryption you have to repeat for everyone.
I don't think it's fair to assume that centralizing encryption makes it
faster. The message still has to be encrypted just as many times; it's just a
matter of where it's done. And we're talking about fraction-of-a-second
delays here, so the lag caused by encrypting isn't likely to be terribly
noticeable.
In any case, what you're asking for is nothing more than using XMPP where
everyone communicating with the XMPP uses HTTPS, and the servers communicate
with each other in an encrypted way as well. So just set up a private XMPP
server and have everyone use HTTPS when they connect to it. (XMPP supports
some kind of group chats, though I haven't actually used it.)
