That's not enough, the client and server need to negotiate the secret key to be able to use encryption (or better AEAD, authenticated encryption).
In libsodium that's in the key exchange session: <https://doc.libsodium.org/key_exchange/> Note that creating protocols like this is the dangerous part of "rolling your own crypto", and they are untestable against a reference implementation so don't use it for sensitive data, only for learning.
