Originally (.Net 1) there was only 'Parse', 'TryParse' came in .Net 2, I guess they had to admit that exceptions are not always practical.
I think TryParse (and anything marked prefixed with Try) is meant for quick stuff. It doesn't return any error. Just a boolean indicating that it failed. The entire function seems to be a wrapper around the actual try mechanism. So Parse will give you an error, TryParse will just tell you it failed.