I've spent several days trying different things and failing at tracking down the actual source of this problem. I have a client and a server communicating over an SSL socket connection. The client sends some data to the server along with an RSA signature to verify it's an authorized request. Upon receiving the request from the client, the server uses the EVP_verify routines to verify the signature. Then the server attempts to send a response back to the client telling it the request succeeded or failed. However, during the transmit routine in Qt, a call to ssl_read fails and gives the following reason:
Error while reading: error:04077068:rsa routines:RSA_verify:bad signature I was originally using the RSA_verify routine, but was having the same problem and thought that maybe it would be solved if I switched over to the EVP_verify routines as they can provide an EVP_MD_CTX structure. I'm using openssl built into the Qt 4.4.1 libraries, which are implemented for thread safety using the locking_function as prescribed. I've verified that the locking_function gets called for both their calls and my calls; so I don't think it's a threading issue. I'm wondering if it could have anything to do with the socket using a different public key than what I'm using to verify the sent data. Another possibility could be that they are using two different decryption algorithms and my calling the EVP_verify routines changes some global value that the ssl_ routine ends up using which is no longer valid for the connection. I've built debug versions of the shared openssl libraries and used those to build a debug version of the QtNetwork.dll; I can step into the Qt calls, but since Qt is dynamically loading the openssl APIs, I'm having difficulty stepping into the openssl routines where I can view the source code. I can only step into the assembly. So I can't really tell which line of code and why the ssl_read routines are actually failing. So, a quick recap to hopefully clarify what I'm seeing... SSL socket connection using public key A recieves signed data EVP_verify routines are called using public key B SSL socket connection attempts to read more data inside the Qt transmission code. ssl_read fails with error:04077068:rsa routines:RSA_verify:bad signature. If I skip the verification calls, everything works fine. Am I attempting to do something invalid? Is Qt making a bad assumption about when they can read from the socket? Is there a bug down in openssl? Or am I skipping some step(s) to put things back in order for the socket after I call the EVP_verify routines? Is there a book I should have read before I tried this? Thanks in advance for any help provided.
