Re: additively computing SHA hash
* Subra Aswathanarayanan wrote on Mon, Jun 07, 2010 at 20:44 -0400:
Both of you mention that OpenSSL doesn't provide such an interface.
May be this question is not appropriate for this forum, but do you
know of any such simpler libraries that I might be able to use?
If it is just SHA1 you need for a particular purpose, you may even use just
some sha1.c (instead of using a big library), for example:
http://www.koders.com/c/fid17B8BC1580A40B49AE494B182AF91A73CBF62B38.aspx?s=sort
http://www.koders.com/c/fid292402EE6741EB2682D9E89AEF3A29B7B2C73F32.aspx?s=sort
the context to be (serialized and) saved is:
struct sha_ctx {
u_int32_t digest[SHA_DIGESTLEN]; /* Message digest */
u_int32_t count_l, count_h; /* 64-bit block
count */
u_int8_t block[SHA_DATASIZE]; /* SHA data
buffer */
int index;/* index into
buffer */
};
your implementation will depend on the structure, i.e. if it
changes, your application breaks. Thus using an libraries private
structure (like including private header files to see what is
inside a struct which has just a forwarder in the public header)
has the disadvantage that as soon as the library is changed, your
application may break, because it uses internal interfaces and
relies on hidden data. When you have a copy of sha1.c, it is your
structure and you have the control. For testing, you might even
use memcpy(dest, ctx, sizeof(struct sha_ctx)).
I assume by serializing such a context, your application depends
on the hardware architecture (e.g. byte order), even if
serialization takes byte order into account the implemenation may
fail (for example, if `block' has to have a specific format).
When Engines are used, for example a smart card, it might even be
impossible to serialize a context in the way you need it (I would
intuitively assume that any hardware token would prohibit such
usage to avoid potential miss-use).
oki,
Steffen
About Ingenico: Ingenico is a leading provider of payment solutions, with over
15 million terminals deployed in more than 125 countries. Its 2,850 employees
worldwide support retailers, banks and service providers to optimize and secure
their electronic payments solutions, develop their offer of services and
increase their point of sales revenue. More information on
http://www.ingenico.com/.
This message may contain confidential and/or privileged information. If you
are not the addressee or authorized to receive this for the addressee, you must
not use, copy, disclose or take any action based on this message or any
information herein. If you have received this message in error, please advise
the sender immediately by reply e-mail and delete this message. Thank you for
your cooperation.
P Please consider the environment before printing this e-mail
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager majord...@openssl.org
Re: additively computing SHA hash
Steve/Victor, You mean you want to do: SHA1(A) and later do: SHA1(A || B) without including A again? That is correct. Thats exactly what I want to do. You need to serialize, save and restore the intermediate state of the digest before you call final if you need to be able to append more data without re-computing the entire checksum. I am open to the idea of serializing, saving and restoring. OpenSSL does not provide a serialization interface for MD_CTX objects. Perhaps you're better off with a simpler library that does not support engines, and other features that make serialization difficult. Both of you mention that OpenSSL doesn't provide such an interface. May be this question is not appropriate for this forum, but do you know of any such simpler libraries that I might be able to use? Has anyone else on this forum ran in to a similar situation and had to dive deep in to the source code to make this work? Thanks a lot for the prompt response! Subra
Re: additively computing SHA hash
On Mon, Jun 07, 2010, Subra Aswathanarayanan wrote: Steve/Victor, You mean you want to do: SHA1(A) and later do: SHA1(A || B) without including A again? That is correct. Thats exactly what I want to do. You need to serialize, save and restore the intermediate state of the digest before you call final if you need to be able to append more data without re-computing the entire checksum. I am open to the idea of serializing, saving and restoring. OpenSSL does not provide a serialization interface for MD_CTX objects. Perhaps you're better off with a simpler library that does not support engines, and other features that make serialization difficult. Both of you mention that OpenSSL doesn't provide such an interface. May be this question is not appropriate for this forum, but do you know of any such simpler libraries that I might be able to use? Has anyone else on this forum ran in to a similar situation and had to dive deep in to the source code to make this work? Well I'd add the BIG disclaimer that will NOT work in future when OpenSSL structures are made opaque and almost certainly will fail if you have an ENGINE. What you need to do is copy the md_ctx-data (which will be a flat buffer for the software SHA1 implementation) for md_ctx-digest-md_size bytes. Save that somewhere and after calling init the second time copy it back. Do NOT try restoring the context with different versions of OpenSSL or different architectures. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: additively computing SHA hash
Stephen, Thanks for your solution. Well I'd add the BIG disclaimer that will NOT work in future when OpenSSL structures are made opaque and almost certainly will fail if you have an ENGINE. Understood. I am new to openssl and I am reading up about 'ENGINE's in openssl. When you say it will fail when there is an 'ENGINE', do you mean if I use EVP_DIGEST functions (from the example on openssl.org), your solution will not work? If yes, is the solution as simple as using SHA1_Init, SHA1_Update, SHA1_Final functions instead? What you need to do is copy the md_ctx-data (which will be a flat buffer for the software SHA1 implementation) for md_ctx-digest-md_size bytes. Save that somewhere and after calling init the second time copy it back. Do NOT try restoring the context with different versions of OpenSSL or different architectures. Will this work with plain old SHA also? I will try to find the answers for some of the questions on my own by writing some code. But any help from your side will be much appreciated. Subra
Re: additively computing SHA hash
On Tue, Jun 08, 2010, Jason Fister wrote: Stephen, Thanks for your solution. Well I'd add the BIG disclaimer that will NOT work in future when OpenSSL structures are made opaque and almost certainly will fail if you have an ENGINE. Understood. I am new to openssl and I am reading up about 'ENGINE's in openssl. When you say it will fail when there is an 'ENGINE', do you mean if I use EVP_DIGEST functions (from the example on openssl.org), your solution will not work? If yes, is the solution as simple as using SHA1_Init, SHA1_Update, SHA1_Final functions instead? If you don't know what an ENGINE is you probably aren't using one. They can contain alternative algorithm implementations in either software or hardware. The reason why this may not work with an ENGINE is the data inside may contain anything including pointers to internal contexts in hardware which wont be properly saved or restored. What you need to do is copy the md_ctx-data (which will be a flat buffer for the software SHA1 implementation) for md_ctx-digest-md_size bytes. Save that somewhere and after calling init the second time copy it back. Do NOT try restoring the context with different versions of OpenSSL or different architectures. Will this work with plain old SHA also? I will try to find the answers for some of the questions on my own by writing some code. But any help from your side will be much appreciated. It should work with any of the standard OpenSSL software implementations. So that includs SHA, MD5, SHA256 etc. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
additively computing SHA hash
Hello folks, I have a service to which people can send data. My service then stores the data and returns the SHA checksum for the data back to the client. I also store the checksum I computed and the offset at which it was computed as part of the metadata. I want to allow clients to send append requests to this data at a later time. When they send the append request, I want to be able to use the checksum that I had already calculated as the starting point and then use that to generate the new checksum for the appended object. I know about the init, update and final functions. But I dont see a way to pass in the checksum value of the original object when computing the checksum of the new appended object. Can someone tell me how I can achive the above? Thanks, Jason
Re: additively computing SHA hash
On Mon, Jun 07, 2010, Jason Fister wrote: Hello folks, I have a service to which people can send data. My service then stores the data and returns the SHA checksum for the data back to the client. I also store the checksum I computed and the offset at which it was computed as part of the metadata. I want to allow clients to send append requests to this data at a later time. When they send the append request, I want to be able to use the checksum that I had already calculated as the starting point and then use that to generate the new checksum for the appended object. I know about the init, update and final functions. But I dont see a way to pass in the checksum value of the original object when computing the checksum of the new appended object. Can someone tell me how I can achive the above? You mean you want to do: SHA1(A) and later do: SHA1(A || B) without including A again? The hash itself does not contain enough information to continue in that fashion. You'd need to store the actual hash context. There isn't a standard OpenSSL function to this you'd need to delve into the internals a little and copy the flat context somewhere and later restore. So you'd do something like... init update(A) save_context hash_A = final init restore_context update(B) hash_AB=final Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: additively computing SHA hash
On Mon, Jun 07, 2010 at 04:07:06PM -0400, Jason Fister wrote: Hello folks, I have a service to which people can send data. My service then stores the data and returns the SHA checksum for the data back to the client. I also store the checksum I computed and the offset at which it was computed as part of the metadata. I want to allow clients to send append requests to this data at a later time. When they send the append request, I want to be able to use the checksum that I had already calculated as the starting point and then use that to generate the new checksum for the appended object. It is not computationally feasible to compute the checksum of a longer message from the checksum of a short message. I know about the init, update and final functions. But I dont see a way to pass in the checksum value of the original object when computing the checksum of the new appended object. You need to serialize, save and restore the intermediate state of the digest before you call final if you need to be able to append more data without re-computing the entire checksum. OpenSSL does not provide a serialization interface for MD_CTX objects. Perhaps you're better off with a simpler library that does not support engines, and other features that make serialization difficult. The source code EVP_MD_CTX_copy_ex() can, with some effort, be adapted to save/restore non-engine OpenSSL digest contexts. -- Viktor. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
