Patches item #1121611, was opened at 2005-02-12 17:33 Message generated for change (Comment added) made by greg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470
Category: Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Gregory P. Smith (greg) Assigned to: Gregory P. Smith (greg) Summary: sha and md5 modules should use OpenSSL when possible Initial Comment: The md5 and sha (sha1) modules should use OpenSSL for the algorithms when it is available as its implementations are much faster than pythons own. Attached is an initial patch to use OpenSSL for the sha module. Its not ready for committing as is yet, but it is setup to be a generic base for all OpenSSL hashes with a little bit of work in the future. Tossing this out there for people to see how trivial it is and enjoy the speedups. diff is against HEAD but it should apply to 2.4 just fine. ---------------------------------------------------------------------- >Comment By: Gregory P. Smith (greg) Date: 2005-03-12 17:13 Message: Logged In: YES user_id=413 I linked a _hashlib.so library statically against openssl and reran the speed test. no change. that means its not shared library overhead causing the higher startup time but just an artifact of the OpenSSL EVP interface. Next up, analyze what size things common heavy sha1 using applications regularly hash (BitTorrent and such). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-10 00:09 Message: Logged In: YES user_id=413 The 007 patch improves the speed of the constructor. There is still a potential speed issue with the constructor/destructor to work on: [EMAIL PROTECTED] src $ ./python Lib/test/test_hashlib_speed.py _sha testing speed of old _sha legacy interface 0.06 seconds [20000 creations] 0.24 seconds [20000 "" digests] 0.15 seconds 20 x 106201 bytes [huge data] 0.15 seconds 200 x 10620 bytes [large data] 0.17 seconds 2000 x 1062 bytes [medium data] 0.35 seconds 20020 x 106 bytes [small data] 1.37 seconds 106200 x 20 bytes [digest_size data] 2.75 seconds 212400 x 10 bytes [tiny data] [EMAIL PROTECTED] src $ ./python Lib/test/test_hashlib_speed.py sha1 testing speed of hashlib.sha1 <built-in function openssl_sha1> 0.22 seconds [20000 creations] 0.57 seconds [20000 "" digests] 0.09 seconds 20 x 106201 bytes [huge data] 0.09 seconds 200 x 10620 bytes [large data] 0.15 seconds 2000 x 1062 bytes [medium data] 0.71 seconds 20020 x 106 bytes [small data] 3.39 seconds 106200 x 20 bytes [digest_size data] 6.70 seconds 212400 x 10 bytes [tiny data] I suspect the cause is either or both of the shared openssl library call overhead or the openssl EVP abstraction interface. The speed results are very similar to the above regardless of which digest is used (the above was a celeron 333mhz running linux). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-03 13:15 Message: Logged In: YES user_id=413 hashlib-006.patch adds fast constructors and a speed test. documentation is the next step. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-01 01:14 Message: Logged In: YES user_id=413 hashlib-005.patch now passes its test suite and no problems appear in valgrind. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-28 10:11 Message: Logged In: YES user_id=413 a much updated patch (hashlib-patch-004.patch). it incorporates some suggestions as well as including sf patch 935454's sha256/224 and sha512/384 implementations. still not complete but shows the direction its going in (i see a segfault part way thru the test suite after running the sha512 tests). as for the private modules being under another package, i see no reason to do that since there aren't very many (how does that work for binary modules anyways?). ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-02-18 11:21 Message: Logged In: YES user_id=764593 Should the private modules (such as _sha) be placed in a crypto package, instead of directly in the parent/everything library? ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-16 22:46 Message: Logged In: YES user_id=413 hashes-openssl-002.patch replaces the sha and md5 modules with a general hashes module that wraps all hashes that OpenSSL supports. note that OpenSSLs implementations are much faster than the previous python versions as it choses versions optimized for your particular hardware. Incase python is compiled without openssl the hashes wrapper falls back on the old python sha and md5 module implementations. side note: This may be sufficient for the Debian folks to work around their random odd licensing issue. just have debian python depend on openssl; use this and remove the old md5 module/code that wouldn't get used anyways. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 _______________________________________________ Patches mailing list [email protected] http://mail.python.org/mailman/listinfo/patches
