Re: binascii.crc32 results not matching
Thanks so much for the offer, I had a friend do this for me and it works great. Regards, Larry Bates Heiko Wundram wrote: Larry Bates wrote: snip lots of code The algorithm looks very much like the source code for binascii.crc32 (but I'm not a C programmer). Well... As you have access to the code, you might actually just create a thin Python-Wrapper around this so that you can get comparable results. In case you're unable to do so, send me the C-file (I'm not so keen on copy-pasting code which was reformatted due to mail), and I'll send you an extension module back. --- Heiko. -- http://mail.python.org/mailman/listinfo/python-list
Re: binascii.crc32 results not matching
I wrote: this prints 0xF032519BL 0xF032519BL 0x90E3070AL 0x90E3070AL no time to sort out the int/long mess for binascii.crc32, but it pro- bably needs the same tweaking as PIL (which handles the CRC as two 16-bit numbers, to avoid sign hell). I realized that I used 2.3 for testing. under 2.4, the following snippet runs without warnings: import binascii def test2(text, crc=0): return hex((binascii.crc32(text, crc^-1)^-1)0x) in other words, crc = binascii.crc32(text, crc^-1)^-1)0x should give the same result as larry's original CalculateCrc function. /F -- http://mail.python.org/mailman/listinfo/python-list
Re: binascii.crc32 results not matching
Peter Hansen wrote: Larry Bates wrote: I'm trying to get the results of binascii.crc32 to match the results of another utility that produces 32 bit unsigned CRCs. What other utility? As Tim says, there are many CRC32s... the background notes on this one happen to stumble out at the top of the list in response to googling for zip file crc32 checksum polynomial, though I'm sure there are easier ways. The binascii docs say its CRC32 is compatible with the Zip file checksum, but they don't describe it further. Generally CRCs are described in terms of their polynomial, though just quoting that isn't sufficient to describe their behaviour, but if you happen to know the polynomial for your utility, someone else can probably point you to a more appropriate routine, or perhaps explain what you were doing wrong if the binascii one is actually the right one.. -Peter Check out the unit test in the following. http://sourceforge.net/projects/crcmod/ I went to a lot of trouble to get the results to match the results of binascii.crc32. As you will see, there are a couple of extra operations even after you get the polynomial and bit ordering correct. Ray -- http://mail.python.org/mailman/listinfo/python-list
Re: binascii.crc32 results not matching
Raymond L. Buvel wrote: Check out the unit test in the following. http://sourceforge.net/projects/crcmod/ I went to a lot of trouble to get the results to match the results of binascii.crc32. As you will see, there are a couple of extra operations even after you get the polynomial and bit ordering correct. Beautiful! -- http://mail.python.org/mailman/listinfo/python-list
Re: binascii.crc32 results not matching
[Raymond L. Buvel] Check out the unit test in the following. http://sourceforge.net/projects/crcmod/ Cool! I went to a lot of trouble to get the results to match the results of binascii.crc32. As you will see, there are a couple of extra operations even after you get the polynomial and bit ordering correct. Nevertheless, the purpose of binascii.crc32 is to compute exactly the same result as most zip programs give. All the details (including what look to you like extra operations ;-)) were specified by RFC 1952 (the GZIP file format specification). As a result, binascii.crc32 matches, e.g., the CRCs reported by WinZip on Windows, and gives the same results as zlib.crc32 (as implemented by the zlib developers). -- http://mail.python.org/mailman/listinfo/python-list
Re: binascii.crc32 results not matching
Tim Peters wrote: [Raymond L. Buvel] Check out the unit test in the following. http://sourceforge.net/projects/crcmod/ Cool! I went to a lot of trouble to get the results to match the results of binascii.crc32. As you will see, there are a couple of extra operations even after you get the polynomial and bit ordering correct. Nevertheless, the purpose of binascii.crc32 is to compute exactly the same result as most zip programs give. All the details (including what look to you like extra operations ;-)) were specified by RFC 1952 (the GZIP file format specification). As a result, binascii.crc32 matches, e.g., the CRCs reported by WinZip on Windows, and gives the same results as zlib.crc32 (as implemented by the zlib developers). Since there are probably others following this thread, it should be pointed out that the specification of those extra operations is to avoid some pathalogical conditions that you can get with a simplistic CRC operation. For example, using 0 as the starting value will give a value of zero for an arbitrary string of zeros. Consequently, a file starting with a string of zeros will have the same CRC as one with the zeros stripped off. While starting with 0x will give a non-zero value. -- http://mail.python.org/mailman/listinfo/python-list
Re: binascii.crc32 results not matching
Peter Hansen wrote:
Larry Bates wrote:
I'm trying to get the results of binascii.crc32
to match the results of another utility that produces
32 bit unsigned CRCs.
What other utility? As Tim says, there are many CRC32s... the
background notes on this one happen to stumble out at the top of the
list in response to googling for zip file crc32 checksum polynomial,
though I'm sure there are easier ways. The binascii docs say its CRC32
is compatible with the Zip file checksum, but they don't describe it
further.
Generally CRCs are described in terms of their polynomial, though just
quoting that isn't sufficient to describe their behaviour, but if you
happen to know the polynomial for your utility, someone else can
probably point you to a more appropriate routine, or perhaps explain
what you were doing wrong if the binascii one is actually the right one..
-Peter
It was a .DLL written by an employee that has long since
left the company. We want to move the code to Linux for
nightly checking of files. I don't know what to do but
post some long code. See below:
/
INCLUDES
/
#include windows.h
#include string.h
#include malloc.h
#include stdio.h
#include stdlib.h
#include filecrc.h
/
ModuleName: filecrc.c
Author: Syscon Computers - Modified by Barry Weck
Project:PowerBuilder External File CRC function
Date created: May. 18, 1999
Last Modified: Jun. 09, 1999
Module Owner: Syscon Computers
Module description:
This module implements a algorithm for calculating the CRC (Cyclic
Redundency Check) of a binary file. This function is meant to be
compiled as a 16 bit Microsoft Windows (Tm) DLL with either the 16 bit C
compilers of Borland or Microsoft. Compilation under other other
compilers has not been tested. The requirement of 16-bit DLL is
nessesitated by the version of PowerBuilder used by syscon.
This module was written by a third party and was then modified by
subcontractor Barry Weck. The code is copyrighted and owned by Syscon
Computers, Inc. of Tuscaloosa Alabama (C) 1999.
/
/
DATA
/
static unsigned long ccitt_32[256] =
{
0xUL, 0x77073096UL, 0xee0e612cUL, 0x990951baUL, 0x076dc419UL,
0x706af48fUL, 0xe963a535UL, 0x9e6495a3UL,
0x0edb8832UL, 0x79dcb8a4UL, 0xe0d5e91eUL, 0x97d2d988UL, 0x09b64c2bUL,
0x7eb17cbdUL, 0xe7b82d07UL, 0x90bf1d91UL,
0x1db71064UL, 0x6ab020f2UL, 0xf3b97148UL, 0x84be41deUL, 0x1adad47dUL,
0x6ddde4ebUL, 0xf4d4b551UL, 0x83d385c7UL,
0x136c9856UL, 0x646ba8c0UL, 0xfd62f97aUL, 0x8a65c9ecUL, 0x14015c4fUL,
0x63066cd9UL, 0xfa0f3d63UL, 0x8d080df5UL,
0x3b6e20c8UL, 0x4c69105eUL, 0xd56041e4UL, 0xa2677172UL, 0x3c03e4d1UL,
0x4b04d447UL, 0xd20d85fdUL, 0xa50ab56bUL,
0x35b5a8faUL, 0x42b2986cUL, 0xdbbbc9d6UL, 0xacbcf940UL, 0x32d86ce3UL,
0x45df5c75UL, 0xdcd60dcfUL, 0xabd13d59UL,
0x26d930acUL, 0x51de003aUL, 0xc8d75180UL, 0xbfd06116UL, 0x21b4f4b5UL,
0x56b3c423UL, 0xcfba9599UL, 0xb8bda50fUL,
0x2802b89eUL, 0x5f058808UL, 0xc60cd9b2UL, 0xb10be924UL, 0x2f6f7c87UL,
0x58684c11UL, 0xc1611dabUL, 0xb6662d3dUL,
0x76dc4190UL, 0x01db7106UL, 0x98d220bcUL, 0xefd5102aUL, 0x71b18589UL,
0x06b6b51fUL, 0x9fbfe4a5UL, 0xe8b8d433UL,
0x7807c9a2UL, 0x0f00f934UL, 0x9609a88eUL, 0xe10e9818UL, 0x7f6a0dbbUL,
0x086d3d2dUL, 0x91646c97UL, 0xe6635c01UL,
0x6b6b51f4UL, 0x1c6c6162UL, 0x856530d8UL, 0xf262004eUL, 0x6c0695edUL,
0x1b01a57bUL, 0x8208f4c1UL, 0xf50fc457UL,
0x65b0d9c6UL, 0x12b7e950UL, 0x8bbeb8eaUL, 0xfcb9887cUL, 0x62dd1ddfUL,
0x15da2d49UL, 0x8cd37cf3UL, 0xfbd44c65UL,
0x4db26158UL, 0x3ab551ceUL, 0xa3bc0074UL, 0xd4bb30e2UL, 0x4adfa541UL,
0x3dd895d7UL, 0xa4d1c46dUL, 0xd3d6f4fbUL,
0x4369e96aUL, 0x346ed9fcUL, 0xad678846UL, 0xda60b8d0UL, 0x44042d73UL,
0x33031de5UL, 0xaa0a4c5fUL, 0xdd0d7cc9UL,
0x5005713cUL, 0x270241aaUL, 0xbe0b1010UL, 0xc90c2086UL, 0x5768b525UL,
0x206f85b3UL, 0xb966d409UL, 0xce61e49fUL,
0x5edef90eUL, 0x29d9c998UL, 0xb0d09822UL, 0xc7d7a8b4UL, 0x59b33d17UL,
0x2eb40d81UL, 0xb7bd5c3bUL, 0xc0ba6cadUL,
0xedb88320UL, 0x9abfb3b6UL, 0x03b6e20cUL, 0x74b1d29aUL, 0xead54739UL,
0x9dd277afUL, 0x04db2615UL, 0x73dc1683UL,
0xe3630b12UL, 0x94643b84UL, 0x0d6d6a3eUL, 0x7a6a5aa8UL, 0xe40ecf0bUL,
0x9309ff9dUL, 0x0a00ae27UL, 0x7d079eb1UL,
0xf00f9344UL, 0x8708a3d2UL, 0x1e01f268UL, 0x6906c2feUL, 0xf762575dUL,
0x806567cbUL, 0x196c3671UL, 0x6e6b06e7UL,
0xfed41b76UL, 0x89d32be0UL, 0x10da7a5aUL, 0x67dd4accUL, 0xf9b9df6fUL,
0x8ebeeff9UL, 0x17b7be43UL, 0x60b08ed5UL,
0xd6d6a3e8UL, 0xa1d1937eUL, 0x38d8c2c4UL,
Re: binascii.crc32 results not matching
Larry Bates wrote: def CalculateCrc(buffer, crc): /snip/ The algorithm looks very much like the source code for binascii.crc32 (but I'm not a C programmer). does your Python version give the right result ? if so, the following might be somewhat helpful: def test1(text, crc=0): # larry's code result = CalculateCrc(text, crc) return hex(result[0]) def test2(text, crc=0): import Image # using pil's crc32 api a = (crc 16) ^ 65535 b = (crc 65535) ^ 65535 a, b = Image.core.crc32(text, (a, b)) a ^= 65535 b ^= 65535 return 0x%04X%04XL % (a, b) def test(text): print test1(text), test2(text) test(hello) test(goodbye) this prints 0xF032519BL 0xF032519BL 0x90E3070AL 0x90E3070AL no time to sort out the int/long mess for binascii.crc32, but it pro- bably needs the same tweaking as PIL (which handles the CRC as two 16-bit numbers, to avoid sign hell). /F -- http://mail.python.org/mailman/listinfo/python-list
Re: binascii.crc32 results not matching
Larry Bates wrote: snip a lot of code Looking over the code, it seems very inefficient and hard to understand. You really should check out the following. http://sourceforge.net/projects/crcmod/ It will allow you to generate efficient CRC functions for use in Python and in C or C++. The only thing you need to input is the polynomial, the bit ordering, and the starting value. The unit test gives a number of polynomials including the one you are using which is: polynomial: 0x104C11DB7, bit reverse algorithm If you are just looking for a utility on Linux to do nightly checking of files, I strongly recommend md5sum. My timing tests indicate that the MD5 algorithm is comparable or slightly faster than a 32-bit CRC and certainly faster than the code you are trying to port. It also has the advantage of being a standard Linux command so you don't need to code anything. Ray -- http://mail.python.org/mailman/listinfo/python-list
Re: binascii.crc32 results not matching
Larry Bates wrote: snip lots of code The algorithm looks very much like the source code for binascii.crc32 (but I'm not a C programmer). Well... As you have access to the code, you might actually just create a thin Python-Wrapper around this so that you can get comparable results. In case you're unable to do so, send me the C-file (I'm not so keen on copy-pasting code which was reformatted due to mail), and I'll send you an extension module back. --- Heiko. -- http://mail.python.org/mailman/listinfo/python-list
binascii.crc32 results not matching
I'm trying to get the results of binascii.crc32 to match the results of another utility that produces 32 bit unsigned CRCs. binascii.crc32 returns results in the range of -2**31-1 and 2**21-1. Has anyone ever worked out any bit twiddling code to get a proper unsigned 32 bit result from binascii.crc32? Output snip from test on three files: binascii.crc32=-1412119273, oldcrc32= 2221277246 binascii.crc32=-647246320, oldcrc32=73793598 binascii.crc32=-1391482316, oldcrc32=79075810 Thanks in advance, Regards, Larry Bates -- http://mail.python.org/mailman/listinfo/python-list
Re: binascii.crc32 results not matching
[Larry Bates] I'm trying to get the results of binascii.crc32 to match the results of another utility that produces 32 bit unsigned CRCs. binascii.crc32 returns results in the range of -2**31-1 and 2**21-1. Has anyone ever worked out any bit twiddling code to get a proper unsigned 32 bit result from binascii.crc32? Just the result with 0x (a string of 32 1-bits). Output snip from test on three files: binascii.crc32=-1412119273, oldcrc32= 2221277246 binascii.crc32=-647246320, oldcrc32=73793598 binascii.crc32=-1391482316, oldcrc32=79075810 Doesn't look like these are using the same CRC algorithms (there are many distinct ways to compute a 32-bit CRC). For example, print -1412119273 0x 2882848023 and that's not equal to 2221277246. Or you're on Windows, and forgot to open the files in binary mode. Or something ;-) -- http://mail.python.org/mailman/listinfo/python-list
Re: binascii.crc32 results not matching
Larry Bates wrote: I'm trying to get the results of binascii.crc32 to match the results of another utility that produces 32 bit unsigned CRCs. What other utility? As Tim says, there are many CRC32s... the background notes on this one happen to stumble out at the top of the list in response to googling for zip file crc32 checksum polynomial, though I'm sure there are easier ways. The binascii docs say its CRC32 is compatible with the Zip file checksum, but they don't describe it further. Generally CRCs are described in terms of their polynomial, though just quoting that isn't sufficient to describe their behaviour, but if you happen to know the polynomial for your utility, someone else can probably point you to a more appropriate routine, or perhaps explain what you were doing wrong if the binascii one is actually the right one.. -Peter -- http://mail.python.org/mailman/listinfo/python-list
