Re: MD5 digests of very big files

[Richmond]

On 06/27/2010 08:46 PM, Alejandro Tejada wrote:

Hi all,

Read this report written by Mark Schonewille:
http://quality.runrev.com/qacenter/show_bug.cgi?id=2410

Mark Waddingham post this function:

function quasiMD5 pFile
   local tMD5s
   open file pFile for binary read
   repeat
 read from file pFile for 4096 chars
 if the result is EOF then
   exit repeat
 end if
 put the md5digest of it after tMD5s
   end repeat
   close file pFile
   return the md5Digest of tMD5s
end quasiMD5

But when i run this function, Rev
consumes all memory available.

   

-

and Mark Waddingham also wrote:

"Some sort of 'stream' abstraction for doing this kind of operation on 
files and
the like would be a useful addition and certainly something to consider 
for a

future version."

in November 2004.

I wonder if anything further has been done in this area over the last 5 
and a half years??

___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: MD5 digests of very big files

[Alejandro Tejada]

Looks like the culprit is the condition
to escape the repeat structure.

The result never reach the EOF "end of file".

Which other condition could i use to
verify that the file have been reading
completely?

Thanks in advance!

Al

> In this enhancement request written by Mark Schonewille: 
> http://quality.runrev.com/qacenter/show_bug.cgi?id=2410
> 
> Mark Waddingham posted this function: 
> 
> function quasiMD5 pFile 
>local tMD5s 
>open file pFile for binary read 
>repeat 
>  read from file pFile for 4096 chars 
>  if the result is EOF then -- Alert!!! the result never reach "EOF"
>exit repeat 
>  end if 
>  put the md5digest of it after tMD5s 
>end repeat 
>close file pFile 
>return the md5Digest of tMD5s 
> end quasiMD5 
> 
> But when i run this function, Rev 
> consumes all memory available. 
-- 
View this message in context: 
http://runtime-revolution.278305.n4.nabble.com/MD5-digests-of-very-big-files-tp2270182p2270228.html
Sent from the Revolution - User mailing list archive at Nabble.com.
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: MD5 digests of very big files

[J. Landman Gay]

Alejandro Tejada wrote:

Looks like the culprit is the condition
to escape the repeat structure.

The result never reach the EOF "end of file".

Which other condition could i use to
verify that the file have been reading
completely?


Try: if it is empty

--
Jacqueline Landman Gay | jac...@hyperactivesw.com
HyperActive Software   | http://www.hyperactivesw.com
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: MD5 digests of very big files

[Alejandro Tejada]

Hi Jacke,

Jacqueline Landman Gay wrote:

> Try: if it is empty 

Many Thanks!
This works. :-D

Now the function will be:

function quasiMD5 pFile
  local tMD5s
  open file pFile for binary read
  repeat
read from file pFile for 4096 chars
if it is empty then
  exit repeat
end if
put the md5digest of it after tMD5s
  end repeat
  close file pFile
  return the md5Digest of tMD5s
end quasiMD5

-- 
View this message in context: 
http://runtime-revolution.278305.n4.nabble.com/MD5-digests-of-very-big-files-tp2270182p2270259.html
Sent from the Revolution - User mailing list archive at Nabble.com.
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: MD5 digests of very big files

[Jan Schenkel]

--- On Sun, 6/27/10, Alejandro Tejada  wrote:
> 
> Hi Jacke,
> 
> Jacqueline Landman Gay wrote:
> 
> > Try: if it is empty 
> 
> Many Thanks!
> This works. :-D
> 
> Now the function will be:
> 
> function quasiMD5 pFile
>   local tMD5s
>   open file pFile for binary read
>   repeat
>     read from file pFile for 4096 chars
>     if it is empty then
>       exit repeat
>     end if
>     put the md5digest of it after tMD5s
>   end repeat
>   close file pFile
>   return the md5Digest of tMD5s
> end quasiMD5
> 
> 

To make this future-proof, you might want to use the byte chunk type for 
reading from the binary file:
   read from file pFile for 4096 bytes  -- was: chars
That way, after Unicode support in the revEngine becomes pervasive and 
transparent, the above will work just fine, whether or not the intricacies of 
what constitutes a 'char' change.

Jan Schenkel
=
Quartam Reports & PDF Library for Revolution


=
"As we grow older, we grow both wiser and more foolish at the same time."  (La 
Rochefoucauld)




___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: MD5 digests of very big files

[Malte Pfaff-Brill]

Hi Al,

I guess the problem you see is that revs MD5s need to be binaryDecoded to have 
the same output like for example PHP. A simple script to see this is:

on mouseUp
   local tCheck,tResult
   put md5Digest("moo") into tCheck
   get binaryDecode("H*",tCheck,tResult)
   put tCheck&cr&tResult
  -- tCheck is revs, tResult is compliant to PHP
end mouseUp

Hope that helps!

Cheers,

Malte___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: MD5 digests of very big files

[Alejandro Tejada]

Hi Malte,

Malte Pfaff Brill wrote:
> I guess the problem you see is that revs MD5s
> need to be binaryDecoded to have the same output
> like for example PHP. A simple script to see this is:
> on mouseUp
>   local tCheck,tResult
>   put md5Digest("moo") into tCheck
>   get binaryDecode("H*",tCheck,tResult)
>   put tCheck&cr&tResult
>  -- tCheck is revs, tResult is compliant to PHP
> end mouseUp

Excellent! :-D
Now the quasiMD5 function reads as follow:

function quasiMD5 pFile 
-- posted by Mark Waddingham 
  local tMD5s,tResult
  open file pFile for binary read 
  repeat 
read from file pFile for 4096 bytes 
if it is empty then 
  exit repeat 
end if 
put the md5digest of it after tMD5s 
  end repeat 
  close file pFile 
  put the md5Digest of tMD5s into tCheck
  get binaryDecode("H*",tCheck,tResult)
  return tResult
end quasiMD5 

Just notice that converting the MD5 binary string
to Hexadecimal numbers do not solve the last
question:

How could we change this  function to get the same MD5
from a big file (like a Linux ISO image) than command
lines programs do?

For example, the MD5 of the more recent version of Puppy
Linux ISO image is 0847331e9cdcb55eaae238924d131a24
ftp://ibiblio.org/pub/linux/distributions/puppylinux/puppy-5.0.1/

This quasiMD5 function does not produce the same MD5
digest and everyone advice that we should not load the
full 128 MB file on memory to calculate it using Rev.

I am sure that command line programs that calculate MD5
digest only read small chunks too (just like this function
does) but they also apply other math or bit operations to
get the correct MD5 digest.

My question is: Which other operations are needed in
this quasiMD5 function to produce true MD5 digests
of big files?

Thanks in advance.

Al
-- 
View this message in context: 
http://runtime-revolution.278305.n4.nabble.com/MD5-digests-of-very-big-files-tp2271355p2272645.html
Sent from the Revolution - User mailing list archive at Nabble.com.
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: MD5 digests of very big files

[Richard Gaskin]

Alejandro Tejada wrote:


My question is: Which other operations are needed in
this quasiMD5 function to produce true MD5 digests
of big files?


The MD5 algo is a great way for quick data validation for internal use, 
but for more serious use (like verifying a distro) I don't think anyone 
uses MD5 anymore - from Wikipedia:



  MD5 was designed by Ron Rivest in 1991 to replace an
  earlier hash function, MD4. In 1996, a flaw was found
  with the design of MD5. While it was not a clearly
  fatal weakness, cryptographers began recommending the
  use of other algorithms, such as SHA-1 (which has
  since been found also to be vulnerable). In 2004, more
  serious flaws were discovered, making further use of
  the algorithm for security purposes questionable.[3][4]
  In 2007 a group of researchers described how to create
  a pair of files that share the same MD5 checksum.[5]
  In an attack on MD5 published in December 2008, a group
  of researchers used this technique to fake SSL certificate
  validity.[6][7]  US-CERT of the U. S. Department of
  Homeland Security said MD5 "should be considered
  cryptographically broken and unsuitable for further
  use,"[8]  and most U.S. government applications will
  be required to move to the SHA-2 family of hash
  functions after 2010.[9]



I've seen some SHA-1 hashes in RevTalk, but nothing for SHA-2 - anyone 
know of one?


--
 Richard Gaskin
 Fourth World
 Rev training and consulting: http://www.fourthworld.com
 Webzine for Rev developers: http://www.revjournal.com
 revJournal blog: http://revjournal.com/blog.irv
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

Re: MD5 digests of very big files

[Bob Sneidar]

Yes MD5 is supremely hackable, or should I say hashable. There are hash tables 
that contain the MD5 hash for an almost unlimited number of possible keys, and 
the algorithm is also easily crackable these days.

Bob


On Jun 29, 2010, at 2:30 PM, Richard Gaskin wrote:

> Alejandro Tejada wrote:
> 
>> My question is: Which other operations are needed in
>> this quasiMD5 function to produce true MD5 digests
>> of big files?
> 
> The MD5 algo is a great way for quick data validation for internal use, but 
> for more serious use (like verifying a distro) I don't think anyone uses MD5 
> anymore - from Wikipedia:
> 
> 
>  MD5 was designed by Ron Rivest in 1991 to replace an
>  earlier hash function, MD4. In 1996, a flaw was found
>  with the design of MD5. While it was not a clearly
>  fatal weakness, cryptographers began recommending the
>  use of other algorithms, such as SHA-1 (which has
>  since been found also to be vulnerable). In 2004, more
>  serious flaws were discovered, making further use of
>  the algorithm for security purposes questionable.[3][4]
>  In 2007 a group of researchers described how to create
>  a pair of files that share the same MD5 checksum.[5]
>  In an attack on MD5 published in December 2008, a group
>  of researchers used this technique to fake SSL certificate
>  validity.[6][7]  US-CERT of the U. S. Department of
>  Homeland Security said MD5 "should be considered
>  cryptographically broken and unsuitable for further
>  use,"[8]  and most U.S. government applications will
>  be required to move to the SHA-2 family of hash
>  functions after 2010.[9]
> 
> 
> 
> I've seen some SHA-1 hashes in RevTalk, but nothing for SHA-2 - anyone know 
> of one?
> 
> --
> Richard Gaskin
> Fourth World
> Rev training and consulting: http://www.fourthworld.com
> Webzine for Rev developers: http://www.revjournal.com
> revJournal blog: http://revjournal.com/blog.irv
> ___
> use-revolution mailing list
> use-revolution@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution

___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution