Perhaps it is a similar bug as this one? https://github.com/scipy/scipy/issues/6999
Basically, it turned out that the CRC was getting computed on an unflushed buffer, or something like that. On Fri, May 14, 2021 at 10:05 AM Isaac Gerg <isaac.g...@gergltd.com> wrote: > I am using 1.19.5 on Windows 10 using Python 3.8.6 (tags/v3.8.6:db45529, > Sep 23 2020, 15:52:53) [MSC v.1927 64 bit (AMD64)]. > > I have two python processes running (i.e. no threads) which do > independent processing jobs and NOT writing to the same directories. Each > process runs for 5-10 hours and then writes out a ~900MB npz file > containing 4 arrays. > > When I go back to read in the npz files, I will sporadically get bad CRC > errors which are related to npz using ziplib. I cannot figure out why this > is happening. Looking through online forums, other folks have had CRC > problems but they seem to be isolated to specifically using ziblib, not > numpy. I have found a few mentions though of ziplib causing headaches if > the same file pointer is used across calls when one uses the file handle > interface to ziblib as opposed to passing in a filename.' > > I have verified with 7zip that the files do in fact have a CRC error so > its not an artifact of the ziblib. I have also used the file handle > interface to np.load and still get the error. > > Aside from writing my own numpy storage file container, I am stumped as to > how to fix this, or reproduce this in a consistent manner. Any suggestions > would be greatly appreciated! > > Thank you, > Isaac > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion >
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion