Dear All, I would like to add a class for writing one (possibly big) .npy file saving multiple (same dtype, compatible shape) arrays. My use case was the saving of slowly accumulating data regularly for a long time into one file.
Please find a first implementation under https://github.com/numpy/numpy/pull/4987 . It currently supports writing a new file only and only in C order in the file. Opening an existing file for append and reading back parts from a very big .npy file would be straightforward next steps for a full featured class. The .npy file format is only affected by leaving some extra space for re-writing the header later with a possibly bigger "shape" field, respecting the 16-byte alignment. Example: ``` A=np.array([[0,1,2,3,4,5,6,7],[8,9,10,11,12,13,14,15]]) with np.IncrementalWriter("testfile.npy",hdrupdate=True,flush=True) as W: W.save(A) W.save(A) ``` Feel free to comment this idea. Cheers, Gabor _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion