Vincent Schut : > Hi, I'm running in this strange issue when using some pretty large > float32 arrays. In the following code I create a large array filled with > ones, and calculate mean and sum, first with a float64 version, then > with a float32 version. Note the difference between the two. NB the > float64 version is obviously right :-) > > > > In [2]: areaGrid = numpy.ones((11334, 16002)) > In [3]: print(areaGrid.dtype) > float64 > In [4]: print(areaGrid.shape, areaGrid.min(), areaGrid.max(), > areaGrid.mean(), areaGrid.sum()) > ((11334, 16002), 1.0, 1.0, 1.0, 181366668.0) > > > In [5]: areaGrid = numpy.ones((11334, 16002), numpy.float32) > In [6]: print(areaGrid.dtype) > float32 > In [7]: print(areaGrid.shape, areaGrid.min(), areaGrid.max(), > areaGrid.mean(), areaGrid.sum()) > ((11334, 16002), 1.0, 1.0, 0.092504406598019437, 16777216.0) > > Yes I also got the same problem. b=npy.ones((11334,16002),dtype='float32') >>> a.shape[0]*a.shape[1] 181366668L >>> b.sum() 16777216.0 >>> print npy.finfo(b.dtype).max 3.40282e+38 Acumulator size is definitely not the problem. I think the float point accuracy actually kicked in. try following code: npy.float32(16777216)+npy.float32(1) You will see the number will not grow any more it is because eps(npy.float32(16777216)) = 2 >1 That is why u cannot accumulate with 1 or smaller number beyound this value. try: npy.float32(16777215)+npy.float32(0.5) and: npy.float64(1e16)+npy.float64(1) You also cannot get bigger number by accumulation anymore The numpy.sum() is simply clumsy in this aspect. It try to simply accumulate all the value together, which should always be avoided for float point value, even with float64 number. Think about add 1e12 with 1e16 values smaller than 0.0001, it will give u 1.0e12, instead of 2e12. Some one try to do smarter things like: 1) put all small value into a group, all big value into another group 2) obtain sum values respectively 3) add the sum values together But it is costy I guess > Can anybody confirm this? And better: explain it? Am I running into a > for me till now hidden ieee float 'feature'? Or is it a bug somewhere? > > Btw I'd like to use float32 arrays, as precision is not really an issue > in this case, but memory usage is... > > > This is using python 2.7, numpy from git (yesterday's checkout), on arch > linux 64bit. > > Best, > Vincent. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion