On Tue, Mar 4, 2014 at 8:53 PM, Markus Metz <markus.metz.gisw...@gmail.com> wrote: > On Tue, Mar 4, 2014 at 5:13 PM, Martin Landa <landa.mar...@gmail.com> wrote: >> Hi, >> >> [I moved this discussion to grass-dev ML which seems to be more appropriate] >> >> 2014-03-03 23:27 GMT+01:00 Markus Metz <markus.metz.gisw...@gmail.com>: >>>> -> after 110min failed with "ERROR: G_calloc: unable to allocate 101 * >>>> 4 bytes of memory at allocation.c:81" (when Finding centroids for OGR >>>> layer <buildings>...), memory peak 3.6GB >>> >>> This works for me on a desktop machine with quadcore AMD Phenom @ 4GHz >>> and 16 GB RAM with Scientific Linux. Memory peak was 6.3 GB and it >>> took 1 hour, but that included snapping which costs time and memory >>> (my first attempt was also successful but I didn't measure processing >>> time), and I processed at the same time a time series of climate data >>> in a different GRASS session. There are a number of topological errors >>> in the shapefile, amongst others there are 6619 overlapping areas >>> which are too large to be removed by snapping and might need manual >>> cleaning. >>> >>> The memory allocation errors you experienced are strange, they should >>> not happen on a system with 48 GB RAM. Can you hack v.in.ogr and make >>> it try to allocate e.g. 8 GB of memory right at the beginning? If that >>> fails, you could check how much RAM is free on your system. Disk space >>> can also be a limiting factor but seems unlikely in your case. >> >> right, this strange, I used code bellow >> >> int i; >> int base = 1e6; >> char *buf = NULL; >> for(i = 1 ; i <= 8000; i++) { >> G_debug(0, "%d -> %ld", i, i * base); >> buf = G_realloc(buf, i * base); >> if (i % 100 == 0) >> sleep(1); >> } >> >> it fails when trying to allocate about 2GB, > > Since i and base are both int, integer overflow will occur for i * > base > 2^31 - 1 and i * base will be converted to 1 by G_realloc(). > The second argument to G_realloc() must be of type size_t, see also > man realloc. > > Thus, no wonder that the test fails when trying to allocate more than > 2GB, but it does not explain why v.in.ogr fails. Did you perform a > memory test with memtest? > >> >> D0/0: 2047 -> 2047000000 >> D0/0: 2048 -> 2048000000 >> Current region rows: 3, cols: 7 >> ERROR: G_realloc: unable to allocate 2048000000 bytes of memory at main.c:101
I was wrong, 2048000000 is smaller than 2^31 - 1, thus no integer overflow. >> >> There is memory available and no limits per process: Are you sure that there are no limits per process? Markus M >> >> $ free -m >> total used free shared buffers cached >> Mem: 48395 4587 43808 0 3272 361 >> -/+ buffers/cache: 953 47441 >> Swap: 47682 0 47682 >> >> Trying to investigate more. In any case thanks for testing... Martin > > I get without GRASS_VECTOR_LOWMEM a memory peak of 8.3GB and 22 min > processing time. The conversion from non-topological vector to > topological vector needs some time. > > With GRASS_VECTOR_LOWMEM I get a memory peak of 6.3GB. I would have > expected a larger difference with regard to GRASS_VECTOR_LOWMEM. > > Markus M _______________________________________________ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev