On Tue, 2008-06-10 at 08:46 +0100, Tom Hughes wrote:
> In message <[EMAIL PROTECTED]>
> Jon Burgess <[EMAIL PROTECTED]> wrote:
>
> > The recent Mapnik builds try to mmap() the shapefiles while
> > rendering[1]. Since the largest shapefile is a few hundred MB this could
> > easily cause some issues if the file was mmap'd multiple times in a
> > single process. This might be a particular problem if you use a 32-bit
> > system in the multi-threaded Apache worker mode.
>
> Um... no. At least not if it does the sensible thing and maps it
> for shared read. Then it will only exist once in memory not matter
> how many times a process maps it, or indeed how many processes map
> it in.
Yes it'll be once in physical memory. My other email in this thread
shows that that each thread maps it to a different virtual address and
32 bit processes can be quite limited in virtual address space.
The attached test program can only map the 313MB shapefile 8 times
before it fails when run on a plain 32 bit machine:
asus:/tmp$ gcc -o mtest mtest.c
asus:/tmp$ ./mtest
File length: 328204288
Mapping: 0 => 0xa46a3000
Mapping: 1 => 0x90da3000
Mapping: 2 => 0x7d4a3000
Mapping: 3 => 0x69ba3000
Mapping: 4 => 0x562a3000
Mapping: 5 => 0x429a3000
Mapping: 6 => 0x2f0a3000
Mapping: 7 => 0x1b7a3000
Mapping: 8 => Mapping failed at 8
asus:/tmp$ uname -a
Linux asus 2.6.23.12-52.fc7 #1 SMP Tue Dec 18 21:18:02 EST 2007 i686
i686 i386 GNU/Linux
> In fact that is probably the ideal way to access a file like that
> as you effectively have one disk backed cache of the file that is
> shared by all users of it.
Yes, I agree with the mmap() approach in principle. It is much faster
than the old one which did thousands of seeks when reading the files.
Jon
#include <stdio.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#define SHP "/home/jburgess/osm/data/world_boundaries/processed_p.shp"
int main(void)
{
void *m[100];
int fd = open(SHP, O_RDONLY);
int i;
size_t len;
if (fd < 0) {
perror(SHP);
exit(-1);
}
len = lseek(fd, 0, SEEK_END);
printf("File length: %zd\n", len);
for (i=0; i<100; i++) {
printf("Mapping: %d => ", i);
m[i] = mmap(NULL, len, PROT_READ, MAP_PRIVATE, fd, 0);
if (m[i] == MAP_FAILED) {
printf("Mapping failed at %d\n", i);
break;
} else
printf("%p\n", m[i]);
}
for(;i>=0; i--)
munmap(m[i], len);
close(fd);
return 0;
}
_______________________________________________
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
