Re: [GRASS-dev] vector large file support

2009-02-23 Thread Markus Metz
Markus Neteler wrote: Hi Markus, On Sun, Feb 22, 2009 at 1:56 PM, Markus Metz markus.metz.gisw...@googlemail.com wrote: This segfault in the rtree lib doesn't happen often, and I don't know if I should submit that patch also to develbranch_6 and releasebranch_6_4. For now I have submitted

Re: [GRASS-dev] vector large file support

2009-02-21 Thread Markus Metz
Markus Metz wrote: Markus Metz wrote: It seems to work, at least I got past the segfault when breaking polygons, but I still have to check spatial index dumps. diff spidx.dump.orig spidx.dump.patched spidx.diff gives me an empty spidx.diff, patched spatial index is identical to original

Re: [GRASS-dev] vector large file support

2009-02-21 Thread Moritz Lennert
On 21/02/09 10:09, Markus Metz wrote: Markus Metz wrote: Markus Metz wrote: It seems to work, at least I got past the segfault when breaking polygons, but I still have to check spatial index dumps. diff spidx.dump.orig spidx.dump.patched spidx.diff gives me an empty spidx.diff, patched

Re: [GRASS-dev] vector large file support

2009-02-18 Thread Markus Metz
Markus Metz wrote: It seems to work, at least I got past the segfault when breaking polygons, but I still have to check spatial index dumps. diff spidx.dump.orig spidx.dump.patched spidx.diff gives me an empty spidx.diff, patched spatial index is identical to original spatial index Waiting

Re: [GRASS-dev] vector large file support

2009-02-15 Thread Markus Metz
Moritz Lennert wrote: On 11/02/09 12:54, Markus Metz wrote: Maybe I should rather try to fix bugs than to add new features... here are my two top candidates: - Keep topology and spatial index in file instead of in memory in the vector ToDo. The fact that it is in memory makes simple

Re: [GRASS-dev] vector large file support

2009-02-12 Thread Markus Metz
Moritz Lennert wrote: On 11/02/09 12:54, Markus Metz wrote: IMHO LFS support for vector libs is not necessarily good, currently. In order to manipulate large vectors, a very fast CPU, fast hard drives and lots of RAM is needed, or a lot of patience, because it can take days or weeks to run

slow vector display in grass7 (was [GRASS-dev] vector large file support)

2009-02-12 Thread Markus Metz
Glynn Clements wrote: One relatively recent change that may cause a slow-down is that the display library no longer tries to automatically simplify vectors (reduce consecutive short line segments with a single segment). This could have a significant impact when viewing detailed vectors at low

Re: [GRASS-dev] vector large file support

2009-02-12 Thread Glynn Clements
Markus Metz wrote: The dglib is still a mystery to me. I don't even know what modules use it. The other vector libs don't use it, that I'm relatively sure of. The vector library (libgrass_vect, aka lib/vector/Vlib) uses it (graph.c, net.c), as does v.generalize. If you want to track symbol

Re: [GRASS-dev] vector large file support

2009-02-12 Thread Markus Metz
Glynn Clements wrote: Markus Metz wrote: The dglib is still a mystery to me. I don't even know what modules use it. The other vector libs don't use it, that I'm relatively sure of. The vector library (libgrass_vect, aka lib/vector/Vlib) uses it (graph.c, net.c), as does

Re: [GRASS-dev] vector large file support

2009-02-11 Thread Glynn Clements
Markus Metz wrote: BTW, display of grass7 with cairo appears to be half the speed of grass65, I think it was faster last week (can't remember the revision). Not sure if this justifies a complaint. Almost everything related to display has changed between 6.x and 7.0. One relatively recent

Re: [GRASS-dev] vector large file support

2009-02-11 Thread Markus Metz
Glynn Clements wrote: Markus Metz wrote: BTW, display of grass7 with cairo appears to be half the speed of grass65, I think it was faster last week (can't remember the revision). Not sure if this justifies a complaint. Almost everything related to display has changed between 6.x

Re: [GRASS-dev] vector large file support

2009-02-11 Thread Moritz Lennert
On 11/02/09 12:54, Markus Metz wrote: IMHO LFS support for vector libs is not necessarily good, currently. In order to manipulate large vectors, a very fast CPU, fast hard drives and lots of RAM is needed, or a lot of patience, because it can take days or weeks to run modules with large

Re: [GRASS-dev] vector large file support

2009-02-11 Thread Glynn Clements
Markus Metz wrote: BTW, display of grass7 with cairo appears to be half the speed of grass65, I think it was faster last week (can't remember the revision). Not sure if this justifies a complaint. Almost everything related to display has changed between 6.x and 7.0. One

Re: [GRASS-dev] vector large file support

2009-02-09 Thread Glynn Clements
Markus Metz wrote: I have read The issues and understand the problem, but some sort of implementation of G_fseek and G_ftell is needed, otherwise modules and libraries need a workaround like the iostream library is doing now. Instead of having many (potentially different) workarounds,

Re: [GRASS-dev] vector large file support

2009-02-09 Thread Markus Metz
Glynn Clements wrote: I have added G_fseek() and G_ftell() to 7.0 in r35818. :-) I got confused by this endian-ness and confused low/high word with first/second word. With the current code, the low word would be the second word when doing 2 32bit reads on a 64bit sized buffer,

Re: [GRASS-dev] vector large file support

2009-02-08 Thread Markus Metz
Glynn Clements wrote: Markus Metz wrote: I like the point of Ivan that off_t is the native type for file offsets. Could G_fseek then use fseeko whenever fseeko is available (ditto for ftello)? Well, that's the general idea. The only advantage of fseek/ftell is that they are always

Re: [GRASS-dev] vector large file support

2009-02-07 Thread Markus Metz
Glynn Clements wrote: lseek() always uses off_t. Originally it used long (hence the name lseek), but that's ancient history; you won't find such a system outside of a museum. _FILE_OFFSET_BITS determines whether off_t is 32 or 64 bits. If it's 64 bits, many of the POSIX I/O functions (open,

Re: [GRASS-dev] vector large file support

2009-02-06 Thread Markus Metz
Glynn Clements wrote: Markus Metz wrote: [...] How about extern off_t G_ftell(FILE *fp) { #ifdef HAVE_LARGEFILES return (ftello(fp); #else return (ftell(fp); #endif } Yep, other than the extraneous open parenthesis (2 open, 1 close). Oops, my sloppy

Re: [GRASS-dev] vector large file support

2009-02-06 Thread Glynn Clements
Markus Metz wrote: What about off_t lseek(int fd, off_t offset, int whence) ? From the GNU C library: The lseek function is the underlying primitive for the fseek, fseeko, ftell, ftello and rewind functions [...] lseek is used in libgis and several modules, I didn't see something like

Re: [GRASS-dev] vector large file support

2009-02-05 Thread Markus Metz
Glynn Clements wrote: Markus Metz wrote: I think I understand. So according to the grass wiki the steps to enable large file support would be 1) add ifneq ($(USE_LARGEFILES),) EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64 endif to all relevant Makefiles Yes. Although this should

Re: [GRASS-dev] vector large file support

2009-02-05 Thread Glynn Clements
Markus Metz wrote: 3) solve the fseek/fseeko and ftell/ftello problem. Get inspiration from libgis and LFS-safe modules? Or as suggested in the grass wiki on LFS, add extern off_t G_ftell(FILE *fp); extern int G_fseek(FILE *stream, off_t offset, int whence); for global use?

Re: [GRASS-dev] vector large file support

2009-02-04 Thread Markus Metz
I tried to understand the grass wiki on Large File Support, sorry for being a bit late with that! Glynn Clements wrote: Markus Metz wrote: If the coor file size stored in the topo file is indeed needed to properly process the coor file, the respective variables must be something else

[GRASS-dev] vector large file support

2009-02-03 Thread Markus Metz
While I looked into possibilities to optimize v.in.ogr I noticed that grass does not support coor files larger than 2 GB. With topological information stored in that file, and often many dead lines wasting space, the coor file can easily exceed 2 GB nowadays. While v.in.ogr was cleaning one

Re: [GRASS-dev] vector large file support

2009-02-03 Thread Glynn Clements
Markus Metz wrote: If the coor file size stored in the topo file is indeed needed to properly process the coor file, the respective variables must be something else than long in order to support coor files larger than 2 GB, maybe long long? Same for all intermediate variables in the