Le mardi 11 février 2014 00:10:20, David Baker (Geoscience) a écrit : > Evan. I am on Windows and only have the binaries installed.
Well, I let Windows developers lurking here answer if they have some good advice. I imagine you would need binaries with debugging symbols to be able to get a usefull stack trace. > > David > > -----Original Message----- > From: Even Rouault [mailto:even.roua...@mines-paris.org] > Sent: Monday, February 10, 2014 8:54 AM > To: David Baker (Geoscience) > Cc: 'Even Rouault'; 'Brian Case'; 'gdal-dev@lists.osgeo.org' > Subject: RE: [gdal-dev] Fast Pixel Access > > Selon "David Baker (Geoscience)" <david.m.ba...@chk.com>: > > Even, > > > > No not an i386... A Dell Precision T3500 w/Intel W3680 @ 3.33GHhz 6x2 > > cores with 12.0GB. Thought the data is on the network, not local, with > > 1Gbps access. The GDAL_DISABLE_READDIR_ON_OPEN = TRUE did significantly > > increase the speed. Does the BIL driver read the whole file into memory > > first? Might a direct read be faster? > > No the BIL driver will just read the line where the pixel is. > > > And Even, please excuse my ignorance, but what is "gdb?" I really would > > like to do the profiling. > > gdb is the GNU debugger ( https://www.gnu.org/software/gdb/ ) . Assuming > you use Linux (likely available on MacOSX too). You should be able to > install it with the usual package management system of the distribution : > apt-get install gdb, yum install gdb, ... . Otherwise on Windows, I'm far > less familiar with the debugging tools. > > gdb --args gdallocationinfo -valonly -geoloc intermap.vrt > > Then type "run" > Type a coordinate "-96.0 36.0" > ctrl+c to suspend execution > "bt" to display the stack trace > "c" to resume execution > > > David > > > > -----Original Message----- > > From: Even Rouault [mailto:even.roua...@mines-paris.org] > > Sent: Sunday, February 09, 2014 6:36 AM > > To: David Baker (Geoscience) > > Cc: 'Brian Case'; 'gdal-dev@lists.osgeo.org' > > Subject: Re: [gdal-dev] Fast Pixel Access > > > > Le samedi 01 février 2014 15:04:46, David Baker (Geoscience) a écrit : > > > Evan, > > > > > > I am not sure how to profile as I do not have access to the code to > > > profile. I did do a timing test... > > > > > > vrt file = 22,970 KB > > > bil file = 35,180 KB * 55,501 > > > > > > I piped five locations from the loc.txt file: > > > -96.0 36.0 > > > -98.0 37.0 > > > -100.0 38.0 > > > -99.0 39.0 > > > -101.0 35.0 > > > > > > gdallocationinfo -valonly -geoloc intermap.vrt < loc.txt > > > 189.841857910156 25.5 sec > > > 384.857452392578 22.6 sec > > > 762.015930175781 22.9 sec > > > 550.719116210938 23.6 sec > > > 883.637023925781 22.9 sec > > > > > > Note: I used a lap timer on my iPhone to capture the split times as the > > > results appeared in the console window. Does this give any insight? > > > > Woo I agree that's utterly slow ! When you mentionned slow I thought it > > was more in the order of 0.1 second ! We can already exclude the parsing > > time of the VRT since you do that in the same gdallocationinfo session > > and that there will be just one parsing. > > And I can't believe that the intersection test for 55 000 rectangles > > takes ~ 20 seconds, unless you have an old i386 at 5 MHz ;-) > > My usual way of profiling stuff that is slow in the order of more than > > one second is to run under gdb, break with Ctrl+C, display the stack > > trace, continue the run, break again, display the stack trace, etc.. If > > you end up breaking in the same function, then you've found the > > bottleneck. > > > > I see now that in that thread GDAL_DISABLE_READDIR_ON_OPEN = TRUE was > > suggested and seems to improve things significantly. Perhaps we should > > try to cache the result of the initial readdir so it can benefits to > > later attempts, but I haven't checked how easily that could be > > miplemented. Or perhaps we should just change the default value of > > GDAL_DISABLE_READDIR_ON_OPEN since it causes problem from time to time. > > But generally filesystems don't behave very well when there are a lot of > > files > > in the same directory. You'd better organizing your tiles in > > subdirectories. But still 1 to 3 seconds sounds a bit slow to me. Would > > be cool if you could try the above suggestion to identify where the time > > is spent. > > > > Even > > > > > David > > > > > > -----Original Message----- > > > From: gdal-dev-boun...@lists.osgeo.org > > > [mailto:gdal-dev-boun...@lists.osgeo.org] On Behalf Of Even Rouault > > > Sent: Saturday, February 01, 2014 1:28 AM > > > To: Brian Case > > > Cc: gdal-dev@lists.osgeo.org > > > Subject: Re: [gdal-dev] Fast Pixel Access > > > > > > Le samedi 01 février 2014 00:23:13, Brian Case a écrit : > > > > evenr > > > > > > > > > > > > what about the use of a tileindex? > > > > > > You really mean a tileindex as produced by gdaltindex ? Well, that's > > > not exactly the same beast as a VRT, but yes if it was recognized as a > > > GDAL dataset then you could potentially save the cost of XML parsing. > > > One could imagine that the VRT driver would accept a tileindex as an > > > altenate connection string. > > > > > > Anyway it would be interesting to first profile where the time is spent > > > in David use case. If it's in the XML parsing, then I can't see what > > > could be easily improved in that area. If it's the intersection, then > > > there's potential for improvement. > > > > > > > seems an intersection with a set of > > > > polys first would be quick > > > > > > > > > > > > > > > > brian > > > > > > > > On Fri, 2014-01-31 at 19:30 +0100, Even Rouault wrote: > > > > > Le vendredi 31 janvier 2014 17:15:53, David Baker (Geoscience) a > > > > > écrit > > > > > > > > > > > Dev's, > > > > > > > > > > > > I have a set of 55,501 bil files in a single directory. They are > > > > > > DEMS data that cover the US in 7.5 minute tiles. I would like to > > > > > > randomly access elevations at a given lat/lon's from the whole > > > > > > dataset. I created a vrt file from the directory of bil files, > > > > > > and have been able to access the elevation at a given lat/lon > > > > > > using gdallocationinfo, but because of the size of the dataset, > > > > > > this operation is somewhat slow. Can the vrt be indexed? > > > > > > > > > > No, it isn't currently, although I think it could be improved to > > > > > have a in- memory index with moderate effort. > > > > > > > > > > But are you sure the slowness is due to the lack of index ? 55,000 > > > > > is a big number, but not that big. Maybe the slowness just comes > > > > > from the opening time (XML parsing) of such a big VRT. That would > > > > > need to be profiled to be sure where the bottleneck is. > > > > > > > > > > > Or, is there a faster, better way to access the pixels? I would > > > > > > first like to do this with the utilities before diving into code > > > > > > (C#). The files are regularly named base on their location within > > > > > > a 1 arc-second grid. > > > > > > > > > > > > Thanks, > > > > > > David > > > > > > > > > > > > David M. Baker > > > > > > Senior Advisor - Geoscience Technology > > > > > > Chesapeake Energy Corporation > > > > > > david.m.ba...@chk.com<mailto:david.m.ba...@chk.com> > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > > > > > This email (and attachments if any) is intended only for the use > > > > > > of the individual or entity to which it is addressed, and may > > > > > > contain information that is confidential or privileged and > > > > > > exempt from disclosure under applicable law. If the reader of > > > > > > this email is not the intended recipient, or the employee or > > > > > > agent responsible for delivering this message to the intended > > > > > > recipient, you are hereby notified that any dissemination, > > > > > > distribution or copying of this communication is strictly > > > > > > prohibited. If you have received this communication in error, > > > > > > please notify the sender immediately by return email and destroy > > > > > > all copies of the email (and attachments if any). > > > > > > -- > > > Geospatial professional services > > > http://even.rouault.free.fr/services.html > > > _______________________________________________ > > > gdal-dev mailing list > > > gdal-dev@lists.osgeo.org > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > > > > > ________________________________ > > > > > > This email (and attachments if any) is intended only for the use of the > > > individual or entity to which it is addressed, and may contain > > > information that is confidential or privileged and exempt from > > > disclosure under applicable law. If the reader of this email is not > > > the intended recipient, or the employee or agent responsible for > > > delivering this message to the intended recipient, you are hereby > > > notified that any dissemination, distribution or copying of this > > > communication is strictly prohibited. If you have received this > > > communication in error, please notify the sender immediately by return > > > email and destroy all copies of the email (and attachments if any). > > > > -- > > Geospatial professional services > > http://even.rouault.free.fr/services.html > > > > ________________________________ > > > > This email (and attachments if any) is intended only for the use of the > > individual or entity to which it is addressed, and may contain > > information that is confidential or privileged and exempt from > > disclosure under applicable law. If the reader of this email is not the > > intended recipient, or the employee or agent responsible for delivering > > this message to the intended recipient, you are hereby notified that any > > dissemination, distribution or copying of this communication is strictly > > prohibited. If you have received this communication in error, please > > notify the sender immediately by return email and destroy all copies of > > the email (and attachments if any). > > ________________________________ > > This email (and attachments if any) is intended only for the use of the > individual or entity to which it is addressed, and may contain information > that is confidential or privileged and exempt from disclosure under > applicable law. If the reader of this email is not the intended recipient, > or the employee or agent responsible for delivering this message to the > intended recipient, you are hereby notified that any dissemination, > distribution or copying of this communication is strictly prohibited. If > you have received this communication in error, please notify the sender > immediately by return email and destroy all copies of the email (and > attachments if any). -- Geospatial professional services http://even.rouault.free.fr/services.html _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev