Re: [gdal-dev] GDAL Speed Optimization
Frank, I have increased the cache as you suggested using GDALSetCacheMax and it produced a performance gain. Thanks. I think I can improve more acting on the virtual datataset that I use to adapt the image to the paper size. I have to look into the tms driver in order to understand how it download the tiles to compose the final image. Stefano On Fri, Jun 11, 2010 at 6:17 PM, Frank Warmerdam warmer...@pobox.comwrote: Stefano Moratto wrote: Frank, The source images are a sequence of tiles from openstreetmaps. Every tiles has a fixed size (256 x 256) and the area they covers depends from the zoom level. It is explained very well in http://www.maptiler.org/google-maps-coordinates-tile-bounds-projection. I use interpolation (via a virtual dataset) because my window size and the paper size are not a multiple of 256 x 256. I use reprojection because I've to mix this raster datasource with a vector datasource that has a different projection. Reprojecting my vector data to openstreet projection do not affect the performance. So I think the key factor is the resizing ( I use bilinear interpolation) and the size of the bitmap that may cause paging. ... Can you suggest some other parameters? Stefano, Sorry, I wasn't able to learn much more skimming the code and referenced web page. Have you tried the cache size setting I suggested? Best regards, ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdamhttp://pobox.com/%7Ewarmerdam and watch the world go round - Rush| Geospatial Programmer for Rent -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL Speed Optimization
Iván, I evaluated the use of mapnik but I develop on Windows and setup it on that platform is not easy as on Unix/Linux. Also I develop in delphi, so I have do create the bindings by myself I would like to use mapnik in my project as a replacement of my custom render but I have to transform the openstreet data in my own format (I use a relational version of the GDF) because I want not to add another db server. I should have to had a new datasource for mapnik and in order to do that I must be able to recompile it ;-) My application is a desktop application and my customer have to produce some reports for the local area government in order to plan alternative path in case of hazard in the highway. The deployment term in near so I have no time to try the direct use of mapnk. The actual solution works but it is not rapid as I would like ( I have to wait for 30 seconds in order to get an a3 bitmap). Have you experience of mapnik on windows? Regards, Stefano 2010/6/11 Iván Sánchez Ortega i...@sanchezortega.es El día Thursday 10 June 2010 09:49:01, Stefano Moratto dijo: I use GDAL to download tiles (jpeg) from OpenstreetMap and to compose a bitmap of the area that is being viewed. The resulting bitmap (not compressed) is warped and displayed. Hhhhmmm. Personally, I think you're doing it wrong. If you really need speed for custom reprojections of OSM data, the best thing to do is to render the data locally with Mapnik. i.e.: - Download a planet file - Put it in PostGIS via osm2pgsql - Configure Mapnik to use the projection you need - Use Mapnik to get the desired area rendered You might incur in a speed penalty due to the rendering itself, but you guarantee that there will be no artifacts due to the image warping. And also, you might get a boost by reprojecting the vector data inside PostGIS so Mapnik doesn't have to reproject it on-the-fly. I realise that setting Mapnik is quite a lot of work at first, but it's an idea that is worth considering in your case. Cheers, -- Iván Sánchez Ortega i...@sanchezortega.es Un ordenador no es una televisión ni un microondas: es una herramienta compleja. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL Speed Optimization
Frank, The source images are a sequence of tiles from openstreetmaps. Every tiles has a fixed size (256 x 256) and the area they covers depends from the zoom level. It is explained very well in http://www.maptiler.org/google-maps-coordinates-tile-bounds-projection. I use interpolation (via a virtual dataset) because my window size and the paper size are not a multiple of 256 x 256. I use reprojection because I've to mix this raster datasource with a vector datasource that has a different projection. Reprojecting my vector data to openstreet projection do not affect the performance. So I think the key factor is the resizing ( I use bilinear interpolation) and the size of the bitmap that may cause paging. I use the following : hWarp := GDALAutoCreateWarpedVRT (hSized, nil, PChar(FMapSRSWkt), GDALResampleAlg_GRA_NearestNeighbour, 0.5, nil); where hSized is a hSized = F (hDataset) ; F (hDataset) := Virtual Dataset of hDataset. I use it for in memory resizing . It is generated by: (I do not write all the details but I think it is clear) VRTAddSimpleSource (poVRTBand, poSrcBand, Round(anSrcWin[0]), Round(anSrcWin[1]), Round(anSrcWin[2]), Round(anSrcWin[3]), 0, 0, size.cx, size.cy,'Bilinear', 0.0 ); and hDataset = GDALOpen( ... openstreet.xml ) Can you suggest some other parameters? Thanks, Stefano On Thu, Jun 10, 2010 at 4:53 PM, Frank Warmerdam warmer...@pobox.comwrote: Stefano Moratto wrote: The performances are quite acceptable but when I try to print they are not. The drawing to be printed has a resolution larger of the screen (Screen : 1024x1024, Printer 4096 x 4094 in A4 and 9000 x 9000 in A3 approximately). I think tha the bottleneck could be found in: 1) jpeg decompression. 2) bitmap interpolation (I use the low qualitiy settings). 3) warping ( I use an approsimated warping function). An increase of performance could be achieved using 1) hw accelerated functions as IPP or GPU (e.g CUDA) 2) parallel alghorithms that takes advantage from multicore CPU Has someone already approched these problems? Stefano, Are the same number of jpeg tiles being used when producing print products as on screen? I'm assuming the answer is yes, and that the warper is just interpolating a higher resolution result from them. If the answer is yes, then presumably the time decoding the jpegs would not be greater for the prints than for screen display, though it is possible that more re-reading is occuring. If re-reading is an issue you might want to increase the size of the GDAL block cache. Programmatically this can be done with the C function GDALSetCacheMax(). Try setting it to 100 (for 100MB). Are you using the warper just to interpolate to different sizes or are you also applying reprojection or some other non-linear warping? If you are just using it to resize the image then I would suggest using something less general-purpose than the warper to do the interpolation. Seth's CUDA work may also be of interest of course, but before committing to that path I would suggest some analysis of where the time is really being spent. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdamhttp://pobox.com/%7Ewarmerdam and watch the world go round - Rush| Geospatial Programmer for Rent -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL Speed Optimization
El 11/06/2010 11:04, Stefano Moratto escribió: I evaluated the use of mapnik but I develop on Windows and [...] The deployment term in near so I have no time to try the direct use of mapnik. OK, if you evaluated using Mapnik, then I said nothing :-) Have you experience of mapnik on windows? Unfortunately, I only use Debian for development. :-( -- Iván Sánchez Ortega i...@sanchezortega.es ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL Speed Optimization
1) hw accelerated functions as IPP or GPU (e.g CUDA) This is my (ongoing) Google Summer of Code project, except I'm using OpenCL. :D What resampling algorithm are you using? ~Seth On Thu, June 10, 2010 1:49 am, Stefano Moratto wrote: I use GDAL in my traffic optimization CAD program. It is a Win32 application written in DELPHI that uses GDAL C API - the binding was autogenerated by my SWIG module for object pascal. I use GDAL to download tiles (jpeg) from OpenstreetMap and to compose a bitmap of the area that is being viewed. The resulting bitmap (not compressed) is warped and displayed. The performances are quite acceptable but when I try to print they are not. The drawing to be printed has a resolution larger of the screen (Screen : 1024x1024, Printer 4096 x 4094 in A4 and 9000 x 9000 in A3 approximately). I think tha the bottleneck could be found in: 1) jpeg decompression. 2) bitmap interpolation (I use the low qualitiy settings). 3) warping ( I use an approsimated warping function). An increase of performance could be achieved using 1) hw accelerated functions as IPP or GPU (e.g CUDA) 2) parallel alghorithms that takes advantage from multicore CPU Has someone already approched these problems? Regards, Stefano -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Re: [gdal-dev] GDAL Speed Optimization
I'm working on a VirtualMachine running on an usb hdd and my customer use iMac laptop so I can't change disk. The primary dataset is composed by a lot of jpeg download from internet and then cached to disk. I think the performance slowdown is caused after the creation of the virtual dataset because and it is present only when the output bitmap has the resolution of the printer (considering alwarys the sampre geographic area). All the output bitmaps, exept the dataset's jpeg, are in memory and the workingset of the application does not exeed the 256 MB Can you suggest a profiler? Thanks, Stefano Il giorno 10/giu/2010 10.31, Seth Price s...@pricepages.org ha scritto: Unfortunately, both nearest neighbor and bilinear are probably not CPU bound, so speeding up their processing won't help. They are probably I/O bound, so you might need faster disks. (Though I can test this later in my project.) Have you tried running a profiling tool on GDAL while you're running the warper? I've just started the OpenCL project, so it's still pretty immature. I'm still discussing the best way to integrate an OpenCL warper into existing code. ~Seth On Thu, June 10, 2010 2:26 am, Stefano Moratto wrote: Seth, You have been choosen a very interesting project. I use the following : hWarp := GDALAutoCreateWarpedVRT (hSized, nil, PChar(FMapSRSWkt), GDALResampleAlg_GRA_NearestNeighbour, 0.5, nil); where hSized is a hSized = F (hDataset) ; F (hDataset) := Virtual Dataset of hDataset. I use it for in memory resizing . It is generated by: (I do not write all the details but I think it is clear) VRTAddSimpleSource (poVRTBand, poSrcBand, Round(anSrcWin[0]), Round(anSrcWin[1]), Round(anSrcWin[2]), Round(anSrcWin[3]), 0, 0, size.cx, size.cy,'Bilinear', 0.0 ); and hDataset = GDALOpen( ... openstreet.xml ) Hence I use 1) ,'Bilinear' for zooming 2) NearestNeighbour for warping. What is the state of your project? Stefano On Thu, Jun 10, 2010 at 9:52 AM, Seth Price s...@pricepages.org wrote: 1) hw accelerated functions as IPP or GPU (eg CUDA) This is my (ongoing) Google Summer of Code project, except I'm using OpenCL. :D What resampling algorithm are you using? ~Seth On Thu, June 10, 2010 1:49 am, Stefano Moratto wrote: I use GDAL in my traffic optimization CAD program. It is a Win32 application written in DELPHI that uses GDAL C API - the binding was autogenerated by my SWIG module for object pascal. I use GDAL to download tiles (jpeg) from OpenstreetMap and to compose a bitmap of the area that is being viewed. The resulting bitmap (not compressed) is warped and displayed. The performances are quite acceptable but when I try to print they are not. The drawing to be printed has a resolution larger of the screen (Screen : 1024x1024, Printer 4096 x 4094 in A4 and 9000 x 9000 in A3 approximately). I think tha the bottleneck could be found in: 1) jpeg decompression. 2) bitmap interpolation (I use the low qualitiy settings). 3) warping ( I use an approsimated warping function). An increase of performance could be achieved using 1) hw accelerated functions as IPP or GPU (eg CUDA) 2) parallel alghorithms that takes advantage from multicore CPU Has someone already approched these problems? Regards, Stefano -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL Speed Optimization
I'm on a Mac, so I normally use Shark for profiling. It's included with Apple's developer tools. I would definitely try running outside of a virtual machine environment. That might be your problem, but you won't know until you try. ~Seth On Jun 10, 2010, at 2:46 AM, stefano.mora...@gmail.com wrote: I'm working on a VirtualMachine running on an usb hdd and my customer use iMac laptop so I can't change disk. The primary dataset is composed by a lot of jpeg download from internet and then cached to disk. I think the performance slowdown is caused after the creation of the virtual dataset because and it is present only when the output bitmap has the resolution of the printer (considering alwarys the sampre geographic area). All the output bitmaps, exept the dataset's jpeg, are in memory and the workingset of the application does not exeed the 256 MB Can you suggest a profiler? Thanks, Stefano Il giorno 10/giu/2010 10.31, Seth Price s...@pricepages.org ha scritto: Unfortunately, both nearest neighbor and bilinear are probably not CPU bound, so speeding up their processing won't help. They are probably I/O bound, so you might need faster disks. (Though I can test this later in my project.) Have you tried running a profiling tool on GDAL while you're running the warper? I've just started the OpenCL project, so it's still pretty immature. I'm still discussing the best way to integrate an OpenCL warper into existing code. ~Seth On Thu, June 10, 2010 2:26 am, Stefano Moratto wrote: Seth, You have been choosen a very interesting project. I use the following : hWarp := GDALAutoCreateWarpedVRT (hSized, nil, PChar(FMapSRSWkt), GDALResampleAlg_GRA_NearestNeighbour, 0.5, nil); where hSized is a hSized = F (hDataset) ; F (hDataset) := Virtual Dataset of hDataset. I use it for in memory resizing . It is generated by: (I do not write all the details but I think it is clear) VRTAddSimpleSource (poVRTBand, poSrcBand, Round(anSrcWin[0]), Round(anSrcWin[1]), Round(anSrcWin[2]), Round(anSrcWin[3]), 0, 0, size.cx, size.cy,'Bilinear', 0.0 ); and hDataset = GDALOpen( ... openstreet.xml ) Hence I use 1) ,'Bilinear' for zooming 2) NearestNeighbour for warping. What is the state of your project? Stefano On Thu, Jun 10, 2010 at 9:52 AM, Seth Price s...@pricepages.org wrote: 1) hw accelerated functions as IPP or GPU (e.g CUDA) This is my (ongoing) Google Summer of Code project, except I'm using OpenCL. :D What resampling algorithm are you using? ~Seth On Thu, June 10, 2010 1:49 am, Stefano Moratto wrote: I use GDAL in my traffic optimization CAD program. It is a Win32 application written in DELPHI that uses GDAL C API - the binding was autogenerated by my SWIG module for object pascal. I use GDAL to download tiles (jpeg) from OpenstreetMap and to compose a bitmap of the area that is being viewed. The resulting bitmap (not compressed) is warped and displayed. The performances are quite acceptable but when I try to print they are not. The drawing to be printed has a resolution larger of the screen (Screen : 1024x1024, Printer 4096 x 4094 in A4 and 9000 x 9000 in A3 approximately). I think tha the bottleneck could be found in: 1) jpeg decompression. 2) bitmap interpolation (I use the low qualitiy settings). 3) warping ( I use an approsimated warping function). An increase of performance could be achieved using 1) hw accelerated functions as IPP or GPU (e.g CUDA) 2) parallel alghorithms that takes advantage from multicore CPU Has someone already approched these problems? Regards, Stefano -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing
Re: [gdal-dev] GDAL Speed Optimization
Stefano Moratto wrote: The performances are quite acceptable but when I try to print they are not. The drawing to be printed has a resolution larger of the screen (Screen : 1024x1024, Printer 4096 x 4094 in A4 and 9000 x 9000 in A3 approximately). I think tha the bottleneck could be found in: 1) jpeg decompression. 2) bitmap interpolation (I use the low qualitiy settings). 3) warping ( I use an approsimated warping function). An increase of performance could be achieved using 1) hw accelerated functions as IPP or GPU (e.g CUDA) 2) parallel alghorithms that takes advantage from multicore CPU Has someone already approched these problems? Stefano, Are the same number of jpeg tiles being used when producing print products as on screen? I'm assuming the answer is yes, and that the warper is just interpolating a higher resolution result from them. If the answer is yes, then presumably the time decoding the jpegs would not be greater for the prints than for screen display, though it is possible that more re-reading is occuring. If re-reading is an issue you might want to increase the size of the GDAL block cache. Programmatically this can be done with the C function GDALSetCacheMax(). Try setting it to 100 (for 100MB). Are you using the warper just to interpolate to different sizes or are you also applying reprojection or some other non-linear warping? If you are just using it to resize the image then I would suggest using something less general-purpose than the warper to do the interpolation. Seth's CUDA work may also be of interest of course, but before committing to that path I would suggest some analysis of where the time is really being spent. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
