Re: [MapServer-users] Speeding up msDrawMap() for shapefiles
I think that for my case, it probably comes down to the S3 access times in general. Going to investigate more of what can be done to improve that side of the situation. Thanks everyone for all the input though! On Mon, Dec 18, 2023 at 2:35 PM Matthew Graber wrote: > I hadn't found that, thanks! It looks like it's largely suggesting what > I've already investigated though, using `shptree` to create the .qix > spatial index and `coshp` to sort the shapefile. > > On Mon, Dec 18, 2023 at 1:37 PM Steve Lime wrote: > >> Did you have a look at https://blog.cleverelephant.ca/2022/04/coshp.html? >> >> On Mon, Dec 18, 2023 at 2:39 PM Matthew Graber via MapServer-users < >> mapserver-users@lists.osgeo.org> wrote: >> >>> Hi Jukka, >>> >>> Thanks for the advice. I was able to confirm through local testing that >>> using "DATA" is consistently faster than using the OGR connection (was >>> getting over 500 ms for OGR vs just over 300 ms for DATA). My results are >>> also suggesting that performing the shapefile optimizations may make my >>> request several hundredths of a second faster, I'm a little less sure on >>> that one. But regardless, it appears that the bulk of the problem with the >>> slow S3 connection is likely S3's overhead, so I don't think there's >>> anything more I can do with MapServer to speed it up. >>> >>> Thanks again, >>> Matt >>> >>> On Fri, Dec 8, 2023 at 11:58 AM Rahkonen Jukka < >>> jukka.rahko...@maanmittauslaitos.fi> wrote: >>> >>>> Hi, >>>> >>>> >>>> >>>> You should test the same shapefiles locally for understanding where the >>>> time gets spent. Normal time with simple styles is perhaps up to 300 ms. >>>> The optimization hints were written long before there were S3 or Amazon >>>> (the company). >>>> >>>> Unfortunately I don’t know how to optimize access to S3. >>>> >>>> >>>> >>>> -Jukka Rahkonen- >>>> >>>> >>>> >>>> *Lähettäjä:* MapServer-users >>>> *Puolesta >>>> *Matthew Graber via MapServer-users >>>> *Lähetetty:* perjantai 8. joulukuuta 2023 21.47 >>>> *Vastaanottaja:* mapserver-users@lists.osgeo.org >>>> *Aihe:* [MapServer-users] Speeding up msDrawMap() for shapefiles >>>> >>>> >>>> >>>> Hi, >>>> >>>> >>>> >>>> I'm trying to speed up the time required for msDrawMap() to execute for >>>> rendering shapefiles. Currently, according to the mapserver logs, it's >>>> taking 2.5-3.5 seconds, sometimes up to 5 seconds for it to execute: >>>> >>>> >>>> >>>> [Wed Dec 6 18:43:47 2023].699791 CGI Request 53 on process 2641 >>>> [Wed Dec 6 18:43:47 2023].717216 msDrawMap(): rendering using >>>> outputformat named png (AGG/PNG). >>>> [Wed Dec 6 18:43:52 2023].299580 msDrawMap(): Layer 1010 >>>> (Shapefile_Layer), 4.582s >>>> [Wed Dec 6 18:43:52 2023].299792 msDrawMap(): Drawing Label Cache, >>>> 0.000s >>>> [Wed Dec 6 18:43:52 2023].299803 msDrawMap() total time: 4.584s >>>> [Wed Dec 6 18:43:52 2023].310853 msSaveImage(stdout) total time: 0.011s >>>> >>>> [Wed Dec 6 18:43:52 2023].310883 mapserv request processing time >>>> (msLoadMap not incl.): 4.611s >>>> >>>> >>>> >>>> The mapfile is currently set up using "CONNECTIONTYPE OGR" and >>>> "CONNECTION" to access the shapefiles from an S3 bucket as follows: >>>> >>>> >>>> >>>> CONNECTIONTYPE OGR >>>> CONNECTION'/vsis3/s3_bucket_path/my_shapefile%.shp' >>>> >>>> >>>> >>>> I noticed at https://mapserver.org/input/vector/shapefiles.html that >>>> it says "Shapefile access is built directly into MapServer. It is also >>>> available through OGR, but direct access without OGR is recommended and >>>> discussed here." >>>> >>>> So I tried removing "CONNECTIONTYPE OGR" and changing "CONNECTION" to >>>> "DATA" as follows: >>>> >>>> >>>> >>>> DATA'/vsis3/s3_bucket_path/my_shapefile%.shp' >>>> >>>> >>>> >>>> I haven't been able to observe any definitive speed improvements by >>>> doing this yet, although I can't say for sure that there aren't any. >>>> >>>> >>>> >>>> Based on https://mapserver.org/optimization/vector.html, I also tried >>>> using the shptree command to create a spatial index and the coshp command >>>> to sort the shapefile and index. Even after doing that, I still see load >>>> msDrawMap() times of up to 5 seconds. >>>> >>>> >>>> >>>> Does anyone know of any other optimizations that could be made here? Or >>>> if not, would using shptree and coshp on the shapefile and then accessing >>>> it in the mapfile using "DATA" to take it from S3 be the optimal way to set >>>> things up? >>>> >>>> >>>> >>>> Thanks! >>>> >>>> Matt >>>> >>> ___ >>> MapServer-users mailing list >>> MapServer-users@lists.osgeo.org >>> https://lists.osgeo.org/mailman/listinfo/mapserver-users >>> >> ___ MapServer-users mailing list MapServer-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [MapServer-users] Speeding up msDrawMap() for shapefiles
Hi Jukka, Thanks for the advice. I was able to confirm through local testing that using "DATA" is consistently faster than using the OGR connection (was getting over 500 ms for OGR vs just over 300 ms for DATA). My results are also suggesting that performing the shapefile optimizations may make my request several hundredths of a second faster, I'm a little less sure on that one. But regardless, it appears that the bulk of the problem with the slow S3 connection is likely S3's overhead, so I don't think there's anything more I can do with MapServer to speed it up. Thanks again, Matt On Fri, Dec 8, 2023 at 11:58 AM Rahkonen Jukka < jukka.rahko...@maanmittauslaitos.fi> wrote: > Hi, > > > > You should test the same shapefiles locally for understanding where the > time gets spent. Normal time with simple styles is perhaps up to 300 ms. > The optimization hints were written long before there were S3 or Amazon > (the company). > > Unfortunately I don’t know how to optimize access to S3. > > > > -Jukka Rahkonen- > > > > *Lähettäjä:* MapServer-users > *Puolesta > *Matthew Graber via MapServer-users > *Lähetetty:* perjantai 8. joulukuuta 2023 21.47 > *Vastaanottaja:* mapserver-users@lists.osgeo.org > *Aihe:* [MapServer-users] Speeding up msDrawMap() for shapefiles > > > > Hi, > > > > I'm trying to speed up the time required for msDrawMap() to execute for > rendering shapefiles. Currently, according to the mapserver logs, it's > taking 2.5-3.5 seconds, sometimes up to 5 seconds for it to execute: > > > > [Wed Dec 6 18:43:47 2023].699791 CGI Request 53 on process 2641 > [Wed Dec 6 18:43:47 2023].717216 msDrawMap(): rendering using > outputformat named png (AGG/PNG). > [Wed Dec 6 18:43:52 2023].299580 msDrawMap(): Layer 1010 > (Shapefile_Layer), 4.582s > [Wed Dec 6 18:43:52 2023].299792 msDrawMap(): Drawing Label Cache, 0.000s > [Wed Dec 6 18:43:52 2023].299803 msDrawMap() total time: 4.584s > [Wed Dec 6 18:43:52 2023].310853 msSaveImage(stdout) total time: 0.011s > > [Wed Dec 6 18:43:52 2023].310883 mapserv request processing time > (msLoadMap not incl.): 4.611s > > > > The mapfile is currently set up using "CONNECTIONTYPE OGR" and > "CONNECTION" to access the shapefiles from an S3 bucket as follows: > > > > CONNECTIONTYPE OGR > CONNECTION'/vsis3/s3_bucket_path/my_shapefile%.shp' > > > > I noticed at https://mapserver.org/input/vector/shapefiles.html that it > says "Shapefile access is built directly into MapServer. It is also > available through OGR, but direct access without OGR is recommended and > discussed here." > > So I tried removing "CONNECTIONTYPE OGR" and changing "CONNECTION" to > "DATA" as follows: > > > > DATA'/vsis3/s3_bucket_path/my_shapefile%.shp' > > > > I haven't been able to observe any definitive speed improvements by doing > this yet, although I can't say for sure that there aren't any. > > > > Based on https://mapserver.org/optimization/vector.html, I also tried > using the shptree command to create a spatial index and the coshp command > to sort the shapefile and index. Even after doing that, I still see load > msDrawMap() times of up to 5 seconds. > > > > Does anyone know of any other optimizations that could be made here? Or if > not, would using shptree and coshp on the shapefile and then accessing it > in the mapfile using "DATA" to take it from S3 be the optimal way to set > things up? > > > > Thanks! > > Matt > ___ MapServer-users mailing list MapServer-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[MapServer-users] Speeding up msDrawMap() for shapefiles
Hi, I'm trying to speed up the time required for msDrawMap() to execute for rendering shapefiles. Currently, according to the mapserver logs, it's taking 2.5-3.5 seconds, sometimes up to 5 seconds for it to execute: [Wed Dec 6 18:43:47 2023].699791 CGI Request 53 on process 2641 [Wed Dec 6 18:43:47 2023].717216 msDrawMap(): rendering using outputformat named png (AGG/PNG). [Wed Dec 6 18:43:52 2023].299580 msDrawMap(): Layer 1010 (Shapefile_Layer), 4.582s [Wed Dec 6 18:43:52 2023].299792 msDrawMap(): Drawing Label Cache, 0.000s [Wed Dec 6 18:43:52 2023].299803 msDrawMap() total time: 4.584s [Wed Dec 6 18:43:52 2023].310853 msSaveImage(stdout) total time: 0.011s [Wed Dec 6 18:43:52 2023].310883 mapserv request processing time (msLoadMap not incl.): 4.611s The mapfile is currently set up using "CONNECTIONTYPE OGR" and "CONNECTION" to access the shapefiles from an S3 bucket as follows: CONNECTIONTYPE OGR CONNECTION'/vsis3/s3_bucket_path/my_shapefile%.shp' I noticed at https://mapserver.org/input/vector/shapefiles.html that it says "Shapefile access is built directly into MapServer. It is also available through OGR, but direct access without OGR is recommended and discussed here." So I tried removing "CONNECTIONTYPE OGR" and changing "CONNECTION" to "DATA" as follows: DATA'/vsis3/s3_bucket_path/my_shapefile%.shp' I haven't been able to observe any definitive speed improvements by doing this yet, although I can't say for sure that there aren't any. Based on https://mapserver.org/optimization/vector.html, I also tried using the shptree command to create a spatial index and the coshp command to sort the shapefile and index. Even after doing that, I still see load msDrawMap() times of up to 5 seconds. Does anyone know of any other optimizations that could be made here? Or if not, would using shptree and coshp on the shapefile and then accessing it in the mapfile using "DATA" to take it from S3 be the optimal way to set things up? Thanks! Matt ___ MapServer-users mailing list MapServer-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [MapServer-users] Changing opacity of one layer in a multilayer WMS request URL with MapServer 8
Glad I could help, Jeff! MapServer.pdf is a great resource! -Matt On Tue, Feb 21, 2023 at 1:02 PM Jeff McKenna wrote: > Wonderful to hear that you are using the MapServer.pdf! (I actually put > much effort into maintaining that PDF output, that but I never hear > feedback...until someone reports a problem, like you did now, ha). > Sometimes my updates differ from website rebuilds, will be sure to > update the PDF for the upcoming 8.0.1 release. > > Again, thanks for this report! > > (also, don't be afraid to report doc issues like this at > https://github.com/MapServer/MapServer-documentation/issues ) > > thanks again for the report! > > -jeff > > > > > -- > Jeff McKenna > GatewayGeo: Developers of MS4W, MapServer Consulting and Training > co-founder of FOSS4G > http://gatewaygeo.com/ > > > > On 2023-02-21 12:52 p.m., Matthew Graber wrote: > > Hello all! > > > > With MapServer 8, I'm wondering what the proper way to change the > > opacity of a single layer in a multilayer WMS request is. > > > > For example, the following URL excerpt shows the old way of > > accomplishing this that worked while using MapServer 7: > > > > > SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&LAYERS=test_a,test_b&map.layer[test_b]=OPACITY+50 > > > > My understanding is that this would have the same effect as adding > > "OPACITY 50" to the test_b layer in the mapfile. The mapfile containing > > test_a and test_b did not have OPACITY included when they were written. > > Obviously, this no longer works with MapServer 8, and now that request > > has the same results as omitting the "&map.layer[test_b]=OPACITY+50" > > portion entirely. > > > > Based on https://mapserver.org/fr/MIGRATION_GUIDE.html > > <https://mapserver.org/fr/MIGRATION_GUIDE.html>, I see that OPACITY now > > belongs inside a COMPOSITE object. I tried the following modifications > > to the request URL: > > > > > SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&LAYERS=test_a,test_b&map.layer[test_b].composite=OPACITY+50 > > > SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&LAYERS=test_a,test_b&map.layer[test_b].composite[0]=OPACITY+50 > > > > I also tried this both with the mapfile left as-is, and with an empty > > COMPOSITE object being included in each layer object, and each time I > > got the same unsuccessful result as before. > > > > Is there a way to accomplish this in MapServer 8? > > > > Also, does this map.layer[layername]=... syntax still work in MapServer > > 8? The documentation at > > https://mapserver.org/cgi/controls.html#using-mapserver-version-8 > > <https://mapserver.org/cgi/controls.html#using-mapserver-version-8> > > suggests it might no longer work for MapServer 8 (based on the heading > > "Using MapServer version < 8"), but the documentation at > > https://mapserver.org/pdf/MapServer.pdf > > <https://mapserver.org/pdf/MapServer.pdf> shows the same examples with > > the heading "Using MapServer version <= 8" on page 1065 of the PDF, so > > I'm unsure if this is still intended to work in MapServer 8. > > > > Thank you, > > Matt > > > > ___ > > MapServer-users mailing list > > MapServer-users@lists.osgeo.org > > https://lists.osgeo.org/mailman/listinfo/mapserver-users > > ___ > MapServer-users mailing list > MapServer-users@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users > ___ MapServer-users mailing list MapServer-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [MapServer-users] Changing opacity of one layer in a multilayer WMS request URL with MapServer 8
Hi all, Thanks for your help! I was able to get the SLD_BODY WMS parameter to serve as a workaround for setting opacity. In case anyone else has this problem, the SLD I used in this case was: http://www.opengis.net/sld"; version="1.0.0" xsi:schemaLocation="http://www.opengis.net/sld http://schemas.opengis.net/sld/1.0.0/StyledLayerDescriptor.xsd"; xmlns:xsi=" http://www.w3.org/2001/XMLSchema-instance"; xmlns:ogc=" http://www.opengis.net/ogc"; xmlns:gml="http://www.opengis.net/gml";> test_b default 0.5 Appreciate everyone's help! -Matt On Wed, Feb 22, 2023 at 11:51 AM Rahkonen Jukka < jukka.rahko...@maanmittauslaitos.fi> wrote: > Hi, > > > > About the SLD, opacity appears in stroke, fill, graphic, and raster in SLD > 1.0. > > > > -Jukka Rahkonen- > > > > *Lähettäjä:* MapServer-users > *Puolesta > *Steve Lime > *Lähetetty:* keskiviikko 22. helmikuuta 2023 17.47 > *Vastaanottaja:* Matthew Graber > *Kopio:* mapserver-users@lists.osgeo.org > *Aihe:* Re: [MapServer-users] Changing opacity of one layer in a > multilayer WMS request URL with MapServer 8 > > > > Hi Matthew: That functionality was indeed removed as part of 8.0. I kind > of expect that we might build back something for more limited use cases if > necessary, Does SLD support opacity/transparency? > > > > --Steve > > > > > > On Tue, Feb 21, 2023 at 10:52 AM Matthew Graber > wrote: > > Hello all! > > > > With MapServer 8, I'm wondering what the proper way to change the opacity > of a single layer in a multilayer WMS request is. > > > > For example, the following URL excerpt shows the old way of accomplishing > this that worked while using MapServer 7: > > > > > SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&LAYERS=test_a,test_b&map.layer[test_b]=OPACITY+50 > > > > My understanding is that this would have the same effect as adding > "OPACITY 50" to the test_b layer in the mapfile. The mapfile containing > test_a and test_b did not have OPACITY included when they were written. > Obviously, this no longer works with MapServer 8, and now that request has > the same results as omitting the "&map.layer[test_b]=OPACITY+50" portion > entirely. > > > > Based on https://mapserver.org/fr/MIGRATION_GUIDE.html > <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmapserver.org%2Ffr%2FMIGRATION_GUIDE.html&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C751680624f65498df41008db14ec07c5%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C1%7C638126776216198394%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CbH%2FfRUQdX9uq8H9QRy48X0BggqTStn4IDQR4jD%2B97Y%3D&reserved=0>, > I see that OPACITY now belongs inside a COMPOSITE object. I tried the > following modifications to the request URL: > > > > > SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&LAYERS=test_a,test_b&map.layer[test_b].composite=OPACITY+50 > > > SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&LAYERS=test_a,test_b&map.layer[test_b].composite[0]=OPACITY+50 > > > > I also tried this both with the mapfile left as-is, and with an empty > COMPOSITE object being included in each layer object, and each time I got > the same unsuccessful result as before. > > > > Is there a way to accomplish this in MapServer 8? > > > > Also, does this map.layer[layername]=... syntax still work in MapServer 8? > The documentation at > https://mapserver.org/cgi/controls.html#using-mapserver-version-8 > <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmapserver.org%2Fcgi%2Fcontrols.html%23using-mapserver-version-8&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C751680624f65498df41008db14ec07c5%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C1%7C638126776216198394%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nu1%2BBml9%2BZG%2BIvjJ8ZpRZ28kDVKzo960hSM%2F18s3Jng%3D&reserved=0> > suggests it might no longer work for MapServer 8 (based on the heading > "Using MapServer version < 8"), but the documentation at > https://mapserver.org/pdf/MapServer.pdf > <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmapserver.org%2Fpdf%2FMapServer.pdf&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C751680624f65498df41008db14ec07c5%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C1%7C638126776216198394%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7
[MapServer-users] Changing opacity of one layer in a multilayer WMS request URL with MapServer 8
Hello all! With MapServer 8, I'm wondering what the proper way to change the opacity of a single layer in a multilayer WMS request is. For example, the following URL excerpt shows the old way of accomplishing this that worked while using MapServer 7: SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&LAYERS=test_a,test_b&map.layer[test_b]=OPACITY+50 My understanding is that this would have the same effect as adding "OPACITY 50" to the test_b layer in the mapfile. The mapfile containing test_a and test_b did not have OPACITY included when they were written. Obviously, this no longer works with MapServer 8, and now that request has the same results as omitting the "&map.layer[test_b]=OPACITY+50" portion entirely. Based on https://mapserver.org/fr/MIGRATION_GUIDE.html, I see that OPACITY now belongs inside a COMPOSITE object. I tried the following modifications to the request URL: SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&LAYERS=test_a,test_b&map.layer[test_b].composite=OPACITY+50 SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&LAYERS=test_a,test_b&map.layer[test_b].composite[0]=OPACITY+50 I also tried this both with the mapfile left as-is, and with an empty COMPOSITE object being included in each layer object, and each time I got the same unsuccessful result as before. Is there a way to accomplish this in MapServer 8? Also, does this map.layer[layername]=... syntax still work in MapServer 8? The documentation at https://mapserver.org/cgi/controls.html#using-mapserver-version-8 suggests it might no longer work for MapServer 8 (based on the heading "Using MapServer version < 8"), but the documentation at https://mapserver.org/pdf/MapServer.pdf shows the same examples with the heading "Using MapServer version <= 8" on page 1065 of the PDF, so I'm unsure if this is still intended to work in MapServer 8. Thank you, Matt ___ MapServer-users mailing list MapServer-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
