Re: [Geoserver-devel] New community module: a faster png output format
Great work Andrea +1 Notice: This email and any attachments are confidential. If received in error please destroy and immediately notify us. Do not copy or disclose the contents. -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk ___ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Re: [Geoserver-devel] New community module: a faster png output format
Very exciting stuff Andrea. +1 as well and very excited to see where it goes from there. On Mon, Aug 12, 2013 at 9:34 AM, Andrea Aime wrote: > On Mon, Aug 12, 2013 at 5:30 PM, Chris Holmes wrote: > >> This sounds awesome Andrea. Psyched to see what improvements it leads to >> when plugged in to the whole wms output pipeline. >> > > Eh well, that will depend a lot on the use case. > If you are trying to paint 3 million road segments into a 256x256 image it > won't make any difference, > if you are building a tile cache, considering that the last level of the > tile cache has often tiles with rather > sparse information (no much time actually spent loading and painting data) > it should make a visible difference instead. > > It's all a matter of what percentage of time is spent encoding the image, > compared to the overall processing time, it > has to be significant to see differences between the various encoders even > in the end to end GetMap request time. > > Cheers > Andrea > > -- > == > Our support, Your Success! Visit http://opensdi.geo-solutions.it for more > information. > == > > Ing. Andrea Aime > @geowolf > Technical Lead > > GeoSolutions S.A.S. > Via Poggio alle Viti 1187 > 55054 Massarosa (LU) > Italy > phone: +39 0584 962313 > fax: +39 0584 1660272 > mob: +39 339 8844549 > > http://www.geo-solutions.it > http://twitter.com/geosolutions_it > > --- > > > -- > Get 100% visibility into Java/.NET code with AppDynamics Lite! > It's a free troubleshooting tool designed for production. > Get down to code-level detail for bottlenecks, with <2% overhead. > Download for free and get started troubleshooting in minutes. > http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk > ___ > Geoserver-devel mailing list > Geoserver-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geoserver-devel > > -- Justin Deoliveira OpenGeo - http://opengeo.org Enterprise support for open source geospatial. -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk___ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Re: [Geoserver-devel] New community module: a faster png output format
On Mon, Aug 12, 2013 at 5:30 PM, Chris Holmes wrote: > This sounds awesome Andrea. Psyched to see what improvements it leads to > when plugged in to the whole wms output pipeline. > Eh well, that will depend a lot on the use case. If you are trying to paint 3 million road segments into a 256x256 image it won't make any difference, if you are building a tile cache, considering that the last level of the tile cache has often tiles with rather sparse information (no much time actually spent loading and painting data) it should make a visible difference instead. It's all a matter of what percentage of time is spent encoding the image, compared to the overall processing time, it has to be significant to see differences between the various encoders even in the end to end GetMap request time. Cheers Andrea -- == Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via Poggio alle Viti 1187 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk___ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Re: [Geoserver-devel] New community module: a faster png output format
This sounds awesome Andrea. Psyched to see what improvements it leads to when plugged in to the whole wms output pipeline. Doing the PNGJ one sounds like the better route especially if the author will solve the raster encoding stuff. Definitely +1 here for a community module. On Sun, Aug 11, 2013 at 2:26 PM, Andrea Aime wrote: > Hi, > during the last couple of months I've spent some spare time investigating > the PNG > format and the encoders generating it. > During the investigation some facts popped up: > * the JDK own encoder performance is horrible > * the CLib one (provided by ImageIO native libs) does a lot better, but > still leaves something > to be desired and generally speaking builds larger PNGs > > The JDK one is so slow because it rigged to provided the maximum possible > compression, > and it spends a lot of effort getting it, without giving any possibility > to configure it. > The thing is, that much effort normally just results in a size that is > less than 10% smaller than > more relaxed approaches, especially with the vector maps that GeoServer is > normally producing. > > Given the above, and also considering that often people just do not > install the ImageIO native > extensions, and on some platforms (Win64, OSX) they are simply not > available, I've setup > to build a faster pure java encoder, concentrating most of my work to make > sure we do the minimum > amount of pixel scans and array copies before writing out the PNG file > (something that apparently > all existing encoders are rather bad at). > > The results are good, today I have two solutions which are both faster > than the JDK one > (by a lot) and normally faster than the CLib native encoder too. > One is based on a minimal encoder that I wrote (called just NG for the > moment), > which is really bare bones, the other one is based on PNGJ, a open > source, low level, > pure java PNG encoder that is fully featured and that I've discovered a > when the basics > for the NG encoder were already laid down. > I've exchanged quite a few mails with the PNGJ author and while at the > beginning it was visibly > slower, after some changes it came to be almost on par. > > The buffered-image.html file attached contains the results of a pure image > encoding > benchmark (the image is ready at the beginning of the banchmark) which > I've built using Carrot Search Lab's > JUnit-Benchmarks (http://labs.carrotsearch.com/junit-benchmarks.html). > The benchmark encodes images using different types of colors and pixels > representations, > ranging from RGB(A) in either bytes or packed ints, grayscale images, and > paletted ones. > > As you can see, either the NG or PNGJ can run circles around the JDK own > writer (between > 5 and 10 times times faster and they still have some advantage over the > CLib one (30-40% faster) > while building images that are, size wise, in between the CLib encoder and > the JDK one. > > I've also made a benchmark based on sample vector (coming from > osm-in-a-box) and raster maps > of different sizes, attached as samples-benchmark.html, in this one I've > spared the JDK encoder > the embarassement and just compared clib, ng and pngj. > As you can see the story is more or less the same, although on raster > images pngj still needs some > work to match my own bare bones encoder. > > All the code wrote so far is available here under the GPL license: > https://github.com/aaime/png-experiments > > The next step would be to turn this into a WMS output format, directly, > without the overhead of going > trough the JAI ImageIO ImageWriter interface, so that's where the > community module comes into > the picture. > > Generally speaking, if at all possible, I'd like to drop my own encoder in > favour of PNGJ to lower > the maintenance overhead and better feature set, hopefully talking a bit > more with the author will solve also the > small raster encoding extra cost. > > Opinions and feedback welcomed (and of course, votes for the new community > module) > > Cheers > Andrea > > -- > == > Our support, Your Success! Visit http://opensdi.geo-solutions.it for more > information. > == > > Ing. Andrea Aime > @geowolf > Technical Lead > > GeoSolutions S.A.S. > Via Poggio alle Viti 1187 > 55054 Massarosa (LU) > Italy > phone: +39 0584 962313 > fax: +39 0584 1660272 > mob: +39 339 8844549 > > http://www.geo-solutions.it > http://twitter.com/geosolutions_it > > --- > > > -- > Get 100% visibility into Java/.NET code with AppDynamics Lite! > It's a free troubleshooting tool designed for production. > Get down to code-level detail for bottlenecks, with <2% overhead. > Download for free and get started troubleshooting in minutes. > http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk > ___ > Geoserver-devel mailing list > Geoserver-deve
Re: [Geoserver-devel] New community module: a faster png output format
Ah hem, it seems that to view the charts you also need to place the json files attached to this mail in the same folder as the html ones Cheers Andrea -- == Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via Poggio alle Viti 1187 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- bufferedimage-benchmark.jsonp Description: Binary data samples-benchmark.jsonp Description: Binary data -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk___ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel
[Geoserver-devel] New community module: a faster png output format
Hi, during the last couple of months I've spent some spare time investigating the PNG format and the encoders generating it. During the investigation some facts popped up: * the JDK own encoder performance is horrible * the CLib one (provided by ImageIO native libs) does a lot better, but still leaves something to be desired and generally speaking builds larger PNGs The JDK one is so slow because it rigged to provided the maximum possible compression, and it spends a lot of effort getting it, without giving any possibility to configure it. The thing is, that much effort normally just results in a size that is less than 10% smaller than more relaxed approaches, especially with the vector maps that GeoServer is normally producing. Given the above, and also considering that often people just do not install the ImageIO native extensions, and on some platforms (Win64, OSX) they are simply not available, I've setup to build a faster pure java encoder, concentrating most of my work to make sure we do the minimum amount of pixel scans and array copies before writing out the PNG file (something that apparently all existing encoders are rather bad at). The results are good, today I have two solutions which are both faster than the JDK one (by a lot) and normally faster than the CLib native encoder too. One is based on a minimal encoder that I wrote (called just NG for the moment), which is really bare bones, the other one is based on PNGJ, a open source, low level, pure java PNG encoder that is fully featured and that I've discovered a when the basics for the NG encoder were already laid down. I've exchanged quite a few mails with the PNGJ author and while at the beginning it was visibly slower, after some changes it came to be almost on par. The buffered-image.html file attached contains the results of a pure image encoding benchmark (the image is ready at the beginning of the banchmark) which I've built using Carrot Search Lab's JUnit-Benchmarks (http://labs.carrotsearch.com/junit-benchmarks.html). The benchmark encodes images using different types of colors and pixels representations, ranging from RGB(A) in either bytes or packed ints, grayscale images, and paletted ones. As you can see, either the NG or PNGJ can run circles around the JDK own writer (between 5 and 10 times times faster and they still have some advantage over the CLib one (30-40% faster) while building images that are, size wise, in between the CLib encoder and the JDK one. I've also made a benchmark based on sample vector (coming from osm-in-a-box) and raster maps of different sizes, attached as samples-benchmark.html, in this one I've spared the JDK encoder the embarassement and just compared clib, ng and pngj. As you can see the story is more or less the same, although on raster images pngj still needs some work to match my own bare bones encoder. All the code wrote so far is available here under the GPL license: https://github.com/aaime/png-experiments The next step would be to turn this into a WMS output format, directly, without the overhead of going trough the JAI ImageIO ImageWriter interface, so that's where the community module comes into the picture. Generally speaking, if at all possible, I'd like to drop my own encoder in favour of PNGJ to lower the maintenance overhead and better feature set, hopefully talking a bit more with the author will solve also the small raster encoding extra cost. Opinions and feedback welcomed (and of course, votes for the new community module) Cheers Andrea -- == Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via Poggio alle Viti 1187 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- Title: Benchmark results for methods in class org.geoserver.png.BufferedImageEncodingBenchmark Benchmark results for methods in class org.geoserver.png.BufferedImageEncodingBenchmark Orientation Click on table headers to change the sorting order and redraw the chart. Properties Run ID: 22 Run timestamp: 2013-08-11 20:03:43.764 JVM: 1.6.0_37-b06 OS: amd64 Title: Benchmark results for methods in class org.geoserver.png.SamplesEncodingBenchmark Benchmark results for methods in class org.geoserver.png.SamplesEncodingBenchmark Orientation Click on table headers to change the sorting order and redraw the chart. Properties Run ID: 57 Run timestamp: 2013-08-11 16:52:19.913 JVM: 1.6.0_37-b06 OS: amd64 -- Get 100% visibility into Java/.NET code
