Hi Andre
No need for imagemosaic jdbc If you serve your image data from one big
geotiff. Geoserver can handle geotiff files directly.
Cheers
Christian
On Fri, May 8, 2015 at 9:33 AM, Mende, Andre <
andre.me...@landkreis-zwickau.de> wrote:
> Hi Christian,
>
>
>
> i have tested it, but i am not able to insert this one big geotiff (300
> GB) in my database because of out of memory error.
>
> *Therefore I need support using image mosaic jdbc without importing the
> raster file only using the path to the image file(s) storing in database.*
>
> *How to do? Because **Paul Ramsey said that I will be an advantage
> storing the image files not in database only using image path.*
>
>
>
> Here are my geoserver configurations:
>
> (0) Dataset: source was 165 GB sum with 592 different geotiff á 10.000 x
> 10.000 pixels.
>
> (1) using gdal_retile with different configurations: everytime
> COMPRESS=JPEG without shape creation and I create 8 different pyramid
> layers (0 original data up to 8 overview image).
>
> Processing gdal_retile with 4 different parameters: A) 500x500 pixel, B)
> 1000x1000 pixels, C) 2000x2000 pixels, D) 5000x5000 pixels
>
> (2) After that I import the 4 different datasets A-D in postgresql
> 9.3.5/postgis (64 Bit) using raster2pgsql
>
> (3) I have 2 different geoservers in use:
>
> A) Windows Server 2008 32 Bit 4 GB RAM, JAVA 7 32 Bit, GeoServer 2.8 with
> native JAI/ImageIO and 1 GB java heap space
>
> B) Windows Server 2012 64 BIT 32 GB RAM, JAVA 7 64 Bit, GeoServer 2.8
> without JAI/ImageIO and 16 GB java heap space
>
> (4) Using in both geoserver image mosaic jdbc and the WMS has no cache for
> testing
>
>
>
> I have defined 40 different requests for the WMS with different scale
> factors (1 x 1:250.000, 4 x 1:100.000, and 5 x for each 1:50k, 1:25k,
> 1:10k, 1:5k, 1:2.5k, 1:1k, 1:0.5k).
>
> The image data type is JPEG. Width/Height is 1.534/776 pixels for request.
>
> Every of the 40 requests a had processed with the 32 and 64 bit geoserver
> and for each image size (500x500, 1000x1000, 2000x2000, 5000x5000)
>
>
>
> Number of Images in database of level 0 (highest resolution):
>
> 500x500: 224.944
>
> 1000x1000: 56.420
>
> 2000x2000: 14.185
>
> 5000x5000: 2.309
>
>
>
> Here are a sum of my results of yesterday:
>
> Note that I have 40 measures for the mean calculations.
>
>
>
> 32 Bit
>
> 64 Bit
>
> Size
>
> Mean
>
> Min
>
> Max
>
> Mean
>
> Min
>
> Max
>
> 500x500
>
> 744 ms
>
> 328 ms
>
> 1797 ms
>
> 598 ms
>
> 375 ms
>
> 1188 ms
>
> 1000x1000
>
> 907 ms
>
> 344 ms
>
> 1469 ms
>
> 779 ms
>
> 437 ms
>
> 1125 ms
>
> 2000x2000
>
> 1501 ms
>
> 656 ms
>
> 2734 ms
>
> 1354 ms
>
> 485 ms
>
> 2765 ms
>
> 5000x5000
>
> 5053 ms
>
> 469 ms
>
> 12919 ms
>
> 4118 ms
>
> 453 ms
>
> 13185 ms
>
>
>
> Now I wait for some tipps using image paths for image mosaic jdbc without
> raster import in database.
>
> Then I will analyse the performance advantages or disadvantages and will
> publish my full results after I am finished.
>
>
>
> Today it is very interesting for me that 225.000 images in database using
> 500x500 pixel size are faster then 1000x1000 pixel size with only 56.000
> images.
>
> And you can see that a small configurated geoserver in 32 bit using native
> JAI/ImageIO is only a bit slower then the 64 bit example I had tested.
>
>
>
> Now I will find out whether the mean of 598 ms (64 Bit, 500x500 pixel
> size) could be faster if only store the meta data of the images and not the
> image in database.
>
> Then will look to the libjpeg turbo plugin because this could be
> accelerate the geoserver requests too.
>
>
>
> Best regards,
>
> André
>
>
>
> *Von:* Christian Mueller [mailto:christian.muel...@os-solutions.at]
> *Gesendet:* Montag, 4. Mai 2015 10:59
>
> *An:* Mende, Andre
> *Cc:* geoserver-users@lists.sourceforge.net
> *Betreff:* Re: [Geoserver-users] WG: ImageMosaicJDBC problem: geoserver
> stop waiting before postgres send the raster information
>
>
>
> Hi Andre
>
>
>
> Some hints:
>
>
>
> To create one big image:
>
>
>
> gdal_merge.py -co COMPRESS=LZW -co BIGTIFF=YES -co TILED=YES -pct -o
> mytiff.tiff --optfile listofimages
>
>
>
> where mytiff.tiff is the result and listofimages is a text file containing
> the name of all your 800 images. (each file name in a separate line).
>
>
>
> Then you can add overviews with
>
>
>
> http://www.gdal.org/gdaladdo.html
>
>
>
> gdaladdo mytiff.tiff 2 4 8 16 32 64
>
>
>
> would add 6 overviews to mytiff.
>
>
>
> Cheers
>
> Christian
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Mon, May 4, 2015 at 10:25 AM, Mende, Andre <
> andre.me...@landkreis-zwickau.de> wrote:
>
> Hi Christian,
>
>
>
> ah ok. My original data is GeoTIFF 800 images á 10.000 x 10.000 pixels RGB
> with 300 MB of each image. If I try to create one big GeoTIFF using
> gdal_retile I got a memory exception… ;)
>
>
>
> if you have some linux scripts I like to look inside the scripts and hope
> that I can find some code snips which helps me to configure my geoserver in
> the right point of view.
>
>
>
> Thanks.
>
> André
>
>
>
> *Von:* Christian Mueller [mailto:christian.muel...@os-solutions.at]
> *Gesendet:* Donnerstag, 30. April 2015 17:40
> *An:* Mende, Andre
> *Cc:* geoserver-users@lists.sourceforge.net
> *Betreff:* Re: [Geoserver-users] WG: ImageMosaicJDBC problem: geoserver
> stop waiting before postgres send the raster information
>
>
>
> Hi Andre
>
>
>
> I meant one big GeoTIFF (no use of gdal_retile) with internal
> compression, tiles , overviews and the CRS. This setup works pretty well
> for my customers. At the end of the day, you have only one geotiff file and
> GeoServer works very good with a geotiff file (my experience).
>
>
>
> Maybe I can send you some Linux scripts I am using.
>
>
>
> What format is your original data ?
>
>
>
> Cheers
>
> Christian
>
>
>
> On Wed, Apr 29, 2015 at 9:49 AM, Mende, Andre <
> andre.me...@landkreis-zwickau.de> wrote:
>
> Hi Christian again,
>
>
>
> Your question was: Did you consider to create one big geotiff file with
> inner tiling and overviews ?
>
> I addition to my last post I will ask you something about big GeoTIFF with
> inner tilling and overviews.
>
> Do you mean that I have to create for every tile of my gdal_retile result
> a inner tilling and overview or should I make one big GeoTIFF (165 GB,
> maybe 20 GB if I reduce with JPEG-compression) without using gdal_retile?
>
>
>
> Thanks.
>
> André
>
>
>
> Hi Christian and Andrea,
>
>
>
> thank you both for your response.
>
>
>
> I had used gdal_retile with different pixelsizes 1000x1000, 2000x2000,
> 5000x5000.
>
> I am interested in the best configuration as a balance between number of
> rows in database for each pyramidal step and the size of the real image.
>
>
>
> @ Andrea:
>
> Do you have this paper of Paul Ramsey or a link with the full results?
>
>
>
> @ Christian:
>
> Is it faster if I use the raster-column in a function like st_envelope to
> get the bounds or maybe we should use a extra column with the geometry
> bounds because a raster is a fixed information without dynamic change of
> the borders?
>
> In my opinion I think it is faster to ask the database whether a point is
> inside a bounding box (geometry column) instead of using a full raster and
> calculate the borders every time…
>
> Another point: Is there a opportunity only use Image Links in database
> using ImageMosaicJDBC? (please look at the answer of Andrea last mail)
>
>
>
> Now I compress my source images using gdal_retile including compression
> attribute because i will find the right balance between resolution and
> image size.
>
> If the process is ready I will tell you some result information. In
> addition I will test a 32 bit server with JAI and a 64 bit server without
> JAI which is faster or what´s the differences.
>
>
>
> Thanks.
>
> André
>
>
>
>
>
> *Von:* Christian Mueller [mailto:christian.muel...@os-solutions.at
> <christian.muel...@os-solutions.at>]
> *Gesendet:* Dienstag, 28. April 2015 11:43
> *An:* Andrea Aime
> *Cc:* Mende, Andre; geoserver-users@lists.sourceforge.net
> *Betreff:* Re: [Geoserver-users] ImageMosaicJDBC problem: geoserver stop
> waiting before postgres send the raster information
>
>
>
> Hi Andre
>
>
>
> Did you consider to create one big geotiff file with inner tiling and
> overviews ?
>
>
>
> I think this is the simplest way to support image data.
>
>
>
> And here is the Swiss army knife you may need
>
> http://www.gdal.org/
>
>
>
>
>
> Cheers
>
> Christian
>
>
>
>
>
>
>
> On Mon, Apr 27, 2015 at 4:45 PM, Andrea Aime <andrea.a...@geo-solutions.it>
> wrote:
>
> On Mon, Apr 27, 2015 at 4:24 PM, Mende, Andre <
> andre.me...@landkreis-zwickau.de> wrote:
>
> Hi Christian,
>
>
>
> yes there are write priveleges… but i will look at this again and tell you
> the answer.
>
>
>
> Now some additional ideas/questions would like to discuss.
>
> A GeoServer-user will use this software with high speed and a useful
> configuration.
>
> Therefore it is very important to know whether a 32 bit geoserver using
> the jai is faster then a 64 bit geoserver without using jai.
>
>
>
> The difference in performance between native and non native jai is
> probably not very relevant given you setup a poorly performing environment
> to start with:
>
> * GeoServer on Windows server is a good 30% slower than on Linux, on the
> same hardware (and other GIS software is too, not specific to GeoServer)
>
> * Keeping raster in PostGIS is slower than keeping it on the file system
> (Paul Ramsey reiterated that at FOSS4G-NA a few months ago, "keep rasters
>
> in the database only to perform spatial analys, not to serve them on the
> web)
>
> * Postgis raster does not have native overviews (they are going to be
> added in the next version), although the imagemosaicjdbc can probably help
> here, since it can work as a image pyramid plugin too
>
>
>
> Cheers
>
> Andrea
>
>
>
>
>
> --
>
> ==
>
> GeoServer Professional Services from the experts! Visit
>
> http://goo.gl/NWWaa2 for more information.
>
> ==
>
>
>
> Ing. Andrea Aime
>
> @geowolf
>
> Technical Lead
>
>
>
> GeoSolutions S.A.S.
>
> Via Poggio alle Viti 1187
>
> 55054 Massarosa (LU)
>
> Italy
>
> phone: +39 0584 962313
>
> fax: +39 0584 1660272
>
> mob: +39 339 8844549
>
>
>
> http://www.geo-solutions.it
>
> http://twitter.com/geosolutions_it
>
>
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
>
>
> -------------------------------------------------------
>
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Geoserver-users mailing list
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>
>
>
>
> --
>
> DI Christian Mueller MSc (GIS), MSc (IT-Security)
>
> OSS Open Source Solutions GmbH
>
>
>
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Geoserver-users mailing list
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>
>
>
>
> --
>
> DI Christian Mueller MSc (GIS), MSc (IT-Security)
>
> OSS Open Source Solutions GmbH
>
>
>
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Geoserver-users mailing list
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>
>
>
>
> --
>
> DI Christian Mueller MSc (GIS), MSc (IT-Security)
>
> OSS Open Source Solutions GmbH
>
>
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Geoserver-users mailing list
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>
--
DI Christian Mueller MSc (GIS), MSc (IT-Security)
OSS Open Source Solutions GmbH
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
