Hi Christian,
i have tested it, but i am not able to insert this one big geotiff (300 GB) in
my database because of out of memory error.
Therefore I need support using image mosaic jdbc without importing the raster
file only using the path to the image file(s) storing in database.
How to do? Because Paul Ramsey said that I will be an advantage storing the
image files not in database only using image path.
Here are my geoserver configurations:
(0) Dataset: source was 165 GB sum with 592 different geotiff á 10.000 x 10.000
pixels.
(1) using gdal_retile with different configurations: everytime COMPRESS=JPEG
without shape creation and I create 8 different pyramid layers (0 original data
up to 8 overview image).
Processing gdal_retile with 4 different parameters: A) 500x500 pixel, B)
1000x1000 pixels, C) 2000x2000 pixels, D) 5000x5000 pixels
(2) After that I import the 4 different datasets A-D in postgresql
9.3.5/postgis (64 Bit) using raster2pgsql
(3) I have 2 different geoservers in use:
A) Windows Server 2008 32 Bit 4 GB RAM, JAVA 7 32 Bit, GeoServer 2.8 with
native JAI/ImageIO and 1 GB java heap space
B) Windows Server 2012 64 BIT 32 GB RAM, JAVA 7 64 Bit, GeoServer 2.8 without
JAI/ImageIO and 16 GB java heap space
(4) Using in both geoserver image mosaic jdbc and the WMS has no cache for
testing
I have defined 40 different requests for the WMS with different scale factors
(1 x 1:250.000, 4 x 1:100.000, and 5 x for each 1:50k, 1:25k, 1:10k, 1:5k,
1:2.5k, 1:1k, 1:0.5k).
The image data type is JPEG. Width/Height is 1.534/776 pixels for request.
Every of the 40 requests a had processed with the 32 and 64 bit geoserver and
for each image size (500x500, 1000x1000, 2000x2000, 5000x5000)
Number of Images in database of level 0 (highest resolution):
500x500: 224.944
1000x1000: 56.420
2000x2000: 14.185
5000x5000: 2.309
Here are a sum of my results of yesterday:
Note that I have 40 measures for the mean calculations.
32 Bit
64 Bit
Size
Mean
Min
Max
Mean
Min
Max
500x500
744 ms
328 ms
1797 ms
598 ms
375 ms
1188 ms
1000x1000
907 ms
344 ms
1469 ms
779 ms
437 ms
1125 ms
2000x2000
1501 ms
656 ms
2734 ms
1354 ms
485 ms
2765 ms
5000x5000
5053 ms
469 ms
12919 ms
4118 ms
453 ms
13185 ms
Now I wait for some tipps using image paths for image mosaic jdbc without
raster import in database.
Then I will analyse the performance advantages or disadvantages and will
publish my full results after I am finished.
Today it is very interesting for me that 225.000 images in database using
500x500 pixel size are faster then 1000x1000 pixel size with only 56.000 images.
And you can see that a small configurated geoserver in 32 bit using native
JAI/ImageIO is only a bit slower then the 64 bit example I had tested.
Now I will find out whether the mean of 598 ms (64 Bit, 500x500 pixel size)
could be faster if only store the meta data of the images and not the image in
database.
Then will look to the libjpeg turbo plugin because this could be accelerate the
geoserver requests too.
Best regards,
André
Von: Christian Mueller [mailto:christian.muel...@os-solutions.at]
Gesendet: Montag, 4. Mai 2015 10:59
An: Mende, Andre
Cc: geoserver-users@lists.sourceforge.net
Betreff: Re: [Geoserver-users] WG: ImageMosaicJDBC problem: geoserver stop
waiting before postgres send the raster information
Hi Andre
Some hints:
To create one big image:
gdal_merge.py -co COMPRESS=LZW -co BIGTIFF=YES -co TILED=YES -pct -o
mytiff.tiff --optfile listofimages
where mytiff.tiff is the result and listofimages is a text file containing the
name of all your 800 images. (each file name in a separate line).
Then you can add overviews with
http://www.gdal.org/gdaladdo.html
gdaladdo mytiff.tiff 2 4 8 16 32 64
would add 6 overviews to mytiff.
Cheers
Christian
On Mon, May 4, 2015 at 10:25 AM, Mende, Andre
<andre.me...@landkreis-zwickau.de<mailto:andre.me...@landkreis-zwickau.de>>
wrote:
Hi Christian,
ah ok. My original data is GeoTIFF 800 images á 10.000 x 10.000 pixels RGB with
300 MB of each image. If I try to create one big GeoTIFF using gdal_retile I
got a memory exception… ;)
if you have some linux scripts I like to look inside the scripts and hope that
I can find some code snips which helps me to configure my geoserver in the
right point of view.
Thanks.
André
Von: Christian Mueller
[mailto:christian.muel...@os-solutions.at<mailto:christian.muel...@os-solutions.at>]
Gesendet: Donnerstag, 30. April 2015 17:40
An: Mende, Andre
Cc:
geoserver-users@lists.sourceforge.net<mailto:geoserver-users@lists.sourceforge.net>
Betreff: Re: [Geoserver-users] WG: ImageMosaicJDBC problem: geoserver stop
waiting before postgres send the raster information
Hi Andre
I meant one big GeoTIFF (no use of gdal_retile) with internal compression,
tiles , overviews and the CRS. This setup works pretty well for my customers.
At the end of the day, you have only one geotiff file and GeoServer works very
good with a geotiff file (my experience).
Maybe I can send you some Linux scripts I am using.
What format is your original data ?
Cheers
Christian
On Wed, Apr 29, 2015 at 9:49 AM, Mende, Andre
<andre.me...@landkreis-zwickau.de<mailto:andre.me...@landkreis-zwickau.de>>
wrote:
Hi Christian again,
Your question was: Did you consider to create one big geotiff file with inner
tiling and overviews ?
I addition to my last post I will ask you something about big GeoTIFF with
inner tilling and overviews.
Do you mean that I have to create for every tile of my gdal_retile result a
inner tilling and overview or should I make one big GeoTIFF (165 GB, maybe 20
GB if I reduce with JPEG-compression) without using gdal_retile?
Thanks.
André
Hi Christian and Andrea,
thank you both for your response.
I had used gdal_retile with different pixelsizes 1000x1000, 2000x2000,
5000x5000.
I am interested in the best configuration as a balance between number of rows
in database for each pyramidal step and the size of the real image.
@ Andrea:
Do you have this paper of Paul Ramsey or a link with the full results?
@ Christian:
Is it faster if I use the raster-column in a function like st_envelope to get
the bounds or maybe we should use a extra column with the geometry bounds
because a raster is a fixed information without dynamic change of the borders?
In my opinion I think it is faster to ask the database whether a point is
inside a bounding box (geometry column) instead of using a full raster and
calculate the borders every time…
Another point: Is there a opportunity only use Image Links in database using
ImageMosaicJDBC? (please look at the answer of Andrea last mail)
Now I compress my source images using gdal_retile including compression
attribute because i will find the right balance between resolution and image
size.
If the process is ready I will tell you some result information. In addition I
will test a 32 bit server with JAI and a 64 bit server without JAI which is
faster or what´s the differences.
Thanks.
André
Von: Christian Mueller [mailto:christian.muel...@os-solutions.at]
Gesendet: Dienstag, 28. April 2015 11:43
An: Andrea Aime
Cc: Mende, Andre;
geoserver-users@lists.sourceforge.net<mailto:geoserver-users@lists.sourceforge.net>
Betreff: Re: [Geoserver-users] ImageMosaicJDBC problem: geoserver stop waiting
before postgres send the raster information
Hi Andre
Did you consider to create one big geotiff file with inner tiling and
overviews ?
I think this is the simplest way to support image data.
And here is the Swiss army knife you may need
http://www.gdal.org/
Cheers
Christian
On Mon, Apr 27, 2015 at 4:45 PM, Andrea Aime
<andrea.a...@geo-solutions.it<mailto:andrea.a...@geo-solutions.it>> wrote:
On Mon, Apr 27, 2015 at 4:24 PM, Mende, Andre
<andre.me...@landkreis-zwickau.de<mailto:andre.me...@landkreis-zwickau.de>>
wrote:
Hi Christian,
yes there are write priveleges… but i will look at this again and tell you the
answer.
Now some additional ideas/questions would like to discuss.
A GeoServer-user will use this software with high speed and a useful
configuration.
Therefore it is very important to know whether a 32 bit geoserver using the jai
is faster then a 64 bit geoserver without using jai.
The difference in performance between native and non native jai is probably not
very relevant given you setup a poorly performing environment to start with:
* GeoServer on Windows server is a good 30% slower than on Linux, on the same
hardware (and other GIS software is too, not specific to GeoServer)
* Keeping raster in PostGIS is slower than keeping it on the file system (Paul
Ramsey reiterated that at FOSS4G-NA a few months ago, "keep rasters
in the database only to perform spatial analys, not to serve them on the web)
* Postgis raster does not have native overviews (they are going to be added in
the next version), although the imagemosaicjdbc can probably help here, since
it can work as a image pyramid plugin too
Cheers
Andrea
--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.
==
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313<tel:%2B39%200584%20962313>
fax: +39 0584 1660272<tel:%2B39%200584%201660272>
mob: +39 339 8844549<tel:%2B39%20%C2%A0339%208844549>
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i
file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo
è consentito esclusivamente al destinatario del messaggio, per le finalità
indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne
il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di
procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro
sistema. Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the
attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act (Legislative
Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in
accord with its purpose, any disclosure, reproduction, copying, distribution,
or either dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the intended
recipient, please contact immediately the sender by telephone, fax or e-mail
and delete the information in this message that has been received in error. The
sender does not give any warranty or accept liability as the content, accuracy
or completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.
-------------------------------------------------------
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net<mailto:Geoserver-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geoserver-users
--
DI Christian Mueller MSc (GIS), MSc (IT-Security)
OSS Open Source Solutions GmbH
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net<mailto:Geoserver-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geoserver-users
--
DI Christian Mueller MSc (GIS), MSc (IT-Security)
OSS Open Source Solutions GmbH
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net<mailto:Geoserver-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geoserver-users
--
DI Christian Mueller MSc (GIS), MSc (IT-Security)
OSS Open Source Solutions GmbH
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
