Thank you for your recommendations! I will work on them.

Joaquín Rodríguez-Guerra Urcelay

Unidad de Sistemas de Procesamiento de Ciencia y de Observación de la Tierra / 
Science and Earth Observing Processing Systems Unit

GMV AEROSPACE AND DEFENCE, S.A.
Isaac Newton, 11
P.T.M. Tres Cantos
E-28760 Madrid
Tel. +34 91 807 21 00 / 7694
Fax +34 91 807 21 99
www.gmv.com
________________________________________
De: Justin Deoliveira [[email protected]]
Enviado el: jueves, 09 de septiembre de 2010 19:29
Para: Joaquín Rodriguez-Guerra Urcelay
CC: [email protected]
Asunto: Re: [Geoserver-users] Reduce the loading time of new data into a heavy 
Geoserver

Andrea and myself just thought of something else. You can save a lot of objects 
by only having a single directory datastore for your shapefiles. To do so you 
would have to use the ftp module in geoserver to first upload the shapefile 
into an appropriate directory. And then use the REST api to configure a new 
feature type. It would be a bit more complicated than your current workflow but 
would scale better since you would have a lot less datastores.

There is also a parameter called "feature type cache size" in geoserver 2.0.x 
that you are going to want to set high if you are using that many feature types.

-Justin

2010/9/9 Joaquín Rodriguez-Guerra Urcelay 
<[email protected]<mailto:[email protected]>>
Thanks a lot Justin for your quick answer!!! :)

Joaquín Rodríguez-Guerra Urcelay

________________________________________
De: Justin Deoliveira [[email protected]<mailto:[email protected]>]
Enviado el: jueves, 09 de septiembre de 2010 19:08
Para: Joaquín Rodriguez-Guerra Urcelay
CC: 
[email protected]<mailto:[email protected]>
Asunto: Re: [Geoserver-users] Reduce the loading time of new data into a heavy 
Geoserver

Hi Joaquín,

You are correct, this is not a REST issue, it is a core GeoServer catalog 
issue. The geoserver catalog is stored in memory and just not designed to scale 
to that number of layers. Especially not the 1.x version of GeoServer which it 
seems you are using.

I think moving to geoserver 2.x would help the problem but it will definitely 
not fix it. In GeoServer 1.7.x when something is saved via rest, *everything* 
is saved which why you are seing things so slow. In geoServer 2.x when 
something new is added to the catalog, only it is saved. So you should 
hopefully not see the same penalty.

That said, you are still going to pay a price querying a catalog that big in 
geoserver 2.0.x. I am not sure there will be much you can do about it besides 
try to reduce the number of layers being served by a single geoserver instance. 
Perhaps you can run multiple geoserver instances behind some sort of proxy that 
can redirect the traffic depending on the layer accessed.

All that said there is likely some work that will start soon to be able to 
store the geoserver catalog in a  database. Which will allow geoserver to 
manage a catalog of this size. But that work is still experimental at this 
point and probably won't be ready to try out for some months.

-Justin

2010/9/9 Joaquín Rodriguez-Guerra Urcelay 
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>
Dear all,

I am really worried about the time that is taking to load new data into 
geoserver and maybe you guys can help me out with some parameters or 
configurations.

I have been inserting a TIF file and 2 SHAPEFILES every 15 minutes in my 
geoserver during the last month automatically using REST. At first the 
operation was taking maybe 10 seconds, but right now at this point it is taking 
more thatn 2 minutes to insert each file. I think this is not a REST issue, 
since inserting a TIF file manually using the user interface following these 
steps is taking (after almost 10000 files were inserted in geoserver) 2 minutes:

- Go to http://localhost:8082/geoserver/config/data/format.do (Coverage Data 
Set Configuration )
- Click on New and add a new tif file ( this step does not take more than 5 
seconds)
- Click on apply (this takes almost a minute)
- Click on save ( this takes almost a minute)

I really dont know know what to do, maybe removing some of the oldest 
shapefiles would lower the loading time?  but my boss would kill me. Do you 
know any configuration or parameters, or maybe some script I might run to 
cleanup things? I am right now running geoserver from tomcat with the following 
parameters

CATALINA_OPTS="-Duser.timezone=Europe/Madrid -Xms512m -Xmx1024m 
-XX:PermSize=256m -XX:MaxPermSize=256m"

The machine has 8GB of RAM and 4 processors , would it make any difference to 
increase the value of PermSize parameters?

I'd really appreciate any idea you share!! Thanks you for your time!

Joaquín Rodríguez-Guerra Urcelay


______________________
This message including any attachments may contain confidential
information, according to our Information Security Management System,
 and intended solely for a specific individual to whom they are addressed.
 Any unauthorised copy, disclosure or distribution of this message
 is strictly forbidden. If you have received this transmission in error,
 please notify the sender immediately and delete it.

______________________
Este mensaje, y en su caso, cualquier fichero anexo al mismo,
 puede contener informacion clasificada por su emisor como confidencial
 en el marco de su Sistema de Gestion de Seguridad de la
Informacion siendo para uso exclusivo del destinatario, quedando
prohibida su divulgacion copia o distribucion a terceros sin la
autorizacion expresa del remitente. Si Vd. ha recibido este mensaje
 erroneamente, se ruega lo notifique al remitente y proceda a su borrado.
Gracias por su colaboracion.

______________________


------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
Geoserver-users mailing list
[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
https://lists.sourceforge.net/lists/listinfo/geoserver-users



--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.


______________________
This message including any attachments may contain confidential
information, according to our Information Security Management System,
 and intended solely for a specific individual to whom they are addressed.
 Any unauthorised copy, disclosure or distribution of this message
 is strictly forbidden. If you have received this transmission in error,
 please notify the sender immediately and delete it.

______________________
Este mensaje, y en su caso, cualquier fichero anexo al mismo,
 puede contener informacion clasificada por su emisor como confidencial
 en el marco de su Sistema de Gestion de Seguridad de la
Informacion siendo para uso exclusivo del destinatario, quedando
prohibida su divulgacion copia o distribucion a terceros sin la
autorizacion expresa del remitente. Si Vd. ha recibido este mensaje
 erroneamente, se ruega lo notifique al remitente y proceda a su borrado.
Gracias por su colaboracion.

______________________


------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
Geoserver-users mailing list
[email protected]<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/geoserver-users



--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.


______________________
This message including any attachments may contain confidential 
information, according to our Information Security Management System,
 and intended solely for a specific individual to whom they are addressed.
 Any unauthorised copy, disclosure or distribution of this message
 is strictly forbidden. If you have received this transmission in error,
 please notify the sender immediately and delete it.

______________________
Este mensaje, y en su caso, cualquier fichero anexo al mismo,
 puede contener informacion clasificada por su emisor como confidencial
 en el marco de su Sistema de Gestion de Seguridad de la 
Informacion siendo para uso exclusivo del destinatario, quedando 
prohibida su divulgacion copia o distribucion a terceros sin la 
autorizacion expresa del remitente. Si Vd. ha recibido este mensaje 
 erroneamente, se ruega lo notifique al remitente y proceda a su borrado. 
Gracias por su colaboracion.

______________________


------------------------------------------------------------------------------
Automate Storage Tiering Simply
Optimize IT performance and efficiency through flexible, powerful, 
automated storage tiering capabilities. View this brief to learn how
you can reduce costs and improve performance. 
http://p.sf.net/sfu/dell-sfdev2dev
_______________________________________________
Geoserver-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Reply via email to