Thank you for your recommendations! I will work on them. Joaquín Rodríguez-Guerra Urcelay
Unidad de Sistemas de Procesamiento de Ciencia y de Observación de la Tierra / Science and Earth Observing Processing Systems Unit GMV AEROSPACE AND DEFENCE, S.A. Isaac Newton, 11 P.T.M. Tres Cantos E-28760 Madrid Tel. +34 91 807 21 00 / 7694 Fax +34 91 807 21 99 www.gmv.com ________________________________________ De: Justin Deoliveira [[email protected]] Enviado el: jueves, 09 de septiembre de 2010 19:29 Para: Joaquín Rodriguez-Guerra Urcelay CC: [email protected] Asunto: Re: [Geoserver-users] Reduce the loading time of new data into a heavy Geoserver Andrea and myself just thought of something else. You can save a lot of objects by only having a single directory datastore for your shapefiles. To do so you would have to use the ftp module in geoserver to first upload the shapefile into an appropriate directory. And then use the REST api to configure a new feature type. It would be a bit more complicated than your current workflow but would scale better since you would have a lot less datastores. There is also a parameter called "feature type cache size" in geoserver 2.0.x that you are going to want to set high if you are using that many feature types. -Justin 2010/9/9 Joaquín Rodriguez-Guerra Urcelay <[email protected]<mailto:[email protected]>> Thanks a lot Justin for your quick answer!!! :) Joaquín Rodríguez-Guerra Urcelay ________________________________________ De: Justin Deoliveira [[email protected]<mailto:[email protected]>] Enviado el: jueves, 09 de septiembre de 2010 19:08 Para: Joaquín Rodriguez-Guerra Urcelay CC: [email protected]<mailto:[email protected]> Asunto: Re: [Geoserver-users] Reduce the loading time of new data into a heavy Geoserver Hi Joaquín, You are correct, this is not a REST issue, it is a core GeoServer catalog issue. The geoserver catalog is stored in memory and just not designed to scale to that number of layers. Especially not the 1.x version of GeoServer which it seems you are using. I think moving to geoserver 2.x would help the problem but it will definitely not fix it. In GeoServer 1.7.x when something is saved via rest, *everything* is saved which why you are seing things so slow. In geoServer 2.x when something new is added to the catalog, only it is saved. So you should hopefully not see the same penalty. That said, you are still going to pay a price querying a catalog that big in geoserver 2.0.x. I am not sure there will be much you can do about it besides try to reduce the number of layers being served by a single geoserver instance. Perhaps you can run multiple geoserver instances behind some sort of proxy that can redirect the traffic depending on the layer accessed. All that said there is likely some work that will start soon to be able to store the geoserver catalog in a database. Which will allow geoserver to manage a catalog of this size. But that work is still experimental at this point and probably won't be ready to try out for some months. -Justin 2010/9/9 Joaquín Rodriguez-Guerra Urcelay <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Dear all, I am really worried about the time that is taking to load new data into geoserver and maybe you guys can help me out with some parameters or configurations. I have been inserting a TIF file and 2 SHAPEFILES every 15 minutes in my geoserver during the last month automatically using REST. At first the operation was taking maybe 10 seconds, but right now at this point it is taking more thatn 2 minutes to insert each file. I think this is not a REST issue, since inserting a TIF file manually using the user interface following these steps is taking (after almost 10000 files were inserted in geoserver) 2 minutes: - Go to http://localhost:8082/geoserver/config/data/format.do (Coverage Data Set Configuration ) - Click on New and add a new tif file ( this step does not take more than 5 seconds) - Click on apply (this takes almost a minute) - Click on save ( this takes almost a minute) I really dont know know what to do, maybe removing some of the oldest shapefiles would lower the loading time? but my boss would kill me. Do you know any configuration or parameters, or maybe some script I might run to cleanup things? I am right now running geoserver from tomcat with the following parameters CATALINA_OPTS="-Duser.timezone=Europe/Madrid -Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=256m" The machine has 8GB of RAM and 4 processors , would it make any difference to increase the value of PermSize parameters? I'd really appreciate any idea you share!! Thanks you for your time! Joaquín Rodríguez-Guerra Urcelay ______________________ This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. ______________________ Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacion clasificada por su emisor como confidencial en el marco de su Sistema de Gestion de Seguridad de la Informacion siendo para uso exclusivo del destinatario, quedando prohibida su divulgacion copia o distribucion a terceros sin la autorizacion expresa del remitente. Si Vd. ha recibido este mensaje erroneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracion. ______________________ ------------------------------------------------------------------------------ This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd _______________________________________________ Geoserver-users mailing list [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> https://lists.sourceforge.net/lists/listinfo/geoserver-users -- Justin Deoliveira OpenGeo - http://opengeo.org Enterprise support for open source geospatial. ______________________ This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. ______________________ Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacion clasificada por su emisor como confidencial en el marco de su Sistema de Gestion de Seguridad de la Informacion siendo para uso exclusivo del destinatario, quedando prohibida su divulgacion copia o distribucion a terceros sin la autorizacion expresa del remitente. Si Vd. ha recibido este mensaje erroneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracion. ______________________ ------------------------------------------------------------------------------ This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd _______________________________________________ Geoserver-users mailing list [email protected]<mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/geoserver-users -- Justin Deoliveira OpenGeo - http://opengeo.org Enterprise support for open source geospatial. ______________________ This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. ______________________ Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacion clasificada por su emisor como confidencial en el marco de su Sistema de Gestion de Seguridad de la Informacion siendo para uso exclusivo del destinatario, quedando prohibida su divulgacion copia o distribucion a terceros sin la autorizacion expresa del remitente. Si Vd. ha recibido este mensaje erroneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracion. ______________________ ------------------------------------------------------------------------------ Automate Storage Tiering Simply Optimize IT performance and efficiency through flexible, powerful, automated storage tiering capabilities. View this brief to learn how you can reduce costs and improve performance. http://p.sf.net/sfu/dell-sfdev2dev _______________________________________________ Geoserver-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geoserver-users
