Andrea and myself just thought of something else. You can save a lot of
objects by only having a single directory datastore for your shapefiles. To
do so you would have to use the ftp module in geoserver to first upload the
shapefile into an appropriate directory. And then use the REST api to
configure a new feature type. It would be a bit more complicated than your
current workflow but would scale better since you would have a lot less
datastores.

There is also a parameter called "feature type cache size" in geoserver
2.0.x that you are going to want to set high if you are using that many
feature types.

-Justin

2010/9/9 Joaquín Rodriguez-Guerra Urcelay <[email protected]>

> Thanks a lot Justin for your quick answer!!! :)
>
> Joaquín Rodríguez-Guerra Urcelay
>
> ________________________________________
> De: Justin Deoliveira [[email protected]]
> Enviado el: jueves, 09 de septiembre de 2010 19:08
> Para: Joaquín Rodriguez-Guerra Urcelay
> CC: [email protected]
> Asunto: Re: [Geoserver-users] Reduce the loading time of new data into a
> heavy Geoserver
>
> Hi Joaquín,
>
> You are correct, this is not a REST issue, it is a core GeoServer catalog
> issue. The geoserver catalog is stored in memory and just not designed to
> scale to that number of layers. Especially not the 1.x version of GeoServer
> which it seems you are using.
>
> I think moving to geoserver 2.x would help the problem but it will
> definitely not fix it. In GeoServer 1.7.x when something is saved via rest,
> *everything* is saved which why you are seing things so slow. In geoServer
> 2.x when something new is added to the catalog, only it is saved. So you
> should hopefully not see the same penalty.
>
> That said, you are still going to pay a price querying a catalog that big
> in geoserver 2.0.x. I am not sure there will be much you can do about it
> besides try to reduce the number of layers being served by a single
> geoserver instance. Perhaps you can run multiple geoserver instances behind
> some sort of proxy that can redirect the traffic depending on the layer
> accessed.
>
> All that said there is likely some work that will start soon to be able to
> store the geoserver catalog in a  database. Which will allow geoserver to
> manage a catalog of this size. But that work is still experimental at this
> point and probably won't be ready to try out for some months.
>
> -Justin
>
> 2010/9/9 Joaquín Rodriguez-Guerra Urcelay <[email protected]<mailto:
> [email protected]>>
> Dear all,
>
> I am really worried about the time that is taking to load new data into
> geoserver and maybe you guys can help me out with some parameters or
> configurations.
>
> I have been inserting a TIF file and 2 SHAPEFILES every 15 minutes in my
> geoserver during the last month automatically using REST. At first the
> operation was taking maybe 10 seconds, but right now at this point it is
> taking more thatn 2 minutes to insert each file. I think this is not a REST
> issue, since inserting a TIF file manually using the user interface
> following these steps is taking (after almost 10000 files were inserted in
> geoserver) 2 minutes:
>
> - Go to http://localhost:8082/geoserver/config/data/format.do (Coverage
> Data Set Configuration )
> - Click on New and add a new tif file ( this step does not take more than 5
> seconds)
> - Click on apply (this takes almost a minute)
> - Click on save ( this takes almost a minute)
>
> I really dont know know what to do, maybe removing some of the oldest
> shapefiles would lower the loading time?  but my boss would kill me. Do you
> know any configuration or parameters, or maybe some script I might run to
> cleanup things? I am right now running geoserver from tomcat with the
> following parameters
>
> CATALINA_OPTS="-Duser.timezone=Europe/Madrid -Xms512m -Xmx1024m
> -XX:PermSize=256m -XX:MaxPermSize=256m"
>
> The machine has 8GB of RAM and 4 processors , would it make any difference
> to increase the value of PermSize parameters?
>
> I'd really appreciate any idea you share!! Thanks you for your time!
>
> Joaquín Rodríguez-Guerra Urcelay
>
>
> ______________________
> This message including any attachments may contain confidential
> information, according to our Information Security Management System,
>  and intended solely for a specific individual to whom they are addressed.
>  Any unauthorised copy, disclosure or distribution of this message
>  is strictly forbidden. If you have received this transmission in error,
>  please notify the sender immediately and delete it.
>
> ______________________
> Este mensaje, y en su caso, cualquier fichero anexo al mismo,
>  puede contener informacion clasificada por su emisor como confidencial
>  en el marco de su Sistema de Gestion de Seguridad de la
> Informacion siendo para uso exclusivo del destinatario, quedando
> prohibida su divulgacion copia o distribucion a terceros sin la
> autorizacion expresa del remitente. Si Vd. ha recibido este mensaje
>  erroneamente, se ruega lo notifique al remitente y proceda a su borrado.
> Gracias por su colaboracion.
>
> ______________________
>
>
>
> ------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> _______________________________________________
> Geoserver-users mailing list
> [email protected]<mailto:
> [email protected]>
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>
>
> --
> Justin Deoliveira
> OpenGeo - http://opengeo.org
> Enterprise support for open source geospatial.
>
>
> ______________________
> This message including any attachments may contain confidential
> information, according to our Information Security Management System,
>  and intended solely for a specific individual to whom they are addressed.
>  Any unauthorised copy, disclosure or distribution of this message
>  is strictly forbidden. If you have received this transmission in error,
>  please notify the sender immediately and delete it.
>
> ______________________
> Este mensaje, y en su caso, cualquier fichero anexo al mismo,
>  puede contener informacion clasificada por su emisor como confidencial
>  en el marco de su Sistema de Gestion de Seguridad de la
> Informacion siendo para uso exclusivo del destinatario, quedando
> prohibida su divulgacion copia o distribucion a terceros sin la
> autorizacion expresa del remitente. Si Vd. ha recibido este mensaje
>  erroneamente, se ruega lo notifique al remitente y proceda a su borrado.
> Gracias por su colaboracion.
>
> ______________________
>
>
>
> ------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> _______________________________________________
> Geoserver-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>



-- 
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.
------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
Geoserver-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Reply via email to