Geoserver Users,
I'm running a GeoServer instance that has significant concurrent load.
It's got a fairly large catalog composed entirely of user-defined SQL
views, against a PostGIS 9.1.5 database. We've enabled flow control, set
garbage collection settings, only hit GWC, etc.
From time to time, especially under a heavy load, we run into a
TP-Processor thread that holds open a lock on a
org.geowebcache.layer.GridLocObj for around 10 minutes. While that
GridLocObj is held hostage, all other GWC requests wait behind it. Other
types of requests run normally. The offending TP-Processor thread hangs at
DatabaseMetaData.getTables. While that thread is waiting for data, I don't
see any running queries on the postgres server.
I have a few questions related to this:
(*) Has anyone run into this before, and did you find a resolution?
(*) Is there any reason that GeoServer/GeoTools needs the list of
tables in order to render a tile? I think this behavior is specific to
user-defined SQL views.
(*) Can the point that GridLocObj is locked be narrowed, so that the
catalog lookups don’t block all the other cached tiles from being served?
Example thread traces for a blocking thread and a blocked thread are
shown below. Notable locks and transitions between frameworks are bolded
and annotated. Thanks in advance for the help.
Best!
Erek
*Offending thread*
"TP-Processor22" daemon prio=10 tid=0x00007f231001e800 nid=0x5ee9 runnable
[0x00007f231f9af000]
java.lang.Thread.State: RUNNABLE
* at java.net.SocketInputStream.socketRead0(Native Method) <---
Hung here for ~10 minutes*
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
at
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
at
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
- locked <0x000000070cad1c60> (a
org.postgresql.core.v3.QueryExecutorImpl)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:252)
at
org.postgresql.jdbc2.AbstractJdbc2DatabaseMetaData.getTables(AbstractJdbc2DatabaseMetaData.java:2062)
at
org.apache.commons.dbcp.DelegatingDatabaseMetaData.getTables(DelegatingDatabaseMetaData.java:604)
* at
org.geotools.jdbc.JDBCDataStore.createTypeNames(JDBCDataStore.java:828)
<--- GeoTools gets list of all tables*
at
org.geotools.data.store.ContentDataStore.entry(ContentDataStore.java:554)
at
org.geotools.data.store.ContentDataStore.ensureEntry(ContentDataStore.java:585)
at
org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:383)
at
org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:350)
at
org.geotools.data.store.ContentDataStore.getSchema(ContentDataStore.java:335)
at
org.geotools.jdbc.JDBCDataStore.addVirtualTable(JDBCDataStore.java:278)
* at
org.geoserver.catalog.ResourcePool.getFeatureSource(ResourcePool.java:840)
<----Geoserver asks GeoTools for FeatureSource object*
at
org.geoserver.catalog.impl.FeatureTypeInfoImpl.getFeatureSource(FeatureTypeInfoImpl.java:91)
at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.geoserver.catalog.impl.ModificationProxy.invoke(ModificationProxy.java:144)
at $Proxy17.getFeatureSource(Unknown Source)
at
org.geoserver.wms.MapLayerInfo.getFeatureSource(MapLayerInfo.java:324)
at org.geoserver.wms.GetMap.run(GetMap.java:235)
at org.geoserver.wms.GetMap.run(GetMap.java:107)
at
org.geoserver.wms.DefaultWebMapService.getMap(DefaultWebMapService.java:353)
at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at
org.geoserver.gwc.wms.CacheSeedingWebMapService.invoke(CacheSeedingWebMapService.java:60)
at
org.geoserver.gwc.wms.CacheSeedingWebMapService.invoke(CacheSeedingWebMapService.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.geoserver.gwc.wms.CachingWebMapService.invoke(CachingWebMapService.java:58)
at
org.geoserver.gwc.wms.CachingWebMapService.invoke(CachingWebMapService.java:40)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.geoserver.ows.util.RequestObjectLogger.invoke(RequestObjectLogger.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy30.getMap(Unknown Source)
at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.geoserver.ows.Dispatcher.execute(Dispatcher.java:630)
at
org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:234)
at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.geoserver.gwc.GWC.dispatchOwsRequest(GWC.java:834)
at
org.geoserver.gwc.layer.CatalogConfiguration.dispatchOwsRequest(CatalogConfiguration.java:309)
at
org.geoserver.gwc.layer.GeoServerTileLayer.dispatchGetMap(GeoServerTileLayer.java:547)
at
org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:521)
- locked <0x00000007f37fbaa8> (a org.geowebcache.layer.GridLocObj)
<--- Holds lock on GridLocObj. Prevents other cached tiles from being
served
at
org.geoserver.gwc.layer.GeoServerTileLayer.getTile(GeoServerTileLayer.java:467)
at
org.geowebcache.GeoWebCacheDispatcher.handleServiceRequest(GeoWebCacheDispatcher.java:350)
at
org.geowebcache.GeoWebCacheDispatcher.handleRequestInternal(GeoWebCacheDispatcher.java:241)
...(Spring and tomcat stack)
at java.lang.Thread.run(Thread.java:722)
*Example Blocked Thread (dozens more of these, however many control flow
allows)*
"TP-Processor21" daemon prio=10 tid=0x00007f231001e000 nid=0x5ee8 waiting
for monitor entry [0x00007f231d9c7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:507)
- waiting to lock <0x00000007f37fbaa8> (a
org.geowebcache.layer.GridLocObj)
at
org.geoserver.gwc.layer.GeoServerTileLayer.getTile(GeoServerTileLayer.java:467)
at
org.geowebcache.GeoWebCacheDispatcher.handleServiceRequest(GeoWebCacheDispatcher.java:350)
at
org.geowebcache.GeoWebCacheDispatcher.handleRequestInternal(GeoWebCacheDispatcher.java:241)
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Geoserver-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
