So thanks to Simone helping me find some sample JMeter data, I was able to piece together a JMeter to find what has been causing my GeoServer 2.17.2 to become non-responsive.
The JMeter test is (5threads/users repeated for 10 iterations) basically as
follows. I will attach it here. I hope that's ok as it's only 15KB.
GET WMS request
GET WMS request
GET WMS request
POST external.imagemosaic (given a directory of images: /data/browse)
Note: the image mosaic is ~100 JPEG granules each w/ a corresponding .jgw
worldfile. We add a few images a few times each day.
It seems once this JMeter is iterated a few times, I will eventually get an
error like below (usually missing .shp or missing .qix file) then my GeoServer
WMS requests will all hang and GeoServer Java process utilization will stay
high. A restart of GeoServer sometimes totally resolves the issue and
sometimes the Shapefile data files of my ImageMosaic Store get hosed and I have
to manually delete and let GeoServer re-create them.
Is there something I could be doing better here? If I could provide the exact
image file path for each external.imagemosaic POST update instead of just
giving it the parent directory of my images, would that help? Any thoughts are
appreciated. Thanks!
17 Nov 01:16:27 ERROR [geoserver.ows] -
org.geoserver.platform.ServiceException: Error rendering coverage on the fast
path
at
org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:362)
at
org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:275)
at
org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:135)
at org.geoserver.wms.GetMap.executeInternal(GetMap.java:749)
at org.geoserver.wms.GetMap.run(GetMap.java:300)
at org.geoserver.wms.GetMap.run(GetMap.java:123)
at
org.geoserver.wms.DefaultWebMapService.getMap(DefaultWebMapService.java:246)
at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.geoserver.kml.WebMapServiceKmlInterceptor.invoke(WebMapServiceKmlInterceptor.java:38)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.geoserver.gwc.wms.CacheSeedingWebMapService.invoke(CacheSeedingWebMapService.java:55)
at
org.geoserver.gwc.wms.CacheSeedingWebMapService.invoke(CacheSeedingWebMapService.java:31)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.geoserver.gwc.wms.CachingWebMapService.invoke(CachingWebMapService.java:61)
at
org.geoserver.gwc.wms.CachingWebMapService.invoke(CachingWebMapService.java:41)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.geoserver.ows.util.RequestObjectLogger.invoke(RequestObjectLogger.java:50)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy104.getMap(Unknown Source)
at sun.reflect.GeneratedMethodAccessor310.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.geoserver.ows.Dispatcher.execute(Dispatcher.java:877)
at
org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:265)
at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177)
at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
at
org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:26)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at
org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:69)
at
org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:70)
at
org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:66)
at
org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:41)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at
org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:37)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
at
org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at
org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74)
at
org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at
org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
at
org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74)
at
org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at
org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:51)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at
org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70)
at
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74)
at
org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
at
org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:81)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at
org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70)
at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at
org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:52)
at
org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74)
at
org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at
org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:142)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:101)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at
org.geoserver.filters.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:77)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:53)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at
org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:46)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at
org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:42)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at
org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:311)
at
org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:265)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.geoserver.platform.ServiceException: java.lang.RuntimeException:
java.io.FileNotFoundException:
/mapdata/geoserver_data_dir/data/ArchiveQuery/BrowseImageMosaicStore/BrowseImageMosaicStore.shp
(No such file or directory)
at
org.geoserver.wms.map.RenderedImageMapOutputFormat.directRasterRender(RenderedImageMapOutputFormat.java:1211)
at
org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:359)
... 129 more
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
/mapdata/geoserver_data_dir/data/ArchiveQuery/BrowseImageMosaicStore/BrowseImageMosaicStore.shp
(No such file or directory)
at
org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:165)
at
org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:52)
at org.geotools.data.DataUtilities.bounds(DataUtilities.java:2725)
at
org.geotools.gce.imagemosaic.RasterLayerRequest.computeCoverageBoundingBox(RasterLayerRequest.java:383)
at
org.geotools.gce.imagemosaic.RasterLayerRequest.<init>(RasterLayerRequest.java:288)
at
org.geotools.gce.imagemosaic.RasterManager.read(RasterManager.java:1315)
at
org.geotools.gce.imagemosaic.ImageMosaicReader.read(ImageMosaicReader.java:905)
at
org.geotools.gce.imagemosaic.ImageMosaicReader.read(ImageMosaicReader.java:886)
at
org.geoserver.catalog.SingleGridCoverage2DReader.read(SingleGridCoverage2DReader.java:132)
at
org.geoserver.catalog.CoverageDimensionCustomizerReader.read(CoverageDimensionCustomizerReader.java:234)
at
org.geoserver.catalog.CoverageDimensionCustomizerReader.read(CoverageDimensionCustomizerReader.java:222)
at
org.geotools.renderer.lite.gridcoverage2d.GridCoverageReaderHelper.readSingleCoverage(GridCoverageReaderHelper.java:642)
at
org.geotools.renderer.lite.gridcoverage2d.GridCoverageReaderHelper.readCoverageInEnvelope(GridCoverageReaderHelper.java:371)
at
org.geotools.renderer.lite.gridcoverage2d.GridCoverageReaderHelper.readCoverages(GridCoverageReaderHelper.java:247)
at
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer.renderImage(GridCoverageRenderer.java:690)
at
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer.renderImage(GridCoverageRenderer.java:639)
at
org.geoserver.wms.map.RenderedImageMapOutputFormat.directRasterRender(RenderedImageMapOutputFormat.java:1059)
... 130 more
Caused by: java.io.FileNotFoundException:
/mapdata/geoserver_data_dir/data/ArchiveQuery/BrowseImageMosaicStore/BrowseImageMosaicStore.shp
(No such file or directory)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at
org.geotools.data.shapefile.files.ShpFiles.getReadChannel(ShpFiles.java:746)
at
org.geotools.data.shapefile.shp.ShapefileReader.<init>(ShapefileReader.java:261)
at
org.geotools.data.shapefile.ShapefileSetManager.openShapeReader(ShapefileSetManager.java:64)
at
org.geotools.data.shapefile.ShapefileFeatureSource.getReaderInternal(ShapefileFeatureSource.java:375)
at
org.geotools.data.shapefile.ShapefileFeatureStore.getReaderInternal(ShapefileFeatureStore.java:133)
at
org.geotools.data.store.ContentFeatureSource.getReader(ContentFeatureSource.java:619)
at
org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:163)
... 146 more
17 Nov 01:16:35 INFO [geoserver.wms] -
Request: getServiceInfo
Ashley Mort
Sr. Software Engineer
972-638-6317 office
814-571-3039 cell
TestPlanGeoServer.jmx
Description: TestPlanGeoServer.jmx
_______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
