It does not sound like there is anything better you could be doing … this may be a race condition in the Shapefile code.
Next time it happens you could ask java for a stack dump (which shows what geoserver is doing) and may provide a clue. There are a number of ways to do this … here is the first google search result ( https://access.redhat.com/solutions/18178). You could also try and use PostGIS or GeoPackage and perform the same test to determine if this is a GeoServer issue, or if the problem is not reproducible with PostGIS we would know it is a Shapefile (and thus GeoTools) issue. Jody On Tue, Nov 16, 2021 at 6:10 PM Ashley Mort via Geoserver-users < geoserver-users@lists.sourceforge.net> wrote: > So thanks to Simone helping me find some sample JMeter data, I was able to > piece together a JMeter to find what has been causing my GeoServer 2.17.2 > to become non-responsive. > > > > The JMeter test is (5threads/users repeated for 10 iterations) basically > as follows. I will attach it here. I hope that’s ok as it’s only 15KB. > > GET WMS request > > GET WMS request > > GET WMS request > > POST external.imagemosaic (given a directory of images: /data/browse) > > *Note: the image mosaic is ~100 JPEG granules each w/ a corresponding .jgw > worldfile. We add a few images a few times each day.* > > It seems once this JMeter is iterated a few times, I will eventually get > an error like below (usually missing .shp or missing .qix file) then my > GeoServer WMS requests will all hang and GeoServer Java process utilization > will stay high. A restart of GeoServer sometimes totally resolves the > issue and sometimes the Shapefile data files of my ImageMosaic Store get > hosed and I have to manually delete and let GeoServer re-create them. > > > > Is there something I could be doing better here? If I could provide the > exact image file path for each external.imagemosaic POST update instead of > just giving it the parent directory of my images, would that help? Any > thoughts are appreciated. Thanks! > > > > > > > > 17 Nov 01:16:27 ERROR [geoserver.ows] - > > org.geoserver.platform.ServiceException: Error rendering coverage on the > fast path > > at > org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:362) > > at > org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:275) > > at > org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:135) > > at org.geoserver.wms.GetMap.executeInternal(GetMap.java:749) > > at org.geoserver.wms.GetMap.run(GetMap.java:300) > > at org.geoserver.wms.GetMap.run(GetMap.java:123) > > at > org.geoserver.wms.DefaultWebMapService.getMap(DefaultWebMapService.java:246) > > at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) > > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > at > org.geoserver.kml.WebMapServiceKmlInterceptor.invoke(WebMapServiceKmlInterceptor.java:38) > > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > > at > org.geoserver.gwc.wms.CacheSeedingWebMapService.invoke(CacheSeedingWebMapService.java:55) > > at > org.geoserver.gwc.wms.CacheSeedingWebMapService.invoke(CacheSeedingWebMapService.java:31) > > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > > at > org.geoserver.gwc.wms.CachingWebMapService.invoke(CachingWebMapService.java:61) > > at > org.geoserver.gwc.wms.CachingWebMapService.invoke(CachingWebMapService.java:41) > > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > > at > org.geoserver.ows.util.RequestObjectLogger.invoke(RequestObjectLogger.java:50) > > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > at com.sun.proxy.$Proxy104.getMap(Unknown Source) > > at sun.reflect.GeneratedMethodAccessor310.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at org.geoserver.ows.Dispatcher.execute(Dispatcher.java:877) > > at > org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:265) > > at > org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177) > > at > org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52) > > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) > > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) > > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) > > at > org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) > > at > org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:26) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > > at > org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:69) > > at > org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:70) > > at > org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:66) > > at > org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:41) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > > at > org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:37) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) > > at > org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70) > > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) > > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) > > at > org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74) > > at > org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91) > > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) > > at > org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70) > > at > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) > > at > org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74) > > at > org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91) > > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) > > at > org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:51) > > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) > > at > org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70) > > at > org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215) > > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) > > at > org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74) > > at > org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91) > > at > org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:81) > > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) > > at > org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70) > > at > org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) > > at > org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:52) > > at > org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74) > > at > org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91) > > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) > > at > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) > > at > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) > > at > org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:142) > > at > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) > > at > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > > at > org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:101) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > > at > org.geoserver.filters.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:77) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > > at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:53) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > > at > org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:46) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > > at > org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:42) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > > at > org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:311) > > at > org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:265) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > > at > org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) > > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) > > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) > > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) > > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) > > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) > > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) > > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) > > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) > > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) > > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) > > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) > > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) > > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) > > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) > > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) > > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) > > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > > at org.eclipse.jetty.server.Server.handle(Server.java:505) > > at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) > > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) > > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) > > at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > > at > org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) > > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) > > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) > > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) > > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) > > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: org.geoserver.platform.ServiceException: > java.lang.RuntimeException: java.io.FileNotFoundException: > /mapdata/geoserver_data_dir/data/ArchiveQuery/BrowseImageMosaicStore/BrowseImageMosaicStore.shp > (No such file or directory) > > at > org.geoserver.wms.map.RenderedImageMapOutputFormat.directRasterRender(RenderedImageMapOutputFormat.java:1211) > > at > org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:359) > > ... 129 more > > Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: > /mapdata/geoserver_data_dir/data/ArchiveQuery/BrowseImageMosaicStore/BrowseImageMosaicStore.shp > (No such file or directory) > > at > org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:165) > > at > org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:52) > > at org.geotools.data.DataUtilities.bounds(DataUtilities.java:2725) > > at > org.geotools.gce.imagemosaic.RasterLayerRequest.computeCoverageBoundingBox(RasterLayerRequest.java:383) > > at > org.geotools.gce.imagemosaic.RasterLayerRequest.<init>(RasterLayerRequest.java:288) > > at > org.geotools.gce.imagemosaic.RasterManager.read(RasterManager.java:1315) > > at > org.geotools.gce.imagemosaic.ImageMosaicReader.read(ImageMosaicReader.java:905) > > at > org.geotools.gce.imagemosaic.ImageMosaicReader.read(ImageMosaicReader.java:886) > > at > org.geoserver.catalog.SingleGridCoverage2DReader.read(SingleGridCoverage2DReader.java:132) > > at > org.geoserver.catalog.CoverageDimensionCustomizerReader.read(CoverageDimensionCustomizerReader.java:234) > > at > org.geoserver.catalog.CoverageDimensionCustomizerReader.read(CoverageDimensionCustomizerReader.java:222) > > at > org.geotools.renderer.lite.gridcoverage2d.GridCoverageReaderHelper.readSingleCoverage(GridCoverageReaderHelper.java:642) > > at > org.geotools.renderer.lite.gridcoverage2d.GridCoverageReaderHelper.readCoverageInEnvelope(GridCoverageReaderHelper.java:371) > > at > org.geotools.renderer.lite.gridcoverage2d.GridCoverageReaderHelper.readCoverages(GridCoverageReaderHelper.java:247) > > at > org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer.renderImage(GridCoverageRenderer.java:690) > > at > org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer.renderImage(GridCoverageRenderer.java:639) > > at > org.geoserver.wms.map.RenderedImageMapOutputFormat.directRasterRender(RenderedImageMapOutputFormat.java:1059) > > ... 130 more > > Caused by: java.io.FileNotFoundException: > /mapdata/geoserver_data_dir/data/ArchiveQuery/BrowseImageMosaicStore/BrowseImageMosaicStore.shp > (No such file or directory) > > at java.io.RandomAccessFile.open0(Native Method) > > at java.io.RandomAccessFile.open(RandomAccessFile.java:316) > > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) > > at > org.geotools.data.shapefile.files.ShpFiles.getReadChannel(ShpFiles.java:746) > > at > org.geotools.data.shapefile.shp.ShapefileReader.<init>(ShapefileReader.java:261) > > at > org.geotools.data.shapefile.ShapefileSetManager.openShapeReader(ShapefileSetManager.java:64) > > at > org.geotools.data.shapefile.ShapefileFeatureSource.getReaderInternal(ShapefileFeatureSource.java:375) > > at > org.geotools.data.shapefile.ShapefileFeatureStore.getReaderInternal(ShapefileFeatureStore.java:133) > > at > org.geotools.data.store.ContentFeatureSource.getReader(ContentFeatureSource.java:619) > > at > org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:163) > > ... 146 more > > 17 Nov 01:16:35 INFO [geoserver.wms] - > > Request: getServiceInfo > > > > > > Ashley Mort > > Sr. Software Engineer > > 972-638-6317 office > > 814-571-3039 cell > > > _______________________________________________ > Geoserver-users mailing list > > Please make sure you read the following two resources before posting to > this list: > - Earning your support instead of buying it, but Ian Turton: > http://www.ianturton.com/talks/foss4g.html#/ > - The GeoServer user list posting guidelines: > http://geoserver.org/comm/userlist-guidelines.html > > If you want to request a feature or an improvement, also see this: > https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer > > > Geoserver-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geoserver-users > -- -- Jody Garnett
_______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
