[ https://issues.apache.org/jira/browse/TIKA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180027#comment-17180027 ]
Tim Allison edited comment on TIKA-3173 at 8/18/20, 7:24 PM: ------------------------------------------------------------- > Should the Watchdog have detected tika was dead in the water and restarted > automatically to avoid the need to have additional doc fail parse before > getting back online? As you point out, the child figured it out (22:54:36), which is good. But, I can't explain why it took 2.5 minutes for the ServerWatchDog to notice that the child is down (22:57:00). What is your {{pingPulseMillis}}? Do the logs showing the server restarting earlier than 22:57:00... Er what's happening between 54:36 and 57:00? was (Author: talli...@mitre.org): > Should the Watchdog have detected tika was dead in the water and restarted > automatically to avoid the need to have additional doc fail parse before > getting back online? As you point out, the child figured it out (22:54:36), which is good. But, I can't explain why it took 3 minutes for the ServerWatchDog to notice that the child is down (22:57:00). What is your {{pingPulseMillis}}? Do the logs showing the server restarting earlier than 22:57:00... Er what's happening between 54:36 and 57:00? > Tika server with spawnChild - server does not recover from OOM until an > additional file comes in > ------------------------------------------------------------------------------------------------ > > Key: TIKA-3173 > URL: https://issues.apache.org/jira/browse/TIKA-3173 > Project: Tika > Issue Type: Improvement > Reporter: Nicholas DiPiazza > Priority: Major > > On latest Tika 1.x latest branch, on Windows 10 with latest patches. > First start a Tika server with spawnChild enabled. > Next run the following test: > (This test is available for run here > https://github.com/nddipiazza/tika/blob/4a094ab310691a68d1c8d270cee4e138eb43a1ed/tika-server/src/test/java/org/apache/tika/server/RecoverFromOutOfMemoryTest.java) > Step 1) Run a normal word doc through the /rmeta/text endpoint. > Step 2) Run a file that is known to crash the tika parser. > Step 3) Pause, give it several minutes to recover. > Step 4) Run another normal word doc through the /rmeta/text endpoint. > Step 5) Run another normal word doc through the /rmeta/text endpoint. > The Step (2) will hang and finally fail with a socket timeout. > The logs at this point show this: > {code} > INFO 2020-08-17 22:53:27,255 [pool-1-thread-1] (TikaServerCli.java:131) - > Starting Apache Tika server > INFO 2020-08-17 22:53:27,276 [pool-1-thread-1] (TikaServerWatchDog.java:69) > - server watch dog is starting up > Aug 17, 2020 10:53:28 PM org.apache.tika.config.InitializableProblemHandler$3 > handleInitializableProblem > WARNING: J2KImageReader not loaded. JPEG2000 files will not be processed. > See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io > for optional dependencies. > Aug 17, 2020 10:53:28 PM org.apache.tika.config.InitializableProblemHandler$3 > handleInitializableProblem > WARNING: org.xerial's sqlite-jdbc is not loaded. > Please provide the jar on your classpath to parse sqlite files. > See tika-parsers/pom.xml for the correct version. > INFO 2020-08-17 22:53:28,766 [main] (RecoverFromOutOfMemoryTest.java:125) - > Still waiting for Tika server on port 54976 to start. Retries remaining: 9 > INFO 2020-08-17 22:53:29,148 [main] (TikaServerCli.java:131) - Starting > Apache Tika server > INFO 2020-08-17 22:53:29,845 [main] (ServerImpl.java:94) - Setting the > server's publish address to be http://192.168.1.39:54976/ > INFO 2020-08-17 22:53:29,952 [main] (Log.java:169) - Logging initialized > @2608ms to org.eclipse.jetty.util.log.Slf4jLog > INFO 2020-08-17 22:53:30,082 [main] (Server.java:359) - > jetty-9.4.27.v20200227; built: 2020-02-27T18:37:21.340Z; git: > a304fd9f351f337e7c0e2a7c28878dd536149c6c; jvm 1.8.0_242-b08 > INFO 2020-08-17 22:53:30,406 [main] (AbstractConnector.java:331) - Started > ServerConnector@46271dd6{HTTP/1.1, (http/1.1)}{192.168.1.39:54976} > INFO 2020-08-17 22:53:30,406 [main] (Server.java:399) - Started @3063ms > WARN 2020-08-17 22:53:30,419 [main] (ContextHandler.java:1534) - Empty > contextPath > INFO 2020-08-17 22:53:30,443 [main] (ContextHandler.java:828) - Started > o.e.j.s.h.ContextHandler@54eb2b70{/,null,AVAILABLE} > INFO 2020-08-17 22:53:30,443 [main] (TikaServerCli.java:352) - Started > Apache Tika server at http://192.168.1.39:54976/ > INFO 2020-08-17 22:53:31,525 [qtp1923999715-18] (TikaResource.java:439) - > rmeta/text (application/msword) > INFO 2020-08-17 22:53:31,797 [main] (RecoverFromOutOfMemoryTest.java:111) - > Normal doc response: [{"Author":"Maxim > Valyanskiy","Content-Type":"application/msword","Creation-Date":"2009-06-11T13:53:04Z","Last-Modified":"1601-01-01T00:00:00Z","Last-Printed":"1601-01-01T00:00:00Z","Last-Save-Date":"1601-01-01T00:00:00Z","Revision-Number":"0","X-Parsed-By":["org.apache.tika.parser.DefaultParser","org.apache.tika.parser.microsoft.OfficeParser"],"X-TIKA:content":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\ntest\n\n","X-TIKA:content_handler":"ToTextContentHandler","X-TIKA:embedded_depth":"0","X-TIKA:parse_time_millis":"210","cp:revision":"0","creator":"Maxim > Valyanskiy","date":"1601-01-01T00:00:00Z","dc:creator":"Maxim > Valyanskiy","dcterms:created":"2009-06-11T13:53:04Z","dcterms:modified":"1601-01-01T00:00:00Z","meta:author":"Maxim > > Valyanskiy","meta:creation-date":"2009-06-11T13:53:04Z","meta:print-date":"1601-01-01T00:00:00Z","meta:save-date":"1601-01-01T00:00:00Z","modified":"1601-01-01T00:00:00Z"}] > INFO 2020-08-17 22:53:31,814 [qtp1923999715-16] (TikaResource.java:439) - > rmeta/text (application/vnd.ms-excel) > WARN 2020-08-17 22:54:32,204 [main] (LogUtils.java:475) - Interceptor for > {http://192.168.1.39:54976/rmeta/text}WebClient has thrown exception, > unwinding now > org.apache.cxf.interceptor.Fault: Could not receive Message. > at > org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:65) > at > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) > at > org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:701) > at > org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1086) > at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:932) > at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:901) > at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:364) > at org.apache.cxf.jaxrs.client.WebClient.put(WebClient.java:382) > at > org.apache.tika.server.RecoverFromOutOfMemoryTest.parseBombFile(RecoverFromOutOfMemoryTest.java:96) > at > org.apache.tika.server.RecoverFromOutOfMemoryTest.testOutOfMemory(RecoverFromOutOfMemoryTest.java:79) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) > at > com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) > at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) > Caused by: java.net.SocketTimeoutException: SocketTimeoutException invoking > http://192.168.1.39:54976/rmeta/text: Read timed out > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1400) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1384) > at > org.apache.cxf.io.AbstractWrappedOutputStream.close(AbstractWrappedOutputStream.java:77) > at > org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) > at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:671) > at > org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63) > ... 35 more > Caused by: java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) > at > org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream$2.run(URLConnectionHTTPConduit.java:377) > at > org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream$2.run(URLConnectionHTTPConduit.java:373) > at java.security.AccessController.doPrivileged(Native Method) > at > org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:373) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1597) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1625) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1570) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1371) > ... 39 more > INFO 2020-08-17 22:54:36,186 [main] (RecoverFromOutOfMemoryTest.java:82) - > Could not parse the bomb file! > javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: > SocketTimeoutException invoking http://192.168.1.39:54976/rmeta/text: Read > timed out > at > org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:629) > at > org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:605) > at org.apache.cxf.jaxrs.client.WebClient.doResponse(WebClient.java:1150) > at > org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1087) > at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:932) > at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:901) > at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:364) > at org.apache.cxf.jaxrs.client.WebClient.put(WebClient.java:382) > at > org.apache.tika.server.RecoverFromOutOfMemoryTest.parseBombFile(RecoverFromOutOfMemoryTest.java:96) > at > org.apache.tika.server.RecoverFromOutOfMemoryTest.testOutOfMemory(RecoverFromOutOfMemoryTest.java:79) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) > at > com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) > at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) > Caused by: java.net.SocketTimeoutException: SocketTimeoutException invoking > http://192.168.1.39:54976/rmeta/text: Read timed out > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1400) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1384) > at > org.apache.cxf.io.AbstractWrappedOutputStream.close(AbstractWrappedOutputStream.java:77) > at > org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) > at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:671) > at > org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63) > at > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) > at > org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:701) > at > org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1086) > ... 32 more > Caused by: java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) > at > org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream$2.run(URLConnectionHTTPConduit.java:377) > at > org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream$2.run(URLConnectionHTTPConduit.java:373) > at java.security.AccessController.doPrivileged(Native Method) > at > org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:373) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1597) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1625) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1570) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1371) > ... 39 more > WARN 2020-08-17 22:54:36,595 [qtp1923999715-16] (TikaResource.java:419) - > rmeta/text: OOM (null) > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOfRange(Arrays.java:3664) > at java.lang.String.<init>(String.java:207) > at java.lang.String.substring(String.java:1969) > at java.lang.String.trim(String.java:2878) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.addTextCell(ExcelExtractor.java:594) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.internalProcessRecord(ExcelExtractor.java:469) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.processRecord(ExcelExtractor.java:367) > at > org.apache.poi.hssf.eventusermodel.FormatTrackingHSSFListener.processRecord(FormatTrackingHSSFListener.java:92) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener$TikaFormatTrackingHSSFListener.processRecord(ExcelExtractor.java:689) > at > org.apache.poi.hssf.eventusermodel.HSSFRequest.processRecord(HSSFRequest.java:106) > at > org.apache.poi.hssf.eventusermodel.HSSFEventFactory.genericProcessEvents(HSSFEventFactory.java:172) > at > org.apache.poi.hssf.eventusermodel.HSSFEventFactory.processEvents(HSSFEventFactory.java:129) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.processFile(ExcelExtractor.java:343) > at > org.apache.tika.parser.microsoft.ExcelExtractor.parse(ExcelExtractor.java:172) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:183) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:131) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280) > at > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143) > at > org.apache.tika.parser.RecursiveParserWrapper.parse(RecursiveParserWrapper.java:234) > at > org.apache.tika.server.resource.TikaResource.parse(TikaResource.java:409) > at > org.apache.tika.server.resource.RecursiveMetadataResource.parseMetadata(RecursiveMetadataResource.java:158) > at > org.apache.tika.server.resource.RecursiveMetadataResource.getMetadata(RecursiveMetadataResource.java:123) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) > at > org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) > WARN 2020-08-17 22:54:36,619 [qtp1923999715-16] (LogUtils.java:475) - > Application {http://resource.server.tika.apache.org/}MetadataResource has > thrown exception, unwinding now > org.apache.cxf.interceptor.Fault: Java heap space > at > org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) > at > org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) > at > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) > at > org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) > at > org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267) > at > org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247) > at > org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.Server.handle(Server.java:500) > at > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOfRange(Arrays.java:3664) > at java.lang.String.<init>(String.java:207) > at java.lang.String.substring(String.java:1969) > at java.lang.String.trim(String.java:2878) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.addTextCell(ExcelExtractor.java:594) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.internalProcessRecord(ExcelExtractor.java:469) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.processRecord(ExcelExtractor.java:367) > at > org.apache.poi.hssf.eventusermodel.FormatTrackingHSSFListener.processRecord(FormatTrackingHSSFListener.java:92) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener$TikaFormatTrackingHSSFListener.processRecord(ExcelExtractor.java:689) > at > org.apache.poi.hssf.eventusermodel.HSSFRequest.processRecord(HSSFRequest.java:106) > at > org.apache.poi.hssf.eventusermodel.HSSFEventFactory.genericProcessEvents(HSSFEventFactory.java:172) > at > org.apache.poi.hssf.eventusermodel.HSSFEventFactory.processEvents(HSSFEventFactory.java:129) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.processFile(ExcelExtractor.java:343) > at > org.apache.tika.parser.microsoft.ExcelExtractor.parse(ExcelExtractor.java:172) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:183) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:131) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280) > at > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143) > at > org.apache.tika.parser.RecursiveParserWrapper.parse(RecursiveParserWrapper.java:234) > at > org.apache.tika.server.resource.TikaResource.parse(TikaResource.java:409) > at > org.apache.tika.server.resource.RecursiveMetadataResource.parseMetadata(RecursiveMetadataResource.java:158) > at > org.apache.tika.server.resource.RecursiveMetadataResource.getMetadata(RecursiveMetadataResource.java:123) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) > at > org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) > WARN 2020-08-17 22:54:36,623 [qtp1923999715-16] > (PhaseInterceptorChain.java:501) - Exception in handleFault on interceptor > org.apache.cxf.jaxrs.interceptor.JAXRSDefaultFaultOutInterceptor@5eee8836 > org.apache.cxf.interceptor.Fault: Java heap space > at > org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) > at > org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) > at > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) > at > org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) > at > org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267) > at > org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247) > at > org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.Server.handle(Server.java:500) > at > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOfRange(Arrays.java:3664) > at java.lang.String.<init>(String.java:207) > at java.lang.String.substring(String.java:1969) > at java.lang.String.trim(String.java:2878) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.addTextCell(ExcelExtractor.java:594) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.internalProcessRecord(ExcelExtractor.java:469) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.processRecord(ExcelExtractor.java:367) > at > org.apache.poi.hssf.eventusermodel.FormatTrackingHSSFListener.processRecord(FormatTrackingHSSFListener.java:92) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener$TikaFormatTrackingHSSFListener.processRecord(ExcelExtractor.java:689) > at > org.apache.poi.hssf.eventusermodel.HSSFRequest.processRecord(HSSFRequest.java:106) > at > org.apache.poi.hssf.eventusermodel.HSSFEventFactory.genericProcessEvents(HSSFEventFactory.java:172) > at > org.apache.poi.hssf.eventusermodel.HSSFEventFactory.processEvents(HSSFEventFactory.java:129) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.processFile(ExcelExtractor.java:343) > at > org.apache.tika.parser.microsoft.ExcelExtractor.parse(ExcelExtractor.java:172) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:183) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:131) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280) > at > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143) > at > org.apache.tika.parser.RecursiveParserWrapper.parse(RecursiveParserWrapper.java:234) > at > org.apache.tika.server.resource.TikaResource.parse(TikaResource.java:409) > at > org.apache.tika.server.resource.RecursiveMetadataResource.parseMetadata(RecursiveMetadataResource.java:158) > at > org.apache.tika.server.resource.RecursiveMetadataResource.getMetadata(RecursiveMetadataResource.java:123) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) > at > org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) > ERROR 2020-08-17 22:54:36,624 [qtp1923999715-16] > (AbstractFaultChainInitiatorObserver.java:114) - An unexpected error occurred > during error handling. No further error processing will occur. > org.apache.cxf.interceptor.Fault: Java heap space > at > org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) > at > org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) > at > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) > at > org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) > at > org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267) > at > org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247) > at > org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.Server.handle(Server.java:500) > at > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOfRange(Arrays.java:3664) > at java.lang.String.<init>(String.java:207) > at java.lang.String.substring(String.java:1969) > at java.lang.String.trim(String.java:2878) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.addTextCell(ExcelExtractor.java:594) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.internalProcessRecord(ExcelExtractor.java:469) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.processRecord(ExcelExtractor.java:367) > at > org.apache.poi.hssf.eventusermodel.FormatTrackingHSSFListener.processRecord(FormatTrackingHSSFListener.java:92) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener$TikaFormatTrackingHSSFListener.processRecord(ExcelExtractor.java:689) > at > org.apache.poi.hssf.eventusermodel.HSSFRequest.processRecord(HSSFRequest.java:106) > at > org.apache.poi.hssf.eventusermodel.HSSFEventFactory.genericProcessEvents(HSSFEventFactory.java:172) > at > org.apache.poi.hssf.eventusermodel.HSSFEventFactory.processEvents(HSSFEventFactory.java:129) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.processFile(ExcelExtractor.java:343) > at > org.apache.tika.parser.microsoft.ExcelExtractor.parse(ExcelExtractor.java:172) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:183) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:131) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280) > at > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143) > at > org.apache.tika.parser.RecursiveParserWrapper.parse(RecursiveParserWrapper.java:234) > at > org.apache.tika.server.resource.TikaResource.parse(TikaResource.java:409) > at > org.apache.tika.server.resource.RecursiveMetadataResource.parseMetadata(RecursiveMetadataResource.java:158) > at > org.apache.tika.server.resource.RecursiveMetadataResource.getMetadata(RecursiveMetadataResource.java:123) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) > at > org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) > WARN 2020-08-17 22:54:36,624 [qtp1923999715-16] (HttpChannel.java:591) - > /rmeta/text > org.apache.cxf.interceptor.Fault: Java heap space > at > org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) > at > org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) > at > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) > at > org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) > at > org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267) > at > org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247) > at > org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.Server.handle(Server.java:500) > at > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOfRange(Arrays.java:3664) > at java.lang.String.<init>(String.java:207) > at java.lang.String.substring(String.java:1969) > at java.lang.String.trim(String.java:2878) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.addTextCell(ExcelExtractor.java:594) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.internalProcessRecord(ExcelExtractor.java:469) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.processRecord(ExcelExtractor.java:367) > at > org.apache.poi.hssf.eventusermodel.FormatTrackingHSSFListener.processRecord(FormatTrackingHSSFListener.java:92) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener$TikaFormatTrackingHSSFListener.processRecord(ExcelExtractor.java:689) > at > org.apache.poi.hssf.eventusermodel.HSSFRequest.processRecord(HSSFRequest.java:106) > at > org.apache.poi.hssf.eventusermodel.HSSFEventFactory.genericProcessEvents(HSSFEventFactory.java:172) > at > org.apache.poi.hssf.eventusermodel.HSSFEventFactory.processEvents(HSSFEventFactory.java:129) > at > org.apache.tika.parser.microsoft.ExcelExtractor$TikaHSSFListener.processFile(ExcelExtractor.java:343) > at > org.apache.tika.parser.microsoft.ExcelExtractor.parse(ExcelExtractor.java:172) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:183) > at > org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:131) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280) > at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280) > at > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143) > at > org.apache.tika.parser.RecursiveParserWrapper.parse(RecursiveParserWrapper.java:234) > at > org.apache.tika.server.resource.TikaResource.parse(TikaResource.java:409) > at > org.apache.tika.server.resource.RecursiveMetadataResource.parseMetadata(RecursiveMetadataResource.java:158) > at > org.apache.tika.server.resource.RecursiveMetadataResource.getMetadata(RecursiveMetadataResource.java:123) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) > at > org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) > WARN 2020-08-17 22:54:36,765 [Thread-1] (ServerStatusWatcher.java:196) - > child process observed ERROR and is shutting down. > INFO 2020-08-17 22:54:36,767 [Thread-1] (ServerStatusWatcher.java:181) - > Shutting down child process with status: ERROR > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 > {code} > So far so good. The logs indicate the child process has recognized the issue. > So after the delay, we perform step (4) and run another working document. > Unfortunatley, this parse fails. Here are the logs from that attempt: > {code} > WARN 2020-08-17 22:57:00,543 [Thread-1] (TikaServerWatchDog.java:128) - > 148488 ms have elapsed since last successful ping. Destroying child now > WARN 2020-08-17 22:57:00,544 [pool-1-thread-1] (TikaServerWatchDog.java:290) > - Received non-operating status from child: ERROR > Aug 17, 2020 10:57:01 PM org.apache.tika.config.InitializableProblemHandler$3 > handleInitializableProblem > WARNING: J2KImageReader not loaded. JPEG2000 files will not be processed. > See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io > for optional dependencies. > Aug 17, 2020 10:57:01 PM org.apache.tika.config.InitializableProblemHandler$3 > handleInitializableProblem > WARNING: org.xerial's sqlite-jdbc is not loaded. > Please provide the jar on your classpath to parse sqlite files. > See tika-parsers/pom.xml for the correct version. > INFO 2020-08-17 22:57:02,300 [main] (TikaServerCli.java:131) - Starting > Apache Tika server > INFO 2020-08-17 22:57:02,373 [pool-1-thread-1] (TikaServerWatchDog.java:83) > - Successfully restarted child process -- 0 restarts so far) > ERROR 2020-08-17 22:57:02,588 [main] (JAXRSUtils.java:1889) - Problem with > writing the data, class java.io.BufferedInputStream, ContentType: > application/msword > WARN 2020-08-17 22:57:02,588 [main] (LogUtils.java:475) - Interceptor for > {http://192.168.1.39:54976/rmeta/text}WebClient has thrown exception, > unwinding now > org.apache.cxf.interceptor.Fault: Problem with writing the data, class > java.io.BufferedInputStream, ContentType: application/msword > at > org.apache.cxf.jaxrs.client.WebClient$BodyWriter.doWriteBody(WebClient.java:1228) > at > org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1233) > at > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) > at > org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:701) > at > org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1086) > at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:932) > at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:901) > at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:364) > at org.apache.cxf.jaxrs.client.WebClient.put(WebClient.java:382) > at > org.apache.tika.server.RecoverFromOutOfMemoryTest.assertCanParseNormalDoc(RecoverFromOutOfMemoryTest.java:102) > at > org.apache.tika.server.RecoverFromOutOfMemoryTest.testOutOfMemory(RecoverFromOutOfMemoryTest.java:89) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) > at > com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) > at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) > Caused by: javax.ws.rs.ProcessingException: Problem with writing the data, > class java.io.BufferedInputStream, ContentType: application/msword > at > org.apache.cxf.jaxrs.client.AbstractClient.reportMessageHandlerProblem(AbstractClient.java:844) > at > org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:529) > at > org.apache.cxf.jaxrs.client.WebClient$BodyWriter.doWriteBody(WebClient.java:1223) > ... 36 more > Caused by: java.net.ConnectException: Connection refused: connect > at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) > at > java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:607) > at sun.net.NetworkClient.doConnect(NetworkClient.java:175) > at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) > at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) > at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) > at sun.net.www.http.HttpClient.New(HttpClient.java:339) > at sun.net.www.http.HttpClient.New(HttpClient.java:357) > at > sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226) > at > sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205) > at > sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) > at > sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990) > at > sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340) > at > sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315) > at > org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:274) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1343) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1304) > at > org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:307) > at > org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47) > at > org.apache.cxf.io.AbstractThresholdOutputStream.unBuffer(AbstractThresholdOutputStream.java:89) > at > org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:63) > at > org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:51) > at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:193) > at org.apache.cxf.helpers.IOUtils.copyAndCloseInput(IOUtils.java:159) > at > org.apache.cxf.jaxrs.provider.BinaryDataProvider.copyInputToOutput(BinaryDataProvider.java:205) > at > org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:159) > at > org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1472) > at > org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:515) > ... 37 more > INFO 2020-08-17 22:57:02,590 [pool-1-thread-1] (TikaServerWatchDog.java:99) > - about to shutdown process > javax.ws.rs.ProcessingException: Problem with writing the data, class > java.io.BufferedInputStream, ContentType: application/msword > at > org.apache.cxf.jaxrs.client.AbstractClient.reportMessageHandlerProblem(AbstractClient.java:844) > at > org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:529) > at > org.apache.cxf.jaxrs.client.WebClient$BodyWriter.doWriteBody(WebClient.java:1223) > at > org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1233) > at > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) > at > org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:701) > at > org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1086) > at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:932) > at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:901) > at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:364) > at org.apache.cxf.jaxrs.client.WebClient.put(WebClient.java:382) > at > org.apache.tika.server.RecoverFromOutOfMemoryTest.assertCanParseNormalDoc(RecoverFromOutOfMemoryTest.java:102) > at > org.apache.tika.server.RecoverFromOutOfMemoryTest.testOutOfMemory(RecoverFromOutOfMemoryTest.java:89) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) > at > com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) > at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) > Caused by: java.net.ConnectException: Connection refused: connect > at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) > at > java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:607) > at sun.net.NetworkClient.doConnect(NetworkClient.java:175) > at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) > at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) > at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) > at sun.net.www.http.HttpClient.New(HttpClient.java:339) > at sun.net.www.http.HttpClient.New(HttpClient.java:357) > at > sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226) > at > sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205) > at > sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) > at > sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990) > at > sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340) > at > sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315) > at > org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:274) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1343) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1304) > at > org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:307) > at > org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47) > at > org.apache.cxf.io.AbstractThresholdOutputStream.unBuffer(AbstractThresholdOutputStream.java:89) > at > org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:63) > at > org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:51) > at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:193) > at org.apache.cxf.helpers.IOUtils.copyAndCloseInput(IOUtils.java:159) > at > org.apache.cxf.jaxrs.provider.BinaryDataProvider.copyInputToOutput(BinaryDataProvider.java:205) > at > org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:159) > at > org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1472) > at > org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:515) > ... 37 more > {code} > The tika server saw the issue, took action by restarting, but we get an extra > connection refused in the meantime. > Finally, we run step (5) happens and the server is back up and able to parse > again at this point. > Should the Watchdog have detected tika was dead in the water and restarted > automatically to avoid the need to have additional doc fail parse before > getting back online? -- This message was sent by Atlassian Jira (v8.3.4#803005)