Bit of a plug for tika-helm here folks... Horizontal pod autoscaling [0] is available (off by default) and can be configured via values.yaml or overridden on the CLI. This would mean that the availability to a tika-server would still be available in the event that one particular pod went down due to OOM. See [1] for more details. lewismc
[0] https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/ [1] https://github.com/apache/tika-helm/blob/main/values.yaml#L99-L105 On 2023/03/08 20:29:49 Tim Allison wrote: > HIT_MAX_FILES is expected. We designed that in to periodically > restart the server to avoid memory leaks in badly behaving parsers. > You can configure a value for the max file threshold if necessary. > > The restart failed, and that's a problem. Let me look into the code, > I thought we offered more grace than 6 seconds to restart the server. > > Can you share any server settings in your config.xml? > > Please remember that as of 2.x tika-server will shutdown on oom, > timeouts and max_files, and clients should be able to handle waiting > for tika-server restarts. > > On Wed, Mar 8, 2023 at 2:58 PM Konstantin Gribov <[email protected]> wrote: > > > > Hello, Artur. > > > > How many concurrent requests did you have and are you running Tika Server > > on Windows? And what kind of files did you use? > > > > You may have hit number of open files limit due to lot of reasons starting > > from known Windows issue (JVM process holds file descriptors for mmaped > > files until process killed) through just too low nofile limit to some Tika > > bug with handling for example stdin/stdout for forked processes. > > > > Could you provide jvm thread dump and lsof output (or Windows analog)? > > > > -- > > Best regards, > > Konstantin Gribov. > > > > > > On Wed, Mar 8, 2023 at 4:26 PM Artur Auhatov via user > > <[email protected]> wrote: > >> > >> Hello! > >> > >> I have a few questions related to Tika server. > >> > >> > >> > >> We’ve started using tika server in our environment. While testing > >> reliability of the tika-server we found it crashes during fork and can’t > >> fork anymore until restart the main process. Is this known problem? > >> > >> In order to start tika-server we use command: java -jar > >> tika-server-2.7.0.jar -c myconfig.xml > >> > >> > >> > >> There is a log message: > >> > >> > >> > >> 14:50:30,272 [INFO] [Thread-9] - Shutting down forked process with > >> status: HIT_MAX_FILES [org.apache.tika.server.core.ServerStatusWatcher] > >> > >> INFO [pool-2-thread-1] 14:50:30,816 > >> org.apache.tika.server.core.TikaServerWatchDog forked process exited with > >> exit value 2 > >> > >> INFO [pool-2-thread-1] 14:50:36,876 > >> org.apache.tika.server.core.TikaServerWatchDog about to shutdown process > >> > >> ERROR [main] 14:50:36,878 org.apache.tika.server.core.TikaServerCli Can't > >> start: > >> > >> java.util.concurrent.ExecutionException: java.lang.RuntimeException: > >> Forked process failed to start after 6022 (ms) > >> > >> at > >> java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?] > >> > >> at > >> java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?] > >> > >> at > >> org.apache.tika.server.core.TikaServerCli.mainLoop(TikaServerCli.java:121) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerCli.execute(TikaServerCli.java:93) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerCli.main(TikaServerCli.java:80) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> Caused by: java.lang.RuntimeException: Forked process failed to start > >> after 6022 (ms) > >> > >> at > >> org.apache.tika.server.core.TikaServerWatchDog$ForkedProcess.<init>(TikaServerWatchDog.java:316) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerWatchDog$ForkedProcess.<init>(TikaServerWatchDog.java:287) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerWatchDog.startForkedProcess(TikaServerWatchDog.java:224) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerWatchDog.call(TikaServerWatchDog.java:143) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerWatchDog.call(TikaServerWatchDog.java:53) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > >> > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > >> ~[?:?] > >> > >> at > >> java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > >> > >> at > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > >> ~[?:?] > >> > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) > >> ~[?:?] > >> > >> at java.lang.Thread.run(Thread.java:832) ~[?:?] > >> > >> >
