>I’ll continue testing, if get some issues I’ll let you know. Please do. You might consider using the MockParser (search our codebase) in tika-core-tests.jar to load-test tika-server.
>However, there is an interesting fact. When the hit max files are coming I >stop calls by JMeter and call manually way and carefully to get the max files >number. So, I’ve noticed if I don’t call while restarting tika server process >a value as 6 sec is enough. And in case I continue calling it is not enough. K. Do we need to do anything on our end? You should leave whatever the default is for a restart under load. If that's not enough, bump it up. On Thu, Mar 9, 2023 at 1:34 AM Artur Auhatov via user <user@tika.apache.org> wrote: > > Hi, Tim and Konstantin! > > Thanks for the quick response. > > I’m testing on my local environment, it’s a sort of home load testing to > understand this service is robust or not. I use a bunch of file with > different mime types such as pdf, docx, txt, mp4, pptx, odt, xsls, rtf, html. > > Env: Mac OS 13, Java 15 > > To test restarts I just override a default value of maxFiles and it doesn’t > matter how many file descriptor OS has. > > Tim, yeah you are right, I also override maxForkedStartupMillis. Now I > returned to a default value and restarting works well. > > I’ll continue testing, if get some issues I’ll let you know. > > > > However, there is an interesting fact. When the hit max files are coming I > stop calls by JMeter and call manually way and carefully to get the max files > number. So, I’ve noticed if I don’t call while restarting tika server process > a value as 6 sec is enough. And in case I continue calling it is not enough. > > > > > > From: Tim Allison <talli...@apache.org> > Date: Wednesday, 8 March 2023, 23:30 > To: user@tika.apache.org <user@tika.apache.org> > Subject: [EXTERNAL] - Re: Tika server crashes > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. If you feel that the email is suspicious, please report it > using PhishAlarm. > > > HIT_MAX_FILES is expected. We designed that in to periodically > restart the server to avoid memory leaks in badly behaving parsers. > You can configure a value for the max file threshold if necessary. > > The restart failed, and that's a problem. Let me look into the code, > I thought we offered more grace than 6 seconds to restart the server. > > Can you share any server settings in your config.xml? > > Please remember that as of 2.x tika-server will shutdown on oom, > timeouts and max_files, and clients should be able to handle waiting > for tika-server restarts. > > On Wed, Mar 8, 2023 at 2:58 PM Konstantin Gribov <gros...@gmail.com> wrote: > > > > Hello, Artur. > > > > How many concurrent requests did you have and are you running Tika Server > > on Windows? And what kind of files did you use? > > > > You may have hit number of open files limit due to lot of reasons starting > > from known Windows issue (JVM process holds file descriptors for mmaped > > files until process killed) through just too low nofile limit to some Tika > > bug with handling for example stdin/stdout for forked processes. > > > > Could you provide jvm thread dump and lsof output (or Windows analog)? > > > > -- > > Best regards, > > Konstantin Gribov. > > > > > > On Wed, Mar 8, 2023 at 4:26 PM Artur Auhatov via user > > <user@tika.apache.org> wrote: > >> > >> Hello! > >> > >> I have a few questions related to Tika server. > >> > >> > >> > >> We’ve started using tika server in our environment. While testing > >> reliability of the tika-server we found it crashes during fork and can’t > >> fork anymore until restart the main process. Is this known problem? > >> > >> In order to start tika-server we use command: java -jar > >> tika-server-2.7.0.jar -c myconfig.xml > >> > >> > >> > >> There is a log message: > >> > >> > >> > >> 14:50:30,272 [INFO] [Thread-9] - Shutting down forked process with > >> status: HIT_MAX_FILES [org.apache.tika.server.core.ServerStatusWatcher] > >> > >> INFO [pool-2-thread-1] 14:50:30,816 > >> org.apache.tika.server.core.TikaServerWatchDog forked process exited with > >> exit value 2 > >> > >> INFO [pool-2-thread-1] 14:50:36,876 > >> org.apache.tika.server.core.TikaServerWatchDog about to shutdown process > >> > >> ERROR [main] 14:50:36,878 org.apache.tika.server.core.TikaServerCli Can't > >> start: > >> > >> java.util.concurrent.ExecutionException: java.lang.RuntimeException: > >> Forked process failed to start after 6022 (ms) > >> > >> at > >> java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?] > >> > >> at > >> java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?] > >> > >> at > >> org.apache.tika.server.core.TikaServerCli.mainLoop(TikaServerCli.java:121) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerCli.execute(TikaServerCli.java:93) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerCli.main(TikaServerCli.java:80) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> Caused by: java.lang.RuntimeException: Forked process failed to start > >> after 6022 (ms) > >> > >> at > >> org.apache.tika.server.core.TikaServerWatchDog$ForkedProcess.<init>(TikaServerWatchDog.java:316) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerWatchDog$ForkedProcess.<init>(TikaServerWatchDog.java:287) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerWatchDog.startForkedProcess(TikaServerWatchDog.java:224) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerWatchDog.call(TikaServerWatchDog.java:143) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> org.apache.tika.server.core.TikaServerWatchDog.call(TikaServerWatchDog.java:53) > >> ~[tika-server-standard-2.7.0.jar:2.7.0] > >> > >> at > >> java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > >> > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > >> ~[?:?] > >> > >> at > >> java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > >> > >> at > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > >> ~[?:?] > >> > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) > >> ~[?:?] > >> > >> at java.lang.Thread.run(Thread.java:832) ~[?:?] > >> > >>
