Re: [dspace-tech] Re: pm2 restarts

[Mark H. Wood]

I'm actually relying on PM2 restarts to help manage seemingly
limitless memory growth.  Front end processes were driving us hard
into swap until the OOM killer would harvest Tomcat, then a couple of
minutes later Monit would start it again.  The memory limit restart
seemed to be ineffective, so I added an hourly Cron restart that seems
to keep memory demand in check.  It's rather crude, though.  I think
it isn't waiting for any sort of "quiet point", as I originally
supposed it might.

I'd appreciate any experience with the interplay of the cache settings
and PM2 process memory limits.

Regardless of that, I think that dspace-angular needs some
graceful-shutdown code to close the HTTP listener neatly at the end of
a response.  I'm looking into doing this with 'http-terminator'.  It
should handle automatic restarts by PM2, manual restarts by the
sysadmin for whatever reason, and normal system shutdown.

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/ZOO4ablAJXJ_VbFA%40IUPUI.Edu.


signature.asc
Description: PGP signature

[dspace-tech] Re: pm2 restarts

[DSpace Technical Support]

Hi Wally,

Based on the error, it sounds to me like PM2 is automatically restarting 
anytime a process goes over the " --max-memory-restart" value.  This is a 
flag that can be used to tell PM2 to automatically restart anytime memory 
usage increases.  You may want to look more at setting that flag to a 
reasonable value (it looks like from the error, it might be defaulting to 
8MB? "max_memory_limit=8388608")

That's my immediate guess as to what might be going on.  There are also 
some good guides out there for PM2 usage in general that might be helpful.  
Here's one which describes the "--max-memory-restart" flag in more 
detail:  
https://blog.appsignal.com/2022/03/09/a-complete-guide-to-nodejs-process-management-with-pm2.html

Hopefully others will add advice here if they've run into this same issue 
before.

Tim

On Thursday, August 17, 2023 at 9:18:57 AM UTC-5 Wally Grotophorst wrote:

> When tailing pm2.log I see this happening every 30 seconds or so (this is 
> just an excerpt, one of the  worker processes out of the 10 I have running):
>  
> 2023-08-17T10:16:20: PM2 log: [PM2][WORKER] Process 6 restarted because it 
> exceeds --max-memory-restart value (current_memory=117948416 
> max_memory_limit=8388608 [octets])
> 2023-08-17T10:16:20: PM2 log: App [dspace-ui:6] starting in -cluster mode-
> 2023-08-17T10:16:20: PM2 log: App [dspace-ui:6] online
> 2023-08-17T10:16:21: PM2 log: -reload- New worker listening
> 2023-08-17T10:16:21: PM2 log: Stopping app:dspace-ui id:_old_6
> 2023-08-17T10:16:21: PM2 log: App name:dspace-ui id:_old_6 disconnected
> 2023-08-17T10:16:21: PM2 log: App [dspace-ui:_old_6] exited with code [0] 
> via signal [SIGINT]
> 2023-08-17T10:16:21: PM2 log: pid=82542 msg=process killed
>
> Server has 16GB unused memory.  Here's my dspace-ui.json file:
>
> "apps": [
> {
>"name": "dspace-ui",
>"cwd": "/opt/local/dspace-ui",
>"script": "dist/server/main.js",
>"instances": "max",
>"exec_mode": "cluster",
>"min_uptime": 3,
>"out_file": "/dev/null",
>"env": {
>   "NODE_ENV": "production"
>},
>"node_args": [
>   "--max_old_space_size=6144"
>]
>
> Any suggestions on what might be wrong (I assume that much reloading is 
> wrong...but I am very new to node.js and pm2...
>

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/e02a0910-93f5-47bc-ac88-1e4a90dbe224n%40googlegroups.com.