Hi Bogdan-Andrei, This does look to be frozen in this state too. I guess GDB is the next step.
Andrew On Wed, 6 Oct 2021 at 17:12, Bogdan-Andrei Iancu <bog...@opensips.org> wrote: > Hi Andrew, > > What you can do is, after getting the FIFO blocked, to do a trap > "opensips-cli trap" to try to see what the FIFO process is doing (or trying > to do). If the trap does not work, let me know, I will give you the > instructions on how to directly attache with GDB to the process. > > Best regards, > > Bogdan-Andrei Iancu > > OpenSIPS Founder and Developer > https://www.opensips-solutions.com > OpenSIPS eBootcamp 2021 > https://opensips.org/training/OpenSIPS_eBootcamp_2021/ > > On 10/6/21 12:24 AM, Andrew Yager wrote: > > Also restarting opensips produces this: > > Restarting opensips (via systemctl): opensips.serviceERROR: communication > exception for 'which' returned: cannot access fifo file /tmp/opensips_fifo: > [Errno 32] Broken pipe! > > Andrew > > > On Wed, 6 Oct 2021 at 08:23, Andrew Yager <and...@rwts.com.au> wrote: > >> Hi, >> >> Just replicated this on the 3.2.2 nightly build. Debug logs are enabled, >> and this is all I can see on the last request that stalled: >> >> Oct 6 04:49:32 hvprxy osips[1186130]: DBG:mi_fifo:mi_fifo_callback: >> running command [{"jsonrpc": "2.0", "id": "5319", "method": "which", >> "params": []}] >> Oct 6 04:49:32 hvprxy osips[1186130]: DBG:mi_fifo:mi_fifo_callback: got >> mi response = [0x558865cc9560] >> Oct 6 04:49:32 hvprxy osips[1186130]: DBG:mi_fifo:mi_fifo_callback: >> running command [{"jsonrpc": "2.0", "id": "8377", "method": >> "get_statistics", "params": [["all"]]}] >> Oct 6 04:49:32 hvprxy osips[1186133]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186134]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186135]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186136]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186137]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186138]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186139]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186140]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186141]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186142]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186143]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186144]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186145]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186146]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186147]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186148]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186149]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186150]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186151]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186152]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186153]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186154]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186155]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186156]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186157]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186158]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186159]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186160]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186161]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186162]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186163]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186164]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186165]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186166]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186167]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186168]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186169]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186170]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186171]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186172]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186173]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> Oct 6 04:49:32 hvprxy osips[1186174]: DBG:core:ipc_handle_job: received >> job type 0[RPC] from process 1 >> >> Andrew >> >> On Mon, 4 Oct 2021 at 22:37, Andrew Yager <and...@rwts.com.au> wrote: >> >>> Just further to this, I think it's the ul_dump command that seems to >>> cause the issue first. >>> >>> Andrew >>> >>> >>> On Sat, 2 Oct 2021 at 13:05, Andrew Yager <and...@rwts.com.au> wrote: >>> >>>> Hi, >>>> >>>> Not entirely sure where to start digging on this one. On 3.1.3 we've >>>> had an issue appear "suddenly" whereby our mi command output seems to >>>> "block" and not return any useful data. Restarting opensips processes >>>> restores comms. >>>> >>>> We end up with a huge number of opensips_fifo_reply_\* files in the >>>> /tmp directory, but effectively no data is ever written and opensips-cli >>>> freezes. >>>> >>>> We've restarted twice to "resolve" now, but the issue appears to >>>> reoccur. We're using the mi get_statistics command to monitor stats/health >>>> and uldump to pull some detail about usrloc data every 5 minutes. >>>> >>>> I don't think I can cause a debug level increase because the MI process >>>> doesn't seem to be communicating (at least it doesn't seem to work). >>>> >>>> opensips 3.1.3-1 >>>> opensips-cli 0.1~20210707~572d2db- >>>> 5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 >>>> x86_64 x86_64 GNU/Linux >>>> >>>> I can restart again, but if this is going to keep happening, I want to >>>> make sure I have something useful in the logs to be able to trace down. >>>> >>>> Andrew >>>> >>> > _______________________________________________ > Users mailing > listUsers@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users > > >
_______________________________________________ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users