Hi Jay, Did you run: "scalr-server-ctl reconfigure" after upgrade?
Regards, Igor On 7 July 2016 at 14:25, Jay Farschman <[email protected]> wrote: > That did not seem to deliver any better information, but I'm not sure I > did what you intended: > > *CHANGED app/src/Scalr/System/Zmq/Cron/Task/ScalarizrMessaging.php* > $logger->error(sprintf("Cannot handle message '%s' (message_id: %s) " . > "from server '%s' (server_id: %s). %s", > $message->getName(), > $message->messageId, > $dbserver->remoteIp ? $dbserver->remoteIp : '*no-ip*', > $dbserver->serverId, > $e->getMessage() . "*$e->getTraceAsString()*" > )); > > *OUTPUT of php-scalarizr_messaging.log* > Jul 07 14:12:09 -07:00 - scalarizr_messaging@24491 - SERVICE - Client > recieved termination SIGNAL:15 > Jul 07 14:12:19 -07:00 - scalarizr_messaging@12760 - SERVICE - Starting > scalarizr_messaging... > Jul 07 14:12:20 -07:00 - scalarizr_messaging@12760 - SERVICE - Memory > usage: 20.93 MB > Jul 07 14:16:48 -07:00 - > Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23926 - ERROR - Cannot > handle message 'HostInit' (message_id: > c48a0c07-ff9e-4a97-b1e0-8447eaff0ba8) from server '50.112.197.90' > (server_id: 371c501c-61fe-4e81-8f0a-6fa82b83239b). Could not find the > service "environment" in the DI > container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91) > Jul 07 14:16:48 -07:00 - scalarizr_messaging@23926 - SERVICE - Memory > usage: 18.25 MB > Jul 07 14:17:23 -07:00 - > Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23617 - ERROR - Cannot > handle message 'HostInit' (message_id: > 0c45e7b8-cb8a-46d6-80ff-19a57a1af31e) from server '50.112.32.218' > (server_id: 0fc7c35f-6f1e-4d9c-a55b-40e6bbaf4f8e). Could not find the > service "environment" in the DI > container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91) > Jul 07 14:17:23 -07:00 - scalarizr_messaging@23617 - SERVICE - Memory > usage: 19.89 MB > Jul 07 14:17:49 -07:00 - > Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23619 - ERROR - Cannot > handle message 'HostInit' (message_id: > c7198f93-abc4-46bb-8d71-a151d0657112) from server '52.38.110.229' > (server_id: d84d3905-5bc9-4454-93e5-40281d4956b7). Could not find the > service "environment" in the DI > container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91) > Jul 07 14:17:49 -07:00 - scalarizr_messaging@23619 - SERVICE - Memory > usage: 18.65 MB > Jul 07 14:17:55 -07:00 - DBServer@23930 - INFO - [FarmID: 128] Sending > message 'HostInitResponse' via REST to server '52.40.134.120' (server_id: > 283b330e-f52a-4479-b1a7-56328bc5447d) completed > Jul 07 14:17:55 -07:00 - scalarizr_messaging@23930 - SERVICE - Memory > usage: 36.08 MB > Jul 07 14:18:01 -07:00 - scalarizr_messaging@23949 - SERVICE - Memory > usage: 24.59 MB > Jul 07 14:18:08 -07:00 - scalarizr_messaging@23962 - SERVICE - Memory > usage: 25.00 MB > Jul 07 14:18:14 -07:00 - scalarizr_messaging@23717 - SERVICE - Memory > usage: 22.81 MB > Jul 07 14:18:40 -07:00 - DBServer@7367 - INFO - [FarmID: 128] Sending > message 'HostInitResponse' via REST to server '52.37.21.229' (server_id: > f5532f40-b52c-4b40-a059-bec7c93fad44) completed > Jul 07 14:18:40 -07:00 - scalarizr_messaging@7367 - SERVICE - Memory > usage: 33.80 MB > Jul 07 14:18:45 -07:00 - Scalr@12174 - FATAL - Exception thrown in > Scalr::FireEvent(Scalr\Observer\MessagingEventObserver:BeforeHostUp, > /opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:119): > keychain() is not callable or does not exist. > Jul 07 14:18:45 -07:00 - > Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@12174 - ERROR - keychain() > is not callable or does not exist. > Jul 07 14:18:45 -07:00 - scalarizr_messaging@12174 - SERVICE - Memory > usage: 23.85 MB > Jul 07 14:19:11 -07:00 - > Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23926 - ERROR - Cannot > handle message 'HostInit' (message_id: > dc60250e-92e3-4e21-aadf-917922f97eb0) from server '52.42.14.129' > (server_id: 0cb11e57-fce4-44d0-93a6-2da95a6ac8e6). Could not find the > service "environment" in the DI > container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91) > > > > On Thursday, July 7, 2016 at 3:01:29 PM UTC-6, Vitaliy Demidov wrote: >> >> Hi, Jay >> May I ask you to replace "({$e->getFile()}:{$e->getLine()})" with >> $e->getTraceAsString() in >> >> app/src/Scalr/System/Zmq/Cron/Task/ScalarizrMessaging.php >> >> $logger->error(sprintf("Cannot handle message '%s' (message_id: %s) " . >> "from server '%s' (server_id: %s). %s", >> $message->getName(), >> $message->messageId, >> $dbserver->remoteIp ? $dbserver->remoteIp : >> '*no-ip*', >> $dbserver->serverId, >> $e->getMessage() . >> "({$e->getFile()}:{$e->getLine()})" >> )); >> >> Restart zmq_services and provide error messages from the log. It should >> include debug trace that can help a lot. >> >> >> On Thursday, July 7, 2016 at 11:01:12 PM UTC+3, Jay Farschman wrote: >>> >>> I'm gonna take a brief break and walk the dog but I can leave it like >>> this for a few more hours. >>> >>> If you want to do a google hangout, let me know. >>> >>> I'm thinking of forcing the a reinstall of the packages >>> >>> On Thursday, July 7, 2016 at 12:29:57 PM UTC-6, DicsyDel wrote: >>>> >>>> Hi Jay, >>>> >>>> We're looking into this. >>>> >>>> Regards, >>>> Igor >>>> >>>> On 7 July 2016 at 11:20, Jay Farschman <[email protected]> wrote: >>>> >>>>> *ON THE SERVER SIDE* >>>>> >>>>> I dug into the server logs and get the following in >>>>> my php-scalarizr_messaging.log: >>>>> >>>>> Jul 07 11:07:39 -07:00 - >>>>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23619 - ERROR - Cannot >>>>> handle message 'HostInit' (message_id: >>>>> 0d58b6be-d0a1-48af-9532-133bbaad76c1) from server '52.40.180.82' >>>>> (server_id: 567174e9-0d52-4e06-b9ca-675f6be3a4e6). Could not find the >>>>> service "environment" in the DI >>>>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91) >>>>> >>>>> and when I did a grep "Could not find the service" >>>>> php-scalarizr_messaging.log I got a lot of error messages from my >>>>> testing today with different messages that could not be handled (HostInit, >>>>> RebootFinish, WinHostdown, etc.). >>>>> >>>>> On Thursday, July 7, 2016 at 11:32:38 AM UTC-6, Jay Farschman wrote: >>>>>> >>>>>> After upgrading to 5.11.22 (Community Edition), I am unable to spin >>>>>> up new instances. The instances become stuck in various states. For >>>>>> example, if I spin up 3 instances I'll get 2 stuck in "pending" status, >>>>>> and >>>>>> 1 will make it to "Initializing". Very rarely, perhaps one time in 10 and >>>>>> instance will properly complete all of the required steps and show a >>>>>> status >>>>>> of "Running". In all cases the instances start in Amazon Ec2 and I can >>>>>> log >>>>>> in. >>>>>> >>>>>> >>>>>> >>>>>> # scalarizr -v >>>>>> Scalarizr 4.7.14 >>>>>> >>>>>> # ps -ef |grep scal >>>>>> root 1354 1 0 16:30 ? 00:00:00 >>>>>> /opt/scalarizr/embedded/bin/python /usr/bin/scalr-upd-client --daemonize >>>>>> --pid-file /var/run/scalr-upd-client.pid --log-file >>>>>> /var/log/scalarizr_update.log >>>>>> root 1415 1 0 16:30 ? 00:00:11 >>>>>> /opt/scalarizr/embedded/bin/python /usr/bin/scalarizr -c /etc/scalr >>>>>> >>>>>> *ON A SERVER STUCK PENDING* >>>>>> The scalarizr_debug.log hangs at: >>>>>> 2016-07-07 16:29:27,764+00:00 - DEBUG - >>>>>> scalarizr.messaging.p2p.consumer - Waiting message acknowledge event: >>>>>> HostInit >>>>>> >>>>>> *ON A SERVER STUCK INITALIZING* >>>>>> The scalarizr_debug.log gets past this because it received and >>>>>> decodes a 'control' message. It does a number of other things, and then >>>>>> shortly after completing a dpkg-query it starts attempts to start some >>>>>> message handlers and the logs stop. >>>>>> >>>>>> 1167 2016-07-07 16:30:48,102+00:00 - DEBUG - >>>>>> scalarizr.messaging.p2p.consumer - Starting message subhandler thread: >>>>>> BeforeHostUpHandler >>>>>> 1168 2016-07-07 16:30:48,102+00:00 - DEBUG - >>>>>> scalarizr.messaging.p2p.consumer - Waiting message subhandler thread: >>>>>> BeforeHostUpHandler >>>>>> 1169 2016-07-07 16:30:48,102+00:00 - DEBUG - >>>>>> scalarizr.messaging.p2p.consumer - Starting message handler >>>>>> >>>>>> I am struggling a bit with reasons this could happen, but I am >>>>>> leaning toward a load issue causing a timeout. >>>>>> >>>>>> top - 17:29:08 up 130 days, 15:58, 2 users, load average: 2.62, >>>>>> 3.27, 3.31 >>>>>> >>>>>> Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie >>>>>> >>>>>> %Cpu(s): 7.1 us, 5.5 sy, 0.0 ni, 79.4 id, 0.3 wa, 0.0 hi, 0.0 >>>>>> si, 7.7 st >>>>>> >>>>>> KiB Mem: 3854816 total, 3436444 used, 418372 free, 103544 >>>>>> buffers >>>>>> >>>>>> KiB Swap: 0 total, 0 used, 0 free. 213720 cached >>>>>> Mem >>>>>> >>>>>> I'm currently testing with Ubuntu Images loaded with the 4.7.14 >>>>>> agent. I'll try a different operating system and perhaps some different >>>>>> agent versions. >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "scalr-discuss" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- > You received this message because you are subscribed to the Google Groups > "scalr-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scalr-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
