What exactly you did with scale-server-manage? Running scalr-server-ctl reconfigure - is a part of a standard upgrade process. Update itself is a pretty much install a new package and run "scalr-server-ctl reconfigure".
Regards, Igor On 7 July 2016 at 14:54, Jay Farschman <[email protected]> wrote: > No, I use scale-server-manage and restarted just zmq_service there. But > I'm doing a reconfigure now. Always a good idea. > > > On Thursday, July 7, 2016 at 3:29:06 PM UTC-6, DicsyDel wrote: >> >> Hi Jay, >> >> Did you run: "scalr-server-ctl reconfigure" after upgrade? >> >> Regards, >> Igor >> >> On 7 July 2016 at 14:25, Jay Farschman <[email protected]> wrote: >> >>> That did not seem to deliver any better information, but I'm not sure I >>> did what you intended: >>> >>> *CHANGED app/src/Scalr/System/Zmq/Cron/Task/ScalarizrMessaging.php* >>> $logger->error(sprintf("Cannot handle message '%s' (message_id: %s) " . >>> "from server '%s' (server_id: %s). %s", >>> $message->getName(), >>> $message->messageId, >>> $dbserver->remoteIp ? $dbserver->remoteIp : '*no-ip*', >>> $dbserver->serverId, >>> $e->getMessage() . "*$e->getTraceAsString()*" >>> )); >>> >>> *OUTPUT of php-scalarizr_messaging.log* >>> Jul 07 14:12:09 -07:00 - scalarizr_messaging@24491 - SERVICE - Client >>> recieved termination SIGNAL:15 >>> Jul 07 14:12:19 -07:00 - scalarizr_messaging@12760 - SERVICE - Starting >>> scalarizr_messaging... >>> Jul 07 14:12:20 -07:00 - scalarizr_messaging@12760 - SERVICE - Memory >>> usage: 20.93 MB >>> Jul 07 14:16:48 -07:00 - >>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23926 - ERROR - Cannot >>> handle message 'HostInit' (message_id: >>> c48a0c07-ff9e-4a97-b1e0-8447eaff0ba8) from server '50.112.197.90' >>> (server_id: 371c501c-61fe-4e81-8f0a-6fa82b83239b). Could not find the >>> service "environment" in the DI >>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91) >>> Jul 07 14:16:48 -07:00 - scalarizr_messaging@23926 - SERVICE - Memory >>> usage: 18.25 MB >>> Jul 07 14:17:23 -07:00 - >>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23617 - ERROR - Cannot >>> handle message 'HostInit' (message_id: >>> 0c45e7b8-cb8a-46d6-80ff-19a57a1af31e) from server '50.112.32.218' >>> (server_id: 0fc7c35f-6f1e-4d9c-a55b-40e6bbaf4f8e). Could not find the >>> service "environment" in the DI >>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91) >>> Jul 07 14:17:23 -07:00 - scalarizr_messaging@23617 - SERVICE - Memory >>> usage: 19.89 MB >>> Jul 07 14:17:49 -07:00 - >>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23619 - ERROR - Cannot >>> handle message 'HostInit' (message_id: >>> c7198f93-abc4-46bb-8d71-a151d0657112) from server '52.38.110.229' >>> (server_id: d84d3905-5bc9-4454-93e5-40281d4956b7). Could not find the >>> service "environment" in the DI >>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91) >>> Jul 07 14:17:49 -07:00 - scalarizr_messaging@23619 - SERVICE - Memory >>> usage: 18.65 MB >>> Jul 07 14:17:55 -07:00 - DBServer@23930 - INFO - [FarmID: 128] Sending >>> message 'HostInitResponse' via REST to server '52.40.134.120' (server_id: >>> 283b330e-f52a-4479-b1a7-56328bc5447d) completed >>> Jul 07 14:17:55 -07:00 - scalarizr_messaging@23930 - SERVICE - Memory >>> usage: 36.08 MB >>> Jul 07 14:18:01 -07:00 - scalarizr_messaging@23949 - SERVICE - Memory >>> usage: 24.59 MB >>> Jul 07 14:18:08 -07:00 - scalarizr_messaging@23962 - SERVICE - Memory >>> usage: 25.00 MB >>> Jul 07 14:18:14 -07:00 - scalarizr_messaging@23717 - SERVICE - Memory >>> usage: 22.81 MB >>> Jul 07 14:18:40 -07:00 - DBServer@7367 - INFO - [FarmID: 128] Sending >>> message 'HostInitResponse' via REST to server '52.37.21.229' (server_id: >>> f5532f40-b52c-4b40-a059-bec7c93fad44) completed >>> Jul 07 14:18:40 -07:00 - scalarizr_messaging@7367 - SERVICE - Memory >>> usage: 33.80 MB >>> Jul 07 14:18:45 -07:00 - Scalr@12174 - FATAL - Exception thrown in >>> Scalr::FireEvent(Scalr\Observer\MessagingEventObserver:BeforeHostUp, >>> /opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:119): >>> keychain() is not callable or does not exist. >>> Jul 07 14:18:45 -07:00 - >>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@12174 - ERROR - >>> keychain() is not callable or does not exist. >>> Jul 07 14:18:45 -07:00 - scalarizr_messaging@12174 - SERVICE - Memory >>> usage: 23.85 MB >>> Jul 07 14:19:11 -07:00 - >>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23926 - ERROR - Cannot >>> handle message 'HostInit' (message_id: >>> dc60250e-92e3-4e21-aadf-917922f97eb0) from server '52.42.14.129' >>> (server_id: 0cb11e57-fce4-44d0-93a6-2da95a6ac8e6). Could not find the >>> service "environment" in the DI >>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91) >>> >>> >>> >>> On Thursday, July 7, 2016 at 3:01:29 PM UTC-6, Vitaliy Demidov wrote: >>>> >>>> Hi, Jay >>>> May I ask you to replace "({$e->getFile()}:{$e->getLine()})" with >>>> $e->getTraceAsString() in >>>> >>>> app/src/Scalr/System/Zmq/Cron/Task/ScalarizrMessaging.php >>>> >>>> $logger->error(sprintf("Cannot handle message '%s' (message_id: %s) " . >>>> "from server '%s' (server_id: %s). %s", >>>> $message->getName(), >>>> $message->messageId, >>>> $dbserver->remoteIp ? $dbserver->remoteIp : >>>> '*no-ip*', >>>> $dbserver->serverId, >>>> $e->getMessage() . >>>> "({$e->getFile()}:{$e->getLine()})" >>>> )); >>>> >>>> Restart zmq_services and provide error messages from the log. It should >>>> include debug trace that can help a lot. >>>> >>>> >>>> On Thursday, July 7, 2016 at 11:01:12 PM UTC+3, Jay Farschman wrote: >>>>> >>>>> I'm gonna take a brief break and walk the dog but I can leave it like >>>>> this for a few more hours. >>>>> >>>>> If you want to do a google hangout, let me know. >>>>> >>>>> I'm thinking of forcing the a reinstall of the packages >>>>> >>>>> On Thursday, July 7, 2016 at 12:29:57 PM UTC-6, DicsyDel wrote: >>>>>> >>>>>> Hi Jay, >>>>>> >>>>>> We're looking into this. >>>>>> >>>>>> Regards, >>>>>> Igor >>>>>> >>>>>> On 7 July 2016 at 11:20, Jay Farschman <[email protected]> wrote: >>>>>> >>>>>>> *ON THE SERVER SIDE* >>>>>>> >>>>>>> I dug into the server logs and get the following in >>>>>>> my php-scalarizr_messaging.log: >>>>>>> >>>>>>> Jul 07 11:07:39 -07:00 - >>>>>>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23619 - ERROR - >>>>>>> Cannot handle message 'HostInit' (message_id: >>>>>>> 0d58b6be-d0a1-48af-9532-133bbaad76c1) from server '52.40.180.82' >>>>>>> (server_id: 567174e9-0d52-4e06-b9ca-675f6be3a4e6). Could not find the >>>>>>> service "environment" in the DI >>>>>>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91) >>>>>>> >>>>>>> and when I did a grep "Could not find the service" >>>>>>> php-scalarizr_messaging.log I got a lot of error messages from my >>>>>>> testing today with different messages that could not be handled >>>>>>> (HostInit, >>>>>>> RebootFinish, WinHostdown, etc.). >>>>>>> >>>>>>> On Thursday, July 7, 2016 at 11:32:38 AM UTC-6, Jay Farschman wrote: >>>>>>>> >>>>>>>> After upgrading to 5.11.22 (Community Edition), I am unable to spin >>>>>>>> up new instances. The instances become stuck in various states. For >>>>>>>> example, if I spin up 3 instances I'll get 2 stuck in "pending" >>>>>>>> status, and >>>>>>>> 1 will make it to "Initializing". Very rarely, perhaps one time in 10 >>>>>>>> and >>>>>>>> instance will properly complete all of the required steps and show a >>>>>>>> status >>>>>>>> of "Running". In all cases the instances start in Amazon Ec2 and I can >>>>>>>> log >>>>>>>> in. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> # scalarizr -v >>>>>>>> Scalarizr 4.7.14 >>>>>>>> >>>>>>>> # ps -ef |grep scal >>>>>>>> root 1354 1 0 16:30 ? 00:00:00 >>>>>>>> /opt/scalarizr/embedded/bin/python /usr/bin/scalr-upd-client >>>>>>>> --daemonize >>>>>>>> --pid-file /var/run/scalr-upd-client.pid --log-file >>>>>>>> /var/log/scalarizr_update.log >>>>>>>> root 1415 1 0 16:30 ? 00:00:11 >>>>>>>> /opt/scalarizr/embedded/bin/python /usr/bin/scalarizr -c /etc/scalr >>>>>>>> >>>>>>>> *ON A SERVER STUCK PENDING* >>>>>>>> The scalarizr_debug.log hangs at: >>>>>>>> 2016-07-07 16:29:27,764+00:00 - DEBUG - >>>>>>>> scalarizr.messaging.p2p.consumer - Waiting message acknowledge event: >>>>>>>> HostInit >>>>>>>> >>>>>>>> *ON A SERVER STUCK INITALIZING* >>>>>>>> The scalarizr_debug.log gets past this because it received and >>>>>>>> decodes a 'control' message. It does a number of other things, and then >>>>>>>> shortly after completing a dpkg-query it starts attempts to start some >>>>>>>> message handlers and the logs stop. >>>>>>>> >>>>>>>> 1167 2016-07-07 16:30:48,102+00:00 - DEBUG - >>>>>>>> scalarizr.messaging.p2p.consumer - Starting message subhandler thread: >>>>>>>> BeforeHostUpHandler >>>>>>>> 1168 2016-07-07 16:30:48,102+00:00 - DEBUG - >>>>>>>> scalarizr.messaging.p2p.consumer - Waiting message subhandler thread: >>>>>>>> BeforeHostUpHandler >>>>>>>> 1169 2016-07-07 16:30:48,102+00:00 - DEBUG - >>>>>>>> scalarizr.messaging.p2p.consumer - Starting message handler >>>>>>>> >>>>>>>> I am struggling a bit with reasons this could happen, but I am >>>>>>>> leaning toward a load issue causing a timeout. >>>>>>>> >>>>>>>> top - 17:29:08 up 130 days, 15:58, 2 users, load average: 2.62, >>>>>>>> 3.27, 3.31 >>>>>>>> >>>>>>>> Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 >>>>>>>> zombie >>>>>>>> >>>>>>>> %Cpu(s): 7.1 us, 5.5 sy, 0.0 ni, 79.4 id, 0.3 wa, 0.0 hi, >>>>>>>> 0.0 si, 7.7 st >>>>>>>> >>>>>>>> KiB Mem: 3854816 total, 3436444 used, 418372 free, 103544 >>>>>>>> buffers >>>>>>>> >>>>>>>> KiB Swap: 0 total, 0 used, 0 free. 213720 cached >>>>>>>> Mem >>>>>>>> >>>>>>>> I'm currently testing with Ubuntu Images loaded with the 4.7.14 >>>>>>>> agent. I'll try a different operating system and perhaps some >>>>>>>> different >>>>>>>> agent versions. >>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "scalr-discuss" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "scalr-discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "scalr-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scalr-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
