No, I use scale-server-manage and restarted just zmq_service there. But
I'm doing a reconfigure now. Always a good idea.
On Thursday, July 7, 2016 at 3:29:06 PM UTC-6, DicsyDel wrote:
>
> Hi Jay,
>
> Did you run: "scalr-server-ctl reconfigure" after upgrade?
>
> Regards,
> Igor
>
> On 7 July 2016 at 14:25, Jay Farschman <jfars...@gmail.com <javascript:>>
> wrote:
>
>> That did not seem to deliver any better information, but I'm not sure I
>> did what you intended:
>>
>> *CHANGED app/src/Scalr/System/Zmq/Cron/Task/ScalarizrMessaging.php*
>> $logger->error(sprintf("Cannot handle message '%s' (message_id: %s) " .
>> "from server '%s' (server_id: %s). %s",
>> $message->getName(),
>> $message->messageId,
>> $dbserver->remoteIp ? $dbserver->remoteIp : '*no-ip*',
>> $dbserver->serverId,
>> $e->getMessage() . "*$e->getTraceAsString()*"
>> ));
>>
>> *OUTPUT of php-scalarizr_messaging.log*
>> Jul 07 14:12:09 -07:00 - scalarizr_messaging@24491 - SERVICE - Client
>> recieved termination SIGNAL:15
>> Jul 07 14:12:19 -07:00 - scalarizr_messaging@12760 - SERVICE - Starting
>> scalarizr_messaging...
>> Jul 07 14:12:20 -07:00 - scalarizr_messaging@12760 - SERVICE - Memory
>> usage: 20.93 MB
>> Jul 07 14:16:48 -07:00 -
>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23926 - ERROR - Cannot handle
>> message 'HostInit' (message_id: c48a0c07-ff9e-4a97-b1e0-8447eaff0ba8) from
>> server '50.112.197.90' (server_id: 371c501c-61fe-4e81-8f0a-6fa82b83239b).
>> Could not find the service "environment" in the DI
>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91)
>> Jul 07 14:16:48 -07:00 - scalarizr_messaging@23926 - SERVICE - Memory
>> usage: 18.25 MB
>> Jul 07 14:17:23 -07:00 -
>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23617 - ERROR - Cannot handle
>> message 'HostInit' (message_id: 0c45e7b8-cb8a-46d6-80ff-19a57a1af31e) from
>> server '50.112.32.218' (server_id: 0fc7c35f-6f1e-4d9c-a55b-40e6bbaf4f8e).
>> Could not find the service "environment" in the DI
>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91)
>> Jul 07 14:17:23 -07:00 - scalarizr_messaging@23617 - SERVICE - Memory
>> usage: 19.89 MB
>> Jul 07 14:17:49 -07:00 -
>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23619 - ERROR - Cannot handle
>> message 'HostInit' (message_id: c7198f93-abc4-46bb-8d71-a151d0657112) from
>> server '52.38.110.229' (server_id: d84d3905-5bc9-4454-93e5-40281d4956b7).
>> Could not find the service "environment" in the DI
>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91)
>> Jul 07 14:17:49 -07:00 - scalarizr_messaging@23619 - SERVICE - Memory
>> usage: 18.65 MB
>> Jul 07 14:17:55 -07:00 - DBServer@23930 - INFO - [FarmID: 128] Sending
>> message 'HostInitResponse' via REST to server '52.40.134.120' (server_id:
>> 283b330e-f52a-4479-b1a7-56328bc5447d) completed
>> Jul 07 14:17:55 -07:00 - scalarizr_messaging@23930 - SERVICE - Memory
>> usage: 36.08 MB
>> Jul 07 14:18:01 -07:00 - scalarizr_messaging@23949 - SERVICE - Memory
>> usage: 24.59 MB
>> Jul 07 14:18:08 -07:00 - scalarizr_messaging@23962 - SERVICE - Memory
>> usage: 25.00 MB
>> Jul 07 14:18:14 -07:00 - scalarizr_messaging@23717 - SERVICE - Memory
>> usage: 22.81 MB
>> Jul 07 14:18:40 -07:00 - DBServer@7367 - INFO - [FarmID: 128] Sending
>> message 'HostInitResponse' via REST to server '52.37.21.229' (server_id:
>> f5532f40-b52c-4b40-a059-bec7c93fad44) completed
>> Jul 07 14:18:40 -07:00 - scalarizr_messaging@7367 - SERVICE - Memory
>> usage: 33.80 MB
>> Jul 07 14:18:45 -07:00 - Scalr@12174 - FATAL - Exception thrown in
>> Scalr::FireEvent(Scalr\Observer\MessagingEventObserver:BeforeHostUp,
>> /opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:119):
>>
>> keychain() is not callable or does not exist.
>> Jul 07 14:18:45 -07:00 -
>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@12174 - ERROR - keychain() is
>> not callable or does not exist.
>> Jul 07 14:18:45 -07:00 - scalarizr_messaging@12174 - SERVICE - Memory
>> usage: 23.85 MB
>> Jul 07 14:19:11 -07:00 -
>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23926 - ERROR - Cannot handle
>> message 'HostInit' (message_id: dc60250e-92e3-4e21-aadf-917922f97eb0) from
>> server '52.42.14.129' (server_id: 0cb11e57-fce4-44d0-93a6-2da95a6ac8e6).
>> Could not find the service "environment" in the DI
>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91)
>>
>>
>>
>> On Thursday, July 7, 2016 at 3:01:29 PM UTC-6, Vitaliy Demidov wrote:
>>>
>>> Hi, Jay
>>> May I ask you to replace "({$e->getFile()}:{$e->getLine()})" with
>>> $e->getTraceAsString() in
>>>
>>> app/src/Scalr/System/Zmq/Cron/Task/ScalarizrMessaging.php
>>>
>>> $logger->error(sprintf("Cannot handle message '%s' (message_id: %s) " .
>>> "from server '%s' (server_id: %s). %s",
>>> $message->getName(),
>>> $message->messageId,
>>> $dbserver->remoteIp ? $dbserver->remoteIp :
>>> '*no-ip*',
>>> $dbserver->serverId,
>>> $e->getMessage() .
>>> "({$e->getFile()}:{$e->getLine()})"
>>> ));
>>>
>>> Restart zmq_services and provide error messages from the log. It should
>>> include debug trace that can help a lot.
>>>
>>>
>>> On Thursday, July 7, 2016 at 11:01:12 PM UTC+3, Jay Farschman wrote:
>>>>
>>>> I'm gonna take a brief break and walk the dog but I can leave it like
>>>> this for a few more hours.
>>>>
>>>> If you want to do a google hangout, let me know.
>>>>
>>>> I'm thinking of forcing the a reinstall of the packages
>>>>
>>>> On Thursday, July 7, 2016 at 12:29:57 PM UTC-6, DicsyDel wrote:
>>>>>
>>>>> Hi Jay,
>>>>>
>>>>> We're looking into this.
>>>>>
>>>>> Regards,
>>>>> Igor
>>>>>
>>>>> On 7 July 2016 at 11:20, Jay Farschman <jfars...@gmail.com> wrote:
>>>>>
>>>>>> *ON THE SERVER SIDE*
>>>>>>
>>>>>> I dug into the server logs and get the following in
>>>>>> my php-scalarizr_messaging.log:
>>>>>>
>>>>>> Jul 07 11:07:39 -07:00 -
>>>>>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23619 - ERROR - Cannot
>>>>>> handle
>>>>>> message 'HostInit' (message_id: 0d58b6be-d0a1-48af-9532-133bbaad76c1)
>>>>>> from
>>>>>> server '52.40.180.82' (server_id: 567174e9-0d52-4e06-b9ca-675f6be3a4e6).
>>>>>> Could not find the service "environment" in the DI
>>>>>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91)
>>>>>>
>>>>>> and when I did a grep "Could not find the service"
>>>>>> php-scalarizr_messaging.log I got a lot of error messages from my
>>>>>> testing today with different messages that could not be handled
>>>>>> (HostInit,
>>>>>> RebootFinish, WinHostdown, etc.).
>>>>>>
>>>>>> On Thursday, July 7, 2016 at 11:32:38 AM UTC-6, Jay Farschman wrote:
>>>>>>>
>>>>>>> After upgrading to 5.11.22 (Community Edition), I am unable to spin
>>>>>>> up new instances. The instances become stuck in various states. For
>>>>>>> example, if I spin up 3 instances I'll get 2 stuck in "pending" status,
>>>>>>> and
>>>>>>> 1 will make it to "Initializing". Very rarely, perhaps one time in 10
>>>>>>> and
>>>>>>> instance will properly complete all of the required steps and show a
>>>>>>> status
>>>>>>> of "Running". In all cases the instances start in Amazon Ec2 and I can
>>>>>>> log
>>>>>>> in.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # scalarizr -v
>>>>>>> Scalarizr 4.7.14
>>>>>>>
>>>>>>> # ps -ef |grep scal
>>>>>>> root 1354 1 0 16:30 ? 00:00:00
>>>>>>> /opt/scalarizr/embedded/bin/python /usr/bin/scalr-upd-client
>>>>>>> --daemonize
>>>>>>> --pid-file /var/run/scalr-upd-client.pid --log-file
>>>>>>> /var/log/scalarizr_update.log
>>>>>>> root 1415 1 0 16:30 ? 00:00:11
>>>>>>> /opt/scalarizr/embedded/bin/python /usr/bin/scalarizr -c /etc/scalr
>>>>>>>
>>>>>>> *ON A SERVER STUCK PENDING*
>>>>>>> The scalarizr_debug.log hangs at:
>>>>>>> 2016-07-07 16:29:27,764+00:00 - DEBUG -
>>>>>>> scalarizr.messaging.p2p.consumer - Waiting message acknowledge event:
>>>>>>> HostInit
>>>>>>>
>>>>>>> *ON A SERVER STUCK INITALIZING*
>>>>>>> The scalarizr_debug.log gets past this because it received and
>>>>>>> decodes a 'control' message. It does a number of other things, and then
>>>>>>> shortly after completing a dpkg-query it starts attempts to start some
>>>>>>> message handlers and the logs stop.
>>>>>>>
>>>>>>> 1167 2016-07-07 16:30:48,102+00:00 - DEBUG -
>>>>>>> scalarizr.messaging.p2p.consumer - Starting message subhandler thread:
>>>>>>> BeforeHostUpHandler
>>>>>>> 1168 2016-07-07 16:30:48,102+00:00 - DEBUG -
>>>>>>> scalarizr.messaging.p2p.consumer - Waiting message subhandler thread:
>>>>>>> BeforeHostUpHandler
>>>>>>> 1169 2016-07-07 16:30:48,102+00:00 - DEBUG -
>>>>>>> scalarizr.messaging.p2p.consumer - Starting message handler
>>>>>>>
>>>>>>> I am struggling a bit with reasons this could happen, but I am
>>>>>>> leaning toward a load issue causing a timeout.
>>>>>>>
>>>>>>> top - 17:29:08 up 130 days, 15:58, 2 users, load average: 2.62,
>>>>>>> 3.27, 3.31
>>>>>>>
>>>>>>> Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie
>>>>>>>
>>>>>>> %Cpu(s): 7.1 us, 5.5 sy, 0.0 ni, 79.4 id, 0.3 wa, 0.0 hi, 0.0
>>>>>>> si, 7.7 st
>>>>>>>
>>>>>>> KiB Mem: 3854816 total, 3436444 used, 418372 free, 103544
>>>>>>> buffers
>>>>>>>
>>>>>>> KiB Swap: 0 total, 0 used, 0 free. 213720 cached
>>>>>>> Mem
>>>>>>>
>>>>>>> I'm currently testing with Ubuntu Images loaded with the 4.7.14
>>>>>>> agent. I'll try a different operating system and perhaps some
>>>>>>> different
>>>>>>> agent versions.
>>>>>>>
>>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "scalr-discuss" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to scalr-discus...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "scalr-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to scalr-discus...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"scalr-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to scalr-discuss+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
