Hi, Jay
May I ask you to replace "({$e->getFile()}:{$e->getLine()})" with
$e->getTraceAsString() in
app/src/Scalr/System/Zmq/Cron/Task/ScalarizrMessaging.php
$logger->error(sprintf("Cannot handle message '%s' (message_id: %s) " .
"from server '%s' (server_id: %s). %s",
$message->getName(),
$message->messageId,
$dbserver->remoteIp ? $dbserver->remoteIp :
'*no-ip*',
$dbserver->serverId,
$e->getMessage() .
"({$e->getFile()}:{$e->getLine()})"
));
Restart zmq_services and provide error messages from the log. It should
include debug trace that can help a lot.
On Thursday, July 7, 2016 at 11:01:12 PM UTC+3, Jay Farschman wrote:
>
> I'm gonna take a brief break and walk the dog but I can leave it like this
> for a few more hours.
>
> If you want to do a google hangout, let me know.
>
> I'm thinking of forcing the a reinstall of the packages
>
> On Thursday, July 7, 2016 at 12:29:57 PM UTC-6, DicsyDel wrote:
>>
>> Hi Jay,
>>
>> We're looking into this.
>>
>> Regards,
>> Igor
>>
>> On 7 July 2016 at 11:20, Jay Farschman <[email protected]> wrote:
>>
>>> *ON THE SERVER SIDE*
>>>
>>> I dug into the server logs and get the following in
>>> my php-scalarizr_messaging.log:
>>>
>>> Jul 07 11:07:39 -07:00 -
>>> Scalr\System\Zmq\Cron\Task\ScalarizrMessaging@23619 - ERROR - Cannot handle
>>> message 'HostInit' (message_id: 0d58b6be-d0a1-48af-9532-133bbaad76c1) from
>>> server '52.40.180.82' (server_id: 567174e9-0d52-4e06-b9ca-675f6be3a4e6).
>>> Could not find the service "environment" in the DI
>>> container.(/opt/scalr-server/embedded/scalr/app/src/Scalr/DependencyInjection/BaseContainer.php:91)
>>>
>>> and when I did a grep "Could not find the service"
>>> php-scalarizr_messaging.log I got a lot of error messages from my
>>> testing today with different messages that could not be handled (HostInit,
>>> RebootFinish, WinHostdown, etc.).
>>>
>>> On Thursday, July 7, 2016 at 11:32:38 AM UTC-6, Jay Farschman wrote:
>>>>
>>>> After upgrading to 5.11.22 (Community Edition), I am unable to spin up
>>>> new instances. The instances become stuck in various states. For example,
>>>> if I spin up 3 instances I'll get 2 stuck in "pending" status, and 1 will
>>>> make it to "Initializing". Very rarely, perhaps one time in 10 and
>>>> instance
>>>> will properly complete all of the required steps and show a status of
>>>> "Running". In all cases the instances start in Amazon Ec2 and I can log in.
>>>>
>>>>
>>>>
>>>> # scalarizr -v
>>>> Scalarizr 4.7.14
>>>>
>>>> # ps -ef |grep scal
>>>> root 1354 1 0 16:30 ? 00:00:00
>>>> /opt/scalarizr/embedded/bin/python /usr/bin/scalr-upd-client --daemonize
>>>> --pid-file /var/run/scalr-upd-client.pid --log-file
>>>> /var/log/scalarizr_update.log
>>>> root 1415 1 0 16:30 ? 00:00:11
>>>> /opt/scalarizr/embedded/bin/python /usr/bin/scalarizr -c /etc/scalr
>>>>
>>>> *ON A SERVER STUCK PENDING*
>>>> The scalarizr_debug.log hangs at:
>>>> 2016-07-07 16:29:27,764+00:00 - DEBUG -
>>>> scalarizr.messaging.p2p.consumer - Waiting message acknowledge event:
>>>> HostInit
>>>>
>>>> *ON A SERVER STUCK INITALIZING*
>>>> The scalarizr_debug.log gets past this because it received and decodes
>>>> a 'control' message. It does a number of other things, and then shortly
>>>> after completing a dpkg-query it starts attempts to start some message
>>>> handlers and the logs stop.
>>>>
>>>> 1167 2016-07-07 16:30:48,102+00:00 - DEBUG -
>>>> scalarizr.messaging.p2p.consumer - Starting message subhandler thread:
>>>> BeforeHostUpHandler
>>>> 1168 2016-07-07 16:30:48,102+00:00 - DEBUG -
>>>> scalarizr.messaging.p2p.consumer - Waiting message subhandler thread:
>>>> BeforeHostUpHandler
>>>> 1169 2016-07-07 16:30:48,102+00:00 - DEBUG -
>>>> scalarizr.messaging.p2p.consumer - Starting message handler
>>>>
>>>> I am struggling a bit with reasons this could happen, but I am leaning
>>>> toward a load issue causing a timeout.
>>>>
>>>> top - 17:29:08 up 130 days, 15:58, 2 users, load average: 2.62, 3.27,
>>>> 3.31
>>>>
>>>> Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie
>>>>
>>>> %Cpu(s): 7.1 us, 5.5 sy, 0.0 ni, 79.4 id, 0.3 wa, 0.0 hi, 0.0 si,
>>>> 7.7 st
>>>>
>>>> KiB Mem: 3854816 total, 3436444 used, 418372 free, 103544
>>>> buffers
>>>>
>>>> KiB Swap: 0 total, 0 used, 0 free. 213720 cached
>>>> Mem
>>>>
>>>> I'm currently testing with Ubuntu Images loaded with the 4.7.14 agent.
>>>> I'll try a different operating system and perhaps some different agent
>>>> versions.
>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "scalr-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"scalr-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
