I've created a brand new instance, installed Scalr, and migrated the scalr-server settings to the new instance using the existing database. I'm going to let this run for a day and see if the issue still occurs.
On Tue, Apr 21, 2015 at 3:59 PM, Dan Thagard <[email protected]> wrote: > That AZ is currently in place. I stopped getting that warning after I > removed and readded the CloudStack credentials last week. > > On Tue, Apr 21, 2015 at 2:30 PM, Thomas Orozco <[email protected]> wrote: > >> Hey Dan, >> >> The issue appears to be caused by an AZ that is missing in your >> CloudStack cloud (QQHQ). Is this an AZ that you used to have and have since >> removed? >> >> I'll let you know next steps based on your answer. >> >> Cheers, >> >> -- Thomas | Product Manager @ Scalr | [email protected] | www.scalr.com | >> blog.scalr.com >> >> On Tue, Apr 21, 2015 at 3:01 PM, Dan Thagard <[email protected]> wrote: >> >>> I sent you a copy of all of the log files. >>> >>> On Tue, Apr 21, 2015 at 4:27 AM, Thomas Orozco <[email protected]> wrote: >>> >>>> Dan >>>> >>>> Can you check */opt/scalr-server/var/log/service/php-cloud_poller.log* for >>>> errors / warnings? Alternatively, can you send that along to me ( >>>> [email protected])? >>>> >>>> Cheers, >>>> >>>> -- Thomas | Product Manager @ Scalr | [email protected] | www.scalr.com >>>> | blog.scalr.com >>>> >>>> On Mon, Apr 20, 2015 at 8:16 PM, Daniel Thagard <[email protected]> >>>> wrote: >>>> >>>>> Same issue as before. It works for a spell and that reverts back to >>>>> invalid. I double checked the IAM role in AWS, and modified it to match >>>>> what you update in March--same issue: >>>>> >>>>> { >>>>> "Version": "2012-10-17", >>>>> "Statement": [ >>>>> { >>>>> "Action": "iam:GetUser", >>>>> "Resource": "*", >>>>> "Effect": "Allow" >>>>> }, >>>>> { >>>>> "Action": "iam:ListInstanceProfiles", >>>>> "Resource": "*", >>>>> "Effect": "Allow" >>>>> }, >>>>> { >>>>> "Action": "iam:ListServerCertificates", >>>>> "Resource": "*", >>>>> "Effect": "Allow" >>>>> }, >>>>> { >>>>> "Action": "iam:PassRole", >>>>> "Effect": "Allow", >>>>> "Resource": "*" >>>>> }, >>>>> { >>>>> "NotAction": "iam:*", >>>>> "Resource": "*", >>>>> "Effect": "Allow" >>>>> } >>>>> ] >>>>> } >>>>> >>>>> >>>>> On Mon, Apr 20, 2015 at 1:29 PM, Thomas Orozco <[email protected]> >>>>> wrote: >>>>> >>>>>> Dan, >>>>>> >>>>>> Can you: >>>>>> >>>>>> 1. Restart Scalr (*scalr-server-manage restart all*). >>>>>> 2. Confirm that your Environment is not reported as "Disabled" when >>>>>> you navigate to the Environments page. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> -- Thomas | Product Manager @ Scalr | [email protected] | >>>>>> www.scalr.com | blog.scalr.com >>>>>> >>>>>> On Mon, Apr 20, 2015 at 3:50 PM, Daniel Thagard <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Yes, these are the same credentials we were using previously. I've >>>>>>> also tried creating a new set of access keys to no avail. I would think >>>>>>> that if the credentials weren't working at all, then I wouldn't be able >>>>>>> to >>>>>>> enumerate the AWS VPC, security groups, etc. Also, the farm launch >>>>>>> wouldn't >>>>>>> work either. >>>>>>> >>>>>>> It appears to be working briefly but some process is knocking the >>>>>>> credentials off and declaring them invalid erroneously. >>>>>>> >>>>>>> On Mon, Apr 20, 2015 at 9:00 AM, Thomas Orozco <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Daniel, >>>>>>>> >>>>>>>> Have you confirmed the credentials are indeed valid / have all the >>>>>>>> permissions Scalr needs? >>>>>>>> >>>>>>>> This error typically happens if the background workers fail to >>>>>>>> access your environment and therefore flag the credentials as not >>>>>>>> working. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> -- Thomas | Product Manager @ Scalr | [email protected] | >>>>>>>> www.scalr.com | blog.scalr.com >>>>>>>> >>>>>>>> On Mon, Apr 20, 2015 at 2:58 PM, Daniel Thagard < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Well, maybe I spoke too soon. Every few minutes, I get the banner >>>>>>>>> warning that "This environment's AWS access credentials have been >>>>>>>>> revoked, and Scalr is no longer able to manage any of its >>>>>>>>> infrastructure. >>>>>>>>> Please click here >>>>>>>>> <http://scalr.qqsolutions.local/#/account/environments/2/clouds?platform=ec2> >>>>>>>>> to >>>>>>>>> update environment with new and functional credentials". If I >>>>>>>>> remove and readd the credentials, then everything works for a short >>>>>>>>> time >>>>>>>>> period before reverting back to this error state, e.g. I can create a >>>>>>>>> new >>>>>>>>> farm, add roles, and launch the farm successfully in AWS. >>>>>>>>> >>>>>>>>> Any suggestions for places to look? Also, the global variables and >>>>>>>>> SSH keys appear to be working as intended. >>>>>>>>> >>>>>>>>> On Fri, Apr 17, 2015 at 12:08 PM, Thomas Orozco <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Glad to hear it's working now. >>>>>>>>>> >>>>>>>>>> You might want to double check that other things that are stored >>>>>>>>>> encrypted (Global Variables, SSH Keys) are looking correct. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> >>>>>>>>>> -- Thomas | Product Manager @ Scalr | [email protected] | >>>>>>>>>> www.scalr.com | blog.scalr.com >>>>>>>>>> >>>>>>>>>> On Fri, Apr 17, 2015 at 5:36 PM, Dan Thagard <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Success! Thanks very much for your help. As a side note, after I >>>>>>>>>>> ran the reconfigure, the web UI came back and said that my AWS >>>>>>>>>>> credentials >>>>>>>>>>> were invalid. However, when I went to edit them, I was greeted with >>>>>>>>>>> a blank >>>>>>>>>>> form (no form fields). I dropped the records from the >>>>>>>>>>> client_environment_properties for that farm ID, and when I >>>>>>>>>>> revisited the >>>>>>>>>>> page I was shown the empty forms fields to complete. >>>>>>>>>>> >>>>>>>>>>> All appears to be working now. Thanks again for your assistance. >>>>>>>>>>> >>>>>>>>>>> On Fri, Apr 17, 2015 at 11:23 AM, Thomas Orozco < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Can you check /opt/scalr-server/var/log/php-errors.log? >>>>>>>>>>>> >>>>>>>>>>>> Additionally: did you run scalr-server-ctl reconfigure after >>>>>>>>>>>> restoring the DB? There's probably some pending DB migrations that >>>>>>>>>>>> must be >>>>>>>>>>>> applied (and running that command will do it for you). In doubt, >>>>>>>>>>>> re-run it. >>>>>>>>>>>> Or run: >>>>>>>>>>>> >>>>>>>>>>>> cd /opt/scalr-server/embedded/scalr/app/bin ; >>>>>>>>>>>> /opt/scalr-server/embedded/bin/php upgrade.php >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> >>>>>>>>>>>> -- Thomas | Product Manager @ Scalr | [email protected] | >>>>>>>>>>>> www.scalr.com | blog.scalr.com >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Apr 17, 2015 at 5:18 PM, Dan Thagard < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Ok, I was able to restore the scalr and analytics database >>>>>>>>>>>>> from a backup, and I removed/readded the cloud keys. I'm now able >>>>>>>>>>>>> to add >>>>>>>>>>>>> new farms to the environments, but when I try to manage an >>>>>>>>>>>>> existing farm, I >>>>>>>>>>>>> get the following error in the web UI: "Database error" >>>>>>>>>>>>> >>>>>>>>>>>>> Any suggestions? >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Apr 17, 2015 at 9:33 AM, Thomas Orozco < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> If you updated the credentials when the old key was in place, >>>>>>>>>>>>>> you might be in a situation where some data is encrypted with >>>>>>>>>>>>>> the old key, >>>>>>>>>>>>>> and other data is encrypted with the new one... >>>>>>>>>>>>>> >>>>>>>>>>>>>> If you still have your DB backup, best approach might be to >>>>>>>>>>>>>> restore "from scratch". If that's not an option, then you might >>>>>>>>>>>>>> want to try >>>>>>>>>>>>>> re-entering cloud credentials. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- Thomas | Product Manager @ Scalr | [email protected] | >>>>>>>>>>>>>> www.scalr.com | blog.scalr.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Apr 17, 2015 at 3:30 PM, Dan Thagard < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I did not, but rerunning the command I still get the same >>>>>>>>>>>>>>> error. I see that in the scalr-server-ctl reconfigure the >>>>>>>>>>>>>>> errant secret key >>>>>>>>>>>>>>> was replaced with the old one. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Apr 17, 2015 at 9:21 AM, Thomas Orozco < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Did you run *scalr-server-ctl reconfigure*? Changes you >>>>>>>>>>>>>>>> make to the configuration files don't take effect until you >>>>>>>>>>>>>>>> run the command. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- Thomas | Product Manager @ Scalr | [email protected] | >>>>>>>>>>>>>>>> www.scalr.com | blog.scalr.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Apr 17, 2015 at 3:20 PM, Dan Thagard < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I had not copied over the old secret_key, *whoops*. I have >>>>>>>>>>>>>>>>> now updated the scalr-server-secrets.json with the old key >>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>> /opt/scalr/current/app/etc/.cryptokey, but I'm still getting >>>>>>>>>>>>>>>>> the same >>>>>>>>>>>>>>>>> results. I also performed a "scalr-server-manage restart all" >>>>>>>>>>>>>>>>> to no avail. >>>>>>>>>>>>>>>>> The admin password and id had previously been copied over. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Apr 17, 2015 at 8:59 AM, Thomas Orozco < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Dan, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Have you followed these instructions: >>>>>>>>>>>>>>>>>> https://scalr-wiki.atlassian.net/wiki/x/SQBJAQ? >>>>>>>>>>>>>>>>>> Specifically, did you copy over the old secret key to the >>>>>>>>>>>>>>>>>> new secrets file? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- Thomas | Product Manager @ Scalr | [email protected] | >>>>>>>>>>>>>>>>>> www.scalr.com | blog.scalr.com >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Apr 17, 2015 at 2:55 PM, Dan Thagard < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I can list the farms but cannot configure them. I can >>>>>>>>>>>>>>>>>>> add new environments, but I can't configure or add any new >>>>>>>>>>>>>>>>>>> farms to those >>>>>>>>>>>>>>>>>>> either. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Thu, Apr 16, 2015 at 11:06 PM, Daniele Testa < >>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi! >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Could you please tell me what steps you took to upgrade >>>>>>>>>>>>>>>>>>>> your installation? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Are you able to list your current farms, but you get >>>>>>>>>>>>>>>>>>>> that error message only when choosing "configure"? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> You should start by viewing all the different Scalr >>>>>>>>>>>>>>>>>>>> logs (/opt/scalr-server/var/log/*). >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>> Daniele >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Friday, April 17, 2015 at 5:05:54 AM UTC+8, Dan >>>>>>>>>>>>>>>>>>>> Thagard wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> After upgrading to 5.3 OSS, I'm no longer able to >>>>>>>>>>>>>>>>>>>>> manage the farms. I get the error "Cannot proceed with >>>>>>>>>>>>>>>>>>>>> your request. Please >>>>>>>>>>>>>>>>>>>>> try again later." when trying to configure the farms. >>>>>>>>>>>>>>>>>>>>> Also, I can't add any >>>>>>>>>>>>>>>>>>>>> new farms. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Any suggestions as to where I should check for errors? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> You received this message because you are subscribed to >>>>>>>>>>>>>>>>>>>> the Google Groups "scalr-discuss" group. >>>>>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving >>>>>>>>>>>>>>>>>>>> emails from it, send an email to >>>>>>>>>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> You received this message because you are subscribed to >>>>>>>>>>>>>>>>>>> the Google Groups "scalr-discuss" group. >>>>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>>>>>> from it, send an email to >>>>>>>>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> You received this message because you are subscribed to >>>>>>>>>>>>>>>>>> the Google Groups "scalr-discuss" group. >>>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>>>>> from it, send an email to >>>>>>>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> You received this message because you are subscribed to >>>>>>>>>>>>>>>>> the Google Groups "scalr-discuss" group. >>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>>>> from it, send an email to >>>>>>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>>> Google Groups "scalr-discuss" group. >>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>>> from it, send an email to >>>>>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>> Google Groups "scalr-discuss" group. >>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>> from it, send an email to >>>>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> Google Groups "scalr-discuss" group. >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>>> it, send an email to >>>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "scalr-discuss" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to >>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "scalr-discuss" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected] >>>>>>>>>>>> . >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "scalr-discuss" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "scalr-discuss" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "scalr-discuss" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "scalr-discuss" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "scalr-discuss" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "scalr-discuss" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "scalr-discuss" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "scalr-discuss" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "scalr-discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "scalr-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "scalr-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
