On Sep 11, 2014, at 3:26 PM, James Martin <jmar...@ansible.com> wrote:
> I think we're probably going to move to a system that uses a tier of proxies
> and two ELBs. That way we can update the idle ELB, change out the AMIs, and
> bring the updated ELB up behind an alternate domain for the blue-green
> testing. Then when everything checks out, switch the proxies to the updated
> ELB and take down the remaining, now idle ELB.
>
>
> Not following this exactly -- what's your tier of proxies? You have a group
> of proxies (haproxy, nginx) behind a load balancer that point to your
> application?
Yes, nginx or some other HA-ish thing. If it's nginx then you can maintain a
brochure site even if something horrible happens to the application.
>
> Amazon would suggest using Route53 to point to the new ELB, but there's too
> great a chance of faulty DNS caching breaking a switch to a new ELB. Plus
> there's a 60s TTL to start with regardless, even in the absence of caching.
>
> Quite right. There are some interesting things you can do with tools you
> could run on the hosts that would redirect traffic from blue hosts to the
> green LB, socat being one. After you notice no more traffic coming to blue,
> you can terminate it.
That's an interesting idea, but it fails if people are behind a caching DNS and
they visit after you've terminated the blue traffic but before their caching
DNS lets go of the record.
> You're right, I did miss that. By checking the AMI, you're only updating the
> instance if the AMI changes. If you a checking the launch config, you are
> updating the instances if any component of the launch config has changed --
> AMI, instance type, address type, etc.
That's true, but if I'm changing instance types I'll generally just cycle_all.
Because of the connection draining and parallelism of the instance creation,
it's just as quick to do all of them instead of the ones that needs changing.
That said, it's an obvious optimization for sure.
> Using the ASG to do the provisioning might be preferable if it's reliable. At
> first I went that route, but I was having problems with the ASG's
> provisioning being non-deterministic. Manually creating the instances seems
> to ensure that things happen in a particular order and with predictable
> speed. As mentioned, the manual method definitely works every time, although
> I need to add some more timeout and error checking (like what happens if I
> ask for 3 new instances and only get 2).
>
>
> I didn't have any issues with the ASG doing the provisioning, but I would say
> nothing is predictable with AWS :).
Very true. Over the past few months I've had several working processes just
fail with no warning. The most recent is AWS sometimes refusing to return the
current list of AMIs. Prior to that it was the Available status on an AMI not
really meaning available. Now I check the list of returned AMIs in a loop until
the one I'm looking for shows up, Available status notwithstanding. Very
frustrating. Things could be worse, however: the API could be run by Facebook...
>
> I have a separate task that cleans up the old AMIs and LCs, incidentally. I
> keep the most recent around as a backup for quick rollbacks.
>
> That's cool, care to share?
>
I think I've posted it before, but here's the important bit. After deleting
everything but the oldest backup AMI (determined by naming convention or tags),
delete any LC that doesn't have an associated AMI:
def delete_launch_configs(asg_connection, ec2_connection, module):
changed = False
launch_configs = asg_connection.get_all_launch_configurations()
for config in launch_configs:
image_id = config.image_id
images = ec2_connection.get_all_images(image_ids=[image_id])
if not images:
config.delete()
changed = True
module.exit_json(changed=changed)
-scott
>
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To post to this group, send email to ansible-project@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/623A534E-0D88-41E7-B86C-78B387041EB0%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
