In the end we added a notification to Kubernetes'es auto-scaling group which publishes to SNS topic which in turn pushes to a Lambda function.
The Lambda function uses the notification payload to get a new list of the members of the auto-scaling group and updates ElasticSearch Service with a new list of public IPs. In addition, a Nginx forward-proxy pod which maps http://elasticsearch-logging requests to AWS ElasticSearch Service address was added to the cluster, and that's how Fluentd and Kibana pods didn't have to be changed at all to support the new installation. On Sunday, July 3, 2016 at 9:50:24 PM UTC+3, Rodrigo Campos wrote: > > Glad! Mail again telling what you ended up doing :-) > > On Sun, Jul 03, 2016 at 09:24:54PM +0300, Ori Popowski wrote: > > Yes, they both sound like good solutions. You've been a great help. > Thanks. > > > > On Sun, Jul 3, 2016 at 9:21 PM, Rodrigo Campos <[email protected] > <javascript:>> wrote: > > > > > Oh, you can probably use something like this, then: > > > https://github.com/kopeio/aws-es-proxy ? > > > > > > Yes, you can watch events using the API. But maybe a daemonSet that > > > registers the node (it will be run on every node) it's enough. > > > > > > > > > On Sunday, July 3, 2016, Ori Popowski <[email protected] <javascript:>> > wrote: > > > > > >> Well, it's actually a long story. > > >> > > >> Amazon's ElasticSearch Service does not support VPC, and the only way > to > > >> protect it from being publicly available on the internet is via > > >> white-listing IP addresses. We are trying to figure out a way to tell > new > > >> cluster nodes to register their IP addresses to it (via AWS CLI or > > >> something) > > >> > > >> Is it possible to tell the Master to run some script when a node > joins > > >> the cluster? The Master must have some mechanism to listen on such > events. > > >> Is there a way to hook onto those events maybe? > > >> > > >> Thanks > > >> > > >> On Sun, Jul 3, 2016 at 8:05 PM, Rodrigo Campos <[email protected] > <javascript:>> > > >> wrote: > > >> > > >>> I think there's (currently at least) no other way than modify the > launch > > >>> configuration changing, for example, the "user data" that Amazon > runs that > > >>> on start (check that there is something now, so be sure to not > remove it or > > >>> the node won't join your k8s cluster, I think). > > >>> > > >>> It depends on what you want to do, it might be possible to just use > a > > >>> daemonSet and no change to the launch configuration > > >>> > > >>> On Sunday, July 3, 2016, Ori Popowski <[email protected] > <javascript:>> wrote: > > >>> > > >>>> Hi, > > >>>> > > >>>> Is there a way to automatically run scripts on nodes that are added > to > > >>>> the cluster? > > >>>> > > >>>> More specifically: > > >>>> Incrementing the auto-scaling group in AWS causes the added > instances > > >>>> to be added automatically to the Kubernetes cluster. Is there a way > to tell > > >>>> Kubernetes to run some scripts on the newly added node? > > >>>> > > >>>> Thanks > > >>>> > > >>>> -- > > >>>> You received this message because you are subscribed to the Google > > >>>> Groups "Containers at Google" group. > > >>>> To unsubscribe from this group and stop receiving emails from it, > send > > >>>> an email to [email protected] <javascript:>. > > >>>> To post to this group, send email to [email protected] > <javascript:> > > >>>> . > > >>>> Visit this group at > https://groups.google.com/group/google-containers. > > >>>> For more options, visit https://groups.google.com/d/optout. > > >>>> > > >>> -- > > >>> You received this message because you are subscribed to a topic in > the > > >>> Google Groups "Containers at Google" group. > > >>> To unsubscribe from this topic, visit > > >>> > https://groups.google.com/d/topic/google-containers/t5sB14kOQ0c/unsubscribe > > >>> . > > >>> To unsubscribe from this group and all its topics, send an email to > > >>> [email protected] <javascript:>. > > >>> To post to this group, send email to [email protected] > <javascript:>. > > >>> Visit this group at > https://groups.google.com/group/google-containers. > > >>> For more options, visit https://groups.google.com/d/optout. > > >>> > > >> > > >> -- > > >> You received this message because you are subscribed to the Google > Groups > > >> "Containers at Google" group. > > >> To unsubscribe from this group and stop receiving emails from it, > send an > > >> email to [email protected] <javascript:>. > > >> To post to this group, send email to [email protected] > <javascript:>. > > >> Visit this group at https://groups.google.com/group/google-containers. > > > >> For more options, visit https://groups.google.com/d/optout. > > >> > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "Containers at Google" group. > > > To unsubscribe from this topic, visit > > > > https://groups.google.com/d/topic/google-containers/t5sB14kOQ0c/unsubscribe > > > . > > > To unsubscribe from this group and all its topics, send an email to > > > [email protected] <javascript:>. > > > To post to this group, send email to [email protected] > <javascript:>. > > > Visit this group at https://groups.google.com/group/google-containers. > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Containers at Google" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] <javascript:>. > > To post to this group, send email to [email protected] > <javascript:>. > > Visit this group at https://groups.google.com/group/google-containers. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Containers at Google" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/google-containers. For more options, visit https://groups.google.com/d/optout.
