Re: [kubernetes-users] Cluster DNS: bottleneck with ~1000 outbound connections per second

Host instead of the confusingly named Default, but it seemed challenging enough that I never got around to it. Evan On Tue, Mar 20, 2018 at 1:55 AM, wrote: > On Thursday, October 5, 2017 at 1:29:28 PM UTC-7, Evan Jones wrote: > > The sustained 1000 qps comes from an application ma

[kubernetes-users] Re: Connecting to Cloud SQL from container app on kubernetes built via google container builder.

I recommend using the Google Cloud SQL proxy container so you don't need to mess with IP whitelists. I don't quite get what you mean about "manual work to edit my pod deployment file": You just need to copy and paste this "sidecar" definition into your .yaml and leave it there. We have been usin

Re: [kubernetes-users] Cluster DNS: bottleneck with ~1000 outbound connections per second

ing a Go DNS query generator, which was able to do 8 DNS queries per second, so dnsmasq does not appear to be the limit. Thanks! Evan On Thu, Oct 5, 2017 at 5:26 PM, Rodrigo Campos wrote: > On Thu, Oct 05, 2017 at 04:29:21PM -0400, Evan Jones wrote: > > The sustained 1000 qps

Re: [kubernetes-users] Cluster DNS: bottleneck with ~1000 outbound connections per second

o tweak timeouts, which users can tweak. > > Sustained 1000 QPS DNS seems artificial. > > On Thu, Oct 5, 2017 at 10:47 AM, Evan Jones > wrote: > > TL;DR: Kubernetes dnsPolicy: ClusterFirst can become a bottleneck with a > > high rate of outbound connections. It seems li

[kubernetes-users] Cluster DNS: bottleneck with ~1000 outbound connections per second

*TL;DR*: Kubernetes dnsPolicy: ClusterFirst can become a bottleneck with a high rate of outbound connections. It seems like the problem is filling the nf_conntrack table, causing client applications to fail to do DNS lookups. I resolved this problem by switching my application to dnsPolicy: Defa

Re: [kubernetes-users] Need some guidance/help: howto diagnose an oomkill

Its been a while since I've dealt with this sort of issue, but there are various libraries that use "native" memory outside the Java heap. The -Xmx flag only limits the Java heap, so it isn't surprising that some processes may need a way higher container memory limit than the Java GC heap limit.

Re: [kubernetes-users] Getting Google's Cloud SQL Proxy to work

:20 AM, Traiano Welcome wrote: > > > On Thursday, 22 June 2017 18:16:22 UTC+4, Evan Jones wrote: >> >> The Cloud SQL Proxy logs suggest to me that it may not be using the right >> credentials? It is possible that it is trying to use the cluster's "default

Re: [kubernetes-users] Getting Google's Cloud SQL Proxy to work

, 2017 at 9:46:52 AM UTC-4, Traiano Welcome wrote: > > Hi Evan > > > On Thu, Jun 22, 2017 at 5:34 PM, Evan Jones > wrote: > >> I know nothing about wordpress, but for what it is worth, we are using >> this Cloud SQL Proxy container with success. A few notes about th

Re: [kubernetes-users] Getting Google's Cloud SQL Proxy to work

I know nothing about wordpress, but for what it is worth, we are using this Cloud SQL Proxy container with success. A few notes about the config you posted: * I'm assuming that where you have "-instances=[INSTANCE_CONNECTION_NAME]=tcp:[PORT]" you've replaced this with your Cloud SQL instance

Re: [kubernetes-users] Authentication on GKE

On the cluster details page on https://console.cloud.google.com/kubernetes , if you have upgraded to 1.6 (I think?), you should see the following drop down to edit an existing cluster. I haven't yet attempted this personally:

[kubernetes-users] Re: Google Compute Engine is unusable, repeated "Failed: Create VM"

A friend of mine ran into something that sounds suspiciously similar to this. I don't recall the details about it, but he did tweet about it: https://twitter.com/nicksantos/status/86997848164864 I seem to recall after the free trial expired, they literally had to delete everything and recre

[kubernetes-users] Re: Help me understand Kubernetes/Google LB options and architectures

This won't directly help answer your questions, since I don't know the answers. However, I found this talk about Kubernetes networking to be extremely helpful to understand the basics. Whenever I'm running into weirdness I end up reviewing it: https://www.youtube.com/watch?v=y2bhV81MfKQ Hopeful

Re: [kubernetes-users] Finding a way to get stable public IP for outbound connections

s > this is also true on an AWS managed equivalent like CoreOS's CloudFormation > scripts. > > On Wed, May 3, 2017 at 8:52 AM, Evan Jones > wrote: > >> As Rodrigo described, we are using Container Engine. I haven't fully >> tested this yet, but my plan is to assig

Re: [kubernetes-users] Finding a way to get stable public IP for outbound connections

As Rodrigo described, we are using Container Engine. I haven't fully tested this yet, but my plan is to assign "dedicated IPs" to a set of nodes, probably in their own Node Pool as part of the cluster. Those are the IPs used by outbound connections from pods running those nodes, if I recalling

Re: [kubernetes-users] Finding a way to get stable public IP for outbound connections

Thank you! I had forgotten about that feature, since we previously have not needed it. That will absolutely solve our problem, and be much better than needing an "exceptional" thing outside of Kubernetes. You are correct about what we need: We have a small number of services where their outboun

Re: [kubernetes-users] Finding a way to get stable public IP for outbound connections

It turns out I've just run into a requirement to have a stable outbound IP address as well. In looking into this: I think we will likely some kind of proxy server running outside of Kubernetes. This will allow services "opt in" to this special handling, rather than doing it for everything in the