It appears that this was buggy until recently:
https://github.com/kubernetes/kubernetes/issues/47990
The original feature issue for terminal size handling has some options for
manual workarounds: https://github.com/kubernetes/kubernetes/issues/13585
/MR
PS: ed(1) is the standard editor.
PPS: sad
We encountered this issue too, and tried to counter it by lowering UDP
conntrack timeouts so that these entries expire more quickly. However, at
the time we found that the corresponding sysctls are not propagated into
network namespaces, so we now patch the global defaults in our kernel build
(patc
You can also run jstatd in a running pod and then attach JVisualVM. I
haven't done it myself, but the general procedure is:
- kubectl exec into the pod
- Write the policy file to disk: echo 'grant codebase
"file:${java.home}/../lib/tools.jar" { permission
java.security.AllPermission; };' > all.pol
The sentence you cite describes push-model metrics (such as Telegraf
writing to InfluxDB). For Prometheus, which uses the pull model, yes – a
service is the recommended way to go. Alternatively you can also use the
pod discovery in Prometheus to select pods based on their labels, but it
just moves
grew from 47% to 75%.
> On Tuesday, September 12, 2017 at 12:36:43 PM UTC+2, Matthias Rampke wrote:
> > A rescheduled pod is a new pod, and there is no logical continuity with
> any specific predecessor. Prometheus 1.x was designed with this in mind –
> one of the main motivations f
A rescheduled pod is a new pod, and there is no logical continuity with any
specific predecessor. Prometheus 1.x was designed with this in mind – one
of the main motivations for developing it were the difficulties we
(SoundCloud) had with per-instance time series in Graphite. However, it's
not perf
TL;DR when you set the cluster domain, this should Just Work™ in Kubernetes
1.7+ but not before
David – what Kubernete version are you running? I just went down a rabbit
hole because our /etc/hosts did not look like this code[0] suggests it
should. Turns out, there was a bug before 1.7[1] that cau
This is set via the `--cluster-domain` flag on the kubelet, as well as in
the kubedns deployment.
/MR
On Tue, Sep 5, 2017 at 10:17 PM David Rosenstrauch
wrote:
> On 2017-09-05 5:39 pm, Matthias Rampke wrote:
> > If it's checking the domain suffix, everything should work if y
If it's checking the domain suffix, everything should work if you set the
cluster domain to a subdomain of yours instead of cluster.local – then the
name will be of the form ..pod.., no? We
use this in all our clusters, but we make a custom distribution, so how to
do this will depend on how you bui
There's not really a way to express that dependency. You can make the
second container wait in a wrapper script, or just let it crash and get
restarted until it can successfully start.
/MR
On Fri, Sep 1, 2017, 16:32 Yong Zhang wrote:
> Hi, all
>
> I have one pod with two containers e.g. A and B
mmendation.
>
> On Tue, Jul 25, 2017 at 8:11 AM, Matthias Rampke
> wrote:
> > Hey,
> >
> > I could not find an easy answer for this, and I suppose it's a bit of a
> > niche question – what's the correct ordering to upgrading kubelet /
> > kube-proxy on li
y one would need to do this. I
> was thinking that if an upgrade is needed, I'd just drain the node, kill
> it, then bring up a totally new machine with the latest version of kubelet
> and kube-proxy I want to use. (Assumes I used pre-baked images).
>
> On Tue, Jul 25, 2017 at 8:1
Hey,
I think audit logs[0] are what you need, potentially with some filtering on
top.
/MR
[0]
https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#audit-logs
On Tue, Aug 29, 2017 at 8:20 AM wrote:
> Hi!
>
> I was wondering whether kubernetes provides a way to track the user
> co
Our biggest cluster has approx. 300 nodes, 7000 pods.
/MR
On Mon, Aug 28, 2017 at 3:24 PM wrote:
> How large is your cluster currently?
>
> > One thing that I did not realise initially is that it is absolutely
> vital to be diligent about securing the etcd peer and client communication.
> In a
nts or
> surprises during implementation of this setup?
>
> I'm curious now about the observed performance/stability differences
> between consistent reads on/off. If anyone else has some insights on that
> matter, please do share. Thanks!
>
> Regards,
>
> Mark
>
&
We have this setup, it works well. We've turned on consistent reads from
etcd, not sure if that's strictly necessary.
/MR
On Sun, Aug 27, 2017 at 2:39 PM wrote:
> Sharing my initial thoughts on HA k8s outside the cloud:
>
> https://www.relaxdiego.com/2017/08/hakube.html
>
> --
> You received th
On Fri, Aug 25, 2017 at 7:04 AM wrote:
> Nice!
>
> Do you do the initial cluster setup using kube-up.sh?
> Or do you use a custom set of scripts?
>
It's all custom as we have a bare metal footprint and at the time there was
nothing usable for that.
/MR
--
You received this message because you
We run HA etcd, apiserver and controllers. What has worked well so far:
- keep etcd upgrades separate
-- Follow the etcd docs for that
-- never skip a minor version
-- just rolling upgrade, once the last etcd is upgraded there is no way back
- for Kubernetes upgrades
-- apiserver first
--- be car
Does GKE let you scale the cluster to 0 nodes? I'm not very familiar with
GKE in particular but since it manages the size of the cluster (and
replacing lost nodes) I think you just need to tell it what you want
instead of going behind it's back.
/MR
On Wed, Aug 16, 2017 at 11:42 AM wrote:
> hel
Oh hold on. the *service cluster IP range* is not for pod IPs at all. It's
for the ClusterIP of services, so you can have up to 64k services in a
cluster at the default setting. The range for pods is the --cluster-cidr
flag on kube-controller-manager.
On Fri, Aug 11, 2017 at 3:05 PM David Rosenst
And yes, with the defaults you are limited to 256 *nodes* per *cluster*. If
you're running that large a cluster, I suppose you can be expected to
twiddle some flags :)
On Fri, Aug 11, 2017 at 2:47 PM David Rosenstrauch
wrote:
> Ah. That makes a bit more sense.
>
> Thanks!
>
> DR
>
> On 2017-08-
, Aug 2, 2017, 09:41 Vinoth Narasimhan wrote:
> Thanks Matthias for your reply.
>
> Can we add a "PreStop" hook on the POD before it going to die, to dump the
> heap to the emptyDir. Is this hook will execute before it crash ?
>
> On Wednesday, August 2, 2017 at 1:02:55
Raise your Kubernetes memory limit, or lower the JVM heap size. If the
container gets OOM-killed there is nothing it can do to still dump
something. By lowering the heap size you may be able to provoke an
OutOfMemoryException within the JVM before it gets killed; with the right
flags set it will do
I haven't tried this, but IIRC the required upgrade is to HTTP/2. Try `curl
--http2`?
/MR
On Sat, Jul 29, 2017, 05:31 Rodrigo Campos wrote:
> If it wasn't resolved as it's not in the documentation wiki (sorry, away
> from laptop now) you can probably check how kubectl exec does it. It might
> n
What issues agree these causing? They are returned pretty quickly, can you
just fetch them all and filter in whatever collects them?
A server side filter API could become quite complex trying to cover all
possible ways to filter, while it's rather easy to filter for any
particular use case on the
Hey,
I could not find an easy answer for this, and I suppose it's a bit of a
niche question – what's the correct ordering to upgrading kubelet /
kube-proxy on live nodes?
Usually (except for the upgrade onto 1.6) we just upgrade either of them
and restart, then the other, all the while the node i
ughts are going into this :)
/MR
On Mon, Jul 24, 2017 at 7:46 AM Matthias Rampke wrote:
> No, we haven't tried handling this manually. We use multiple runtimes (Go,
> JVM, Ruby) so the pain would be multiplied. We put monitoring on container
> CPU requests vs. usage but this is far
7;t be
> a problem.
>
> I'd also like to know, as you say, how is it handled on Borg :-)
>
>
> On Saturday, July 22, 2017, Matthias Rampke wrote:
>
>> With a CPU request+limit of 2, I would want /proc to look like the
>> machine only had 2 cores, no matter what i
With a CPU request+limit of 2, I would want /proc to look like the machine
only had 2 cores, no matter what it physically has.
Some applications self-scale according to the resources available (such as
the JVM, or Go via the default GOMAXPROCS).
A problem we have encountered is that even if you l
blem with how it set up iptables, but
> comparing the table between a working and a non-working machine, i find no
> difference.
>
>
> On Thursday, July 20, 2017 at 3:03:54 AM UTC-5, Matthias Rampke wrote:
>
>> Are there any errors in either `kubectl describe pod` or the kub
Are there any errors in either `kubectl describe pod` or the kubelet logs?
Is the kubelet at the version you expect (it logs this on start)? 1.6.2 is
pretty old, have you considered upgrading to 1.6.7?
/MR
On Wed, Jul 19, 2017, 22:02 Jeremy Derr wrote:
> We recently had an instance check failur
On Tue, Jul 18, 2017 at 12:09 AM 'David Oppenheimer' via Kubernetes user
discussion and Q&A wrote:
>
> Why would having the signed certificate cause a problem? Wouldn't you need
> the private key to cause a problem?
>
Yes, me being inexact again. Something that has certificate authentication
*cr
On Sun, Jul 16, 2017 at 7:08 PM 'David Oppenheimer' via Kubernetes user
discussion and Q&A wrote:
>
> Why would using the same CA for the etcd cluster members and the
> Kubernetes components allow "anyone" to attach to the etcd cluster?
>
Sorry, I somewhat oversimplified there. Anyone who can ge
First of all, do you *really* need an HA setup? HA introduces some
consistency, performance and authentication issues that you just don't need
to deal with in a single controller setup.
A Kubernetes API outage will not disrupt applications running in the
cluster. Everything will just run on for th
Philosophically, the problem is what Kubernetes could do with the reclaimed
CPU. The pod could restart at any time, so it can't really promise this CPU
time to a different pod. It can let others use this on a best effort basis,
but that's already the case when you make a request and don't use it fu
If you want the dataset to be available on every host, you could use a
DaemonSet to update it (again via a hostpath) independently of the
individual processes. This is assuming they don't need to be told that it's
new.
Alternatively, you could use init containers and sidecars to do the
updating fr
On Wed, Jun 28, 2017, 01:52 Rodrigo Campos wrote:
>
> I don't know, though, how to change those options for the docker
> daemon. Probably the kubelet might be involved, but never changed
> those myself in kubernetes.
>
The kubelet doesn't start Docker itself, so you'll have to do this wherever
y
t; Thanks for your answer!
> I am using Prometheus to grab metrics from cadvisor, if I just want to
> grab the redis docker container, can you give a example how to apply metric
> relabeling to keep metrics matching redis container? Thank you very much!
>
>
> On Monday, June 26, 201
Just drop the data you need when you collect it. For example, with
Prometheus, you can apply metric relabeling to keep metrics matching any
label combination you specify. If you are using the API fro. custom code,
you have even more freedom.
/MR
On Mon, Jun 26, 2017, 09:14 wangpeng007 wrote:
>
Is this difference a problem for your use case? Kubernetes does do more
work before a pod starts. If you need low-latency execution you'll have to
use long-running worker processes of some form. Once it's started, it
should be just as fast.
On Fri, Jun 2, 2017 at 2:13 PM Diego Lapiduz wrote:
> T
Note that the probe command is executed directly, not in a shell, but you
are using a pipe. try this:
command:
- sh
- -ec
- ps -ef | grep my_process_name
aside from that - don't run multiple processes in a container; use multiple
containers in a pod instead. Make sure that when your process exits
There is no "step by step" guide for this, because this is an unorthodox
use case. I would suggest you familiarise yourself with Kubernetes'
networking model and implementations thereof, so you can make an informed
decision about possible solutions.
/MR
On Thu, Jun 1, 2017 at 9:09 AM wrote:
> C
You can span a cluster over public networks if you use some kind of overlay
network (like flannel), as long as the nodes can talk to the apiserver and
each other.
When a node disappears (stops checking in with the apiserver), it will
eventually be drained and the pods on it deleted; controllers wi
How does actual utilization relate to billing, compared to resource
reservations? Kubernetes schedules by requests (more or less), what should
a customer be billed who reserves N cores but only uses a small fraction?
Or a customer who under-reserves but uses spare capacity? You can also
forbid the
You are adding the annotations in the Service, but checking on the pods.
Add them to the template section in websocket-replication-controller.yaml
to get the pods created from this controller annotated.
/MR
On Wed, May 24, 2017, 16:36 wrote:
> My Kubernetes versions is ::
>
> # kubectl --ve
In our clusters, we have solved this in the following (not elegant, very
roundabout) way:
On each node (as a daemonset) we run a BIND pod that grabs port 53 in the
host network namespace. We needed this anyway because some applications did
not deal well with the latency of going over the network f
One drawback of both NodePort and the cluster internal ClusterIP
loadbalancing (they are essentially the same) is that they are Layer 4 only
and purely probabilistic, so the load distribution between the actual
service pods can only get so good. Using HAProxy as an intermediary, the
distribution be
At SoundCloud, we use multiple environments in one cluster, using
namespaces and different configuration.
We have a weaker notion of environments – there is no global "staging".
Therefore, the "env" dimension is grouped *under* the system (~ app). One
system's staging and production may be another
If I recall correctly, the implicit selector is created from the daemonset
metadata.
A daemonset needs a selector to function, so it knows which pods already
exist. You are probably better off being explicit about it.
/MR
On Tue, Apr 11, 2017, 23:07 Nate Rook wrote:
> So, I'm updating a daemon
The HAProxy adds a hop, so at least in the low-traffic case it won't make
it faster.
We essentially use this setup, with auto-generated HAProxy configurations.
There are two benefits for this: the network load (which is considerable in
our case) is spread over more Kubernetes nodes, and HAProxy ha
r case the probability
of a collision is low enough that I can deal with it manually if need be.
Thank you!
MR
On Wed, Mar 8, 2017 at 8:41 AM Matthias Rampke wrote:
> How does it manage the AWS route table? Will each node only manage its own
> entry, or will they try to remove entrie
t; This is not something we planned a migration for. In general trying to do
> some sort of networking migration on a running Kubernetes cluster is going
> to lead to a bad time.
>
> Adding Casey and Tom in case they have any brilliant ideas.
>
> Brandon
>
> On Mon, Mar 6, 2017 at
Hi,
I am trying to work out how to upgrade a Kubernetes cluster to use the
kube-subnet-mgr from flannel 0.7.0 safely. This is running with the aws-vpc
backend, so flannel itself is only the control plane.
I plan to whole-sale replace the nodes in the process, for unrelated
reasons (this is part o
You can find out everything there is to know about the node from the
Kubernetes API. A simple way to access that may be to include the kubectl
binary in the container, rely on the auth token it gets from the default
ServiceAccount, and script the registration around that.
The use the Downward API[
"DNS not working" is likely a symptom. First, (by `kubectl exec`ing
around), verify that pods on multiple nodes can talk to each other, and
that they can talk to the internet (`ping 8.8.8.8; host google.com
8.8.8.8`).
Also a common problem is that the kubedns pods inherit the DNS settings
from the
I see three containers in this.
Yes, 0.0.0.0: should work if the graphql container binds to all
interfaces. Try it out?
On Mon, Feb 20, 2017, 21:01 Norman Khine wrote:
> Hello, I have the following template file which has 2 containers:
>
>
> containers:
> - name: api
>
EIKHROUHOU <
yasmine.cheikhrou...@enis.tn> wrote:
> Thank you Sir
> I have an other question, how containers in different pod can communicate
> and is it possible to autoscale node?
>
> 2017-02-16 11:59 GMT+01:00 Matthias Rampke :
>
> Hi,
>
> no, containers never co
Hi,
no, containers never come by themselves – they are always in a pod.
However, it is very common to have only one container in a pod.
/MR
On Thu, Feb 16, 2017 at 10:37 AM YASMINE CHEIKHROUHOU <
yasmine.cheikhrou...@enis.tn> wrote:
> Hi everyone :)
> I am new to kubernetes.
> I am an engineer
Several minutes really is quite long.
Is the master side of Kubernetes functional the whole time? (at least etcd,
kube-apiserver, kube-controller-manager?) If that's not the case, the
cluster cannot react to external events properly.
How are the replicas distributed? 3 replicas on 3 nodes should
No, this is not possible. You can either run multiple containers in a
single pod per node, or use multiple daemonsets.
/MR
On Sun, Feb 12, 2017, 09:45 'David Oppenheimer' via Kubernetes user
discussion and Q&A wrote:
> [+kubernetes-users; kubernetes-dev to bcc]
>
>
> On Sun, Feb 12, 2017 at 12:
I second that you should look at helm.
As a slightly more low tech variant, we effectively `kubectl apply -f` the
definition from CI, and let Kubernetes take care of filling in the blanks
and converging the state. This works great with Namespaces, Deployments,
Services and Ingresses.
/MR
On Sat,
Asking as someone with such a replicated master setup – how do quorum reads
(from etcd) interact with this? Would they prevent these stale reads?
On Wed, Feb 1, 2017 at 2:57 PM 'Eric Tune' via Kubernetes user discussion
and Q&A wrote:
> In some configurations, the API servers are replicated, wit
For the Prometheus ecosystem, directly instrument your own app and have
Prometheus scrape that. for redis and nginx add the relevant exporters* to
the pod, let them connect to the service over localhost, and scrape that.
They don't need to be in the same container, the pod concept is there for
exac
Is the `--service-account-private-key-file` flag set (correctly) on
kube-controller-manager? It needs to match `--service-account-key-file` on
kube-apiserver if set, or `--tls-private-key-file` otherwise.
/MR
On Thu, Dec 22, 2016 at 12:05 PM Christopher Stelly
wrote:
> Hi,
>
> When I create a s
As an alternative approach to "401 OK", what would you need to make
authenticated probe requests?
On Fri, Dec 16, 2016, 20:09 Adam Daughterson
wrote:
> The microservice is entirely set up to prevent access without auth, and I
> don't have control over that. In our use-case, 401 is perfect becaus
No, it doesn't. We `ulimit -n 10240` in our start script and it works just
fine.
Where do you set the raised limit? Apparently upstart ignores limits.conf
and you need to set this in your job definition:
http://bryanmarty.com/2012/02/10/setting-nofile-limit-upstart/
/MR
On Wed, Nov 30, 2016 at
*Jignesh*
>
> *+91 80 44291600 <+91%2080%204429%201600>*
>
> [image: image003]
>
>
>
> *From:* Matthias Rampke [mailto:m...@soundcloud.com]
> *Sent:* Wednesday, November 16, 2016 2:22 PM
> *To:* Kubernetes user discussion and Q&A <
> kubernetes-user
There is nothing special that you need to do. If you use an LB, you
probably want to enable sticky sessions of some sort[0]. We just stick all
the master IPs in one DNS record and call it a day, that's good enough for
us.
/MR
[0] for HAProxy:
http://blog.haproxy.com/2012/03/29/load-balancing-af
To debug, you can `kubectl exec` into a kubedns pod and inspect the
/etc/resolv.conf it got.
On Tue, Oct 18, 2016, 01:52 'Tim Hockin' via Kubernetes user discussion and
Q&A wrote:
> kube-dns should pass-thru to whatever nameserver(s) are configured in
> the node's resolv.conf
>
> On Mon, Oct 17,
I want something like this too, for the following use case:
We create one namespace for each system (a collection of components) and
environment (staging/production/green/blue). Now, the meaning of
environment is specific to the system. Some systems create new ephemeral
environments for each branc
On Sat, Oct 8, 2016, 00:15 ravi prasad l r wrote:
> Matthias, how are disabling swap for the containers ?
>
We disabled and removed all swap on the nodes, and changed new installs not
to include any.
Kubelet seems to set MemorySwap (libcontainer config) to -1 by default here:
>
> https://gith
On Thu, Oct 6, 2016 at 5:06 PM Vinoth Narasimhan
wrote:
> Later if we want to add some more address in it means, we update the
> configmap.
>
How often does that happen? Keep in mind that Kubernetes makes changing a
deployment *easy*, and just having all the pods replaced is the baseline
design.
ere. This is for a non-production cluster
> (basically our staging/dev cluster), and we just want to keep costs low by
> using smaller instance sizes.
>
> Kind regards,
> June
>
> On Thu., 6 Oct. 2016, 7:45 pm Matthias Rampke, wrote:
>
> This is aside from the actual qu
This is aside from the actual question (I'm not very familiar with GCE) but:
Be very careful with swap and containers. We found that having swap
available changes the behaviour when a container hits its memory limit –
container memory was being paged out even though the host had free memory;
we ac
On 1.3.4, each of these tables has a last entry that does not use "mode
random" at all (effectively accomplishing the same as the 1.0 probability):
Chain KUBE-SVC-P5LYC7UHSUKOR4AD (1 references)
target prot opt source destination
KUBE-SEP-5FITWMPKBFFPPBLJ all -- anywhere
Hi,
has anyone found a way to make pod IPs in AWS directly reachable from a
network that is connected to a VPC via VPN (or DirectConnect)?
What we want to do:
* run Kubernetes clusters in a few AWS VPCs
* the VPCs are connected back to our datacenter via VPN or DirectConnect
* we want to reach t
Hi,
this only applies to the DNS pod (technically, any pod with `dnsPolicy:
Default`), but since it does the resolution for all the other pods they
actually inherit this.
So, first you either set the nameserver you want in /etc/resolv.conf or use
a different resolv.conf using the flag. Then resta
rs+unsubscr...@googlegroups.com.
>>> To post to this group, send email to kubernetes-users@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>
--
Matthias Rampke
Engineer
SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany | +49 173
6395215
Managing Director: Alexander Ljung | Incorporated in England & Wales with
Company No. 6343600 | Local Branch Office | AG Charlottenburg | HRB 110657B
--
You received this message bec
nfigure a readiness probe, which you should).
Kubernetes is very much geared towards making this rolling-replace model of
deployment easy and correct, although you can of course do anything you
want.
/MR
--
Matthias Rampke
Engineer
SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, German
80 matches
Mail list logo
