Re: oc -w timeout

[Philippe Lafoucrière]

Hi,

You might want to take a look at this thread:
https://lists.openshift.redhat.com/openshift-archives/users/2017-June/msg00135.html
​
Cheers
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

Andrew, YOU MADE MY DAY!

The issue is GONE (sorry, I'm very excited and relieved at the same time).
We tried to nuke docker completely on the node (also removing
/var/lib/docker), but we hadn't removed /var/lib/origin/.

So, for an obscur reason, we had a lot of old volumes from May, June and
July.
After removing these folders, our deploys now take less than 5s (time to
run the deploy pod + actually starting the services). We havent seen our
cluster running like that since a long time.

For the record, here's the command we've been using on all nodes:

find /var/lib/origin/openshift.local.volumes/pods/ -type d -maxdepth 1
-mtime +30 -exec rm -rf \{\} \;

It tooks more than 30s on some nodes, so I suspect some folders to be
completely full of sh...

Anyway, that's a relief, thanks again for your pugnacity :)
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

On Tue, Jul 25, 2017 at 12:21 PM, Andrew Lau  wrote:

> I think your issue may come from https://github.com/
> kubernetes/kubernetes/issues/38498
>
> Too many orphaned volumes causing the timeout. I guess the downgrade
> doesn't help with the increased number of volumes(?)
>

That's a good path to follow, indeed.
We tried to downgrade, with the same result (timeout).

I've been looking on a node, and we're seeing volumes from May in
/var/lib/origin/openshift.local.volumes/pods/

These folders have ~5000 items, and we have ~300 per nodes. So maybe we're
hitting a limit somewhere.

I'll try to purge docker completely, and this folder too to see if it helps.
I'll keep you updated, and thanks again for pointing us to this.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

I'm pretty sure it's not related, but I took a look at the git log from the
day we started to have issues, and noticed it was the first time we were
using env vars set from secrets (env[x].valueFrom.secretKeyRef).
Maybe it will ring a bell for someone.
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

We have tried to rollback the master to 1.4, and it worked for a few
moments.
And now, again, we can't deploy anything unless we restart origin-node for
every deploy.

I guess now the solution will be to restart origin-node every 5 minutes on
all nodes to make sure deploy are not blocked :(
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

And the problem occurs on all our nodes btw.
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: oc rsh or oc get pod -w disconnection after few minutes

[Philippe Lafoucrière]

We have achieved a lot of tests, and the connection is dropped somewhere in
Openshift, not by the firewall.

As we don't have any proxy, except haproxy.

We've seen
https://docs.openshift.com/container-platform/3.3/install_config/router/default_haproxy_router.html#preventing-connection-failures-during-restarts

Could it be related?
​
We're seeing a disconnection of `oc get events -w` after exactly 30s, which
is exactly the reload time of haproxy.

thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

Could it be related to this?
https://github.com/openshift/origin/issues/11016
​
Sounds definitely like our issue, I just don't understand why would we hit
this suddenly.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

On the master, we're seeing this on a regular basis:
https://gist.github.com/gravis/cae52e763cd5cdac19a8456f9208aa34

I don't know if it can be related
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

Our nodes are up-to-date already, but we're not using docker-latest (1.13).
I don't think that's an issue, since everything was fine with 1.12 last
week.
​
The only thing having changed lately are PVs, we are migrating some
datastores. I wonder if one of them could be an issue, and openshift is
waiting for a volume until timeout.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

Hi,

We have this issue on Openshift 1.5 (with 1.4 nodes because of this crazy
bug https://github.com/openshift/origin/issues/14092).
It started a few days ago, and nothing really changed in our cluster. We
just added a bunch of secrets, and noticed longer and longer deploys.

We have nothing fancy in the logs, and the only relevent event is :

Unable to mount volumes for pod "xx": timeout expired waiting for
volumes to attach/mount for pod ""/"". list of unattached/unmounted
volumes=[xxx-secrets -secrets -secrets ssl-certs -secrets
default-token-n6pbo]
​
We have this event several times (it varies, let's say around 5 times),
then the container starts as expected. It's an issue when it comes to
single DB pod, the application is down for 5 minutes if the pod needs to
restart.

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

And... it's starting again.
Pods are getting stuck because volumes (secrets) can't be mounted, then
after a few minutes, everything starts.
I really don't get it :(
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

After a lot of tests, we discovered the pending pods were always on the
same node.
There were some (usual) "thin: Deletion of thin device" messages.
After draining the node, nuking /var/lib/docker, a hard reboot, everything
went back to normal.

I suspect devicemapper to be the source of all our troubles, and we'll
certainly try overlayfs instead when 3.6 will be ready.
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

timeout expired waiting for volumes to attach/mount for pod

[Philippe Lafoucrière]

Hi,

Since a few days, we have pods waiting for volumes to be mounted, and get
stuck for several minutes.

https://www.dropbox.com/s/9vuge2t9llr7u6h/Screenshot%202017-07-11%2011.29.19.png?dl=0

After 3-10 minutes, the pod eventually starts, with no obvious reason. Any
idea what could cause this?

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: oc whoami bug?

[Philippe Lafoucrière]

Just to be clear, my point is: if `oc whoami` returns "error: You must be
logged in to the server (the server has asked for the client to provide
credentials)", `oc whoami -t` should return the same if the session has
timed out ;)​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: oc whoami bug?

[Philippe Lafoucrière]

On Mon, Jun 19, 2017 at 4:56 PM, Louis Santillan 
wrote:

> The default user for any request is `system:anonymous` a user is not
> logged in or a valid token is not found.  Depending on your cluster, this
> usually has almost no access (less than `system:authenticated`).  Maybe an
> RFE is order (oc could suggest logging in if request is unsuccessful and
> the found user happens to be `system:anonymous`).


That's what I suspect, but when I'm logged, I expect the token to be mine.
In this particular case, the session had expired, and nothing warned that
the issued token was for `system:anonymous` instead of me.

Thanks,
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

oc whoami bug?

[Philippe Lafoucrière]

Hi,

I think I have hit a bug (or a lack of warning message) with `oc whoami
-t`: I tried to login on our registry, and only got "unauthorized:
authentication required" responses. After a couple of tries, I launched `oc
whoami` without -t:
"error: You must be logged in to the server (the server has asked for the
client to provide credentials)"
The server was probably returning a token for an anonymous user, but this
is a bit disturbing :)
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Cronjob: Failed to pull image "...": unauthorized: authentication required

[Philippe Lafoucrière]

Answering to myself:

https://github.com/openshift/origin/issues/13161
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Cronjob: Failed to pull image "...": unauthorized: authentication required

[Philippe Lafoucrière]

ps: the image is an imageStream in the same namespace.​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Cronjob: Failed to pull image "...": unauthorized: authentication required

[Philippe Lafoucrière]

Hi,

I created a cronJob (
https://docs.openshift.com/container-platform/3.5/dev_guide/cron_jobs.html)
in a project on openshift 1.5, and the job fails to start with:

Failed to pull image "...": unauthorized: authentication required

Error syncing pod, skipping: failed to "StartContainer" for "..." with
ErrImagePull: "unauthorized: authentication required"

Back-off pulling image "..."

Any idea why this is not working out of the box?

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Possible bug with haproxy?

[Philippe Lafoucrière]

On Fri, May 26, 2017 at 4:24 PM, Clayton Coleman 
wrote:

> HAProxy doesn't currently support HTTP/2 connections - so unless you've
> done something custom, you shouldn't even be able to connect to HAProxy as
> http/2


That's maybe the issue then. Since A.domain is a passthrough route, http/2
is supported (by our backend).

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Possible bug with haproxy?

[Philippe Lafoucrière]

Hi, could you take a look at this please:
https://stackoverflow.com/questions/44162263/request-cached-when-using-http-2/44163462

I wonder if the problem could come from haproxy?
We're using the images "openshift/origin-haproxy-router:v1.5.0"

Thanks,
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Pods has connectivity to other pod and service only when I run an additional pod

[Philippe Lafoucrière]

Do you know if it's possible to run 1.4 nodes with 1.5 masters?
We need to start rolling back, we have too many issues with our clients :(

Thanks
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Pods has connectivity to other pod and service only when I run an additional pod

[Philippe Lafoucrière]

@Clayton, it sounds like a critical bug here, other people are seeing this
issue too.
Thanks,
Philippe
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Passthrough and insecure route

[Philippe Lafoucrière]

Hmm, tested with OS 1.4.1:

Route "https" is invalid: spec.tls.insecureEdgeTerminationPolicy: Invalid
value: "Redirect": InsecureEdgeTerminationPolicy is only allowed for
edge-terminated routes

but the doc (
https://docs.openshift.com/container-platform/3.4/architecture/core_concepts/routes.html)
is telling the same as you:

"passthrough routes can also have an insecureEdgeTerminationPolicy the only
valid values are None or empty (for disabled) or Redirect."

​Any idea? :(

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Passthrough and insecure route

[Philippe Lafoucrière]

On Wed, Apr 19, 2017 at 2:35 PM, Clayton Coleman 
wrote:

> 1.4 added the ability to specify insecureEdgeTerminationPolicy for
> passthrough


Good, thanks :)
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Passthrough and insecure route

[Philippe Lafoucrière]

Hi,

It seem impossible to register a route with both "passthrough" tls
termination and standard http.
The "insecure" route is being rejected because the https route already
exposes the same host.
Is there a reason for that?
I'm using passthrough because the underlying service has its own ssl certs
and routing, but that should not block a route to hit another non-https
port.
Please note I'm using OS 1.3

Thanks,
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Volumes with subpath

[Philippe Lafoucrière]

ps : we're using NFS in our test, but it should not matter.​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Volumes with subpath

[Philippe Lafoucrière]

Hi,

We're testing the new k8s SubPath for volumes, and it's not working as what
we had in mind.

We'd like to have something like:

volumeMounts:

  - name: app

mountPath: /var/data

subPath: /data

  - name: app

mountPath: /opt/data2

subPath: /data2

But it seems like the PV is only mountable once?
Is this on purpose?


Thanks
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Let's encrypt routers

[Philippe Lafoucrière]

Thanks for your efforts and investment Tomas, I'm sure it will benefit to a
large number of users!​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Let's encrypt routers

[Philippe Lafoucrière]

Hi,

Is there any plan to support automatic Let's Encrypt ssl cert creation /
renewal in Openshift?
Web servers like caddy (https://caddyserver.com/docs/automatic-https) are
supporting this natively.
(Caddy has, like all modern web servers, proxy/reverseproxy/loadbalancing
capabilities https://caddyserver.com/docs/proxy).

Thanks,
Philippe

-- 
Philippe Lafoucrière - CEO
http://www.tech-angels.com
https://gemnasium.com
France : +33 (0) 3 65 96 02 92
Canada: +1 (418) 478-1175
USA: +1 (954) 607-7443
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: S2I and git-lfs

[Philippe Lafoucrière]

On Mon, Jan 16, 2017 at 12:40 PM, Jonathan Yu  wrote:

> What about git-annex? https://apps.fedoraproject.org/packages/git-annex
>

Unfortunately, we don't manage what our clients are using. LFS is now
supported directly in Github, Gitlab, and Bitbucket (and probably others),
which makes the best candidate for this feature.

Thanks,
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: S2I and git-lfs

[Philippe Lafoucrière]

@Clayton, does it sound reasonable to you to ask for LFS support in s2i?
If yes, where should this ticket go?
https://github.com/openshift/source-to-image ?

Thanks

​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: S2I and git-lfs

[Philippe Lafoucrière]

On Thu, Jan 12, 2017 at 3:50 PM, Ben Parees  wrote:

> ​it's part of the origin/sti-builder infrastructure image.  if git-lfs
> requires client side logic, it's not going to work with openshift s2i
> builds i'm afraid.  (at least not w/o some hackery like doing the git clone
> of the source in your s2i builder's assemble script, instead of letting
> openshift do it.  then your s2i builder image could have git-lfs installed)


There's no "logic" per se. The plugin just needs to be installed, and git
will clone the repo as usual, with no extra command to run.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: S2I and git-lfs

[Philippe Lafoucrière]

On Thu, Jan 12, 2017 at 3:43 PM, Ben Parees  wrote:

> ​no known issues but no known uses either.  s2i is basically just git
> cloning your repo, it sounds like the lfs pointers didn't resolve properly
> during the clone operation.  I don't know the inner workings of how that is
> supposed to happen.  If you git clone the repo yourself, does the correct
> binary content get cloned to your local dir?  Are any additional args
> required during the clone process to make it work?
>

If I clone the repo, I have the .war files with their expected content, but
I have git-lfs installed on my machine (from https://git-lfs.github.com/).
I wonder where the git used by s2i is coming from, I need to read (again)
the docs.

Thanks,
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: ceph pv

[Philippe Lafoucrière]

On Mon, Jan 9, 2017 at 3:42 AM, James Eckersall 
wrote:

> Our use case would be utilisation of openshift clusters with untrusted
> clients in distinct projects, so we’re trying to ensure they can’t access
> each/others storage.


We are in the same situation, and we generally let our clients access their
projects without permissions for secrets :)
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Link to other files in yaml configuration

[Philippe Lafoucrière]

On Mon, Jan 9, 2017 at 3:18 PM, Ben Parees  wrote:

> or references to secrets from within the DC?


Sorry, I meant routes, I mixed up two messages.
So no, I don't think I can use a secret in a route :)

Going with a template is probably the only way to achieve this.

Thanks!
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Link to other files in yaml configuration

[Philippe Lafoucrière]

Hi,

Does anyone know if I can use external files in a yaml file?
I'd like to keep out the certificate files (crt, CA and key) from my
DeploymentConfig yaml file.

Thanks!
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Can't push images after 1.3.0 upgrade

[Philippe Lafoucrière]

Hmm, I got it working by removing the :443 in our repo url (while this port
was needed before to be able to push...)
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Can't push images after 1.3.0 upgrade

[Philippe Lafoucrière]

It's really the images layers patching which is blocking. I can pull an
image from the registry, and push it again with no error:

The push refers to a repository
[our-registry:443/projectname/theimagestream]
38731c91ef63: Layer already exists
7d7e09f222b3: Layer already exists
latest: digest:
sha256:fa1f5a94b89552b2c9d370c7ff779d658e09547eae00d931a3b1f2502f1f7260
size: 3339



Cordialement,
Philippe Lafoucrière

-- 
Philippe Lafoucrière - CEO
http://www.tech-angels.com
https://gemnasium.com
France : +33 (0) 3 65 96 02 92
Canada: +1 (418) 478-1175
USA: +1 (954) 607-7443



On Thu, Jan 5, 2017 at 8:49 PM, Philippe Lafoucrière <
philippe.lafoucri...@tech-angels.com> wrote:

> I'm digging this up.
> We just upgraded our production cluster to OS 1.3, and having this issue
> again.
> Builds are working as expected, but we can't push using our CI any more,
> with the same exact symptoms as above :(
>
> Any idea?
> I have tried to reconcile roles, with no success.
>
> Thanks
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Can't push images after 1.3.0 upgrade

[Philippe Lafoucrière]

I'm digging this up.
We just upgraded our production cluster to OS 1.3, and having this issue
again.
Builds are working as expected, but we can't push using our CI any more,
with the same exact symptoms as above :(

Any idea?
I have tried to reconcile roles, with no success.

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

oc .env files

[Philippe Lafoucrière]

Hi,

We're having fun with the "oc apply" command, which solves a lot of
configuration issues we've had in the past.
There's just one thing I would like to have in oc: a local .oc_env file to
define some defaults, like:

- Current cluster url (make sure we're hitting the production cluster, not
any of the test clusters)
- Current namespace (to make sure apply won't fu.. up another project if we
forget to specify it).

Is there something we could see in the future?

Thanks,
Philippe

-- 
Philippe Lafoucrière - CEO
http://www.tech-angels.com
https://gemnasium.com
France : +33 (0) 3 65 96 02 92
Canada: +1 (418) 478-1175
USA: +1 (954) 607-7443
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Upgrading to 1.3 / 3.3

[Philippe Lafoucrière]

On Mon, Nov 7, 2016 at 5:31 PM, Diego Castro 
wrote:

> Hi Philippe i have upgraded without major issues using blue/green upgrade.
> The only thing was the secured registry that i managed to solve turning off
> the hosted registry at inventory file.
>

Thanks for sharing this. Do you use metrics in your cluster? Did it went
well?
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Metrics - Could not connect to Cassandra cluster

[Philippe Lafoucrière]

The problem is we removed everything in the project, and tried to reinstall
with ansible.
Since we can't deploy the metrics (even with the v1.2.1 image, note that
v1.2.2 doesn't exist:
https://hub.docker.com/r/openshift/origin-metrics-deployer/tags/), we're
screwed :(
We'll try this on another 1.3.1 cluster.

Thanks for tip anyway!
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Metrics - Could not connect to Cassandra cluster

[Philippe Lafoucrière]

We have the exact same problem, except hawkular-metrics never starts in our
case.
We tried a lot of things, ending with removing everything in
openshift-infra, and now metrics aren't deploying anymore (with or without
ansible).
The deployer always fails:

[...]

Adding password for user hawkular
Creating the Hawkular Metrics Secrets configuration json file
Creating the Hawkular Metrics Certificate Secrets configuration json file
Creating the Hawkular Metrics User Account Secrets
Creating the Cassandra Secrets configuration file
Creating the Cassandra Certificate Secrets configuration json file
Creating Hawkular Metrics & Cassandra Secrets
Error from server: error when creating
"/etc/deploy/_output/hawkular-metrics-secrets.json": secrets
"hawkular-metrics-secrets" already exists


Of course, if we remove "hawkular-metrics-secrets", the deployer doesn't
start. It's been recreated if we run ansible again, but fails with the same
error as above.

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How can I put logstash config files in ConfigMap ?

[Philippe Lafoucrière]

It seems to have been fixed in k8s 1.4.0, so with OS 1.3.1 as well.
https://github.com/kubernetes/kubernetes/blob/v1.4.0/pkg/util/validation/validation.go#L273
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Why I don't have debug information in DockerRegistry logs?

[Philippe Lafoucrière]

See: https://github.com/openshift/openshift-ansible/issues/2648
The registry is not logging anything because the ssl handshake failed.
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: HELP - Image pull back off

[Philippe Lafoucrière]

For the record, I'm pretty sure that's the issue blocking us from upgrading
to 1.3.X (the symptoms are exactly the same + this issue
https://github.com/openshift/origin/issues/11164).

We're using ansible to upgrade, and this change is part of it (yes, we
should run ansible more often).
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: HELP - Image pull back off

[Philippe Lafoucrière]

It's definitely ansible :(
We have updated by hand all nodes, and after an ansible run, the wrong
config is back
​We'll investigate from here
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Managing OpenShift Configuration with Puppet/Ansible… what are your best practices?

[Philippe Lafoucrière]

Thanks Clayton!
Really looking forward to seeing this released :)
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Managing OpenShift Configuration with Puppet/Ansible… what are your best practices?

[Philippe Lafoucrière]

Just to clarify our need here:

We want the projects config inside a configuration tool. There's currently
nothing preventing from modifying the config of a project (let's say, a
DC), and no one will be notified of the change.
We're looking for something to keep track of changes, and make sure the
config deployed is the config we have in our git repo.

Thanks

​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Secrets not available anymore with 1.3.0

[Philippe Lafoucrière]

Any news on this?
This is really a blocking issue to upgrade to 1.3.0 for us.
​
Thanks
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Can't push images after 1.3.0 upgrade

[Philippe Lafoucrière]

We're using :443/namespace/imagestream
We had to rollback to previous snapshots, we never managed to get it
working :(
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Secrets not available anymore with 1.3.0

[Philippe Lafoucrière]

It's definitely an issue related to 1.3.0. I have downgraded the cluster to
1.2.1, and it works again :(​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Can't push images after 1.3.0 upgrade

[Philippe Lafoucrière]

Note that I can pull the image with this account.
I have tried to readd the role to the user:

$ oadm policy add-cluster-role-to-user system:image-builder our_ci_user

with no success.
According to
https://docs.openshift.com/container-platform/3.3/admin_guide/manage_authorization_policy.html,
I
should be able to update the layers.

$ oadm policy who-can update imagestreams/layers
-> my ci user is listed here
​
Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Can't push images after 1.3.0 upgrade

[Philippe Lafoucrière]

On Tue, Sep 27, 2016 at 4:29 PM, Jordan Liggitt  wrote:
>
> Do you have the registry logs available from the timeframe during the
push?


10.1.0.1 - - [27/Sep/2016:20:59:57
+] time="2016-09-27T20:59:58.948672089Z" level=error msg="error
authorizing context: authorization header required" go.version=go1.6.3
http.request.host=redacted http.request.id=24db7eaf-f66f-462a-9d2e-434b77ca7a30
http.request.method=PATCH http.request.remoteaddr=172.29.13.4
http.request.uri="/v2/gemnasium-staging/registry-scanner/blobs/uploads/a9c303fc-e85b-428c-b799-9cba00a40f77?_state=FguTxOGl3FNUtqk1-RNJvR8E7fvACwiGW_MQetCuFRp7Ik5hbWUiOiJnZW1uYXNpdW0tc3RhZ2luZy9yZWdpc3RyeS1zY2FubmVyIiwiVVVJRCI6ImE5YzMwM2ZjLWU4NWItNDI4Yy1iNzk5LTljYmEwMGE0MGY3NyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wOS0yN1QyMDo1OTo1OC45MTEyMDE4MjhaIn0%3D"
http.request.useragent="docker/1.12.1 go/go1.6.3 git-commit/23cf638
kernel/3.16.0-4-amd64 os/linux arch/amd64
UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"
instance.id=093a0322-b4bf-4b2a-bed3-5f02d0b2b0d7
vars.name="gemnasium-staging/registry-scanner"
vars.uuid=a9c303fc-e85b-428c-b799-9cba00a40f77
10.1.0.1 - - [27/Sep/2016:20:59:58 +] "PATCH
/v2/gemnasium-staging/registry-scanner/blobs/uploads/a9c303fc-e85b-428c-b799-9cba00a40f77?_state=FguTxOGl3FNUtqk1-RNJvR8E7fvACwiGW_MQetCuFRp7Ik5hbWUiOiJnZW1uYXNpdW0tc3RhZ2luZy9yZWdpc3RyeS1zY2FubmVyIiwiVVVJRCI6ImE5YzMwM2ZjLWU4NWItNDI4Yy1iNzk5LTljYmEwMGE0MGY3NyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wOS0yN1QyMDo1OTo1OC45MTEyMDE4MjhaIn0%3D
HTTP/1.1" 401 248 "" "docker/1.12.1 go/go1.6.3 git-commit/23cf638
kernel/3.16.0-4-amd64 os/linux arch/amd64
UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"
10.1.0.1 - - [27/Sep/2016:20:59:58 +] "PATCH
/v2/gemnasium-staging/registry-scanner/blobs/uploads/66b43d12-ad91-4c48-9e74-4d7fcc2f6eb8?_state=XTu8Xy0JVxFNNHlaCwnssuOkev1Vc_xy_iyGsSwtI5t7Ik5hbWUiOiJnZW1uYXNpdW0tc3RhZ2luZy9yZWdpc3RyeS1zY2FubmVyIiwiVVVJRCI6IjY2YjQzZDEyLWFkOTEtNGM0OC05ZTc0LTRkN2ZjYzJmNmViOCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wOS0yN1QyMDo1OTo1OC45MTAyNDgzNzlaIn0%3D
HTTP/1.1" 401 248 "" "docker/1.12.1 go/go1.6.3 git-commit/23cf638
kernel/3.16.0-4-amd64 os/linux arch/amd64
UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Secrets not available anymore with 1.3.0

[Philippe Lafoucrière]

Is this what you're looking for?

 secret.go:152] Setting up volume airbrake-secrets for pod
41cdd02f-84ea-11e6-be87-005056b17dcc at
/var/lib/origin/openshift.local.volumes/pods/41cdd02f-84ea-11e6-be87-005056b17dcc/volumes/
kubernetes.io~secret/airbrake-secrets
 nsenter_mount.go:183] findmnt command: nsenter
[--mount=/rootfs/proc/1/ns/mnt -- /bin/findmnt -o target,fstype
--noheadings --first-only --target
/var/lib/origin/openshift.local.volumes/pods/41cdd02f-84ea-11e6-be87-005056b17dcc/volumes/
kubernetes.io~secret/airbrake-secrets]


secret.go:179] Received secret gemnasium-staging/airbrake containing (2)
pieces of data, 40 total bytes
atomic_writer.go:316]
/var/lib/origin/openshift.local.volumes/pods/41cdd02f-84ea-11e6-be87-005056b17dcc/volumes/
kubernetes.io~secret/airbrake-secrets: current paths:   [airbrake-key
airbrake-project-id]
atomic_writer.go:328]
/var/lib/origin/openshift.local.volumes/pods/41cdd02f-84ea-11e6-be87-005056b17dcc/volumes/
kubernetes.io~secret/airbrake-secrets: new paths:   [airbrake-key
airbrake-project-id]
atomic_writer.go:331]
/var/lib/origin/openshift.local.volumes/pods/41cdd02f-84ea-11e6-be87-005056b17dcc/volumes/
kubernetes.io~secret/airbrake-secrets: paths to remove: map[]
atomic_writer.go:136] pod gemnasium-staging/gemnasium-api-v1-3-xxi0j volume
airbrake-secrets: no update required for target directory
/var/lib/origin/openshift.local.volumes/pods/41cdd02f-84ea-11e6-be87-005056b17dcc/volumes/
kubernetes.io~secret/airbrake-secrets

I can't find any error related to that :(
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Can't push images after 1.3.0 upgrade

[Philippe Lafoucrière]

Hi,

Another issue we're facing after the upgrade to 1.3.0:
our CI service account can't push images to the registry anymore.
I have tried to push the image by hand:

202bc3fd6fe4: Pushing [==>]
7.114 MB
be16db112b16: Pushing [==>]
280.6 kB
unauthorized: authentication required

In the sa description, the tokens seem to be the same (at least they have
the same names).
I have triedto reconcile policies :

oadm policy reconcile-cluster-roles \
--additive-only=true \
--confirm

oadm policy reconcile-cluster-role-bindings \
--exclude-groups=system:authenticated \
--exclude-groups=system:authenticated:oauth \
--exclude-groups=system:unauthenticated \
--exclude-users=system:anonymous \
--additive-only=true \
--confirm

oadm policy reconcile-sccs \
--additive-only=true \
--confirm

(but it should done by the playbook I think), and yet, I can't push any
more :(

Did we miss something during the upgrade?

Thanks,
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Secrets not available anymore with 1.3.0

[Philippe Lafoucrière]

Hi,

We're testing OS 1.3.0 on our test cluster, and have something weird
happening.
The secrets are mounted, but apparently not readable anymore in _some_ pods:

This is on openshift 1.2.1:

{
"Source":
"/var/lib/origin/openshift.local.volumes/pods/3f7a5adc-84b1-11e6-8101-005056b12d45/volumes/
kubernetes.io~secret/airbrake-secrets",
"Destination": "/etc/secrets/airbrake",
"Mode": "ro,Z",
"RW": false
}

and on openshift 1.3.0:

 {
 "Source":
"/var/lib/origin/openshift.local.volumes/pods/19df38db-84e9-11e6-be87-005056b17dcc/volumes/
kubernetes.io~secret/airbrake-secrets",
 "Destination": "/etc/secrets/airbrake",
 "Mode": "ro,Z",
 "RW": false,
 "Propagation": "rslave"
 },

Only the propagation is different, but it should not be an issue.
I can't get a shell inside the container, because it's just an executable
wrapped inside a "scratch" docker image.

The pods with a shell don't seem to this problem, and I can see the secrets
mounted and used as usual.

Any hints?

Thanks,
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Adding master to 3 node install

[Philippe Lafoucrière]

On Thu, Aug 11, 2016 at 8:42 PM, Jason DeTiberus 
wrote:

> This sounds like your registry was using ephemeral storage rather than
> being backed by a PV or object storage.


It's should not.
We're using:

docker_register_volume_source='{"nfs": { "server": "10.x.x.x", "path":
"/zpool-1234/registry/"}}'

Anyway, it seems this variable isn't used anymore :( (in favor of the
portion you mentionned in your link)
I will investigate that.

Yet, I can see the volume present in the yaml manifest:

spec:
  volumes:
-
  name: registry-storage
  nfs:
server: 10.x.x.x
path: /zpool-1234/registry/
-
  name: registry-token-xmulp
  secret:
secretName: registry-token-xmulp
[...]

volumeMounts:
  -
name: registry-storage
mountPath: /registry
  -
name: registry-token-xmulp
readOnly: true
mountPath: /var/run/secrets/kubernetes.io/serviceaccount


but not in the console:

[image: Inline image 1]

Weird :)
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: configuring periodic import of images

[Philippe Lafoucrière]

https://docs.openshift.com/enterprise/3.2/install_config/install/docker_registry.html

" The manifest v2 schema 2

 (*schema2*) is not yet supported."

Sorry :)
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: configuring periodic import of images

[Philippe Lafoucrière]

On Thu, Aug 11, 2016 at 5:15 PM, Tony Saxon  wrote:

> Damn, I just went through having to downgrade my registry because I was
> pushing with 1.12 and openshift (running docker 1.10) wasn't able to pull
> the image due the the sha256 hash that it was referencing not existing
> because of the v1/v2 issues. I guess my only option if I don't want to
> upgrade my openshift is to push from a machine running docker 1.10?


I'm afraid so. We're running into the same issue, and haven't found a
solution yet :(
You could also try the 1.3.0-alpha3, but I wouldn't recommend it for
production of course...
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: configuring periodic import of images

[Philippe Lafoucrière]

If you are using different versions of docker on openshift, and the server
where the image was build, you will fall into this (known) problem.
Check out https://trello.com/c/CJiUnVUm/136-3-docker-1-10-
push-force-schema1-manifest-via-daemon-flag

Hopefuly, this seems to be fixed in the upcoming 1.3. Check the section :
Upgrading to Docker Registry 2.4, cross-repository linking, and better
usage tooling
in https://github.com/openshift/origin/releases/tag/v1.3.0-alpha.3
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: unexpected fault address 0x0

[Philippe Lafoucrière]

Hmm, indeed, the container will be always Up:

# cat /usr/local/bin/origin-node-run.sh
#!/bin/sh

set -eu

conf=${CONFIG_FILE:-/etc/origin/node/node-config.yaml}
opts=${OPTIONS:---loglevel=2}

function quit {
pkill -g 0 openshift
exit 0
}

trap quit SIGTERM

if [ ! -f ${HOST_ETC}/systemd/system/docker.service.d/docker-sdn-ovs.conf
]; then
mkdir -p ${HOST_ETC}/systemd/system/docker.service.d
cp /usr/lib/systemd/system/docker.service.d/docker-sdn-ovs.conf
${HOST_ETC}/systemd/system/docker.service.d
fi

/usr/bin/openshift start node "--config=${conf}" "${opts}" &

while true; do sleep 5; done


The while loop at the end is endless, even if openshift exited. Strange
idea.
What's the point of this?
Apparently, just handle the SIGTERM signal, and send it to all processes in
the same group:

# ps x -o  "%p %r %c" | grep 9003
 9003  9003 origin-node-run
 9012  9003 openshift
32347  9003 sleep

Maybe it lacks a test in the while loop to ensure at least "openshift" is
running?

​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Adding master to 3 node install

[Philippe Lafoucrière]

Did you check out this?
https://docs.openshift.com/enterprise/3.1/install_config/install/advanced_install.html#adding-nodes-advanced


​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Recreate strategy optimization

[Philippe Lafoucrière]

Hi,

I have noticed something that could be optimized in recreate strategy:
During a deploy, Openshift will set the current running replicas to zero,
and then pull the new image, and eventually starts it.
The problem is, sometimes, the pull can very (very) long, and during that
period, the pods are unavailable. I would be more efficient to start
pulling the image, and IF the image is pulled successfully, start to touch
replicas.
What do you think?

Thanks,
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Atomic Centos, can't upgrade

[Philippe Lafoucrière]

We're using postgres 9.5.
It was working fine before the upgrade. Unfortunately, we upgraded atomic
AND Openshift at the same time, so I can't tell if it's a problem with
docker 1.10 or openshift 1.2.1.
I'd tend to say Docker 1.10, but we need to isolate this first.

Thanks,
Philippe
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Atomic Centos, can't upgrade

[Philippe Lafoucrière]

https://docs.openshift.org/latest/dev_guide/shared_memory.html

fixed the issue, but It seems something changed regarding /dev or shm
docker mounts between 1.2.0 and 1.2.1.
Can someone confirm?
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Atomic Centos, can't upgrade

[Philippe Lafoucrière]

I confirm: it's fixed :)​
thanks!
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Atomic Centos, can't upgrade

[Philippe Lafoucrière]

On Tue, Jul 12, 2016 at 5:22 PM, Scott Dodson  wrote:

> I'll see if I can get openshift/node:v1.2.0 rebuilt with this fix but
> you can also rebuild the node image placing the docker wrapper script
> in /usr/local/bin
>

Any news on this?
Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Atomic Centos, can't upgrade

[Philippe Lafoucrière]

Testing it right away.
Thanks guys :)
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Atomic Centos, can't upgrade

[Philippe Lafoucrière]

Good catch Scott:


[plafoucriere@atomic-test-node-1 origin]# docker info
/usr/bin/docker-current: error while loading shared libraries:
libseccomp.so.2: cannot open shared object file: No such file or directory
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Atomic Centos, can't upgrade

[Philippe Lafoucrière]

On Mon, Jul 11, 2016 at 9:56 AM, Scott Dodson  wrote:

> That commit is mostly related to the fact that we cannot
> upgrade/downgrade docker on atomic host like can on RHEL so abort the
> docker upgrade playbook early.
>

Ok, I get it now, thanks.

Anyway, we couldn't fix our beta cluster, and had to restore snapshots, as
nothing was deploying anymore ("Failed to setup network for pod [...]").
Even with the latest version of the playbook :(
Should I open an issue in openshift, or openshift-ansible project for that?

Thanks
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Atomic Centos, can't upgrade

[Philippe Lafoucrière]

Sounds like docker 1.10 is a bad idea, I found this commit:

https://github.com/openshift/openshift-ansible/commit/b377f9d85df11c532281c213eda1869596642204

I was probably running openshift-ansible with a wrong tag :(
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: PVs, Endpoints, and GlusterFS

[Philippe Lafoucrière]

That's right, you'll have to create endpoints inside each project, and a
service to make sure the endpoints are not cleaned-up.

We have switched back to simple NFS after having a LOT of issues with
glusterfs, especially data corruption. It's a lot easier to setup in
openshift (no endpoints, no service required), and the recycle retain
policy works out of the box...
Maybe CEPH is a good option, but I've heard NFS is better for small files.

​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Atomic Centos, can't upgrade

[Philippe Lafoucrière]

We have updated our beta cluster to latest atomic centos:

-bash-4.2# atomic host status
  TIMESTAMP (UTC) VERSIONID OSNAME
REFSPEC
* 2016-07-07 21:23:41 7.20160707 cd47a72eb5 centos-atomic-host
centos-atomic-host:centos-atomic-host/7/x86_64/standard
  2016-06-10 13:15:00 7.20160610 3c3786d1dd centos-atomic-host
centos-atomic-host:centos-atomic-host/7/x86_64/standard

GPG: Found 1 signature on the booted deployment (*):

  Signature made Thu Jul  7 23:34:40 2016 using RSA key ID F17E745691BA8335
  Good signature from "CentOS Atomic SIG "


And the problem re-appeared:

Jul 10 01:40:08 atomic-test-node-1 origin-node[3150]: I0710 01:40:08.000177
   3201 manager.go:1400] Container
"0cf256d23de1b837a295233491e6650c90519fa2d0807d37f95a8164a842257b
gemnasium-enterprise/gemnasium-enterprise-7-8unp4" exited after 121.527557ms
Jul 10 01:40:08 atomic-test-node-1.priv.tech-angels.net origin-node[3150]:
E0710 01:40:08.0002433201 pod_workers.go:138] Error syncing pod
6e4dd3f7-462d-11e6-89a2-005056b17dcc, skipping: failed to "SetupNetwork"
for "gemnasium-enterprise-7-8unp4_gemnasium-enterprise" with
SetupNetworkError: "Failed to setup network for pod
\"gemnasium-enterprise-7-8unp4_gemnasium-enterprise(6e4dd3f7-462d-11e6-89a2-005056b17dcc)\"
using network plugins \"redhat/openshift-ovs-multitenant\": exit status 1;
Skipping pod"


Running the playbook doesn't seem to fix the problem this time. I've seen
docker has been updated to 1.10, could it be an issue?
​
Thanks
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Error pulling image (v1.2.0) from docker.io/openshift/origin-metrics-hawkular-metrics, HTTP code 400

[Philippe Lafoucrière]

Hi,

We can't pull this image anymore on our nodes:

Error pulling image (v1.2.0) from
docker.io/openshift/origin-metrics-hawkular-metrics, HTTP code 400

We're still using docker 1.9, and it seems we can pull the image everywhere
else (docker 1.9, 1.11, etc.), but not on the nodes where it was previously
deployed... We have therefore an event:

Failed to pull image "openshift/origin-metrics-hawkular-metrics:v1.2.0":
Error pulling image (v1.2.0) from
docker.io/openshift/origin-metrics-hawkular-metrics, HTTP code 400

Is anyone already had this? Googling for this issue only returns old things.

Thanks
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Labels removed from private registry

[Philippe Lafoucrière]

Hi,

We have labeled our images with a git commit sha, and apparently this label
disappear when the image is pushed on our openshift private registry. Is
this something on purpose?

Thanks,
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: logs aggregation

[Philippe Lafoucrière]

Great!
I will take a look
Thanks
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: All my image stream are bad docker registry IP, where is my mistake ?

[Philippe Lafoucrière]

On Thu, Jun 23, 2016 at 12:37 PM, Clayton Coleman 
wrote:

> Did you delete and recreate your docker registry?
>
>
yes, several times.
And we can't find any clue from where this IP is coming from.
We have grep all files, searched in etcd, and nothing.
It's a mystery :)
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: What is the consequence if I switch from ovs-subnet to ovs-multitenant on production cluster ?

[Philippe Lafoucrière]

Thanks!
We would love some feedback from people having done this before.
We have a test cluster, with snapshots, but sometimes it's all about the
details, and something could fail after a while :)

​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: What is the consequence if I switch from ovs-subnet to ovs-multitenant on production cluster ?

[Philippe Lafoucrière]

@Clayton, any idea on this?​
Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: prune tags

[Philippe Lafoucrière]

On Tue, Jun 21, 2016 at 2:33 PM, Clayton Coleman 
wrote:

> I don't think we have anything to prune tags (since we don't know what
> tag names to prune).  We'd need some way of knowing the tag was "not
> important" before it could be pruned.
>

I would say, if no stream nor container is using a specific tag?
I admit it's a bit harder than just untagged layers for images :(
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

prune tags

[Philippe Lafoucrière]

Hi,

As we're using openshift for continuous deployments, we often have a lot of
tags per ImageStream (to test branches). We would love to see these unused
tags pruned, like other objects.
Is there something in the roadmap?

Thanks,
Philippe

-- 
Philippe Lafoucrière - CEO
http://www.tech-angels.com
https://gemnasium.com
main : +33 (0) 970 444 643
mobile CA: +1 (581) 986-7540
mobile FR: +33 (0) 6 72 63 75 40
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: MySQL: Readiness probe failed

[Philippe Lafoucrière]

Have you tried to raise initialDelaySeconds ?
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Build/Deploy notifications

[Philippe Lafoucrière]

Hi,

We're using Openshift to do continuous deployment, and it's working great.
Anyway, we should be notified (email, slack, etc.) when something went
wrong, like a failing deployment.
Is there anyway to achieve that?

Thanks
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Atomic Centos, can't upgrade

[Philippe Lafoucrière]

Thanks Tobias for the detailed help!
I should have thought of running again ansible, I was focused on the error.


​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Atomic Centos, can't upgrade

[Philippe Lafoucrière]

Hi,

We have tried to update our atomic host centos 7, with the tree 3c3786d1dd
(from the tree e39c28570a), but deployments are all failing after the
updates on the nodes:

Error syncing pod, skipping: failed to "SetupNetwork" for "some_deploy"
with SetupNetworkError: "Failed to setup network for pod
\"some_deploy(d080f8d4-3498-11e6-8512-005056b1755a)\" using network plugins
\"redhat/openshift-ovs-subnet\": exit status 1; Skipping pod"
Where can I fill a report for that? Openshift or Atomic host bugtracker
(whereever it is)?

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: ImagePullBackOff - API error (500): manifest unknown

[Philippe Lafoucrière]

Thanks!​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: ImagePullBackOff - API error (500): manifest unknown

[Philippe Lafoucrière]

Is there a github issue we can follow to have updates on this?

Thanks
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: 503 - Maintenance page

[Philippe Lafoucrière]

On Tue, Jun 7, 2016 at 6:45 PM, Ram Ranganathan  wrote:

> Is your server always returning 503 - example for a GET/HEAD on / ? That
> could cause haproxy to mark it as down.
>
> You can also see the stats in haproxy to look at if the server has been
> marked down:
> cmd="echo 'show stat' | socat
> unix-connect:/var/lib/haproxy/run/haproxy.sock stdio"
> echo "$cmd"  | oc rsh #  replace with router pod
> name.
>

Of course my server is returning a 503 for "/' :) (it's down for
maintenance). Haproxy thinks no server is available, so it's not even
trying to pass the page. Make sense.
Ok, so I guess I'll to use a custom router then :(

Thanks for your help.
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: 503 - Maintenance page

[Philippe Lafoucrière]

On Tue, Jun 7, 2016 at 3:46 PM, Luke Meyer  wrote:

> It sounds like what he wants is for the router to simply not interfere
> with passing along something that's already returning a 503. It sounds like
> haproxy is replacing the page content with its own in that use case.


THANKS Luke :))
I don't want to change the router, I just want it to point to a specific
service returning 503 for most URLs.
On the other hand, the SAME router is used (with another route) to point to
the production service, with a different URL if we want to test the change.
Imagine a migration from pg 9.4 to 9.5, you have to shutdown your site.
That doesn't mean traffic can't be routed any more, we like to test the
site after the migration, and before resuming all the public traffic.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: 503 - Maintenance page

[Philippe Lafoucrière]

@Clayton:
Sorry for the confusion. I'm not updating the routeR, I'm updating the
route directly. The route to our website is pointing to a "maintenance"
service during maintenance. This service serves 503 pages for most URLs,
except a few for testing purprose.

The problem is: If I browse my website, I get the expected 503 code, but a
blank page, instead of the desired maintenance page served by the
"maintenance" pods. I don't understand this blank page, it's like haproxy
is not forwarding it because the pods responded with a 503.

@v: Can I use a dedicated router per project?
​
Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Limiting scaling of a pod

[Philippe Lafoucrière]

On Sun, May 22, 2016 at 7:05 PM, Clayton Coleman 
wrote:

> What are you trying to defend against?  The more concrete examples you
> can provide (where the app would be broken if there is >1 copy) the
> more we can do to ensure the use case is solved.
>

We have a legacy Resque Scheduler pod running. If this pod is running more
than once at a time, jobs are schedule (queued) more than once, and it
leads to other issues.
This pod doesn't have any PV, so we can't use a ReadWriteOnce PV to ensure
that.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Error updating deployment [deploy] status to Pending

[Philippe Lafoucrière]

Hi,

Yes the node is ready. I have tried to unschedule it, and evacuate pods,
it's not working either. I  don't use any PV in the test I'm doing. The
other node seems to have the same problem, so I guess it's somewhere else
than the node. Maybe corrupted data in etcd?
This cluster has been working for months now, I don't understand why it's
suddenly failing. I have absolutely no clue.
Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Error updating deployment [deploy] status to Pending

[Philippe Lafoucrière]

If I make this node unschedulable, I get an event: "node 'openshift-node-1'
is not in cache"
Does this ring a bell? (and the pod is still pending)
For the record, there's PV used in this pod, and all pods have the same
behavior now on this cluster. Only a restart of origin-node can unlock them.
​Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Error updating deployment [deploy] status to Pending

[Philippe Lafoucrière]

I have this in the logs (with loglevel=4):
https://gist.github.com/gravis/7454a743cb988f6d192bf5a5c9890a82
So, nothing fancy :(

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Error updating deployment [deploy] status to Pending

[Philippe Lafoucrière]

Hello,

I can't deploy any on os 1.1.3: the deploy (and/or build) is stuck on
pending, with the message:

Error updating deployment [project]/[deploy] status to Pending

in the events. Nothing specific in the logs.
Using oc describe doesn't give any reason, and I see a different event:

Events:
  FirstSeen LastSeenCount   From
 SubobjectPath   TypeReason  Message
  - -   
 -   --  ---
  6m6m  1   {default-scheduler }
 Normal  Scheduled   Successfully assigned
slots-site-8-deploy to node-1

The deploy is starting if I restart origin-node on node-1. If I deploy
again, I get stuck at the same place, and only a origin-node restart can
unlock it.

Any hints to fix that?

Thanks
Philippe
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: RWO mounted on multiple hosts

[Philippe Lafoucrière]

ho, and btw, Openshift was mentioned MANY times ;)​
Thanks for the hard work guys.

http://www.slideshare.net/plafoucriere/rails-monolithtomicroservicesdesign
(With speaker notes:)
https://speakerdeck.com/jipiboily/from-rails-to-microservices-with-go-our-experience-with-gemnasium-enterprise
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: RWO mounted on multiple hosts

[Philippe Lafoucrière]

I'm at a conference this week, will try to send you something next week.
Thanks
Philippe
​
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users