Re: Node startup Failure on SDN

[Jonathan Yu]

On Aug 15, 2016 11:08, "Skarbek, John"  wrote:
>
> So I figured it out. Ntp went kaboom on one of our master nodes.
>
> ERROR: [DCli0015 from diagnostic 
> ConfigContexts@openshift/origin/pkg/diagnostics/client/config_contexts.go:285]
For client config context 'default/cluster:8443/system:admin': The server
URL is 'https://cluster:8443' The user authentication is
'system:admin/cluster:8443' The current project is 'default' (*url.Error)
Get https://cluster:8443/api: x509: certificate has expired or is not yet
valid Diagnostics does not have an explanation for what this means. Please
report this error so one can be added.
>
> I ended up finding that the master node clock just…. I have no idea:
>
> [/etc/origin/master]# date Wed Feb 14 12:23:13 UTC 2001
>
> I’d like to suggest that diagnostics checks the date and time of all the
certificates and perhaps do some sort of ntp check and maybe even go the
extra mile and compare the time on the server to …life. I have no idea why
my master node decided to back to Valentines day in 2001. I think I was
single way back when.

Good idea. At minimum it seems like a good idea to record the build date
for the binary and check against that. I think Chrome does something
similar - perhaps figuring out how Chrome handles this is a reasonable
starting point
>
>
>
> --
> John Skarbek
>
> On August 15, 2016 at 13:32:13, Skarbek, John (john.skar...@ca.com) wrote:
>>
>> It would appear the certificate is valid 2018:
>>
>> `[/etc/origin/node]# openssl x509 -enddate -in system:node:node-001.crt
notAfter=Mar 21 15:18:10 2018 GMT
>>
>> Got any other ideas?
>>
>>
>>
>> --
>> John Skarbek
>>
>> On August 15, 2016 at 13:27:57, Clayton Coleman (ccole...@redhat.com)
wrote:
>>>
>>> The node's client certificate may have expired - that a common failure
mode.
>>>
>>> On Aug 15, 2016, at 1:23 PM, Skarbek, John  wrote:
>>>
 Good Morning,

 We recently had a node go down, upon trying to get it back online, the
origin-node service fails to start. The rest of the cluster appears to be
just fine, so with the desire to troubleshoot, what can I look at to
determine the root cause of the following error:

 Aug 15 17:12:59 node-001 origin-node[14536]: E0815 17:12:59.469682
14536 common.go:194] Failed to obtain ClusterNetwork: the server has asked
for the client to provide credentials (get clusterNetworks default) Aug 15
17:12:59 node-001 origin-node[14536]: F0815 17:12:59.469705 14536
node.go:310] error: SDN node startup failed: the server has asked for the
client to provide credentials (get clusterNetworks default)



 --
 John Skarbek

 ___
 users mailing list
 users@lists.openshift.redhat.com
 http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: no items in "browse catalog" page

[Jonathan Yu]

Hey Ravi,

On Mon, Aug 15, 2016 at 8:35 PM, Ravi Kapoor 
wrote:

> Thanks Jessica, that was helpful. I had to do following
>
> 1. oc cluster up
> it gave error
>  Error: did not detect an --insecure-registry argument on the Docker daemon
>Solution:
>
>  Ensure that the Docker daemon is running with the following argument:
> --insecure-registry 172.30.0.0/16
>

You will need to modify your /etc/sysconfig/docker file (on Fedora and I
think also CentOS and RHEL) to add this flag.  Other OSes will have these
flags potentially stored elsewhere, like /etc/default on Debian and Ubuntu.

>
> 2. Google search reveals too many places where this should be edited.
>

Yes, unfortunately different distros do things differently as there's no
"standard" approach. I would suggest adding your distro name to your Google
searches to get the most relevant results.


> For me, I had to add it to ExecStart in file /etc/systemd/system/docker.
> service
>
> Now I do see images as shown in walkthrough https://www.
> youtube.com/watch?v=yFPYGeKwmpk
>
> 3. However as I follow the tutorial, at time 6:30 (
> https://youtu.be/yFPYGeKwmpk?t=390), it fails to deploy. Log shows
> following error
> error: cannot connect to the server: open /var/run/secrets/kubernetes.
> io/serviceaccount/token: no such file or directory
>

Hmm, this looks interesting.  Unfortunately I'm not an expert so I'm not
sure where to look for this.  A quick search for the error message yields
this bug: https://github.com/kubernetes/kubernetes/issues/10620 - does
anything in there help?

>
> Once again, I find too many suggestions to how this can be fixed. I am
> afraid trying all those will take me days.
> One of the pages said this has been fixed in OS 1.3, however I am running
> OS 1.3
>
> thanks so much
>
>
>
> On Mon, Aug 15, 2016 at 4:33 PM, Jessica Forrester 
> wrote:
>
>> So if you are just running a local dev cluster to try things out, I
>> recommend using 'oc cluster up' instead, it will set up a lot for you,
>> including all the example image streams and templates.
>>
>> If you want to add them in an existing cluster see step 10 here
>> https://github.com/openshift/origin/blob/master/CONTRIB
>> UTING.adoc#develop-on-virtual-machine-using-vagrant
>>
>> Where the files it is referring to are in the origin repo under examples.
>>
>> On Mon, Aug 15, 2016 at 7:09 PM, Ravi Kapoor 
>> wrote:
>>
>>>
>>> I am a newbie, so excuse my ignorance. I have tried for 2 days now to
>>> get "browse catalog" page to show me catalog. I do not see any errors nor
>>> images.
>>> Due to this I am not able to follow any tutorials.
>>> thanks for helping
>>>
>>>
>>> This is what my page looks like (text copy in case image attachments are
>>> not allowed)
>>> 
>>> No images or templates.
>>>
>>> No images or templates are loaded for this project or the shared
>>> openshift namespace. An image or template is required to add content.
>>>
>>> To add an image stream or template from a file, use the editor in the
>>> Import YAML / JSON tab, or run the following command:
>>>
>>> oc create -f  -n test
>>> Back to overview
>>> 
>>>
>>> Here is a screenshot
>>>
>>> [image: Inline image 1]
>>>
>>> ___
>>> users mailing list
>>> users@lists.openshift.redhat.com
>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>>
>>>
>>
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>


-- 
Jonathan Yu, P.Eng. / Software Engineer, OpenShift by Red Hat / Twitter
(@jawnsy) is the quickest way to my heart 

*“A master in the art of living draws no sharp distinction between his work
and his play; his labor and his leisure; his mind and his body; his
education and his recreation. He hardly knows which is which. He simply
pursues his vision of excellence through whatever he is doing, and leaves
others to determine whether he is working or playing. To himself, he always
appears to be doing both.”* — L. P. Jacks, Education through Recreation
(1932), p. 1
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: no items in "browse catalog" page

[Ravi Kapoor]

Thanks Jessica, that was helpful. I had to do following

1. oc cluster up
it gave error
 Error: did not detect an --insecure-registry argument on the Docker daemon
   Solution:

 Ensure that the Docker daemon is running with the following argument:
--insecure-registry 172.30.0.0/16

2. Google search reveals too many places where this should be edited.
For me, I had to add it to ExecStart in file
/etc/systemd/system/docker.service

Now I do see images as shown in walkthrough
https://www.youtube.com/watch?v=yFPYGeKwmpk

3. However as I follow the tutorial, at time 6:30 (
https://youtu.be/yFPYGeKwmpk?t=390), it fails to deploy. Log shows
following error
error: cannot connect to the server: open /var/run/secrets/
kubernetes.io/serviceaccount/token: no such file or directory

Once again, I find too many suggestions to how this can be fixed. I am
afraid trying all those will take me days.
One of the pages said this has been fixed in OS 1.3, however I am running
OS 1.3

thanks so much



On Mon, Aug 15, 2016 at 4:33 PM, Jessica Forrester 
wrote:

> So if you are just running a local dev cluster to try things out, I
> recommend using 'oc cluster up' instead, it will set up a lot for you,
> including all the example image streams and templates.
>
> If you want to add them in an existing cluster see step 10 here
> https://github.com/openshift/origin/blob/master/
> CONTRIBUTING.adoc#develop-on-virtual-machine-using-vagrant
>
> Where the files it is referring to are in the origin repo under examples.
>
> On Mon, Aug 15, 2016 at 7:09 PM, Ravi Kapoor 
> wrote:
>
>>
>> I am a newbie, so excuse my ignorance. I have tried for 2 days now to get
>> "browse catalog" page to show me catalog. I do not see any errors nor
>> images.
>> Due to this I am not able to follow any tutorials.
>> thanks for helping
>>
>>
>> This is what my page looks like (text copy in case image attachments are
>> not allowed)
>> 
>> No images or templates.
>>
>> No images or templates are loaded for this project or the shared
>> openshift namespace. An image or template is required to add content.
>>
>> To add an image stream or template from a file, use the editor in the
>> Import YAML / JSON tab, or run the following command:
>>
>> oc create -f  -n test
>> Back to overview
>> 
>>
>> Here is a screenshot
>>
>> [image: Inline image 1]
>>
>> ___
>> users mailing list
>> users@lists.openshift.redhat.com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>
>>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: no items in "browse catalog" page

[Jessica Forrester]

So if you are just running a local dev cluster to try things out, I
recommend using 'oc cluster up' instead, it will set up a lot for you,
including all the example image streams and templates.

If you want to add them in an existing cluster see step 10 here
https://github.com/openshift/origin/blob/master/CONTRIBUTING.adoc#develop-on-virtual-machine-using-vagrant

Where the files it is referring to are in the origin repo under examples.

On Mon, Aug 15, 2016 at 7:09 PM, Ravi Kapoor 
wrote:

>
> I am a newbie, so excuse my ignorance. I have tried for 2 days now to get
> "browse catalog" page to show me catalog. I do not see any errors nor
> images.
> Due to this I am not able to follow any tutorials.
> thanks for helping
>
>
> This is what my page looks like (text copy in case image attachments are
> not allowed)
> 
> No images or templates.
>
> No images or templates are loaded for this project or the shared openshift
> namespace. An image or template is required to add content.
>
> To add an image stream or template from a file, use the editor in the
> Import YAML / JSON tab, or run the following command:
>
> oc create -f  -n test
> Back to overview
> 
>
> Here is a screenshot
>
> [image: Inline image 1]
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

no items in "browse catalog" page

[Ravi Kapoor]

I am a newbie, so excuse my ignorance. I have tried for 2 days now to get
"browse catalog" page to show me catalog. I do not see any errors nor
images.
Due to this I am not able to follow any tutorials.
thanks for helping


This is what my page looks like (text copy in case image attachments are
not allowed)

No images or templates.

No images or templates are loaded for this project or the shared openshift
namespace. An image or template is required to add content.

To add an image stream or template from a file, use the editor in the
Import YAML / JSON tab, or run the following command:

oc create -f  -n test
Back to overview


Here is a screenshot

[image: Inline image 1]
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Dynamic Provisioning PVs in AWS across Zones

[Andrew Lau]

Here was the original issue
https://github.com/kubernetes/kubernetes/issues/23330

On Tue, 16 Aug 2016, 7:50 AM Isaac Christoffersen <
ichristoffer...@vizuri.com> wrote:

> Ugh.  Alright, back to the 2-stage process now.  Do you have the issue
> reference in kubernetes?  I looked but couldn't find it.
>
>
>
> On Mon, Aug 15, 2016 at 5:43 PM, Andrew Lau  wrote:
>
>> This something thats tracked in kubernetes, iirc its something that was
>> planned to be fixed in 1.4
>>
>> On Tue, 16 Aug 2016, 7:08 AM Isaac Christoffersen <
>> ichristoffer...@vizuri.com> wrote:
>>
>>> I have an Origin v1.2.1 setup in AWS and I have my masters and nodes
>>> spread across zones in AWS.
>>>
>>> I'm looking to use the Dynamic provisioning of Persistent Volumes when a
>>> claim is made as per
>>> https://docs.openshift.org/latest/install_config/persistent_storage/dynamically_provisioning_pvs.html
>>>
>>> However, I'm getting an error as the PV is being created in a difference
>>> zone than the node.
>>>
>>> "Unable to mount volumes for pod
>>> "jenkins-cicd-2-9n2fl_cicd(3e69d228-632b-11e6-bab3-0ee2d3ada485)": Could
>>> not attach EBS Disk "aws://us-east-1a/vol-f1b04356": Error attaching EBS
>>> volume: InvalidVolume.ZoneMismatch: The volume 'vol-f1b04356' is not in the
>>> same availability zone as instance 'i-' status code: 400, request
>>> id:"
>>>
>>> Can I control the zone where the EBS is provisioned with dynamic
>>> provisioning?
>>>
>>> ___
>>> users mailing list
>>> users@lists.openshift.redhat.com
>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>>
>>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Dynamic Provisioning PVs in AWS across Zones

[Isaac Christoffersen]

Ugh.  Alright, back to the 2-stage process now.  Do you have the issue
reference in kubernetes?  I looked but couldn't find it.



On Mon, Aug 15, 2016 at 5:43 PM, Andrew Lau  wrote:

> This something thats tracked in kubernetes, iirc its something that was
> planned to be fixed in 1.4
>
> On Tue, 16 Aug 2016, 7:08 AM Isaac Christoffersen <
> ichristoffer...@vizuri.com> wrote:
>
>> I have an Origin v1.2.1 setup in AWS and I have my masters and nodes
>> spread across zones in AWS.
>>
>> I'm looking to use the Dynamic provisioning of Persistent Volumes when a
>> claim is made as per https://docs.openshift.org/latest/install_config/
>> persistent_storage/dynamically_provisioning_pvs.html
>>
>> However, I'm getting an error as the PV is being created in a difference
>> zone than the node.
>>
>> "Unable to mount volumes for pod "jenkins-cicd-2-9n2fl_cicd(
>> 3e69d228-632b-11e6-bab3-0ee2d3ada485)": Could not attach EBS Disk
>> "aws://us-east-1a/vol-f1b04356": Error attaching EBS volume:
>> InvalidVolume.ZoneMismatch: The volume 'vol-f1b04356' is not in the same
>> availability zone as instance 'i-' status code: 400, request id:"
>>
>> Can I control the zone where the EBS is provisioned with dynamic
>> provisioning?
>>
>> ___
>> users mailing list
>> users@lists.openshift.redhat.com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Dynamic Provisioning PVs in AWS across Zones

[Andrew Lau]

This something thats tracked in kubernetes, iirc its something that was
planned to be fixed in 1.4

On Tue, 16 Aug 2016, 7:08 AM Isaac Christoffersen <
ichristoffer...@vizuri.com> wrote:

> I have an Origin v1.2.1 setup in AWS and I have my masters and nodes
> spread across zones in AWS.
>
> I'm looking to use the Dynamic provisioning of Persistent Volumes when a
> claim is made as per
> https://docs.openshift.org/latest/install_config/persistent_storage/dynamically_provisioning_pvs.html
>
> However, I'm getting an error as the PV is being created in a difference
> zone than the node.
>
> "Unable to mount volumes for pod
> "jenkins-cicd-2-9n2fl_cicd(3e69d228-632b-11e6-bab3-0ee2d3ada485)": Could
> not attach EBS Disk "aws://us-east-1a/vol-f1b04356": Error attaching EBS
> volume: InvalidVolume.ZoneMismatch: The volume 'vol-f1b04356' is not in the
> same availability zone as instance 'i-' status code: 400, request
> id:"
>
> Can I control the zone where the EBS is provisioned with dynamic
> provisioning?
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Dynamic Provisioning PVs in AWS across Zones

[Isaac Christoffersen]

I have an Origin v1.2.1 setup in AWS and I have my masters and nodes spread
across zones in AWS.

I'm looking to use the Dynamic provisioning of Persistent Volumes when a
claim is made as per
https://docs.openshift.org/latest/install_config/persistent_storage/dynamically_provisioning_pvs.html

However, I'm getting an error as the PV is being created in a difference
zone than the node.

"Unable to mount volumes for pod
"jenkins-cicd-2-9n2fl_cicd(3e69d228-632b-11e6-bab3-0ee2d3ada485)": Could
not attach EBS Disk "aws://us-east-1a/vol-f1b04356": Error attaching EBS
volume: InvalidVolume.ZoneMismatch: The volume 'vol-f1b04356' is not in the
same availability zone as instance 'i-' status code: 400, request
id:"

Can I control the zone where the EBS is provisioned with dynamic
provisioning?
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: configuring periodic import of images

[Tony Saxon]

I'm using a registry deployed from a docker compose:

registry:
  restart: always
  image: registry:2.2.1
  ports:
- 5000:5000
  environment:
REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt
REGISTRY_HTTP_TLS_KEY: /certs/domain.key
REGISTRY_AUTH: htpasswd
REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
  volumes:
- /var/docker/registry/data:/var/lib/registry
- /var/docker/registry/certs:/certs
- /var/docker/registry/auth:/auth


It was originally using "image:2" but that was the one that I had problems
even importing the docker imaged due to the schema v1/v2 issue. After
changing it to 2.2.1 and repushing the image it worked.

On Mon, Aug 15, 2016 at 4:00 PM, Clayton Coleman 
wrote:

> Did a test, but the import looks like it works correctly for hub images.
> In this case are you using a regular Docker registry, the integrated
> registry, or a third party Docker registry?
>
> On Mon, Aug 15, 2016 at 3:34 PM, Clayton Coleman 
> wrote:
>
>> It's currently 15 minutes:
>>
>> imagePolicyConfig:
>>   disableScheduledImport: false
>>   maxImagesBulkImportedPerRepository: 5
>>   maxScheduledImageImportsPerMinute: 60
>>   scheduledImageImportMinimumIntervalSeconds: 900
>>
>> Will take a look and see if I can recreate this issue.
>>
>>
>> On Mon, Aug 15, 2016 at 2:33 PM, Tony Saxon  wrote:
>>
>>>
>>> So I've found that if I tag the imagestream manually, that it is able to
>>> pull down the latest changes and deploys them to my app:
>>>
>>> oc tag --source=docker --scheduled=true docker-lab.example.com:5000/te
>>> stwebapp:latest testwebapp:latest
>>>
>>> [root@os-master ~]# oc describe is
>>> Name:   testwebapp
>>> Created:4 days ago
>>> Labels: 
>>> Annotations:openshift.io/image.dockerRepos
>>> itoryCheck=2016-08-15T17:49:36Z
>>> Docker Pull Spec:   172.30.11.167:5000/testwebapp/testwebapp
>>>
>>> Tag Spec
>>> Created PullSpec
>>>   Image
>>> latest  docker-lab.example.com:5000/testwebapp:latest *38
>>> minutes ago  docker-lab.example.com:5000/te
>>> stwebapp@sha256:dd75ff58184489...
>>> About
>>> an hour ago   docker-lab.example.com:5000/te
>>> stwebapp@sha256:2a4f9e1262e377...
>>> 4 days
>>> ago  docker-lab.example.com:5000/te
>>> stwebapp@sha256:c1c8c6c3e1c672...
>>>
>>>   * tag is scheduled for periodic import
>>>   ! tag is insecure and can be imported over HTTP or self-signed HTTPS
>>>
>>>
>>> This updates the tags, redeploys the pods and all my new changes are
>>> visible once the new containers are up. It appears that it's not doing the
>>> periodic import despite being configured to. What is the default period
>>> that it uses to check the source registry?
>>>
>>>
>>> On Mon, Aug 15, 2016 at 2:29 PM, Tony Saxon 
>>> wrote:
>>>
 So I've found that if I tag the imagestream manually, that it is able
 to pull down the latest changes and deploys them to my app:

 On Mon, Aug 15, 2016 at 8:46 AM, Tony Saxon 
 wrote:

> There are logs showing that it's detecting that the imagestream has
> changed, but doesn't seem like there's any explanation of why it can't get
> it:
>
> Aug 15 08:18:10 os-master origin-master: I0815 08:18:10.446822   77042
> image_change_controller.go:47] Build image change controller detected
> ImageStream change 172.30.11.167:5000/testwebapp/testwebapp
> Aug 15 08:20:01 os-master origin-master: ation":2}]},{"tag":"8.1","item
> s":[{"created":"2016-08-02T18:21:31Z","dockerImageReference"
> :"openshift/wildfly-81-centos7@sha256:68a27d407fd1ead3b8a9e3
> 3aa2054c948ad3a54556d28bb4caaf704a0f651f96","image":"sha256:
> 68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f65
> 1f96","generation":2}]},{"tag":"9.0","items":[{"created":"20
> 16-08-02T18:21:31Z","dockerImageReference":"openshift/wildfl
> y-90-centos7@sha256:212d8e093d50b44cf8dd3101d22e7efce6293d74
> 1a6dc30fced9cd27b70c7c22","image":"sha256:212d8e093d50b44
> cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c22","generati
> on":2}]},{"tag":"latest","items":[{"created":"2016-08-02T18:
> 21:31Z","dockerImageReference":"openshift/wildfly-100-centos7@sha256
> :5a428b5b36d4cd98dce8603d5accb30ff014b7d4fb73c2bb895
> edda89cabbb3d","image":"sha256:5a428b5b36d4cd98dce8603d5accb
> 30ff014b7d4fb73c2bb895edda89cabbb3d","generation":2}]}]}},{"
> metadata":{"name":"testwebapp","namespace":"testwebapp","
> selfLink":"/oapi/v1/namespaces/testwebapp/imagestr
> eams/testwebapp","uid":"dae5b8d1-5fc3-11e6-88da-525400f41cdb
> ","resourceVersion":"359311","generation":2,"creationTimesta
> mp":"2016-08-11T13:02:27Z","annotations":{"openshift.io/
>

Re: configuring periodic import of images

[Clayton Coleman]

Did a test, but the import looks like it works correctly for hub images.
In this case are you using a regular Docker registry, the integrated
registry, or a third party Docker registry?

On Mon, Aug 15, 2016 at 3:34 PM, Clayton Coleman 
wrote:

> It's currently 15 minutes:
>
> imagePolicyConfig:
>   disableScheduledImport: false
>   maxImagesBulkImportedPerRepository: 5
>   maxScheduledImageImportsPerMinute: 60
>   scheduledImageImportMinimumIntervalSeconds: 900
>
> Will take a look and see if I can recreate this issue.
>
>
> On Mon, Aug 15, 2016 at 2:33 PM, Tony Saxon  wrote:
>
>>
>> So I've found that if I tag the imagestream manually, that it is able to
>> pull down the latest changes and deploys them to my app:
>>
>> oc tag --source=docker --scheduled=true docker-lab.example.com:5000/te
>> stwebapp:latest testwebapp:latest
>>
>> [root@os-master ~]# oc describe is
>> Name:   testwebapp
>> Created:4 days ago
>> Labels: 
>> Annotations:openshift.io/image.dockerRepos
>> itoryCheck=2016-08-15T17:49:36Z
>> Docker Pull Spec:   172.30.11.167:5000/testwebapp/testwebapp
>>
>> Tag Spec
>> Created PullSpec
>>   Image
>> latest  docker-lab.example.com:5000/testwebapp:latest *38
>> minutes ago  docker-lab.example.com:5000/te
>> stwebapp@sha256:dd75ff58184489...
>> About an
>> hour ago   docker-lab.example.com:5000/te
>> stwebapp@sha256:2a4f9e1262e377...
>> 4 days
>> ago  docker-lab.example.com:5000/te
>> stwebapp@sha256:c1c8c6c3e1c672...
>>
>>   * tag is scheduled for periodic import
>>   ! tag is insecure and can be imported over HTTP or self-signed HTTPS
>>
>>
>> This updates the tags, redeploys the pods and all my new changes are
>> visible once the new containers are up. It appears that it's not doing the
>> periodic import despite being configured to. What is the default period
>> that it uses to check the source registry?
>>
>>
>> On Mon, Aug 15, 2016 at 2:29 PM, Tony Saxon  wrote:
>>
>>> So I've found that if I tag the imagestream manually, that it is able to
>>> pull down the latest changes and deploys them to my app:
>>>
>>> On Mon, Aug 15, 2016 at 8:46 AM, Tony Saxon 
>>> wrote:
>>>
 There are logs showing that it's detecting that the imagestream has
 changed, but doesn't seem like there's any explanation of why it can't get
 it:

 Aug 15 08:18:10 os-master origin-master: I0815 08:18:10.446822   77042
 image_change_controller.go:47] Build image change controller detected
 ImageStream change 172.30.11.167:5000/testwebapp/testwebapp
 Aug 15 08:20:01 os-master origin-master: ation":2}]},{"tag":"8.1","item
 s":[{"created":"2016-08-02T18:21:31Z","dockerImageReference"
 :"openshift/wildfly-81-centos7@sha256:68a27d407fd1ead3b8a9e3
 3aa2054c948ad3a54556d28bb4caaf704a0f651f96","image":"sha256:
 68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f65
 1f96","generation":2}]},{"tag":"9.0","items":[{"created":"20
 16-08-02T18:21:31Z","dockerImageReference":"openshift/
 wildfly-90-centos7@sha256:212d8e093d50b44cf8dd3101d22e7efce6
 293d741a6dc30fced9cd27b70c7c22","image":"sha256:212d8e093d50
 b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c22","gener
 ation":2}]},{"tag":"latest","items":[{"created":"2016-08-
 02T18:21:31Z","dockerImageReference":"openshift/wildfly-100-
 centos7@sha256:5a428b5b36d4cd98dce8603d5accb30ff014b7d4fb73c
 2bb895edda89cabbb3d","image":"sha256:5a428b5b36d4cd98dce8603
 d5accb30ff014b7d4fb73c2bb895edda89cabbb3d","generation":2}]}
 ]}},{"metadata":{"name":"testwebapp","namespace":"
 testwebapp","selfLink":"/oapi/v1/namespaces/testwebapp/image
 streams/testwebapp","uid":"dae5b8d1-5fc3-11e6-88da-525400
 f41cdb","resourceVersion":"359311","generation":2,"creati
 onTimestamp":"2016-08-11T13:02:27Z","annotations":{"opensh
 ift.io/image.dockerRepositoryCheck":"2016-08-11T13:02:27Z"}}
 ,"spec":{"tags":[{"name":"latest","annotations":null,"
 from":{"kind":"DockerImage","name":"docker-lab.example.com:
 5000/testwebapp:latest"},"generation":1,"importPolicy":{
 "scheduled":true}}]},"status":{"dockerImageRepository":"172.
 30.11.167:5000/testwebapp/testwebapp","tags":[{"tag":"
 latest","items":[{"created":"2016-08-11T13:02:27Z","
 dockerImageReference":"docker-lab.example.com:5000/testwebap
 p@sha256:c1c8c6c3e1c6729d1366acaf54c9772b4849f35d971e73449cf
 9044f3af06074","image":"sha256:c1c8c6c3e1c6729d1366aca
 f54c9772b4849f35d971e73449cf9044f3af06074","generation":1}]}]}}]}
 Aug 15 08:22:00 os-master origin-master: ation":2}]},{"tag":"8.1","item
 s":[{"created":"2016-08-02T18:21:31Z","dockerImageReference"
 :"openshift/wildfly-81-centos7@sha25

Re: configuring periodic import of images

[Clayton Coleman]

It's currently 15 minutes:

imagePolicyConfig:
  disableScheduledImport: false
  maxImagesBulkImportedPerRepository: 5
  maxScheduledImageImportsPerMinute: 60
  scheduledImageImportMinimumIntervalSeconds: 900

Will take a look and see if I can recreate this issue.


On Mon, Aug 15, 2016 at 2:33 PM, Tony Saxon  wrote:

>
> So I've found that if I tag the imagestream manually, that it is able to
> pull down the latest changes and deploys them to my app:
>
> oc tag --source=docker --scheduled=true docker-lab.example.com:5000/
> testwebapp:latest testwebapp:latest
>
> [root@os-master ~]# oc describe is
> Name:   testwebapp
> Created:4 days ago
> Labels: 
> Annotations:openshift.io/image.dockerRepositoryCheck=2016-08-
> 15T17:49:36Z
> Docker Pull Spec:   172.30.11.167:5000/testwebapp/testwebapp
>
> Tag Spec
> Created PullSpec
>   Image
> latest  docker-lab.example.com:5000/testwebapp:latest *38 minutes
> ago  docker-lab.example.com:5000/testwebapp@sha256:dd75ff58184489...
> 
> About an
> hour ago   docker-lab.example.com:5000/testwebapp@sha256:
> 2a4f9e1262e377...
> 4 days
> ago  docker-lab.example.com:5000/testwebapp@sha256:
> c1c8c6c3e1c672...
>
>   * tag is scheduled for periodic import
>   ! tag is insecure and can be imported over HTTP or self-signed HTTPS
>
>
> This updates the tags, redeploys the pods and all my new changes are
> visible once the new containers are up. It appears that it's not doing the
> periodic import despite being configured to. What is the default period
> that it uses to check the source registry?
>
>
> On Mon, Aug 15, 2016 at 2:29 PM, Tony Saxon  wrote:
>
>> So I've found that if I tag the imagestream manually, that it is able to
>> pull down the latest changes and deploys them to my app:
>>
>> On Mon, Aug 15, 2016 at 8:46 AM, Tony Saxon  wrote:
>>
>>> There are logs showing that it's detecting that the imagestream has
>>> changed, but doesn't seem like there's any explanation of why it can't get
>>> it:
>>>
>>> Aug 15 08:18:10 os-master origin-master: I0815 08:18:10.446822   77042
>>> image_change_controller.go:47] Build image change controller detected
>>> ImageStream change 172.30.11.167:5000/testwebapp/testwebapp
>>> Aug 15 08:20:01 os-master origin-master: ation":2}]},{"tag":"8.1","item
>>> s":[{"created":"2016-08-02T18:21:31Z","dockerImageReference"
>>> :"openshift/wildfly-81-centos7@sha256:68a27d407fd1ead3b8a9e3
>>> 3aa2054c948ad3a54556d28bb4caaf704a0f651f96","image":"sha256:
>>> 68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f65
>>> 1f96","generation":2}]},{"tag":"9.0","items":[{"created":"
>>> 2016-08-02T18:21:31Z","dockerImageReference":"openshi
>>> ft/wildfly-90-centos7@sha256:212d8e093d50b44cf8dd3101d22e7e
>>> fce6293d741a6dc30fced9cd27b70c7c22","image":"sha256:212d8e09
>>> 3d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c22","
>>> generation":2}]},{"tag":"latest","items":[{"created":"2016-
>>> 08-02T18:21:31Z","dockerImageReference":"openshift/wildfly-
>>> 100-centos7@sha256:5a428b5b36d4cd98dce8603d5accb30ff014b7d4f
>>> b73c2bb895edda89cabbb3d","image":"sha256:5a428b5b36d4cd9
>>> 8dce8603d5accb30ff014b7d4fb73c2bb895edda89cabbb3d","generati
>>> on":2}]}]}},{"metadata":{"name":"testwebapp","namespace"
>>> :"testwebapp","selfLink":"/oapi/v1/namespaces/testwebapp/
>>> imagestreams/testwebapp","uid":"dae5b8d1-5fc3-11e6-88da-
>>> 525400f41cdb","resourceVersion":"359311","generation":2,"
>>> creationTimestamp":"2016-08-11T13:02:27Z","annotations":{"
>>> openshift.io/image.dockerRepositoryCheck":"2016-
>>> 08-11T13:02:27Z"}},"spec":{"tags":[{"name":"latest","annot
>>> ations":null,"from":{"kind":"DockerImage","name":"docker-la
>>> b.example.com:5000/testwebapp:latest"},"generation":1,"impor
>>> tPolicy":{"scheduled":true}}]},"status":{"dockerImageRepository":"
>>> 172.30.11.167:5000/testwebapp/testwebapp","tags":
>>> [{"tag":"latest","items":[{"created":"2016-08-11T13:02:
>>> 27Z","dockerImageReference":"docker-lab.example.com:5000/
>>> testwebapp@sha256:c1c8c6c3e1c6729d1366acaf54c9772b4849f35d97
>>> 1e73449cf9044f3af06074","image":"sha256:c1c8c6c3e1c6729
>>> d1366acaf54c9772b4849f35d971e73449cf9044f3af06074","generati
>>> on":1}]}]}}]}
>>> Aug 15 08:22:00 os-master origin-master: ation":2}]},{"tag":"8.1","item
>>> s":[{"created":"2016-08-02T18:21:31Z","dockerImageReference"
>>> :"openshift/wildfly-81-centos7@sha256:68a27d407fd1ead3b8a9e3
>>> 3aa2054c948ad3a54556d28bb4caaf704a0f651f96","image":"sha256:
>>> 68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f65
>>> 1f96","generation":2}]},{"tag":"9.0","items":[{"created":"
>>> 2016-08-02T18:21:31Z","dockerImageReference":"openshi
>>> ft/wildfly-90-centos7@sha256:212d8e093d50b44cf8dd3101d22e7e
>>> fce6293d741a6dc

Re: configuring periodic import of images

[Tony Saxon]

So I've found that if I tag the imagestream manually, that it is able to
pull down the latest changes and deploys them to my app:

oc tag --source=docker --scheduled=true
docker-lab.example.com:5000/testwebapp:latest testwebapp:latest

[root@os-master ~]# oc describe is
Name:   testwebapp
Created:4 days ago
Labels: 
Annotations:
openshift.io/image.dockerRepositoryCheck=2016-08-15T17:49:36Z
Docker Pull Spec:   172.30.11.167:5000/testwebapp/testwebapp

Tag Spec
Created
PullSpec
Image
latest  docker-lab.example.com:5000/testwebapp:latest *38 minutes
ago  docker-lab.example.com:5000/testwebapp@sha256:dd75ff58184489...

About an
hour ago   docker-lab.example.com:5000/testwebapp@sha256:2a4f9e1262e377...

4 days
ago
docker-lab.example.com:5000/testwebapp@sha256:c1c8c6c3e1c672...


  * tag is scheduled for periodic import
  ! tag is insecure and can be imported over HTTP or self-signed HTTPS


This updates the tags, redeploys the pods and all my new changes are
visible once the new containers are up. It appears that it's not doing the
periodic import despite being configured to. What is the default period
that it uses to check the source registry?


On Mon, Aug 15, 2016 at 2:29 PM, Tony Saxon  wrote:

> So I've found that if I tag the imagestream manually, that it is able to
> pull down the latest changes and deploys them to my app:
>
> On Mon, Aug 15, 2016 at 8:46 AM, Tony Saxon  wrote:
>
>> There are logs showing that it's detecting that the imagestream has
>> changed, but doesn't seem like there's any explanation of why it can't get
>> it:
>>
>> Aug 15 08:18:10 os-master origin-master: I0815 08:18:10.446822   77042
>> image_change_controller.go:47] Build image change controller detected
>> ImageStream change 172.30.11.167:5000/testwebapp/testwebapp
>> Aug 15 08:20:01 os-master origin-master: ation":2}]},{"tag":"8.1","item
>> s":[{"created":"2016-08-02T18:21:31Z","dockerImageReference"
>> :"openshift/wildfly-81-centos7@sha256:68a27d407fd1ead
>> 3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f651f96","image":"
>> sha256:68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf7
>> 04a0f651f96","generation":2}]},{"tag":"9.0","items":[{"
>> created":"2016-08-02T18:21:31Z","dockerImageReference":"o
>> penshift/wildfly-90-centos7@sha256:212d8e093d50b44cf8dd3101d
>> 22e7efce6293d741a6dc30fced9cd27b70c7c22","image":"sha256:212
>> d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c2
>> 2","generation":2}]},{"tag":"latest","items":[{"created":"
>> 2016-08-02T18:21:31Z","dockerImageReference":"openshift/
>> wildfly-100-centos7@sha256:5a428b5b36d4cd98dce8603d5accb30ff
>> 014b7d4fb73c2bb895edda89cabbb3d","image":"sha256:5a428b5b36d
>> 4cd98dce8603d5accb30ff014b7d4fb73c2bb895edda89cabbb3d","
>> generation":2}]}]}},{"metadata":{"name":"testwebapp","
>> namespace":"testwebapp","selfLink":"/oapi/v1/namespaces/
>> testwebapp/imagestreams/testwebapp","uid":"dae5b8d1-
>> 5fc3-11e6-88da-525400f41cdb","resourceVersion":"359311","gen
>> eration":2,"creationTimestamp":"2016-08-11T13:02:27Z","annotations":{"
>> openshift.io/image.dockerRepositoryCheck":"2016-08-11T13:02:27Z"}},"spec"
>> :{"tags":[{"name":"latest","annotations":null,"from":{"kind"
>> :"DockerImage","name":"docker-lab.example.com:5000/testwebapp:latest
>> "},"generation":1,"importPolicy":{"scheduled":true}}]},"status":{"
>> dockerImageRepository":"172.30.11.167:5000/testwebapp/testwebapp
>> ","tags":[{"tag":"latest","items":[{"created":"2016-08-
>> 11T13:02:27Z","dockerImageReference":"docker-lab.example.
>> com:5000/testwebapp@sha256:c1c8c6c3e1c6729d1366acaf54c9772b4
>> 849f35d971e73449cf9044f3af06074","image":"sha256:c1c8c6c3e1c
>> 6729d1366acaf54c9772b4849f35d971e73449cf9044f3af06074","
>> generation":1}]}]}}]}
>> Aug 15 08:22:00 os-master origin-master: ation":2}]},{"tag":"8.1","item
>> s":[{"created":"2016-08-02T18:21:31Z","dockerImageReference"
>> :"openshift/wildfly-81-centos7@sha256:68a27d407fd1ead
>> 3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f651f96","image":"
>> sha256:68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf7
>> 04a0f651f96","generation":2}]},{"tag":"9.0","items":[{"
>> created":"2016-08-02T18:21:31Z","dockerImageReference":"o
>> penshift/wildfly-90-centos7@sha256:212d8e093d50b44cf8dd3101d
>> 22e7efce6293d741a6dc30fced9cd27b70c7c22","image":"sha256:212
>> d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c2
>> 2","generation":2}]},{"tag":"latest","items":[{"created":"
>> 2016-08-02T18:21:31Z","dockerImageReference":"openshift/
>> wildfly-100-centos7@sha256:5a428b5b36d4cd98dce8603d5accb30ff
>> 014b7d4fb73c2bb895edda89cabbb3d","image":"sha256:5a428b5b36d
>> 4cd98dce8603d5accb30ff014b7d4fb73c2bb895edda89cabbb3d","
>> generation":2}]}]}},{"metadata":{"name":"testwebapp","
>> namespace":"testwebapp","selfLink":"/oapi/v1/namespaces/
>> testwebapp/i

Re: configuring periodic import of images

[Tony Saxon]

So I've found that if I tag the imagestream manually, that it is able to
pull down the latest changes and deploys them to my app:

On Mon, Aug 15, 2016 at 8:46 AM, Tony Saxon  wrote:

> There are logs showing that it's detecting that the imagestream has
> changed, but doesn't seem like there's any explanation of why it can't get
> it:
>
> Aug 15 08:18:10 os-master origin-master: I0815 08:18:10.446822   77042
> image_change_controller.go:47] Build image change controller detected
> ImageStream change 172.30.11.167:5000/testwebapp/testwebapp
> Aug 15 08:20:01 os-master origin-master: ation":2}]},{"tag":"8.1","
> items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"
> openshift/wildfly-81-centos7@sha256:68a27d407fd1ead3b8a9e33aa2054c
> 948ad3a54556d28bb4caaf704a0f651f96","image":"sha256:
> 68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f65
> 1f96","generation":2}]},{"tag":"9.0","items":[{"created":"
> 2016-08-02T18:21:31Z","dockerImageReference":"
> openshift/wildfly-90-centos7@sha256:212d8e093d50b44cf8dd3101d22e7e
> fce6293d741a6dc30fced9cd27b70c7c22","image":"sha256:
> 212d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c
> 7c22","generation":2}]},{"tag":"latest","items":[{"created":
> "2016-08-02T18:21:31Z","dockerImageReference":"
> openshift/wildfly-100-centos7@sha256:5a428b5b36d4cd98dce8603d5accb3
> 0ff014b7d4fb73c2bb895edda89cabbb3d","image":"sha256:
> 5a428b5b36d4cd98dce8603d5accb30ff014b7d4fb73c2bb895edda89cab
> bb3d","generation":2}]}]}},{"metadata":{"name":"testwebapp"
> ,"namespace":"testwebapp","selfLink":"/oapi/v1/namespaces/testwebapp/
> imagestreams/testwebapp","uid":"dae5b8d1-5fc3-11e6-88da-525400f41cdb","
> resourceVersion":"359311","generation":2,"creationTimestamp":"2016-08-
> 11T13:02:27Z","annotations":{"openshift.io/image.dockerRepositoryCheck
> ":"2016-08-11T13:02:27Z"}},"spec":{"tags":[{"name":"latest","
> annotations":null,"from":{"kind":"DockerImage","name":"do
> cker-lab.example.com:5000/testwebapp:latest"},"
> generation":1,"importPolicy":{"scheduled":true}}]},"status":
> {"dockerImageRepository":"172.30.11.167:5000/testwebapp/testwebapp
> ","tags":[{"tag":"latest","items":[{"created":"2016-08-11T13:02:27Z","
> dockerImageReference":"docker-lab.example.com:5000/testwebapp@sha256:
> c1c8c6c3e1c6729d1366acaf54c9772b4849f35d971e73449cf9044f3af06074
> ","image":"sha256:c1c8c6c3e1c6729d1366acaf54c977
> 2b4849f35d971e73449cf9044f3af06074","generation":1}]}]}}]}
> Aug 15 08:22:00 os-master origin-master: ation":2}]},{"tag":"8.1","
> items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"
> openshift/wildfly-81-centos7@sha256:68a27d407fd1ead3b8a9e33aa2054c
> 948ad3a54556d28bb4caaf704a0f651f96","image":"sha256:
> 68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f65
> 1f96","generation":2}]},{"tag":"9.0","items":[{"created":"
> 2016-08-02T18:21:31Z","dockerImageReference":"
> openshift/wildfly-90-centos7@sha256:212d8e093d50b44cf8dd3101d22e7e
> fce6293d741a6dc30fced9cd27b70c7c22","image":"sha256:
> 212d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c
> 7c22","generation":2}]},{"tag":"latest","items":[{"created":
> "2016-08-02T18:21:31Z","dockerImageReference":"
> openshift/wildfly-100-centos7@sha256:5a428b5b36d4cd98dce8603d5accb3
> 0ff014b7d4fb73c2bb895edda89cabbb3d","image":"sha256:
> 5a428b5b36d4cd98dce8603d5accb30ff014b7d4fb73c2bb895edda89cab
> bb3d","generation":2}]}]}},{"metadata":{"name":"testwebapp"
> ,"namespace":"testwebapp","selfLink":"/oapi/v1/namespaces/testwebapp/
> imagestreams/testwebapp","uid":"dae5b8d1-5fc3-11e6-88da-525400f41cdb","
> resourceVersion":"359311","generation":2,"creationTimestamp":"2016-08-
> 11T13:02:27Z","annotations":{"openshift.io/image.dockerRepositoryCheck
> ":"2016-08-11T13:02:27Z"}},"spec":{"tags":[{"name":"latest","
> annotations":null,"from":{"kind":"DockerImage","name":"do
> cker-lab.example.com:5000/testwebapp:latest"},"
> generation":1,"importPolicy":{"scheduled":true}}]},"status":
> {"dockerImageRepository":"172.30.11.167:5000/testwebapp/testwebapp
> ","tags":[{"tag":"latest","items":[{"created":"2016-08-11T13:02:27Z","
> dockerImageReference":"docker-lab.example.com:5000/testwebapp@sha256:
> c1c8c6c3e1c6729d1366acaf54c9772b4849f35d971e73449cf9044f3af06074
> ","image":"sha256:c1c8c6c3e1c6729d1366acaf54c977
> 2b4849f35d971e73449cf9044f3af06074","generation":1}]}]}}]}
> Aug 15 08:23:59 os-master origin-master: ation":2}]},{"tag":"8.1","
> items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"
> openshift/wildfly-81-centos7@sha256:68a27d407fd1ead3b8a9e33aa2054c
> 948ad3a54556d28bb4caaf704a0f651f96","image":"sha256:
> 68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f65
> 1f96","generation":2}]},{"tag":"9.0","items":[{"created":"
> 2016-08-02T18:21:31Z","dockerImageReference":"
> openshift/wildfly-90-centos7@sha256:212d8e093d50b44cf8dd3101d22e7e
> fce6293d741a6dc30fced9cd27b70c7c22","image":"sha256:
> 212d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c
> 7c22","gener

Re: Node startup Failure on SDN

[Skarbek, John]

So I figured it out. Ntp went kaboom on one of our master nodes.

ERROR: [DCli0015 from diagnostic 
ConfigContexts@openshift/origin/pkg/diagnostics/client/config_contexts.go:285]
   For client config context 'default/cluster:8443/system:admin':
   The server URL is 'https://cluster:8443'
   The user authentication is 'system:admin/cluster:8443'
   The current project is 'default'
   (*url.Error) Get https://cluster:8443/api: x509: certificate has expired 
or is not yet valid
   Diagnostics does not have an explanation for what this means. Please 
report this error so one can be added.



I ended up finding that the master node clock just…. I have no idea:

[/etc/origin/master]# date
Wed Feb 14 12:23:13 UTC 2001


I’d like to suggest that diagnostics checks the date and time of all the 
certificates and perhaps do some sort of ntp check and maybe even go the extra 
mile and compare the time on the server to …life. I have no idea why my master 
node decided to back to Valentines day in 2001. I think I was single way back 
when.


--
John Skarbek


On August 15, 2016 at 13:32:13, Skarbek, John 
(john.skar...@ca.com) wrote:

It would appear the certificate is valid 2018:

`[/etc/origin/node]# openssl x509 -enddate -in system:node:node-001.crt 
notAfter=Mar 21 15:18:10 2018 GMT

Got any other ideas?


--
John Skarbek


On August 15, 2016 at 13:27:57, Clayton Coleman 
(ccole...@redhat.com) wrote:

The node's client certificate may have expired - that a common failure mode.

On Aug 15, 2016, at 1:23 PM, Skarbek, John 
mailto:john.skar...@ca.com>> wrote:


Good Morning,

We recently had a node go down, upon trying to get it back online, the 
origin-node service fails to start. The rest of the cluster appears to be just 
fine, so with the desire to troubleshoot, what can I look at to determine the 
root cause of the following error:

Aug 15 17:12:59 node-001 origin-node[14536]: E0815 17:12:59.469682   14536 
common.go:194] Failed to obtain ClusterNetwork: the server has asked for the 
client to provide credentials (get clusterNetworks default)
Aug 15 17:12:59 node-001 origin-node[14536]: F0815 17:12:59.469705   14536 
node.go:310] error: SDN node startup failed: the server has asked for the 
client to provide credentials (get clusterNetworks default)



--
John Skarbek
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Hybrid Cloud Hostname Issues (AWS, Co-Lo)

[Jason DeTiberus]

On Mon, Aug 15, 2016 at 12:09 PM, Frank Liauw  wrote:

> Thanks Jordan.
>
> The Ansible playbooks do not provide a means of setting nodeName when
> performing Advanced Install or scaleup? I can add new nodes with the
> appropriate nodeName set and remove the existing ones if possible.
>

Setting the openshift_hostname variable should override the nodeName value
when using the playbooks, both for install and scaleup.



>
> Otherwise, would you care to provide some details on what node client
> credentials needs to be updated? The documentation revolves largely around
> assisted setups; didn't manage to find anything on manual node
> configuration and introduction into an existing cluster (might look into
> how scaleup works in Ansible for some idea).
>

I'm cc'ing Andrew Butcher, work that he has been doing recently should
allow for you to replace the certificates/kubeconfigs that are used.


>
> Frank
> Systems Engineer
>
> VSee: fr...@vsee.com  | Cell: +65 9338 0035
>
> Join me on VSee for Free 
>
>
>
>
> On Mon, Aug 15, 2016 at 11:32 PM, Jordan Liggitt 
> wrote:
>
>> Node names are immutable in the API. Changing the node name would require
>> updating the node client credentials, the node would register another node
>> with the API when it started, and any pods scheduled to the old node name
>> would get evicted once the old node object's status got stale enough.
>>
>>
>>
>> On Aug 15, 2016, at 11:25 AM, Frank Liauw  wrote:
>>
>> Thanks Jason.
>>
>> Can I update nodeName in config.yaml and restart the EC2 nodes? Will that
>> update the metadata of my nodes automatically across the entire cluster?
>>
>> Frank
>> Systems Engineer
>>
>> VSee: fr...@vsee.com  | Cell: +65 9338 0035
>>
>> Join me on VSee for Free 
>>
>>
>>
>>
>> On Mon, Aug 15, 2016 at 11:10 PM, Jason DeTiberus 
>> wrote:
>>
>>>
>>>
>>> On Mon, Aug 15, 2016 at 4:17 AM, Frank Liauw  wrote:
>>>
 Hi All,

 I have a 5 node Openshift cluster split across 2 AZs; our colocation
 center and AWS, with a master in each AZ and the rest being nodes.

 We setup our cluster with the Ansible script, and somewhere during the
 setup, the EC2 instance's private hostname were picked up and registered as
 node names of the nodes in AWS, which is a bit annoying as that deviates
 from our hostname conventions and is rather difficult to read, and it's not
 something that can be changed post setup.

 It didn't help that parts of the admin operations seem to be using the
 EC2 instance's private hostname, so I get errors like this:

 # oc logs logging-fluentd-shfnu
 Error from server: Get https://ip-10-20-128-101.us-we
 st-1.compute.internal:10250/containerLogs/logging/logging-fl
 uentd-shfnu/fluentd-elasticsearch: dial tcp 198.90.20.95:10250: i/o
 timeout

 Scheduling system related pods on the AWS instances works (router,
 fluentd), though any build pods that lands up on EC2s never gets built, and
 just eventually times out; my suspicion is that the build process monitors
 depends on the hostname which can't be reached from our colocation center
 master (which we use as a primary), and hence breaks.

 I'm unable to find much detail on this behaviour.

 1. Can we manually change the hostname of certain nodes?

>>>
>>> The nodeName value overrides this, however if you are relying on cloud
>>> provider integration there are limitations, see below.
>>>
>>>

 2. How do we avoid registering EC2 nodes with their private hostnames?

>>>
>>> f you are willing to give up the native cloud provider integration
>>> (ability to leverage EBS volumes as PVs), then you can override this using
>>> the openshift_hostname variable when installing the cluster. At least as of
>>> Kubernetes/Origin 1.2, the nodeName value in the node config needed to
>>> match the private dns name of the host.
>>>
>>> --
>>> Jason DeTiberus
>>>
>>
>> ___
>> users mailing list
>> users@lists.openshift.redhat.com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>
>>
>


-- 
Jason DeTiberus
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Node startup Failure on SDN

[Skarbek, John]

It would appear the certificate is valid 2018:

` [/etc/origin/node]# openssl x509 -enddate -in system:node:node-001.crt 
notAfter=Mar 21 15:18:10 2018 GMT

Got any other ideas?


--
John Skarbek


On August 15, 2016 at 13:27:57, Clayton Coleman 
(ccole...@redhat.com) wrote:

The node's client certificate may have expired - that a common failure mode.

On Aug 15, 2016, at 1:23 PM, Skarbek, John 
mailto:john.skar...@ca.com>> wrote:


Good Morning,

We recently had a node go down, upon trying to get it back online, the 
origin-node service fails to start. The rest of the cluster appears to be just 
fine, so with the desire to troubleshoot, what can I look at to determine the 
root cause of the following error:

Aug 15 17:12:59 node-001 origin-node[14536]: E0815 17:12:59.469682   14536 
common.go:194] Failed to obtain ClusterNetwork: the server has asked for the 
client to provide credentials (get clusterNetworks default)
Aug 15 17:12:59 node-001 origin-node[14536]: F0815 17:12:59.469705   14536 
node.go:310] error: SDN node startup failed: the server has asked for the 
client to provide credentials (get clusterNetworks default)



--
John Skarbek
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Node startup Failure on SDN

[Clayton Coleman]

The node's client certificate may have expired - that a common failure mode.

On Aug 15, 2016, at 1:23 PM, Skarbek, John  wrote:

Good Morning,

We recently had a node go down, upon trying to get it back online, the
origin-node service fails to start. The rest of the cluster appears to be
just fine, so with the desire to troubleshoot, what can I look at to
determine the root cause of the following error:

Aug 15 17:12:59 node-001 origin-node[14536]: E0815 17:12:59.469682
14536 common.go:194] Failed to obtain ClusterNetwork: the server has
asked for the client to provide credentials (get clusterNetworks
default)
Aug 15 17:12:59 node-001 origin-node[14536]: F0815 17:12:59.469705
14536 node.go:310] error: SDN node startup failed: the server has
asked for the client to provide credentials (get clusterNetworks
default)



-- 
John Skarbek

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Node startup Failure on SDN

[Skarbek, John]

Good Morning,

We recently had a node go down, upon trying to get it back online, the 
origin-node service fails to start. The rest of the cluster appears to be just 
fine, so with the desire to troubleshoot, what can I look at to determine the 
root cause of the following error:

Aug 15 17:12:59 node-001 origin-node[14536]: E0815 17:12:59.469682   14536 
common.go:194] Failed to obtain ClusterNetwork: the server has asked for the 
client to provide credentials (get clusterNetworks default)
Aug 15 17:12:59 node-001 origin-node[14536]: F0815 17:12:59.469705   14536 
node.go:310] error: SDN node startup failed: the server has asked for the 
client to provide credentials (get clusterNetworks default)



--
John Skarbek
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Hybrid Cloud Hostname Issues (AWS, Co-Lo)

[Frank Liauw]

Thanks Jordan.

The Ansible playbooks do not provide a means of setting nodeName when
performing Advanced Install or scaleup? I can add new nodes with the
appropriate nodeName set and remove the existing ones if possible.

Otherwise, would you care to provide some details on what node client
credentials needs to be updated? The documentation revolves largely around
assisted setups; didn't manage to find anything on manual node
configuration and introduction into an existing cluster (might look into
how scaleup works in Ansible for some idea).

Frank
Systems Engineer

VSee: fr...@vsee.com  | Cell: +65 9338 0035

Join me on VSee for Free 




On Mon, Aug 15, 2016 at 11:32 PM, Jordan Liggitt 
wrote:

> Node names are immutable in the API. Changing the node name would require
> updating the node client credentials, the node would register another node
> with the API when it started, and any pods scheduled to the old node name
> would get evicted once the old node object's status got stale enough.
>
>
>
> On Aug 15, 2016, at 11:25 AM, Frank Liauw  wrote:
>
> Thanks Jason.
>
> Can I update nodeName in config.yaml and restart the EC2 nodes? Will that
> update the metadata of my nodes automatically across the entire cluster?
>
> Frank
> Systems Engineer
>
> VSee: fr...@vsee.com  | Cell: +65 9338 0035
>
> Join me on VSee for Free 
>
>
>
>
> On Mon, Aug 15, 2016 at 11:10 PM, Jason DeTiberus 
> wrote:
>
>>
>>
>> On Mon, Aug 15, 2016 at 4:17 AM, Frank Liauw  wrote:
>>
>>> Hi All,
>>>
>>> I have a 5 node Openshift cluster split across 2 AZs; our colocation
>>> center and AWS, with a master in each AZ and the rest being nodes.
>>>
>>> We setup our cluster with the Ansible script, and somewhere during the
>>> setup, the EC2 instance's private hostname were picked up and registered as
>>> node names of the nodes in AWS, which is a bit annoying as that deviates
>>> from our hostname conventions and is rather difficult to read, and it's not
>>> something that can be changed post setup.
>>>
>>> It didn't help that parts of the admin operations seem to be using the
>>> EC2 instance's private hostname, so I get errors like this:
>>>
>>> # oc logs logging-fluentd-shfnu
>>> Error from server: Get https://ip-10-20-128-101.us-we
>>> st-1.compute.internal:10250/containerLogs/logging/logging-fl
>>> uentd-shfnu/fluentd-elasticsearch: dial tcp 198.90.20.95:10250: i/o
>>> timeout
>>>
>>> Scheduling system related pods on the AWS instances works (router,
>>> fluentd), though any build pods that lands up on EC2s never gets built, and
>>> just eventually times out; my suspicion is that the build process monitors
>>> depends on the hostname which can't be reached from our colocation center
>>> master (which we use as a primary), and hence breaks.
>>>
>>> I'm unable to find much detail on this behaviour.
>>>
>>> 1. Can we manually change the hostname of certain nodes?
>>>
>>
>> The nodeName value overrides this, however if you are relying on cloud
>> provider integration there are limitations, see below.
>>
>>
>>>
>>> 2. How do we avoid registering EC2 nodes with their private hostnames?
>>>
>>
>> f you are willing to give up the native cloud provider integration
>> (ability to leverage EBS volumes as PVs), then you can override this using
>> the openshift_hostname variable when installing the cluster. At least as of
>> Kubernetes/Origin 1.2, the nodeName value in the node config needed to
>> match the private dns name of the host.
>>
>> --
>> Jason DeTiberus
>>
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Hybrid Cloud Hostname Issues (AWS, Co-Lo)

[Jordan Liggitt]

Node names are immutable in the API. Changing the node name would require
updating the node client credentials, the node would register another node
with the API when it started, and any pods scheduled to the old node name
would get evicted once the old node object's status got stale enough.



On Aug 15, 2016, at 11:25 AM, Frank Liauw  wrote:

Thanks Jason.

Can I update nodeName in config.yaml and restart the EC2 nodes? Will that
update the metadata of my nodes automatically across the entire cluster?

Frank
Systems Engineer

VSee: fr...@vsee.com  | Cell: +65 9338 0035

Join me on VSee for Free 




On Mon, Aug 15, 2016 at 11:10 PM, Jason DeTiberus 
wrote:

>
>
> On Mon, Aug 15, 2016 at 4:17 AM, Frank Liauw  wrote:
>
>> Hi All,
>>
>> I have a 5 node Openshift cluster split across 2 AZs; our colocation
>> center and AWS, with a master in each AZ and the rest being nodes.
>>
>> We setup our cluster with the Ansible script, and somewhere during the
>> setup, the EC2 instance's private hostname were picked up and registered as
>> node names of the nodes in AWS, which is a bit annoying as that deviates
>> from our hostname conventions and is rather difficult to read, and it's not
>> something that can be changed post setup.
>>
>> It didn't help that parts of the admin operations seem to be using the
>> EC2 instance's private hostname, so I get errors like this:
>>
>> # oc logs logging-fluentd-shfnu
>> Error from server: Get https://ip-10-20-128-101.us-we
>> st-1.compute.internal:10250/containerLogs/logging/logging-fl
>> uentd-shfnu/fluentd-elasticsearch: dial tcp 198.90.20.95:10250: i/o
>> timeout
>>
>> Scheduling system related pods on the AWS instances works (router,
>> fluentd), though any build pods that lands up on EC2s never gets built, and
>> just eventually times out; my suspicion is that the build process monitors
>> depends on the hostname which can't be reached from our colocation center
>> master (which we use as a primary), and hence breaks.
>>
>> I'm unable to find much detail on this behaviour.
>>
>> 1. Can we manually change the hostname of certain nodes?
>>
>
> The nodeName value overrides this, however if you are relying on cloud
> provider integration there are limitations, see below.
>
>
>>
>> 2. How do we avoid registering EC2 nodes with their private hostnames?
>>
>
> f you are willing to give up the native cloud provider integration
> (ability to leverage EBS volumes as PVs), then you can override this using
> the openshift_hostname variable when installing the cluster. At least as of
> Kubernetes/Origin 1.2, the nodeName value in the node config needed to
> match the private dns name of the host.
>
> --
> Jason DeTiberus
>

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Hybrid Cloud Hostname Issues (AWS, Co-Lo)

[Frank Liauw]

Thanks Jason.

Can I update nodeName in config.yaml and restart the EC2 nodes? Will that
update the metadata of my nodes automatically across the entire cluster?

Frank
Systems Engineer

VSee: fr...@vsee.com  | Cell: +65 9338 0035

Join me on VSee for Free 




On Mon, Aug 15, 2016 at 11:10 PM, Jason DeTiberus 
wrote:

>
>
> On Mon, Aug 15, 2016 at 4:17 AM, Frank Liauw  wrote:
>
>> Hi All,
>>
>> I have a 5 node Openshift cluster split across 2 AZs; our colocation
>> center and AWS, with a master in each AZ and the rest being nodes.
>>
>> We setup our cluster with the Ansible script, and somewhere during the
>> setup, the EC2 instance's private hostname were picked up and registered as
>> node names of the nodes in AWS, which is a bit annoying as that deviates
>> from our hostname conventions and is rather difficult to read, and it's not
>> something that can be changed post setup.
>>
>> It didn't help that parts of the admin operations seem to be using the
>> EC2 instance's private hostname, so I get errors like this:
>>
>> # oc logs logging-fluentd-shfnu
>> Error from server: Get https://ip-10-20-128-101.us-we
>> st-1.compute.internal:10250/containerLogs/logging/logging-fl
>> uentd-shfnu/fluentd-elasticsearch: dial tcp 198.90.20.95:10250: i/o
>> timeout
>>
>> Scheduling system related pods on the AWS instances works (router,
>> fluentd), though any build pods that lands up on EC2s never gets built, and
>> just eventually times out; my suspicion is that the build process monitors
>> depends on the hostname which can't be reached from our colocation center
>> master (which we use as a primary), and hence breaks.
>>
>> I'm unable to find much detail on this behaviour.
>>
>> 1. Can we manually change the hostname of certain nodes?
>>
>
> The nodeName value overrides this, however if you are relying on cloud
> provider integration there are limitations, see below.
>
>
>>
>> 2. How do we avoid registering EC2 nodes with their private hostnames?
>>
>
> f you are willing to give up the native cloud provider integration
> (ability to leverage EBS volumes as PVs), then you can override this using
> the openshift_hostname variable when installing the cluster. At least as of
> Kubernetes/Origin 1.2, the nodeName value in the node config needed to
> match the private dns name of the host.
>
> --
> Jason DeTiberus
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Hybrid Cloud Hostname Issues (AWS, Co-Lo)

[Jason DeTiberus]

On Mon, Aug 15, 2016 at 4:17 AM, Frank Liauw  wrote:

> Hi All,
>
> I have a 5 node Openshift cluster split across 2 AZs; our colocation
> center and AWS, with a master in each AZ and the rest being nodes.
>
> We setup our cluster with the Ansible script, and somewhere during the
> setup, the EC2 instance's private hostname were picked up and registered as
> node names of the nodes in AWS, which is a bit annoying as that deviates
> from our hostname conventions and is rather difficult to read, and it's not
> something that can be changed post setup.
>
> It didn't help that parts of the admin operations seem to be using the EC2
> instance's private hostname, so I get errors like this:
>
> # oc logs logging-fluentd-shfnu
> Error from server: Get https://ip-10-20-128-101.us-
> west-1.compute.internal:10250/containerLogs/logging/logging-
> fluentd-shfnu/fluentd-elasticsearch: dial tcp 198.90.20.95:10250: i/o
> timeout
>
> Scheduling system related pods on the AWS instances works (router,
> fluentd), though any build pods that lands up on EC2s never gets built, and
> just eventually times out; my suspicion is that the build process monitors
> depends on the hostname which can't be reached from our colocation center
> master (which we use as a primary), and hence breaks.
>
> I'm unable to find much detail on this behaviour.
>
> 1. Can we manually change the hostname of certain nodes?
>

The nodeName value overrides this, however if you are relying on cloud
provider integration there are limitations, see below.


>
> 2. How do we avoid registering EC2 nodes with their private hostnames?
>

f you are willing to give up the native cloud provider integration (ability
to leverage EBS volumes as PVs), then you can override this using the
openshift_hostname variable when installing the cluster. At least as of
Kubernetes/Origin 1.2, the nodeName value in the node config needed to
match the private dns name of the host.

--
Jason DeTiberus
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: configuring periodic import of images

[Tony Saxon]

There are logs showing that it's detecting that the imagestream has
changed, but doesn't seem like there's any explanation of why it can't get
it:

Aug 15 08:18:10 os-master origin-master: I0815 08:18:10.446822   77042
image_change_controller.go:47] Build image change controller detected
ImageStream change 172.30.11.167:5000/testwebapp/testwebapp
Aug 15 08:20:01 os-master origin-master:
ation":2}]},{"tag":"8.1","items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"openshift/wildfly-81-centos7@sha256
:68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f651f96","image":"sha256:68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f651f96","generation":2}]},{"tag":"9.0","items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"openshift/wildfly-90-centos7@sha256
:212d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c22","image":"sha256:212d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c22","generation":2}]},{"tag":"latest","items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"openshift/wildfly-100-centos7@sha256
:5a428b5b36d4cd98dce8603d5accb30ff014b7d4fb73c2bb895edda89cabbb3d","image":"sha256:5a428b5b36d4cd98dce8603d5accb30ff014b7d4fb73c2bb895edda89cabbb3d","generation":2}]}]}},{"metadata":{"name":"testwebapp","namespace":"testwebapp","selfLink":"/oapi/v1/namespaces/testwebapp/imagestreams/testwebapp","uid":"dae5b8d1-5fc3-11e6-88da-525400f41cdb","resourceVersion":"359311","generation":2,"creationTimestamp":"2016-08-11T13:02:27Z","annotations":{"
openshift.io/image.dockerRepositoryCheck
":"2016-08-11T13:02:27Z"}},"spec":{"tags":[{"name":"latest","annotations":null,"from":{"kind":"DockerImage","name":"
docker-lab.example.com:5000/testwebapp:latest
"},"generation":1,"importPolicy":{"scheduled":true}}]},"status":{"dockerImageRepository":"
172.30.11.167:5000/testwebapp/testwebapp
","tags":[{"tag":"latest","items":[{"created":"2016-08-11T13:02:27Z","dockerImageReference":"
docker-lab.example.com:5000/testwebapp@sha256:c1c8c6c3e1c6729d1366acaf54c9772b4849f35d971e73449cf9044f3af06074
","image":"sha256:c1c8c6c3e1c6729d1366acaf54c9772b4849f35d971e73449cf9044f3af06074","generation":1}]}]}}]}
Aug 15 08:22:00 os-master origin-master:
ation":2}]},{"tag":"8.1","items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"openshift/wildfly-81-centos7@sha256
:68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f651f96","image":"sha256:68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f651f96","generation":2}]},{"tag":"9.0","items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"openshift/wildfly-90-centos7@sha256
:212d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c22","image":"sha256:212d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c22","generation":2}]},{"tag":"latest","items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"openshift/wildfly-100-centos7@sha256
:5a428b5b36d4cd98dce8603d5accb30ff014b7d4fb73c2bb895edda89cabbb3d","image":"sha256:5a428b5b36d4cd98dce8603d5accb30ff014b7d4fb73c2bb895edda89cabbb3d","generation":2}]}]}},{"metadata":{"name":"testwebapp","namespace":"testwebapp","selfLink":"/oapi/v1/namespaces/testwebapp/imagestreams/testwebapp","uid":"dae5b8d1-5fc3-11e6-88da-525400f41cdb","resourceVersion":"359311","generation":2,"creationTimestamp":"2016-08-11T13:02:27Z","annotations":{"
openshift.io/image.dockerRepositoryCheck
":"2016-08-11T13:02:27Z"}},"spec":{"tags":[{"name":"latest","annotations":null,"from":{"kind":"DockerImage","name":"
docker-lab.example.com:5000/testwebapp:latest
"},"generation":1,"importPolicy":{"scheduled":true}}]},"status":{"dockerImageRepository":"
172.30.11.167:5000/testwebapp/testwebapp
","tags":[{"tag":"latest","items":[{"created":"2016-08-11T13:02:27Z","dockerImageReference":"
docker-lab.example.com:5000/testwebapp@sha256:c1c8c6c3e1c6729d1366acaf54c9772b4849f35d971e73449cf9044f3af06074
","image":"sha256:c1c8c6c3e1c6729d1366acaf54c9772b4849f35d971e73449cf9044f3af06074","generation":1}]}]}}]}
Aug 15 08:23:59 os-master origin-master:
ation":2}]},{"tag":"8.1","items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"openshift/wildfly-81-centos7@sha256
:68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f651f96","image":"sha256:68a27d407fd1ead3b8a9e33aa2054c948ad3a54556d28bb4caaf704a0f651f96","generation":2}]},{"tag":"9.0","items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"openshift/wildfly-90-centos7@sha256
:212d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c22","image":"sha256:212d8e093d50b44cf8dd3101d22e7efce6293d741a6dc30fced9cd27b70c7c22","generation":2}]},{"tag":"latest","items":[{"created":"2016-08-02T18:21:31Z","dockerImageReference":"openshift/wildfly-100-centos7@sha256
:5a428b5b36d4cd98dce8603d5accb30ff014b7d4fb73c2bb895edda89cabbb3d","image":"sha256:5a428b5b36d4cd98dce8603d5accb30ff014b7d4fb73c2bb895edda89cabbb3d","generation":2}]}]}},{"metadata":{"name":"testwebapp","namespace":"testwebapp","selfLink":"/oapi/v1/na

Hybrid Cloud Hostname Issues (AWS, Co-Lo)

[Frank Liauw]

Hi All,

I have a 5 node Openshift cluster split across 2 AZs; our colocation center
and AWS, with a master in each AZ and the rest being nodes.

We setup our cluster with the Ansible script, and somewhere during the
setup, the EC2 instance's private hostname were picked up and registered as
node names of the nodes in AWS, which is a bit annoying as that deviates
from our hostname conventions and is rather difficult to read, and it's not
something that can be changed post setup.

It didn't help that parts of the admin operations seem to be using the EC2
instance's private hostname, so I get errors like this:

# oc logs logging-fluentd-shfnu
Error from server: Get
https://ip-10-20-128-101.us-west-1.compute.internal:10250/containerLogs/logging/logging-fluentd-shfnu/fluentd-elasticsearch:
dial tcp 198.90.20.95:10250: i/o timeout

Scheduling system related pods on the AWS instances works (router,
fluentd), though any build pods that lands up on EC2s never gets built, and
just eventually times out; my suspicion is that the build process monitors
depends on the hostname which can't be reached from our colocation center
master (which we use as a primary), and hence breaks.

I'm unable to find much detail on this behaviour.

1. Can we manually change the hostname of certain nodes?

2. How do we avoid registering EC2 nodes with their private hostnames?

Frank
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Kibana Logs Empty

[Frank Liauw]

Hi All,

I followed through the instructions on
https://docs.openshift.org/latest/install_config/aggregate_logging.html and
have setup a 3 node ES cluster. Fluentd is also deployed on all my nodes.

I am getting kibana logs on the logging project, but all my other projects
do not have any logs; kibana shows "No results found", with occasional
errors reading "Discover: An error occurred with your request. Reset your
inputs and try again."

Probing the requests made by kibana, some calls to
/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471245075265
are
failing from time to time.

Looking into the ES logs for all 3 cluster pods, I don't see much errors to
be concerned, with the last error of 2 nodes similar to the following which
seems to be a known issue with Openshift's setup (
https://lists.openshift.redhat.com/openshift-archives/users/2015-December/msg00078.html)
and possibly explains the failed requests made by kibana on auto-refresh,
but that's a problem for another day:

[2016-08-15 06:53:49,130][INFO ][cluster.service  ] [Gremlin] added
{[Quicksilver][t2l6Oz8uT-WS8Fa7S7jzfQ][logging-es-d7r1t3dm-2-a0cf0][inet[/10.1.3.3:9300]],},
reason: zen-disco-receive(from master [[One Above
All][CyFgyTTtS_S85yYRom2wVQ][logging-es-0w45va6n-2-8m85p][inet[/10.1.2.5:9300
]]])
[2016-08-15
06:59:27,727][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter]
Error while apply() due to
com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no
bypass or execute filters at all for action
indices:admin/mappings/fields/get
com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no
bypass or execute filters at all

Looking into fluentd logs, one of my nodes is complaining of a
"getaddrinfo" error:

2016-08-15 03:45:18 -0400 [error]: unexpected error error="getaddrinfo:
Name or service not known"
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in
`initialize'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in
`open'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:878:in
`block in connect'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/timeout.rb:52:in
`timeout'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:877:in
`connect'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:862:in
`do_start'
  2016-08-15 03:45:18 -0400 [error]: /usr/share/ruby/net/http.rb:851:in
`start'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:766:in
`transmit'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:215:in
`execute'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/request.rb:52:in
`execute'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/rest-client-2.0.0/lib/restclient/resource.rb:51:in
`get'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:328:in
`block in api'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:58:in
`handle_exception'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:327:in
`api'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/kubeclient-1.1.4/lib/kubeclient/common.rb:322:in
`api_valid?'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluent-plugin-kubernetes_metadata_filter-0.24.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:167:in
`configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:144:in
`add_filter'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:61:in `block in
configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `each'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in
`block in configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in `each'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:83:in
`configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:129:in
`configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/engine.rb:103:in
`run_configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:483:in
`run_configure'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:154:in
`block in start'
  2016-08-15 03:45:18 -0400 [error]:
/opt/app-root/src/gems/fluentd-0.12.23/lib/fluent/supervi

Re: Resolving localhost (IPv6 issue?)

[Ulf Lilleengen]

sh-4.3$ dig +search localhost

; <<>> DiG 9.10.3-P4-RedHat-9.10.3-13.P4.fc23 <<>> +search localhost
;; global options: +cmd
;; connection timed out; no servers could be reached

On 08/11/2016 08:12 PM, Clayton Coleman wrote:

Can you run dig in a container on localhost (dig +search localhost, I
think) and contrast that?  Is it only one tool that fails to read
/etc/hosts, or all tools?

On Thu, Aug 11, 2016 at 2:00 PM, Ulf Lilleengen mailto:l...@redhat.com>> wrote:

My bad. This was inside the same container, but it was one I ran
manually with --net=host.

Here is from the one running in openshift:
sh-4.3$ cat /etc/resolv.conf
search enmasse.svc.cluster.local svc.cluster.local cluster.local
nameserver 192.168.1.17
options ndots:5

On 08/11/2016 07:44 PM, Clayton Coleman wrote:

Sorry, I meant resolve.conf inside of one of your containers

On Thu, Aug 11, 2016 at 1:35 PM, Ulf Lilleengen mailto:l...@redhat.com>
>> wrote:

[root@strappi /]# cat /etc/resolv.conf
# Generated by NetworkManager
search redhat.com  

nameserver 10.38.5.26
nameserver 10.35.255.14
nameserver 192.168.1.1
# NOTE: the libc resolver may not support more than 3
nameservers.
# The nameservers listed below may not be recognized.
nameserver 2001:4662:afe3:0:8a1f:a1ff:fe2d:34e6

Note that I set the disable_ipv6=1, but I didn't restart
openshift/docker after doing it though.

On 08/11/2016 05:48 PM, Clayton Coleman wrote:

Tried this with Fedora 24 and very similar config (but
centos7
image)
and I'm able to ping localhost.

$ cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1localhost
::1localhost ip6-localhost ip6-loopback
fe00::0ip6-localnet
fe00::0ip6-mcastprefix
fe00::1ip6-allnodes
fe00::2ip6-allrouters
172.17.0.2centoscentos7-debug

I have net.ipv6.conf.all.disable_ipv6 = 1 set - should
be possible.

Can you provide your /etc/resolv.conf from inside that
image?

On Thu, Aug 11, 2016 at 11:29 AM, Ulf Lilleengen
mailto:l...@redhat.com>
>


>

>>

>

 wrote:

Hi,

We were debugging an issue yesterday where
'localhost' could
not be
resolved inside a container in openshift origin
v1.3.0-alpha.3. I'm
not sure if this is openshift or
kubernetes-related, but
thought I'd
ask here first.

We have two containers running on a pod, and one
container is
connecting to the other using 'localhost'.
This has
worked
fine for
several months, but stopped working
yesterday. We
resolved
the issue
by usin