Re: Unable to start openshift in VM, AWS or google cloud

[Cesar Wong]

Looks like we're not displaying the public address after it starts.
However, you should be able to access it at the server's public address:

https://ec2-54-227-129-18.compute-1.amazonaws.com:8443/

On Nov 3, 2016, at 4:23 PM, Ravi Kapoor  wrote:

I tried this, it still starts at private IP address. My logs are copied
below notice that I want to start at 54.227.129.18 but server starts at
https://172.31.32.131:8443

What am I doing wrong?
--

[ec2-user@ip-172-31-32-131 openshift131]$ ./oc cluster up --public-hostname=
ec2-54-227-129-18.compute-1.amazonaws.com --routing-suffix=
54.227.129.18.xip.io --metrics
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.3.1 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ...
   Using nsenter mounter for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ...
   Using 172.31.32.131 as the server IP
-- Starting OpenShift container ...
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
   OpenShift server started
-- Installing registry ... OK
-- Installing router ... OK
-- Install Metrics ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Login to server ... OK
-- Creating initial project "myproject" ... OK
-- Server Information ...
   OpenShift server started.
   The server is accessible via web console at:
   https://172.31.32.131:8443

   The metrics service is available at:
   https://metrics-openshift-infra.54.227.129.18.xip.io

   You are logged in as:
   User: developer
   Password: developer

   To login as administrator:
   oc login -u system:admin


On Thu, Nov 3, 2016 at 9:00 AM, Cesar Wong  wrote:

> Hi Ravi,
>
> On AWS, the magic incantation is:
>
> oc cluster up --public-hostname=[public dns name] --routing-suffix=[ip
> address].xip.io
>
> Don't specify the numeric ip address in --public-hostname, rather the dns
> name.
>
> You can then access the web console at https://[public dns name]:8443/
>
>
> On Windows 7 + VirtualBox + Ubuntu ... is it unable to connect to a route
> because the address/port is not reachable or because the dns name is not
> found?
>
> On Nov 2, 2016, at 9:34 PM, Ravi  wrote:
>
>
> I am not able to start openshift, I tried three different ways.
>
> 1. Windows 7 + Virtual Box + Ubuntu
> oc cluster up works well. I went to console and launched nodejs-ex
> example. Console shows it is up, however when I click on route, it says
> "unable to connect". I tried going directly to POD's IP address and it does
> work. In other words, somehow load balancer was failing in virtualbox
> Ubuntu VM.
>
> 2. Then I moved on to AWS. I launched a RedHat image and installed docker
> and started openshift. Here, OC starts on private IP address, so I am not
> able to access it from public internet. I even tried
> oc cluster up --public-hostname='my ip address' but since the public ip
> address is some magic, oc is not able to detect etcd etc and fails.
>
> 3. Then I tried on google cloud. I faced exactly same issue as AWS.
>
> If any one of them works, I will be ok but no idea how to get past these
> issues.
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Unable to start openshift in VM, AWS or google cloud

[Ravi Kapoor]

> Have you installed a router? Does DNS or /etc/hosts for the route direct
your browser to the host's IP?

my browser is running within the same VM where openshift is starting. Also
it is able to load the console.
The issue is that when I do a deployment, I am not able to access the
route. I am able to ping the route hostname. I am also able to access pods
directly through their individual ip addresses.
Does that help?



On Thu, Nov 3, 2016 at 8:54 AM, Luke Meyer  wrote:

>
>
> On Wed, Nov 2, 2016 at 11:34 PM, Ravi  wrote:
>
>>
>> I am not able to start openshift, I tried three different ways.
>>
>> 1. Windows 7 + Virtual Box + Ubuntu
>> oc cluster up works well. I went to console and launched nodejs-ex
>> example. Console shows it is up, however when I click on route, it says
>> "unable to connect". I tried going directly to POD's IP address and it does
>> work. In other words, somehow load balancer was failing in virtualbox
>> Ubuntu VM.
>>
>
> Have you installed a router? Does DNS or /etc/hosts for the route direct
> your browser to the host's IP?
>
>
>>
>> 2. Then I moved on to AWS. I launched a RedHat image and installed docker
>> and started openshift. Here, OC starts on private IP address, so I am not
>> able to access it from public internet. I even tried
>> oc cluster up --public-hostname='my ip address' but since the public ip
>> address is some magic, oc is not able to detect etcd etc and fails.
>>
>> 3. Then I tried on google cloud. I faced exactly same issue as AWS.
>>
>> If any one of them works, I will be ok but no idea how to get past these
>> issues.
>>
>> ___
>> users mailing list
>> users@lists.openshift.redhat.com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>>
>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Unable to start openshift in VM, AWS or google cloud

[Ravi Kapoor]

I tried this, it still starts at private IP address. My logs are copied
below notice that I want to start at 54.227.129.18 but server starts at
https://172.31.32.131:8443

What am I doing wrong?
--

[ec2-user@ip-172-31-32-131 openshift131]$ ./oc cluster up --public-hostname=
ec2-54-227-129-18.compute-1.amazonaws.com --routing-suffix=
54.227.129.18.xip.io --metrics
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.3.1 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ...
   Using nsenter mounter for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ...
   Using 172.31.32.131 as the server IP
-- Starting OpenShift container ...
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
   OpenShift server started
-- Installing registry ... OK
-- Installing router ... OK
-- Install Metrics ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Login to server ... OK
-- Creating initial project "myproject" ... OK
-- Server Information ...
   OpenShift server started.
   The server is accessible via web console at:
   https://172.31.32.131:8443

   The metrics service is available at:
   https://metrics-openshift-infra.54.227.129.18.xip.io

   You are logged in as:
   User: developer
   Password: developer

   To login as administrator:
   oc login -u system:admin


On Thu, Nov 3, 2016 at 9:00 AM, Cesar Wong  wrote:

> Hi Ravi,
>
> On AWS, the magic incantation is:
>
> oc cluster up --public-hostname=[public dns name] --routing-suffix=[ip
> address].xip.io
>
> Don't specify the numeric ip address in --public-hostname, rather the dns
> name.
>
> You can then access the web console at https://[public dns name]:8443/
>
>
> On Windows 7 + VirtualBox + Ubuntu ... is it unable to connect to a route
> because the address/port is not reachable or because the dns name is not
> found?
>
> On Nov 2, 2016, at 9:34 PM, Ravi  wrote:
>
>
> I am not able to start openshift, I tried three different ways.
>
> 1. Windows 7 + Virtual Box + Ubuntu
> oc cluster up works well. I went to console and launched nodejs-ex
> example. Console shows it is up, however when I click on route, it says
> "unable to connect". I tried going directly to POD's IP address and it does
> work. In other words, somehow load balancer was failing in virtualbox
> Ubuntu VM.
>
> 2. Then I moved on to AWS. I launched a RedHat image and installed docker
> and started openshift. Here, OC starts on private IP address, so I am not
> able to access it from public internet. I even tried
> oc cluster up --public-hostname='my ip address' but since the public ip
> address is some magic, oc is not able to detect etcd etc and fails.
>
> 3. Then I tried on google cloud. I faced exactly same issue as AWS.
>
> If any one of them works, I will be ok but no idea how to get past these
> issues.
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Network segmentation

[Clayton Coleman]

Fluentd runs on the host network and communicates out (today) to reach
elastic search.  Elastic search is protected by authorization that denies
read/write access from random parties based on cluster level permissions.

On Thu, Nov 3, 2016 at 4:52 PM, Josh Baird  wrote:

> There would already be required 'holes' in the firewall for master<>node
> communication, so I don't see that putting them on separate networks really
> solves many security related issues.  Maybe I'm missing something, but I
> don't know that I'm gaining too much by placing the masters in one network,
> and the app/infra nodes in another (separated by physical firewalls).
>
> On a separate note - with ovs-multitenant enabled, how would
> logging/metrics work since they have to be in their own dedicated projects
> (not default), thus preventing them from accessing other projects where the
> application pods are located?
>
> Josh
>
> On Thu, Nov 3, 2016 at 4:19 PM, Ben Bennett  wrote:
>
>> The question is how much you trust container security.
>>
>> If someone managed to escape from a container they can send traffic to
>> any pod in the network from the node.  If the networks were physically
>> separated that would not be possible...
>>
>> OVS is enforcing the pod connectivity rules, and it can't be programmed
>> unless you have root on the box.  And the rules are pretty standard to
>> read, so they can be audited (there are just a bunch of them for each
>> pod).  In fact networking tends to fail such that packets don't flow rather
>> than flow too widely (I've never seen a case where pods can be reached that
>> should not; but I have debugged many many cases where networking doesn't
>> send packets).
>>
>> -ben
>>
>> On Thu, Nov 3, 2016 at 3:39 PM, Josh Baird  wrote:
>>
>>> Hi Ben,
>>>
>>> Thanks for the detailed response.
>>>
>>> Do you think it's worth the effort to have physical network separation
>>> between the infra/app nodes and the masters?  Or.. can the masters and
>>> nodes both coexist in the same physical network without security concerns?
>>>
>>> Thanks,
>>>
>>> Josh
>>>
>>> On Thu, Nov 3, 2016 at 9:44 AM, Ben Bennett  wrote:
>>>
 Good question, I'll take a crack below, and I've cc'ed the rest of the
 networking team in case they have any other ideas.


 On Tue, Nov 1, 2016 at 6:30 PM, Josh Baird  wrote:

> Hi,
>
> As a follow-up to my previous post [1], I'm curious to hear how others
> are handling network segmentation in general with OpenShift.  I'm 
> concerned
> with segmenting external based applications from internal
> assets/networks/hosts.
>

>
 A general practice is to place external/public facing applications in
> some type of 'DMZ' network to protect your internal assets.  How are you
> accomplishing this with OpenShift?
>
> A few ways that I can think of:
>
> * Deploy entirely separate clusters for public facing applications and
> internal (private) applications and completely isolate the networks 
> between
> the two. This seems like it may be over-kill and costly both technically
> and financially.
>

 I agree that should be unnecessary, and will be rather expensive.  But
 depending upon the regulatory / security environment I suppose it could be
 necessary.



> * Deploy separate nodes (same cluster) for public facing applications
> and internal applications.  The nodes dedicated to public facing apps 
> would
> be located in a DMZ/restricted network.  In the event of a container break
> or compromise on the public nodes, this option would at least provide some
> level of protection for your internal assets.
>
>
 This is a reasonable approach, but will require that people are
 fastidious about labeling things correctly.



> * Deploy dedicated routers for 'internal' and 'external' and use a
> combination of edge load balancing/reverse proxy/firewall ACLs to separate
> the two (as my previous post suggests).  This approach really just 
> protects
> internal applications from being exposed to the internet, but doesn't
> protect internal assets from being exposed should a public application 
> node
> (or container) be compromised.
>
>
 If you use the mutli-tenant networking plugin, each project is isolated
 from the others by the OVS rules.  Pods in the default project can talk to
 the any project (and vice-versa) so if you place multiple routers into the
 default project and use sharding to select which namespaces are served by
 each router, then you could have a private router (or routers) that are
 reachable only within the cluster (or you could expose them externally, but
 only within your corporate network).  Then if a pod reachable through the
 public router was compromised, it could only talk to things in its project
 and the default project.

Re: Network segmentation

[Josh Baird]

There would already be required 'holes' in the firewall for master<>node
communication, so I don't see that putting them on separate networks really
solves many security related issues.  Maybe I'm missing something, but I
don't know that I'm gaining too much by placing the masters in one network,
and the app/infra nodes in another (separated by physical firewalls).

On a separate note - with ovs-multitenant enabled, how would
logging/metrics work since they have to be in their own dedicated projects
(not default), thus preventing them from accessing other projects where the
application pods are located?

Josh

On Thu, Nov 3, 2016 at 4:19 PM, Ben Bennett  wrote:

> The question is how much you trust container security.
>
> If someone managed to escape from a container they can send traffic to any
> pod in the network from the node.  If the networks were physically
> separated that would not be possible...
>
> OVS is enforcing the pod connectivity rules, and it can't be programmed
> unless you have root on the box.  And the rules are pretty standard to
> read, so they can be audited (there are just a bunch of them for each
> pod).  In fact networking tends to fail such that packets don't flow rather
> than flow too widely (I've never seen a case where pods can be reached that
> should not; but I have debugged many many cases where networking doesn't
> send packets).
>
> -ben
>
> On Thu, Nov 3, 2016 at 3:39 PM, Josh Baird  wrote:
>
>> Hi Ben,
>>
>> Thanks for the detailed response.
>>
>> Do you think it's worth the effort to have physical network separation
>> between the infra/app nodes and the masters?  Or.. can the masters and
>> nodes both coexist in the same physical network without security concerns?
>>
>> Thanks,
>>
>> Josh
>>
>> On Thu, Nov 3, 2016 at 9:44 AM, Ben Bennett  wrote:
>>
>>> Good question, I'll take a crack below, and I've cc'ed the rest of the
>>> networking team in case they have any other ideas.
>>>
>>>
>>> On Tue, Nov 1, 2016 at 6:30 PM, Josh Baird  wrote:
>>>
 Hi,

 As a follow-up to my previous post [1], I'm curious to hear how others
 are handling network segmentation in general with OpenShift.  I'm concerned
 with segmenting external based applications from internal
 assets/networks/hosts.

>>>

>>> A general practice is to place external/public facing applications in
 some type of 'DMZ' network to protect your internal assets.  How are you
 accomplishing this with OpenShift?

 A few ways that I can think of:

 * Deploy entirely separate clusters for public facing applications and
 internal (private) applications and completely isolate the networks between
 the two. This seems like it may be over-kill and costly both technically
 and financially.

>>>
>>> I agree that should be unnecessary, and will be rather expensive.  But
>>> depending upon the regulatory / security environment I suppose it could be
>>> necessary.
>>>
>>>
>>>
 * Deploy separate nodes (same cluster) for public facing applications
 and internal applications.  The nodes dedicated to public facing apps would
 be located in a DMZ/restricted network.  In the event of a container break
 or compromise on the public nodes, this option would at least provide some
 level of protection for your internal assets.


>>> This is a reasonable approach, but will require that people are
>>> fastidious about labeling things correctly.
>>>
>>>
>>>
 * Deploy dedicated routers for 'internal' and 'external' and use a
 combination of edge load balancing/reverse proxy/firewall ACLs to separate
 the two (as my previous post suggests).  This approach really just protects
 internal applications from being exposed to the internet, but doesn't
 protect internal assets from being exposed should a public application node
 (or container) be compromised.


>>> If you use the mutli-tenant networking plugin, each project is isolated
>>> from the others by the OVS rules.  Pods in the default project can talk to
>>> the any project (and vice-versa) so if you place multiple routers into the
>>> default project and use sharding to select which namespaces are served by
>>> each router, then you could have a private router (or routers) that are
>>> reachable only within the cluster (or you could expose them externally, but
>>> only within your corporate network).  Then if a pod reachable through the
>>> public router was compromised, it could only talk to things in its project
>>> and the default project.
>>>
>>>   https://docs.openshift.com/container-platform/3.3/architectu
>>> re/additional_concepts/sdn.html
>>>   https://docs.openshift.com/container-platform/3.3/architectu
>>> re/core_concepts/routes.html#router-sharding
>>>
>>> If you need some pods to access resources that are outside the cluster,
>>> but in your private network... but not all pods.  The new "egress firewall"
>>> feature allows you to put rules on names

Re: Wrong resource consumption on scheduler

[Clayton Coleman]

No, but the global default for any project will steer you away from infra
nodes.

On Wed, Nov 2, 2016 at 9:32 AM, Frank Liauw  wrote:

> No, it does not. Are nodes without region labels automatically classified
> as infra region?
>
> Frank
> Systems Engineer
>
> VSee: fr...@vsee.com  | Cell: +65 9338 0035
>
> Join me on VSee for Free 
>
>
>
>
> On Wed, Nov 2, 2016 at 9:24 PM, Clayton Coleman 
> wrote:
>
>> Does your namespace have a namespace restriction annotation set?  If not,
>> you'll be defaulted to the global restriction which is usually excluding
>> the infra region.
>>
>> On Nov 2, 2016, at 8:14 AM, Frank Liauw  wrote:
>>
>> There's no node selector on the pod. The pod is under a service.
>>
>> The affinity rules are left unmodified from install:
>>
>> {
>> "predicates": [{
>> "name": "MatchNodeSelector"
>> }, {
>> "name": "PodFitsResources"
>> }, {
>> "name": "PodFitsPorts"
>> }, {
>> "name": "NoDiskConflict"
>> }, {
>> "argument": {
>> "serviceAffinity": {
>> "labels": ["region"]
>> }
>> },
>> "name": "Region"
>> }],
>> "kind": "Policy",
>> "priorities": [{
>> "name": "LeastRequestedPriority",
>> "weight": 1
>> }, {
>> "name": "SelectorSpreadPriority",
>> "weight": 1
>> }, {
>> "argument": {
>> "serviceAntiAffinity": {
>> "label": "zone"
>> }
>> },
>> "weight": 2,
>> "name": "Zone"
>> }],
>> "apiVersion": "v1"
>> }
>>
>> It puzzles me more that my custom labels as well as the extra labels were
>> not introduced in previous scaleup runs by ansible; it's not the first time
>> I'm adding new nodes to the cluster.
>>
>> Frank
>> Systems Engineer
>>
>> VSee: fr...@vsee.com  | Cell: +65 9338 0035
>>
>> Join me on VSee for Free 
>>
>>
>>
>>
>> On Fri, Oct 28, 2016 at 9:47 PM, Clayton Coleman 
>> wrote:
>>
>>> What node selector / tolerations / affinity rules were on your pod?  Is
>>> the pod under a service?
>>>
>>> On Oct 28, 2016, at 4:03 AM, Frank Liauw  wrote:
>>>
>>> After giving the event log some thought, I realised that the 'Region fit
>>> failure' was a different error as 'Node didn't have enough resource', and
>>> realised that openshift is trying to force deployment onto
>>> node4.openshift.internal, a node I added recently.
>>>
>>> My nodes had these sets of labels:
>>>
>>> kubernetes.io/hostname=node1.openshift.internal,logging-infr
>>> a-fluentd=true
>>> kubernetes.io/hostname=node2.openshift.internal,logging-infr
>>> a-fluentd=true,public-router=true
>>> core-router=true,kubernetes.io/hostname=node3.openshift.inte
>>> rnal,logging-infra-fluentd=true
>>> beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ku
>>> bernetes.io/hostname=node4.openshift.internal,logging-infra-
>>> fluentd=true,region=primary,zone=layer42
>>> kubernetes.io/hostname=node2.openshift.ncal,logging-infra-fl
>>> uentd=true,public-router=true
>>>
>>> Removing the labels 'beta.kubernetes.io/arch=amd64
>>> ,beta.kubernetes.io/os=linux' fixed the issue.
>>>
>>> Why is it so?
>>>
>>> Frank
>>> Systems Engineer
>>>
>>> VSee: fr...@vsee.com  | Cell: +65 9338 0035
>>>
>>> Join me on VSee for Free 
>>>
>>>
>>>
>>>
>>> On Fri, Oct 28, 2016 at 3:50 PM, Frank Liauw  wrote:
>>>
 Hi,

 My pods are not deploying despite there being plenty of spare resources
 on my nodes; the event log on the pod seems to report much higher resource
 usage than what I'm seeing on my node:

 pod (redshiftlogger-14-8zaty) failed to fit in any node fit failure on
 node (node1.openshift.internal): Region fit failure on node
 (node2.openshift.internal): Region fit failure on node
 (node2.openshift.ncal): Region fit failure on node
 (node3.openshift.internal): Region fit failure on node
 (node4.openshift.internal): Node didn't have enough resource: Memory,
 requested: 2147483648, used: 32448771072, capacity: 32511942656

 Node   CPU Requests CPU Limits Memory
 Requests Memory Limits
 --    -- --- -
 node1.openshift.internal  100m (1%) 100m (1%) 2560Mi (8%) 10Gi (33%)
 node2.openshift.internal  100m (1%) 100m (1%) 5623141Ki (46%) 13487461Ki
 (111%)
 node3.openshift.internal  200m (2%) 200m (2%) 7680Mi (48%) 15Gi (96%)
 node4.openshift.internal  100m (1%) 100m (1%) 32448771072 (99%) 32448771072
 (99%)
 node2.openshift.ncal   100m (2%) 100m (2%) 4147483648 (25%)
 4147483648 (25%)

 Restarting the master didn't help; I restarted one of two masters.

 Frank
 Systems Engineer

 VSee: fr...@vsee.com  | Cell: +65 9338 0035

Re: Network segmentation

[Ben Bennett]

The question is how much you trust container security.

If someone managed to escape from a container they can send traffic to any
pod in the network from the node.  If the networks were physically
separated that would not be possible...

OVS is enforcing the pod connectivity rules, and it can't be programmed
unless you have root on the box.  And the rules are pretty standard to
read, so they can be audited (there are just a bunch of them for each
pod).  In fact networking tends to fail such that packets don't flow rather
than flow too widely (I've never seen a case where pods can be reached that
should not; but I have debugged many many cases where networking doesn't
send packets).

-ben

On Thu, Nov 3, 2016 at 3:39 PM, Josh Baird  wrote:

> Hi Ben,
>
> Thanks for the detailed response.
>
> Do you think it's worth the effort to have physical network separation
> between the infra/app nodes and the masters?  Or.. can the masters and
> nodes both coexist in the same physical network without security concerns?
>
> Thanks,
>
> Josh
>
> On Thu, Nov 3, 2016 at 9:44 AM, Ben Bennett  wrote:
>
>> Good question, I'll take a crack below, and I've cc'ed the rest of the
>> networking team in case they have any other ideas.
>>
>>
>> On Tue, Nov 1, 2016 at 6:30 PM, Josh Baird  wrote:
>>
>>> Hi,
>>>
>>> As a follow-up to my previous post [1], I'm curious to hear how others
>>> are handling network segmentation in general with OpenShift.  I'm concerned
>>> with segmenting external based applications from internal
>>> assets/networks/hosts.
>>>
>>
>>>
>> A general practice is to place external/public facing applications in
>>> some type of 'DMZ' network to protect your internal assets.  How are you
>>> accomplishing this with OpenShift?
>>>
>>> A few ways that I can think of:
>>>
>>> * Deploy entirely separate clusters for public facing applications and
>>> internal (private) applications and completely isolate the networks between
>>> the two. This seems like it may be over-kill and costly both technically
>>> and financially.
>>>
>>
>> I agree that should be unnecessary, and will be rather expensive.  But
>> depending upon the regulatory / security environment I suppose it could be
>> necessary.
>>
>>
>>
>>> * Deploy separate nodes (same cluster) for public facing applications
>>> and internal applications.  The nodes dedicated to public facing apps would
>>> be located in a DMZ/restricted network.  In the event of a container break
>>> or compromise on the public nodes, this option would at least provide some
>>> level of protection for your internal assets.
>>>
>>>
>> This is a reasonable approach, but will require that people are
>> fastidious about labeling things correctly.
>>
>>
>>
>>> * Deploy dedicated routers for 'internal' and 'external' and use a
>>> combination of edge load balancing/reverse proxy/firewall ACLs to separate
>>> the two (as my previous post suggests).  This approach really just protects
>>> internal applications from being exposed to the internet, but doesn't
>>> protect internal assets from being exposed should a public application node
>>> (or container) be compromised.
>>>
>>>
>> If you use the mutli-tenant networking plugin, each project is isolated
>> from the others by the OVS rules.  Pods in the default project can talk to
>> the any project (and vice-versa) so if you place multiple routers into the
>> default project and use sharding to select which namespaces are served by
>> each router, then you could have a private router (or routers) that are
>> reachable only within the cluster (or you could expose them externally, but
>> only within your corporate network).  Then if a pod reachable through the
>> public router was compromised, it could only talk to things in its project
>> and the default project.
>>
>>   https://docs.openshift.com/container-platform/3.3/architectu
>> re/additional_concepts/sdn.html
>>   https://docs.openshift.com/container-platform/3.3/architectu
>> re/core_concepts/routes.html#router-sharding
>>
>> If you need some pods to access resources that are outside the cluster,
>> but in your private network... but not all pods.  The new "egress firewall"
>> feature allows you to put rules on namespaces to control what they can and
>> can not reach.
>>
>>   https://docs.openshift.com/container-platform/3.3/admin_guid
>> e/managing_pods.html#admin-guide-limit-pod-access-egress
>>
>> Or if you must have IP-based access to an external resource, but only
>> want some pods to be able to reach it, the "egress router" can be useful:
>>
>>   https://docs.openshift.com/container-platform/3.3/admin_guid
>> e/managing_pods.html#admin-guide-limit-pod-access-egress-router
>>
>>
>> The problem with this approach is that you can't say that one project can
>> talk to a service in a different project.  We are working on this by being
>> heavily involved with the Kubernetes Networking SIG and defining the
>> NetworkPolicy and the EgressPolicy objects.  You can use a private router
>> (in

Re: Network segmentation

[Josh Baird]

Hi Ben,

Thanks for the detailed response.

Do you think it's worth the effort to have physical network separation
between the infra/app nodes and the masters?  Or.. can the masters and
nodes both coexist in the same physical network without security concerns?

Thanks,

Josh

On Thu, Nov 3, 2016 at 9:44 AM, Ben Bennett  wrote:

> Good question, I'll take a crack below, and I've cc'ed the rest of the
> networking team in case they have any other ideas.
>
>
> On Tue, Nov 1, 2016 at 6:30 PM, Josh Baird  wrote:
>
>> Hi,
>>
>> As a follow-up to my previous post [1], I'm curious to hear how others
>> are handling network segmentation in general with OpenShift.  I'm concerned
>> with segmenting external based applications from internal
>> assets/networks/hosts.
>>
>
>>
> A general practice is to place external/public facing applications in some
>> type of 'DMZ' network to protect your internal assets.  How are you
>> accomplishing this with OpenShift?
>>
>> A few ways that I can think of:
>>
>> * Deploy entirely separate clusters for public facing applications and
>> internal (private) applications and completely isolate the networks between
>> the two. This seems like it may be over-kill and costly both technically
>> and financially.
>>
>
> I agree that should be unnecessary, and will be rather expensive.  But
> depending upon the regulatory / security environment I suppose it could be
> necessary.
>
>
>
>> * Deploy separate nodes (same cluster) for public facing applications and
>> internal applications.  The nodes dedicated to public facing apps would be
>> located in a DMZ/restricted network.  In the event of a container break or
>> compromise on the public nodes, this option would at least provide some
>> level of protection for your internal assets.
>>
>>
> This is a reasonable approach, but will require that people are fastidious
> about labeling things correctly.
>
>
>
>> * Deploy dedicated routers for 'internal' and 'external' and use a
>> combination of edge load balancing/reverse proxy/firewall ACLs to separate
>> the two (as my previous post suggests).  This approach really just protects
>> internal applications from being exposed to the internet, but doesn't
>> protect internal assets from being exposed should a public application node
>> (or container) be compromised.
>>
>>
> If you use the mutli-tenant networking plugin, each project is isolated
> from the others by the OVS rules.  Pods in the default project can talk to
> the any project (and vice-versa) so if you place multiple routers into the
> default project and use sharding to select which namespaces are served by
> each router, then you could have a private router (or routers) that are
> reachable only within the cluster (or you could expose them externally, but
> only within your corporate network).  Then if a pod reachable through the
> public router was compromised, it could only talk to things in its project
> and the default project.
>
>   https://docs.openshift.com/container-platform/3.3/
> architecture/additional_concepts/sdn.html
>   https://docs.openshift.com/container-platform/3.3/
> architecture/core_concepts/routes.html#router-sharding
>
> If you need some pods to access resources that are outside the cluster,
> but in your private network... but not all pods.  The new "egress firewall"
> feature allows you to put rules on namespaces to control what they can and
> can not reach.
>
>   https://docs.openshift.com/container-platform/3.3/admin_
> guide/managing_pods.html#admin-guide-limit-pod-access-egress
>
> Or if you must have IP-based access to an external resource, but only want
> some pods to be able to reach it, the "egress router" can be useful:
>
>   https://docs.openshift.com/container-platform/3.3/admin_
> guide/managing_pods.html#admin-guide-limit-pod-access-egress-router
>
>
> The problem with this approach is that you can't say that one project can
> talk to a service in a different project.  We are working on this by being
> heavily involved with the Kubernetes Networking SIG and defining the
> NetworkPolicy and the EgressPolicy objects.  You can use a private router
> (in the default namespace) but then anyone can access the resource, and it
> must be http/https/tls (with SNI).  You could also make a NodePort for a
> service and then it could carry any IP traffic... but again, it would be
> visible to all pods in the cluster.
>
>
>
>> * Another option may be combination of these where the entire OpenShift
>> environment (application nodes only?) is isolated from the rest of the
>> internal/private network for both public and private applications.  You
>> then deploy multiple router-sets for internal/external as described above.
>>
>>
> That could work too, but I would look at the egress router first, unless I
> am misunderstanding what you mean by "rest of the internal/private network"
> (I assume you mean, outside the cluster, but inside my organization's
> private network).
>
>
> -ben
>
>
>
>> What are everyone's 

Re: Openshift discovery

[Clayton Coleman]

prerequisites.html

covers
some of the implications here.

On Thu, Nov 3, 2016 at 1:36 PM, Clayton Coleman  wrote:

> Not sure from your output but ff the first entry in the same server isn't
> the openshift master address, then Alpine will fail because it doesn't try
> multiple name servers for things under cluster.local.  But it *might* be
> trying a random one, in which case the only solution for alpine is to set
> up dnsmasq on the node and have only a single entry in resolv.conf
> (dnsmasq) that points to the master and then your internal dns entries.
>
> On Thu, Nov 3, 2016 at 1:30 PM, Srinivas Naga Kotaru (skotaru) <
> skot...@cisco.com> wrote:
>
>>
>>
>>
>>
>> % oc get svc
>>
>> NAMECLUSTER-IP EXTERNAL-IP   PORT(S)AGE
>>
>> net-tools   172.30.112.9   8080/TCP   18h
>>
>>
>>
>> / $ cat /etc/resolv.conf
>>
>> search sd-testing.svc.cluster.local svc.cluster.local cluster.local
>> cisco.com
>>
>> nameserver 173.36.96.19
>>
>> nameserver 173.37.137.85
>>
>> nameserver 173.37.142.73
>>
>> nameserver 173.37.87.157
>>
>> options timeout:1 attempts:1
>>
>> options ndots:5
>>
>>
>>
>> / $ dig +short net-tools.sd-testing.svc.cluster.local
>>
>> 172.30.112.9
>>
>>
>>
>> / $ dig +short yahoo.com
>>
>>
>>
>> / $ curl -I yahoo.com
>>
>> HTTP/1.1 301 Moved Permanently
>>
>> Date: Thu, 03 Nov 2016 17:22:30 GMT
>>
>> Server: ATS
>>
>> Location: https://www.yahoo.com/
>>
>> Content-Language: en
>>
>> Cache-Control: no-store, no-cache
>>
>> Content-Length: 304
>>
>> Content-Type: text/html
>>
>> Via: https/1.1 ir37.fp.ne1.yahoo.com (ApacheTrafficServer), 1.1
>> alln01-mda1-dmz-wsa-2.cisco.com:80 (Cisco-WSA/9.0.1-162)
>>
>> Connection: keep-alive
>>
>>
>>
>>
>>
>> $ nslookup 173.37.137.85
>>
>> Server:   173.36.96.19
>>
>> Address:173.36.96.19#53
>>
>>
>>
>> ** server can't find 85.137.37.173.in-addr.arpa: REFUSED
>>
>>
>>
>> / $ nslookup 173.36.96.19
>>
>> Server:   173.36.96.19
>>
>> Address:173.36.96.19#53
>>
>>
>>
>> 19.96.36.173.in-addr.arpa  name = l3ipn-id2-002.cisco.com.
>>
>>
>>
>>
>>
>> It seems to be working but didn’t understand why dns resolution against
>> other entries in /etc/resolve.conf saying server can’t find. Last 3 entries
>> in /etc/resolve.conf are our enterprise DNS servers, which might be
>> automatically added to container /etc/resolv.conf from host /etc/resolv.conf
>>
>>
>>
>> --
>>
>> *Srinivas Kotaru*
>>
>>
>>
>> *From: *"ccole...@redhat.com" 
>> *Date: *Thursday, November 3, 2016 at 10:11 AM
>>
>> *To: *Srinivas Naga Kotaru 
>> *Cc: *"users@lists.openshift.redhat.com" > com>
>> *Subject: *Re: Openshift discovery
>>
>>
>>
>> Can you show me the output of dig for kubernetes.default.svc.cluster.local
>> AND contents of resolv.conf?
>>
>>
>>
>> On Thu, Nov 3, 2016 at 12:38 PM, Srinivas Naga Kotaru (skotaru) <
>> skot...@cisco.com> wrote:
>>
>> SKOTARU-M-H06U:~ $ oc get pods
>>
>> NAMEREADY STATUS RESTARTS   AGE
>>
>> net-tools-1-pp4t4   0/1   CrashLoopBackOff   20817h
>>
>> SKOTARU-M-H06U:~ $
>>
>>
>>
>> SKOTARU-M-H06U:~ $ oc debug net-tools-1-pp4t4
>>
>> Debugging with pod/net-tools-1-pp4t4-debug, original command: sh
>>
>> Waiting for pod to start ...
>>
>> Pod IP: 10.1.4.10
>>
>> If you don't see a command prompt, try pressing enter.
>>
>>
>>
>> / $ dig
>>
>>
>>
>> ; <<>> DiG 9.10.4-P3 <<>>
>>
>> ;; global options: +cmd
>>
>> ;; Got answer:
>>
>> ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 18102
>>
>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
>>
>>
>>
>> ;; QUESTION SECTION:
>>
>> ;.
>> INNS
>>
>>
>>
>> ;; Query time: 0 msec
>>
>> ;; SERVER: 173.36.96.19#53(173.36.96.19)
>>
>> ;; WHEN: Thu Nov 03 16:37:12 UTC 2016
>>
>> ;; MSG SIZE  rcvd: 17
>>
>>
>>
>>
>>
>> --
>>
>> *Srinivas Kotaru*
>>
>>
>>
>> *From: *"ccole...@redhat.com" 
>> *Date: *Thursday, November 3, 2016 at 7:02 AM
>> *To: *Srinivas Naga Kotaru 
>> *Cc: *"users@lists.openshift.redhat.com" > com>
>> *Subject: *Re: Openshift discovery
>>
>>
>>
>> If you "oc debug" the crashing pods, do you get a shell up?
>>
>>
>> On Nov 3, 2016, at 9:56 AM, Srinivas Naga Kotaru (skotaru) <
>> skot...@cisco.com> wrote:
>>
>> Clayton
>>
>>
>>
>> Sorry for confusion. Original problem was, Service discovery not working
>> in regular openshift apps. Out of the box images as well as custom images.
>>
>>
>>
>> I was trying to build a image with a net tools for debugging, so it is
>> easy for troubleshoot as out of the box images does not have basic net
>> tools. Openshift throwing crash recovery for any image I build, so I might
>> be doing some mistake.  These images working fine in standard docker.
>>
>>
>>
>>
>> Sent from my iPhone
>>
>>
>> On Nov 3, 2016, at 6:24 AM, Clayton Coleman  wrote:
>>
>> Alpine uses musl which has known differences from glibc in how i

Re: Openshift discovery

[Clayton Coleman]

Not sure from your output but ff the first entry in the same server isn't
the openshift master address, then Alpine will fail because it doesn't try
multiple name servers for things under cluster.local.  But it *might* be
trying a random one, in which case the only solution for alpine is to set
up dnsmasq on the node and have only a single entry in resolv.conf
(dnsmasq) that points to the master and then your internal dns entries.

On Thu, Nov 3, 2016 at 1:30 PM, Srinivas Naga Kotaru (skotaru) <
skot...@cisco.com> wrote:

>
>
>
>
> % oc get svc
>
> NAMECLUSTER-IP EXTERNAL-IP   PORT(S)AGE
>
> net-tools   172.30.112.9   8080/TCP   18h
>
>
>
> / $ cat /etc/resolv.conf
>
> search sd-testing.svc.cluster.local svc.cluster.local cluster.local
> cisco.com
>
> nameserver 173.36.96.19
>
> nameserver 173.37.137.85
>
> nameserver 173.37.142.73
>
> nameserver 173.37.87.157
>
> options timeout:1 attempts:1
>
> options ndots:5
>
>
>
> / $ dig +short net-tools.sd-testing.svc.cluster.local
>
> 172.30.112.9
>
>
>
> / $ dig +short yahoo.com
>
>
>
> / $ curl -I yahoo.com
>
> HTTP/1.1 301 Moved Permanently
>
> Date: Thu, 03 Nov 2016 17:22:30 GMT
>
> Server: ATS
>
> Location: https://www.yahoo.com/
>
> Content-Language: en
>
> Cache-Control: no-store, no-cache
>
> Content-Length: 304
>
> Content-Type: text/html
>
> Via: https/1.1 ir37.fp.ne1.yahoo.com (ApacheTrafficServer), 1.1
> alln01-mda1-dmz-wsa-2.cisco.com:80 (Cisco-WSA/9.0.1-162)
>
> Connection: keep-alive
>
>
>
>
>
> $ nslookup 173.37.137.85
>
> Server:   173.36.96.19
>
> Address:173.36.96.19#53
>
>
>
> ** server can't find 85.137.37.173.in-addr.arpa: REFUSED
>
>
>
> / $ nslookup 173.36.96.19
>
> Server:   173.36.96.19
>
> Address:173.36.96.19#53
>
>
>
> 19.96.36.173.in-addr.arpa  name = l3ipn-id2-002.cisco.com.
>
>
>
>
>
> It seems to be working but didn’t understand why dns resolution against
> other entries in /etc/resolve.conf saying server can’t find. Last 3 entries
> in /etc/resolve.conf are our enterprise DNS servers, which might be
> automatically added to container /etc/resolv.conf from host /etc/resolv.conf
>
>
>
> --
>
> *Srinivas Kotaru*
>
>
>
> *From: *"ccole...@redhat.com" 
> *Date: *Thursday, November 3, 2016 at 10:11 AM
>
> *To: *Srinivas Naga Kotaru 
> *Cc: *"users@lists.openshift.redhat.com"  >
> *Subject: *Re: Openshift discovery
>
>
>
> Can you show me the output of dig for kubernetes.default.svc.cluster.local
> AND contents of resolv.conf?
>
>
>
> On Thu, Nov 3, 2016 at 12:38 PM, Srinivas Naga Kotaru (skotaru) <
> skot...@cisco.com> wrote:
>
> SKOTARU-M-H06U:~ $ oc get pods
>
> NAMEREADY STATUS RESTARTS   AGE
>
> net-tools-1-pp4t4   0/1   CrashLoopBackOff   20817h
>
> SKOTARU-M-H06U:~ $
>
>
>
> SKOTARU-M-H06U:~ $ oc debug net-tools-1-pp4t4
>
> Debugging with pod/net-tools-1-pp4t4-debug, original command: sh
>
> Waiting for pod to start ...
>
> Pod IP: 10.1.4.10
>
> If you don't see a command prompt, try pressing enter.
>
>
>
> / $ dig
>
>
>
> ; <<>> DiG 9.10.4-P3 <<>>
>
> ;; global options: +cmd
>
> ;; Got answer:
>
> ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 18102
>
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
>
>
>
> ;; QUESTION SECTION:
>
> ;.
> INNS
>
>
>
> ;; Query time: 0 msec
>
> ;; SERVER: 173.36.96.19#53(173.36.96.19)
>
> ;; WHEN: Thu Nov 03 16:37:12 UTC 2016
>
> ;; MSG SIZE  rcvd: 17
>
>
>
>
>
> --
>
> *Srinivas Kotaru*
>
>
>
> *From: *"ccole...@redhat.com" 
> *Date: *Thursday, November 3, 2016 at 7:02 AM
> *To: *Srinivas Naga Kotaru 
> *Cc: *"users@lists.openshift.redhat.com"  >
> *Subject: *Re: Openshift discovery
>
>
>
> If you "oc debug" the crashing pods, do you get a shell up?
>
>
> On Nov 3, 2016, at 9:56 AM, Srinivas Naga Kotaru (skotaru) <
> skot...@cisco.com> wrote:
>
> Clayton
>
>
>
> Sorry for confusion. Original problem was, Service discovery not working
> in regular openshift apps. Out of the box images as well as custom images.
>
>
>
> I was trying to build a image with a net tools for debugging, so it is
> easy for troubleshoot as out of the box images does not have basic net
> tools. Openshift throwing crash recovery for any image I build, so I might
> be doing some mistake.  These images working fine in standard docker.
>
>
>
>
> Sent from my iPhone
>
>
> On Nov 3, 2016, at 6:24 AM, Clayton Coleman  wrote:
>
> Alpine uses musl which has known differences from glibc in how it handles
> DNS resolution.  *usually* this is because multiple  nameservers are listed
> in resolv.conf and the first one doesn't answer queries for
> *svc.cluster.local.  You can check that by execing into containers and
> looking at the resolv.conf.
>
>
>
> In 3.3, at the host level we configure dnsmasq by default to offer a
> single resolver (so musl doesn't get confused).  You can check how that is
> configured on your hosts.
>
>
> On Nov 2, 2016, at 5:06 PM, S

Re: Openshift discovery

[Srinivas Naga Kotaru (skotaru)]

% oc get svc
NAMECLUSTER-IP EXTERNAL-IP   PORT(S)AGE
net-tools   172.30.112.9   8080/TCP   18h

/ $ cat /etc/resolv.conf
search sd-testing.svc.cluster.local svc.cluster.local cluster.local cisco.com
nameserver 173.36.96.19
nameserver 173.37.137.85
nameserver 173.37.142.73
nameserver 173.37.87.157
options timeout:1 attempts:1
options ndots:5

/ $ dig +short net-tools.sd-testing.svc.cluster.local
172.30.112.9

/ $ dig +short yahoo.com

/ $ curl -I yahoo.com
HTTP/1.1 301 Moved Permanently
Date: Thu, 03 Nov 2016 17:22:30 GMT
Server: ATS
Location: https://www.yahoo.com/
Content-Language: en
Cache-Control: no-store, no-cache
Content-Length: 304
Content-Type: text/html
Via: https/1.1 ir37.fp.ne1.yahoo.com (ApacheTrafficServer), 1.1 
alln01-mda1-dmz-wsa-2.cisco.com:80 (Cisco-WSA/9.0.1-162)
Connection: keep-alive


$ nslookup 173.37.137.85
Server:   173.36.96.19
Address:173.36.96.19#53

** server can't find 85.137.37.173.in-addr.arpa: REFUSED

/ $ nslookup 173.36.96.19
Server:   173.36.96.19
Address:173.36.96.19#53

19.96.36.173.in-addr.arpa  name = l3ipn-id2-002.cisco.com.


It seems to be working but didn’t understand why dns resolution against other 
entries in /etc/resolve.conf saying server can’t find. Last 3 entries in 
/etc/resolve.conf are our enterprise DNS servers, which might be automatically 
added to container /etc/resolv.conf from host /etc/resolv.conf

--
Srinivas Kotaru

From: "ccole...@redhat.com" 
Date: Thursday, November 3, 2016 at 10:11 AM
To: Srinivas Naga Kotaru 
Cc: "users@lists.openshift.redhat.com" 
Subject: Re: Openshift discovery

Can you show me the output of dig for kubernetes.default.svc.cluster.local AND 
contents of resolv.conf?

On Thu, Nov 3, 2016 at 12:38 PM, Srinivas Naga Kotaru (skotaru) 
mailto:skot...@cisco.com>> wrote:
SKOTARU-M-H06U:~ $ oc get pods
NAMEREADY STATUS RESTARTS   AGE
net-tools-1-pp4t4   0/1   CrashLoopBackOff   20817h
SKOTARU-M-H06U:~ $

SKOTARU-M-H06U:~ $ oc debug net-tools-1-pp4t4
Debugging with pod/net-tools-1-pp4t4-debug, original command: sh
Waiting for pod to start ...
Pod IP: 10.1.4.10
If you don't see a command prompt, try pressing enter.

/ $ dig

; <<>> DiG 9.10.4-P3 <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 18102
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;. INNS

;; Query time: 0 msec
;; SERVER: 173.36.96.19#53(173.36.96.19)
;; WHEN: Thu Nov 03 16:37:12 UTC 2016
;; MSG SIZE  rcvd: 17


--
Srinivas Kotaru

From: "ccole...@redhat.com" 
mailto:ccole...@redhat.com>>
Date: Thursday, November 3, 2016 at 7:02 AM
To: Srinivas Naga Kotaru mailto:skot...@cisco.com>>
Cc: "users@lists.openshift.redhat.com" 
mailto:users@lists.openshift.redhat.com>>
Subject: Re: Openshift discovery

If you "oc debug" the crashing pods, do you get a shell up?

On Nov 3, 2016, at 9:56 AM, Srinivas Naga Kotaru (skotaru) 
mailto:skot...@cisco.com>> wrote:
Clayton

Sorry for confusion. Original problem was, Service discovery not working in 
regular openshift apps. Out of the box images as well as custom images.

I was trying to build a image with a net tools for debugging, so it is easy for 
troubleshoot as out of the box images does not have basic net tools. Openshift 
throwing crash recovery for any image I build, so I might be doing some 
mistake.  These images working fine in standard docker.


Sent from my iPhone

On Nov 3, 2016, at 6:24 AM, Clayton Coleman 
mailto:ccole...@redhat.com>> wrote:
Alpine uses musl which has known differences from glibc in how it handles DNS 
resolution.  *usually* this is because multiple  nameservers are listed in 
resolv.conf and the first one doesn't answer queries for *svc.cluster.local.  
You can check that by execing into containers and looking at the resolv.conf.

In 3.3, at the host level we configure dnsmasq by default to offer a single 
resolver (so musl doesn't get confused).  You can check how that is configured 
on your hosts.

On Nov 2, 2016, at 5:06 PM, Srinivas Naga Kotaru (skotaru) 
mailto:skot...@cisco.com>> wrote:
Trying to debug below issue reported by client. For some reason, service 
discover never working in our platform.

Building an image with net tools for easy troubleshooting these issues from 
platform side. I’m sure making silly mistake, but image build from below code 
always throws CrashLoopBackOff error.

Wondering what mistake am doing here?

FROM alpine:latest
RUN apk update && apk add bind-tools net-tools curl
ENTRYPOINT ["sh"]

I observed any image build throwing the same error. Example ubuntu image from 
dockerhub. What preventing oepnshfit to run ?

--
Srinivas Kotaru



Tried all of those options.


In fact, even the first one sho

Re: Openshift discovery

[Clayton Coleman]

Can you show me the output of dig for kubernetes.default.svc.cluster.local
AND contents of resolv.conf?

On Thu, Nov 3, 2016 at 12:38 PM, Srinivas Naga Kotaru (skotaru) <
skot...@cisco.com> wrote:

> SKOTARU-M-H06U:~ $ oc get pods
>
> NAMEREADY STATUS RESTARTS   AGE
>
> net-tools-1-pp4t4   0/1   CrashLoopBackOff   20817h
>
> SKOTARU-M-H06U:~ $
>
>
>
> SKOTARU-M-H06U:~ $ oc debug net-tools-1-pp4t4
>
> Debugging with pod/net-tools-1-pp4t4-debug, original command: sh
>
> Waiting for pod to start ...
>
> Pod IP: 10.1.4.10
>
> If you don't see a command prompt, try pressing enter.
>
>
>
> / $ dig
>
>
>
> ; <<>> DiG 9.10.4-P3 <<>>
>
> ;; global options: +cmd
>
> ;; Got answer:
>
> ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 18102
>
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
>
>
>
> ;; QUESTION SECTION:
>
> ;.
> INNS
>
>
>
> ;; Query time: 0 msec
>
> ;; SERVER: 173.36.96.19#53(173.36.96.19)
>
> ;; WHEN: Thu Nov 03 16:37:12 UTC 2016
>
> ;; MSG SIZE  rcvd: 17
>
>
>
>
>
> --
>
> *Srinivas Kotaru*
>
>
>
> *From: *"ccole...@redhat.com" 
> *Date: *Thursday, November 3, 2016 at 7:02 AM
> *To: *Srinivas Naga Kotaru 
> *Cc: *"users@lists.openshift.redhat.com"  >
> *Subject: *Re: Openshift discovery
>
>
>
> If you "oc debug" the crashing pods, do you get a shell up?
>
>
> On Nov 3, 2016, at 9:56 AM, Srinivas Naga Kotaru (skotaru) <
> skot...@cisco.com> wrote:
>
> Clayton
>
>
>
> Sorry for confusion. Original problem was, Service discovery not working
> in regular openshift apps. Out of the box images as well as custom images.
>
>
>
> I was trying to build a image with a net tools for debugging, so it is
> easy for troubleshoot as out of the box images does not have basic net
> tools. Openshift throwing crash recovery for any image I build, so I might
> be doing some mistake.  These images working fine in standard docker.
>
>
>
>
> Sent from my iPhone
>
>
> On Nov 3, 2016, at 6:24 AM, Clayton Coleman  wrote:
>
> Alpine uses musl which has known differences from glibc in how it handles
> DNS resolution.  *usually* this is because multiple  nameservers are listed
> in resolv.conf and the first one doesn't answer queries for
> *svc.cluster.local.  You can check that by execing into containers and
> looking at the resolv.conf.
>
>
>
> In 3.3, at the host level we configure dnsmasq by default to offer a
> single resolver (so musl doesn't get confused).  You can check how that is
> configured on your hosts.
>
>
> On Nov 2, 2016, at 5:06 PM, Srinivas Naga Kotaru (skotaru) <
> skot...@cisco.com> wrote:
>
> Trying to debug below issue reported by client. For some reason, service
> discover never working in our platform.
>
>
>
> Building an image with net tools for easy troubleshooting these issues
> from platform side. I’m sure making silly mistake, but image build from
> below code always throws CrashLoopBackOff error.
>
>
>
> Wondering what mistake am doing here?
>
>
>
> FROM alpine:latest
>
> RUN apk update && apk add bind-tools net-tools curl
>
> ENTRYPOINT ["sh"]
>
>
>
> I observed any image build throwing the same error. Example ubuntu image
> from dockerhub. What preventing oepnshfit to run ?
>
>
>
> --
>
> *Srinivas Kotaru*
>
>
>
>
>
>
>
> Tried all of those options.
>
> 
>
>
>
> In fact, even the first one should work, since resolve.conf has search
> domains configured.  That would be ideal, since it makes the configuration
> of pods dependencies easier to port across projects.
>
>
> Regards,
>
> Tom.
>
>
>
>
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Openshift discovery

[Srinivas Naga Kotaru (skotaru)]

SKOTARU-M-H06U:~ $ oc get pods
NAMEREADY STATUS RESTARTS   AGE
net-tools-1-pp4t4   0/1   CrashLoopBackOff   20817h
SKOTARU-M-H06U:~ $

SKOTARU-M-H06U:~ $ oc debug net-tools-1-pp4t4
Debugging with pod/net-tools-1-pp4t4-debug, original command: sh
Waiting for pod to start ...
Pod IP: 10.1.4.10
If you don't see a command prompt, try pressing enter.

/ $ dig

; <<>> DiG 9.10.4-P3 <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 18102
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;. INNS

;; Query time: 0 msec
;; SERVER: 173.36.96.19#53(173.36.96.19)
;; WHEN: Thu Nov 03 16:37:12 UTC 2016
;; MSG SIZE  rcvd: 17


--
Srinivas Kotaru

From: "ccole...@redhat.com" 
Date: Thursday, November 3, 2016 at 7:02 AM
To: Srinivas Naga Kotaru 
Cc: "users@lists.openshift.redhat.com" 
Subject: Re: Openshift discovery

If you "oc debug" the crashing pods, do you get a shell up?

On Nov 3, 2016, at 9:56 AM, Srinivas Naga Kotaru (skotaru) 
mailto:skot...@cisco.com>> wrote:
Clayton

Sorry for confusion. Original problem was, Service discovery not working in 
regular openshift apps. Out of the box images as well as custom images.

I was trying to build a image with a net tools for debugging, so it is easy for 
troubleshoot as out of the box images does not have basic net tools. Openshift 
throwing crash recovery for any image I build, so I might be doing some 
mistake.  These images working fine in standard docker.


Sent from my iPhone

On Nov 3, 2016, at 6:24 AM, Clayton Coleman 
mailto:ccole...@redhat.com>> wrote:
Alpine uses musl which has known differences from glibc in how it handles DNS 
resolution.  *usually* this is because multiple  nameservers are listed in 
resolv.conf and the first one doesn't answer queries for *svc.cluster.local.  
You can check that by execing into containers and looking at the resolv.conf.

In 3.3, at the host level we configure dnsmasq by default to offer a single 
resolver (so musl doesn't get confused).  You can check how that is configured 
on your hosts.

On Nov 2, 2016, at 5:06 PM, Srinivas Naga Kotaru (skotaru) 
mailto:skot...@cisco.com>> wrote:
Trying to debug below issue reported by client. For some reason, service 
discover never working in our platform.

Building an image with net tools for easy troubleshooting these issues from 
platform side. I’m sure making silly mistake, but image build from below code 
always throws CrashLoopBackOff error.

Wondering what mistake am doing here?

FROM alpine:latest
RUN apk update && apk add bind-tools net-tools curl
ENTRYPOINT ["sh"]

I observed any image build throwing the same error. Example ubuntu image from 
dockerhub. What preventing oepnshfit to run ?

--
Srinivas Kotaru



Tried all of those options.


In fact, even the first one should work, since resolve.conf has search domains 
configured.  That would be ideal, since it makes the configuration of pods 
dependencies easier to port across projects.

Regards,
Tom.


___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Unable to start openshift in VM, AWS or google cloud

[Cesar Wong]

Hi Ravi,

On AWS, the magic incantation is:

oc cluster up --public-hostname=[public dns name] --routing-suffix=[ip 
address].xip.io 

Don't specify the numeric ip address in --public-hostname, rather the dns name.

You can then access the web console at https://[public  dns 
name]:8443/


On Windows 7 + VirtualBox + Ubuntu ... is it unable to connect to a route 
because the address/port is not reachable or because the dns name is not found?

> On Nov 2, 2016, at 9:34 PM, Ravi  wrote:
> 
> 
> I am not able to start openshift, I tried three different ways.
> 
> 1. Windows 7 + Virtual Box + Ubuntu
> oc cluster up works well. I went to console and launched nodejs-ex example. 
> Console shows it is up, however when I click on route, it says "unable to 
> connect". I tried going directly to POD's IP address and it does work. In 
> other words, somehow load balancer was failing in virtualbox Ubuntu VM.
> 
> 2. Then I moved on to AWS. I launched a RedHat image and installed docker and 
> started openshift. Here, OC starts on private IP address, so I am not able to 
> access it from public internet. I even tried
> oc cluster up --public-hostname='my ip address' but since the public ip 
> address is some magic, oc is not able to detect etcd etc and fails.
> 
> 3. Then I tried on google cloud. I faced exactly same issue as AWS.
> 
> If any one of them works, I will be ok but no idea how to get past these 
> issues.
> 
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Upcoming: backwards incompatible change of handling --env/--param/--value oc options

[Martin Milata]

In upcoming Origin 1.4 and OpenShift Container Platform 3.4 releases, it
will no longer be possible to use single command-line line option to
pass several comma-separated environment variables or template
parameters. Examples of commands that will not work as before:

  oc start-build hello-world --env 
HTTP_PROXY=http://192.168.1.1:8080,HTTPS_PROXY=http://192.168.1.1:8443
  oc new-app mysql --param MYSQL_USER=user,MYSQL_PASSWORD=pass
  oc process mysql --value MYSQL_USER=user,MYSQL_PASSWORD=pass

You can use multiple --env/--param/--value options to provide multiple
values instead:

  oc start-build hello-world --env HTTP_PROXY=http://192.168.1.1:8080 --env 
HTTPS_PROXY=http://192.168.1.1:8443
  oc new-app mysql --param MYSQL_USER=user --param MYSQL_PASSWORD=pass
  oc process mysql --value MYSQL_USER=user --value MYSQL_PASSWORD=pass

The main reason for this change is to make it possible to set a
variable/parameter to arbitrary value without worrying whether it
contains commas, etc. Example:

  oc process sometemplate --value VARIABLE=$(cat value-in-file.txt)

Please update your scripts if you are using the old form.

Best regards,
Martin Milata

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Unable to start openshift in VM, AWS or google cloud

[Luke Meyer]

On Wed, Nov 2, 2016 at 11:34 PM, Ravi  wrote:

>
> I am not able to start openshift, I tried three different ways.
>
> 1. Windows 7 + Virtual Box + Ubuntu
> oc cluster up works well. I went to console and launched nodejs-ex
> example. Console shows it is up, however when I click on route, it says
> "unable to connect". I tried going directly to POD's IP address and it does
> work. In other words, somehow load balancer was failing in virtualbox
> Ubuntu VM.
>

Have you installed a router? Does DNS or /etc/hosts for the route direct
your browser to the host's IP?


>
> 2. Then I moved on to AWS. I launched a RedHat image and installed docker
> and started openshift. Here, OC starts on private IP address, so I am not
> able to access it from public internet. I even tried
> oc cluster up --public-hostname='my ip address' but since the public ip
> address is some magic, oc is not able to detect etcd etc and fails.
>
> 3. Then I tried on google cloud. I faced exactly same issue as AWS.
>
> If any one of them works, I will be ok but no idea how to get past these
> issues.
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use SCC and HostPath ?

[Stéphane Klein]

2016-11-03 15:03 GMT+01:00 Stéphane Klein :

>
>
> 2016-11-03 14:56 GMT+01:00 Clayton Coleman :
>
>> That RC is creating pods under service account cassandra.  So you need to
>> give "cassandra" access to privileged
>>
>>
> Yes ! it's here: https://gist.github.com/harobed/
> 76dc697e1658afd934c107aadc4f09a6#file-replicationcontrollers-yaml-L87
>


I removed this lines:

serviceAccount: cassandra
serviceAccountName: cassandra

Now it's working with:

$ oc adm policy add-scc-to-user privileged -z default -n openshift-infra

Thanks
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use SCC and HostPath ?

[Stéphane Klein]

2016-11-03 15:03 GMT+01:00 Stéphane Klein :

>
>
> 2016-11-03 14:56 GMT+01:00 Clayton Coleman :
>
>> That RC is creating pods under service account cassandra.  So you need to
>> give "cassandra" access to privileged
>>
>>
> Yes ! it's here: https://gist.github.com/harobed/
> 76dc697e1658afd934c107aadc4f09a6#file-replicationcontrollers-yaml-L87
>


How can I see this log info
https://github.com/openshift/origin/blob/85eb37b34f0657631592356d020cef5a58470f8e/pkg/security/admission/admission.go#L88
from oc cli ?
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use SCC and HostPath ?

[Slava Semushin]

No, I'm sorry, I've just tried and it's not because of indentation.

-- 
Slava Semushin | OpenShift


- Original Message -
From: "Stéphane Klein" 
To: "Slava Semushin" 
Cc: "users" 
Sent: Thursday, November 3, 2016 3:00:36 PM
Subject: Re: How to use SCC and HostPath ?

2016-11-03 14:55 GMT+01:00 Slava Semushin < semus...@redhat.com > : 


I suspect that it can be caused by wrong indentation. Could you try to reduce 
the indentation of the volumes: block by 2 spaces? 

Here? 
https://gist.github.com/harobed/76dc697e1658afd934c107aadc4f09a6#file-replicationcontrollers-yaml-L93
 

It's already 2 spaces indentation. I don't understand. 

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Openshift discovery

[Srinivas Naga Kotaru (skotaru)]

Clayton

Sorry for confusion. Original problem was, Service discovery not working in 
regular openshift apps. Out of the box images as well as custom images.

I was trying to build a image with a net tools for debugging, so it is easy for 
troubleshoot as out of the box images does not have basic net tools. Openshift 
throwing crash recovery for any image I build, so I might be doing some 
mistake.  These images working fine in standard docker.


Sent from my iPhone

On Nov 3, 2016, at 6:24 AM, Clayton Coleman 
mailto:ccole...@redhat.com>> wrote:

Alpine uses musl which has known differences from glibc in how it handles DNS 
resolution.  *usually* this is because multiple  nameservers are listed in 
resolv.conf and the first one doesn't answer queries for *svc.cluster.local.  
You can check that by execing into containers and looking at the resolv.conf.

In 3.3, at the host level we configure dnsmasq by default to offer a single 
resolver (so musl doesn't get confused).  You can check how that is configured 
on your hosts.

On Nov 2, 2016, at 5:06 PM, Srinivas Naga Kotaru (skotaru) 
mailto:skot...@cisco.com>> wrote:

Trying to debug below issue reported by client. For some reason, service 
discover never working in our platform.

Building an image with net tools for easy troubleshooting these issues from 
platform side. I'm sure making silly mistake, but image build from below code 
always throws CrashLoopBackOff error.

Wondering what mistake am doing here?

FROM alpine:latest
RUN apk update && apk add bind-tools net-tools curl
ENTRYPOINT ["sh"]

I observed any image build throwing the same error. Example ubuntu image from 
dockerhub. What preventing oepnshfit to run ?

--
Srinivas Kotaru



Tried all of those options.


In fact, even the first one should work, since resolve.conf has search domains 
configured.  That would be ideal, since it makes the configuration of pods 
dependencies easier to port across projects.

Regards,
Tom.


___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use SCC and HostPath ?

[Clayton Coleman]

That RC is creating pods under service account cassandra.  So you need to
give "cassandra" access to privileged

On Nov 3, 2016, at 9:23 AM, Stéphane Klein 
wrote:

Hi,

This my SCC:

$ oc get scc
NAME   PRIV  CAPS  SELINUX RUNASUSER
FSGROUP SUPGROUPPRIORITY   READONLYROOTFS   VOLUMES
anyuid false []MustRunAs   RunAsAny
RunAsAnyRunAsAny10 false[configMap downwardAPI
emptyDir persistentVolumeClaim secret]
hostaccess false []MustRunAs   MustRunAsRange
MustRunAs   RunAsAny false[configMap downwardAPI
emptyDir hostPath persistentVolumeClaim secret]
hostmount-anyuid   false []MustRunAs   RunAsAny
RunAsAnyRunAsAny false[configMap downwardAPI
emptyDir hostPath nfs persistentVolumeClaim secret]
hostnetworkfalse []MustRunAs   MustRunAsRange
MustRunAs   MustRunAsfalse[configMap downwardAPI
emptyDir persistentVolumeClaim secret]
nonrootfalse []MustRunAs   MustRunAsNonRoot
RunAsAnyRunAsAny false[configMap downwardAPI
emptyDir persistentVolumeClaim secret]
privileged true  []RunAsAnyRunAsAny
RunAsAnyRunAsAny false[*]
restricted false []MustRunAs   MustRunAsRange
MustRunAs   RunAsAny false[configMap downwardAPI
emptyDir persistentVolumeClaim secret]

I see that hostaccess, hostmount-anyuid and privileged have access to
hostPath volume.

I've removed all SCC from admin user and default SA:

$ oc adm policy remove-scc-from-user anyuid -z default -n openshift-infra
$ oc adm policy remove-scc-from-user hostaccess -z default -n
openshift-infra
$ oc adm policy remove-scc-from-user hostmount-anyuid -z default -n
openshift-infra
$ oc adm policy remove-scc-from-user hostnetwork -z default -n
openshift-infra
$ oc adm policy remove-scc-from-user nonroot -z default -n openshift-infra
$ oc adm policy remove-scc-from-user privileged -z default -n
openshift-infra
$ oc adm policy remove-scc-from-user restricted -z default -n
openshift-infra
$ oc adm policy remove-scc-from-user anyuid admin -n openshift-infra
$ oc adm policy remove-scc-from-user hostaccess admin -n openshift-infra
$ oc adm policy remove-scc-from-user hostmount-anyuid admin -n
openshift-infra
$ oc adm policy remove-scc-from-user hostnetwork admin -n openshift-infra
$ oc adm policy remove-scc-from-user nonroot admin -n openshift-infra
$ oc adm policy remove-scc-from-user privileged admin -n openshift-infra
$ oc adm policy remove-scc-from-user restricted admin -n openshift-infra
$ oc adm policy add-scc-to-user privileged admin -n openshift-infra
$ oc adm policy add-scc-to-user privileged -z default -n openshift-infra

Now I add privileged SCC to admin user and default SA:

$ oc adm policy add-scc-to-user privileged admin -n openshift-infra
$ oc adm policy add-scc-to-user privileged -z default -n openshift-infra

My replication controller file:
https://gist.github.com/harobed/76dc697e1658afd934c107aadc4f09a6

Next, I create ReplicationController:

$ oc delete rc hawkular-cassandra-1
$ oc delete event --all
$ oc apply -n openshift-infra -f replicationcontrollers.yaml
$ oc get events
FIRSTSEEN   LASTSEEN   COUNT NAME
KINDSUBOBJECT   TYPE  REASON
SOURCE  MESSAGE
3d  3d 4 hawkular-cassandra-1
ReplicationController   Warning   FailedCreate
{replication-controller }   Error creating: pods "hawkular-cassandra-1-" is
forbidden: unable to validate against any security context constraint:
[spec.containers[0].securityContext.volumes[0]: Invalid value: "hostPath":
hostPath volumes are not allowed to be used]

Why ? I set policy on bad user ?

Is it this bug? https://github.com/openshift/origin/issues/11153

Best regards,
Stéphane
-- 
Stéphane Klein 
blog: http://stephane-klein.info
cv : http://cv.stephane-klein.info
Twitter: http://twitter.com/klein_stephane

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use SCC and HostPath ?

[Stéphane Klein]

2016-11-03 14:56 GMT+01:00 Clayton Coleman :

> That RC is creating pods under service account cassandra.  So you need to
> give "cassandra" access to privileged
>
>
Yes ! it's here:
https://gist.github.com/harobed/76dc697e1658afd934c107aadc4f09a6#file-replicationcontrollers-yaml-L87

Thanks!

I don't understand why I've this config.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Openshift discovery

[Clayton Coleman]

If you "oc debug" the crashing pods, do you get a shell up?

On Nov 3, 2016, at 9:56 AM, Srinivas Naga Kotaru (skotaru) <
skot...@cisco.com> wrote:

Clayton

Sorry for confusion. Original problem was, Service discovery not working in
regular openshift apps. Out of the box images as well as custom images.

I was trying to build a image with a net tools for debugging, so it is easy
for troubleshoot as out of the box images does not have basic net tools.
Openshift throwing crash recovery for any image I build, so I might be
doing some mistake.  These images working fine in standard docker.


Sent from my iPhone

On Nov 3, 2016, at 6:24 AM, Clayton Coleman  wrote:

Alpine uses musl which has known differences from glibc in how it handles
DNS resolution.  *usually* this is because multiple  nameservers are listed
in resolv.conf and the first one doesn't answer queries for
*svc.cluster.local.  You can check that by execing into containers and
looking at the resolv.conf.

In 3.3, at the host level we configure dnsmasq by default to offer a single
resolver (so musl doesn't get confused).  You can check how that is
configured on your hosts.

On Nov 2, 2016, at 5:06 PM, Srinivas Naga Kotaru (skotaru) <
skot...@cisco.com> wrote:

Trying to debug below issue reported by client. For some reason, service
discover never working in our platform.



Building an image with net tools for easy troubleshooting these issues from
platform side. I’m sure making silly mistake, but image build from below
code always throws CrashLoopBackOff error.



Wondering what mistake am doing here?



FROM alpine:latest

RUN apk update && apk add bind-tools net-tools curl

ENTRYPOINT ["sh"]



I observed any image build throwing the same error. Example ubuntu image
from dockerhub. What preventing oepnshfit to run ?



-- 

*Srinivas Kotaru*







Tried all of those options.





In fact, even the first one should work, since resolve.conf has search
domains configured.  That would be ideal, since it makes the configuration
of pods dependencies easier to port across projects.


Regards,

Tom.





___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use SCC and HostPath ?

[Stéphane Klein]

2016-11-03 14:55 GMT+01:00 Slava Semushin :

> I suspect that it can be caused by wrong indentation. Could you try to
> reduce the indentation of the volumes: block by 2 spaces?


Here?
https://gist.github.com/harobed/76dc697e1658afd934c107aadc4f09a6#file-replicationcontrollers-yaml-L93

It's already 2 spaces indentation. I don't understand.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: How to use SCC and HostPath ?

[Slava Semushin]

Hi,

I suspect that it can be caused by wrong indentation. Could you try to reduce 
the indentation of the volumes: block by 2 spaces?

-- 
Slava Semushin | OpenShift

- Original Message -
From: "Stéphane Klein" 
To: "users" 
Sent: Thursday, November 3, 2016 2:21:13 PM
Subject: How to use SCC and HostPath ?

Hi, 

This my SCC: 

$ oc get scc 
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP PRIORITY READONLYROOTFS 
VOLUMES 
anyuid false [] MustRunAs RunAsAny RunAsAny RunAsAny 10 false [configMap 
downwardAPI emptyDir persistentVolumeClaim secret] 
hostaccess false [] MustRunAs MustRunAsRange MustRunAs RunAsAny  false 
[configMap downwardAPI emptyDir hostPath persistentVolumeClaim secret] 
hostmount-anyuid false [] MustRunAs RunAsAny RunAsAny RunAsAny  false 
[configMap downwardAPI emptyDir hostPath nfs persistentVolumeClaim secret] 
hostnetwork false [] MustRunAs MustRunAsRange MustRunAs MustRunAs  false 
[configMap downwardAPI emptyDir persistentVolumeClaim secret] 
nonroot false [] MustRunAs MustRunAsNonRoot RunAsAny RunAsAny  false 
[configMap downwardAPI emptyDir persistentVolumeClaim secret] 
privileged true [] RunAsAny RunAsAny RunAsAny RunAsAny  false [*] 
restricted false [] MustRunAs MustRunAsRange MustRunAs RunAsAny  false 
[configMap downwardAPI emptyDir persistentVolumeClaim secret] 

I see that hostaccess, hostmount-anyuid and privileged have access to hostPath 
volume. 

I've removed all SCC from admin user and default SA: 

$ oc adm policy remove-scc-from-user anyuid -z default -n openshift-infra 
$ oc adm policy remove-scc-from-user hostaccess -z default -n openshift-infra 
$ oc adm policy remove-scc-from-user hostmount-anyuid -z default -n 
openshift-infra 
$ oc adm policy remove-scc-from-user hostnetwork -z default -n openshift-infra 
$ oc adm policy remove-scc-from-user nonroot -z default -n openshift-infra 
$ oc adm policy remove-scc-from-user privileged -z default -n openshift-infra 
$ oc adm policy remove-scc-from-user restricted -z default -n openshift-infra 
$ oc adm policy remove-scc-from-user anyuid admin -n openshift-infra 
$ oc adm policy remove-scc-from-user hostaccess admin -n openshift-infra 
$ oc adm policy remove-scc-from-user hostmount-anyuid admin -n openshift-infra 
$ oc adm policy remove-scc-from-user hostnetwork admin -n openshift-infra 
$ oc adm policy remove-scc-from-user nonroot admin -n openshift-infra 
$ oc adm policy remove-scc-from-user privileged admin -n openshift-infra 
$ oc adm policy remove-scc-from-user restricted admin -n openshift-infra 
$ oc adm policy add-scc-to-user privileged admin -n openshift-infra 
$ oc adm policy add-scc-to-user privileged -z default -n openshift-infra 

Now I add privileged SCC to admin user and default SA: 

$ oc adm policy add-scc-to-user privileged admin -n openshift-infra 
$ oc adm policy add-scc-to-user privileged -z default -n openshift-infra 

My replication controller file: 
https://gist.github.com/harobed/76dc697e1658afd934c107aadc4f09a6 

Next, I create ReplicationController: 

$ oc delete rc hawkular-cassandra-1 
$ oc delete event --all 
$ oc apply -n openshift-infra -f replicationcontrollers.yaml 
$ oc get events 
FIRSTSEEN LASTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE 
3d 3d 4 hawkular-cassandra-1 ReplicationController Warning FailedCreate 
{replication-controller } Error creating: pods "hawkular-cassandra-1-" is 
forbidden: unable to validate against any security context constraint: 
[spec.containers[0].securityContext.volumes[0]: Invalid value: "hostPath": 
hostPath volumes are not allowed to be used] 

Why ? I set policy on bad user ? 

Is it this bug? https://github.com/openshift/origin/issues/11153 

Best regards, 
Stéphane 
-- 
Stéphane Klein < cont...@stephane-klein.info > 
blog: http://stephane-klein.info 
cv : http://cv.stephane-klein.info 
Twitter: http://twitter.com/klein_stephane 

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Network segmentation

[Ben Bennett]

Good question, I'll take a crack below, and I've cc'ed the rest of the
networking team in case they have any other ideas.


On Tue, Nov 1, 2016 at 6:30 PM, Josh Baird  wrote:

> Hi,
>
> As a follow-up to my previous post [1], I'm curious to hear how others are
> handling network segmentation in general with OpenShift.  I'm concerned
> with segmenting external based applications from internal
> assets/networks/hosts.
>

>
A general practice is to place external/public facing applications in some
> type of 'DMZ' network to protect your internal assets.  How are you
> accomplishing this with OpenShift?
>
> A few ways that I can think of:
>
> * Deploy entirely separate clusters for public facing applications and
> internal (private) applications and completely isolate the networks between
> the two. This seems like it may be over-kill and costly both technically
> and financially.
>

I agree that should be unnecessary, and will be rather expensive.  But
depending upon the regulatory / security environment I suppose it could be
necessary.



> * Deploy separate nodes (same cluster) for public facing applications and
> internal applications.  The nodes dedicated to public facing apps would be
> located in a DMZ/restricted network.  In the event of a container break or
> compromise on the public nodes, this option would at least provide some
> level of protection for your internal assets.
>
>
This is a reasonable approach, but will require that people are fastidious
about labeling things correctly.



> * Deploy dedicated routers for 'internal' and 'external' and use a
> combination of edge load balancing/reverse proxy/firewall ACLs to separate
> the two (as my previous post suggests).  This approach really just protects
> internal applications from being exposed to the internet, but doesn't
> protect internal assets from being exposed should a public application node
> (or container) be compromised.
>
>
If you use the mutli-tenant networking plugin, each project is isolated
from the others by the OVS rules.  Pods in the default project can talk to
the any project (and vice-versa) so if you place multiple routers into the
default project and use sharding to select which namespaces are served by
each router, then you could have a private router (or routers) that are
reachable only within the cluster (or you could expose them externally, but
only within your corporate network).  Then if a pod reachable through the
public router was compromised, it could only talk to things in its project
and the default project.


https://docs.openshift.com/container-platform/3.3/architecture/additional_concepts/sdn.html

https://docs.openshift.com/container-platform/3.3/architecture/core_concepts/routes.html#router-sharding

If you need some pods to access resources that are outside the cluster, but
in your private network... but not all pods.  The new "egress firewall"
feature allows you to put rules on namespaces to control what they can and
can not reach.


https://docs.openshift.com/container-platform/3.3/admin_guide/managing_pods.html#admin-guide-limit-pod-access-egress

Or if you must have IP-based access to an external resource, but only want
some pods to be able to reach it, the "egress router" can be useful:


https://docs.openshift.com/container-platform/3.3/admin_guide/managing_pods.html#admin-guide-limit-pod-access-egress-router


The problem with this approach is that you can't say that one project can
talk to a service in a different project.  We are working on this by being
heavily involved with the Kubernetes Networking SIG and defining the
NetworkPolicy and the EgressPolicy objects.  You can use a private router
(in the default namespace) but then anyone can access the resource, and it
must be http/https/tls (with SNI).  You could also make a NodePort for a
service and then it could carry any IP traffic... but again, it would be
visible to all pods in the cluster.



> * Another option may be combination of these where the entire OpenShift
> environment (application nodes only?) is isolated from the rest of the
> internal/private network for both public and private applications.  You
> then deploy multiple router-sets for internal/external as described above.
>
>
That could work too, but I would look at the egress router first, unless I
am misunderstanding what you mean by "rest of the internal/private network"
(I assume you mean, outside the cluster, but inside my organization's
private network).


-ben



> What are everyone's thoughts on this?  I would appreciate any
> feedback/suggestions.
>
> Thanks,
>
> Josh
>
> [1] https://lists.openshift.redhat.com/openshift-archives/us
> ers/2016-October/msg00263.html
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/

Re: Openshift discovery

[Clayton Coleman]

Alpine uses musl which has known differences from glibc in how it handles
DNS resolution.  *usually* this is because multiple  nameservers are listed
in resolv.conf and the first one doesn't answer queries for
*svc.cluster.local.  You can check that by execing into containers and
looking at the resolv.conf.

In 3.3, at the host level we configure dnsmasq by default to offer a single
resolver (so musl doesn't get confused).  You can check how that is
configured on your hosts.

On Nov 2, 2016, at 5:06 PM, Srinivas Naga Kotaru (skotaru) <
skot...@cisco.com> wrote:

Trying to debug below issue reported by client. For some reason, service
discover never working in our platform.



Building an image with net tools for easy troubleshooting these issues from
platform side. I’m sure making silly mistake, but image build from below
code always throws CrashLoopBackOff error.



Wondering what mistake am doing here?



FROM alpine:latest

RUN apk update && apk add bind-tools net-tools curl

ENTRYPOINT ["sh"]



I observed any image build throwing the same error. Example ubuntu image
from dockerhub. What preventing oepnshfit to run ?



-- 

*Srinivas Kotaru*







Tried all of those options.





In fact, even the first one should work, since resolve.conf has search
domains configured.  That would be ideal, since it makes the configuration
of pods dependencies easier to port across projects.


Regards,

Tom.





___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

How to use SCC and HostPath ?

[Stéphane Klein]

Hi,

This my SCC:

$ oc get scc
NAME   PRIV  CAPS  SELINUX RUNASUSER
FSGROUP SUPGROUPPRIORITY   READONLYROOTFS   VOLUMES
anyuid false []MustRunAs   RunAsAny
RunAsAnyRunAsAny10 false[configMap downwardAPI
emptyDir persistentVolumeClaim secret]
hostaccess false []MustRunAs   MustRunAsRange
MustRunAs   RunAsAny false[configMap downwardAPI
emptyDir hostPath persistentVolumeClaim secret]
hostmount-anyuid   false []MustRunAs   RunAsAny
RunAsAnyRunAsAny false[configMap downwardAPI
emptyDir hostPath nfs persistentVolumeClaim secret]
hostnetworkfalse []MustRunAs   MustRunAsRange
MustRunAs   MustRunAsfalse[configMap downwardAPI
emptyDir persistentVolumeClaim secret]
nonrootfalse []MustRunAs   MustRunAsNonRoot
RunAsAnyRunAsAny false[configMap downwardAPI
emptyDir persistentVolumeClaim secret]
privileged true  []RunAsAnyRunAsAny
RunAsAnyRunAsAny false[*]
restricted false []MustRunAs   MustRunAsRange
MustRunAs   RunAsAny false[configMap downwardAPI
emptyDir persistentVolumeClaim secret]

I see that hostaccess, hostmount-anyuid and privileged have access to
hostPath volume.

I've removed all SCC from admin user and default SA:

$ oc adm policy remove-scc-from-user anyuid -z default -n openshift-infra
$ oc adm policy remove-scc-from-user hostaccess -z default -n
openshift-infra
$ oc adm policy remove-scc-from-user hostmount-anyuid -z default -n
openshift-infra
$ oc adm policy remove-scc-from-user hostnetwork -z default -n
openshift-infra
$ oc adm policy remove-scc-from-user nonroot -z default -n openshift-infra
$ oc adm policy remove-scc-from-user privileged -z default -n
openshift-infra
$ oc adm policy remove-scc-from-user restricted -z default -n
openshift-infra
$ oc adm policy remove-scc-from-user anyuid admin -n openshift-infra
$ oc adm policy remove-scc-from-user hostaccess admin -n openshift-infra
$ oc adm policy remove-scc-from-user hostmount-anyuid admin -n
openshift-infra
$ oc adm policy remove-scc-from-user hostnetwork admin -n openshift-infra
$ oc adm policy remove-scc-from-user nonroot admin -n openshift-infra
$ oc adm policy remove-scc-from-user privileged admin -n openshift-infra
$ oc adm policy remove-scc-from-user restricted admin -n openshift-infra
$ oc adm policy add-scc-to-user privileged admin -n openshift-infra
$ oc adm policy add-scc-to-user privileged -z default -n openshift-infra

Now I add privileged SCC to admin user and default SA:

$ oc adm policy add-scc-to-user privileged admin -n openshift-infra
$ oc adm policy add-scc-to-user privileged -z default -n openshift-infra

My replication controller file:
https://gist.github.com/harobed/76dc697e1658afd934c107aadc4f09a6

Next, I create ReplicationController:

$ oc delete rc hawkular-cassandra-1
$ oc delete event --all
$ oc apply -n openshift-infra -f replicationcontrollers.yaml
$ oc get events
FIRSTSEEN   LASTSEEN   COUNT NAME
KINDSUBOBJECT   TYPE  REASON
SOURCE  MESSAGE
3d  3d 4 hawkular-cassandra-1
ReplicationController   Warning   FailedCreate
{replication-controller }   Error creating: pods "hawkular-cassandra-1-" is
forbidden: unable to validate against any security context constraint:
[spec.containers[0].securityContext.volumes[0]: Invalid value: "hostPath":
hostPath volumes are not allowed to be used]

Why ? I set policy on bad user ?

Is it this bug? https://github.com/openshift/origin/issues/11153

Best regards,
Stéphane
-- 
Stéphane Klein 
blog: http://stephane-klein.info
cv : http://cv.stephane-klein.info
Twitter: http://twitter.com/klein_stephane
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: containters with host volumes from controllers

[Stéphane Klein]

Same issue here:

oc adm policy add-scc-to-user anyuid -z default -n openshift-infra
oc adm policy add-scc-to-user privileged -z default -n openshift-infra
oc adm policy add-scc-to-user hostmount-anyuid
system:serviceaccount:openshift-infra:default

oc apply -n openshift-infra -f replicationcontrollers.yaml

replicationcontrollers.yaml content:
https://gist.github.com/harobed/76dc697e1658afd934c107aadc4f09a6

I've this error:

Error creating: pods "hawkular-cassandra-1-" is forbidden: unable to
validate against any security context constraint:
[securityContext.runAsUser: Invalid value: 0: UID on container
hawkular-cassandra-1 does not match required range. Found 0, required min:
10 max: 10 spec.containers[0].securityContext.volumes[0]:
Invalid value: "hostPath": hostPath volumes are not allowed to be used]

What is my mistake ?

Best regards,
Stéphane

2016-05-19 23:07 GMT+02:00 Clayton Coleman :

> Users don't have a "preferred namespace", you'll have to provide that
> yourself.  oc project sets it in the config.  You can use the -n flag to
> set it.
>
> On May 19, 2016, at 11:36 AM, Alan Jones  wrote:
>
> Of all the command's I've tried, I think the following from another tread
> did the magic:
> oadm policy add-scc-to-user privileged -z default
> In addition, I had to provide kubelet with --allow-privileged=true, which
> wasn't required in the stock K8 1.2 config.
> Perhaps OpenShift is adding something to the pod spec that kubelet is
> validating.
> What I'd really like to do now is wipe the OpenShift config to rerun
> 'atomic-openshift-installer install' and confirm the particular steps that
> make it work.
> If you have any insight into the best way to wipe my OpenShift config,
> please share.
>
> On getting my user and project; the replication set is submitted by one of
> our system daemons that is run out of systemd with our own user and the
> node certificates I described earlier.
> Looking at the CLI code, it seems the command 'oc project' gets it from
> the context before any REST API call is made.
> However, 'oc whoami' seems to call GET on the 'users' resource with the
> name '~'.
> Can my daemon can make that call and get the project name or namespace
> from the user details?
>
> Thank for helping me get this right!
> Alan
>
> On Wed, May 18, 2016 at 7:43 PM, Clayton Coleman 
> wrote:
>
>> The node is running as a user, but every pod / rc has to be created in
>> a namespace (or project, which is the same thing but with some
>> additional controls).  When you create an RC from your credentials,
>> you are either creating it in the "default" namespace (in which case
>> you need to grant system:serviceaccount:default:default access to
>> hostmount-anyuid) or in whatever namespace was the default.  If you
>> run "oc project", which project does it say you are in?
>>
>> On Wed, May 18, 2016 at 8:16 PM, Alan Jones  wrote:
>> > I now reproduced the issue with OpenShift 3.2 on RHEL 7, as apposed to
>> my
>> > few week old origin on CentOS.
>> > Unfortunately, my magic command isn't working.
>> > Here is my procedure:
>> > 1) Create node certs with `oadm create-node-config`
>> > 2) Use these certs from said node to create a replication set for a
>> > container that requires a host mount.
>> > 3) See event with 'hostPath volumes are not allowed to be used'
>> > Note, this process works with standard Kubernetes; so navigating the
>> > OpenShift authentication & permissions is what I'm trying to accomplish.
>> > Also note that there is not *project* specified in this procedure; the
>> node
>> > being certified belongs to system:node, should I use that?
>> > I feel like I'm flying blind because there is no feedback:
>> > 1) The command to add privileges doesn't verify that the project or user
>> > exists.
>> > 2) The failure doesn't tell me which project/user was attempting to do
>> the
>> > unpermitted task.
>> > Alan
>> > [root@alan-lnx ~]# cat /etc/redhat-release
>> > Red Hat Enterprise Linux Server release 7.2 (Maipo)
>> > [root@alan-lnx ~]# openshift version
>> > openshift v3.2.0.20
>> > kubernetes v1.2.0-36-g4a3f9c5
>> > etcd 2.2.5
>> >
>> >
>> > On Wed, May 18, 2016 at 3:08 PM, Alan Jones 
>> wrote:
>> >>
>> >> I think I'm making progress:
>> >> oadm policy add-scc-to-user hostmount-anyuid
>> >> system:serviceaccount:openshift-infra:default
>> >> Now when I submit the replica set I get a different mount error that I
>> >> think I understand.
>> >> Note, the context I'm submitting the request in is using the node host
>> >> certs under /openshift.local/config/ to the API server.
>> >> There is no specified project.
>> >> Thank you!
>> >> Alan
>> >>
>> >> On Wed, May 18, 2016 at 2:48 PM, Clayton Coleman 
>> >> wrote:
>> >>>
>> >>>
>> >>>
>> >>> On May 18, 2016, at 5:26 PM, Alan Jones  wrote:
>> >>>
>> >>> > oadm policy ... -z default
>> >>> In the version of openshift origin I'm using the oadm command doesn't
>> >>> take '-z'.
>> >>> Can you fill in the dot, dot, dot for me?
>> >>> I'