Re: haproxy logs

[Ram Ranganathan]

As Phil mentioned, you can check if the iptables rule is blocking it.

Simple test would be to rsh into the router pod and use netcat to send a
message.
$ oc rsh 
pod>  echo '<14> user test message from router pod' | nc -w 2 -u
 514

And maybe try from the host (openshift-node) or another node as well.
$ echo '<14> user test message from the node' | nc -w 2 -u 
514

And if you need to open the port you could use something like:
$ sudo iptables -I INPUT -p udp -s  --dport 514 -j ACCEPT
$ sudo service iptables save
$ sudo service iptables restart

HTH

On Mon, Feb 20, 2017 at 3:16 AM, Julio Saura  wrote:

> hello
>
> any clue please?
>
> thanks
>
>
> El 17 feb 2017, a las 10:04, Julio Saura  escribió:
>
> Hello
>
> i need to enable haproxy access logs on my openshift routers..
>
> i followed the guide and enabled a syslog server on my net ..
>
> after adding env variables on my router dc for poiting to my syslog server
> i don’t see any packet sent to my syslog server ( tcpdump on my syslog
> servers shows no traffic on syslog port tcp or udp ) y put haproxy log
> level to debug for being sure it generates logs.
>
> if i describe my router pods y see env variables are passed and filled
> with the right values ,  and the routers have been redeployed by the router
> DC ..
>
> anything else i am missing?
>
> thanks
>
> Best regards
>
>
>
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>


-- 
Ram//
main(O,s){s=--O;10>4*s)*(O++?-1:1):10)&&\
main(++O,s++);}
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: source IP restriction on routes

[Ram Ranganathan]

*Sorry for the duplicate email Sebastian - the users list rejected the
original mail*

You would need a customized haproxy config template but you could add
something like this in the 2 frontends public[_ssl] (or to specific
backends if you need more granular control on a per-backend basis):

acl allowed 10.1.2.3 10.4.5.6 172.16.10.0/24 192.168.1.0/24
block if !allowed

Or alternatively you can check if the src is in a whitelist ala:

tcp-request connection accept if { src -f /path/to/allowed.lst } #  or
... connection reject if { src -f /path/to/denied.lst }

And its also possible to do the same with maps (and map_ip) - allow/deny
list.

You'd need to use a config map for your customized template. See:
https://docs.openshift.org/latest/install_config/router/customized_haproxy_
router.html#using-configmap-replace-template

HTH

On Mon, Oct 17, 2016 at 2:39 AM, Sebastian Wieseler <
sebast...@myrepublic.com.sg> wrote:

> Hi guys,
>
> Is it possible with router (s, sharding) to restrict access on IP level?
>
> We want to expose various applications via various routers, but
> restrict access via source IP addresses,
> so that different source IP addresses can only access allowed applications.
>
> How can we do that?
>
> Thanks a lot in advance.
> Greetings,
>   Sebastian
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>



-- 
Ram//
main(O,s){s=--O;10>4*s)*(O++?-1:1):10)&&\
main(++O,s++);}
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: All my ipfailover pods are in "Entering MASTER STATE", it's not fair ?

[Ram Ranganathan]

Couldn't figure out if you have a problem or not  (or it was just a
question) from the email thread.

What does "ip addr show" on all the nodes show?  This is the nodes where
your ipfailover pods are running.
Are the VIPs allocated to both nodes (assuming you have from the logs),
then it is likely some of the VRRP instances would be in master state.

If that is not the case, then can you please check if multicast is enabled
and traffic (firewall/iptables) is allowed for/to 224.0.0.18

Thanks,

Ram//

On Mon, Jun 20, 2016 at 8:14 AM, Stéphane Klein  wrote:

> In this documentation
> https://github.com/acassen/keepalived/blob/master/doc/source/case_study_failover.rst#architecture-specification
> I read:
>
> « 4 VRRP Instances per LVS director: 2 VRRP Instance in the MASTER state
> and 2 in BACKUP state. We use a symmetric state on each LVS directors. »
>
> then I think it's normal to have many Master instances.
>
> 2016-06-20 9:35 GMT+02:00 Stéphane Klein :
>
>> I would like to say "It's a problem, it's abnormal ?"
>>
>> 2016-06-17 16:26 GMT+02:00 Stéphane Klein :
>>
>>> Hi,
>>>
>>> I've:
>>>
>>> * one cluster with 2 nodes
>>> * ipfailover replicas=2
>>>
>>> I execute:
>>>
>>> * oc logs ipfailover-rbx-1-bh3kn
>>> https://gist.github.com/harobed/2ab152ed98f95285d549cbc7af3a#file-oc-logs-ipfailover-rbx-1-bh3kn
>>> * oc logs ipfailover-rbx-1-mmp36
>>> https://gist.github.com/harobed/2ab152ed98f95285d549cbc7af3a#file-oc-logs-ipfailover-rbx-1-mmp36
>>>
>>> and I see that all ipfailover pod are in "Entering MASTER STATE".
>>>
>>> It's not fair ?
>>>
>>> Best regards,
>>> Stéphane
>>>
>>
>>
>>
>> --
>> Stéphane Klein 
>> blog: http://stephane-klein.info
>> cv : http://cv.stephane-klein.info
>> Twitter: http://twitter.com/klein_stephane
>>
>
>
>
> --
> Stéphane Klein 
> blog: http://stephane-klein.info
> cv : http://cv.stephane-klein.info
> Twitter: http://twitter.com/klein_stephane
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>


-- 
Ram//
main(O,s){s=--O;10>4*s)*(O++?-1:1):10)&&\
main(++O,s++);}
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: 503 - Maintenance page

[Ram Ranganathan]

So an alternative might be to use a temporary redirect on '/' - 302 to some
site-under-maintenance-page  (which can return a 503 http code w/ whatever
custom page content you want). And who knows, that might also make a http
purist happier!! ;^)

On Wed, Jun 8, 2016 at 7:31 AM, Philippe Lafoucrière <
philippe.lafoucri...@tech-angels.com> wrote:

>
> On Tue, Jun 7, 2016 at 6:45 PM, Ram Ranganathan <rrang...@redhat.com>
> wrote:
>
>> Is your server always returning 503 - example for a GET/HEAD on / ? That
>> could cause haproxy to mark it as down.
>>
>> You can also see the stats in haproxy to look at if the server has been
>> marked down:
>> cmd="echo 'show stat' | socat
>> unix-connect:/var/lib/haproxy/run/haproxy.sock stdio"
>> echo "$cmd"  | oc rsh #  replace with router pod
>> name.
>>
>
> Of course my server is returning a 503 for "/' :) (it's down for
> maintenance). Haproxy thinks no server is available, so it's not even
> trying to pass the page. Make sense.
> Ok, so I guess I'll to use a custom router then :(
>
> Thanks for your help.
> Philippe
>



-- 
Ram//
main(O,s){s=--O;10<putchar(3^O?97-(15&7183>>4*s)*(O++?-1:1):10)&&\
main(++O,s++);}
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: 503 - Maintenance page

[Ram Ranganathan]

Hmm, so 503 is also returned by haproxy if no server is available to
service a request (example for a backend with no servers or if the server
is not available failing the health check).  As I recall, we did the error
page on a request as it gives the ability to override it in a custom
template.

Now that said, if the app (server associated with a haproxy backend) is
returning 503s, that content should get passed back as is. Meaning you
should see your custom error page being returned back to the server.

I just tested this out with a repo I have:
https://github.com/ramr/nodejs-header-echo/blob/master/server.js#L12
and it returns the content + status code back to the requester.

If that's not the case - from what you are seeing, it is more than likely
that haproxy has marked the backend server down as unavailable - which
means its failing health checks.

Is your server always returning 503 - example for a GET/HEAD on / ? That
could cause haproxy to mark it as down.

You can also see the stats in haproxy to look at if the server has been
marked down:
cmd="echo 'show stat' | socat
unix-connect:/var/lib/haproxy/run/haproxy.sock stdio"
echo "$cmd"  | oc rsh #  replace with router pod
name.

HTH


On Tue, Jun 7, 2016 at 12:56 PM, Philippe Lafoucrière <
philippe.lafoucri...@tech-angels.com> wrote:

>
> On Tue, Jun 7, 2016 at 3:46 PM, Luke Meyer  wrote:
>
>> It sounds like what he wants is for the router to simply not interfere
>> with passing along something that's already returning a 503. It sounds like
>> haproxy is replacing the page content with its own in that use case.
>
>
> THANKS Luke :))
> I don't want to change the router, I just want it to point to a specific
> service returning 503 for most URLs.
> On the other hand, the SAME router is used (with another route) to point
> to the production service, with a different URL if we want to test the
> change.
> Imagine a migration from pg 9.4 to 9.5, you have to shutdown your site.
> That doesn't mean traffic can't be routed any more, we like to test the
> site after the migration, and before resuming all the public traffic.
>
>


-- 
Ram//
main(O,s){s=--O;10>4*s)*(O++?-1:1):10)&&\
main(++O,s++);}
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: 503 - Maintenance page

[Ram Ranganathan]

Not clear if you want the router to automatically serve the 503 page or
not. If you do, this line in the haproxy config template:
https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L198

automatically sends a 503 page if your service is down (example has 0 pods
backing the service).
The actual error page template is at:

https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/error-page-503.http


You could customize the template and/or error page (and the router image)
to use a different page.

Alternatively, if you desire some other behavior, you can disable it by
removing that haproxy directive. Does still need a custom template + router
image.

HTH.


On Mon, Jun 6, 2016 at 12:58 PM, Philippe Lafoucrière <
philippe.lafoucri...@tech-angels.com> wrote:

> @Clayton:
> Sorry for the confusion. I'm not updating the routeR, I'm updating the
> route directly. The route to our website is pointing to a "maintenance"
> service during maintenance. This service serves 503 pages for most URLs,
> except a few for testing purprose.
>
> The problem is: If I browse my website, I get the expected 503 code, but a
> blank page, instead of the desired maintenance page served by the
> "maintenance" pods. I don't understand this blank page, it's like haproxy
> is not forwarding it because the pods responded with a 503.
>
> @v: Can I use a dedicated router per project?
> ​
> Thanks
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>


-- 
Ram//
main(O,s){s=--O;10>4*s)*(O++?-1:1):10)&&\
main(++O,s++);}
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Router warning for default ssl

[Ram Ranganathan]

It is warning you that the router will be serving up the default cert for
test-prjtest.getup.io   - which may or may not
be a problem depending on  your env (config + domain + routes).

If your default certificate is a wild-card cert for *.getup.io, then its ok
- at least for the test-prjtest.getup.io route.
Otherwise the router would be serving up a certificate that would/may not
match the route hostname.


On Sun, May 15, 2016 at 1:14 PM, Diego Castro 
wrote:

> I updated the default ssl certificate/private key to the router, using the
> following command:
>
> $ cat cert.crt key.key ca.cert > cert.pem
>
> $ adm router --default-cert=cert.pem --subdomain='${name}-${namespace}.
> getup.io' --replicas=2 --selector='role=router'
> --images='openshift/origin-${component}:v1.1.6'
>
> The router starts and the routes with edge encryption are being served
> with the default certificate. But i see lots os messages on the router
> container:
>
> W0515 20:03:47.648109   1 router.go:572] a edge terminated route with
> host test-prjtest.getup.io does not have the required certificates.  The
> route will still be created but no certificates will be written
>
> Is it something to worry ?
>
> ---
> Diego Castro / The CloudFather
> GetupCloud.com - Eliminamos a Gravidade
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>


-- 
Ram//
main(O,s){s=--O;10>4*s)*(O++?-1:1):10)&&\
main(++O,s++);}
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Openshift HA environment - keepalived high number of close syscalls

[Ram Ranganathan]

Haven't seen this till you mentioned it. I can see the close calls in my
local env. It looks like it happens in a new process - after a clone()
syscall at about a couple of seconds apart. So it is likely part of the
script that does the health check:
 script "
wrote:

> Using OSEv3.1.1
>
> I'm looking to setup sysdig in our native HA openshift environment, but
> having issues getting the agent to run on our infra nodes hosting
> keepalived and ha-proxy -- agent runs without issue on all the other nodes
> in our env.
>
> After the agent has been running about an hour or two, the node hangs and
> our hypervisor reports 100% cpu utilization. A power reset is the only
> option to bring the node back to life. The problem may be with keepalived
> doing an extremely large number(around 17 million in a minute) of "close"
> syscall operations, and it looks like those close operations are on any
> available fd. Is this expected behavior of keepalived running in an
> OSEv3.1.1 HA environment?
>
> Thanks!
>
>
>
>
>
>
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>



-- 
Ram//
main(O,s){s=--O;10>4*s)*(O++?-1:1):10)&&\
main(++O,s++);}
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: 503 service unavailable

[Ram Ranganathan]

Yeah, that should not matter. The routes + namespaces you would see are
based on the permissions of the service account.

I was able to get Dean on irc and ssh into his instance seeing something
wonky with the permissions.
CCing Jordan and Paul  for some help.

Inside the router container, I tried running this:
curl -k -vvv https://127.0.0.1:8443/api/v1/endpoints -H "Authorization:
Bearer $(http://fpaste.org/332733/45699454/


The token info from inside the router container (/var/run/secrets/
kubernetes.io/serviceaccount/token) seems to work if I use it
with oc login but not with the curl command - so it feels a bit odd.   Any
ideas what's amiss here?

Thanks,

Ram//



On Wed, Mar 2, 2016 at 11:56 PM, Dean Peterson <peterson.d...@gmail.com>
wrote:

> The router is on default namespace but the service pods are running on a
> different namespace.
>
> On Thu, Mar 3, 2016 at 1:53 AM, Julio Saura <jsa...@hiberus.com> wrote:
>
>> seems your router is running on default namespace, your pods are also
>> running on namespace default?
>>
>>
>> El 3 mar 2016, a las 7:58, Dean Peterson <peterson.d...@gmail.com>
>> escribió:
>>
>> I did do an "oc edit scc privileged" and made sure this was at the end:
>>
>> users:
>> - system:serviceaccount:openshift-infra:build-controller
>> - system:serviceaccount:management-infra:management-admin
>> - system:serviceaccount:default:router
>> - system:serviceaccount:default:registry
>>
>> router has always been a privileged user service account.
>>
>> On Thu, Mar 3, 2016 at 12:55 AM, Ram Ranganathan <rrang...@redhat.com>
>> wrote:
>>
>>> So you have no app level backends in that gist (haproxy.config file).
>>> That would explain the 503s - there's nothing there for haproxy to route
>>> to.  Most likely its due to the router service account has no permissions
>>> to get the routes/endpoints info from etcd.
>>> Check that the router service account (router default or whatever
>>> service account you used to start the router) is
>>> part of the privileged SCC and has read permissions to etcd.
>>>
>>>
>>> On Wed, Mar 2, 2016 at 10:43 PM, Dean Peterson <peterson.d...@gmail.com>
>>> wrote:
>>>
>>>> I created a public gist from the output:
>>>> https://gist.github.com/deanpeterson/76aa9abf2c7fa182b56c
>>>>
>>>> On Thu, Mar 3, 2016 at 12:35 AM, Ram Ranganathan <rrang...@redhat.com>
>>>> wrote:
>>>>
>>>>> You shouldn't need to restart the router. It should have created a new
>>>>> deployment and redeployed the router.
>>>>> So looks like the cause for your 503 errors is something else.
>>>>>
>>>>> Can you check that your haproxy.config file is correct (has the
>>>>> correct backends and servers).
>>>>> Either nsenter into your router docker container and cat the file or
>>>>> then run:
>>>>> oc exec  cat /var/lib/haproxy/conf/haproxy.config
>>>>>#  router-pod-name as shown in oc get pods
>>>>>
>>>>> Ram//
>>>>>
>>>>> On Wed, Mar 2, 2016 at 10:10 PM, Dean Peterson <
>>>>> peterson.d...@gmail.com> wrote:
>>>>>
>>>>>> I ran that "oc env dc router RELOAD_INTERVAL=5s" but I still get the
>>>>>> 503 error.  Do I need to restart anything?
>>>>>>
>>>>>> On Wed, Mar 2, 2016 at 11:47 PM, Ram Ranganathan <rrang...@redhat.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Dean, we did have a recent change to coalesce router reloads
>>>>>>> (default is 0s) and it looks like with that default we are more 
>>>>>>> aggressive
>>>>>>> with the reloads which could be causing this problem.
>>>>>>>
>>>>>>> Could you please try setting an environment variable ala:
>>>>>>> oc env dc router RELOAD_INTERVAL=5s
>>>>>>>#  or even 2s or 3s  - that's reload interval in seconds btw
>>>>>>># if you have a custom deployment config then replace the dc
>>>>>>> name router to that deployment config name.
>>>>>>>
>>>>>>> and see if that helps.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 2, 2016 at 6:21 PM, Dean Peterson <
>>>>>>> peterson.d...@gmail.com> wrote: