Re: Dynamic backends decided by an external service
Thanks a lot Thierry, that was it, changing to http-request solved my issue. I am now able to leverage a fully dynamic backend :) Thanks Sachin On 7/21/16, 10:52 PM, "thierry.fourn...@arpalert.org" wrote: >On Tue, 19 Jul 2016 15:28:25 +0530 >Sachin Shetty wrote: > >> Hi, >> >> We always had a unique requirement of picking a backend based on >>response >> from a external http service. In the past we have got this working by >> routing requests via a modified apache and caching the headers in maps >>for >> further request, but now I am trying to simplify our topology and >>trying to >> get this done using just haproxy and lua. >> >> I am running in to some problems: >> >> Lua method: >> >> function choose_backend(txn) >> >> local host = txn.http:req_get_headers()["host"][0] >> >> core.Alert("Getting Info:" .. host) >> >> local sock = core.tcp() >> >> sock:connect("127.0.0.1", 6280) >> >> sock:send("GET /eos/rest/private/gds/l1/1.0/domain/" .. host .. >> "\r\n") >> >> result = sock:receive("*a") >> >> sock:close() >> >> core.Alert("Received Response:" .. result .. "<") >> >> core.set_map("/tmp/proxy_webui.map", host, result) >> >> core.Alert("Map Set:" .. host .. "-->" .. result .. "<") >> >> end >> >> >> >> core.register_action("choose_backend", { "http-req" }, choose_backend) >> >> >> Haprpxy Conf: >> >> frontend luatest >> >> mode http >> >> maxconn 1 >> >> bind *:9000 >> >> >> >> use_backend %[hdr(host),lower,map(/tmp/proxy_webui.map)] if >>FALSE # >> To declare the map >> >> >> http-request lua.choose_backend >> >> >> tcp-request content capture hdr(host),map(/tmp/proxy_webui.map) >>len >> 80 >> >> acl is_ez_pod capture.req.hdr(0) http://127.0.0.1:6280 >> >> use_backend ez_pod if is_ez_pod >> >> >> >> >> >> backend ez_pod >> >> server ez_pod 192.168.56.101:6280 maxconn 2 >> >> >> >> >> >> There are some issues: >> 1. I do see Map Set called correctly in the logs, but the haproxy >>capture >> does not find the key in the map for the first request. Is this related >>to >> async execution of luna routine? > > >Hi, > >The processing of the request by haproxy is pending while the execution >of the lua action. > >The execution of "tcp-request content" directives are done before the >execution of "http-request". So HAProxy executes first the capture and >them the action which populate the map. > >Normally, HAProxy display a warning when it start, if it detects a >configuration order different than a execution order. Try to replace the >"tcp-request" by "http-request". > > >> 2. I expected subsequent requests to see the value atleast, but even >>that is >> not consistent, some request see the value in the map and some don¹t >> Is there a better way to do this? > > >It seems that you use a good way to do this. > > >> I already found that I cannot invoke the >> luna routine with use_backend because yield is not allowed in a sample >>fetch >> context. > > >Yes, the usage of core.tcp() require some yields, and haproxu doesn't >support yield during the execution of sample-fetches. For bypassing >this problem, I use variables. I store the expected result in a >variable during the execution of the action, and I use these variable >with the use_backend directive. > > >Thierry > > >> What would be the best way to achieve this, our requirement is similar >>to >> what can be done with redis+nginx here: >> http://openresty.org/en/dynamic-routing-based-on-redis.html except for >>we >> have an http service that decides the backend instead of a redis >>service. >> >> Thanks >> Sachin >> >>
Re: Dynamic backends decided by an external service
On Tue, 19 Jul 2016 15:28:25 +0530 Sachin Shetty wrote: > Hi, > > We always had a unique requirement of picking a backend based on response > from a external http service. In the past we have got this working by > routing requests via a modified apache and caching the headers in maps for > further request, but now I am trying to simplify our topology and trying to > get this done using just haproxy and lua. > > I am running in to some problems: > > Lua method: > > function choose_backend(txn) > > local host = txn.http:req_get_headers()["host"][0] > > core.Alert("Getting Info:" .. host) > > local sock = core.tcp() > > sock:connect("127.0.0.1", 6280) > > sock:send("GET /eos/rest/private/gds/l1/1.0/domain/" .. host .. > "\r\n") > > result = sock:receive("*a") > > sock:close() > > core.Alert("Received Response:" .. result .. "<") > > core.set_map("/tmp/proxy_webui.map", host, result) > > core.Alert("Map Set:" .. host .. "-->" .. result .. "<") > > end > > > > core.register_action("choose_backend", { "http-req" }, choose_backend) > > > Haprpxy Conf: > > frontend luatest > > mode http > > maxconn 1 > > bind *:9000 > > > > use_backend %[hdr(host),lower,map(/tmp/proxy_webui.map)] if FALSE # > To declare the map > > > http-request lua.choose_backend > > > tcp-request content capture hdr(host),map(/tmp/proxy_webui.map) len > 80 > > acl is_ez_pod capture.req.hdr(0) http://127.0.0.1:6280 > > use_backend ez_pod if is_ez_pod > > > > > > backend ez_pod > > server ez_pod 192.168.56.101:6280 maxconn 2 > > > > > > There are some issues: > 1. I do see Map Set called correctly in the logs, but the haproxy capture > does not find the key in the map for the first request. Is this related to > async execution of luna routine? Hi, The processing of the request by haproxy is pending while the execution of the lua action. The execution of "tcp-request content" directives are done before the execution of "http-request". So HAProxy executes first the capture and them the action which populate the map. Normally, HAProxy display a warning when it start, if it detects a configuration order different than a execution order. Try to replace the "tcp-request" by "http-request". > 2. I expected subsequent requests to see the value atleast, but even that is > not consistent, some request see the value in the map and some don¹t > Is there a better way to do this? It seems that you use a good way to do this. > I already found that I cannot invoke the > luna routine with use_backend because yield is not allowed in a sample fetch > context. Yes, the usage of core.tcp() require some yields, and haproxu doesn't support yield during the execution of sample-fetches. For bypassing this problem, I use variables. I store the expected result in a variable during the execution of the action, and I use these variable with the use_backend directive. Thierry > What would be the best way to achieve this, our requirement is similar to > what can be done with redis+nginx here: > http://openresty.org/en/dynamic-routing-based-on-redis.html except for we > have an http service that decides the backend instead of a redis service. > > Thanks > Sachin > >
Re: Dynamic backends decided by an external service
Hi, Any suggestions? Basically if I call http-request lua.choose_backend which seeds a map, will the map values be available in the subsequent map look up in the same request? Thanks Sachin From: Sachin Shetty Date: Tuesday, July 19, 2016 at 3:28 PM To: "haproxy@formilux.org" Subject: Dynamic backends decided by an external service Hi, We always had a unique requirement of picking a backend based on response from a external http service. In the past we have got this working by routing requests via a modified apache and caching the headers in maps for further request, but now I am trying to simplify our topology and trying to get this done using just haproxy and lua. I am running in to some problems: Lua method: function choose_backend(txn) local host = txn.http:req_get_headers()["host"][0] core.Alert("Getting Info:" .. host) local sock = core.tcp() sock:connect("127.0.0.1", 6280) sock:send("GET /eos/rest/private/gds/l1/1.0/domain/" .. host .. "\r\n") result = sock:receive("*a") sock:close() core.Alert("Received Response:" .. result .. "<") core.set_map("/tmp/proxy_webui.map", host, result) core.Alert("Map Set:" .. host .. "-->" .. result .. "<") end core.register_action("choose_backend", { "http-req" }, choose_backend) Haprpxy Conf: frontend luatest mode http maxconn 1 bind *:9000 use_backend %[hdr(host),lower,map(/tmp/proxy_webui.map)] if FALSE # To declare the map http-request lua.choose_backend tcp-request content capture hdr(host),map(/tmp/proxy_webui.map) len 80 acl is_ez_pod capture.req.hdr(0) http://127.0.0.1:6280 use_backend ez_pod if is_ez_pod backend ez_pod server ez_pod 192.168.56.101:6280 maxconn 2 There are some issues: 1. I do see Map Set called correctly in the logs, but the haproxy capture does not find the key in the map for the first request. Is this related to async execution of luna routine? 2. I expected subsequent requests to see the value atleast, but even that is not consistent, some request see the value in the map and some don¹t Is there a better way to do this? I already found that I cannot invoke the luna routine with use_backend because yield is not allowed in a sample fetch context. What would be the best way to achieve this, our requirement is similar to what can be done with redis+nginx here: http://openresty.org/en/dynamic-routing-based-on-redis.html except for we have an http service that decides the backend instead of a redis service. Thanks Sachin
Dynamic backends decided by an external service
Hi, We always had a unique requirement of picking a backend based on response from a external http service. In the past we have got this working by routing requests via a modified apache and caching the headers in maps for further request, but now I am trying to simplify our topology and trying to get this done using just haproxy and lua. I am running in to some problems: Lua method: function choose_backend(txn) local host = txn.http:req_get_headers()["host"][0] core.Alert("Getting Info:" .. host) local sock = core.tcp() sock:connect("127.0.0.1", 6280) sock:send("GET /eos/rest/private/gds/l1/1.0/domain/" .. host .. "\r\n") result = sock:receive("*a") sock:close() core.Alert("Received Response:" .. result .. "<") core.set_map("/tmp/proxy_webui.map", host, result) core.Alert("Map Set:" .. host .. "-->" .. result .. "<") end core.register_action("choose_backend", { "http-req" }, choose_backend) Haprpxy Conf: frontend luatest mode http maxconn 1 bind *:9000 use_backend %[hdr(host),lower,map(/tmp/proxy_webui.map)] if FALSE # To declare the map http-request lua.choose_backend tcp-request content capture hdr(host),map(/tmp/proxy_webui.map) len 80 acl is_ez_pod capture.req.hdr(0) http://127.0.0.1:6280 use_backend ez_pod if is_ez_pod backend ez_pod server ez_pod 192.168.56.101:6280 maxconn 2 There are some issues: 1. I do see Map Set called correctly in the logs, but the haproxy capture does not find the key in the map for the first request. Is this related to async execution of luna routine? 2. I expected subsequent requests to see the value atleast, but even that is not consistent, some request see the value in the map and some don¹t Is there a better way to do this? I already found that I cannot invoke the luna routine with use_backend because yield is not allowed in a sample fetch context. What would be the best way to achieve this, our requirement is similar to what can be done with redis+nginx here: http://openresty.org/en/dynamic-routing-based-on-redis.html except for we have an http service that decides the backend instead of a redis service. Thanks Sachin