Hi Thierry, haproxy-list,
Op 19-10-2015 om 11:24 schreef thierry.fourn...@arpalert.org:
On Mon, 19 Oct 2015 01:31:42 +0200
PiBa-NL <piba.nl....@gmail.com> wrote:
Hi Thierry,
Op 18-10-2015 om 21:37 schreef thierry.fourn...@arpalert.org:
On Sun, 18 Oct 2015 00:07:13 +0200
PiBa-NL <piba.nl....@gmail.com> wrote:
Hi haproxy list,
For testing purposes i am trying to 'modify' a response of a webserver
but only having limited success. Is this supposed to work?
As a more usefull goal than the current LAL to TST replacement i imagine
rewriting absolute links on a webpage could be possible which is
sometimes problematic with 'dumb' webapplications..
Or is it outside of the current scope of implemented functionality? If
so, it on the 'lua todo list' ?
I tried for example a configuration like below. And get several
different results in the browser.
-Sometimes i get 4 times TSTA
-Sometimes i see after the 8th TSTA- Connection: keep-alive << this
happens most of the time..
-Sometimes i get 9 times TSTA + STOP << this would be the desired
outcome (only seen very few times..)
Probably due to the response-buffer being filled differently due to
'timing'..
The "connection: keep-alive" text is probably from the actual server
reply which is 'appended' behind the response generated by my lua
script.?. However shouldn't the .done() prevent that from being send to
the client?
Ive tried putting a loop into the lua script to call res:get() multiple
times but that didnt seem to work..
Also to properly modify a page i would need to know all changes before
sending the headers with changed content-length back to the client..
Can someone confirm this is or isn't (reliably) possible? Or how this
can be scripted in lua differently?
Hello,
Your script replace 3 bytes by 3 bytes, this must run with HTTP, but if
your replacement change the length of the response, you can have some
difficulties with clients, or with keepalive.
Yes i started with replacing with the same number of bytes to avoid some
of the possible troubles caused by changing the length.. And as seen in
the haproxy.cfg it is configured with 'mode http'.
The res:get(), returns the current content of the response buffer.
Maybe it not contains the full response. You must execute a loop with
regular "core.yield()" to get back the hand to HAProxy and wait for new
Calling yield does allow to 'wait' for more data to come in.. No
guarantee that it only takes 1 yield for data to 'grow'..
[info] 278/055943 (77431) : luahttpresponse Content-Length XYZ: 14115
[info] 278/055943 (77431) : luahttpresponse SIZE: 2477
[info] 278/055943 (77431) : luahttpresponse LOOP
[info] 278/055943 (77431) : luahttpresponse SIZE: 6221
[info] 278/055943 (77431) : luahttpresponse LOOP
[info] 278/055943 (77431) : luahttpresponse SIZE: 7469
[info] 278/055943 (77431) : luahttpresponse LOOP
[info] 278/055943 (77431) : luahttpresponse SIZE: 7469
[info] 278/055943 (77431) : luahttpresponse LOOP
[info] 278/055943 (77431) : luahttpresponse SIZE: 7469
[info] 278/055943 (77431) : luahttpresponse LOOP
[info] 278/055943 (77431) : luahttpresponse SIZE: 7469
[info] 278/055943 (77431) : luahttpresponse LOOP
[info] 278/055943 (77431) : luahttpresponse SIZE: 7469
[info] 278/055943 (77431) : luahttpresponse LOOP
[info] 278/055943 (77431) : luahttpresponse SIZE: 7469
[info] 278/055943 (77431) : luahttpresponse LOOP
[info] 278/055943 (77431) : luahttpresponse SIZE: 7469
[info] 278/055943 (77431) : luahttpresponse LOOP
[info] 278/055943 (77431) : luahttpresponse SIZE: 8717
[info] 278/055943 (77431) : luahttpresponse LOOP
[info] 278/055943 (77431) : luahttpresponse SIZE: 14337
[info] 278/055943 (77431) : luahttpresponse DONE?: 14337
data. When all the data are read, res:get() returns an error.
Not sure when/how this error would happen.? The result of res:get only
seems to get bigger while the webserver is sending the response..
The res:send() is dangerous because it send data directly to the client
without the end of haproxy analysis. Maybe it is the cause o your
problem.
Try to use res:set().
Ok tried that, new try with function below.
The difficulty is that another "res:get()" returns the same data that
these you put.
I don't known if you can modify an http response greater than one
buffer.
Would be nice if that was somehow possible. But my current lua script
cannot..
The function res:close() closes the connection even if HAProxy want to
keep the connection alive. I suggest that you don't use this function.
It seems txn.res:close() does not exist? txn:done()
I reproduce the error message using curl. By default curl tries
to transfer data with keepalive, and it is not happy if the all the
announced data are not transfered.
Connection: keep-alive curl: (18) transfer closed with outstanding
read data remaining
It seems that i reproduce a bug. I'm looking for.
Ok if you can create a patch, let me know. Happy to test if it solves
some of the issues i see.
Hi,
I catch the error. While we execute http actions, the content of the
buffer must contains the http headers, otherwise, the request is
invalid.
It is because the http action are applied on the content and not on
the stream. So this is th behaviour:
-> The function res:get() remove the data from the haproxy input buffer
-> The function res:send() send the modified data directly to the
client (throught the output buffer), this data is not yet avalaible
in the input buffer.
-> Now HAProxy is locked in the processing, because it wait for a
valid reponse header for continuing the Lua processing.
This behaviour is not really a bug, it is the process who garantee that
a valid http response is present before continuing the processing.
Maybe, a patch will send an error in the case of we enter in an action
with a valid request/response and we out from the processing without a
valid request/response.
So, regarding the HAProxy global behaviour, the action are designed for
manipulating http header, and not the body.
If you want to manipulates the body, you can use a tcp action.
Ok the script below works, even for larger responses and sets the new
content-length.. That is.. for my single test page i tried it with.
It could be troublesome that the lua script must take into account every
variation that might happen like chunked/compressed bodies, and parsing
the headers. Was kinda hoping that haproxy in http mode would be able to
take care of at least some of these kind of things. All advantages of
running a haproxy frontent in 'mode http' are now nolonger available so
thats a downside in my opinion..
Also the script below buffers the whole response (which must have a
content-length header) before forwarding it to the client. This could
significantly increase the memoryusage of a single connection through
haproxy.. But hey, it does open up a world of new 'rewriting'
possibilities for those backends that really really really need it..
Anyway i wanted to share this for those people that might be in need of
doing something similar..
And if anyone knows a way to simplify any part of the lua script please
let me know :)
Yes searching the start and end of the content-length header twice isnt
needed, and the txn:Info() statements should be removed.. but im asking
more for other 'major' improvements.
Best regards,
PiBa-NL
listen proxytcpresponse
bind :10009
mode tcp
tcp-response content lua.luatcpresponse
server x 192.168.0.40:302
function luatcpresponse(txn)
txn:Info("#### TCP RESPONSE ENTERING LUA FUNCTION ###\n")
local responsecomplete = ""
local headersparsed = false
local headerlength = 0
local contentsize = 0
local response2 = txn.res:get()
local headers = ""
while (response2) and (string.len(response2) > 0) do
txn:Info("lua tcp response LOOP")
txn:Info("lua tcp response SIZE: " .. string.len(response2))
response2 = string.gsub(response2,"LAL","TST")
responsecomplete = responsecomplete .. response2
if (not headersparsed) and
(string.find(responsecomplete,"\r\n\r\n") > 0) then
txn:Info("headers complete")
headersparsed = true
_ , headerlength = string.find(responsecomplete,"\r\n\r\n")
headers = string.sub(responsecomplete,0,headerlength)
local cl,cle = string.find(headers, "Content%-Length: ")
local sizeend = string.find(responsecomplete,"\r\n", cle)
contentsize = tonumber(string.sub(headers, cle+1,
sizeend-1))
txn:Info("Headersize: " .. headerlength)
txn:Info("Content-Length: " .. contentsize)
responsecomplete = string.sub(responsecomplete,
headerlength+1)
end
txn:Info("RES size: " .. string.len(responsecomplete) .. "
H:" .. headerlength .. " S:" .. contentsize)
if (string.len(responsecomplete) == contentsize) then
txn:Info("RESPONSE COMPLETE")
local i = 0
-- remove the scheme and host part from response href
links.. (nothing checked..)
responsecomplete =
string.gsub(responsecomplete,"href=\"http://localhost:302/","href=\"/";)
local ressize = string.len(responsecomplete)
local cl,cle = string.find(headers, "Content%-Length: ")
local sizeend = string.find(headers,"\r\n", cle)
local firstheaders = string.sub(headers, 0, cl-1)
local lastheaders = string.sub(headers, sizeend)
headers = firstheaders .. "Content-Length: " .. ressize
.. lastheaders
txn.res:send(headers .. responsecomplete)
responsecomplete = ""
headersparsed = false
end
response2 = txn.res:get()
end
txn:Info("TCP RESPONSE LUA FUNCTION, EXIT\n")
end
core.register_action("luatcpresponse" , { "tcp-res" }, luatcpresponse);
You can use also an http action, but the buffer must contains a valid
http request each time when HAProxy give back the hand. In this case,
you cannot modify requests greater than an haproxy buffer.
res:get() -> get all the data and remove it from the input buffer
res:dup() -> just duplicates data
res:set() -> set data in the input buffer, HAProxy will continue the
analysis.
res:send()-> send data in the output buffer, HAProy cannot analyse
these data.
Thierry
Thierry
This function seems to work for responses up to +-15KB.
Sometimes the number of loops it runs is different, and it seems kinda
in-efficient to just run loops until the response is 'complete', another
strange observation is that the res:set inside the loop is required,
even though it doesn't set a modified response, eventually the complete
response is modified in the browser result. Second request over a
keep-alive connection also fails. Adding http-server-close also closes
the client connection, but does avoid the problem with the second
request.. In the lua script I dont account for headers size yet when
checking if the response is completely read, but i dont think thats
affected the test..
Is there anything i can do to improve this function? (Besides removing
the txn:Info() lines.)
function luahttpresponse(txn)
local resheaders = txn.http:res_get_headers()
local contentlength = tonumber(resheaders["content-length"][0])
local response2 = txn.res:get()
txn:Info("luahttpresponse Content-Length XYZ: " .. contentlength)
txn:Info("luahttpresponse SIZE: " .. string.len(response2))
while string.len(response2) < contentlength do
txn:Info("luahttpresponse LOOP")
txn.res:set(response2)
core.yield()
response2 = txn.res:get()
txn:Info("luahttpresponse SIZE: " .. string.len(response2))
end
response2 = string.gsub(response2,"LAL","TST")
txn.res:set(response2)
txn:Info("luahttpresponse DONE?: " .. string.len(response2))
end
Regards,
PiBa-NL
Thanks in advance,
PiBa-NL
## haproxy.cfg
listen proxyresponse
bind :10006
mode http
http-response lua.luahttpresponse
server x 192.168.0.40:302
## script.lua
function luahttpresponse(txn)
local response2 = txn.res:get()
response2 = string.gsub(response2,"LAL","TST")
txn.res:send(response2)
txn.done()
end
core.register_action("luahttpresponse" , { "http-res" }, luahttpresponse);
## webpage.aspx , with 2.7KB output.
------------------
<%@ Page Language="C#" %>
<html><body>START
<% var x = new String('x', 250);
x = "<input type=\"hidden\" value=\""+x+"\" />"; %>
1 -LALA- <% Response.Write(x); %>
2 -LALA- <% Response.Write(x); %>
3 -LALA- <% Response.Write(x); %>
4 -LALA- <% Response.Write(x); %>
5 -LALA- <% Response.Write(x); %>
6 -LALA- <% Response.Write(x); %>
7 -LALA- <% Response.Write(x); %>
8 -LALA- <% Response.Write(x); %>
9 -LALA- <% Response.Write(x); %>
STOP
--------------------
