Hi everyone, Thanks Willy for the introduction.
I join the first part of the document. All the function are not yet available. The doc uses a markdown format like that uses GitHub. I join the original format, and the HTML conversion. The next steps are: - collecting users ideas and bug reports, - add missing functions like native header manipulations, - check the usage of existing libraries, - change the 'sleep' engine, - add execution timeout to prevent infinite loops, - some other tasks ... Below, I add the HAProxy configuration documentation: lua-load <file> scope: global This function declares and load the Lua file. A syntax parsing is executed and this directive fails if the syntax contain errors. The main Lua code is executed. This code can be used to initialize the Lua bindings in HAProxy. tcp-request content lua <function> scope: frontend / backend / listen This keyword registers and Lua function executed when a tcp connection receive content. tcp-response content lua <function> scope: frontend / backend / listen This keyword registers and Lua function executed when a tcp connection receive content from the response channel. http-request lua <function> scope: frontend / backend / listen This keyword registers and Lua function executed when an http request is sent. http-response lua <function> scope: frontend / backend / listen This keyword registers and Lua function executed when an http response is received. enjoy Thierry On Sat, 28 Feb 2015 23:33:56 +0100 Willy Tarreau <w...@1wt.eu> wrote: > Hi folks, > > this is just a quick mail to let you know that I've just merged the recent > work from my coworker Thierry Fournier (in Cc) about supporting Lua scripting > in haproxy. I won't go into lengths explaining how that's supposed to be used > because I don't really know myself, but I'll let him provide some explanations > and working examples. The real purpose would be to get some feedback, > suggestions, criticisms to get further into this. > > At the moment it is possible to declare some Lua-based sample fetch functions, > some converters as well and even to create new tcp-request and http-req/resp > actions. These last ones (the actions) support yielding so that it is possible > to make blocking calls from the Lua code and benefit from haproxy's native > session management and scheduling to switch to something else while things > are making progress. Thus the whole system is totally non-blocking and > event-driven, even if in Lua you appear to be blocking. > > The long term goal is to be able to quickly provide some features that are > not yet possible natively, and eventually to implement some of them natively > if they're often requested or have a performance impact. Some of the most > complex parts such as writing authentication systems relying on side-traffic > would possibly remain written in Lua only, except if we manage to see a big > use of certain things and we see a way to implement them cleanly. > > Thierry is busy these days, and that's why I asked him to push his code > prior to switching to other tasks despite the doc not being there yet. So > it's possible we'll have to wait a bit for some doc, but we'll really value > testing, bug reports and feedback. Nothing is set in stone yet, he does have > some ideas on how to improve things. I do still have some concerns about > long-term stability and resource usage, as you can expect. So use it as a > toy first, play with it and tell us if you like it or not, and if not, why. > > Thanks! > Willy > >
channel.fig
Description: application/xfig
How Lua runs in HAProxy
HAProxy Lua running contexts
The Lua code executed in HAProxy can be processed in 2 main modes. The first one is the initialisation mode, and the second is the runtime mode.
In the initialisation mode, we can perform DNS solves, but we cannot perform socket I/O. In this initialisation mode, HAProxy still blocked during the execution of the Lua program.
In the runtime mode, we cannot perform DNS solves, but we can use sockets. The execution of the Lua code is multiplexed with the requests processing, so the Lua code seems to be run in blocking, but it is not the case.
The Lua code is loaded in one or more files. These files contains main code and functions. Lua have 6 execution context.
The Lua file body context. It is executed during the load of the Lua file in the HAProxy
[global]section with the directivelua-load. It is executed in initialisation mode. This section is use for configuring Lua bindings in HAProxy.The Lua init context. It is an Lua function executed just after the HAProxy configuration parsing. The execution is in initialisation mode. In this context the HAProxy environment are already initialized. It is useful to check configuration, or initializing socket connections or tasks. These functions are declared in the body context with the Lua function
core.register_init(). The prototype of the function is a simple function without return value and without parameters, like this:function fcn().The Lua task context. It is an Lua function executed after the start of the HAProxy scheduler, and just after the declaration of the task with the Lua function
core.register_task(). This context can be concurrent with the traffic processing. It is executed in runtime mode. The prototype of the function is a simple function without return value and without parameters, like this:function fcn().The action context. It is an Lua function conditionally executed. These actions are declared by the HAProxy directives "
tcp-request content lua <function>", "tcp-response content lua <function>", "http-request lua <function>" and "http-response lua <function>". The prototype of the Lua called function is a function with doesn't returns anything and that take an object of class TXN as entry.function fcn(txn)The sample-fetch context. This function takes a TXN object as entry argument and returns a string. These types of function cannot execute any blocking function. They are useful to aggregate some of original HAProxy sample-fetches and return the result. The prototype of the function is
function string fcn(txn). These functions can be registered with the Lua functioncore.register_fetches(). Each declared sample-fetch is prefixed by the string "lua.".NOTE: It is possible that this function cannot found the required data in the original HAProxy sample-fetches, in this case, it cannot return the result. This case is not yet supported
The converter context. It is an Lua function that takes a string as input and returns another string as output. These types of function are stateless, it cannot access to any context. They don't execute any blocking function. The call prototype is
function string fcn(string). This function can be registered with the Lua functioncore.register_converters(). Each declared converter is prefixed by the string "lua.".
HAProxy Lua Hello world
HAProxy configuration file (hello_world.conf):
global
lua-load hello_world.lua
listen proxy
bind 127.0.0.1:10001
tcp-request content lua hello_world
HAProxy Lua file (hello_world.lua):
function hello_world(txn)
local res = txn:res_channel()
res:send("hello world\n")
end
How to start HAProxy for testing this configuration:
./haproxy -f hello_world.conf
On other terminal, you can test with telnet:
#:~ telnet 127.0.0.1 10001
coucou
core class
The "core" class contains all the HAProxy core functions. These function are useful for the controlling the execution flow, registering hooks, manipulating global maps or ACL, ...
core.add_acl(filename, key)
context: runtime, init except main code
Add the ACL key in the ACLs list referenced by the file filename.
core.del_acl(filename, key)
context: runtime, init except main code
Delete the ACL entry referenced by the key key in the list of ACLs referenced by filename.
core.del_map(filename, key)
context: runtime, init except main code
Delete the map entry indexed with the key key in the list of maps referenced by filename.
core.msleep(int milliseconds)
context: runtime except sample-fetch and converters
The core.sleep stop the Lua execution between milliseconds argument.
core.register_converters(string name, function func)
context: init
Register an Lua function executed as converter. All the registered converters can be used in HAProxy with the prefix "lua.". An converter get a string as input and return a string as output. The registered function can take up to 9 values as parameter. All the value are strings.
The prototype of the Lua function used as argument is:
function(object txn, string param1, string param2, ...)
core.register_fetches(string name, function func)
context: init
Register an Lua function executed as sample fetch. All the registered sample fetchs can be used in HAProxy with the prefix "lua.". A Lua sample fetch return a string as output. The registered function can take up to 9 values as parameter. All the value are strings.
The prototype of the Lua function used as argument is:
string function(object txn, string param1, string param2, ...)
lua example code:
core.register_fetches("hello", function()
return "hello"
end)
HAProxy example configuration:
frontend example
http-request redirect location "/%[lua.hello]"
core.register_init(function func)
context: init
Register a function executed after the configuration parsing. This is useful to check any parameters.
The prototype of the Lua function used as argument is:
function()
core.register_task(function func)
context: init, runtime
Register and start independent task. The task is started when the HAProxy main scheduler starts. For example this type of tasks can be executed to perform complex health checks.
The prototype of the Lua function used as argument is:
function()
core.set_nice(int nice)
context: runtime, init except main code
Change the nice of the current task or current session. The nice must be between -1024 and 1024.
core.set_map(filename, key, value)
context: runtime, init except main code
set the value value associated to the key key in the map referenced by filename.
core.sleep(int seconds)
context: init, runtime except sample-fetch and converters
The core.sleep() functions stop the Lua execution between seconds argument.
socket core.tcp()
context: runtime, init except main code
This function returns a new object of a socket class.
socket core.yield()
context: runtime
Give back the hand at the HAProxy scheduler. It is used when the LUA processing consumes a lot of processing time.
txn class
The txn class contain all the functions relative to the http or tcp transaction (Note than a tcp stream is the same than a tcp transaction, but an HTTP transaction is not the same than a tcp stream).
The functions found permits to retrieve data from the requests, alter it and forward it.
All the functions provided by this class are available in the converters, sample-fetches, http-request, http-response and tcp-request.
data txn.get_priv(txn)
Return Lua data stored in the current transaction. If no data are stored, it returns a nil value.
txn.set_priv(txn, data)
Store any data in the current HAProxy transaction. This action replace the old stored data.
txn.<fetch>(txn, ...)
The transaction class (txn) contains a function for each HAProxy native sample-fetch. The list of usable sample-fetches and their usage are available in the HAProxy main documention.
txn.get_headers(txn)
This function returns an array of headers.
channel txn.get_req_buf(txn)
This function returns an object able to manipulate the request buffer.
channel txn.get_res_buf(txn)
This function returns an object able to manipulate the response buffer.
txn.finish(txn)
This function finish the current transaction. The data waiting in the buffers is send and the transaction is terminated.
txn.close(txn)
This function close the transaction and the associated session. It can be used when a critical error is detected.
txn.http.redirect(txn, location)
txn.http.req.add_header(txn, name, value)
txn.http.req.set_method(txn, string)
txn.http.req.set_path(txn, string)
txn.http.req.set_query(txn, string)
txn.http.req.set_uri(txn, string)
txn.http.req.set_header(txn, name, value)
txn.http.req.del_header(txn, name)
txn.http.req.replace_header(txn, name, regex, string)
txn.http.req.replace_value(txn, name, regex, string)
txn.http.res.set_header(txn, name, value)
txn.http.res.del_header(txn, name)
txn.http.res.replace_header(txn, name, regex, string)
txn.http.res.replace_value(txn, name, regex, string)
channel class
HAProxy uses two buffers for the processing of the requests. The first one is used with the request data (from the client to the server) and the second is used for the response data (from the server to the client).
Each buffer contains two types of data. The first type is the incoming data waiting for a processing. The second part is the outgoing data already processed. Usually, the incoming data is processed, after it is tagged as outgoing data, and finally it is sent. The following functions provides tools for manipulating these data in a buffer.
The following diagram shows where the channel class function are applied.
string channel.dup(channel)
This function returns a string that contain the entire buffer. The data is not remove from the buffer and can be reprocessed later.
If the buffer cant receive more data, a 'nil' value is returned.
string channel.get(channel)
This function returns a string that contain the entire buffer. The data is consumed from the buffer.
If the buffer cant receive more data, a 'nil' value is returned.
string channel.get_line(channel)
This function returns a string that contain the first line of the buffer. The data is consumed. If the data returned doesn't contains a final '\n' its assumed than its the last avalaible data in the buffer.
If the buffer cant receive more data, a 'nil' value is returned.
int channel.set(channel, string)
This function replace the content of the buffer by the string. The function returns the copied length, otherwise, it returns -1.
The data set with this function are not send. They wait for the end of HAProxy processing, so the buffer can be full.
int channel.append(channel, string)
This function append the string argument to the content of the buffer. The function returns the copied length, otherwise, it returns -1.
The data set with this function are not send. They wait for the end of HAProxy processing, so the buffer can be full.
int channel.send(channel, string)
This function required immediate send of the data. Unless if the connection is close, the buffer is regularly flushed and all the string can be sent.
int channel.get_in_length(channel)
This function returns the length of the input part of the buffer.
int channel.get_out_length(channel)
This function returns the length of the output part of the buffer.
channel.forward(channel, int)
This function transfer bytes from the input part of the buffer to the output part.
socket class
This class must be compatible with the Lua Socket class. Only the 'client' functions are available. See the Lua Socket documentation:
http://w3.impa.br/~diego/software/luasocket/tcp.html
socket.close(socket)
Closes a TCP object. The internal socket used by the object is closed and the local address to which the object was bound is made available to other applications. No further operations (except for further calls to the close method) are allowed on a closed socket.
Note: It is important to close all used sockets once they are not needed, since, in many systems, each socket uses a file descriptor, which are limited system resources. Garbage-collected objects are automatically closed before destruction, though.
socket.connect(socket, address, port)
Attempts to connect a socket object to a remote host.
Address can be an IP address or a host name. Port must be an integer number in the range [1..64K).
In case of error, the method returns nil followed by a string describing the error. In case of success, the method returns 1.
Note: The function socket.connect is available and is a shortcut for the creation of client sockets.
Note: Starting with LuaSocket 2.0, the settimeout method affects the behavior of connect, causing it to return with an error in case of a timeout. If that happens, you can still call socket.select with the socket in the sendt table. The socket will be writable when the connection is established.
socket.connect_ssl(socket, address, port)
Same behavior than the function socket:connect, but uses SSL.
socket.getpeername(socket)
Returns information about the remote side of a connected client object.
Returns a string with the IP address of the peer, followed by the port number that peer is using for the connection. In case of error, the method returns nil.
socket.getsockname(socket)
Returns the local address information associated to the object.
The method returns a string with local IP address and a number with the port. In case of error, the method returns nil.
socket.receive(socket, [pattern [, prefix]])
Reads data from a client object, according to the specified read pattern. Patterns follow the Lua file I/O format, and the difference in performance between all patterns is negligible.
Pattern can be any of the following:
*a: reads from the socket until the connection is closed. No end-of-line translation is performed;*l: reads a line of text from the socket. The line is terminated by a LF character (ASCII 10), optionally preceded by a CR character (ASCII 13). The CR and LF characters are not included in the returned line. In fact, all CR characters are ignored by the pattern. This is the default pattern.number: causes the method to read a specified number of bytes from the socket. Prefix is an optional string to be concatenated to the beginning of any received data before return.
If successful, the method returns the received pattern. In case of error, the method returns nil followed by an error message which can be the string 'closed' in case the connection was closed before the transmission was completed or the string 'timeout' in case there was a timeout during the operation. Also, after the error message, the function returns the partial result of the transmission.
Important note: This function was changed severely. It used to support multiple patterns (but I have never seen this feature used) and now it doesn't anymore. Partial results used to be returned in the same way as successful results. This last feature violated the idea that all functions should return nil on error. Thus it was changed too.
socket.send(socket, data [, i [, j]])
Sends data through client object.
Data is the string to be sent. The optional arguments i and j work exactly like the standard string.sub Lua function to allow the selection of a substring to be sent.
If successful, the method returns the index of the last byte within [i, j] that has been sent. Notice that, if i is 1 or absent, this is effectively the total number of bytes sent. In case of error, the method returns nil, followed by an error message, followed by the index of the last byte within [i, j] that has been sent. You might want to try again from the byte following that. The error message can be 'closed' in case the connection was closed before the transmission was completed or the string 'timeout' in case there was a timeout during the operation.
Note: Output is not buffered. For small strings, it is always better to concatenate them in Lua (with the '..' operator) and send the result in one call instead of calling the method several times.
socket.setoption(socket, option [, value])
Just implemented for compatibility, this cal does nothing.
socket.settimeout(socket, value [, mode])
Changes the timeout values for the object. All I/O operations are blocking. That is, any call to the methods send, receive, and accept will block indefinitely, until the operation completes. The settimeout method defines a limit on the amount of time the I/O methods can block. When a timeout time has elapsed, the affected methods give up and fail with an error code.
The amount of time to wait is specified as the value parameter, in seconds.
The timeout modes are bot implemented, the only settable timeout is the inactivity time waiting for complete the internal buffer send or waiting for receive data.
External Lua libraries
A lot of useful lua libraries can be found here:
Redis acces:
Operations on bits:
OpenSSL:
lua_api.md
Description: Binary data
