Re: [ANNOUNCE] haproxy 1.5-dev9
Hi, Le 09/05/2012 07:04, Baptiste a écrit : Hi, Yes, appsession has been obsoleted by cookie and set-cookie stick tables pattern extraction (in HAProxy 1.5-dev7 as far as I remember). As an example: stick-table type string len 32 size 10K stick store-response set-cookie(PHPSESSID) stick on cookie(PHPSESSID) or, better, if your cookie is presented on the query string by the key session_id, then this would do the persistence as well: stick on url_param(session_id) I may be wrong but I think there are still some points where appsession can't be replaced by stick tables. - for example the use of path parameters : http://example.com/url;jsessionid=XXX - the use of cookie prefixes : I don't see how to match cookies (or path parameters/query string) like ASPSESSIONIDXXX=value (where XXX is dynamic). Maybe it still requires some development if we want to replace it by an equivalent with stick tables. -- Cyril Bonté
Re: [ANNOUNCE] haproxy 1.5-dev9
Hi Willy, On 09-05-2012 05:47, Willy Tarreau wrote: Hi Aleks, On Wed, May 09, 2012 at 12:26:38AM +0200, Aleksandar Lazic wrote: After all this changes is it still necessary to have the appsession directive in haproxy? Could it not be removed to avoid confusions and future question what should be used appsession or *cook*? I remember that there were a few special tricks in appsession, such as request-learn and things like this. I think that right now we have everything ready to emulate appsession, so maybe we could replace the whole thing with stick-tables and have the appsession keyword just alias the required configuration statements with a warning. This would enable appsession sharing between LBs, the thing we wanted to implement back in 2005 :-) Yes with the peers we are now in a much better setup ;-) Still I think a plugin witch is able to get/set data to memcached or Kyoto Tycoon would be a hugh benefit ;-) Are you interested in looking at this ? Yes, I will think about it an keep you informed ;-). Cheers
appsession redesign (was: Re: [ANNOUNCE] haproxy 1.5-dev9)
Hi all, On 09-05-2012 14:08, Willy Tarreau wrote: Hi Cyril, On Wed, May 09, 2012 at 08:55:45AM +0200, Cyril Bonté wrote: Hi, Le 09/05/2012 07:04, Baptiste a écrit : Hi, Yes, appsession has been obsoleted by cookie and set-cookie stick tables pattern extraction (in HAProxy 1.5-dev7 as far as I remember). As an example: stick-table type string len 32 size 10K stick store-response set-cookie(PHPSESSID) stick on cookie(PHPSESSID) or, better, if your cookie is presented on the query string by the key session_id, then this would do the persistence as well: stick on url_param(session_id) I may be wrong but I think there are still some points where appsession can't be replaced by stick tables. - for example the use of path parameters : http://example.com/url;jsessionid=XXX - the use of cookie prefixes : I don't see how to match cookies (or path parameters/query string) like ASPSESSIONIDXXX=value (where XXX is dynamic). That was one of the primary Idea, dynamic cookie/url Sessions. Maybe it still requires some development if we want to replace it by an equivalent with stick tables. You're perfectly right. Also the logic in appsession is somewhat different to the logic in stick-tables. For instance, if a client comes with an inexistant cookie and the server replaces it, then the client's cookie won't be learned. With the request-learn option, it's even trickier. The client's cookie will be learned from the URL parameter unless the server overwrites it in the response. I think it is fairly acceptable to change this behaviour a bit provided that what currently works still works. After all it's not really a problem if we store old expired entries in the table once in a while. But I want to insist (like you did) on the fact that there is not a 1:1 match right now. I would like to reuse as much code as possible from the current haproxy framework. I like the idea from Baptiste ### stick-table type string len 32 size 10K stick store-response set-cookie(PHPSESSID) stick on cookie(PHPSESSID) ### which only must be added to the appsession options ;-) http://haproxy.1wt.eu/git?p=haproxy.git;a=blob;f=doc/configuration.txt;h=9aa7e1c6d27b373aece27ef61b965d0b62a9b5a2;hb=HEAD#l1147 What I currently see that we need is following: 1.) We need a similar function like appsession_refresh() to clean up OLD entries from the $STORE (peers, anything else). http://haproxy.1wt.eu/git?p=haproxy.git;a=blob;f=src/appsession.c;h=ec97336e3f6873741c75e39c052fbe6df36473b3;hb=HEAD#l86 2.) Is there a similar hash function in Haproxy like appsession_hash_f() to avoid duplicate functions. http://haproxy.1wt.eu/git?p=haproxy.git;a=blob;f=src/sessionhash.c;h=80683900a1085e6f8e0d84555495576fd77ed423;hb=HEAD#l26 How difficult is it to use Consistent Hash implementation, is it useful to replace the appsession_hash_f with chash? http://haproxy.1wt.eu/git?p=haproxy.git;a=blob;f=src/lb_chash.c;h=58f1c9eaae8b15e984b9a5a341b2bdb11d8656f1;hb=HEAD 3.) Add the appsession to the balance keyword. http://haproxy.1wt.eu/git?p=haproxy.git;a=blob;f=doc/configuration.txt;h=9aa7e1c6d27b373aece27ef61b965d0b62a9b5a2;hb=HEAD#l1240 4.) What do you think to extend cook_beg(name) string with cook_beg(name) string [len] [prefix] [request-learn] [save_the_rest_of_the_cookie_after_prefix] scook_beg(name) string [len] [prefix] [save_the_rest_of_the_cookie_after_prefix] or with regex or so. I would start to dig again into haproxy framework and try to replace the dedicated solutions in **/appsession* **/appsession* with common haproxy functions. Another solution could be that the appsession is one of the first haproxy plugin ;-) ;-) What are your opinions? Is there the possibility for a chat or irc session? Cheers Aleks
[ANNOUNCE] haproxy 1.5-dev9
Hi all, I found some time to work on haproxy last weeks and to perform a number of fundamental changes that have been needed for a long time. First, while working on SSL and Compression at Exceliance, we found that the way the internal buffers and the HTTP message interact is really annoying. It comes from a long leftover of the migration which happened in 1.3 but it now had to come to an end. Some buffer manipulation functions have to deal with pointers that are copied into other places and because of this, some operations such as a simple realign are not possible. So I've changed the way it works. Now a buffer has a base (or origin) pointer, everything below it is from the past and is leaving the buffer. Everything above it is new and waiting for being forwarded. And HTTP messages don't hold and absolute pointer anymore, just offsets relative to the base pointer. The change was complex but the code is much more manageable and offers much more flexibility right now. Some of these changes conflicted with the ACL and pattern frameworks, so it was the right moment to merge them together. We now have a single sample fetch function for each type of data we want to extract, and both ACLs and patterns rely on this. The first user-visible benefit from this is that ACLs can now match cookies, URL parameters and arbitrary payloads. In practice, the current code is almost ready to enable session tracking on any input criteria. I thought I could make the track-sc1 and track-sc2 actions track headers but some more changes were needed that were out of the scope of all these changes, so I left them for later. Since some ACLs and pattern fetch methods supported an argument, a new argument management framework was implemented, making it very easy to declare variable number of typed arguments for new keywords. Thanks to this extension, I could bring new optional arguments to hdr() and cook() fetch methods to specify an occurrence number. This allows stick-tables to extract an IP address from a precise occurrence of the X-Forwarded-For header for instance, and to write ACLs which match such headers against networks found in files. Another point which had to be done was to automatically type the samples. Since the pattern framework supported automatic type casts, it was easy to complete this. Thanks to these types, we now support IPv6 ACLs, and the src and dst ACL/patterns are IPv4 or IPv6 depending on the data found. This is important because it means that it is now possible to mix v4 and v6 addresses in ACL patterns. As a side effect, the src6 and dst6 pattern fetches have been removed because they were redundant with src and dst. All these extensions required some improved parsing and error reporting. Thus I have implemented a simple and convenient error reporting framework based on a new memprintf() function which acts on a single pointer that is automatically reallocated and freed. A large number of config parsing options (specifically the ACL ones) which used to report error at line X are now able to say something like occurrence -20 too negative at argument 2 of hdr_ip(), must be = -10. I wish I've done this earlier, it's so simple, it took far less time to implement than the time it took to design without it in the past ! Along these things, the long-awaited use-server directive was introduced. It works as an exception to load balancing and persistence. It is convenient to avoid creating many backends when you want to select a server for a specific purpose (eg: monitoring). The log framework now learned to create, emit and log a unique request ID. Using the same syntax as log-format, it is possible to build a string which is supposed to uniquely identify a request in a given environment. This string is logged and emitted in headers so that everyone along the chain can log the same information, making it much easier to correlate events across large infrastructures. The error capture system was lacking a number of important information. I discovered this while trying to track a bug I have on my server, which causes invalid contents to sometimes be emitted and blocked by haproxy which logs them. Unfortunately, the level of information made these traces inexploitable. Now there are additional information such as the client's source port, all known internal flags, the position in the stream and the length of the last chunk. This will probably help when I get the error again. Another point, I found an uninitialized entry in a structure which made me waste 2 hours because on one machine, the first malloc() returned a zeroed area while on another one it was not the case. So I have added a command line option to enable memory poisonning. It immediately gave me another occurrence which I fixed :-) However I think the code is safe now. A number of other minor issues were fixed : - balance source did not properly hash IPv6 addresses (Alex Markham) - logformat could
Re: [ANNOUNCE] haproxy 1.5-dev9
I thought I could make the track-sc1 and track-sc2 actions track headers but some more changes were needed that were out of the scope of all these changes, so I left them for later. That is really sad :) Hopefully you'll be able to add string tracking to track-sc[12] soon, cause we'll be able to do great things :) cheers
Re: [ANNOUNCE] haproxy 1.5-dev9
On Tue, May 08, 2012 at 11:38:53PM +0200, Baptiste wrote: I thought I could make the track-sc1 and track-sc2 actions track headers but some more changes were needed that were out of the scope of all these changes, so I left them for later. That is really sad :) No it's not sad because the code is really taking shape and such features become easier to add day after day. Hopefully you'll be able to add string tracking to track-sc[12] soon, cause we'll be able to do great things :) I know but as you're well aware, the most important for me is to ensure that we can concurrently work on this code. So I sometimes prefer delay minor features to focus on architectural changes which allow multiple persons to develop in parallel. This is the most important as I'm still too much the bottleneck. Willy
Re: [ANNOUNCE] haproxy 1.5-dev9
I know but as you're well aware, the most important for me is to ensure that we can concurrently work on this code. So I sometimes prefer delay minor features to focus on architectural changes which allow multiple persons to develop in parallel. This is the most important as I'm still too much the bottleneck. Willy I know, but I was expecting that we could play with strings in stick tables with this release and so I'm just a bit disappointed :) Well, I'll wait a bit more time for it. cheers
Re: [ANNOUNCE] haproxy 1.5-dev9
Hi, On 08-05-2012 22:33, Willy Tarreau wrote: Hi all, I found some time to work on haproxy last weeks and to perform a number of fundamental changes that have been needed for a long time. [snipp] Some of these changes conflicted with the ACL and pattern frameworks, so it was the right moment to merge them together. We now have a single sample fetch function for each type of data we want to extract, and both ACLs and patterns rely on this. The first user-visible benefit from this is that ACLs can now match cookies, URL parameters and arbitrary payloads. In practice, the current code is almost ready to enable session tracking on any input criteria. I thought I could make the track-sc1 and track-sc2 actions track headers but some more changes were needed that were out of the scope of all these changes, so I left them for later. Since some ACLs and pattern fetch methods supported an argument, a new argument management framework was implemented, making it very easy to declare variable number of typed arguments for new keywords. Thanks to this extension, I could bring new optional arguments to hdr() and cook() fetch methods to specify an occurrence number. This allows stick-tables to extract an IP address from a precise occurrence of the X-Forwarded-For header for instance, and to write ACLs which match such headers against networks found in files. [snipp] After all this changes is it still necessary to have the appsession directive in haproxy? Could it not be removed to avoid confusions and future question what should be used appsession or *cook*? Cheers Aleks
Re: [ANNOUNCE] haproxy 1.5-dev9
Hi, Yes, appsession has been obsoleted by cookie and set-cookie stick tables pattern extraction (in HAProxy 1.5-dev7 as far as I remember). As an example: stick-table type string len 32 size 10K stick store-response set-cookie(PHPSESSID) stick on cookie(PHPSESSID) or, better, if your cookie is presented on the query string by the key session_id, then this would do the persistence as well: stick on url_param(session_id) You can use peers section and to share the content of the table between two LBs and to recover your table after a reload of haproxy. regards
