Re: Complete rewrite of HAProxy in Lua
Hi Aleks, On Thu, Apr 02, 2015 at 03:39:27PM +0200, Aleksandar Lazic wrote: (...) I think it would be worth to check how much time costs to switch from c to lua and back. I already tried. On my laptop I was seeing 55k conn/s on a simple test which was building a redirect from a sample fetch declared in Lua which was concatenating 3 other fetch results and calling crc32 on them. I did it just for fun and was pleased to see that the overhead is very low. (...) But as every time then comes the Users and say haproxy is slow because there LUA-Script is full of flaws ;-/ This is inevitable. Several years ago, I've seen people saturate an F5-8800 with just a few hundreds of requests per second while this device was supposed to be capable of hundreds of thousands, just because they completely filled the machine with thousands of iRules. I was disgusted just by thinking about it... ordering a machine designed for high traffic, involving hardware accelerators for L4/SSL, just to immediately set a bottleneck using a massive amount of L7 rules is absurd. I would have liked to exchange it for a PC running haproxy to resell it later and get rich, but they preferred to upgrade it. Some people like to use the wrong tool for a given job, just because it's possible. So we'll see it as well with haproxy/lua (and we'll be able to point them to this thread). BTW the absolute record I've seen in a real configuration for haproxy was 45 named ACLs and their respective use_backend rules to implement geolocation by hand! It did not run very fast to be honnest, just enough for a medium site :-) Cheers, Willy
Re: Complete rewrite of HAProxy in Lua
Hi Willy. Am 03-04-2015 13:24, schrieb Willy Tarreau: Hi Aleks, On Thu, Apr 02, 2015 at 03:39:27PM +0200, Aleksandar Lazic wrote: (...) I think it would be worth to check how much time costs to switch from c to lua and back. I already tried. On my laptop I was seeing 55k conn/s on a simple test which was building a redirect from a sample fetch declared in Lua which was concatenating 3 other fetch results and calling crc32 on them. I did it just for fun and was pleased to see that the overhead is very low. Thanks for sharing the information. (...) But as every time then comes the Users and say haproxy is slow because there LUA-Script is full of flaws ;-/ This is inevitable. Several years ago, I've seen people saturate an F5-8800 with just a few hundreds of requests per second while this device was supposed to be capable of hundreds of thousands, just because they completely filled the machine with thousands of iRules. I was disgusted just by thinking about it... ordering a machine designed for high traffic, involving hardware accelerators for L4/SSL, just to immediately set a bottleneck using a massive amount of L7 rules is absurd. I would have liked to exchange it for a PC running haproxy to resell it later and get rich, but they preferred to upgrade it. Some people like to use the wrong tool for a given job, just because it's possible. So we'll see it as well with haproxy/lua (and we'll be able to point them to this thread). Full ack. BTW the absolute record I've seen in a real configuration for haproxy was 45 named ACLs and their respective use_backend rules to implement geolocation by hand! It did not run very fast to be honest, just enough for a medium site :-) I was glad that I sat when I read this. *WOW* Good work Willy and contributors ;-) Cheers Aleks
Re: Complete rewrite of HAProxy in Lua
Hi. nice try ;-) But some parts are not that bad, IMHO. Am 01-04-2015 10:43, schrieb Willy Tarreau: Hi, As some might have noticed, HAProxy development is progressively slowing down over time. I have analyzed the situation and came to the following conclusions : [snipp] Recently with the Lua addition, we found it to be quite fast. Maybe not as fast as C, but Lua is improving and C skills are diminishing, so I guess that in a few years the code written in Lua will be much faster than the code we'll be able to write in C. Thus I found it wise to declare a complete rewrite of HAProxy in Lua. It comes with many benefits. First, Lua is easy to learn, we'll get many more developers and contributors. One of the reason is that you don't need to care about resource allocation anymore. What's the benefit of doing an strdup() to keep a copy of a string when you can simply do a = b without having to care about the memory used behind. Machines are huge nowadays, much larger than the old Athlon XP I was using 10 years ago. Second, Lua doesn't require a compiler, so we'll save 30 minutes a day per 200 builds, this will definitely speed up development for each developer. And we won't depend on a given C compiler, won't be subject to its bugs, and more importantly we'll be able to get rid of the few lines of assembly that we currently have in some performance-critical parts. Third, last version of HAProxy saw a lot of new sample fetch functions and converters. This will not be needed anymore, because the code and the configuration will be mixed together, just as everyone does with Shell scripts. This means that any config will just look like an include directive for the haproxy code, followed by some code to declare the configuration. It will then be possible to create infinite combinations of new functions, and the configuration will have access to anything internal to HAProxy. This part. Sounds like mod_perl/mod_lua for apache or mod_lua for nginx. I think it would be worth to check how much time costs to switch from c to lua and back. Looks like this questions is answerd. 1.8 Optimization tips? http://www.luafaq.org/#T1.8 6.1 LuaJIT is significantly faster than vanilla Lua. What's the catch? http://www.luafaq.org/#T6.1 But as every time then comes the Users and say haproxy is slow because there LUA-Script is full of flaws ;-/ They should read http://www.luafaq.org/ before code *SHOULD*. Who do what he/she should ;-) In the end, of the current HAProxy will only remain the Lua engine, and probably by then we'll find even better ones so that haproxy will be distributed as a Lua library to use anywhere, maybe even on IoT devices if that makes sense (anyone ever dreamed of having haproxy in their watches ?). From my point of view this was the funniest part ;-). [snipp] Say us good luck! Willy Good luck. Cheers Aleks
Complete rewrite of HAProxy in Lua
Hi, As some might have noticed, HAProxy development is progressively slowing down over time. I have analyzed the situation and came to the following conclusions : - the code base is increasing and is becoming slower to build day after day. Ten years ago, version 1.1.31 was only 6716 lines everything included. Today, mainline is 108395 lines, or 16 times larger. - gcc is getting slower over time. Since version 2.7.2 I used to rely on ten years ago, we've seen important slowdowns with v2.95, several v3.x then v4.x. I'm currently on 4.7 and afraid to upgrade. - while the whole code base used to build in less than a second ten years ago on an Athlon XP-1800, now it takes about 10 seconds on a core i5 at 3 GHz. Multiply this by about 200 builds a day and you see that half an hour is wasted every single day dedicated to development. That's about 1/4 of the available time if you count the small amount of time available after processing e-mails. - people don't learn C anymore at school and this makes it harder to get new contributors. In fact, most of those who are proficient in C already have a job and little spare time to dedicate to an opensource project. In parallel, I'm seeing I'm getting old, I turned 40 last year and it's obvious that I'm not as much capable of optimizing code as I used to be. I'm of the old school, still counting the CPU cycles it takes a function to execute, the nanoseconds required to append an X-Forwarded-For header or to parse a cookie. And all of this is totally wasted when people run the software in virtual machines which only allocate portions of CPUs (ie they switch between multiple VMs at high rate), or install it in front of applications which saturate at 100 requests a second. Recently with the Lua addition, we found it to be quite fast. Maybe not as fast as C, but Lua is improving and C skills are diminishing, so I guess that in a few years the code written in Lua will be much faster than the code we'll be able to write in C. Thus I found it wise to declare a complete rewrite of HAProxy in Lua. It comes with many benefits. First, Lua is easy to learn, we'll get many more developers and contributors. One of the reason is that you don't need to care about resource allocation anymore. What's the benefit of doing an strdup() to keep a copy of a string when you can simply do a = b without having to care about the memory used behind. Machines are huge nowadays, much larger than the old Athlon XP I was using 10 years ago. Second, Lua doesn't require a compiler, so we'll save 30 minutes a day per 200 builds, this will definitely speed up development for each developer. And we won't depend on a given C compiler, won't be subject to its bugs, and more importantly we'll be able to get rid of the few lines of assembly that we currently have in some performance-critical parts. Third, last version of HAProxy saw a lot of new sample fetch functions and converters. This will not be needed anymore, because the code and the configuration will be mixed together, just as everyone does with Shell scripts. This means that any config will just look like an include directive for the haproxy code, followed by some code to declare the configuration. It will then be possible to create infinite combinations of new functions, and the configuration will have access to anything internal to HAProxy. In the end, of the current HAProxy will only remain the Lua engine, and probably by then we'll find even better ones so that haproxy will be distributed as a Lua library to use anywhere, maybe even on IoT devices if that makes sense (anyone ever dreamed of having haproxy in their watches ?). This step forward will save us from having to continue to do any code versionning, because everyone will have his own fork and the code will grow much faster this way. That also means that Git will become useless for us. In terms of security, it will be much better as it will not be possible to exploit a vulnerability common to all versions anymore since each version will be different. HAProxy Technologies is going to assign a lot of resources to this task. Obviously all the development team will work on this full time, but we also realize that since customers will not be interested in the C version anymore after this public announce, we'll train the sales people to write Lua as well in order to speed up development. We'll continue to provide an enterprise version forked from HAPEE that we'll rename Luapee. It will still provide all the extras that make it a professional solution such as VRRP, SNMP etc and over the long term we expect to rewrite all of these components in Lua as well. The ALOHA appliances will change a little bit, they'll mostly be a Lua engine to run all that code, so we'll probably rename them HALUA. And given that the appliance's goal has always been to take profit of the hardware and kernel to further improve the capabilities
Re: Complete rewrite of HAProxy in Lua
I bought the whole thing ... until I realized the date ... (and I'm older than you Willy ;-) ) You should've gone turbopascal on me :-D Franky
Re: Complete rewrite of HAProxy in Lua
On Wed, Apr 01, 2015 at 10:43:47AM +0200, Willy Tarreau wrote: In the end, of the current HAProxy will only remain the Lua engine, and probably by then we'll find even better ones so that haproxy will be distributed as a Lua library to use anywhere, maybe even on IoT devices if that makes sense (anyone ever dreamed of having haproxy in their watches ?). That's good news, we'll finally get ride of these filthy pointers! -- William Lallemand
Re: Complete rewrite of HAProxy in Lua
On Wed, Apr 01, 2015 at 11:00:10AM +0200, Franky Van Liedekerke wrote: I bought the whole thing ... until I realized the date ... (and I'm older than you Willy ;-) ) You should've gone turbopascal on me :-D I loved it as well, it was easy to write clean and efficient code. Cheers, Willy
Re: Complete rewrite of HAProxy in Lua
Great one :) I was 10 seconds away from forwarding this to our web teams :) On 01/04/2015 09:43, Willy Tarreau wrote: Hi, As some might have noticed, HAProxy development is progressively slowing down over time. I have analyzed the situation and came to the following conclusions : - the code base is increasing and is becoming slower to build day Due to the huge amount of work, we'll postpone the 1.6 release to 1st April 2016, which leaves us exactly 366 days to complete this task. I hope everyone understands that we have no other choice. Now I have to go back to work, there's no time to waste. Say us good luck! Willy
Re: Complete rewrite of HAProxy in Lua
I'd prefer a complete rewrite of HAProxy in JavaScript. We would be able to make it run on all platform (hello windows), and with a simple browser tab left open :( 2015-04-01 11:40 GMT+02:00 Chris Allen ch...@cjx.com: Great one :) I was 10 seconds away from forwarding this to our web teams :) On 01/04/2015 09:43, Willy Tarreau wrote: Hi, As some might have noticed, HAProxy development is progressively slowing down over time. I have analyzed the situation and came to the following conclusions : - the code base is increasing and is becoming slower to build day Due to the huge amount of work, we'll postpone the 1.6 release to 1st April 2016, which leaves us exactly 366 days to complete this task. I hope everyone understands that we have no other choice. Now I have to go back to work, there's no time to waste. Say us good luck! Willy
Re: Complete rewrite of HAProxy in Lua
Good luck! (good to hear about this progress) 2015-04-01 10:43 GMT+02:00 Willy Tarreau w...@1wt.eu: Hi, As some might have noticed, HAProxy development is progressively slowing down over time. I have analyzed the situation and came to the following conclusions : - the code base is increasing and is becoming slower to build day after day. Ten years ago, version 1.1.31 was only 6716 lines everything included. Today, mainline is 108395 lines, or 16 times larger. - gcc is getting slower over time. Since version 2.7.2 I used to rely on ten years ago, we've seen important slowdowns with v2.95, several v3.x then v4.x. I'm currently on 4.7 and afraid to upgrade. - while the whole code base used to build in less than a second ten years ago on an Athlon XP-1800, now it takes about 10 seconds on a core i5 at 3 GHz. Multiply this by about 200 builds a day and you see that half an hour is wasted every single day dedicated to development. That's about 1/4 of the available time if you count the small amount of time available after processing e-mails. - people don't learn C anymore at school and this makes it harder to get new contributors. In fact, most of those who are proficient in C already have a job and little spare time to dedicate to an opensource project. In parallel, I'm seeing I'm getting old, I turned 40 last year and it's obvious that I'm not as much capable of optimizing code as I used to be. I'm of the old school, still counting the CPU cycles it takes a function to execute, the nanoseconds required to append an X-Forwarded-For header or to parse a cookie. And all of this is totally wasted when people run the software in virtual machines which only allocate portions of CPUs (ie they switch between multiple VMs at high rate), or install it in front of applications which saturate at 100 requests a second. Recently with the Lua addition, we found it to be quite fast. Maybe not as fast as C, but Lua is improving and C skills are diminishing, so I guess that in a few years the code written in Lua will be much faster than the code we'll be able to write in C. Thus I found it wise to declare a complete rewrite of HAProxy in Lua. It comes with many benefits. First, Lua is easy to learn, we'll get many more developers and contributors. One of the reason is that you don't need to care about resource allocation anymore. What's the benefit of doing an strdup() to keep a copy of a string when you can simply do a = b without having to care about the memory used behind. Machines are huge nowadays, much larger than the old Athlon XP I was using 10 years ago. Second, Lua doesn't require a compiler, so we'll save 30 minutes a day per 200 builds, this will definitely speed up development for each developer. And we won't depend on a given C compiler, won't be subject to its bugs, and more importantly we'll be able to get rid of the few lines of assembly that we currently have in some performance-critical parts. Third, last version of HAProxy saw a lot of new sample fetch functions and converters. This will not be needed anymore, because the code and the configuration will be mixed together, just as everyone does with Shell scripts. This means that any config will just look like an include directive for the haproxy code, followed by some code to declare the configuration. It will then be possible to create infinite combinations of new functions, and the configuration will have access to anything internal to HAProxy. In the end, of the current HAProxy will only remain the Lua engine, and probably by then we'll find even better ones so that haproxy will be distributed as a Lua library to use anywhere, maybe even on IoT devices if that makes sense (anyone ever dreamed of having haproxy in their watches ?). This step forward will save us from having to continue to do any code versionning, because everyone will have his own fork and the code will grow much faster this way. That also means that Git will become useless for us. In terms of security, it will be much better as it will not be possible to exploit a vulnerability common to all versions anymore since each version will be different. HAProxy Technologies is going to assign a lot of resources to this task. Obviously all the development team will work on this full time, but we also realize that since customers will not be interested in the C version anymore after this public announce, we'll train the sales people to write Lua as well in order to speed up development. We'll continue to provide an enterprise version forked from HAPEE that we'll rename Luapee. It will still provide all the extras that make it a professional solution such as VRRP, SNMP etc and over the long term we expect to rewrite all of these components in Lua as well. The ALOHA appliances will change a little bit, they'll mostly be a Lua engine to run all
Re: Complete rewrite of HAProxy in Lua
Hi Willy, On 1 April 2015 at 09:43, Willy Tarreau w...@1wt.eu wrote: Recently with the Lua addition, we found it to be quite fast. Maybe not as fast as C, but Lua is improving and C skills are diminishing, so I guess that in a few years the code written in Lua will be much faster than the code we'll be able to write in C. Thus I found it wise to declare a complete rewrite of HAProxy in Lua. It comes with many benefits. As the author of an open-source network server project written in Lua, I'm delighted at this news. Currently in today's deployments, people who use haproxy have to put it on a separate machine in front of the application. With this move though, we will be able to merge haproxy directly into our code, so everyone can benefit from haproxy's performance without the need for extra hardware. Really looking forward to this, good luck! Regards, Matthew
Re: Complete rewrite of HAProxy in Lua
On 1 Apr 2015, at 09:43, Willy Tarreau w...@1wt.eu wrote: we'll train the sales people to write Lua as well in order to speed up development. You got me going good until I read this. :-) Nice one Willy. Pedro.
Re: Complete rewrite of HAProxy in Lua
Ok. My immediate thought was.. Oh crap we are going to have to fork haproxy and hire loads of C developers! Then when someone mentioned what day it was I felt incredible relief :-). Nice joke. Well executed. On 1 Apr 2015 10:06, Baptiste bed...@gmail.com wrote: I'll have to find a way to code buffer overflows in LUA! Baptiste
Re: Complete rewrite of HAProxy in Lua
I'll have to find a way to code buffer overflows in LUA! Baptiste
