Re: About the SPOE
On Wed, Jul 24, 2024 at 03:48:15PM +0200, Christopher Faulet wrote: > As announced, the SPOE was finally refactored. This new SPOE will be shipped > with the 3.1-dev4. It is a full rewrite of the engine, based on a dedicated > SPOP multiplexer. It means a "spop" proxy mode, used for SPOE backends, was > added. It is now the default mode for backends used by SPOE engines. "mode > spop" can be explicitly specified, but otherwise, the mode is detected > during post-parsing. So, these backends cannot be mixed for a raw TCP usage. > > Thanks to this refactoring, SPOP connections are now properly reused, > similarly to HTTP connections. So, it is now possible to properly balance > connections when several servers are configured. The management of IDLE > connections used on HTTP backends is used for SPOP connections. It is a huge > improvement compared to what was performed by hand with the pool of SPOE > applets. In addition to these, what's important to understand is that previously, connections were load-balanced between agents while now it's messages. Thus the control is much finer. For example "balance leastconn" can make some sense for always sending a request to the least loaded agent; "balance hash var(ptxn.XXX)" can make sense as well to perform some affinity and improve context reuse on the backend, and so on. It's never easy to enumerate all possibilities offered by switching from per-connection processing to per-request processing, but it's basically the same as we gained long ago by switching from forwarding SSL as raw TCP connections and later decoding SSL to process each individual request! That's why I second Christopher regarding this quest for testers. It's possible some will feel that something tiny is missing to achieve a great step forward in their use cases, it'd better be expressed early ;-) Willy
Re: About the SPOE
Le 28/03/2024 à 18:14, Christopher Faulet a écrit : Thanks Lokesh, Abhijeet and Aleksandar for your feedback. This truly help us. Thanks too to Pierre and Mattia for their feedback on the request mirroring. Rest assured that we take this into account in our reflections. After some internal discussions and also regarding to feedback we had internally, we've decided to invest some time to rewrite the engine for the 3.1. A feature request was created to keep the conversation going (https://github.com/haproxy/haproxy/issues/2502). But to sum up, the idea is to rethink the engine to make it simpler. Of course, the engine will be based on recent core features. But features will also be reduced to remain maintainable. Among other things, the asynchronous mode will be removed because it is far too complex and most probably unused. It is clearly an over-designed feature. Then we will not invest time on the payload streaming support. This feature is pretty complex to implement and this is mainly why we never made the current SPOE evolved. Let's be reasonable this time. This will probably influence the design. For people who have invested time in SPOAs development, the protocol will be kept compatible. The purpose here is to rewrite the engine itself. So it is above all internal to HAProxy. For the 3.0, the warning about the SPOE will be changed to notify users some configuration changes should be expected in future versions. Some other small changes should be expected for the 3.0. But the heavy lifting will be performed on the 3.1. Of course, it is still a subject under discussion. The above issue is here to collect ideas for the next steps but also for more long term features. Feel free to feed it. Hi, As announced, the SPOE was finally refactored. This new SPOE will be shipped with the 3.1-dev4. It is a full rewrite of the engine, based on a dedicated SPOP multiplexer. It means a "spop" proxy mode, used for SPOE backends, was added. It is now the default mode for backends used by SPOE engines. "mode spop" can be explicitly specified, but otherwise, the mode is detected during post-parsing. So, these backends cannot be mixed for a raw TCP usage. Thanks to this refactoring, SPOP connections are now properly reused, similarly to HTTP connections. So, it is now possible to properly balance connections when several servers are configured. The management of IDLE connections used on HTTP backends is used for SPOP connections. It is a huge improvement compared to what was performed by hand with the pool of SPOE applets. In parallel, the notion of child and parent streams was added. In this context, the SPOE stream, used to exchange messages with SPOA is the child while the main stream is the parent. The child stream is created by the parent stream to perform some processing for its parent. In addition, it is possible from a child stream to retrieve variables of the parent stream. This feature can be used to perform some processing in the child stream based on properties of the parent stream. The first and probably the main usage is to apply stickiness rules in a child stream based on info from the parent stream. dedicated scopes was added to retrieve parent variables (psess, ptxn, preq and pres). Here is a very simple example: frontend front-http bind *:8889 tcp-request content set-var(txn.client_src) src filter spoe engine ip-reputation config /spoe-ip-reputation.conf http-request send-spoe-group ip-reputation check-client-ip http-response set-header X-Ip-Score %[var(sess.iprep.ip_score)] default_backend back-http backend back-http server www 127.0.0.1:8000 backend iprep-backend mode spop timeout server 10s stick-table type ip size 200k expire 30m stick on var(ptxn.client_src) server iprep-srv1 127.0.0.1:12345 server iprep-srv2 127.0.0.1:12345 server iprep-srv3 127.0.0.1:12345 server iprep-srv4 127.0.0.1:12345 With this configuration, SPOP sessions are sticky on servers based on the client source address. The configuration manual was updated accordingly. Check it for details. About other changes, the fragmentation support and the asynchronous mode were removed. Without the streaming feature, the fragmentation is just useless. And the asynchronous mode is overkill and just incompatible with the new architecture. Following SPOE parameters were also removed and are silently ignored when found in SPOE configuration: maxconnrate, maxerrrate, max-waiting-frames, timeout hello and timeout idle. SPOE log format was adapted because several info no longer exist. The first and main step is finished. But there are some missing parts. For instance, connection closures are a bit dirty for now because the internal API must evolve a bit first. This will be achieved before the 3.1 release. Stats regarding the SPOE must be enriched. Some parameters must be added (in configuration and
Re: About the SPOE
Thanks Lokesh, Abhijeet and Aleksandar for your feedback. This truly help us. Thanks too to Pierre and Mattia for their feedback on the request mirroring. Rest assured that we take this into account in our reflections. After some internal discussions and also regarding to feedback we had internally, we've decided to invest some time to rewrite the engine for the 3.1. A feature request was created to keep the conversation going (https://github.com/haproxy/haproxy/issues/2502). But to sum up, the idea is to rethink the engine to make it simpler. Of course, the engine will be based on recent core features. But features will also be reduced to remain maintainable. Among other things, the asynchronous mode will be removed because it is far too complex and most probably unused. It is clearly an over-designed feature. Then we will not invest time on the payload streaming support. This feature is pretty complex to implement and this is mainly why we never made the current SPOE evolved. Let's be reasonable this time. This will probably influence the design. For people who have invested time in SPOAs development, the protocol will be kept compatible. The purpose here is to rewrite the engine itself. So it is above all internal to HAProxy. For the 3.0, the warning about the SPOE will be changed to notify users some configuration changes should be expected in future versions. Some other small changes should be expected for the 3.0. But the heavy lifting will be performed on the 3.1. Of course, it is still a subject under discussion. The above issue is here to collect ideas for the next steps but also for more long term features. Feel free to feed it. Regards, -- Christopher Faulet
About the SPOE
Hi list, We have a few usages of SPOE, mainly: * HTTP traffic mirroring. It's being discussed on GH, more details here: https://github.com/haproxy/haproxy/issues/2471#issuecomment-2007261494 * authentication (because we used to have 2 auth methods and it helped developing one backend to handle them both). I guess if one provide pure SSO/OAuth2/OIDC it can be handled natively nowadays with all the converters that have been added (also seen that multiple official blogposts have been published on the topic). We also envisaged SPOE to offload some heavy filtering workload, especially to separate the ownership of components (one spoa could have been owned and deployed by ). As long as there's a sort of equivalent of SPOE in the long-run, we're not shocked that this comes to be deprecated. -- Pierre
Re: About the SPOE
Hi Lokesh, On Tue, Mar 26, 2024 at 12:10:53AM +, Lokesh Jindal wrote: > Hey Willy > > Resending this email in case you missed the last one. Let me know if you had > any follow up questions/comments. > I saw https://github.com/haproxy/haproxy/issues/2502 created by Christopher - > looking forward to discussion on that issue. Sorry for the delay, yes I had a quick look but last week was full of meetings here and I'm slowly trying to catch up with the lag, I'm even late on -dev6 :-( Back on this soon. There could be good news from our side given that some of the meetings were precisely about all of this ;-) Stay tuned, Willy
Re: About the SPOE
Hey Willy Resending this email in case you missed the last one. Let me know if you had any follow up questions/comments. I saw https://github.com/haproxy/haproxy/issues/2502 created by Christopher - looking forward to discussion on that issue. - Lokesh From: Lokesh Jindal Date: Monday, March 18, 2024 at 4:55 PM To: Willy Tarreau Cc: Abhijeet Rastogi , Christopher Faulet , haproxy@formilux.org , Aleksandar Lazic Subject: Re: About the SPOE Hey Willy Please see my response inline below. - Lokesh From: Willy Tarreau Date: Monday, March 18, 2024 at 4:08 AM To: Lokesh Jindal Cc: Abhijeet Rastogi , Christopher Faulet , haproxy@formilux.org , Aleksandar Lazic Subject: Re: About the SPOE Hi Lokesh, Abhijeet, Alex, First, thanks for jumping into this thread, the purpose of the deprecation is in a big part to try to collect the requirements of possibly existing users. Mind you that the rare times we hear about SPOE is only because of problems, so it's difficult to figure what to keep and what to cut from the existing design. More on that below: On Fri, Mar 15, 2024 at 05:15:14PM +, Lokesh Jindal wrote: > Hey Christopher > > Adding to what my colleague, Abhijeet, said. > > > 1. We plan to ramp traffic to HAProxy very soon where we will heavily rely > on SPOA. In our testing, we are satisfied with SPOE in terms of > performance. The flexibility to write SPOA in any language not only allows > us to handle "complex use cases" like connecting to non-http downstreams, > but also helps in observability - metrics and logging. Interesting, the initial internal e-mail in 2016 that ignited the SPOE design was driven from the same observations: in web environments it's quite rare to find developers who are at ease with system-level languages, yet they are the ones most likely to request to extend the proxy. For this reason we wanted to offer the possibility to call code written in other languages. In addition it was estimated that the ability to connect to the agent over the network and using secure connections was absolutely essential. It brings the ability to scale the processing engine without adding more LB nodes, and even to offload that to another DC, infrastructure or even to delegate it to a 3rd party. [Lokesh]: One of our major flows requires us to connect to couchbase for every request. Today, we are running one HAProxy process + one golang SPOA process on every host of the reverse proxy layer. This keeps the setup simple and avoids making a network call for HAProxy-SPOA communication, reducing latency. We have a couple of other flows where the SPOA connects to HTTP services for things like SSO, some type of cookie renewal, etc. These are not as latency sensitive as the flow mentioned above. Further, we anticipate the interaction of SPOA with these external services will change often – how we parse the http response from the services, how we make the request to these services, addition of new calls to services, etc. In these cases, the SPOA will give us ability to develop features fast. Among the use cases which immediately came to mind were authentication, database accesses, name resolution, "remote maps", request classification, IP reputation, etc. In addition we thought that limiting ourselves to short request/responses like this was probably limiting and that it would have been useful to support streaming so that we could implement image recompression, caching, WAFs etc. The first PoC implementation that was merged in version 1.7 lacked the streaming abilities, and it's still the current implementation. It took a while before we received feedback on it, since then caching was implemented, the demand for image recompression is close to non-existing, and WAFs users have well accommodated to dealing with extra layers by now it seems. So basically we're left with something ultra-complex that deals with short request-responses most of the time, and that suffers from the initial design which was way bigger than the use cases. [Lokesh]: I think this applies to us, as explained in my comment above. > 2. What is the best alternative to SPOE? I don't know yet, and the purpose of this deprecation precisely is to engage discussion with users. One could think about various HTTP-based protocols for which it is easier to implement a server, some gRPC maybe, possibly even a stripped-down version of the SPOP protocol if we figure that everyone is running on a reasonable subset that's much easier to deal with than the whole stuff we have now. > Two options that we are aware of > - Write fetchers/converters in lua or write filters in other languages > using the Lua API. In your experience, how do they compare to SPOE in terms > of: > * Performance > * Fault isolation The benefits of SPOE as you've found, clearly are in terms of flexibility as i
Re: About the SPOE
Hey Willy Please see my response inline below. - Lokesh From: Willy Tarreau Date: Monday, March 18, 2024 at 4:08 AM To: Lokesh Jindal Cc: Abhijeet Rastogi , Christopher Faulet , haproxy@formilux.org , Aleksandar Lazic Subject: Re: About the SPOE Hi Lokesh, Abhijeet, Alex, First, thanks for jumping into this thread, the purpose of the deprecation is in a big part to try to collect the requirements of possibly existing users. Mind you that the rare times we hear about SPOE is only because of problems, so it's difficult to figure what to keep and what to cut from the existing design. More on that below: On Fri, Mar 15, 2024 at 05:15:14PM +, Lokesh Jindal wrote: > Hey Christopher > > Adding to what my colleague, Abhijeet, said. > > > 1. We plan to ramp traffic to HAProxy very soon where we will heavily rely > on SPOA. In our testing, we are satisfied with SPOE in terms of > performance. The flexibility to write SPOA in any language not only allows > us to handle "complex use cases" like connecting to non-http downstreams, > but also helps in observability - metrics and logging. Interesting, the initial internal e-mail in 2016 that ignited the SPOE design was driven from the same observations: in web environments it's quite rare to find developers who are at ease with system-level languages, yet they are the ones most likely to request to extend the proxy. For this reason we wanted to offer the possibility to call code written in other languages. In addition it was estimated that the ability to connect to the agent over the network and using secure connections was absolutely essential. It brings the ability to scale the processing engine without adding more LB nodes, and even to offload that to another DC, infrastructure or even to delegate it to a 3rd party. [Lokesh]: One of our major flows requires us to connect to couchbase for every request. Today, we are running one HAProxy process + one golang SPOA process on every host of the reverse proxy layer. This keeps the setup simple and avoids making a network call for HAProxy-SPOA communication, reducing latency. We have a couple of other flows where the SPOA connects to HTTP services for things like SSO, some type of cookie renewal, etc. These are not as latency sensitive as the flow mentioned above. Further, we anticipate the interaction of SPOA with these external services will change often – how we parse the http response from the services, how we make the request to these services, addition of new calls to services, etc. In these cases, the SPOA will give us ability to develop features fast. Among the use cases which immediately came to mind were authentication, database accesses, name resolution, "remote maps", request classification, IP reputation, etc. In addition we thought that limiting ourselves to short request/responses like this was probably limiting and that it would have been useful to support streaming so that we could implement image recompression, caching, WAFs etc. The first PoC implementation that was merged in version 1.7 lacked the streaming abilities, and it's still the current implementation. It took a while before we received feedback on it, since then caching was implemented, the demand for image recompression is close to non-existing, and WAFs users have well accommodated to dealing with extra layers by now it seems. So basically we're left with something ultra-complex that deals with short request-responses most of the time, and that suffers from the initial design which was way bigger than the use cases. [Lokesh]: I think this applies to us, as explained in my comment above. > 2. What is the best alternative to SPOE? I don't know yet, and the purpose of this deprecation precisely is to engage discussion with users. One could think about various HTTP-based protocols for which it is easier to implement a server, some gRPC maybe, possibly even a stripped-down version of the SPOP protocol if we figure that everyone is running on a reasonable subset that's much easier to deal with than the whole stuff we have now. > Two options that we are aware of > - Write fetchers/converters in lua or write filters in other languages > using the Lua API. In your experience, how do they compare to SPOE in terms > of: > * Performance > * Fault isolation The benefits of SPOE as you've found, clearly are in terms of flexibility as it allows to scale the number of analysers independently on the number of LBs, and it almost makes problems almost unnoticeable. For example, if your code relied on unstable 3rd party libraries, crashing your SPOA doesn't bring down the whole proxy. Similarly in terms of added latency, all the latency is in the external component, the rest of the traffic is not affected as it would be by placing some heavy processing directly inside the haproxy proces
Re: About the SPOE
Hi Lokesh, Abhijeet, Alex, First, thanks for jumping into this thread, the purpose of the deprecation is in a big part to try to collect the requirements of possibly existing users. Mind you that the rare times we hear about SPOE is only because of problems, so it's difficult to figure what to keep and what to cut from the existing design. More on that below: On Fri, Mar 15, 2024 at 05:15:14PM +, Lokesh Jindal wrote: > Hey Christopher > > Adding to what my colleague, Abhijeet, said. > > > 1. We plan to ramp traffic to HAProxy very soon where we will heavily rely > on SPOA. In our testing, we are satisfied with SPOE in terms of > performance. The flexibility to write SPOA in any language not only allows > us to handle "complex use cases" like connecting to non-http downstreams, > but also helps in observability - metrics and logging. Interesting, the initial internal e-mail in 2016 that ignited the SPOE design was driven from the same observations: in web environments it's quite rare to find developers who are at ease with system-level languages, yet they are the ones most likely to request to extend the proxy. For this reason we wanted to offer the possibility to call code written in other languages. In addition it was estimated that the ability to connect to the agent over the network and using secure connections was absolutely essential. It brings the ability to scale the processing engine without adding more LB nodes, and even to offload that to another DC, infrastructure or even to delegate it to a 3rd party. Among the use cases which immediately came to mind were authentication, database accesses, name resolution, "remote maps", request classification, IP reputation, etc. In addition we thought that limiting ourselves to short request/responses like this was probably limiting and that it would have been useful to support streaming so that we could implement image recompression, caching, WAFs etc. The first PoC implementation that was merged in version 1.7 lacked the streaming abilities, and it's still the current implementation. It took a while before we received feedback on it, since then caching was implemented, the demand for image recompression is close to non-existing, and WAFs users have well accommodated to dealing with extra layers by now it seems. So basically we're left with something ultra-complex that deals with short request-responses most of the time, and that suffers from the initial design which was way bigger than the use cases. > 2. What is the best alternative to SPOE? I don't know yet, and the purpose of this deprecation precisely is to engage discussion with users. One could think about various HTTP-based protocols for which it is easier to implement a server, some gRPC maybe, possibly even a stripped-down version of the SPOP protocol if we figure that everyone is running on a reasonable subset that's much easier to deal with than the whole stuff we have now. > Two options that we are aware of > - Write fetchers/converters in lua or write filters in other languages > using the Lua API. In your experience, how do they compare to SPOE in terms > of: > * Performance > * Fault isolation The benefits of SPOE as you've found, clearly are in terms of flexibility as it allows to scale the number of analysers independently on the number of LBs, and it almost makes problems almost unnoticeable. For example, if your code relied on unstable 3rd party libraries, crashing your SPOA doesn't bring down the whole proxy. Similarly in terms of added latency, all the latency is in the external component, the rest of the traffic is not affected as it would be by placing some heavy processing directly inside the haproxy process. > 3. As Abhijeet said, can you share a list of issues with SPOE that make it > hard to maintain? On the top of my head I can enumerate a few: - the load balancing is integraly reimplemented at the SPOE level between applets - the load balancing is affected by the support of response-less messages, which prevent haproxy from having an estimate of the server's load, which means that an algo such as leastconn would not make sense in such a condition - the idle connections management is integraly reimplemented at the SPOE level using applets as well. An idle SPOE connection is in fact a full application-layer stream with one applet on one side and a connection on the other side. These cannot be migrated between threads, which require even more complex stream-to-stream thread-safe communication mechanisms and synchronisation. - it's possible to receive a response on a different connection than the one that saw the request. This adds complexity in the matching request lookup, needs for extra synchronisation so that the other stream doesn't vanish while we're delivering the reponse, and it also makes it harder to keep track of the number of in-flight requests.
Re: About the SPOE
Hi. On 2024-03-15 (Fr.) 15:09, Christopher Faulet wrote: Hi all, It was evoked on the ML by Willy and mentioned in few issues on GH. It is now official. The SPOE was marked as deprecated for the 3.0. It is not a pleasant announce because it is always an admission of failure to remove a feature. Sadly, this filter should be refactored to work properly. It was implemented as a functional PoC for the 1.7 and since then, no time was invest to improve it and make it truly maintainable in time. Worst, other parts of HAProxy evolved, especially applets part, making maintenance ever more expensive. We must be realistic on the subject, there was no real adoption on the SPOE and this partly explains why no time was invest on it. So we are really sorry for users relying on it. But we cannot continue in this direction. The 3.0 is be an LTS version. It means the SPOE will still be maintained on this version and lower ones for 5 years. On the 3.1, it will be marked as unmaintained and possibly removed if an alternative solution is implemented. It remains few months before the 3.0 release to change our mind. Maybe this announce will be an electroshock to give it a new lease of life. Otherwise it is time to find an alternative solution based on an existing protocol. For all 3.0 users, there is now a warning if a SPOE filter is configured. But there is also a global option to silent it. To do so, "expose-deprecated-directives" must be added in the global section. Now we are open to discussion on this subject. Let us know your feeling and if you have any suggestion, we will be happy to talk about it. As I fully understand this step it would he very helpful to have a filter which have the possibility to run some tasks outside of HAProxy in async way. There was a short discussion, in the past, about the future of filters https://www.mail-archive.com/haproxy@formilux.org/msg44164.html maybe there are some Ideas which can be reused. From my point of view would be a http-filter (1,2,!3 imho), with all the pros and cons, one of the best way for a filter, because this protocol is so widely used and a lot of knowledge could be reused. One of the biggest benefit is also that, even in Enterprise environments, could this filter be used as this protocol is able to run across a proxy. FCGI is also another option as it's already part of the Filter chain :-). I don't know too much about grpc but maybe this protocol could also be used as filter ¯\_(ツ)_/¯. Lua-API with some external Daemons could be also used to move the workload out of HAProxy. From my point of view, what ever solution is chosen the Idea behind the SPOE should be kept because it's a good concept to scale the filters outside of HAProxy. I see a lot of possibilities here the main point is always how much work it it to maintain the filter chain. Regards, Jm2c Regards Alex
Re: About the SPOE
Hey Christopher Adding to what my colleague, Abhijeet, said. 1. We plan to ramp traffic to HAProxy very soon where we will heavily rely on SPOA. In our testing, we are satisfied with SPOE in terms of performance. The flexibility to write SPOA in any language not only allows us to handle “complex use cases” like connecting to non-http downstreams, but also helps in observability – metrics and logging. 2. What is the best alternative to SPOE? Two options that we are aware of – Write fetchers/converters in lua or write filters in other languages using the Lua API. In your experience, how do they compare to SPOE in terms of: * Performance * Fault isolation 3. As Abhijeet said, can you share a list of issues with SPOE that make it hard to maintain? Be it SPOE or an alternate solution that allows us to handle complex use cases with good performance, fault isolation (as much as possible) and observability, we will be happy to help develop/maintain it. - Lokesh From: Abhijeet Rastogi Date: Friday, March 15, 2024 at 8:23 AM To: Christopher Faulet Cc: haproxy@formilux.org Subject: Re: About the SPOE Hi Christopher, Thank you for starting the discussion here. >Worst, other parts of HAProxy evolved, especially applets part, making maintenance ever more expensive. Can we elaborate on what makes maintenance expensive? >We must be realistic on the subject, there was no real adoption on the SPOE and this partly explains why no time was invest on it. My company primarily uses SPOE in cases where we've to connect with non-http downstreams, for example, connecting to the data layer (Couchbase etc) that makes the proxy stateless. Other uses-cases are SSO implementation etc. We are a new user to HAproxy, with multi-million QPS on HAproxy already, and we've plans to ramp multiples of that in the coming year or two. >Now we are open to discussion on this subject. Let us know your feeling and if you have any suggestion, we will be happy to talk about it. Our goal at my company has been to not modify HAproxy core and still add features to it without running in the same memory space as HAproxy for safety. Of course, we use config DSL and lua as much as possible, but it falls short in terms of writing complex business logic. (non-http downstreams etc). If it's not SPOE, what is a true replacement? With my limited knowledge, not a true replacement though, maybe lua filter API and then write these complex in-flight operations in Rust? With the learnings from today, what would be the ideal way to replace it, considering we had engineers to maintain it? Slightly off-topic, SPOE has its flaws too though. One of our use-cases is to stream (kafka) HTTP traffic information (cookies, request data etc) in an efficient fashion and we actually did not use SPOE as the cost to exchange information was too high. In this particular case, we ended up streaming data via txn/req variables to logs instead (ring buffer). Thanks, Abhijeet On Fri, Mar 15, 2024 at 7:10 AM Christopher Faulet wrote: > > Hi all, > > It was evoked on the ML by Willy and mentioned in few issues on GH. It is now > official. The SPOE was marked as deprecated for the 3.0. It is not a pleasant > announce because it is always an admission of failure to remove a feature. > Sadly, this filter should be refactored to work properly. It was implemented > as > a functional PoC for the 1.7 and since then, no time was invest to improve it > and make it truly maintainable in time. Worst, other parts of HAProxy evolved, > especially applets part, making maintenance ever more expensive. > > We must be realistic on the subject, there was no real adoption on the SPOE > and > this partly explains why no time was invest on it. So we are really sorry for > users relying on it. But we cannot continue in this direction. > > The 3.0 is be an LTS version. It means the SPOE will still be maintained on > this > version and lower ones for 5 years. On the 3.1, it will be marked as > unmaintained and possibly removed if an alternative solution is implemented. > > It remains few months before the 3.0 release to change our mind. Maybe this > announce will be an electroshock to give it a new lease of life. Otherwise it > is > time to find an alternative solution based on an existing protocol. > > For all 3.0 users, there is now a warning if a SPOE filter is configured. But > there is also a global option to silent it. To do so, > "expose-deprecated-directives" must be added in the global section. > > Now we are open to discussion on this subject. Let us know your feeling and if > you have any suggestion, we will be happy to talk about it. > > Regards, > -- > Christopher Faulet > -- Cheers, Abhijeet (https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fabhi.host%2
Re: About the SPOE
Hi Christopher, Thank you for starting the discussion here. >Worst, other parts of HAProxy evolved, especially applets part, making maintenance ever more expensive. Can we elaborate on what makes maintenance expensive? >We must be realistic on the subject, there was no real adoption on the SPOE and this partly explains why no time was invest on it. My company primarily uses SPOE in cases where we've to connect with non-http downstreams, for example, connecting to the data layer (Couchbase etc) that makes the proxy stateless. Other uses-cases are SSO implementation etc. We are a new user to HAproxy, with multi-million QPS on HAproxy already, and we've plans to ramp multiples of that in the coming year or two. >Now we are open to discussion on this subject. Let us know your feeling and if you have any suggestion, we will be happy to talk about it. Our goal at my company has been to not modify HAproxy core and still add features to it without running in the same memory space as HAproxy for safety. Of course, we use config DSL and lua as much as possible, but it falls short in terms of writing complex business logic. (non-http downstreams etc). If it's not SPOE, what is a true replacement? With my limited knowledge, not a true replacement though, maybe lua filter API and then write these complex in-flight operations in Rust? With the learnings from today, what would be the ideal way to replace it, considering we had engineers to maintain it? Slightly off-topic, SPOE has its flaws too though. One of our use-cases is to stream (kafka) HTTP traffic information (cookies, request data etc) in an efficient fashion and we actually did not use SPOE as the cost to exchange information was too high. In this particular case, we ended up streaming data via txn/req variables to logs instead (ring buffer). Thanks, Abhijeet On Fri, Mar 15, 2024 at 7:10 AM Christopher Faulet wrote: > > Hi all, > > It was evoked on the ML by Willy and mentioned in few issues on GH. It is now > official. The SPOE was marked as deprecated for the 3.0. It is not a pleasant > announce because it is always an admission of failure to remove a feature. > Sadly, this filter should be refactored to work properly. It was implemented > as > a functional PoC for the 1.7 and since then, no time was invest to improve it > and make it truly maintainable in time. Worst, other parts of HAProxy evolved, > especially applets part, making maintenance ever more expensive. > > We must be realistic on the subject, there was no real adoption on the SPOE > and > this partly explains why no time was invest on it. So we are really sorry for > users relying on it. But we cannot continue in this direction. > > The 3.0 is be an LTS version. It means the SPOE will still be maintained on > this > version and lower ones for 5 years. On the 3.1, it will be marked as > unmaintained and possibly removed if an alternative solution is implemented. > > It remains few months before the 3.0 release to change our mind. Maybe this > announce will be an electroshock to give it a new lease of life. Otherwise it > is > time to find an alternative solution based on an existing protocol. > > For all 3.0 users, there is now a warning if a SPOE filter is configured. But > there is also a global option to silent it. To do so, > "expose-deprecated-directives" must be added in the global section. > > Now we are open to discussion on this subject. Let us know your feeling and if > you have any suggestion, we will be happy to talk about it. > > Regards, > -- > Christopher Faulet > -- Cheers, Abhijeet (https://abhi.host)
About the SPOE
Hi all, It was evoked on the ML by Willy and mentioned in few issues on GH. It is now official. The SPOE was marked as deprecated for the 3.0. It is not a pleasant announce because it is always an admission of failure to remove a feature. Sadly, this filter should be refactored to work properly. It was implemented as a functional PoC for the 1.7 and since then, no time was invest to improve it and make it truly maintainable in time. Worst, other parts of HAProxy evolved, especially applets part, making maintenance ever more expensive. We must be realistic on the subject, there was no real adoption on the SPOE and this partly explains why no time was invest on it. So we are really sorry for users relying on it. But we cannot continue in this direction. The 3.0 is be an LTS version. It means the SPOE will still be maintained on this version and lower ones for 5 years. On the 3.1, it will be marked as unmaintained and possibly removed if an alternative solution is implemented. It remains few months before the 3.0 release to change our mind. Maybe this announce will be an electroshock to give it a new lease of life. Otherwise it is time to find an alternative solution based on an existing protocol. For all 3.0 users, there is now a warning if a SPOE filter is configured. But there is also a global option to silent it. To do so, "expose-deprecated-directives" must be added in the global section. Now we are open to discussion on this subject. Let us know your feeling and if you have any suggestion, we will be happy to talk about it. Regards, -- Christopher Faulet
