Re: [PATCH] Added support for Arrays in sample_conv_json_query in sample.c
Hello, On 18/09/2023 10:27, Jens Popp wrote: Method now returns the content of Json Arrays, if it is specified in Json Path as String. The start and end character is a square bracket. Any complex object in the array is returned as Json, so that you might get Arrays of Array or objects. Only recommended for Arrays of simple types (e.g., String or int) which will be returned as CSV String. --- src/sample.c | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/sample.c b/src/sample.c index 07c881dcf..ecc4a961d 100644 --- a/src/sample.c +++ b/src/sample.c @@ -4159,8 +4159,16 @@ static int sample_conv_json_query(const struct arg *args, struct sample *smp, vo return 1; } - case MJSON_TOK_NULL: case MJSON_TOK_ARRAY: + // We copy the complete array, including square brackets into the return buffer + // result looks like: ["manage-account","manage-account-links","view-profile"] + strncpy( trash->area, token, token_size); + trash->data = token_size; + trash->size = token_size; + smp->data.u.str = *trash; + smp->data.type = SMP_T_STR; + return 1; + case MJSON_TOK_NULL: case MJSON_TOK_OBJECT: /* We cannot handle these. */ return 0; -- 2.39.3 [X] You might want to use the b_putblk function instead of strncpy since you don't manage the case where you don't have enough space in the trash buffer. The trash->size member should never be changed as well since it represents the buffer size, not the number of bytes written in it. Please update the "json_query" documentation as well. Thanks Rémi LB
Re: mux-h2: Backend stream is not fully closed if frontend keeps stream open
Le 20/09/2023 à 22:14, Valters Jansons a écrit : On Mon, Sep 18, 2023 at 9:32 AM Christopher Faulet wrote: Please find a new patch. It should be good ( or better at least :) The provided patch resolves my scenario for both the gRPC server-client as well as the Java client to nginx server. There is an improvement for sure, and HTTP 200 OK is delivered to the client! However, I want to bring attention that HAProxy consistently reports connection state as SD-- (the TCP session was unexpectedly aborted by the server, or the server explicitly refused it, in the DATA phase). To me it sounds a bit confusing, as the server doesn't do anything "unexpected" but HAProxy believes it did. Is this something that you have ideas about, or if that's something that gets dealt with in a separate patch? Thanks for the feedback ! I pushed the fix to 2.9-DEV. About the termination state, unfortunately it is a side effect of the error reported by te H2 multiplexer. We've planned to rework how errors are handled to improve the situation, to be able to report more accurate information in the logs but also to be able to properly forward errors from a side to the other one. So the situation will evolved and should be improved. But it still requires huge work. And it is hard to said when this will be achieved. -- Christopher Faulet
Re: maxconn limit not working after reload / sighup
On Thu, 21 Sept 2023 at 01:20, Björn Jacke wrote: > > Hello, > > I just experienced that maxconn can easily not work as expected and lead > to unavailable services. Take this example backend configuration of a > 2.8.3 haproxy setup: > > backend bk_example >balance first >server server1 192.168.4.1:8000 id 1 maxconn 10 >server server2 192.168.4.2:8000 id 2 maxconn 10 >server server3 192.168.4.3:8000 id 3 maxconn 10 >... > > Each server here is only able to handle 10 requests, if it receives more > requests it will just return an error. So usually the above > configuration works fine, server1 receives up to 10 connections, after > that connections are sent to server2, if also that has the maxconn limit > reached, server3 receives requests and so on. > > So far so good. If haproxy however receives a SIGHUP because of some > reconfiguration, then all the connections to the backend servers are > kept alive but haproxy thinks that the servers have 0 connections and it > will send up to 10 new connections to backend servers, even if they > already had 10 connections, which are still active and still correctly > processed by haproxy. So each server receives up to 20 connections and > the backend servers just return errors in this case. > > This is very unexpected and it looks like unintended behavior actually. > I also never heard about this and never read a warning note for such a > side effect when a haproxy reload is being done. Maybe a > server-state-file configuration might work around this problem but it > was not obvious till now that this is a requirement if maxconn is being > used. Can someone shed some light on this? This is expected behaviour. global maxconn docs do define that this is per-process, although frontend or backend server maxconn docs do not specifically reiterate that we are always talking about a per process counter. I think it's pretty much impossible to solve, actually. These are counters that can change every millisecond. Haproxy doesn't do any IO at all during run time. The state file is useless for this and I think it would be really hard to sync this between different instances, considering how time critical this is. But even if the master process would somehow pass information back and forth every second, this could only theoretically work when the configuration remains the same. What if the old configuration has: backend http server s1 s1:80 maxconn 5 server s2 s2:80 maxconn 10 server s3 s3:80 maxconn 20 and the new configuration has: backend http server s1 s1:80 maxconn 3 server s2 s2:80 maxconn 7 server s3 s3:80 maxconn 30 Heuristics between old and new configuration to find middle ground? I don't think that's even theoretically possible, but it would also be completely unexpected. I'm all for improving the docs where we can, but I doubt we can do more than that in this case. There are features to limit the amount of time old processes are running (hard-stop-after) as well as features that ramp up the numbers in new processes slowly. Other than that you'd have to account for this overhead. Lukas
Re: Error While deviceatlas compile/make from source
So the 20230802_compact.json only works with v2 versions, even from the page here it is still the case. Your inquiry had been acknowledged by the dev team but I can't say exactly when the new v3 version will be available. But we ll keep you posted. Kindest regards. On Wed, 20 Sept 2023 at 23:50, Mahendra Patil wrote: > *After installation haproxy not able to start , show below error* > > [root@govinda opt]# /opt/haproxy-ssl/sbin/haproxy -f > /opt/haproxy-ssl/conf/haproxy.cfg > [NOTICE] (42915) : haproxy version is 2.8.3-86e043a > [NOTICE] (42915) : path to executable is /opt/haproxy-ssl/sbin/haproxy > [ALERT](42915) : config : deviceatlas : > '/opt/deviceatlas/20230802_compact.json' json file is invalid. > > On Tue, Sep 19, 2023 at 5:36 PM David CARLIER wrote: > >> We are almost there, now remains the linkage part to pass. >> In your /opt/deviceatlas-enterprise-c-3.2 folder, there should be a >> `include` and a `lib` or a `lib64` folder >> So for the last command, what needs to be done instead is one of the >> following (no need to repeat step 1) : >> >> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib >> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib -lda" >> >> or >> >> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib64 >> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib64 >> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib64 -lda" >> >> The only difference is `lib` vs `lib64` >> >> Hope it's useful. >> >> Kindest regards. >> >> >> >> On Tue, 19 Sept 2023 at 12:52, Mahendra Patil >> wrote: >> >>> >>> 1) >>> cd /opt/deviceatlas-enterprise-c-3.2 >>> cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 >>> -DCMAKE_C_FLAGS="-std=gnu99" >>> make >>> make install >>> >>> 2) >>> cd haproxy-2.8.3 >>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>> >>> LD haproxy >>> /usr/bin/ld: cannot find -lda >>> collect2: error: ld returned 1 exit status >>> make: *** [haproxy] Error 1 >>> >>> >>> >>> >>> >>> On Tue, Sep 19, 2023 at 10:19 AM David CARLIER >>> wrote: >>> Hi, Since you build the DeviceAtlas library from cmake prior, the following steps are more appropriate : 1/ Inside the deviceatlas-enterprise-c-3.2 folder cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 -DCMAKE_C_FLAGS="-std=gnu99" make make install 2/ Inside the haproxy-2.8.3 folder make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib Let me know if you have any further question. On Tue, 19 Sept 2023 at 00:02, Mahendra Patil < mahendra.pa...@naaptol.com> wrote: > Thanks for yours prompt reply about deviceatlas-enterprise-c-3.2 > error while make > after yours suggestion deviceatlas-enterprise-c-3.2 *make *works fine > > But while using with haproxy-2.8.3 version gives following error > > [root@govinda opt]# cd haproxy-2.8.3 > [root@govinda haproxy-2.8.3]# make TARGET=linux-glibc USE_PCRE=1 > USE_OPENSSL=1 USE_ZLIB=1 USE_DEVICEATLAS=1 > DEVICEATLAS_SRC=/opt/deviceatlas-enterprise-c-3.2/Src/ > CC src/ev_poll.o > CC src/ev_epoll.o > CC src/cpuset.o > CC src/ssl_sock.o > CC src/ssl_ckch.o > CC src/ssl_sample.o > CC src/ssl_crtlist.o > CC src/cfgparse-ssl.o > CC src/ssl_utils.o > CC src/jwt.o > CC src/ssl_ocsp.o > CC /opt/deviceatlas/Src//Os/daunix.o > CC /opt/deviceatlas/Src//dadwcom.o > CC /opt/deviceatlas/Src//dasch.o > CC /opt/deviceatlas/Src//json.o > CC /opt/deviceatlas/Src//dac.o > /opt/deviceatlas/Src//dac.c:1033:0: warning: ignoring #pragma optimize > [-Wunknown-pragmas] > #pragma optimize("Ofast") > ^ > /opt/deviceatlas/Src//dac.c:1041:0: warning: ignoring #pragma optimize > [-Wunknown-pragmas] > #pragma optimize("Ofast") > ^ > /opt/deviceatlas/Src//dac.c:1049:0: warning: ignoring #pragma optimize > [-Wunknown-pragmas] > #pragma optimize("Ofast") > ^ > /opt/deviceatlas/Src//dac.c:1057:0: warning: ignoring #pragma optimize > [-Wunknown-pragmas] > #pragma optimize("Ofast") > ^ > /opt/deviceatlas/Src//dac.c:1065:0: warning: ignoring #pragma optimize > [-Wunkno
Re: Error While deviceatlas compile/make from source
Sorry I forgot the link https://deviceatlas.com/deviceatlas-haproxy-module. On Thu, 21 Sept 2023 at 17:10, David CARLIER wrote: > So the 20230802_compact.json only works with v2 versions, even from the > page here it is still the case. > Your inquiry had been acknowledged by the dev team but I can't say exactly > when the new v3 version will be available. > But we ll keep you posted. > > Kindest regards. > > On Wed, 20 Sept 2023 at 23:50, Mahendra Patil > wrote: > >> *After installation haproxy not able to start , show below error* >> >> [root@govinda opt]# /opt/haproxy-ssl/sbin/haproxy -f >> /opt/haproxy-ssl/conf/haproxy.cfg >> [NOTICE] (42915) : haproxy version is 2.8.3-86e043a >> [NOTICE] (42915) : path to executable is /opt/haproxy-ssl/sbin/haproxy >> [ALERT](42915) : config : deviceatlas : >> '/opt/deviceatlas/20230802_compact.json' json file is invalid. >> >> On Tue, Sep 19, 2023 at 5:36 PM David CARLIER wrote: >> >>> We are almost there, now remains the linkage part to pass. >>> In your /opt/deviceatlas-enterprise-c-3.2 folder, there should be a >>> `include` and a `lib` or a `lib64` folder >>> So for the last command, what needs to be done instead is one of the >>> following (no need to repeat step 1) : >>> >>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib >>> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib >>> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib -lda" >>> >>> or >>> >>> make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 >>> USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include >>> DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib64 >>> DEVICEATLAS_LDFLAGS="-L/opt/deviceatlas-enterprise-c-3.2/lib64 >>> -Wl,-rpath,/opt/deviceatlas-enterprise-c-3.2/lib64 -lda" >>> >>> The only difference is `lib` vs `lib64` >>> >>> Hope it's useful. >>> >>> Kindest regards. >>> >>> >>> >>> On Tue, 19 Sept 2023 at 12:52, Mahendra Patil < >>> mahendra.pa...@naaptol.com> wrote: >>> 1) cd /opt/deviceatlas-enterprise-c-3.2 cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 -DCMAKE_C_FLAGS="-std=gnu99" make make install 2) cd haproxy-2.8.3 make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_DEVICEATLAS=1 DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib LD haproxy /usr/bin/ld: cannot find -lda collect2: error: ld returned 1 exit status make: *** [haproxy] Error 1 On Tue, Sep 19, 2023 at 10:19 AM David CARLIER wrote: > Hi, > > Since you build the DeviceAtlas library from cmake prior, the > following steps are more appropriate : > 1/ Inside the deviceatlas-enterprise-c-3.2 folder > cmake . -DCMAKE_INSTALL_PREFIX=/opt/deviceatlas-enterprise-c-3.2 > -DCMAKE_C_FLAGS="-std=gnu99" > make > make install > > 2/ Inside the haproxy-2.8.3 folder > > make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 > USE_DEVICEATLAS=1 > DEVICEATLAS_INC=/opt/deviceatlas-enterprise-c-3.2/include > DEVICEATLAS_LIB=/opt/deviceatlas-enterprise-c-3.2/lib > > Let me know if you have any further question. > > On Tue, 19 Sept 2023 at 00:02, Mahendra Patil < > mahendra.pa...@naaptol.com> wrote: > >> Thanks for yours prompt reply about deviceatlas-enterprise-c-3.2 >> error while make >> after yours suggestion deviceatlas-enterprise-c-3.2 *make *works fine >> >> But while using with haproxy-2.8.3 version gives following error >> >> [root@govinda opt]# cd haproxy-2.8.3 >> [root@govinda haproxy-2.8.3]# make TARGET=linux-glibc USE_PCRE=1 >> USE_OPENSSL=1 USE_ZLIB=1 USE_DEVICEATLAS=1 >> DEVICEATLAS_SRC=/opt/deviceatlas-enterprise-c-3.2/Src/ >> CC src/ev_poll.o >> CC src/ev_epoll.o >> CC src/cpuset.o >> CC src/ssl_sock.o >> CC src/ssl_ckch.o >> CC src/ssl_sample.o >> CC src/ssl_crtlist.o >> CC src/cfgparse-ssl.o >> CC src/ssl_utils.o >> CC src/jwt.o >> CC src/ssl_ocsp.o >> CC /opt/deviceatlas/Src//Os/daunix.o >> CC /opt/deviceatlas/Src//dadwcom.o >> CC /opt/deviceatlas/Src//dasch.o >> CC /opt/deviceatlas/Src//json.o >> CC /opt/deviceatlas/Src//dac.o >> /opt/deviceatlas/Src//dac.c:1033:0: warning: ignoring #pragma >> optimize [-Wunknown-pragmas] >> #pragma optimize("Ofast") >> ^ >> /opt/deviceatlas/Src//dac.c:1041:0: warning: ignoring #pragma >> optimize [-Wunknown-pragmas] >> #pragma optimize("Ofast") >> ^ >> /opt/deviceatlas/Src//dac.c:1049:0: warning: ignoring #pragma >>
Re: [PATCH 1/1] CI: cirrus-ci: display gdb bt if any
ping :) пт, 8 сент. 2023 г. в 22:57, Ilya Shipitsin : > previously, if test process crashes (either BUG_ON or segfault), no > coredump were collected and analysed > --- > .cirrus.yml | 7 ++- > 1 file changed, 6 insertions(+), 1 deletion(-) > > diff --git a/.cirrus.yml b/.cirrus.yml > index 2993b943a..4bf3fb672 100644 > --- a/.cirrus.yml > +++ b/.cirrus.yml > @@ -6,8 +6,13 @@ FreeBSD_task: >install_script: > - pkg update -f && pkg upgrade -y && pkg install -y openssl git gmake > lua53 socat pcre >script: > +- sudo sysctl kern.corefile=/tmp/%N.%P.core > +- sudo sysctl kern.sugid_coredump=1 > - scripts/build-vtest.sh > - gmake CC=clang V=1 ERR=1 TARGET=freebsd USE_ZLIB=1 USE_PCRE=1 > USE_OPENSSL=1 USE_LUA=1 LUA_INC=/usr/local/include/lua53 > LUA_LIB=/usr/local/lib LUA_LIB_NAME=lua-5.3 > - ./haproxy -vv > - ldd haproxy > -- env VTEST_PROGRAM=../vtest/vtest gmake reg-tests > REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do > cat $folder/INFO $folder/LOG; done && exit 1) > + test_script: > +- env VTEST_PROGRAM=../vtest/vtest gmake reg-tests > REGTESTS_TYPES=default,bug,devel > + on_failure: > +debug_script: (for folder in /tmp/*regtest*/vtc.*; do cat > $folder/INFO $folder/LOG; done && ls /tmp/haproxy.*.core && gdb -ex 'thread > apply all bt full' ./haproxy /tmp/haproxy.*.core) > -- > 2.35.3.windows.1 > >
Re: [PATCH 1/1] CI: cirrus-ci: display gdb bt if any
On Thu, Sep 21, 2023 at 11:55:03PM +0200, ??? wrote: > ping :) yeah sorry, I noticed it just after previous release and thought I had merged it already. It's done now, thanks! Willy
Re: [PATCH] MEDIUM: sample: Enhances converter "bytes" to take variable names as arguments
Hi Lokesh, On Sun, Sep 17, 2023 at 09:45:31PM -0700, Lokesh Jindal wrote: > Hello Willy > > Thanks for the detailed explanations and the feedback. > PFA the two patches based on your feedback. My apologies for the delay. Thanks, I've applied them now. > > - maybe a combination of both, i.e. return 0 as long as the caller does > > not pass SMP_OPT_FINAL, or return a truncated response. > > > Sounds good. The revised patch uses option 3 above. OK I also think it's the most versatile choice. > > > /* Try to decode a variable. */ > > > - if (vars_check_arg(&args[0], NULL)) > > > + if (vars_check_arg(&args[0], err)) > > > return 1; > > > > I can't say if this change was intentional or not, nor if the NULL had a > > particular reason or not, but at least this is unrelated to your patch > > and would need to be in its own patch, so that in case of regression we > > can decide whether to fix or revert it. And if it really fixes an issue > > maybe we'll want to backport it. > > This was intentional. I thought it was a "miss" in the existing code. This > change is now part of the second patch. Perfect, thanks. > I have also fixed indentation/tabs, column width, etc. based on your > feedback. > > > I have a Q for you. Pasting code from the revised patch. > + // arg1 value is greater than the remaining length > + if (smp->opt & SMP_OPT_FINAL) { > + // truncate to remaining length > + length = smp->data.u.str.data - start_idx; > + } else { > + smp->data.u.str.data = 0; > + return 0; > + } > > I tried to write a test in the vtc file for the "else" clause above, but > could not figure out when we will hit that condition. Here is what I tried. > Added following the config section. > + http-response set-header hdr_bytes_0_7 > "%[res.hdr(input),bytes(0,7)]" > + http-response set-header hdr_bytes_0_7_concat > "%[res.hdr(input),bytes(0,7),concat()]" > > Added following in the client section > + # since "bytes" is the non-final operator, response would be empty > + expect resp.http.hdr_bytes_0_7_concat == "" > > But the test failed: > c2EXPECT resp.http.hdr_bytes_0_7_concat (012345) == "" failed > > Anyway, I can add this such a test in another patch if needed and time > permits. Let me know what you think. You can't do these using HTTP rules because they're only evaluated once the message is complete. The only case these are used is in TCP rules, because TCP rules can also process HTTP by decoding it on the fly. For example you can have a TCP frontend that inspects protocol, and if it detects HTTP with a certain header or content, routes it to an HTTP backend, otherwise routes it to a TCP frontend. And more generally at the TCP level it's more common to process bytes than at the HTTP level. We could for instance imagine that you extract bytes from an SSL Client Hello message to figure an offset and a length, then assign these to a pair of variables, and reuse them with bytes() with your modification. But raw TCP is less easy to manipulate in VTC, and stuff like this is prone to time races, which regularly cause false positives, and the best is the ennemy of good, so no worries :-) Thanks! Willy