Re: haproxy 2.1.2 and 2.1.3
On 02/21/2020 12:32 PM, Tim Düsterhus wrote: Miroslav, I order to save you duplicated effort in helping I'd like to let you know that Marcel also created a GitHub issue upon request from Ilya: https://github.com/haproxy/haproxy/issues/512 Hello Tim, ah.. okay, i missed that. Thank you. :) Best regards, -- Zaga What can change the nature of a man?
RE: haproxy 2.1.2 and 2.1.3
Hi all, I tried to recompile all previously compiled haproxy version. All gave same output during compilation, but all newly compiled versions do not seem to work... As far as I can see in my history I compiled haproxy 2.0.0 with gcc version 4.8.2: root@SYSTEM:/export/home/USER/haproxy-2.0.0# gcc --version gcc (GCC) 4.8.2 Copyright (C) 2013 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. And now it's using 7.3.0: root@sz4203:/export/home/bhr_kloos605/haproxy-2.0.0# gcc --version gcc (GCC) 7.3.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Not sure if that could be of any impact... Met vriendelijke groet/Kind regards, Marcel Kloosterman Specialist Technisch Beheer KPN N ACN P DC Cloud 1 Phone: +31612413284 Email: marcel.klooster...@kpn.com TeamWiki (with spoc planning): https://confluence.kpn.org/display/KDU/KPN+DCLOUD+SOLARIS+CLOUD+TEAM+HOMEPAGE (JIRA access is required and can be requested using IAM portal) BTW nr.: NL0092.92.056B01 KvK: 27124701 The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you. -Oorspronkelijk bericht- Van: William Lallemand Verzonden: vrijdag 21 februari 2020 10:55 Aan: Kloosterman, Marcel CC: haproxy@formilux.org Onderwerp: Re: haproxy 2.1.2 and 2.1.3 Hello, On Fri, Feb 21, 2020 at 08:52:05AM +, marcel.klooster...@kpn.com wrote: > Hi all, > > At this moment we’re using HA-Proxy version 2.0.0 2019/06/16 - > https://haproxy.org/ And we’re running solaris 11.4 sru 950 for SPARC and I > used developer studio version 12.6 to compile the newer versions and 12.5 to > compile the 2.0.0 version. > > And because of a known bug in haproxy, we wanted to upgrade to: > HA-Proxy version 2.1.2 2019/12/21 - https://haproxy.org/ On our test-server > we had no problems but on the production server it kept crashing. > The problem is somehow related to ssl, because as soon as we add “ssl crt > /etc/certs/self.pem” to the bind it crashes also on the test-server. > So then I checked and version 2.1.3 is already available, solving lots > of ssl bugs. So I tried that one also…but without any luck ☹ > > Starting haproxy now gives a Bus Error (with both versons, 2.1.2 and 2.1.3): > > root@sz4203:~ # /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f > /usr/local/etc/haproxy.cfg Bus Error (core dumped) > > Can you please help me solving this issue ? > > This is the stack-dump of the 2.1.3 version I compiled: > root@s00501:/var/core# pstack core_sz4203_haproxy_0_0_1582273479_18461 > core 'core_sz4203_haproxy_0_0_1582273479_18461' of 18461: > /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f /usr/local/etc/haproxy. > 000100240c30 shctx_init (100366170, 4e20, c4, , > 7ec00144, 0) + b0 (shctx.c:368) 000100086e7c > ssl_sock_prepare_bind_conf (1005bbb80, , fffefffc, > 29729948, , 100366000) + 1bc > 000100135278 check_config_validity (fffefffc, 1400, > 1005bbb80, 1005bb750, 100357238, 0) + 25b8 (cfgparse.c:3753) > 000100173450 init (100357468, 1000596a0, 10005a940, 100357238, > 10003aa90, 0) + 610 (haproxy.c:1884) > 000100178058 main (5, 7688, 76b8, 10, > 10046d680, 0) + 178 (haproxy.c:2858) > 00010006caa4 _start (0, 0, 0, 0, 0, 0) + 64 > That's really odd, what is crashing is the shctx_init(), which is the initialization of the shared memory used by the SSL, it didn't changed much in 2 years so I'm surprised that you didn't have this problem with haproxy 2.0. Since its running on SPARC and that the Bus Error appears in the the shctx block access that could be an alignment issue. But if it's working on your test server it might be something else. Do the test server and the production one share the same hardware? (CPU, RAM etc.) Could you display the stacktrace with gdb? The arguments looks weird to me, (too much arguments and weird values) and I don't know how reliable is pstack about this. Did you try to build haproxy 2.0 with your new compiler and do the same tests? Do you have the same issue? -- William Lallemand
Re: haproxy 2.1.2 and 2.1.3
Miroslav, Am 21.02.20 um 12:22 schrieb Miroslav Zagorac: > You can start gdb like this: > > % gdb -core core-file program > > , and in your case it would look like this: > > % gdb -core core_sz4203_haproxy_0_0_1582273479_18461 > /usr/local/sbin/haproxy > > After that, in gdb program you write command 'where' (without quotes), > which will print backtrace of all stack frames. It will work better if > you have debug symbols included in the program you want to debug. :) > I order to save you duplicated effort in helping I'd like to let you know that Marcel also created a GitHub issue upon request from Ilya: https://github.com/haproxy/haproxy/issues/512 Best regards Tim Düsterhus
Re: haproxy 2.1.2 and 2.1.3
On 02/21/2020 11:42 AM, marcel.klooster...@kpn.com wrote: Hi, It's not working on test-server also. Test-server is a solaris zone on T7 with 11.4 sru 950, production is a solaris zone on T8 with 11.4 sru 950. You can start gdb like this: % gdb -core core-file program , and in your case it would look like this: % gdb -core core_sz4203_haproxy_0_0_1582273479_18461 /usr/local/sbin/haproxy After that, in gdb program you write command 'where' (without quotes), which will print backtrace of all stack frames. It will work better if you have debug symbols included in the program you want to debug. :) -- Zaga What can change the nature of a man?
RE: haproxy 2.1.2 and 2.1.3
Hi, It's not working on test-server also. Test-server is a solaris zone on T7 with 11.4 sru 950, production is a solaris zone on T8 with 11.4 sru 950. What I also tried is: root@SYSTEM:~# gdb GNU gdb (GDB) 8.0 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "sparc-sun-solaris2.11". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word". (gdb) exec-file No executable file now. (gdb) exec-file /usr/local/sbin/haproxy (gdb) run -p /tmp/haproxy.pid -f /usr/local/etc/haproxy.cfg Starting program: /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f /usr/local/etc/haproxy.cfg [Thread debugging using libthread_db enabled] [New Thread 1 (LWP 1)] Thread 2 received signal SIGSEGV, Segmentation fault. [Switching to Thread 1 (LWP 1)] 0x000100242c90 in ?? () (gdb) bt full #0 0x000100242c90 in ?? () No symbol table info available. Backtrace stopped: previous frame identical to this frame (corrupt stack?) As already stated in the other mail. I'm not really familiar with gdb. Can you guide me. Met vriendelijke groet/Kind regards, Marcel Kloosterman Specialist Technisch Beheer KPN N ACN P DC Cloud 1 Phone: +31612413284 Email: marcel.klooster...@kpn.com TeamWiki (with spoc planning): https://confluence.kpn.org/display/KDU/KPN+DCLOUD+SOLARIS+CLOUD+TEAM+HOMEPAGE (JIRA access is required and can be requested using IAM portal) BTW nr.: NL0092.92.056B01 KvK: 27124701 The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you. -Oorspronkelijk bericht- Van: William Lallemand Verzonden: vrijdag 21 februari 2020 10:55 Aan: Kloosterman, Marcel CC: haproxy@formilux.org Onderwerp: Re: haproxy 2.1.2 and 2.1.3 Hello, On Fri, Feb 21, 2020 at 08:52:05AM +, marcel.klooster...@kpn.com wrote: > Hi all, > > At this moment we’re using HA-Proxy version 2.0.0 2019/06/16 - > https://haproxy.org/ And we’re running solaris 11.4 sru 950 for SPARC and I > used developer studio version 12.6 to compile the newer versions and 12.5 to > compile the 2.0.0 version. > > And because of a known bug in haproxy, we wanted to upgrade to: > HA-Proxy version 2.1.2 2019/12/21 - https://haproxy.org/ On our test-server > we had no problems but on the production server it kept crashing. > The problem is somehow related to ssl, because as soon as we add “ssl crt > /etc/certs/self.pem” to the bind it crashes also on the test-server. > So then I checked and version 2.1.3 is already available, solving lots > of ssl bugs. So I tried that one also…but without any luck ☹ > > Starting haproxy now gives a Bus Error (with both versons, 2.1.2 and 2.1.3): > > root@sz4203:~ # /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f > /usr/local/etc/haproxy.cfg Bus Error (core dumped) > > Can you please help me solving this issue ? > > This is the stack-dump of the 2.1.3 version I compiled: > root@s00501:/var/core# pstack core_sz4203_haproxy_0_0_1582273479_18461 > core 'core_sz4203_haproxy_0_0_1582273479_18461' of 18461: > /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f /usr/local/etc/haproxy. > 000100240c30 shctx_init (100366170, 4e20, c4, , > 7ec00144, 0) + b0 (shctx.c:368) 000100086e7c > ssl_sock_prepare_bind_conf (1005bbb80, , fffefffc, > 29729948, , 100366000) + 1bc > 000100135278 check_config_validity (fffefffc, 1400, > 1005bbb80, 1005bb750, 100357238, 0) + 25b8 (cfgparse.c:3753) > 000100173450 init (100357468, 1000596a0, 10005a940, 100357238, > 10003aa90, 0) + 610 (haproxy.c:1884) > 000100178058 main (5, 7688, 76b8, 10, > 10046d680, 0) + 178 (haproxy.c:2858) > 00010006caa4 _start (0, 0, 0, 0, 0, 0) + 64 > That's really odd, what is crashing is the shctx_init(), which is the initialization of the shared memory used by the SSL, it didn't changed much in 2 years so
RE: haproxy 2.1.2 and 2.1.3
Hi, I created issue: haproxy 2.1.3 issue with ssl on solaris 11 sru 950 #512 Nut really sure about gdb and bt full. First time using it… had to install it first. Is this what you need: root@SYSTEM:/var/core# gdb --core=core_sz4203_haproxy_0_0_1582279635_18282 GNU gdb (GDB) 8.0 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "sparc-sun-solaris2.11". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word". [New LWP 1] [New LWP 1] Core was generated by `/usr/local/sbin/haproxy -p /tmp/haproxy.pid -f /usr/local/etc/haproxy.cfg'. Program terminated with signal SIGBUS, Bus error. #0 0x000100242c90 in ?? () (gdb) bt full #0 0x000100242c90 in ?? () No symbol table info available. Backtrace stopped: previous frame identical to this frame (corrupt stack?) Can you please guide me… Met vriendelijke groet/Kind regards, [cid:image001.png@01D5E8AA.6AD7A760] Marcel Kloosterman Specialist Technisch Beheer KPN N ACN P DC Cloud 1 Phone: +31612413284 Email: marcel.klooster...@kpn.com<mailto:marcel.klooster...@kpn.com> TeamWiki (with spoc planning): https://confluence.kpn.org/display/KDU/KPN+DCLOUD+SOLARIS+CLOUD+TEAM+HOMEPAGE (JIRA access is required and can be requested using IAM portal) BTW nr.: NL0092.92.056B01 KvK: 27124701 The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you. Van: Илья Шипицин Verzonden: vrijdag 21 februari 2020 10:24 Aan: Kloosterman, Marcel CC: HAProxy Onderwerp: Re: haproxy 2.1.2 and 2.1.3 useful things to debug would be 1) haorpxy -vv 2) config 3) bt full can you please open a github issue at https://github.com/haproxy/haproxy/issues ? пт, 21 февр. 2020 г. в 13:55, mailto:marcel.klooster...@kpn.com>>: Hi all, At this moment we’re using HA-Proxy version 2.0.0 2019/06/16 - https://haproxy.org/ And we’re running solaris 11.4 sru 950 for SPARC and I used developer studio version 12.6 to compile the newer versions and 12.5 to compile the 2.0.0 version. And because of a known bug in haproxy, we wanted to upgrade to: HA-Proxy version 2.1.2 2019/12/21 - https://haproxy.org/ On our test-server we had no problems but on the production server it kept crashing. The problem is somehow related to ssl, because as soon as we add “ssl crt /etc/certs/self.pem” to the bind it crashes also on the test-server. So then I checked and version 2.1.3 is already available, solving lots of ssl bugs. So I tried that one also…but without any luck ☹ Starting haproxy now gives a Bus Error (with both versons, 2.1.2 and 2.1.3): root@sz4203:~ # /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f /usr/local/etc/haproxy.cfg Bus Error (core dumped) Can you please help me solving this issue ? This is the stack-dump of the 2.1.3 version I compiled: root@s00501:/var/core# pstack core_sz4203_haproxy_0_0_1582273479_18461 core 'core_sz4203_haproxy_0_0_1582273479_18461' of 18461: /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f /usr/local/etc/haproxy. 000100240c30 shctx_init (100366170, 4e20, c4, , 7ec00144, 0) + b0 (shctx.c:368) 000100086e7c ssl_sock_prepare_bind_conf (1005bbb80, , fffefffc, 29729948, , 100366000) + 1bc 000100135278 check_config_validity (fffefffc, 1400, 1005bbb80, 1005bb750, 100357238, 0) + 25b8 (cfgparse.c:3753) 000100173450 init (100357468, 1000596a0, 10005a940, 100357238, 10003aa90, 0) + 610 (haproxy.c:1884) 000100178058 main (5, 7688, 76b8, 10, 10046d680, 0) + 178 (haproxy.c:2858) 00010006caa4 _start (0, 0, 0, 0, 0, 0) + 64 And this is how I compiled the 2.1.3 version: PATH=/usr/gnu/bin:${PATH}:/opt/developerstudio12.6/bin/ cd ~bhr_kloos605/haproxy-2.1.3/ vi Makefile Just to make sure that I can compile…changes: root@sz4203:/export/home/bhr_kloos605# diff haproxy-2.1.3/Makefile haproxy-2.1.3.compiled/Makefile 137c137 < DESTDIR = --- > DESTDIR = /tmp/haproxy 662c662,663 < OPTIONS_LDF
Re: haproxy 2.1.2 and 2.1.3
Hello, On Fri, Feb 21, 2020 at 08:52:05AM +, marcel.klooster...@kpn.com wrote: > Hi all, > > At this moment we’re using HA-Proxy version 2.0.0 2019/06/16 - > https://haproxy.org/ > And we’re running solaris 11.4 sru 950 for SPARC and I used developer studio > version 12.6 to compile the newer versions and 12.5 to compile the 2.0.0 > version. > > And because of a known bug in haproxy, we wanted to upgrade to: HA-Proxy > version 2.1.2 2019/12/21 - https://haproxy.org/ > On our test-server we had no problems but on the production server it kept > crashing. > The problem is somehow related to ssl, because as soon as we add “ssl crt > /etc/certs/self.pem” to the bind it crashes also on the test-server. > So then I checked and version 2.1.3 is already available, solving lots of ssl > bugs. So I tried that one also…but without any luck ☹ > > Starting haproxy now gives a Bus Error (with both versons, 2.1.2 and 2.1.3): > > root@sz4203:~ # /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f > /usr/local/etc/haproxy.cfg > Bus Error (core dumped) > > Can you please help me solving this issue ? > > This is the stack-dump of the 2.1.3 version I compiled: > root@s00501:/var/core# pstack core_sz4203_haproxy_0_0_1582273479_18461 > core 'core_sz4203_haproxy_0_0_1582273479_18461' of 18461: > /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f /usr/local/etc/haproxy. > 000100240c30 shctx_init (100366170, 4e20, c4, , 7ec00144, > 0) + b0 (shctx.c:368) > 000100086e7c ssl_sock_prepare_bind_conf (1005bbb80, , > fffefffc, 29729948, , 100366000) + 1bc > 000100135278 check_config_validity (fffefffc, 1400, 1005bbb80, > 1005bb750, 100357238, 0) + 25b8 (cfgparse.c:3753) > 000100173450 init (100357468, 1000596a0, 10005a940, 100357238, 10003aa90, > 0) + 610 (haproxy.c:1884) > 000100178058 main (5, 7688, 76b8, 10, > 10046d680, 0) + 178 (haproxy.c:2858) > 00010006caa4 _start (0, 0, 0, 0, 0, 0) + 64 > That's really odd, what is crashing is the shctx_init(), which is the initialization of the shared memory used by the SSL, it didn't changed much in 2 years so I'm surprised that you didn't have this problem with haproxy 2.0. Since its running on SPARC and that the Bus Error appears in the the shctx block access that could be an alignment issue. But if it's working on your test server it might be something else. Do the test server and the production one share the same hardware? (CPU, RAM etc.) Could you display the stacktrace with gdb? The arguments looks weird to me, (too much arguments and weird values) and I don't know how reliable is pstack about this. Did you try to build haproxy 2.0 with your new compiler and do the same tests? Do you have the same issue? -- William Lallemand
Re: haproxy 2.1.2 and 2.1.3
useful things to debug would be 1) haorpxy -vv 2) config 3) bt full can you please open a github issue at https://github.com/haproxy/haproxy/issues ? пт, 21 февр. 2020 г. в 13:55, : > Hi all, > > > > At this moment we’re using HA-Proxy version 2.0.0 2019/06/16 - > https://haproxy.org/ > > And we’re running solaris 11.4 sru 950 for SPARC and I used developer > studio version 12.6 to compile the newer versions and 12.5 to compile the > 2.0.0 version. > > > > And because of a known bug in haproxy, we wanted to upgrade to: HA-Proxy > version 2.1.2 2019/12/21 - https://haproxy.org/ > > On our test-server we had no problems but on the production server it kept > crashing. > > The problem is somehow related to ssl, because as soon as we add “ssl crt > /etc/certs/self.pem” to the bind it crashes also on the test-server. > > So then I checked and version 2.1.3 is already available, solving lots of > ssl bugs. So I tried that one also…but without any luck ☹ > > > > Starting haproxy now gives a Bus Error (with both versons, 2.1.2 and > 2.1.3): > > > > root@sz4203:~ # /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f > /usr/local/etc/haproxy.cfg > > Bus Error (core dumped) > > > > Can you please help me solving this issue ? > > > > This is the stack-dump of the 2.1.3 version I compiled: > > root@s00501:/var/core# pstack core_sz4203_haproxy_0_0_1582273479_18461 > > core 'core_sz4203_haproxy_0_0_1582273479_18461' of 18461: > /usr/local/sbin/haproxy -p /tmp/haproxy.pid -f /usr/local/etc/haproxy. > > 000100240c30 shctx_init (100366170, 4e20, c4, , > 7ec00144, 0) + b0 (shctx.c:368) > > 000100086e7c ssl_sock_prepare_bind_conf (1005bbb80, , > fffefffc, 29729948, , 100366000) + 1bc > > 000100135278 check_config_validity (fffefffc, 1400, 1005bbb80, > 1005bb750, 100357238, 0) + 25b8 (cfgparse.c:3753) > > 000100173450 init (100357468, 1000596a0, 10005a940, 100357238, > 10003aa90, 0) + 610 (haproxy.c:1884) > > 000100178058 main (5, 7688, 76b8, 10, > 10046d680, 0) + 178 (haproxy.c:2858) > > 00010006caa4 _start (0, 0, 0, 0, 0, 0) + 64 > > > > And this is how I compiled the 2.1.3 version: > > > > PATH=/usr/gnu/bin:${PATH}:/opt/developerstudio12.6/bin/ > > cd ~bhr_kloos605/haproxy-2.1.3/ > > vi Makefile > > Just to make sure that I can compile…changes: > > root@sz4203:/export/home/bhr_kloos605# diff haproxy-2.1.3/Makefile > haproxy-2.1.3.compiled/Makefile > > 137c137 > > < DESTDIR = > > --- > > > DESTDIR = /tmp/haproxy > > 662c662,663 > > < OPTIONS_LDFLAGS += $(if $(PCRE_LIB),-L$(PCRE_LIB)) -Wl,-Bstatic > -lpcreposix -lpcre -Wl,-Bdynamic > > --- > > > #OPTIONS_LDFLAGS += $(if $(PCRE_LIB),-L$(PCRE_LIB)) -Wl,-Bstatic > -lpcreposix -lpcre -Wl,-Bdynamic > > > OPTIONS_LDFLAGS += $(if $(PCRE_LIB),-L$(PCRE_LIB)) -Bstatic -lpcreposix > -lpcre > > make TARGET=solaris CPU=ultrasparc PCRE_LIB=/usr/lib > PCRE_INC=/usr/include/pcre USE_STATIC_PCRE=1 USE_OPENSSL=1 > > make install > > > > Can you please help ? > > > > > > Met vriendelijke groet/Kind regards, > > > > *Marcel Kloosterman* > > *Specialist Technisch Beheer* > > > > > > *KPN* > > N ACN P DC Cloud 1 > > Phone: +31612413284 > > Email: marcel.klooster...@kpn.com > > TeamWiki (with spoc planning): > https://confluence.kpn.org/display/KDU/KPN+DCLOUD+SOLARIS+CLOUD+TEAM+HOMEPAGE > > (JIRA access is required and can be requested using IAM portal) > > *BTW nr.: NL0092.92.056B01* > > *KvK: 27124701* > > > > *The information transmitted is intended only for use by the addressee and > may contain confidential and/or privileged material. Any review, > re-transmission, dissemination or other use of it, or the taking of any > action in reliance upon this information by persons and/or entities other > than the intended recipient is prohibited. If you received this in error, > please inform the sender and/or addressee immediately and delete the > material. Thank you.* > > >