Hi Gustaf thanks again for the advice. Today I made some more progress on this. There does appear to be some differences between your script and the Oupfiz5 installer e.g. his ns-build.sh script https://github.com/oupfiz5/tcl-build/blob/master/src/builds/ns-build.sh I have reached the conclusion that I will be wasting your time if I can't reproduce this problem using your scripts, so my next task will be to run your script and try to reproduce. I am now seeing the downsides to using a non-official Docker approach!
Today I took the approach of installing (through the APM) our OpenACS packages one by one. For example, we use packages such as Categories, General Comments etc as well as many of our own custom packages. After each package I bounced Naviserver and tested the site. The system worked perfectly until after I installed the last package, which is our main core of our product, very large and old with a lot of features. This makes me very confident that Oracle and nsoracle are working fine. The problem could be some API call in our custom package that maybe changed in 4.99.25. To answer some of your questions: * did you run at this state any Oracle queries? Yes, I did. I'm 95% confident that Oracle and nsoracle are working fine. * did you recompile in the "clean install" also the oracle driver? Yes, I'm building nsoracle from scratch (I am also running the same version of nsoracle in the 4.99.24 build that is working without issue) * you mean the crash happens in the plain openacs-config.tcl, with no additional drivers etc, no oracle involved? No, this does use Oracle, sorry for not being clear. We have our own heavily modified config file, so I wanted to rule that out by using the openacs-config.tcl that you provide. I just changed the database to Oracle and left everything else as is. The fact that it crashed too means that I can eliminate some strange configuration setting in our custom config file as a possible cause. * My request in the last mail was to try to reproduce the problem with nsd-config.tcl (i.e. no OpenACS involved). Yes, I replied previously that it runs fine. And also a simple OpenACS install on Oracle runs fine. The problems only start with our custom OpenACS package. * To be on the safe side, all /usr/local/ns/bin/*.so files should be newly compiled. Yes, these all appear to be freshly compiled. # ls -l /usr/local/ns/bin/*.so -rwxr-xr-x 1 nsadmin nsadmin 32560 Aug 10 15:31 /usr/local/ns/bin/nscgi.so -rwxr-xr-x 1 nsadmin nsadmin 27360 Aug 10 15:31 /usr/local/ns/bin/nscp.so -rwxr-xr-x 1 nsadmin nsadmin 15808 Aug 10 15:31 /usr/local/ns/bin/nsdb.so -rwxr-xr-x 1 nsadmin nsadmin 50808 Aug 10 15:31 /usr/local/ns/bin/nsdbpg.so -rwxr-xr-x 1 nsadmin nsadmin 16176 Aug 10 15:31 /usr/local/ns/bin/nsdbtest.so -rwxr-xr-x 1 nsadmin nsadmin 32640 Aug 10 15:31 /usr/local/ns/bin/nslog.so -rwxr-xr-x 1 nsadmin nsadmin 90688 Aug 10 15:42 /usr/local/ns/bin/nsoracle.so -rwxr-xr-x 1 nsadmin nsadmin 90848 Aug 10 15:42 /usr/local/ns/bin/nsoraclecass.so -rwxr-xr-x 1 nsadmin nsadmin 31712 Aug 10 15:31 /usr/local/ns/bin/nsperm.so -rwxr-xr-x 1 nsadmin nsadmin 15888 Aug 10 15:31 /usr/local/ns/bin/nsproxy.so -rwxr-xr-x 1 nsadmin nsadmin 16536 Aug 10 15:31 /usr/local/ns/bin/nssock.so -rwxr-xr-x 1 nsadmin nsadmin 26624 Aug 10 15:31 /usr/local/ns/bin/nsssl.so So my next steps are to try to reproduce the problem using your install-ns.sh script. Then I can compile with debugging and have some fun with gdb. thanks Brian ________________________________ From: Gustaf Neumann <neum...@wu.ac.at> Sent: Saturday 12 August 2023 11:55 am To: naviserver-devel@lists.sourceforge.net <naviserver-devel@lists.sourceforge.net> Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu On 11.08.23 20:15, Brian Fenton wrote: Hi Gustaf thanks for the response. I've been looking at this in more detail this afternoon and it does appear to be caused by something in the interaction of our OpenACS application with 4.99.27. As I previously mentioned, it has been running fine on 4.99.24 on the same Ubuntu version. I realise that I may not have been clear on this point on my previous email: this is Naviserver running on Ubuntu in a Docker container. The version of Naviserver is based on this Docker build https://github.com/oupfiz5/naviserver-s6 which I have forked and updated to 4.99.27 (I may well have missed something in updating NS version - maybe I should have waited until oupfiz updates his build). * I can confirm that nsd-config.tcl runs fine with 4.99.27 * Some good news: I am able to do an OpenACS clean install on Oracle with 4.99.27. I then successfully installed our application using the APM. did you run at this state any Oracle queries? did you recompile in the "clean install" also the oracle driver? * However, once I restart Naviserver the problems start. * I tried using the openacs-config.tcl that ships with 4.99.27 and the problems are happening with that too. you mean the crash happens in the plain openacs-config.tcl, with no additional drivers etc, no oracle involved? this can get us closer to something i might be able to reproduce. My request in the last mail was to try to reproduce the problem with nsd-config.tcl (i.e. no OpenACS involved). If you can reproduce the crash, you should compile with debugging turned on and run nsd under gdb or lldb. First one should get he most simple case causing the crash. What is odd is that it seems to be able to handle one request before crashing. Eg. I type in the URL, it shows the /register page but then crashes. After restarting, I enter my login details on the register page, press return. It then crashes. After restarting, it successfully logs me, then crashes again. the memory errors or normally hinting on some buffer overflow, or a mixture between 32bit and 64bit compilation, etc. There is no clear pattern in the logs. I thought it might be related to OCSP and disabled that, but the problems continued to occur. if you suspect nsssl, then one potential problem might be a mixture during of different OpenSSL versions during compilation (when using install_ns.sh, this will not happen). Turning on debug hasn't helped - but maybe there is so much information in the log that I have missed something important. What drivers are you referring to in your question? actually all naviserver modules you are using, including the db drivers (since you mentioned nsoracle, which is not part of the regular regression tests). To be on the safe side, all /usr/local/ns/bin/*.so files should be newly compiled. all the best -gn thanks Brian ________________________________ From: Gustaf Neumann <neum...@wu.ac.at><mailto:neum...@wu.ac.at> Sent: Thursday 10 August 2023 7:27 pm To: naviserver-devel@lists.sourceforge.net<mailto:naviserver-devel@lists.sourceforge.net> <naviserver-devel@lists.sourceforge.net><mailto:naviserver-devel@lists.sourceforge.net> Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu Hi Brian, The new NaviServer versions are running fine on Ubuntu 22.04. Have you recompiled the drivers you are using with the updated version? A good test for the NaviServer binary is to test it with one of the packaged configuration files, e.g. nsd-config.tcl. all the best -gn On 10.08.23 18:23, Brian Fenton wrote: Hello we have been testing out our OpenACS application on Ubuntu 22.04.2 LTS (previously we only ran on Windows). It was working great with Naviserver 4.99.24 but I have been getting constant crashes on more recent versions. I get this error on 4.99.25, 4.99.26 and today I also got it on 4.99.27. The server runs fine until I click on a page, then it immediately crashes. The log has only the following error: free(): invalid size and today I got this one: [10/Aug/2023:15:02:23][303.7fa3a64ee640][-conn:openacs:default:1:119-] Fatal: received fatal signal 11 We have an Oracle application and are using the latest nsoracle driver, which might be a factor here. We have been running it with a pretty old OpenACS config file, so I am currently looking to merge in all the latest changes to ensure that is not an issue. Also note that I am running Naviserver on Docker on Windows, but as mentioned it was running great on 4.99.24. thanks for any help Brian _______________________________________________ naviserver-devel mailing list naviserver-devel@lists.sourceforge.net<mailto:naviserver-devel@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/naviserver-devel -- Univ.Prof. Dr. Gustaf Neumann Head of the Institute of Information Systems and New Media of Vienna University of Economics and Business Program Director of MSc "Information Systems"
_______________________________________________ naviserver-devel mailing list naviserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/naviserver-devel
