Am 25.09.15 um 15:58 schrieb Zoran Vasiljevic: > On 25 Sep 2015, at 15:44, Gustaf Neumann <neum...@wu.ac.at> wrote: > >> kind of. the fix was addressing a problem in rivet under Linux, >> the consequences were not easy to spot from the code but hit >> all multi-threaded tcl applications, including NaviServer. >> It also lead to bug-reports from bsd about "massive slowdowns". >> > I see... so you had to fix that mess... > At this point I believe you've done the > fix so that it will not harm us! it was already causing harm. i started to look at the problem, when one quite large NaviServer based site was suddenly running our of memory. This site was using [exec ...] without nsproxy (for no good reason). The [exec] call hang sometimes in a fork(), after duplicating the memory. This site had a maximum of 90 connection threads configured, using a footprint of several GB. When forking such a monster multiple times, and the fork hangs, one runs soon out of memory... The obvious fix was to use nsproxy for that site, but still, the Tcl behavior was bad enough to investigate further.. > I guess they wanted to make [fork] working > which simply cannot be done MT-safe(ly). A few people tried to argue for that, but not long :).
The problem was that rivet requires after a fork() a notifier thread running, such that [vwait ...] etc. works. Therefore, it was necessary to use a hook that starts the notifier thread after a fork() operation. That bad thing is that the implementation in 2013 created a notifier thread after every fork, (even for [exec], where it is completely useless). Furthermore, starting a notifier threads requires Tcl_MutexLock(), and even Tcl_MutexLock() requires a (master) lock to function (Tcl maintains a list of mutexes). If one locks this master mutex before fork (which takes a while), all parallelism is gone, since every mutex operation is serialized ... and that happened for every exec! Finally i implemented the notifier thread creation in a lazy manner (create the notifier thread when needed, i.e. not during [exec]), and used the pthread-API for the notifier thread management instead of the Tcl_Mutex* interface. > I've been telling this for years but people > won't listen. And there is some substantional > portion in politics there, which I regret :-( i don't think that politics was a main blocker. It was rather missing problem awareness (most application are not using threads), and the hope for a general [fork] call in Tcl, which makes me shudder when i think about heavy multi-threading applications. -g ------------------------------------------------------------------------------ _______________________________________________ naviserver-devel mailing list naviserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/naviserver-devel
