Re: [Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
The obvious problem with while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev—; is that once something does the http server thing, you'll be running Tcl_DoOneEvent max_ev times, _every_ time you hit TclSpinLoop. I wonder it we could some sort of hybrid between this and Tcl_ServiceAll()? Like, calling Tcl_DoOneEvent once before/after Tcl_ServiceAll(). - Peter > On 21 Feb 2024, at 17:04 , Tomas Kalibera wrote: > > > On 2/21/24 08:01, webmail.gandi.net wrote: >> Thank you, Ivan for this investigation. I inspected the R changes file >> (https://cran.r-project.org/doc/manuals/r-devel/NEWS.html) and found nothing >> about this. I should inspect the sources too! >> >> It could possibly break other Tcl/Tk related stuff. The doc about >> Tcl_ServiceAll and Tcl_DoOneEvent is confusing. On one hand, it says when >> Tcl is used from an external program, Tcl_ServiceAll should be used in its >> event loop instead of Tcl_DoOneEvent (and the change in the latest R >> versions goes in that direction). But in the other hand, it is indicated >> that Tcl_ServiceAll does not always handle all Tcl events and extra >> Tcl_DoOneEvent should be called in this case. I think we spotted one case >> where Tcl_ServiceAll is not doing its job correctly. There may be others. >> >> Since the {tcltk} package was working fine with "while >> (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev—;", unless there is a clear >> performance enhancement with "while (i-- && Tcl_ServiceAll())", it would >> perhaps be wise to revert this back. > > Yes, for now I've done that in R-devel. The Tcl documentation is really hard > to follow, but debugging reveals that Tcl_ServiceAll() doesn't queue new > events, as also Ivan reports. > > Tomas > >> >> Indeed, when I use this on the server side with R 4.3.2: >> >> library(tcltk) >> cmd <- r"( >> proc accept {chan addr port} { ;# Make a proc to accept connections >>puts "$addr:$port says [gets $chan]" ;# Receive a string >>puts $chan goodbye ;# Send a string >>close $chan ;# Close the socket (automatically >> flushes) >> } ;# >> socket -server accept 12345 ;# Create a server socket)" >> .Tcl(cmd) >> .Tcl("vwait myvar") >> >> It works again as expected. And vwait is known to call Tcl_DoOneEvent. Of >> course, in this case, R is blocked and waits for the `myvar` variable on the >> Tcl side. Anyway, the problem seems to be really in Tcl_ServiceAll not >> catching all Tcl events. >> >> All the best, >> >> Philippe >> >> ..<°}))>< >> ) ) ) ) ) >> ( ( ( ( (Prof. Philippe Grosjean >> ) ) ) ) ) >> ( ( ( ( (Numerical Ecology >> ) ) ) ) ) Mons University, Belgium >> ( ( ( ( ( >> .. >> >>> Le 20 févr. 2024 à 17:13, Ivan Krylov via R-devel a >>> écrit : >>> >>> В Tue, 20 Feb 2024 12:27:35 +0100 >>> "webmail.gandi.net" пишет: >>> When R process #1 is R 4.2.3, it works as expected (whatever version of R #2). When R process #1 is R 4.3.2, nothing is sent or received through the socket apparently, but no error is issued and process #2 seems to be able to connect to the socket. >>> The difference is related to the change in >>> src/library/tcltk/src/tcltk_unix.c. >>> >>> In R-4.2.1, the function static void TclSpinLoop(void *data) says: >>> >>>int max_ev = 100; >>>/* Tcl_ServiceAll is not enough here, for reasons that escape me */ >>>while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev--; >>> >>> In R-devel, the function instead says: >>> >>>int i = R_TCL_SPIN_MAX; >>>while (i-- && Tcl_ServiceAll()) >>>; >>> >>> Manually calling Tcl_DoOneEvent(0) from the debugger at this point >>> makes the Tcl code respond to the connection. Tcl_ServiceAll() seems to >>> be still not enough. I'll try reading Tcl documentation to investigate >>> this further. >>> >>> -- >>> Best regards, >>> Ivan >>> >>> __ >>> R-devel@r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-devel >> >> [[alternative HTML version deleted]] >> >> __ >> R-devel@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel -- Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Office: A 4.23 Email: pd@cbs.dk Priv: pda...@gmail.com __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
On 2/21/24 08:01, webmail.gandi.net wrote: Thank you, Ivan for this investigation. I inspected the R changes file (https://cran.r-project.org/doc/manuals/r-devel/NEWS.html) and found nothing about this. I should inspect the sources too! It could possibly break other Tcl/Tk related stuff. The doc about Tcl_ServiceAll and Tcl_DoOneEvent is confusing. On one hand, it says when Tcl is used from an external program, Tcl_ServiceAll should be used in its event loop instead of Tcl_DoOneEvent (and the change in the latest R versions goes in that direction). But in the other hand, it is indicated that Tcl_ServiceAll does not always handle all Tcl events and extra Tcl_DoOneEvent should be called in this case. I think we spotted one case where Tcl_ServiceAll is not doing its job correctly. There may be others. Since the {tcltk} package was working fine with "while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev—;", unless there is a clear performance enhancement with "while (i-- && Tcl_ServiceAll())", it would perhaps be wise to revert this back. Yes, for now I've done that in R-devel. The Tcl documentation is really hard to follow, but debugging reveals that Tcl_ServiceAll() doesn't queue new events, as also Ivan reports. Tomas Indeed, when I use this on the server side with R 4.3.2: library(tcltk) cmd <- r"( proc accept {chan addr port} { ;# Make a proc to accept connections puts "$addr:$port says [gets $chan]" ;# Receive a string puts $chan goodbye ;# Send a string close $chan ;# Close the socket (automatically flushes) } ;# socket -server accept 12345 ;# Create a server socket)" .Tcl(cmd) .Tcl("vwait myvar") It works again as expected. And vwait is known to call Tcl_DoOneEvent. Of course, in this case, R is blocked and waits for the `myvar` variable on the Tcl side. Anyway, the problem seems to be really in Tcl_ServiceAll not catching all Tcl events. All the best, Philippe ..<°}))>< ) ) ) ) ) ( ( ( ( (Prof. Philippe Grosjean ) ) ) ) ) ( ( ( ( (Numerical Ecology ) ) ) ) ) Mons University, Belgium ( ( ( ( ( .. Le 20 févr. 2024 à 17:13, Ivan Krylov via R-devel a écrit : В Tue, 20 Feb 2024 12:27:35 +0100 "webmail.gandi.net" пишет: When R process #1 is R 4.2.3, it works as expected (whatever version of R #2). When R process #1 is R 4.3.2, nothing is sent or received through the socket apparently, but no error is issued and process #2 seems to be able to connect to the socket. The difference is related to the change in src/library/tcltk/src/tcltk_unix.c. In R-4.2.1, the function static void TclSpinLoop(void *data) says: int max_ev = 100; /* Tcl_ServiceAll is not enough here, for reasons that escape me */ while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev--; In R-devel, the function instead says: int i = R_TCL_SPIN_MAX; while (i-- && Tcl_ServiceAll()) ; Manually calling Tcl_DoOneEvent(0) from the debugger at this point makes the Tcl code respond to the connection. Tcl_ServiceAll() seems to be still not enough. I'll try reading Tcl documentation to investigate this further. -- Best regards, Ivan __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
I don't think we're going to fix this before 4.3.3. Given that it has gone unnoticed since June 2022 (yes '22) and that tampering in this area has a history of popping up complications in other areas, I think we should leave it alone until 4.4.0. (I see that Ivan and Tomas has been on the issue since I started writing, but the above probably still holds true.) - Peter D. > On 21 Feb 2024, at 08:01 , webmail.gandi.net wrote: > > Thank you, Ivan for this investigation. I inspected the R changes file > (https://cran.r-project.org/doc/manuals/r-devel/NEWS.html) and found nothing > about this. I should inspect the sources too! > > It could possibly break other Tcl/Tk related stuff. The doc about > Tcl_ServiceAll and Tcl_DoOneEvent is confusing. On one hand, it says when Tcl > is used from an external program, Tcl_ServiceAll should be used in its event > loop instead of Tcl_DoOneEvent (and the change in the latest R versions goes > in that direction). But in the other hand, it is indicated that > Tcl_ServiceAll does not always handle all Tcl events and extra Tcl_DoOneEvent > should be called in this case. I think we spotted one case where > Tcl_ServiceAll is not doing its job correctly. There may be others. > > Since the {tcltk} package was working fine with "while > (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev—;", unless there is a clear > performance enhancement with "while (i-- && Tcl_ServiceAll())", it would > perhaps be wise to revert this back. > > Indeed, when I use this on the server side with R 4.3.2: > > library(tcltk) > cmd <- r"( > proc accept {chan addr port} { ;# Make a proc to accept connections > puts "$addr:$port says [gets $chan]" ;# Receive a string > puts $chan goodbye ;# Send a string > close $chan ;# Close the socket (automatically > flushes) > } ;# > socket -server accept 12345 ;# Create a server socket)" > .Tcl(cmd) > .Tcl("vwait myvar") > > It works again as expected. And vwait is known to call Tcl_DoOneEvent. Of > course, in this case, R is blocked and waits for the `myvar` variable on the > Tcl side. Anyway, the problem seems to be really in Tcl_ServiceAll not > catching all Tcl events. > > All the best, > > Philippe > > ..<°}))>< > ) ) ) ) ) > ( ( ( ( (Prof. Philippe Grosjean > ) ) ) ) ) > ( ( ( ( (Numerical Ecology > ) ) ) ) ) Mons University, Belgium > ( ( ( ( ( > .. > >> Le 20 févr. 2024 à 17:13, Ivan Krylov via R-devel a >> écrit : >> >> В Tue, 20 Feb 2024 12:27:35 +0100 >> "webmail.gandi.net" пишет: >> >>> When R process #1 is R 4.2.3, it works as expected (whatever version >>> of R #2). When R process #1 is R 4.3.2, nothing is sent or received >>> through the socket apparently, but no error is issued and process #2 >>> seems to be able to connect to the socket. >> >> The difference is related to the change in >> src/library/tcltk/src/tcltk_unix.c. >> >> In R-4.2.1, the function static void TclSpinLoop(void *data) says: >> >> int max_ev = 100; >> /* Tcl_ServiceAll is not enough here, for reasons that escape me */ >> while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev--; >> >> In R-devel, the function instead says: >> >> int i = R_TCL_SPIN_MAX; >> while (i-- && Tcl_ServiceAll()) >> ; >> >> Manually calling Tcl_DoOneEvent(0) from the debugger at this point >> makes the Tcl code respond to the connection. Tcl_ServiceAll() seems to >> be still not enough. I'll try reading Tcl documentation to investigate >> this further. >> >> -- >> Best regards, >> Ivan >> >> __ >> R-devel@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel > > > [[alternative HTML version deleted]] > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel -- Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Office: A 4.23 Email: pd@cbs.dk Priv: pda...@gmail.com __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
В Wed, 21 Feb 2024 08:01:16 +0100 "webmail.gandi.net" пишет: > Since the {tcltk} package was working fine with "while > (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev—;", unless there is > a clear performance enhancement with "while (i-- && > Tcl_ServiceAll())", it would perhaps be wise to revert this back. I forgot to mention the comment in the new version of the function explaining the switch: >> [Tcl_DoOneEvent(TCL_DONT_WAIT)] <...> causes infinite recursion with >> R handlers that have a re-entrancy guard, when TclSpinLoop is >> invoked from such a handler (seen with Rhttp server) The difference between Tcl_ServiceAll() and Tcl_DoOneEvent() is that the latter calls Tcl_WaitForEvent(). The comments say that it is called for the side effect of queuing the events detected by select(). The function can indeed be observed to access the fileHandlers via the thread-specific data pointer, which contain the file descriptors and the instructions saying what to do with them. Without Tcl_WaitForEvent, the only event sources known to Tcl are RTcl_{setup,check}Proc (which only checks file descriptors owned by R), Display{Setup,Check}Proc (which seems to be owned by Tk), and Timer{Setup,Check}Proc (for which there doesn't seem to be any timers by default). As far as I understand the problem, while the function worker_input_handler() from src/modules/internet/Rhttpd.c is running, TclHandler() might be invoked, causing Tcl_DoOneEvent() to call RTcl_checkProc() and therefore trying to run worker_input_handler() again. The Rhttpd handler prevents this and doesn't clear the condition, which causes the event loop to keep calling it. Is that correct? Are there easy ways to reproduce the problem? -- Best regards, Ivan __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
Thank you, Ivan for this investigation. I inspected the R changes file (https://cran.r-project.org/doc/manuals/r-devel/NEWS.html) and found nothing about this. I should inspect the sources too! It could possibly break other Tcl/Tk related stuff. The doc about Tcl_ServiceAll and Tcl_DoOneEvent is confusing. On one hand, it says when Tcl is used from an external program, Tcl_ServiceAll should be used in its event loop instead of Tcl_DoOneEvent (and the change in the latest R versions goes in that direction). But in the other hand, it is indicated that Tcl_ServiceAll does not always handle all Tcl events and extra Tcl_DoOneEvent should be called in this case. I think we spotted one case where Tcl_ServiceAll is not doing its job correctly. There may be others. Since the {tcltk} package was working fine with "while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev—;", unless there is a clear performance enhancement with "while (i-- && Tcl_ServiceAll())", it would perhaps be wise to revert this back. Indeed, when I use this on the server side with R 4.3.2: library(tcltk) cmd <- r"( proc accept {chan addr port} { ;# Make a proc to accept connections puts "$addr:$port says [gets $chan]" ;# Receive a string puts $chan goodbye ;# Send a string close $chan ;# Close the socket (automatically flushes) } ;# socket -server accept 12345 ;# Create a server socket)" .Tcl(cmd) .Tcl("vwait myvar") It works again as expected. And vwait is known to call Tcl_DoOneEvent. Of course, in this case, R is blocked and waits for the `myvar` variable on the Tcl side. Anyway, the problem seems to be really in Tcl_ServiceAll not catching all Tcl events. All the best, Philippe ..<°}))>< ) ) ) ) ) ( ( ( ( (Prof. Philippe Grosjean ) ) ) ) ) ( ( ( ( (Numerical Ecology ) ) ) ) ) Mons University, Belgium ( ( ( ( ( .. > Le 20 févr. 2024 à 17:13, Ivan Krylov via R-devel a > écrit : > > В Tue, 20 Feb 2024 12:27:35 +0100 > "webmail.gandi.net" пишет: > >> When R process #1 is R 4.2.3, it works as expected (whatever version >> of R #2). When R process #1 is R 4.3.2, nothing is sent or received >> through the socket apparently, but no error is issued and process #2 >> seems to be able to connect to the socket. > > The difference is related to the change in > src/library/tcltk/src/tcltk_unix.c. > > In R-4.2.1, the function static void TclSpinLoop(void *data) says: > >int max_ev = 100; >/* Tcl_ServiceAll is not enough here, for reasons that escape me */ >while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev--; > > In R-devel, the function instead says: > >int i = R_TCL_SPIN_MAX; >while (i-- && Tcl_ServiceAll()) >; > > Manually calling Tcl_DoOneEvent(0) from the debugger at this point > makes the Tcl code respond to the connection. Tcl_ServiceAll() seems to > be still not enough. I'll try reading Tcl documentation to investigate > this further. > > -- > Best regards, > Ivan > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
В Tue, 20 Feb 2024 12:27:35 +0100 "webmail.gandi.net" пишет: > When R process #1 is R 4.2.3, it works as expected (whatever version > of R #2). When R process #1 is R 4.3.2, nothing is sent or received > through the socket apparently, but no error is issued and process #2 > seems to be able to connect to the socket. The difference is related to the change in src/library/tcltk/src/tcltk_unix.c. In R-4.2.1, the function static void TclSpinLoop(void *data) says: int max_ev = 100; /* Tcl_ServiceAll is not enough here, for reasons that escape me */ while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev--; In R-devel, the function instead says: int i = R_TCL_SPIN_MAX; while (i-- && Tcl_ServiceAll()) ; Manually calling Tcl_DoOneEvent(0) from the debugger at this point makes the Tcl code respond to the connection. Tcl_ServiceAll() seems to be still not enough. I'll try reading Tcl documentation to investigate this further. -- Best regards, Ivan __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
On 20 February 2024 at 12:27, webmail.gandi.net wrote: | Dear list, | | It seems that something changed between R 4.2.3 and R 4.3 (tested with 4.3.2) that broke the Tcl socket server. Here is a reproducible example: | | - R process #1 (Tcl socket server): | | library(tcltk) | cmd <- r"( | proc accept {chan addr port} { ;# Make a proc to accept connections |puts "$addr:$port says [gets $chan]" ;# Receive a string |puts $chan goodbye ;# Send a string |close $chan ;# Close the socket (automatically flushes) | } ;# | socket -server accept 12345 ;# Create a server socket)" | .Tcl(cmd) | | - R process #2 (socket client): | | con <- socketConnection(host = "localhost", port = 12345, blocking = FALSE) | writeLines("Hello, world!", con) # Should print something in R #1 stdout | readLines(con) # Should receive "goodbye" | close(con) | | When R process #1 is R 4.2.3, it works as expected (whatever version of R #2). When R process #1 is R 4.3.2, nothing is sent or received through the socket apparently, but no error is issued and process #2 seems to be able to connect to the socket. | | I am stuck with this. Thanks in advance for help. >From a quick check this issue seems to persist in the (current) R-devel 2024-02-20 r85951 too. Dirk | Regards, | | Philippe | | > .Tcl("puts [info patchlevel]") | 8.6.13 | | | > sessionInfo() | R version 4.3.2 (2023-10-31) | Platform: aarch64-apple-darwin20 (64-bit) | Running under: macOS Sonoma 14.2.1 | | Matrix products: default | BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib | LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.11.0 | | locale: | [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 | | time zone: Europe/Brussels | tzcode source: internal | | attached base packages: | [1] tcltk stats graphics grDevices utils datasets methods base | | loaded via a namespace (and not attached): | [1] compiler_4.3.2 tools_4.3.2glue_1.7.0 | __ | R-devel@r-project.org mailing list | https://stat.ethz.ch/mailman/listinfo/r-devel -- dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
Dear list, It seems that something changed between R 4.2.3 and R 4.3 (tested with 4.3.2) that broke the Tcl socket server. Here is a reproducible example: - R process #1 (Tcl socket server): library(tcltk) cmd <- r"( proc accept {chan addr port} { ;# Make a proc to accept connections puts "$addr:$port says [gets $chan]" ;# Receive a string puts $chan goodbye ;# Send a string close $chan ;# Close the socket (automatically flushes) } ;# socket -server accept 12345 ;# Create a server socket)" .Tcl(cmd) - R process #2 (socket client): con <- socketConnection(host = "localhost", port = 12345, blocking = FALSE) writeLines("Hello, world!", con) # Should print something in R #1 stdout readLines(con) # Should receive "goodbye" close(con) When R process #1 is R 4.2.3, it works as expected (whatever version of R #2). When R process #1 is R 4.3.2, nothing is sent or received through the socket apparently, but no error is issued and process #2 seems to be able to connect to the socket. I am stuck with this. Thanks in advance for help. Regards, Philippe > .Tcl("puts [info patchlevel]") 8.6.13 > sessionInfo() R version 4.3.2 (2023-10-31) Platform: aarch64-apple-darwin20 (64-bit) Running under: macOS Sonoma 14.2.1 Matrix products: default BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.11.0 locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 time zone: Europe/Brussels tzcode source: internal attached base packages: [1] tcltk stats graphics grDevices utils datasets methods base loaded via a namespace (and not attached): [1] compiler_4.3.2 tools_4.3.2glue_1.7.0 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel