Re: [O] [Geiser-users] bug#33403: Data length limit in Guile/Geiser/Scheme evaluation
On Sat, Nov 17 2018, Mark H Weaver wrote: [...] >> Ah, system* is a scheme call! So yeah, maybe we could add that call to >> Geiser's guile initialization... i don't really see how that would cause >> any problem elsewhere. > > I think something like this should be done, not only in the Guile > initialization, but ideally in the generic Geiser code that connects to > inferior processes via pseudo-tty. After the pseudo-tty is allocated > but before launching the inferior Scheme process, something like "stty > --file=/dev/pts/N -icanon" should be run. > > However, before doing this, some warnings are in order: > > When in noncanonical mode, the normal processing of ERASE (usually DEL > or Ctrl-H) and KILL (usually Ctrl-U) characters are disabled, and input > characters are delivered to the subprocess immediately as they are > typed, rather than waiting for the newline as normally happens in > canonical mode. > > At least in the case of the Guile REPL, one notable side effect of > running in noncanonical mode is that when a list is entered at the REPL, > the 'read' returns as soon as the final close parenthesis is entered. > Nothing after that is read, not even the usual newline. The final > newline is only read if the reader is not yet sure that it has finished > reading the token, e.g. if a number or symbol is entered. In those > cases, typically any delimiter may be typed to terminate the read, > e.g. space. > > To see this, you can try running Guile from a traditional terminal > program (e.g. xterm or GNOME Terminal), and type: > > (system* "stty" "-icanon") > > and then: > > (display "hello") > > You will see that as soon as you type that close paren, "hello" is > immediately printed, followed by another REPL prompt, all on the same > line. I think this is not an actual problem in Geiser. In comint mode, the stdin of the Guile process doesn't receive any input bytes until higher level elisp functions send them, and that's totally under our control. Repeating that experiment in a Geiser REPL, nothing is printed before a RET (and, in fact, we might not even send the RET to Guile). > > You might also try (use-modules (ice-9 rdelim)) and then: > > (read-line) > > and you'll see that the newline you type at the end of that line is read > by 'read-line', which then immediately returns the empty string. The > input that you wish for 'read-line' to see must be typed on the same > line, immediately after the close parenthesis. Again, a comint/geiser REPL doesn't have this problem. > So, it might be that Geiser needs to be adjusted somewhat to deal with > these differences. Seems we're lucky enough to be already adjusted :) > Finally, you might consider the possibility that 'stty' might not be > available in PATH, or fails for some reason, and ideally this case > should be handled as well. Yes, that's a real concern. We should at least provide some kind of warning. > It might be simpler to always use REPL servers over a socket, than to > deal with these headaches, although I don't know if that will be an > option for the other Scheme implementations. Not for all of them. For Guile it's doable, but definitely not "simpler", it requires some work and solving some unrelated corner cases; but it might be worth the effort, because it's a cleaner interaction mode (for instance, behaves much better in the presence of multiple threads). Cheers, jao -- Beware of the stories you read or tell; subtly, at night, beneath the waters of consciousness, they are altering your world. -Ben Okri, poet and novelist
Re: [O] [Geiser-users] bug#33403: Data length limit in Guile/Geiser/Scheme evaluation
A few more notes: I wrote earlier: > However, before doing this, some warnings are in order: > > When in noncanonical mode, the normal processing of ERASE (usually DEL > or Ctrl-H) and KILL (usually Ctrl-U) characters are disabled, Also the handling of Ctrl-D appears to be disabled in noncanonical mode on my system, although this wasn't clear to me from the docs. > At least in the case of the Guile REPL, one notable side effect of > running in noncanonical mode is that when a list is entered at the REPL, > the 'read' returns as soon as the final close parenthesis is entered. > Nothing after that is read, not even the usual newline. There's an additional wrinkle here: after 'read' returns, Guile tries to read optional whitespace followed by a newline, but only if it's immediately available. See 'flush-to-newline' at the end of module/system/repl/repl.scm in Guile. So, unfortunately there's a race condition here, but typically if you send the newline immediately after the final character of input, it is likely that the newline will be consumed by the REPL reader and not by the code that is subsequently run. Finally, I should note that I consider this race condition suboptimal, and will likely change how Guile behaves in the future, so please don't rely on the behavior I have described above. I will likely change Guile's REPL reader to wait for the final newline in all cases. Mark
Re: [O] [Geiser-users] bug#33403: Data length limit in Guile/Geiser/Scheme evaluation
Hello all, "Jose A. Ortega Ruiz" writes: > On Fri, Nov 16 2018, Neil Jerram wrote: > >> Neil Jerram writes: >> >>> Mark H Weaver writes: >>> This is a documented limitation in Linux's terminal handling when in canonical mode. See the termios(3) man page, which includes this text: Canonical and noncanonical mode The setting of the ICANON canon flag in c_lflag determines whether the terminal is operating in canonical mode (ICANON set) or noncanonical mode (ICANON unset). By default, ICANON is set. >>> [...] * The maximum line length is 4096 chars (including the terminating newline character); lines longer than 4096 chars are truncated. After 4095 characters, input processing (e.g., ISIG and ECHO* processing) continues, but any input data after 4095 characters up to (but not including) any terminating newline is discarded. This ensures that the terminal can always receive more input until at least one line can be read. Note that last item above. >>> >>> Awesome; thank you Mark. >>> >>> So possibly this limit can be removed, in my Org/Geiser context, by >>> evaluating (system* "stty" "-icanon") when initializing the Geiser-Guile >>> connection. I'll try that. Will the terminal that that 'stty' sees be >>> the same as Guile's stdin? >>> >>> Jao, if that works, I wonder if it should be the default for Geiser? It >>> appears to me that Geiser shouldn't ever need the features of canonical >>> mode. Is that right? >>> >>> Anyway, I'll see first if the stty call is effective. >> >> Yes, with this in my ~/.guile-geiser - >> >> (system* "stty" "-icanon") >> >> - I can do evaluations past the 4K line length limit, and the Org-driven >> problem that I first reported [1] has disappeared. > > Ah, system* is a scheme call! So yeah, maybe we could add that call to > Geiser's guile initialization... i don't really see how that would cause > any problem elsewhere. I think something like this should be done, not only in the Guile initialization, but ideally in the generic Geiser code that connects to inferior processes via pseudo-tty. After the pseudo-tty is allocated but before launching the inferior Scheme process, something like "stty --file=/dev/pts/N -icanon" should be run. However, before doing this, some warnings are in order: When in noncanonical mode, the normal processing of ERASE (usually DEL or Ctrl-H) and KILL (usually Ctrl-U) characters are disabled, and input characters are delivered to the subprocess immediately as they are typed, rather than waiting for the newline as normally happens in canonical mode. At least in the case of the Guile REPL, one notable side effect of running in noncanonical mode is that when a list is entered at the REPL, the 'read' returns as soon as the final close parenthesis is entered. Nothing after that is read, not even the usual newline. The final newline is only read if the reader is not yet sure that it has finished reading the token, e.g. if a number or symbol is entered. In those cases, typically any delimiter may be typed to terminate the read, e.g. space. To see this, you can try running Guile from a traditional terminal program (e.g. xterm or GNOME Terminal), and type: (system* "stty" "-icanon") and then: (display "hello") You will see that as soon as you type that close paren, "hello" is immediately printed, followed by another REPL prompt, all on the same line. You might also try (use-modules (ice-9 rdelim)) and then: (read-line) and you'll see that the newline you type at the end of that line is read by 'read-line', which then immediately returns the empty string. The input that you wish for 'read-line' to see must be typed on the same line, immediately after the close parenthesis. So, it might be that Geiser needs to be adjusted somewhat to deal with these differences. Finally, you might consider the possibility that 'stty' might not be available in PATH, or fails for some reason, and ideally this case should be handled as well. It might be simpler to always use REPL servers over a socket, than to deal with these headaches, although I don't know if that will be an option for the other Scheme implementations. Regards, Mark
Re: [O] [Geiser-users] bug#33403: Data length limit in Guile/Geiser/Scheme evaluation
On Fri, Nov 16 2018, Neil Jerram wrote: > Neil Jerram writes: > >> Mark H Weaver writes: >> >>> This is a documented limitation in Linux's terminal handling when in >>> canonical mode. See the termios(3) man page, which includes this text: >>> >>>Canonical and noncanonical mode >>> >>>The setting of the ICANON canon flag in c_lflag determines >>>whether the terminal is operating in canonical mode (ICANON set) >>>or noncanonical mode (ICANON unset). By default, ICANON is set. >> [...] >>>* The maximum line length is 4096 chars (including the >>> terminating newline character); lines longer than 4096 chars >>> are truncated. After 4095 characters, input processing (e.g., >>> ISIG and ECHO* processing) continues, but any input data after >>> 4095 characters up to (but not including) any terminating >>> newline is discarded. This ensures that the terminal can >>> always receive more input until at least one line can be read. >>> >>> Note that last item above. >> >> Awesome; thank you Mark. >> >> So possibly this limit can be removed, in my Org/Geiser context, by >> evaluating (system* "stty" "-icanon") when initializing the Geiser-Guile >> connection. I'll try that. Will the terminal that that 'stty' sees be >> the same as Guile's stdin? >> >> Jao, if that works, I wonder if it should be the default for Geiser? It >> appears to me that Geiser shouldn't ever need the features of canonical >> mode. Is that right? >> >> Anyway, I'll see first if the stty call is effective. > > Yes, with this in my ~/.guile-geiser - > > (system* "stty" "-icanon") > > - I can do evaluations past the 4K line length limit, and the Org-driven > problem that I first reported [1] has disappeared. Ah, system* is a scheme call! So yeah, maybe we could add that call to Geiser's guile initialization... i don't really see how that would cause any problem elsewhere. > Thanks to Nicolas, Jao and Mark for your help in understanding this. And thanks to Nicolas, Mark and you for yours :) Cheers, jao -- The vast majority of human beings dislike and even dread all notions with which they are not familiar. Hence it comes about that at their first appearance innovators have always been derided as fools and madmen. -Aldous Huxley, novelist (1894-1963)
