Re: First two bytes of 'stdout' are lost
On 11Apr2024 14:42, Olivier B. wrote: I am trying to use StringIO to capture stdout, in code that looks like this: import sys from io import StringIO old_stdout = sys.stdout sys.stdout = mystdout = StringIO() print( "patate") mystdout.seek(0) sys.stdout = old_stdout print(mystdout.read()) Well, it is not exactly like this, since this works properly Aye, I just tried that. All good. This code is actually run from C++ using the C Python API. This worked quite well, so the code was right at some point. But now, two things changed: - Now using python 3.11.7 instead of 3.7.12 - Now using only the python limited C API Maybe you should post the code then: the exact Python code and the exact C++ code. And it seems that now, mystdout.read() always misses the first two characters that have been written to stdout. My first ideas was something related to the BOM improperly truncated at some point, but i am manipulating UTF-8, so the bom would be 3 bytes, not 2. I didn't think UTF-8 needed a BOM. Somone will doubtless correct me. However, does the `mystdout.read()` code _know_ you're using UTF-8? I have the vague impression that eg some Windows systems default to UTF-16 of some flavour, possibly _with_ a BOM. I'm suggesting that you rigorously check that the bytes->text bits know what text encoding they're using. If you've left an encoding out anywhere, put it in explicitly. Hopefully someone has a clue on what would have changed in Python for this to stop working compared to python 3.7? None at all, alas. My experience with the Python C API is very limited. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to Add ANSI Color to User Response
On 10Apr2024 23:41, Alan Gauld wrote: Normally, for any kind of fancy terminal work, I'd say use curses. My problem with curses is that it takes over the whole terminal - you need to manage everything from that point on. Great if you want it (eg some full-terminal tool like `top`) but complex overkill for small interactive (or not interactive) commands which are basicly printing lines of text. Which is what many of my scripts are. That said, you don't _have_ to use curses to run the whole terminal. You can use it to just look up the terminal capabilities and use those strings. I haven't tried that for colours, but here's some same code from my `cs.upd` module using curses to look up various cursor motion type things: ... up the top ... try: import curses except ImportError as curses_e: warning("cannot import curses: %s", curses_e) curses = None ... later we cache the available motions ... try: # pylint: disable=no-member curses.setupterm(fd=backend_fd) except TypeError: pass else: for ti_name in ( 'vi', # cursor invisible 'vs', # cursor visible 'cuu1', # cursor up one line 'dl1', # delete one line 'il1', # insert one line 'el', # clear to end of line ): # pylint: disable=no-member s = curses.tigetstr(ti_name) if s is not None: s = s.decode('ascii') self._ti_strs[ti_name] = s ... then a method to access the cache ... def ti_str(self, ti_name): ''' Fetch the terminfo capability string named `ti_name`. Return the string or `None` if not available. ''' return self._ti_strs.get(ti_name, None) ... and even later, use the method ... # emit cursor_up cursor_up = self.ti_str('cuu1') movetxts.append(cursor_up * (to_slot - from_slot)) Generally, when I add ANSI colours I do it via a little module of my own, `cs.ansi_colour`, which you can get from PyPI using `pip`. The two most useful items in it for someone else are probably `colourise` and `colourise_patterns`. Link: https://github.com/cameron-simpson/css/blob/26504f1df55e1bbdef00c3ff7f0cb00b2babdc01/lib/python/cs/ansi_colour.py#L96 I particularly use it to automatically colour log messages on a terminal, example code: https://github.com/cameron-simpson/css/blob/26504f1df55e1bbdef00c3ff7f0cb00b2babdc01/lib/python/cs/logutils.py#L824 -- https://mail.python.org/mailman/listinfo/python-list
Re: First two bytes of 'stdout' are lost
On 4/11/2024 8:42 AM, Olivier B. via Python-list wrote: I am trying to use StringIO to capture stdout, in code that looks like this: import sys from io import StringIO old_stdout = sys.stdout sys.stdout = mystdout = StringIO() print( "patate") mystdout.seek(0) sys.stdout = old_stdout print(mystdout.read()) Well, it is not exactly like this, since this works properly This code is actually run from C++ using the C Python API. This worked quite well, so the code was right at some point. But now, two things changed: - Now using python 3.11.7 instead of 3.7.12 - Now using only the python limited C API And it seems that now, mystdout.read() always misses the first two characters that have been written to stdout. My first ideas was something related to the BOM improperly truncated at some point, but i am manipulating UTF-8, so the bom would be 3 bytes, not 2. I ruled out wrong C++ code to extract the string from the python variable, since running a python print of the content of mystdout in the real stdout also misses the two first characters. Hopefully someone has a clue on what would have changed in Python for this to stop working compared to python 3.7? I've not used the C API, so just for fun I asked ChatGPT about this and it suggested that a flush after writing to StringIO might do it. It suggested using a custom class for this purpose: class MyStringIO(StringIO): def write(self, s): # Override write method to ensure all characters are written correctly super().write(s) self.flush() You would use it like this: sys.stdout = mystdout = MyStringIO() I haven't tested it but it seems reasonable, although I would have naively expected to lose bytes from the end, not the beginning. -- https://mail.python.org/mailman/listinfo/python-list
Re: First two bytes of 'stdout' are lost
Partly answering myself: For some reason, right after mystdout has been created, i now have to do mystdout.seek(0) and this solves the issue. No idea why though.. Le jeu. 11 avr. 2024 à 14:42, Olivier B. a écrit : > > I am trying to use StringIO to capture stdout, in code that looks like this: > > import sys > from io import StringIO > old_stdout = sys.stdout > sys.stdout = mystdout = StringIO() > print( "patate") > mystdout.seek(0) > sys.stdout = old_stdout > print(mystdout.read()) > > Well, it is not exactly like this, since this works properly > > This code is actually run from C++ using the C Python API. > This worked quite well, so the code was right at some point. But now, > two things changed: > - Now using python 3.11.7 instead of 3.7.12 > - Now using only the python limited C API > > And it seems that now, mystdout.read() always misses the first two > characters that have been written to stdout. > > My first ideas was something related to the BOM improperly truncated > at some point, but i am manipulating UTF-8, so the bom would be 3 > bytes, not 2. > > I ruled out wrong C++ code to extract the string from the python > variable, since running a python print of the content of mystdout in > the real stdout also misses the two first characters. > > Hopefully someone has a clue on what would have changed in Python for > this to stop working compared to python 3.7? -- https://mail.python.org/mailman/listinfo/python-list
First two bytes of 'stdout' are lost
I am trying to use StringIO to capture stdout, in code that looks like this: import sys from io import StringIO old_stdout = sys.stdout sys.stdout = mystdout = StringIO() print( "patate") mystdout.seek(0) sys.stdout = old_stdout print(mystdout.read()) Well, it is not exactly like this, since this works properly This code is actually run from C++ using the C Python API. This worked quite well, so the code was right at some point. But now, two things changed: - Now using python 3.11.7 instead of 3.7.12 - Now using only the python limited C API And it seems that now, mystdout.read() always misses the first two characters that have been written to stdout. My first ideas was something related to the BOM improperly truncated at some point, but i am manipulating UTF-8, so the bom would be 3 bytes, not 2. I ruled out wrong C++ code to extract the string from the python variable, since running a python print of the content of mystdout in the real stdout also misses the two first characters. Hopefully someone has a clue on what would have changed in Python for this to stop working compared to python 3.7? -- https://mail.python.org/mailman/listinfo/python-list