[PATCH 1/3] diagnostics: Enable escape sequence processing on windows consoles

[Peter Damianov]

Since windows 10 release v1511, the windows console has had support for VT100
escape sequences. We should try to enable this, and utilize it where possible.

gcc/ChangeLog:
* diagnostic-color.cc (should_colorize): Enable processing of VT100
escape sequences on windows consoles

Signed-off-by: Peter Damianov 
---
 gcc/diagnostic-color.cc | 21 -
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/gcc/diagnostic-color.cc b/gcc/diagnostic-color.cc
index f01a0fc2e37..3af198654af 100644
--- a/gcc/diagnostic-color.cc
+++ b/gcc/diagnostic-color.cc
@@ -213,12 +213,23 @@ should_colorize (void)
  pp_write_text_to_stream() in pretty-print.cc calls fputs() on
  that stream.  However, the code below for non-Windows doesn't seem
  to care about it either...  */
-  HANDLE h;
-  DWORD m;
+  HANDLE handle;
+  DWORD mode;
+  BOOL isconsole = false;
 
-  h = GetStdHandle (STD_ERROR_HANDLE);
-  return (h != INVALID_HANDLE_VALUE) && (h != NULL)
- && GetConsoleMode (h, &m);
+  handle = GetStdHandle (STD_ERROR_HANDLE);
+
+  if ((handle != INVALID_HANDLE_VALUE) && (handle != NULL))
+isconsole = GetConsoleMode (handle, &mode);
+
+  if (isconsole)
+{
+  /* Try to enable processing of VT100 escape sequences */
+  mode |= ENABLE_PROCESSED_OUTPUT | ENABLE_VIRTUAL_TERMINAL_PROCESSING;
+  SetConsoleMode (handle, mode);
+}
+
+  return isconsole;
 #else
   char const *t = getenv ("TERM");
   /* emacs M-x shell sets TERM="dumb".  */
-- 
2.39.2

Re: [PATCH 1/3] diagnostics: Enable escape sequence processing on windows consoles

[LIU Hao]

在 2024-05-09 22:48, Peter Damianov 写道:

-  h = GetStdHandle (STD_ERROR_HANDLE);
-  return (h != INVALID_HANDLE_VALUE) && (h != NULL)
- && GetConsoleMode (h, &m);
+  handle = GetStdHandle (STD_ERROR_HANDLE);
+  isconsole = GetConsoleMode (handle, &mode);


Shouldn't `GetConsoleMode()` be called only when `handle` is valid? I think you may initialize 
`isconsole` to `false`; then only if the handle is valid, should it be set accordingly; and this 
function just returns `isconsole`.


The other two patches look good to me.


--
Best regards,
LIU Hao



OpenPGP_signature.asc
Description: OpenPGP digital signature

[PATCH 1/3] diagnostics: Enable escape sequence processing on windows consoles

[Peter Damianov]

Since windows 10 release v1511, the windows console has had support for VT100
escape sequences. We should try to enable this, and utilize it where possible.

gcc/ChangeLog:
* diagnostic-color.cc (should_colorize): Enable processing of VT100
escape sequences on windows consoles

Signed-off-by: Peter Damianov 
---
 gcc/diagnostic-color.cc | 20 +++-
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/gcc/diagnostic-color.cc b/gcc/diagnostic-color.cc
index f01a0fc2e37..0a4845406f5 100644
--- a/gcc/diagnostic-color.cc
+++ b/gcc/diagnostic-color.cc
@@ -213,12 +213,22 @@ should_colorize (void)
  pp_write_text_to_stream() in pretty-print.cc calls fputs() on
  that stream.  However, the code below for non-Windows doesn't seem
  to care about it either...  */
-  HANDLE h;
-  DWORD m;
+  HANDLE handle;
+  DWORD mode;
+  BOOL isconsole;
 
-  h = GetStdHandle (STD_ERROR_HANDLE);
-  return (h != INVALID_HANDLE_VALUE) && (h != NULL)
- && GetConsoleMode (h, &m);
+  handle = GetStdHandle (STD_ERROR_HANDLE);
+  isconsole = GetConsoleMode (handle, &mode);
+
+  if (isconsole)
+{
+  /* Try to enable processing of VT100 escape sequences */
+  mode |= ENABLE_PROCESSED_OUTPUT | ENABLE_VIRTUAL_TERMINAL_PROCESSING;
+  SetConsoleMode (handle, mode);
+}
+
+  return (handle != INVALID_HANDLE_VALUE) && (handle != NULL)
+ && isconsole;
 #else
   char const *t = getenv ("TERM");
   /* emacs M-x shell sets TERM="dumb".  */
-- 
2.39.2