Re: [PATCH v1] Add -ftime-report-wall

[David Malcolm]

On Thu, 2024-10-03 at 11:15 -0700, Andi Kleen wrote:
> > The only consumer I know of for the JSON time report data is in the
> > integration tests I wrote for -fanalyzer, which assumes that all
> > fields
> > are present when printing, and then goes on to use the "user" times
> > for
> > summarizing; see this commit FWIW:
> > https://github.com/davidmalcolm/gcc-analyzer-integration-tests/commit/5420ce968e6eae886e61486555b54fd460e0d35f
> 
> It seems to be broken even without my changes:
> 
> 
> % ./gcc/cc1plus -ftime-report -fdiagnostics-format=sarif-file
> ../tsrc/tramp3d-v4.i
> cc1plus: internal compiler error: Segmentation fault

Oops, thanks; I'm tracking this as
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=116978
and working on a fix.

Dave

Re: [PATCH v1] Add -ftime-report-wall

[David Malcolm]

On Thu, 2024-10-03 at 00:37 -0700, Andi Kleen wrote:
> > Note that if the user requests SARIF output e.g. with
> >   -fdiagnostics-format=sarif-stderr
> > then any timevar data from -ftime-report is written in JSON form as
> > part of the SARIF, rather than in text form to stderr (see
> > 75d623946d4b6ea80a777b789b116d4b4a2298dc).
> > 
> > I see that the proposed patch leaves the user and sys stats as
> > zero,
> > and conditionalizes what's printed for text output as part of
> > timer::print.  Should it also do something similar in
> > make_json_for_timevar_time_def for the json output, and not add the
> > properties for "user" and "sys" if the data hasn't been gathered?
> 
> > Hope I'm reading the patch correctly.
> 
> Yes that's right.

Thanks.

> 
> I mainly adjusted the human output for cosmetic reasons.
> 
> For machine readable i guess it is better to have a stable schema 
> and not skip fields to avoid pain for parsers. So I left it alone.

The only consumer I know of for the JSON time report data is in the
integration tests I wrote for -fanalyzer, which assumes that all fields
are present when printing, and then goes on to use the "user" times for
summarizing; see this commit FWIW:
https://github.com/davidmalcolm/gcc-analyzer-integration-tests/commit/5420ce968e6eae886e61486555b54fd460e0d35f

I'm not planning to use -ftime-report-wall, but given that my
summarization code is using the "user" times I think I'd prefer it the
property wasn't present rather than contained a bogus value that I
might mistakenly use.  The existing docs do say: "The precise format of
this JSON data is subject to change":
https://gcc.gnu.org/onlinedocs/gcc/Developer-Options.html#index-ftime-report
and there isn't a formal schema for this written down anywhere.

Hope this is constructive
Dave

Re: [PATCH v1] Add -ftime-report-wall

[David Malcolm]

On Wed, 2024-10-02 at 14:14 -0700, Andi Kleen wrote:
> From: Andi Kleen 
> 
> Time vars normally use times(2) to get the user/sys/wall time, which
> is always a
> system call. I don't think the system time is very useful because
> most overhead
> is in user time. If we only use the wall (or monotonic) time modern
> OS have an
> optimized path to get it directly from a CPU instruction like RDTSC
> without system call, which is much faster.
> 
> Comparing the overhead with tramp3d:
> 
>   ./gcc/cc1plus -quiet  ../tsrc/tramp3d-v4.i ran
>     1.03 ± 0.00 times faster than ./gcc/cc1plus -quiet -ftime-report-
> wall ../tsrc/tramp3d-v4.i
>     1.18 ± 0.00 times faster than ./gcc/cc1plus -quiet -ftime-report
> ../tsrc/tramp3d-v4.i
> 
> -ftime-report costs 18% (excluding the output), while -ftime-report-
> wall
> only costs 3%, so is nearly free. So it would be feasible for some
> build
> system to always enable it and break down the build time into passes.
> 
> The drawback is that if there is context switching with other
> programs
> the time will be overestimated, however for the common case that the
> system is not oversubscribed it is more accurate because each
> measurement has less overhead.
> 
> Add a -ftime-report-wall option. It actually uses the POSIX monotonic
> time,
> so strictly it's not wall clock, but it's still a reasonable name.
> 
> Bootstrapped on x86_64-linux with full test suite run.

Note that if the user requests SARIF output e.g. with
  -fdiagnostics-format=sarif-stderr
then any timevar data from -ftime-report is written in JSON form as
part of the SARIF, rather than in text form to stderr (see
75d623946d4b6ea80a777b789b116d4b4a2298dc).

I see that the proposed patch leaves the user and sys stats as zero,
and conditionalizes what's printed for text output as part of
timer::print.  Should it also do something similar in
make_json_for_timevar_time_def for the json output, and not add the
properties for "user" and "sys" if the data hasn't been gathered?

Hope I'm reading the patch correctly.

Thanks
Dave

> 
> gcc/ChangeLog:
> 
>   * common.opt (ftime-report-wall): Add.
>   * common.opt.urls: Regenerate.
>   * doc/invoke.texi: (ftime-report-wall): Document
>   * gcc.cc (try_generate_repro): Check for -ftime-report-wall.
>   * timevar.cc (get_time): Use clock_gettime if enabled.
>   (timer::print): Print only wall time for time_report_wall.
>   * toplev.cc (toplev::start_timevars): Check for
> time_report_wall.
> 
> gcc/testsuite/ChangeLog:
> 
>   * g++.dg/ext/timevar3.C: New test.
> ---
>  gcc/common.opt  |  4 
>  gcc/common.opt.urls |  3 +++
>  gcc/doc/invoke.texi |  7 +++
>  gcc/gcc.cc  |  3 ++-
>  gcc/testsuite/g++.dg/ext/timevar3.C | 14 +
>  gcc/timevar.cc  | 31 +++
> --
>  gcc/toplev.cc   |  3 ++-
>  7 files changed, 57 insertions(+), 8 deletions(-)
>  create mode 100644 gcc/testsuite/g++.dg/ext/timevar3.C
> 
> diff --git a/gcc/common.opt b/gcc/common.opt
> index d270e524ff45..e9fb15e28d80 100644
> --- a/gcc/common.opt
> +++ b/gcc/common.opt
> @@ -3010,6 +3010,10 @@ ftime-report
>  Common Var(time_report)
>  Report the time taken by each compiler pass.
>  
> +ftime-report-wall
> +Common Var(time_report_wall)
> +Report the wall time taken by each compiler.
> +
>  ftime-report-details
>  Common Var(time_report_details)
>  Record times taken by sub-phases separately.
> diff --git a/gcc/common.opt.urls b/gcc/common.opt.urls
> index e31736cd9945..6e79a8f9390b 100644
> --- a/gcc/common.opt.urls
> +++ b/gcc/common.opt.urls
> @@ -1378,6 +1378,9 @@ UrlSuffix(gcc/Optimize-Options.html#index-
> fthread-jumps)
>  ftime-report
>  UrlSuffix(gcc/Developer-Options.html#index-ftime-report)
>  
> +ftime-report-wall
> +UrlSuffix(gcc/Developer-Options.html#index-ftime-report-wall)
> +
>  ftime-report-details
>  UrlSuffix(gcc/Developer-Options.html#index-ftime-report-details)
>  
> diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> index e199522f62c7..80cb355f5d79 100644
> --- a/gcc/doc/invoke.texi
> +++ b/gcc/doc/invoke.texi
> @@ -784,6 +784,7 @@ Objective-C and Objective-C++ Dialects}.
>  -frandom-seed=@var{string}  -fsched-verbose=@var{n}
>  -fsel-sched-verbose  -fsel-sched-dump-cfg  -fsel-sched-pipelining-
> verbose
>  -fstats  -fstack-usage  -ftime-report  -ftime-report-details
> +-ftime-report-wall
>  -fvar-tracking-assignments-toggle  -gtoggle
>  -print-file-name=@var{library}  -print-libgcc-file-name
>  -print-multi-directory  -print-multi-lib  -print-multi-os-directory
> @@ -21026,6 +21027,12 @@ slightly different place within the
> compiler.
>  @item -ftime-report-details
>  Record the time consumed by infrastructure parts separately for each
> pass.
>  
> +@opindex ftime-report-wall
> +@item -ftime-report-wall
> +Report statistics about compiler pass time consumpion, but only
> using wall
> +time.  This

Re: [PATCH] opts: Fix up regenerate-opt-urls dependencies

[David Malcolm]

On Sat, 2024-09-21 at 19:43 +0200, Jakub Jelinek wrote:
> Hi!
> 
> It seems that we currently require
> 1) enabling at least c,c++,fortran,d in --enable-languages
> 2) first doing make html
> before one can successfully regenerate-opt-urls, otherwise without 2)
> one gets
> make regenerate-opt-urls
> make: *** No rule to make target
> '/home/jakub/src/gcc/obj12x/gcc/HTML/gcc-15.0.0/gcc/Option-
> Index.html', needed by 'regenerate-opt-urls'.  Stop.
> or say if not configuring d after make html one still gets
> make regenerate-opt-urls
> make: *** No rule to make target
> '/home/jakub/src/gcc/obj12x/gcc/HTML/gcc-15.0.0/gdc/Option-
> Index.html', needed by 'regenerate-opt-urls'.  Stop.
> 
> Now, I believe neither 1) nor 2) is really necessary.
> The regenerate-opt-urls goal has dependency on 3 Option-Index.html
> files,
> but those files don't have dependencies how to generate them.
> make html has dependency on $(HTMLS_BUILD) which adds
> $(build_htmldir)/gcc/index.html and lang.html among other things,
> where
> the former actually builds not just index.html but also Option-
> Index.html
> and tons of other files, and lang.html is filled in by configure
> depending
> on configured languages, so sometimes will include gfortran.html and
> sometimes d.html.
> 
> The following patch adds dependencies of the Option-Index.html on
> their
> corresponding index.html files and that is all that seems to be
> needed,
> make regenerate-opt-urls then works even without prior make html and
> even if just a subset of c/c++, fortran and d is enabled.
> 
> Ok for trunk?

Thanks, LGTM.

Dave

> 
> 2024-09-21  Jakub Jelinek  
> 
>   * Makefile.in ($(OPT_URLS_HTML_DEPS)): Add dependencies of
> the
>   Option-Index.html files on the corresponding index.html
> files.
>   Don't mention the requirement that all languages that have
> their own
>   HTML manuals to be enabled.
> 
> --- gcc/Makefile.in.jj2024-09-18 15:03:25.979207519 +0200
> +++ gcc/Makefile.in   2024-09-21 19:26:31.160949856 +0200
> @@ -3640,12 +3640,12 @@ $(build_htmldir)/gccinstall/index.html:
>   $(SHELL) $(srcdir)/doc/install.texi2html
>  
>  # Regenerate the .opt.urls files from the generated html, and from
> the .opt
> -# files.  Doing so requires all languages that have their own HTML
> manuals
> -# to be enabled.
> +# files.
>  .PHONY: regenerate-opt-urls
>  OPT_URLS_HTML_DEPS = $(build_htmldir)/gcc/Option-Index.html \
>   $(build_htmldir)/gdc/Option-Index.html \
>   $(build_htmldir)/gfortran/Option-Index.html
> +$(OPT_URLS_HTML_DEPS): %/Option-Index.html: %/index.html
>  
>  regenerate-opt-urls: $(srcdir)/regenerate-opt-urls.py
> $(OPT_URLS_HTML_DEPS)
>   $(srcdir)/regenerate-opt-urls.py $(build_htmldir) $(shell
> dirname $(srcdir))
> 
>   Jakub
> 

[pushed: r15-3976] diagnostics: isolate diagnostic_context with interface classes [PR116613]

[David Malcolm]

g_newline): Likewise.
(selftest::test_fixit_insert_containing_newline_2): Likewise.
(selftest::test_fixit_replace_containing_newline): Likewise.
(selftest::test_fixit_deletion_affecting_newline): Likewise.
(selftest::test_tab_expansion): Likewise.
(selftest::test_escaping_bytes_1): Likewise.
(selftest::test_escaping_bytes_2): Likewise.
(selftest::test_line_numbers_multiline_range): Likewise.
* diagnostic.cc
(diagnostic_column_policy::diagnostic_column_policy): New.
(diagnostic_context::converted_column): Convert to...
(diagnostic_column_policy::converted_column): ...this.
(diagnostic_context::get_location_text): Convert to...
(diagnostic_column_policy::get_location_text): ...this, adding
"show_column" param.
(diagnostic_location_print_policy::diagnostic_location_print_policy):
New ctors.
(default_diagnostic_start_span_fn): Convert param from
diagnostic_context * to const diagnostic_location_print_policy &.
Add "pp" param.
(selftest::assert_location_text): Update for above changes.
(selftest::test_diagnostic_get_location_text): Rename to...
(selftest::test_get_location_text): ...this.
(selftest::c_diagnostic_cc_tests): Update for renaming.
* diagnostic.h (class diagnostic_location_print_policy): New
forward decl.
(class diagnostic_source_print_policy): New forward decl.
(diagnostic_start_span_fn): Convert first param from
diagnostic_context * to const diagnostic_location_print_policy &
and add pretty_printer * param.
(class diagnostic_column_policy): New.
(class diagnostic_location_print_policy): New.
(class diagnostic_source_print_policy): New.
(class diagnostic_context): Add friend class
diagnostic_source_print_policy.
(diagnostic_context::converted_column): Drop decl in favor of
diagnostic_column_policy::converted_column.
(diagnostic_context::get_location_text): Drop decl in favor of
diagnostic_column_policy::get_location_text.
(diagnostic_context::show_locus): Drop decl in favor of
diagnostic_source_print_policy::print.
(default_diagnostic_start_span_fn): Update for change to
diagnostic_start_span_fn.
* gcc-rich-location.h (class diagnostic_source_print_policy): New
forward decl.
(gcc_rich_location::add_location_if_nearby): Convert first param
from diagnostic_context to diagnostic_source_print_policy.  Add
overload taking diagnostic_context.
* selftest-diagnostic.cc
(selftest::test_diagnostic_context::test_diagnostic_context): Turn
off colorization.
(selftest::test_diagnostic_context::start_span_cb): Update for
change to callback type.
(test_diagnostic_context::test_show_locus): New.
* selftest-diagnostic.h
(selftest::test_diagnostic_context::start_span_cb): Update for
change to callback type.
(test_diagnostic_context::test_show_locus): New decl.

gcc/fortran/ChangeLog:
PR other/116613
* error.cc (gfc_diagnostic_build_locus_prefix): Convert first
param from diagnostic_context * to
const diagnostic_location_print_policy &.  Add colorize param.
Likewise for the "two expanded_locations" overload.
(gfc_diagnostic_text_starter): Update for above changes.
(gfc_diagnostic_start_span): Update for change to callback type.

gcc/testsuite/ChangeLog:
PR other/116613
* gcc.dg/plugin/diagnostic_group_plugin.c
(test_diagnostic_start_span_fn): Update for change to callback
type.

Signed-off-by: David Malcolm 
---
 gcc/analyzer/program-point.cc |   4 +-
 gcc/diagnostic-format-json.cc |   3 +-
 gcc/diagnostic-format-sarif.cc|   3 +-
 gcc/diagnostic-format-text.cc |   9 +-
 gcc/diagnostic-format-text.h  |   9 +
 gcc/diagnostic-path.cc| 130 ++-
 gcc/diagnostic-show-locus.cc  | 788 ++
 gcc/diagnostic.cc |  61 +-
 gcc/diagnostic.h  | 121 ++-
 gcc/fortran/error.cc  |  41 +-
 gcc/gcc-rich-location.h   |   9 +-
 gcc/selftest-diagnostic.cc|  19 +-
 gcc/selftest-diagnostic.h |   6 +-
 .../gcc.dg/plugin/diagnostic_group_plugin.c   |   9 +-
 14 files changed, 765 insertions(+), 447 deletions(-)

diff --git a/gcc/analyzer/program-point.cc b/gcc/analyzer/program-point.cc
index cfb59e1ee4ea..d1535703e0e9 100644
--- a/gcc/analyzer/program-point.cc
+++ b/gcc/analyzer/program-point.cc
@@ -277,7 +277,9 @@ function_point::print_source_line (pretty_printer *pp) const
   // TODO: monospace font
   de

[pushed: r15-3977] diagnostics: require callers of diagnostic_show_locus to be explicit about the printer [PR116613]

[David Malcolm]

As work towards supporting multiple diagnostic outputs (where each
output has its own pretty_printer), update diagnostic_show_locus
so that the pretty_printer must always be explicitly passed in.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3977-ge7a8fbe2fed83b.

gcc/c-family/ChangeLog:
PR other/116613
* c-format.cc (selftest::test_type_mismatch_range_labels):
Explicitly pass in dc.m_printer to diagnostic_show_locus.

gcc/ChangeLog:
PR other/116613
* diagnostic-show-locus.cc (diagnostic_context::maybe_show_locus):
Convert param "pp" from * to &.  Drop logic for using the
context's m_printer when the param is null.
* diagnostic.h (diagnostic_context::maybe_show_locus): Convert
param "pp" from * to &.
(diagnostic_show_locus): Drop default "nullptr" value for pp
param.  Assert that it and context are nonnull.  Pass pp by
reference to maybe_show_locus.

gcc/testsuite/ChangeLog:
PR other/116613
* gcc.dg/plugin/expensive_selftests_plugin.c (test_richloc):
Explicitly pass in dc.m_printer to diagnostic_show_locus.

Signed-off-by: David Malcolm 
---
 gcc/c-family/c-format.cc | 2 +-
 gcc/diagnostic-show-locus.cc | 8 ++--
 gcc/diagnostic.h | 8 +---
 gcc/testsuite/gcc.dg/plugin/expensive_selftests_plugin.c | 2 +-
 4 files changed, 9 insertions(+), 11 deletions(-)

diff --git a/gcc/c-family/c-format.cc b/gcc/c-family/c-format.cc
index 614b43266a31..f4a65a5019c3 100644
--- a/gcc/c-family/c-format.cc
+++ b/gcc/c-family/c-format.cc
@@ -5578,7 +5578,7 @@ test_type_mismatch_range_labels ()
   richloc.add_range (param, SHOW_RANGE_WITHOUT_CARET, ¶m_label);
 
   test_diagnostic_context dc;
-  diagnostic_show_locus (&dc, &richloc, DK_ERROR);
+  diagnostic_show_locus (&dc, &richloc, DK_ERROR, dc.m_printer);
   if (c_dialect_cxx ())
 /* "char*", without a space.  */
 ASSERT_STREQ ("   printf (\"msg: %i\\n\", msg);\n"
diff --git a/gcc/diagnostic-show-locus.cc b/gcc/diagnostic-show-locus.cc
index a1d66cf493d6..b575dc51a78c 100644
--- a/gcc/diagnostic-show-locus.cc
+++ b/gcc/diagnostic-show-locus.cc
@@ -3265,7 +3265,7 @@ add_location_if_nearby (const diagnostic_context &dc,
 void
 diagnostic_context::maybe_show_locus (const rich_location &richloc,
  diagnostic_t diagnostic_kind,
- pretty_printer *pp,
+ pretty_printer &pp,
  diagnostic_source_effect_info *effects)
 {
   const location_t loc = richloc.get_loc ();
@@ -3287,12 +3287,8 @@ diagnostic_context::maybe_show_locus (const 
rich_location &richloc,
 
   m_last_location = loc;
 
-  if (!pp)
-pp = m_printer;
-  gcc_assert (pp);
-
   diagnostic_source_print_policy source_policy (*this);
-  source_policy.print (*pp, richloc, diagnostic_kind, effects);
+  source_policy.print (pp, richloc, diagnostic_kind, effects);
 }
 
 diagnostic_source_print_policy::
diff --git a/gcc/diagnostic.h b/gcc/diagnostic.h
index 54b7f307f849..447e3b183d92 100644
--- a/gcc/diagnostic.h
+++ b/gcc/diagnostic.h
@@ -529,7 +529,7 @@ public:
 
   void maybe_show_locus (const rich_location &richloc,
 diagnostic_t diagnostic_kind,
-pretty_printer *pp,
+pretty_printer &pp,
 diagnostic_source_effect_info *effect_info);
 
   void emit_diagram (const diagnostic_diagram &diagram);
@@ -970,11 +970,13 @@ inline void
 diagnostic_show_locus (diagnostic_context *context,
   rich_location *richloc,
   diagnostic_t diagnostic_kind,
-  pretty_printer *pp = nullptr,
+  pretty_printer *pp,
   diagnostic_source_effect_info *effect_info = nullptr)
 {
+  gcc_assert (context);
   gcc_assert (richloc);
-  context->maybe_show_locus (*richloc, diagnostic_kind, pp, effect_info);
+  gcc_assert (pp);
+  context->maybe_show_locus (*richloc, diagnostic_kind, *pp, effect_info);
 }
 
 /* Because we read source files a second time after the frontend did it the
diff --git a/gcc/testsuite/gcc.dg/plugin/expensive_selftests_plugin.c 
b/gcc/testsuite/gcc.dg/plugin/expensive_selftests_plugin.c
index 3c534005a419..554dad6fa35a 100644
--- a/gcc/testsuite/gcc.dg/plugin/expensive_selftests_plugin.c
+++ b/gcc/testsuite/gcc.dg/plugin/expensive_selftests_plugin.c
@@ -48,7 +48,7 @@ test_richloc (rich_location *richloc)
 {
   /* Run the diagnostic and fix-it printing code.  */
   test_diagnostic_context dc;
-  diagnostic_show_locus (&dc, richloc, DK_ERROR);
+  diagnostic_show_locus (&dc, richloc, DK_ERROR, dc.m_printer);
 
   /* Generate a diff.  */
   edit_context ec (global_dc->get_file_cache ());
-- 
2.26.3

[pushed: r15-3974] diagnostics: use "%e" to avoid intermediate strings [PR116613]

[David Malcolm]

Various diagnostics build an intermediate string, potentially with
colorization, and then use this in a diagnostic message.

This won't work if we have multiple diagnostic sinks, where some might
be colorized and some not.

This patch reworks such places using "%e" and pp_element subclasses, so
that any colorization happens within report_diagnostic's call to
pp_format.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3974-g3d3d20ccd83659.

gcc/analyzer/ChangeLog:
PR other/116613
* kf-analyzer.cc: Include "pretty-print-markup.h".
(kf_analyzer_dump_escaped::impl_call_pre): Defer colorization
choices by eliminating the construction of a intermediate string,
replacing it with a new pp_element subclass via "%e".

gcc/ChangeLog:
PR other/116613
* attribs.cc: Include "pretty-print-markup.h".
(decls_mismatched_attributes): Defer colorization choices by
replacing printing to a pretty_printer * param with appending
to a vec of strings.
(maybe_diag_alias_attributes): As above, replacing pretty_printer
with usage of pp_markup::comma_separated_quoted_strings and "%e"
in two places.
* attribs.h (decls_mismatched_attributes): Update decl.
* gimple-ssa-warn-access.cc: Include "pretty-print-markup.h".
(pass_waccess::maybe_warn_memmodel): Defer colorization choices by
replacing printing to a pretty_printer * param with use of
pp_markup::comma_separated_quoted_strings and "%e".
(pass_waccess::maybe_warn_memmodel): Likewise, replacing printing
to a temporary buffer.
* pretty-print-markup.h
(class pp_markup::comma_separated_quoted_strings): New.
* pretty-print.cc
(pp_markup::comma_separated_quoted_strings::add_to_phase_2): New.
(selftest::test_pp_printf_within_pp_element): New.
(selftest::test_comma_separated_quoted_strings): New.
(selftest::pretty_print_cc_tests): Call the new tests.

gcc/cp/ChangeLog:
PR other/116613
* pt.cc: Include "pretty-print-markup.h".
(warn_spec_missing_attributes): Defer colorization choices by
replacing printing to a pretty_printer * param with appending
to a vec of strings.  Replace pretty_printer with usage of
pp_markup::comma_separated_quoted_strings and "%e".

gcc/testsuite/ChangeLog:
PR other/116613
* c-c++-common/analyzer/escaping-1.c: Update expected results to
remove type information from C++ results.  Previously we were
using %qD with default_tree_printer, which used
lang_hooks.decl_printable_name, whereas now we're using %qD with
a clone of the cxx_pretty_printer.

Signed-off-by: David Malcolm 
---
 gcc/analyzer/kf-analyzer.cc   | 42 ++---
 gcc/attribs.cc| 33 ---
 gcc/attribs.h |  2 +-
 gcc/cp/pt.cc  | 18 ++--
 gcc/gimple-ssa-warn-access.cc | 21 ++---
 gcc/pretty-print-markup.h | 17 
 gcc/pretty-print.cc   | 92 +++
 .../c-c++-common/analyzer/escaping-1.c|  9 +-
 8 files changed, 179 insertions(+), 55 deletions(-)

diff --git a/gcc/analyzer/kf-analyzer.cc b/gcc/analyzer/kf-analyzer.cc
index 26c2e41da6ff..da49baa5bff1 100644
--- a/gcc/analyzer/kf-analyzer.cc
+++ b/gcc/analyzer/kf-analyzer.cc
@@ -36,6 +36,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "analyzer/pending-diagnostic.h"
 #include "analyzer/call-details.h"
 #include "make-unique.h"
+#include "pretty-print-markup.h"
 
 #if ENABLE_ANALYZER
 
@@ -176,23 +177,40 @@ public:
probably most user-friendly.  */
 escaped_decls.qsort (cmp_decls_ptr_ptr);
 
-pretty_printer pp;
-pp_format_decoder (&pp) = default_tree_printer;
-pp_show_color (&pp) = pp_show_color (global_dc->m_printer);
-bool first = true;
-for (auto iter : escaped_decls)
+class escaped_list_element : public pp_element
+{
+public:
+  escaped_list_element (auto_vec &escaped_decls)
+  : m_escaped_decls (escaped_decls)
   {
-   if (first)
- first = false;
-   else
- pp_string (&pp, ", ");
-   pp_printf (&pp, "%qD", iter);
   }
+
+  void add_to_phase_2 (pp_markup::context &ctxt) final override
+  {
+   /* We can't call pp_printf directly on ctxt.m_pp from within
+  formatting.  As a workaround, work with a clone of the pp.  */
+   std::unique_ptr pp (ctxt.m_pp.clone ());
+   bool first = true;
+   for (auto iter : m_escaped_decls)
+ {
+   if (first)
+ first = false;
+

[pushed: r15-3973] diagnostics: add "dump" to pretty_printer and output_buffer

[David Malcolm]

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3973-g4c7a58ac2617e2.

gcc/ChangeLog:
* pretty-print.cc (output_buffer::dump): New.
(pretty_printer::dump): New.
* pretty-print.h (output_buffer::dump): New decls.
(pretty_printer::dump): New decls.

Signed-off-by: David Malcolm 
---
 gcc/pretty-print.cc | 23 +++
 gcc/pretty-print.h  |  6 ++
 2 files changed, 29 insertions(+)

diff --git a/gcc/pretty-print.cc b/gcc/pretty-print.cc
index 998e06e155f7..2b865212ac55 100644
--- a/gcc/pretty-print.cc
+++ b/gcc/pretty-print.cc
@@ -790,6 +790,21 @@ output_buffer::pop_formatted_chunks ()
   obstack_free (&m_chunk_obstack, old_top);
 }
 
+/* Dump state of this output_buffer to OUT, for debugging.  */
+
+void
+output_buffer::dump (FILE *out) const
+{
+  int depth = 0;
+  for (pp_formatted_chunks *iter = m_cur_formatted_chunks;
+   iter;
+   iter = iter->m_prev, depth++)
+{
+  fprintf (out, "pp_formatted_chunks: depth %i\n", depth);
+  iter->dump (out);
+}
+}
+
 #ifndef PTRDIFF_MAX
 #define PTRDIFF_MAX INTTYPE_MAXIMUM (ptrdiff_t)
 #endif
@@ -3013,6 +3028,14 @@ pretty_printer::end_url ()
 pp_string (this, get_end_url_string (this));
 }
 
+/* Dump state of this pretty_printer to OUT, for debugging.  */
+
+void
+pretty_printer::dump (FILE *out) const
+{
+  m_buffer->dump (out);
+}
+
 /* class pp_markup::context.  */
 
 void
diff --git a/gcc/pretty-print.h b/gcc/pretty-print.h
index b5ded5cdd5e0..ec64a167327b 100644
--- a/gcc/pretty-print.h
+++ b/gcc/pretty-print.h
@@ -93,6 +93,9 @@ public:
   pp_formatted_chunks *push_formatted_chunks ();
   void pop_formatted_chunks ();
 
+  void dump (FILE *out) const;
+  void DEBUG_FUNCTION dump () const { dump (stderr); }
+
   /* Obstack where the text is built up.  */
   struct obstack m_formatted_obstack;
 
@@ -313,6 +316,9 @@ public:
   void set_real_maximum_length ();
   int remaining_character_count_for_line ();
 
+  void dump (FILE *out) const;
+  void DEBUG_FUNCTION dump () const { dump (stderr); }
+
 private:
   /* Where we print external representation of ENTITY.  */
   output_buffer *m_buffer;
-- 
2.26.3

[pushed: r15-3978] diagnostics: return text buffer from test_show_locus [PR116613]

[David Malcolm]

As work towards supporting multiple diagnostic outputs (where each
output has its own pretty_printer), avoid referencing dc.m_printer
throughout the selftests of diagnostic-show-locus.cc.  Instead
have test_diagnostic_context::test_show_locus return the result
buffer, hiding the specifics of which printer is in use in such
test cases.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3978-g9c14f9a9c19957.

gcc/ChangeLog:
PR other/116613
* diagnostic-show-locus.cc
(selftest::test_diagnostic_show_locus_unknown_location): Move call
to dc.test_show_locus into ASSERT_STREQ, and compare against its
result, rather than explicitly using dc.m_printer.
(selftest::test_one_liner_simple_caret): Likewise.
(selftest::test_one_liner_no_column): Likewise.
(selftest::test_one_liner_caret_and_range): Likewise.
(selftest::test_one_liner_multiple_carets_and_ranges): Likewise.
(selftest::test_one_liner_fixit_insert_before): Likewise.
(selftest::test_one_liner_fixit_insert_after): Likewise.
(selftest::test_one_liner_fixit_remove): Likewise.
(selftest::test_one_liner_fixit_replace): Likewise.
(selftest::test_one_liner_fixit_replace_non_equal_range):
Likewise.
(selftest::test_one_liner_fixit_replace_equal_secondary_range):
Likewise.
(selftest::test_one_liner_fixit_validation_adhoc_locations):
Likewise.
(selftest::test_one_liner_many_fixits_1): Likewise.
(selftest::test_one_liner_many_fixits_2): Likewise.
(selftest::test_one_liner_labels): Likewise.
(selftest::test_one_liner_simple_caret_utf8): Likewise.
(selftest::test_one_liner_multiple_carets_and_ranges_utf8):
Likewise.
(selftest::test_one_liner_fixit_insert_before_utf8): Likewise.
(selftest::test_one_liner_fixit_insert_after_utf8): Likewise.
(selftest::test_one_liner_fixit_remove_utf8): Likewise.
(selftest::test_one_liner_fixit_replace_utf8): Likewise.
(selftest::test_one_liner_fixit_replace_non_equal_range_utf8):
Likewise.
(selftest::test_one_liner_fixit_replace_equal_secondary_range_utf8):
Likewise.
(selftest::test_one_liner_fixit_validation_adhoc_locations_utf8):
Likewise.
(selftest::test_one_liner_many_fixits_1_utf8): Likewise.
(selftest::test_one_liner_many_fixits_2_utf8): Likewise.
(selftest::test_one_liner_labels_utf8): Likewise.
(selftest::test_one_liner_colorized_utf8): Likewise.
(selftest::test_add_location_if_nearby): Likewise.
(selftest::test_diagnostic_show_locus_fixit_lines): Likewise.
(selftest::test_overlapped_fixit_printing): Likewise.
(selftest::test_overlapped_fixit_printing_utf8): Likewise.
(selftest::test_overlapped_fixit_printing_utf8): Likewise.
(selftest::test_overlapped_fixit_printing_2): Likewise.
(selftest::test_fixit_insert_containing_newline): Likewise.
(selftest::test_fixit_insert_containing_newline_2): Likewise.
(selftest::test_fixit_replace_containing_newline): Likewise.
(selftest::test_fixit_deletion_affecting_newline): Likewise.
(selftest::test_tab_expansion): Likewise.
(selftest::test_escaping_bytes_1): Likewise.
(selftest::test_escaping_bytes_2): Likewise.
(selftest::test_line_numbers_multiline_range): Likewise.
* selftest-diagnostic.cc
(selftest::test_diagnostic_context::test_show_locus): Return the
formatted text of m_printer.
* selftest-diagnostic.h
(selftest::test_diagnostic_context::test_show_locus): Convert
return type from void to const char *.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-show-locus.cc | 248 ---
 gcc/selftest-diagnostic.cc   |   6 +-
 gcc/selftest-diagnostic.h|   2 +-
 3 files changed, 88 insertions(+), 168 deletions(-)

diff --git a/gcc/diagnostic-show-locus.cc b/gcc/diagnostic-show-locus.cc
index b575dc51a78c..415de42cbc7b 100644
--- a/gcc/diagnostic-show-locus.cc
+++ b/gcc/diagnostic-show-locus.cc
@@ -3742,8 +3742,7 @@ test_diagnostic_show_locus_unknown_location ()
 {
   test_diagnostic_context dc;
   rich_location richloc (line_table, UNKNOWN_LOCATION);
-  dc.test_show_locus (richloc);
-  ASSERT_STREQ ("", pp_formatted_text (dc.m_printer));
+  ASSERT_STREQ ("", dc.test_show_locus (richloc));
 }
 
 /* Verify that diagnostic_show_locus works sanely for various
@@ -3764,10 +3763,9 @@ test_one_liner_simple_caret ()
   test_diagnostic_context dc;
   location_t caret = linemap_position_for_column (line_table, 10);
   rich_location richloc (line_table, caret);
-  dc.test_show_locus (richloc);
   ASSERT_STREQ (" foo = bar.field;\n"
"  ^\n",
-   pp_formatted_text (dc.m_p

[pushed: r15-3975] diagnostics: avoid using diagnostic_context's m_printer [PR116613]

[David Malcolm]

As work towards supporting multiple diagnostic outputs (where each
output has its own pretty_printer), avoid using diagnostic_context's
m_printer field.  Instead, use the output format's printer.  Currently
this *is* the dc's printer, but eventually it might not be.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3975-gcce52867d1892c.

gcc/ChangeLog:
PR other/116613
* diagnostic-format-json.cc (diagnostic_output_format_init_json):
Pass in the format.  Use the format's printer when disabling
colorization.  Move the call to set_output_format into here.
(diagnostic_output_format_init_json_stderr): Update for above
change.
(diagnostic_output_format_init_json_file): Likewise.
* diagnostic-format-sarif.cc
(diagnostic_output_format_init_sarif): Use the format's printer
when disabling colorization.
* diagnostic-path.cc (selftest::test_empty_path): Use the
text_output's printer.
(selftest::test_intraprocedural_path): Likewise.
(selftest::test_interprocedural_path_1): Likewise.
(selftest::test_interprocedural_path_2): Likewise.
(selftest::test_recursion): Likewise.
(selftest::test_control_flow_1): Likewise.
(selftest::test_control_flow_2): Likewise.
(selftest::test_control_flow_3): Likewise.
(selftest::assert_cfg_edge_path_streq): Likewise.
(selftest::test_control_flow_5): Likewise.
(selftest::test_control_flow_6): Likewise.

gcc/testsuite/ChangeLog:
PR other/116613
* gcc.dg/plugin/diagnostic_group_plugin.c
(test_output_format::on_begin_group): Use get_printer () rather
than accessing m_context.m_printer.
(test_output_format::on_end_group): Likewise.
* gcc.dg/plugin/diagnostic_plugin_xhtml_format.c
(xhtml_builder::m_printer): New field.
(xhtml_builder::xhtml_builder): Add "pp" param and use it to
initialize m_printer.
(xhtml_builder::on_report_diagnostic): Drop "context" param.
(xhtml_builder::make_element_for_diagnostic): Likewise.  Use
this->m_printer rather than the context's m_printer.  Pass
m_printer to call to diagnostic_show_locus.
(xhtml_builder::emit_diagram): Drop "context" param.
(xhtml_output_format::on_report_diagnostic): Drop context param
from call to m_builder.
(xhtml_output_format::on_diagram): Likewise.
(xhtml_output_format::xhtml_output_format): Pass result of
get_printer as printer for builder.
(diagnostic_output_format_init_xhtml): Use the fmt's printer
rather than the context's.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-json.cc | 23 +
 gcc/diagnostic-format-sarif.cc|  2 +-
 gcc/diagnostic-path.cc| 38 +++
 .../gcc.dg/plugin/diagnostic_group_plugin.c   | 12 +++--
 .../plugin/diagnostic_plugin_xhtml_format.c   | 47 +--
 5 files changed, 64 insertions(+), 58 deletions(-)

diff --git a/gcc/diagnostic-format-json.cc b/gcc/diagnostic-format-json.cc
index 448b6cb54eee..cf900a9ecba2 100644
--- a/gcc/diagnostic-format-json.cc
+++ b/gcc/diagnostic-format-json.cc
@@ -393,14 +393,17 @@ private:
to a file).  */
 
 static void
-diagnostic_output_format_init_json (diagnostic_context &context)
+diagnostic_output_format_init_json (diagnostic_context &context,
+   std::unique_ptr fmt)
 {
   /* Suppress normal textual path output.  */
   context.set_path_format (DPF_NONE);
 
   /* Don't colorize the text.  */
-  pp_show_color (context.m_printer) = false;
+  pp_show_color (fmt->get_printer ()) = false;
   context.set_show_highlight_colors (false);
+
+  context.set_output_format (fmt.release ());
 }
 
 /* Populate CONTEXT in preparation for JSON output to stderr.  */
@@ -409,9 +412,10 @@ void
 diagnostic_output_format_init_json_stderr (diagnostic_context &context,
   bool formatted)
 {
-  diagnostic_output_format_init_json (context);
-  context.set_output_format (new json_stderr_output_format (context,
-   formatted));
+  diagnostic_output_format_init_json
+(context,
+ ::make_unique (context,
+  formatted));
 }
 
 /* Populate CONTEXT in preparation for JSON output to a file named
@@ -422,10 +426,11 @@ diagnostic_output_format_init_json_file 
(diagnostic_context &context,
 bool formatted,
 const char *base_file_name)
 {
-  diagnostic_output_format_init_json (context);
-  context.set_output_format 

[pushed: r15-3972] diagnostics: fix typo in XHTML output [PR116792]

[David Malcolm]

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3972-g3286b6724ec1d0.

gcc/testsuite/ChangeLog:
PR other/116792
* gcc.dg/plugin/diagnostic_plugin_xhtml_format.c: Fix stray
reference to JSON.

Signed-off-by: David Malcolm 
---
 gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_xhtml_format.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_xhtml_format.c 
b/gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_xhtml_format.c
index 192288aff1bc..0f13e8d6d01a 100644
--- a/gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_xhtml_format.c
+++ b/gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_xhtml_format.c
@@ -751,7 +751,7 @@ diagnostic_output_format_init_xhtml_file 
(diagnostic_context &context,
 namespace selftest {
 
 /* A subclass of xhtml_output_format for writing selftests.
-   The JSON output is cached internally, rather than written
+   The XML output is cached internally, rather than written
out to a file.  */
 
 class test_xhtml_diagnostic_context : public test_diagnostic_context
-- 
2.26.3

[pushed: r15-3971] diagnostics: fix memory leak in SARIF selftests

[David Malcolm]

"make selftest-valgrind" was complaining about leaks of artifact objects
in SARIF's selftest::test_make_location_object:

-fself-test: 7638695 pass(es) in 89.999249 seconds
==3306525==
==3306525== HEAP SUMMARY:
==3306525== in use at exit: 1,215,639 bytes in 2,808 blocks
==3306525==   total heap usage: 2,860,898 allocs, 2,858,090 frees, 
1,336,446,579 bytes allocated
==3306525==
==3306525== 11,728 (1,536 direct, 10,192 indirect) bytes in 16 blocks are 
definitely lost in loss record 353 of 375
==3306525==at 0x514FE7D: operator new(unsigned long) 
(vg_replace_malloc.c:342)
==3306525==by 0x36E5FD2: sarif_builder::get_or_create_artifact(char const*, 
diagnostic_artifact_role, bool) (diagnostic-format-sarif.cc:2884)
==3306525==by 0x36E3D57: 
sarif_builder::maybe_make_physical_location_object(unsigned int, 
diagnostic_artifact_role, int, content_renderer const*) 
(diagnostic-format-sarif.cc:2097)
==3306525==by 0x36E34CE: 
sarif_builder::make_location_object(sarif_location_manager&, rich_location 
const&, logical_location const*, diagnostic_artifact_role) 
(diagnostic-format-sarif.cc:1922)
==3306525==by 0x36E72C6: 
selftest::test_make_location_object(selftest::line_table_case const&) 
(diagnostic-format-sarif.cc:3500)
==3306525==by 0x375609B: selftest::for_each_line_table_case(void 
(*)(selftest::line_table_case const&)) (input.cc:3898)
==3306525==by 0x36E9668: selftest::diagnostic_format_sarif_cc_tests() 
(diagnostic-format-sarif.cc:3910)
==3306525==by 0x3592A11: selftest::run_tests() (selftest-run-tests.cc:100)
==3306525==by 0x17DBEF3: toplev::run_self_tests() (toplev.cc:2268)
==3306525==by 0x17DC2BF: toplev::main(int, char**) (toplev.cc:2376)
==3306525==by 0x36A1919: main (main.cc:39)
==3306525==
==3306525== 12,400 (1,536 direct, 10,864 indirect) bytes in 16 blocks are 
definitely lost in loss record 355 of 375
==3306525==at 0x514FE7D: operator new(unsigned long) 
(vg_replace_malloc.c:342)
==3306525==by 0x36E5FD2: sarif_builder::get_or_create_artifact(char const*, 
diagnostic_artifact_role, bool) (diagnostic-format-sarif.cc:2884)
==3306525==by 0x36E2323: sarif_builder::sarif_builder(diagnostic_context&, 
line_maps const*, char const*, bool) (diagnostic-format-sarif.cc:1500)
==3306525==by 0x36E70AA: 
selftest::test_make_location_object(selftest::line_table_case const&) 
(diagnostic-format-sarif.cc:3469)
==3306525==by 0x375609B: selftest::for_each_line_table_case(void 
(*)(selftest::line_table_case const&)) (input.cc:3898)
==3306525==by 0x36E9668: selftest::diagnostic_format_sarif_cc_tests() 
(diagnostic-format-sarif.cc:3910)
==3306525==by 0x3592A11: selftest::run_tests() (selftest-run-tests.cc:100)
==3306525==by 0x17DBEF3: toplev::run_self_tests() (toplev.cc:2268)
==3306525==by 0x17DC2BF: toplev::main(int, char**) (toplev.cc:2376)
==3306525==by 0x36A1919: main (main.cc:39)
==3306525==
==3306525== LEAK SUMMARY:
==3306525==definitely lost: 3,072 bytes in 32 blocks
==3306525==indirectly lost: 21,056 bytes in 368 blocks
==3306525==  possibly lost: 0 bytes in 0 blocks
==3306525==still reachable: 1,191,511 bytes in 2,408 blocks
==3306525== suppressed: 0 bytes in 0 blocks
==3306525== Reachable blocks (those to which a pointer was found) are not shown.
==3306525== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3306525==
==3306525== For lists of detected and suppressed errors, rerun with: -s
==3306525== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

Fixed thusly.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3971-gab6c7a329d4958.

gcc/ChangeLog:
* diagnostic-format-sarif.cc (sarif_builder::~sarif_builder): New,
deleting any remaining artifact objects.
(sarif_builder::make_run_object): Empty the artifact map.
* ordered-hash-map.h (ordered_hash_map::empty): New.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc | 14 ++
 gcc/ordered-hash-map.h |  2 ++
 2 files changed, 16 insertions(+)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 6cd18cef6c89..7b11dfd89a31 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -649,6 +649,7 @@ public:
 const line_maps *line_maps,
 const char *main_input_filename_,
 bool formatted);
+  ~sarif_builder ();
 
   void on_report_diagnostic (const diagnostic_info &diagnostic,
 diagnostic_t orig_diag_kind);
@@ -1500,6 +1501,18 @@ sarif_builder::sarif_builder (diagnostic_context 
&context,
  false);
 }
 
+sarif_builder::~sarif_builder ()
+{
+  /* Normally m_filename_to_artifact_map will have been emptied as part
+ of make_run_object, but this isn't run by all the selftests.
+ Ensure the artifact 

Re: [RFC PATCH] More detailed diagnostics for section type conflicts

[David Malcolm]

On Mon, 2024-09-30 at 09:33 +0200, Florian Weimer wrote:
> * David Malcolm:
> 
> > I'm not quite sure what you mean by "non-error" and "non-
> > anchored". 
> 
> Sorry, I'm not familiar with the appropriate terminology.
> 
> > By "non-error", do you mean that this should this be a warning?  If
> > so,
> > use warning_at.  You can use 0 for the option_id whilst
> > prototyping. 
> > Or use "inform" to get a note.
> 
> I meant that I want to attach something like a note to another error.
> There is just one error here (an ICE really, in case of PR116887). 
> If
> it's okay I can call the error function without a location multiple
> times to provide the additional information.  But the thing the
> alternative below seems more appropriate.

To attach notes to another diagnostic, use auto_diagnostic_group, a
RAII class that, during its lifetime, put all diagnostics into a group:

   {
 auto_diagnostic_group d;
 error_at (somewhere, "can't find %qs", "foo");
 inform (somewhere_else, "here's where I last remember seeing it");
}

The effect isn't visible on the standard text output format, but is in
SARIF.

> 
> > By "non-anchored", do you mean "not associated with any particular
> > source location"?  If so, use error_at/warning_at and use
> > UNKNOWN_LOCATION for the location_t ("error" and "warning"
> > implicitly
> > use the "input_location" global variable; it's usually best to
> > instead
> > specify a location, or use UNKNOWN_LOCATION for "global" problems).
> 
> Ahh, so use inform (UNKNOWN_LOCATION, …)?  I see a couple of examples
> like that.

Yes.

Using UNKNOWN_LOCATION will lead to output like:

  cc1: note: message goes here

which might be appropriate, but can be unhelpful to the user for
tracking down the problem.  Is there no source location that's
relevant?

I got the impression from Richi's comments that this might be more of a
GCC developer thing rather than an end-user thing, so perhaps using the
dumpfile might be more appropriate?  (I'm not sure)


Dave

> 
> I'll wait for further comments from Richi and repost.  There are also
> some test cases that need adjusting.
> 
> Thanks,
> Florian
> 

Re: [RFC PATCH] More detailed diagnostics for section type conflicts

[David Malcolm]

On Sun, 2024-09-29 at 17:12 +0200, Florian Weimer wrote:
> Sometimes this is a user error, sometimes it is more of an ICE.
> In either case, more information about the conflict is helpful.
> 
> I used to this to get a better idea about what is going on with
> PR116887.  The original diagnostics look like this:
> 
> dl-find_object.c: In function ‘_dlfo_process_initial’:
> dl-find_object.c:507:1: error: section type conflict with
> ‘_dlfo_nodelete_mappings’
>   507 | _dlfo_process_initial (void)
>   | ^
> dl-find_object.c:73:40: note: ‘_dlfo_nodelete_mappings’ was declared
> here
>    73 | static struct dl_find_object_internal
> *_dlfo_nodelete_mappings
>   |   
> ^~~
> 
> 
> I don't quite understand what is going on (the symbol that's being
> flagged for conflict is somewhat unstable), but at least the new
> diagnostics show that the sectio name, and maybe the flags are
> useful,
> too:
> 
> /tmp/bug.i:6798:1: error: section ‘.data.rel.ro’ type conflict with
> ‘_dlfo_main’
>  6798 | }
>   | ^
> /tmp/bug.i:6190:39: note: ‘_dlfo_main’ was declared here
>  6190 | static struct dl_find_object_internal _dlfo_main
> __attribute__ ((section (".data.rel.ro")));
>   |   ^~
> /tmp/bug.i:6798:1: error: previous section type:
> WRITE|NOTYPE|DECLARED|NAMED
>  6798 | }
>   | ^
> /tmp/bug.i:6798:1: error: new section type: WRITE|NOTYPE|NAMED|RELRO
> 
> I still need help with producing a non-error, non-anchored
> diagnostic.
> I don't know how to do that.

I'm not quite sure what you mean by "non-error" and "non-anchored". 

By "non-error", do you mean that this should this be a warning?  If so,
use warning_at.  You can use 0 for the option_id whilst prototyping. 
Or use "inform" to get a note.

By "non-anchored", do you mean "not associated with any particular
source location"?  If so, use error_at/warning_at and use
UNKNOWN_LOCATION for the location_t ("error" and "warning" implicitly
use the "input_location" global variable; it's usually best to instead
specify a location, or use UNKNOWN_LOCATION for "global" problems).

Or am I misunderstanding?

Hope this is helpful
Dave




> 
> Thanks,
> Florian
> 
>   * varasm.cc (section_flags_to_string):  New function.
>   (get_section): Include name of section in diagnostics.
>   Print old and new section flags, as rendered by
>   section_flags_to_string.
> 
> ---
>  gcc/varasm.cc | 80
> ---
>  1 file changed, 76 insertions(+), 4 deletions(-)
> 
> diff --git a/gcc/varasm.cc b/gcc/varasm.cc
> index 4426e7ce6c6..deba15933aa 100644
> --- a/gcc/varasm.cc
> +++ b/gcc/varasm.cc
> @@ -25,6 +25,7 @@ along with GCC; see the file COPYING3.  If not see
>     We also output the assembler code for constants stored in memory
>     and are responsible for combining constants with the same value. 
> */
>  
> +#define INCLUDE_STRING
>  #include "config.h"
>  #include "system.h"
>  #include "coretypes.h"
> @@ -276,6 +277,65 @@ get_noswitch_section (unsigned int flags,
> noswitch_section_callback callback)
>    return sect;
>  }
>  
> +/* Return a string describing the section flags.  */
> +
> +static std::string
> +section_flags_to_string (unsigned int flags)
> +{
> +  if (flags == 0)
> +    return "UNNAMED";
> +  std::string result;
> +  auto append = [&result] (bool f, const char *name)
> +  {
> +    if (f)
> +  {
> + if (!result.empty ())
> +   result += '|';
> + result += name;
> +  }
> +  };
> +
> +  append (flags & SECTION_CODE, "CODE");
> +  append (flags & SECTION_WRITE, "WRITE");
> +  append (flags & SECTION_DEBUG, "DEBUG");
> +  append (flags & SECTION_LINKONCE, "LINKONCE");
> +  append (flags & SECTION_SMALL, "SMALL");
> +  append (flags & SECTION_BSS, "BSS");
> +  append (flags & SECTION_MERGE, "MERGE");
> +  append (flags & SECTION_STRINGS, "STRINGS");
> +  append (flags & SECTION_OVERRIDE, "OVERRIDE");
> +  append (flags & SECTION_TLS, "TLS");
> +  append (flags & SECTION_NOTYPE, "NOTYPE");
> +  append (flags & SECTION_DECLARED, "DECLARED");
> +  append (flags & SECTION_NAMED, "NAMED");
> +  append (flags & SECTION_NOSWITCH, "NOSWITCH");
> +  append (flags & SECTION_COMMON, "COMMON");
> +  append (flags & SECTION_RELRO, "RELRO");
> +  append (flags & SECTION_EXCLUDE, "EXCLUDE");
> +  append (flags & SECTION_RETAIN, "RETAIN");
> +  append (flags & SECTION_LINK_ORDER, "LINK_ORDER");
> +
> +  unsigned int entsize = flags & SECTION_ENTSIZE;
> +  if (entsize != 0)
> +    {
> +  if (!result.empty ())
> + result += ',';
> +  result += "size=";
> +  result += std::to_string (entsize);
> +    }
> +
> +  unsigned int mach_dep = flags / SECTION_MACH_DEP;
> +  if (mach_dep != 0)
> +    {
> +  if (!result.empty ())
> + result += ',';
> +  result += "mach=";
> +  result += std::to_string (mach_dep);
> +    }
> +
> +  return result;
> +}

Re: [PATCH] diagnostic: Save/restore diagnostic context history and push/pop state for PCH [PR116847]

[David Malcolm]

On Fri, 2024-09-27 at 09:54 -0400, Lewis Hyatt wrote:
> On Fri, Sep 27, 2024 at 9:41 AM David Malcolm 
> wrote:
> > 
> > On Thu, 2024-09-26 at 23:28 +0200, Jakub Jelinek wrote:
> > > Hi!
> > > 
> > > The following patch on top of the just posted cleanup patch
> > > saves/restores the m_classification_history and m_push_list
> > > vectors for PCH.  Without that as the testcase shows during
> > > parsing
> > > of the templates we don't report ignored diagnostics, but after
> > > loading
> > > PCH header when instantiating those templates those warnings can
> > > be
> > > emitted.  This doesn't show up on x86_64-linux build because
> > > configure
> > > injects there -fcf-protection -mshstk flags during library build
> > > (and
> > > so
> > > also during PCH header creation), but make check doesn't use
> > > those
> > > flags
> > > and so the PCH header is ignored.
> > > 
> > > Bootstrapped on i686-linux so far, bootstrap/regtest on x86_64-
> > > linux
> > > and
> > > i686-linux still pending, ok for trunk if it passes it?
> > 
> > Thanks, yes please
> > 
> > Dave
> > 
> 
> A couple comments that may be helpful...
> 
> -This is also PR 64117
> (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64117)
> 
> -I submitted a patch last year for that but did not get any response
> (
> https://gcc.gnu.org/pipermail/gcc-patches/2023-November/635648.html).
> I guess I never pinged it because I am still trying to ping two other
> ones :). 

Gahhh, I'm sorry about this.

What are the other two patches?

> My patch did not switch to vec so it was not as nice as this
> one. I wonder though, if some of the testcases I added could be
> incorporated? In particular the testcase from my patch
> pragma-diagnostic-3.C I believe will still be failing after this one.
> There is an issue with C++ because it processes the pragmas twice,
> once in early mode and once in normal mode, that makes it do the
> wrong
> thing for this case:
> 
> t.h:
> 
>  #pragma GCC diagnostic push
>  #pragma GCC diagnostic ignored...
>  //no pop at end of the file
> 
> t.c
> 
>  #include "t.h"
>  #pragma GCC diagnostic pop
>  //expect to be at the initial state here, but won't be if t.h is a
> PCH
> 
> In my patch I had separated the PCH restore from a more general
> "state
> restore" logic so that the C++ frontend can restore the state after
> the first pass through the data.

It sounds like the ideal here would be to incorporate the test cases
from Lewis's patch into Jakub's, if the latter can be tweaked to fix 
pragma-diagnostic-3.C

Dave

Re: [PATCH 1/2] JSON Dumping of GENERIC trees

[David Malcolm]

On Fri, 2024-09-27 at 10:21 -0500, Thor Preimesberger wrote:
> That's all correct. I think I got it.
> 
> There are times where the code is emitting a json::object* that is
> contained in another json object. Is it good style to return these
> still as a unique_ptr? 

Probably not.  If you have a json::value "contained" in another json
value, either as an array alement, or as an object property, then the
parent "owns" the child: note how the destructor for json::object calls
delete on its property values, and how the destructor for json::array
calls delete on its elements.

So if you have code that is creating a new json value that's about to
be added somewhere in the value tree, that new json value should
probably use std::unique_ptr to indicate ownership (responsibility to
delete), whereas if you're borrowing a value that's already owned by
something in the value tree, just use a regular pointer (or a
reference, if it's guaranteed to be non-null).

Hopefully that makes sense.
Dave


> I'm looking over what I wrote again, and in
> some parts I wrap the new json object in a unique_ptr (as a return in
> some function calls) and in others I use new and delete.
> 
> Thanks,
> Thor Preimesberger
> 
> On Fri, Sep 27, 2024 at 9:18 AM David Malcolm 
> wrote:
> > 
> > On Sat, 2024-09-21 at 22:49 -0500, -thor wrote:
> > > From: thor 
> > > 
> > > This is the second revision of:
> > > 
> > > 
> > > https://gcc.gnu.org/pipermail/gcc-patches/2024-September/662849.html
> > > 
> > > I've incorporated the feedback given both by Richard and David -
> > > I
> > > didn't
> > > find any memory leaks when testing in valgrind :)
> > 
> > Thanks for the updated patch.
> > 
> > [...snip...]
> > 
> > > diff --git a/gcc/tree-emit-json.cc b/gcc/tree-emit-json.cc
> > > new file mode 100644
> > > index 000..df97069b922
> > > --- /dev/null
> > > +++ b/gcc/tree-emit-json.cc
> > 
> > [...snip...]
> > 
> > Thanks for using std::unique_ptr, but I may have been unclear in my
> > earlier email - please use it to indicate ownership of a heap-
> > allocated
> > pointer...
> > 
> > > +/* Adds Identifier information to JSON object */
> > > +
> > > +void
> > > +identifier_node_add_json (tree t, std::unique_ptr
> > > & json_obj)
> > > +  {
> > > +    const char* buff = IDENTIFIER_POINTER (t);
> > > +    json_obj->set_string("id_to_locale",
> > > identifier_to_locale(buff));
> > > +    buff = IDENTIFIER_POINTER (t);
> > > +    json_obj->set_string("id_point", buff);
> > > +  }
> > 
> > ...whereas here (and in many other places), the patch has a
> > 
> >    std::unique_ptr &json_obj
> > 
> > (expressing a reference to a a unique_ptr i.e. a non-modifiable
> > non-
> > null pointer to a pointer that manages the lifetime of a modifiable
> > json::object)
> > 
> > where, if I'm reading the code correctly,
> > 
> >    json::object &json_obj
> > 
> > (expressing a non-modifiable non-null pointer to a modifiable
> > json::object).
> > 
> > would be much clearer and simpler.
> > 
> > [...snip...]
> > 
> > Hope the above makes sense; sorry if I'm being unclear.
> > Dave
> > 
> 

Re: [PATCH] diagnostic: Save/restore diagnostic context history and push/pop state for PCH [PR116847]

[David Malcolm]

On Fri, 2024-09-27 at 10:23 -0400, David Malcolm wrote:
> On Fri, 2024-09-27 at 09:54 -0400, Lewis Hyatt wrote:
> > On Fri, Sep 27, 2024 at 9:41 AM David Malcolm 
> > wrote:
> > > 
> > > On Thu, 2024-09-26 at 23:28 +0200, Jakub Jelinek wrote:
> > > > Hi!
> > > > 
> > > > The following patch on top of the just posted cleanup patch
> > > > saves/restores the m_classification_history and m_push_list
> > > > vectors for PCH.  Without that as the testcase shows during
> > > > parsing
> > > > of the templates we don't report ignored diagnostics, but after
> > > > loading
> > > > PCH header when instantiating those templates those warnings
> > > > can
> > > > be
> > > > emitted.  This doesn't show up on x86_64-linux build because
> > > > configure
> > > > injects there -fcf-protection -mshstk flags during library
> > > > build
> > > > (and
> > > > so
> > > > also during PCH header creation), but make check doesn't use
> > > > those
> > > > flags
> > > > and so the PCH header is ignored.
> > > > 
> > > > Bootstrapped on i686-linux so far, bootstrap/regtest on x86_64-
> > > > linux
> > > > and
> > > > i686-linux still pending, ok for trunk if it passes it?
> > > 
> > > Thanks, yes please
> > > 
> > > Dave
> > > 
> > 
> > A couple comments that may be helpful...
> > 
> > -This is also PR 64117
> > (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64117)
> > 
> > -I submitted a patch last year for that but did not get any
> > response
> > (
> > https://gcc.gnu.org/pipermail/gcc-patches/2023-November/635648.html
> > ).
> > I guess I never pinged it because I am still trying to ping two
> > other
> > ones :). 
> 
> Gahhh, I'm sorry about this.
> 
> What are the other two patches?
> 
> > My patch did not switch to vec so it was not as nice as this
> > one. I wonder though, if some of the testcases I added could be
> > incorporated? In particular the testcase from my patch
> > pragma-diagnostic-3.C I believe will still be failing after this
> > one.
> > There is an issue with C++ because it processes the pragmas twice,
> > once in early mode and once in normal mode, that makes it do the
> > wrong
> > thing for this case:
> > 
> > t.h:
> > 
> >  #pragma GCC diagnostic push
> >  #pragma GCC diagnostic ignored...
> >  //no pop at end of the file
> > 
> > t.c
> > 
> >  #include "t.h"
> >  #pragma GCC diagnostic pop
> >  //expect to be at the initial state here, but won't be if t.h is a
> > PCH
> > 
> > In my patch I had separated the PCH restore from a more general
> > "state
> > restore" logic so that the C++ frontend can restore the state after
> > the first pass through the data.
> 
> It sounds like the ideal here would be to incorporate the test cases
> from Lewis's patch into Jakub's, if the latter can be tweaked to fix 
> pragma-diagnostic-3.C

...and I see that Jakub has pushed his patch (as
https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=64072e60b1599ae7d347c2cdee46c3b0e37fc338
so is there a way to do Lewis's patch on top of that?

Sorry about this
Dave

Re: [PATCH 1/2] JSON Dumping of GENERIC trees

[David Malcolm]

On Sat, 2024-09-21 at 22:49 -0500, -thor wrote:
> From: thor 
> 
> This is the second revision of:
> 
>  
> https://gcc.gnu.org/pipermail/gcc-patches/2024-September/662849.html
> 
> I've incorporated the feedback given both by Richard and David - I
> didn't
> find any memory leaks when testing in valgrind :)

Thanks for the updated patch.

[...snip...]

> diff --git a/gcc/tree-emit-json.cc b/gcc/tree-emit-json.cc
> new file mode 100644
> index 000..df97069b922
> --- /dev/null
> +++ b/gcc/tree-emit-json.cc

[...snip...]

Thanks for using std::unique_ptr, but I may have been unclear in my
earlier email - please use it to indicate ownership of a heap-allocated
pointer...

> +/* Adds Identifier information to JSON object */
> +
> +void
> +identifier_node_add_json (tree t, std::unique_ptr & json_obj)
> +  {
> +    const char* buff = IDENTIFIER_POINTER (t);
> +    json_obj->set_string("id_to_locale", identifier_to_locale(buff));
> +    buff = IDENTIFIER_POINTER (t);
> +    json_obj->set_string("id_point", buff);
> +  }

...whereas here (and in many other places), the patch has a

   std::unique_ptr &json_obj

(expressing a reference to a a unique_ptr i.e. a non-modifiable non-
null pointer to a pointer that manages the lifetime of a modifiable
json::object)

where, if I'm reading the code correctly,

   json::object &json_obj

(expressing a non-modifiable non-null pointer to a modifiable
json::object).

would be much clearer and simpler.

[...snip...]

Hope the above makes sense; sorry if I'm being unclear.
Dave

Re: [PATCH 2/2] HTML Dumping of trees from gdb

[David Malcolm]

On Sat, 2024-09-21 at 22:49 -0500, -thor wrote:
> From: thor 
> 
> This patch allows one to dump a tree as HTML from within gdb by
> invoking,
> i.e,
>   htlml-tree tree
> 
> gcc/ChangeLog:
>     * gcc/gdbhooks.py: Rudimentary dumping of GENERIC trees as html
> through
>   one new python function (jsonNodeToHtml) and one new gdb
> command 
>   (html-tree). There is also a parameter to allow html-tree to 
>   automatically open a browser to view the HTML, but that needs a
> fix
>   or workaround that I don't understand.
> 
> Signed-off-by: Thor C Preimesberger 


Hi Thor, thanks for the patch.

I didn't try running it, but I notice that the patch is building the
HTML directly by writing strings to the output file, using python f-
strings, and there's no escaping of values.  Hence if a value contains
characters like '"', '<', or '>' the resulting HTML will be ill-formed
(similar to a SQL injection attack).

You probably need to use html.escape when writing string values from
the JSON into the HTML; see:
  https://docs.python.org/3/library/html.html#html.escape

Another approach would be to the HTML as a DOM tree in the python
script, and then serialize that; see e.g.:
  https://docs.python.org/3/library/xml.etree.elementtree.html
for a relatively simple API that's readily available in the Python
standard library - but that would be a rewrite of jsonNodeToHtml (but
probably be more robust in the long term).

Hope this is helpful
Dave

> 
> ---
>  gcc/gdbhooks.py | 113
> 
>  1 file changed, 113 insertions(+)
> 
> diff --git a/gcc/gdbhooks.py b/gcc/gdbhooks.py
> index 904ee28423a..fff85d738b4 100644
> --- a/gcc/gdbhooks.py
> +++ b/gcc/gdbhooks.py
> @@ -143,6 +143,7 @@ import os.path
>  import re
>  import sys
>  import tempfile
> +import json
>  
>  import gdb
>  import gdb.printing
> @@ -889,6 +890,118 @@ class DotFn(gdb.Command):
>  
>  DotFn()
>  
> +# Quick and dirty way to turn a tree as JSON object into HTML.
> +# Used in treeToHtml.
> +
> +def jsonNodeToHtml(node_list, html_doc):
> +    for node in node_list:
> +    id = node["addr"]
> +    html_doc.write("" % id)
> +    for key, value in node.items():
> +    if (key == "addr"):
> +    html_doc.write("addr:")
> +    html_doc.write(f"")
> +    html_doc.write(f"{value}")
> +    html_doc.write(f"")
> +    if (type(value) == dict):
> +    html_doc.write(f"{key}:")
> +    sub = value
> +    if "ref_addr" in sub.keys():
> +    html_doc.write(f" 10px\">") 
> +    subAddress = sub["ref_addr"]
> +    subCode = sub["tree_code"]
> +    html_doc.write(f"ref_addr:  href=#{subAddress}>{subAddress}")
> +    html_doc.write(f"tree_code: {subCode}")
> +    html_doc.write("")
> +    # Currently, child tree nodes that are referred to
> by OMP
> +    # accsessors are not dumped recursively by
> +    # dump_generic_node_json, i.e. they have no
> corresponding
> +    # entry in node_list. So we just read it out key-
> value pairs.
> +    else:
> +    html_doc.write(f" px\">") 
> +    for key, value in sub.items():
> +    html_doc.write(f"{key}: {value}")
> +    html_doc.write("")
> +    elif (type(value) == list):
> +    html_doc.write(f"{key}:")
> +    html_doc.write(f"") 
> +    for i in value:
> +    for key, value in i.items():
> +    if (key == "ref_addr"):
> +    html_doc.write("ref_addr:")
> +    html_doc.write(f"")
> +    html_doc.write(f"{value}")
> +    html_doc.write(f"")
> +    else:
> +    html_doc.write(f"{key}: {value}")
> +    html_doc.write("")
> +    elif (key != "addr"):
> +    html_doc.write(f"{key}: {value}")
> +    html_doc.write("")
> +
> +class GCChtml (gdb.Parameter):
> +    """
> +    This parameter defines what program is used to view HTML files
> +    by the html-tree command. It will be invoked as gcc-html  file>.
> +    """
> +    def __init__(self):
> +    super(GCChtml, self).__init__('gcc-html',
> +    gdb.COMMAND_NONE, gdb.PARAM_STRING)
> +    self.value = "firefox"
> +
> +gcc_html_cmd = GCChtml()
> +
> +class treeToHtml (gdb.Command):
> +    """
> +    A custom command that converts a tree to html after it is
> +    first parsed to JSON. The html is saved in cwd as  +
> ".html".
> +    
> +    TODO : It'd be nice if we then open the html with the program
> specified
> +    by the GCChtml parameter, but there's an error thrown whenever I
> try
> +    to do this while attached 

Re: [PATCH] diagnostic: Save/restore diagnostic context history and push/pop state for PCH [PR116847]

[David Malcolm]

On Thu, 2024-09-26 at 23:28 +0200, Jakub Jelinek wrote:
> Hi!
> 
> The following patch on top of the just posted cleanup patch
> saves/restores the m_classification_history and m_push_list
> vectors for PCH.  Without that as the testcase shows during parsing
> of the templates we don't report ignored diagnostics, but after
> loading
> PCH header when instantiating those templates those warnings can be
> emitted.  This doesn't show up on x86_64-linux build because
> configure
> injects there -fcf-protection -mshstk flags during library build (and
> so
> also during PCH header creation), but make check doesn't use those
> flags
> and so the PCH header is ignored.
> 
> Bootstrapped on i686-linux so far, bootstrap/regtest on x86_64-linux
> and
> i686-linux still pending, ok for trunk if it passes it?

Thanks, yes please

Dave

Re: [PATCH] diagnostic: Use vec instead of custom array reallocations for m_classification_history/m_push_list [PR116847]

[David Malcolm]

On Thu, 2024-09-26 at 23:24 +0200, Jakub Jelinek wrote:
> Hi!
> 
> diagnostic.h already relies on vec.h, it uses auto_vec in one spot.
> 
> The following patch converts m_classification_history and m_push_list
> hand-managed arrays to vec templates.
> The main advantage is exponential rather than linear reallocation,
> e.g. with current libstdc++ headers if one includes all the standard
> headers there could be ~ 300 reallocations of the
> m_classification_history
> array (sure, not all of them will result in actually copying the
> data, but
> still).
> In addition to that it fixes some formatting issues in the code.
> 
> Bootstrapped on i686-linux so far, bootstrap/regtest on x86_64-linux
> and
> i686-linux still pending, ok for trunk if it passes it?

Thanks, yes please.

Dave

[pushed: r15-3751] analyzer: simplify dumps using tree_dump_pretty_printer [PR116613]

[David Malcolm]

There are numerous "dump" member functions in the analyzer with
copied-and-pasted logic.  Simplify them by moving the shared code
to a new class tree_dump_pretty_printer.

As well as reducing code duplication, this eliminates numerous
uses of pp_show_color (global_dc->m_printer), which should
ultimately help with supporting multiple diagnostic sinks.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3751-g39f7703fffee0c.

gcc/analyzer/ChangeLog:
PR other/116613
* access-diagram.cc (access_range::dump): Simplify using
tree_dump_pretty_printer.
* call-details.cc (call_details::dump): Likewise.
* call-summary.cc (call_summary::dump): Likewise.
(call_summary_replay::dump): Likewise.
* checker-event.cc (checker_event::debug): Likewise.
* constraint-manager.cc (range::dump): Likewise.
(bounded_range::dump): Likewise.
(bounded_ranges::dump): Likewise.
(constraint_manager::dump): Likewise.
* engine.cc (exploded_node::dump): Likewise.
(exploded_path::dump): Likewise.
* program-point.cc (program_point::dump): Likewise.
* program-state.cc (extrinsic_state::dump_to_file): Likewise.
(sm_state_map::dump): Likewise.
(program_state::dump_to_file): Likewise.
* ranges.cc (symbolic_byte_offset::dump): Likewise.
(symbolic_byte_range::dump): Likewise.
* record-layout.cc (record_layout::dump): Likewise.
* region-model-reachability.cc (reachable_regions::dump):
Likewise.
* region-model.cc (region_to_value_map::dump): Likewise.
(region_model::dump): Likewise.
(model_merger::dump): Likewise.
* region.cc (region_offset::dump): Likewise.
(region::dump): Likewise.
* sm-malloc.cc (deallocator_set::dump): Likewise.
* store.cc (uncertainty_t::dump): Likewise.
(binding_key::dump): Likewise.
(bit_range::dump): Likewise.
(byte_range::dump): Likewise.
(binding_map::dump): Likewise.
(binding_cluster::dump): Likewise.
(store::dump): Likewise.
* supergraph.cc (superedge::dump): Likewise.
* svalue.cc (svalue::dump): Likewise.

gcc/ChangeLog:
PR other/116613
* text-art/dump.h (dump_to_file): Simplify using
tree_dump_pretty_printer.
* tree-diagnostic.h (class tree_dump_pretty_printer): New.

Signed-off-by: David Malcolm 
---
 gcc/analyzer/access-diagram.cc|  6 +---
 gcc/analyzer/call-details.cc  |  6 +---
 gcc/analyzer/call-summary.cc  | 12 ++-
 gcc/analyzer/checker-event.cc |  6 +---
 gcc/analyzer/constraint-manager.cc| 24 +++---
 gcc/analyzer/engine.cc| 12 ++-
 gcc/analyzer/program-point.cc |  5 +--
 gcc/analyzer/program-state.cc | 19 ++--
 gcc/analyzer/ranges.cc| 12 ++-
 gcc/analyzer/record-layout.cc |  5 +--
 gcc/analyzer/region-model-reachability.cc |  6 +---
 gcc/analyzer/region-model.cc  | 18 ++-
 gcc/analyzer/region.cc| 12 ++-
 gcc/analyzer/sm-malloc.cc |  5 +--
 gcc/analyzer/store.cc | 38 +--
 gcc/analyzer/supergraph.cc|  6 +---
 gcc/analyzer/svalue.cc|  6 +---
 gcc/text-art/dump.h   |  8 +
 gcc/tree-diagnostic.h | 20 
 19 files changed, 55 insertions(+), 171 deletions(-)

diff --git a/gcc/analyzer/access-diagram.cc b/gcc/analyzer/access-diagram.cc
index ddeb45a94b24..4822ae392845 100644
--- a/gcc/analyzer/access-diagram.cc
+++ b/gcc/analyzer/access-diagram.cc
@@ -544,13 +544,9 @@ access_range::dump_to_pp (pretty_printer *pp, bool simple) 
const
 DEBUG_FUNCTION void
 access_range::dump (bool simple) const
 {
-  pretty_printer pp;
-  pp_format_decoder (&pp) = default_tree_printer;
-  pp_show_color (&pp) = pp_show_color (global_dc->m_printer);
-  pp.set_output_stream (stderr);
+  tree_dump_pretty_printer pp (stderr);
   dump_to_pp (&pp, simple);
   pp_newline (&pp);
-  pp_flush (&pp);
 }
 
 void
diff --git a/gcc/analyzer/call-details.cc b/gcc/analyzer/call-details.cc
index e543eda347a5..a9c613bc1822 100644
--- a/gcc/analyzer/call-details.cc
+++ b/gcc/analyzer/call-details.cc
@@ -363,12 +363,8 @@ call_details::dump_to_pp (pretty_printer *pp, bool simple) 
const
 DEBUG_FUNCTION void
 call_details::dump (bool simple) const
 {
-  pretty_printer pp;
-  pp_format_decoder (&pp) = default_tree_printer;
-  pp_show_color (&pp) = pp_show_color (global_dc->m_printer);
-  pp.set_output_stream (stderr);
+  tree_dump_pretty_printer pp (stderr);
   dump_to_pp (&pp, simple);
-  pp_flush (&pp);
 }
 
 /* Get a conjured_svalue for this call for REG,
diff --git a/

[pushed: r15-3750] diagnostics: isolate SARIF output's pretty_printer [PR116613]

[David Malcolm]

Add an m_printer to sarif_builder and use throughout, rather than
using the context's printer.  For now this is the same printer, but
eventually this should help with transitioning to multiple output sinks.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3750-g725231e16768bd.

gcc/ChangeLog:
PR other/116613
* diagnostic-format-sarif.cc (sarif_builder::m_printer): New
field.
(sarif_invocation::add_notification_for_ice): Drop context param.
(sarif_invocation::prepare_to_flush): Convert param from context
to builder.
(sarif_result::on_nested_diagnostic): Drop context param.  Use
builder's printer.
(sarif_result::on_diagram): Drop context param.
(sarif_ice_notification::sarif_ice_notification): Drop context
param.  Use builder's printer.
(sarif_builder::sarif_builder): Initialize m_printer.
(sarif_builder::on_report_diagnostic): Drop context param.  Use
builder's printer.
(sarif_builder::emit_diagram): Drop context param.
(sarif_builder::flush_to_object): Use this rather than context
for call to prepare_to_flush.
(sarif_builder::make_result_object): Drop context param.  Use
builder's printer.
(sarif_builder::make_reporting_descriptor_object_for_warning):
Drop context param.
(sarif_builder::make_message_object_for_diagram): Likewise.
Use builder's printer.
(sarif_output_format::on_report_diagnostic): Drop context param
from call to sarif_builder::on_report_diagnostic.
(sarif_output_format::on_diagram): Drop context param from call to
sarif_builder::emit_diagram.
* diagnostic.h (diagnostic_conetxt::get_client_data_hooks): Make const.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc | 116 ++---
 gcc/diagnostic.h   |   2 +-
 2 files changed, 51 insertions(+), 67 deletions(-)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index eda67a6f6583..6cd18cef6c89 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -166,10 +166,9 @@ public:
   sarif_invocation (sarif_builder &builder,
const char * const *original_argv);
 
-  void add_notification_for_ice (diagnostic_context &context,
-const diagnostic_info &diagnostic,
+  void add_notification_for_ice (const diagnostic_info &diagnostic,
 sarif_builder &builder);
-  void prepare_to_flush (diagnostic_context &context);
+  void prepare_to_flush (sarif_builder &builder);
 
 private:
   std::unique_ptr m_notifications_arr;
@@ -395,12 +394,10 @@ public:
   unsigned get_index_within_parent () const { return m_idx_within_parent; }
 
   void
-  on_nested_diagnostic (diagnostic_context &context,
-   const diagnostic_info &diagnostic,
+  on_nested_diagnostic (const diagnostic_info &diagnostic,
diagnostic_t orig_diag_kind,
sarif_builder &builder);
-  void on_diagram (diagnostic_context &context,
-  const diagnostic_diagram &diagram,
+  void on_diagram (const diagnostic_diagram &diagram,
   sarif_builder &builder);
 
   void
@@ -583,8 +580,7 @@ class sarif_replacement : public sarif_object {};
 class sarif_ice_notification : public sarif_location_manager
 {
 public:
-  sarif_ice_notification (diagnostic_context &context,
- const diagnostic_info &diagnostic,
+  sarif_ice_notification (const diagnostic_info &diagnostic,
  sarif_builder &builder);
 
   void
@@ -654,11 +650,9 @@ public:
 const char *main_input_filename_,
 bool formatted);
 
-  void on_report_diagnostic (diagnostic_context &context,
-const diagnostic_info &diagnostic,
+  void on_report_diagnostic (const diagnostic_info &diagnostic,
 diagnostic_t orig_diag_kind);
-  void emit_diagram (diagnostic_context &context,
-const diagnostic_diagram &diagram);
+  void emit_diagram (const diagnostic_diagram &diagram);
   void end_group ();
 
   std::unique_ptr take_current_result ()
@@ -685,8 +679,7 @@ public:
   std::unique_ptr
   make_message_object (const char *msg) const;
   std::unique_ptr
-  make_message_object_for_diagram (diagnostic_context &context,
-  const diagnostic_diagram &diagram);
+  make_message_object_for_diagram (const diagnostic_diagram &diagram);
   std::unique_ptr
   maybe_make_artifact_content_object (const char *filename) const;
 
@@ -699,6 +692,8 @@ public:
 return m_current_code_flow;
   }

[pushed: r15-3749] diagnostics: convert text hooks to use diagnostic_text_output_format [PR116613]

[David Malcolm]

_text_starter): ...this.  Convert first
param from diagnostic_context * to diagnostic_text_output_format &
and update accordingly.
(tree_diagnostics_defaults): Update for renamings.

gcc/cp/ChangeLog:
PR other/116613
* cp-tree.h (cxx_print_error_function): Convert first param
from diagnostic_context * to diagnostic_text_output_format &.
* error.cc: Include "diagnostic-format-text.h".
(cxx_initialize_diagnostics): Update for renamings.
(cxx_print_error_function): Convert first param from
diagnostic_context * to diagnostic_text_output_format & and update
accordingly
(cp_diagnostic_starter): Rename to...
(cp_diagnostic_text_starter): ...this.  Convert first
param from diagnostic_context * to diagnostic_text_output_format &
and update accordingly.
(cp_print_error_function): Likewise.
(print_instantiation_full_context): Likewise.
(print_instantiation_partial_context_line): Likewise.
(print_instantiation_partial_context): Likewise.
(maybe_print_instantiation_context): Likewise.
(maybe_print_constexpr_context): Likewise.
(print_location): Likewise.
(print_constrained_decl_info): Likewise.
(print_concept_check_info): Likewise.
(print_constraint_context_head): Likewise.
(print_requires_expression_info): Likewise.
(maybe_print_single_constraint_context): Likewise.

gcc/fortran/ChangeLog:
PR other/116613
* error.cc: Include "diagnostic-format-text.h".
(gfc_diagnostic_starter): Rename to...
(gfc_diagnostic_text_starter): ...this.  Convert first
param from diagnostic_context * to diagnostic_text_output_format &
and update accordingly.
(gfc_diagnostic_finalizer, gfc_diagnostic_text_finalizer):
Likewise.
(gfc_diagnostics_init): Update for renamings.
(gfc_diagnostics_finish): Likewise.

gcc/jit/ChangeLog:
PR other/116613
* dummy-frontend.cc: Include "diagnostic-format-text.h".
(jit_begin_diagnostic): Convert first param from
diagnostic_context * to diagnostic_text_output_format &
(jit_end_diagnostic): Likewise.  Update accordingly.
(jit_langhook_init): Update for renamings.

gcc/rust/ChangeLog:
PR other/116613
* resolve/rust-ast-resolve-expr.cc
(funny_ice_finalizer): : Convert first param from
diagnostic_context * to diagnostic_text_output_format &.
(ResolveExpr::visit): Update for renaming.

gcc/testsuite/ChangeLog:
PR other/116613
* g++.dg/plugin/show_template_tree_color_plugin.c
(noop_starter_fn): Rename to...
(noop_text_starter_fn): ...this.  Update first param from dc to
text_output.
(plugin_init): Update for renamings.
* gcc.dg/plugin/diagnostic_group_plugin.c
(test_diagnostic_starter): Rename to...
(test_diagnostic_text_starter): ...this.  Update first param from
dc to text_output.
(plugin_init): Update for renaming.
* gcc.dg/plugin/diagnostic_plugin_test_show_locus.c: Include
"diagnostic-format-text.h".
(custom_diagnostic_finalizer): Rename to...
(custom_diagnostic_text_finalizer): ...this.  Update first param
from dc to text_output.
(test_show_locus): Update for renamings.
* gcc.dg/plugin/location_overflow_plugin.c: Include
"diagnostic-format-text.h".
(original_finalizer): Rename to...
(original_text_finalizer): ...this and update type.
(verify_unpacked_ranges): Update first param from dc to
text_output.  Update for this and for renamings.
(verify_no_columns): Likewise.
    (plugin_init): Update for renamings.

libcc1/ChangeLog:
PR other/116613
* context.cc: Include "diagnostic-format-text.h".
(plugin_print_error_function): Update first param from
diagnostic_context * to diagnostic_text_output_format &.

Signed-off-by: David Malcolm 
---
 gcc/c-family/c-opts.cc|  16 +-
 gcc/coretypes.h   |   1 +
 gcc/cp/cp-tree.h  |   2 +-
 gcc/cp/error.cc   | 149 ++--
 gcc/diagnostic-format-json.cc |   4 +
 gcc/diagnostic-format-sarif.cc|   4 +
 gcc/diagnostic-format-text.cc | 186 ++-
 gcc/diagnostic-format-text.h  |  31 ++-
 gcc/diagnostic-format.h   |   7 +
 gcc/diagnostic-macro-unwinding.cc |  19 +-
 gcc/diagnostic-macro-unwinding.h  |   4 +-
 gcc/diagnostic-path.cc| 214 ++
 gcc/diagnostic.cc | 205 +++--
 gcc/diagnostic.h

[pushed: r15-3752] diagnostics: add HTML output format as a plugin [PR116792]

[David Malcolm]

This patch adds an experimental diagnostics output format that
writes HTML.  It isn't ready yet for end-users, but seems worth
keeping in the tree as I refactor the diagnostics subsystem, to
ensure that this code still builds, and to verify that it's possible to
implement new diagnostic output formats via GCC plugins. Hence
this patch merely adds it to the testsuite as an example of a GCC
plugin, rather than exposing it as a feature for end-users.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3752-g48261bd26df624.

gcc/testsuite/ChangeLog:
PR other/116792
* gcc.dg/plugin/diagnostic-test-xhtml-1.c: New test.
* gcc.dg/plugin/diagnostic_plugin_xhtml_format.c: New test plugin.
* gcc.dg/plugin/plugin.exp (plugin_test_list): Add the above.

Signed-off-by: David Malcolm 
---
 .../gcc.dg/plugin/diagnostic-test-xhtml-1.c   |  19 +
 .../plugin/diagnostic_plugin_xhtml_format.c   | 866 ++
 gcc/testsuite/gcc.dg/plugin/plugin.exp|   2 +
 3 files changed, 887 insertions(+)
 create mode 100644 gcc/testsuite/gcc.dg/plugin/diagnostic-test-xhtml-1.c
 create mode 100644 gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_xhtml_format.c

diff --git a/gcc/testsuite/gcc.dg/plugin/diagnostic-test-xhtml-1.c 
b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-xhtml-1.c
new file mode 100644
index ..da069ff4789d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-xhtml-1.c
@@ -0,0 +1,19 @@
+/* { dg-do compile } */
+
+int missing_semicolon (void)
+{
+  return 42
+}
+
+/* Verify some properties of the generated HTML.  */
+
+/* { dg-begin-multiline-output "" }
+
+http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";>
+http://www.w3.org/1999/xhtml";>
+  
+   { dg-end-multiline-output "" } */
+
+/* { dg-excess-errors "" } */
diff --git a/gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_xhtml_format.c 
b/gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_xhtml_format.c
new file mode 100644
index ..192288aff1bc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_xhtml_format.c
@@ -0,0 +1,866 @@
+/* Verify that we can write a non-trivial diagnostic output format
+   as a plugin (XHTML).
+   Copyright (C) 2018-2024 Free Software Foundation, Inc.
+   Contributed by David Malcolm .
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+<http://www.gnu.org/licenses/>.  */
+
+
+#include "config.h"
+#define INCLUDE_LIST
+#define INCLUDE_MAP
+#define INCLUDE_MEMORY
+#define INCLUDE_VECTOR
+#include "system.h"
+#include "coretypes.h"
+#include "diagnostic.h"
+#include "diagnostic-metadata.h"
+#include "diagnostic-path.h"
+#include "cpplib.h"
+#include "logical-location.h"
+#include "diagnostic-client-data-hooks.h"
+#include "diagnostic-diagram.h"
+#include "text-art/canvas.h"
+#include "diagnostic-format.h"
+#include "ordered-hash-map.h"
+#include "sbitmap.h"
+#include "make-unique.h"
+#include "selftest.h"
+#include "selftest-diagnostic.h"
+#include "selftest-diagnostic-show-locus.h"
+#include "text-range-label.h"
+#include "pretty-print-format-impl.h"
+#include "pretty-print-urlifier.h"
+#include "intl.h"
+#include "gcc-plugin.h"
+#include "plugin-version.h"
+
+namespace xml {
+
+/* Disable warnings about quoting issues in the pp_xxx calls below
+   that (intentionally) don't follow GCC diagnostic conventions.  */
+#if __GNUC__ >= 10
+#  pragma GCC diagnostic push
+#  pragma GCC diagnostic ignored "-Wformat-diag"
+#endif
+
+static void write_escaped_text (const char *text);
+
+struct node
+{
+  virtual ~node () {}
+  virtual void write_as_xml (pretty_printer *pp,
+int depth, bool indent) const = 0;
+  void dump (FILE *out) const;
+  void DEBUG_FUNCTION dump () const { dump (stderr); }
+};
+
+struct text : public node
+{
+  text (label_text str)
+  : m_str (std::move (str))
+  {}
+
+  void write_as_xml (pretty_printer *pp,
+int depth, bool indent) const final override;
+
+  label_text m_str;
+};
+
+struct node_with_children : public node
+{
+  void add_child (std::unique_ptr node);
+  void add_text (label_text s

[pushed: r15-3748] analyzer: remove redundant 'pp' [PR116613]

[David Malcolm]

diagnostic_manager::emit_saved_diagnostic makes a useless clone
of global_dc->m_printer; remove it.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3748-g37604edf37b379.

gcc/analyzer/ChangeLog:
PR other/116613
* diagnostic-manager.cc (diagnostic_manager::emit_saved_diagnostic):
Remove remove redundant 'pp'.

Signed-off-by: David Malcolm 
---
 gcc/analyzer/diagnostic-manager.cc | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/gcc/analyzer/diagnostic-manager.cc 
b/gcc/analyzer/diagnostic-manager.cc
index 4a5a95136156..2363ae5df19c 100644
--- a/gcc/analyzer/diagnostic-manager.cc
+++ b/gcc/analyzer/diagnostic-manager.cc
@@ -1561,8 +1561,6 @@ diagnostic_manager::emit_saved_diagnostic (const 
exploded_graph &eg,
sd.get_index (), sd.m_d->get_kind (), sd.m_snode->m_index);
   log ("num dupes: %i", sd.get_num_dupes ());
 
-  pretty_printer *pp = global_dc->m_printer->clone ();
-
   const exploded_path *epath = sd.get_best_epath ();
   gcc_assert (epath);
 
@@ -1645,7 +1643,6 @@ diagnostic_manager::emit_saved_diagnostic (const 
exploded_graph &eg,
  free (filename);
}
 }
-  delete pp;
 }
 
 /* Emit a "path" of events to EMISSION_PATH describing the exploded path
-- 
2.26.3

Re: [PATCH] libcpp: Implement clang -Wheader-guard warning [PR96842]

[David Malcolm]

On Thu, 2024-09-12 at 17:18 +0200, Jakub Jelinek wrote:
> On Thu, Sep 12, 2024 at 11:12:26AM -0400, David Malcolm wrote:
> > We were chatting on IRC about how it would be nice to be able to
> > use
> > %qs in libcppp diagnostics; here is an example (rather than using
> > \"%s\").
> 
> Yeah, I'm working on a patch for that.

Thanks.

> 
> > Not a blocker, but it occurs to me that ideally we'd group the
> > warning
> > and note into a diagnostic group, but unfortunately there's no way
> > to
> > express that currently via the interface libcpp has.  We would need
> > to
> > add {begin,end}_group hooks, which in turn suggests that maybe that
> > libcpp's interface into diagnostics should be an abstract base
> > class
> > with various vfuncs, rather than a callback.
> 
> I haven't added auto_diagnostic_group because nothing in libcpp does
> that,
> yes, we need some solution for that.

(nods)

> 
> > Also not a blocker, but it would nice to have a fix-it hint here,
> > by
> > using the rich_location overload of cpp_error_at and adding a fix-
> > it
> > hint to the rich_location.
> 
> And yes, I was thinking about fix-it hint, but I think that depends
> on
> better locations there first, currently the patch uses just the lines
> with the directives.
> I was hoping that can be done incrementally.

Indeed, let's defer the fix-it hint to a possible followup.


Thanks
Dave

Re: [PATCH] libcpp: Implement clang -Wheader-guard warning [PR96842]

[David Malcolm]

On Wed, 2024-09-11 at 23:26 +0200, Jakub Jelinek wrote:
> Hi!
> 
> The following patch implements the clang -Wheader-guard warning,
> which warns
> if a valid multiple inclusion header guard's #ifndef/#if !defined
> directive
> is immediately (no other non-line directives nor other (non-comment)
> tokens in between) followed by #define directive for some different
> macro,
> which in get_suggestion rules is close enough to the actual header
> guard
> macro (i.e. likely misspelling), the #define is object-like with
> empty
> definition (I've followed what clang implements) and the macro isn't
> defined
> later on (at least not on the final #endif at the end of a header).
> 
> In this case it emits a warning, so that
> #ifndef STDIO_H
> #define STDOI_H
> ...
> #endif
> or similar misspellings can be caught.
> 
> clang enables this warning by default, but I've put it into -Wall
> instead
> as it still seems to be a style warning, nothing more severe; if a
> header
> doesn't survive multiple inclusion because of the misspelling, users
> will
> get different diagnostics.
> 
> Bootstrapped/regtested on x86_64-linux and i686-linux, ok for trunk?
> 

Overall, LGTM, but I'm not as familiar with libcpp's implementation
details.

> 
> --- libcpp/files.cc.jj2024-09-03 16:47:47.322031849 +0200
> +++ libcpp/files.cc   2024-09-11 19:32:43.754868132 +0200
> @@ -1664,7 +1664,28 @@ _cpp_pop_file_buffer (cpp_reader *pfile,
>    /* Record the inclusion-preventing macro, which could be NULL
>   meaning no controlling macro.  */
>    if (pfile->mi_valid && file->cmacro == NULL)
> -    file->cmacro = pfile->mi_cmacro;
> +    {
> +  file->cmacro = pfile->mi_cmacro;
> +  if (pfile->mi_cmacro
> +   && pfile->mi_def_cmacro
> +   && pfile->cb.get_suggestion)
> + {
> +   const char *names[]
> +     = { (const char *) NODE_NAME (pfile->mi_def_cmacro),
> NULL };
> +   if (pfile->cb.get_suggestion (pfile,
> + (const char *)
> + NODE_NAME (pfile-
> >mi_cmacro), names)
> +   && cpp_warning_with_line (pfile, CPP_W_HEADER_GUARD,
> + pfile->mi_loc, 0,
> + "header guard \"%s\"
> followed by "
> + "\"#define\" of a different
> macro",
> + NODE_NAME (pfile-
> >mi_cmacro)))
> +     cpp_error_at (pfile, CPP_DL_NOTE, pfile->mi_def_loc,
> +   "\"%s\" is defined here; did you mean
> \"%s\"?",
> +   NODE_NAME (pfile->mi_def_cmacro),
> +   NODE_NAME (pfile->mi_cmacro));
> + }

We were chatting on IRC about how it would be nice to be able to use
%qs in libcppp diagnostics; here is an example (rather than using
\"%s\").

Not a blocker, but it occurs to me that ideally we'd group the warning
and note into a diagnostic group, but unfortunately there's no way to
express that currently via the interface libcpp has.  We would need to
add {begin,end}_group hooks, which in turn suggests that maybe that
libcpp's interface into diagnostics should be an abstract base class
with various vfuncs, rather than a callback.

Also not a blocker, but it would nice to have a fix-it hint here, by
using the rich_location overload of cpp_error_at and adding a fix-it
hint to the rich_location.

Hope this is constructive
Dave

Re: [PATCH] JSON dumping for GENERIC trees

[David Malcolm]

On Wed, 2024-09-11 at 20:49 -0500, tcpreimesber...@gmail.com wrote:
> From: Thor C Preimesberger 
> 
> This patch allows the compiler to dump GENERIC trees as JSON objects.
> 
> The dump flag -fdump-tree-original-json dumps each fndecl node in the
> C frontend's gimplifier as a JSON object and traverses related nodes 
> in an analagous manner as to raw-dumping.

Thanks for posting this patch.

Are you able to upload somewhere some examples of what the dumps look
like?

Some high level thoughts:

* the patch uses "dummy" throughout as a variable name.  To me the name
"dummy" suggests something unimportant that we had to give a name to,
or something that we'd prefer didn't exist but had to create.  However
in most(all?) cases "dummy" seems to refer to the json object being
created or having properties added to it, and thus the most interesting
thing in the function.  I suspect that renaming "dummy" to "js_obj" or
"json_obj" throughout would be an improvement in readability in terms
of capturing the intent of the code (assuming that all of them are
indeed json objects).

* I think the code is leaking memory for all of the json values created
- there are lots of uses of "naked new" in this code, but I don't see
any uses of "delete".  For example, in 

> +void
> +dump_node_json (const_tree t, dump_flags_t flags, FILE *stream)
> +{
> +  struct dump_info di;
> +  dump_queue_p dq;
> +  dump_queue_p next_dq;
> +  pretty_printer pp;
> +  /* Initialize the dump-information structure.  */
> +  di.stream = stream;
> +  di.index = 0;
> +  di.column = 0;
> +  di.queue = 0;
> +  di.queue_end = 0;
> +  di.free_list = 0;
> +  di.flags = flags;
> +  di.node = t;
> +  di.nodes = splay_tree_new (splay_tree_compare_pointers, 0,
> +  splay_tree_delete_pointers);
> +  di.json_dump = new json::array ();
    ^^
 allocated with naked new here

> +  /* Queue up the first node.  */
> +  queue (&di, t);
> +
> +  /* Until the queue is empty, keep dumping nodes.  */
> +  while (di.queue)
> +dequeue_and_dump (&di);
> +
> +  di.json_dump->dump(stream, true);
> +  fputs("\n", stream);
> +  /* Now, clean up.  */
> +  for (dq = di.free_list; dq; dq = next_dq)
> +{
> +  next_dq = dq->next;
> +  free (dq);
> +}
> +  splay_tree_delete (di.nodes);

and di.json_dump goes out of scope here and is leaked, I think.  So I
*think* all of the json values being created during dumping are being
leaked.

> +}

Similarly, in:

> +DEBUG_FUNCTION void
> +debug_tree_json (tree t)
> +{
> +  json::object* _x = node_emit_json(t);
> +  _x->dump(stderr, true);
> +  fprintf(stderr, "\n");
> +}

if I'm reading things right, node_emit_json doesn't "emit" json so much
as create a new json::object on the heap via "new", and when "_x" goes
out of scope, it's leaked.

The pattern in the code seems to be that node_emit_json creates a new
json::object and populates it with properties (sometimes recursively).

Given that, and that we can use C++11, I recommend using
std::unique_ptr for it, to capture the intent that this
is a heap-allocated pointer with responsibility for being "delete"-d at
some point.

That way, rather that:

  json::object* 
  node_emit_json(tree t)
  {
tree op0, op1, type;
enum tree_code code;
expanded_location xloc;
json::object *dummy;
json::array* holder;
char address_buffer[sizeof(&t)] = {"\0"};
  
dummy = new json::object ();
holder = new json::array ();

[...snip...]

return dummy;
  }


we could have (perhaps renaming to "node_to_json"):

  std::unique_ptr
  node_to_json(tree t)
  {
tree op0, op1, type;
enum tree_code code;
expanded_location xloc;
char address_buffer[sizeof(&t)] = {"\0"};
  
auto js_obj = ::make_unique (); // to implicitly use 
std::unique_ptr
auto holder = ::make_unique ();  // likewise 
std::unique_ptr

[...snip...]

return js_obj;
  }

...assuming that I'm correctly understanding the ownership of the json
values in the patch.  Note that we have to use ::make_unique from our
make-unique.h, rather than std::make_unique from  since the
latter was only added in C++14.

Many of our data structures don't properly handle objects with
destructors, and I suspect splay_tree is one of these.  You can use
js_obj.release () to transfer ownership to such data structures, and
will (probably) need to manually use "delete" on the pointers in the
right places.

What happens to "holder" in that function?  It seems to get populated
with json objects for the various tree nodes found recursively, but
then it seems to simply be leaked (or populated then auto-deleted, if
we use std::unique_ptr>.  Or am I missing something?

In case it's helpful, a couple of months ago I converted the SARIF
output code from using "naked" json pointers to using std::unique_ptr
in:
  https://gcc.gnu.org/pipermail/gcc-patches/2024-July/658204.html
and I found it helped a *lot* with documenting ownership and avoiding
leaks.

[pushed: r15-3556] diagnostics: introduce struct diagnostic_option_id

[David Malcolm]

p): Likewise.

gcc/fortran/ChangeLog:
* cpp.cc (cb_cpp_diagnostic_cpp_option): Convert return type from
"int" to "diagnostic_option_id".
(cb_cpp_diagnostic): Update for renaming of
diagnostic_override_option_index to diagnostic_set_option_id.
* error.cc (gfc_warning): Update for renaming of diagnostic_info
field.
(gfc_warning_now_at): Likewise.
(gfc_warning_now): Likewise.
(gfc_warning_internal): Likewise.

gcc/ChangeLog:
* ipa-pure-const.cc: Replace include of "opts.h" with
"opts-diagnostic.h".
(suggest_attribute): Convert param from int to
diagnostic_option_id.
* lto-wrapper.cc (class lto_diagnostic_option_manager): Use
diagnostic_option_id rather than "int".
* opts-common.cc
(compiler_diagnostic_option_manager::option_enabled_p): Likewise.
* opts-diagnostic.h (class gcc_diagnostic_option_manager):
Likewise.
(class compiler_diagnostic_option_manager): Likewise.
* opts.cc (compiler_diagnostic_option_manager::make_option_name):
Likewise.
(gcc_diagnostic_option_manager::make_option_url): Likewise.
* substring-locations.cc
(format_string_diagnostic_t::emit_warning_n_va): Likewise.
(format_string_diagnostic_t::emit_warning_va): Likewise.
(format_string_diagnostic_t::emit_warning): Likewise.
(format_string_diagnostic_t::emit_warning_n): Likewise.
* substring-locations.h
(format_string_diagnostic_t::emit_warning_va): Likewise.
(format_string_diagnostic_t::emit_warning_n_va): Likewise.
(format_string_diagnostic_t::emit_warning): Likewise.
(format_string_diagnostic_t::emit_warning_n): Likewise.

Signed-off-by: David Malcolm 
---
 gcc/c-family/c-common.cc |   7 +--
 gcc/c/c-errors.cc|  38 +++-
 gcc/c/c-tree.h   |   8 +--
 gcc/cp/constexpr.cc  |   2 +-
 gcc/cp/cp-tree.h |   4 +-
 gcc/cp/error.cc  |  10 +--
 gcc/d/d-diagnostic.cc|   2 +-
 gcc/diagnostic-core.h|  78 +++-
 gcc/diagnostic-format-json.cc|   4 +-
 gcc/diagnostic-format-sarif.cc   |   4 +-
 gcc/diagnostic-format-text.cc|   4 +-
 gcc/diagnostic-global-context.cc | 101 +++
 gcc/diagnostic.cc|  55 +
 gcc/diagnostic.h |  78 +---
 gcc/fortran/cpp.cc   |   6 +-
 gcc/fortran/error.cc |   8 +--
 gcc/ipa-pure-const.cc|   5 +-
 gcc/lto-wrapper.cc   |   6 +-
 gcc/opts-common.cc   |   5 +-
 gcc/opts-diagnostic.h|   6 +-
 gcc/opts.cc  |  16 ++---
 gcc/substring-locations.cc   |  21 ---
 gcc/substring-locations.h|  16 +++--
 23 files changed, 295 insertions(+), 189 deletions(-)

diff --git a/gcc/c-family/c-common.cc b/gcc/c-family/c-common.cc
index e7e371fd26f6..ec6a5da892dd 100644
--- a/gcc/c-family/c-common.cc
+++ b/gcc/c-family/c-common.cc
@@ -6787,7 +6787,7 @@ c_parse_error (const char *gmsgid, enum cpp_ttype 
token_type,
 /* Return the gcc option code associated with the reason for a cpp
message, or 0 if none.  */
 
-static int
+static diagnostic_option_id
 c_option_controlling_cpp_diagnostic (enum cpp_warning_reason reason)
 {
   const struct cpp_reason_option_codes_t *entry;
@@ -6870,9 +6870,8 @@ c_cpp_diagnostic (cpp_reader *pfile ATTRIBUTE_UNUSED,
 richloc->set_range (0, input_location, SHOW_RANGE_WITH_CARET);
   diagnostic_set_info_translated (&diagnostic, msg, ap,
  richloc, dlevel);
-  diagnostic_override_option_index
-(&diagnostic,
- c_option_controlling_cpp_diagnostic (reason));
+  diagnostic_set_option_id (&diagnostic,
+   c_option_controlling_cpp_diagnostic (reason));
   ret = diagnostic_report_diagnostic (global_dc, &diagnostic);
   if (level == CPP_DL_WARNING_SYSHDR)
 global_dc->m_warn_system_headers = save_warn_system_headers;
diff --git a/gcc/c/c-errors.cc b/gcc/c/c-errors.cc
index f36e7f9780a3..c6b7c108354b 100644
--- a/gcc/c/c-errors.cc
+++ b/gcc/c/c-errors.cc
@@ -32,7 +32,9 @@ along with GCC; see the file COPYING3.  If not see
when C2Y is specified.  */
 
 bool
-pedwarn_c23 (location_t location, int opt, const char *gmsgid, ...)
+pedwarn_c23 (location_t location,
+diagnostic_option_id option_id,
+const char *gmsgid, ...)
 {
   diagnostic_info diagnostic;
   va_list ap;
@@ -47,7 +49,7 @@ pedwarn_c23 (location_t location, int opt, const char 
*gmsgid, ...)
   diagnostic_set_info (&diagnostic, gmsgid, &ap, &richloc,
   (pedantic && !flag_isoc2y)
   ? DK_PEDWARN : DK_WARNING);
-  diagnostic.option_index = OPT_Wc23

[pushed: r15-3555] diagnostics: replace option_hooks with a diagnostic_option_manager class

[David Malcolm]

Introduce a diagnostic_option_manager class to help isolate the
diagnostics subsystem from GCC's option handling.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3555-ga97448e92eb76a.

gcc/ChangeLog:
* diagnostic.cc (diagnostic_context::initialize): Replace
m_options_callbacks with m_option_mgr.
(diagnostic_context::set_option_hooks): Replace with...
(diagnostic_context::set_option_manager): ...this.
* diagnostic.h (diagnostic_option_enabled_cb): Delete.
(diagnostic_make_option_name_cb): Delete.
(diagnostic_make_option_url_cb): Delete.
(class diagnostic_option_manager): New.
(diagnostic_manager::option_enabled_p): Convert from using
m_option_callbacks to m_option_mgr.
(diagnostic_manager::make_option_name): Likewise.
(diagnostic_manager::make_option_url): Likewise.
(diagnostic_manager::set_option_hooks): Replace with...
(diagnostic_manager::set_option_manager): ...this.
(diagnostic_manager::get_lang_mask): Update for field changes.
(diagnostic_manager::m_option_callbacks): Replace with...
(diagnostic_manager::m_option_mgr): ...this and...
(diagnostic_manager::m_lang_mask): ...this.
* lto-wrapper.cc (class lto_diagnostic_option_manager): New.
(main): Port from option hooks to diagnostic_option_manager.
* opts-common.cc: Include "opts-diagnostic.h".
(compiler_diagnostic_option_manager::option_enabled_p): New.
* opts-diagnostic.h (option_name): Drop decl.
(get_option_url): Drop decl.
(class gcc_diagnostic_option_manager): New.
(class compiler_diagnostic_option_manager): New.
* opts.cc (option_name): Convert to...
(compiler_diagnostic_option_manager::make_option_name): ...this.
(get_option_url): Convert to...
(gcc_diagnostic_option_manager::make_option_url): ...this.
* toplev.cc (general_init): Port from option hooks to
diagnostic_option_manager.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic.cc | 23 ---
 gcc/diagnostic.h  | 94 +++
 gcc/lto-wrapper.cc| 24 ---
 gcc/opts-common.cc|  7 
 gcc/opts-diagnostic.h | 44 +---
 gcc/opts.cc   | 18 -
 gcc/toplev.cc | 10 ++---
 7 files changed, 126 insertions(+), 94 deletions(-)

diff --git a/gcc/diagnostic.cc b/gcc/diagnostic.cc
index 32eab7d5407a..0e0ab7aeb838 100644
--- a/gcc/diagnostic.cc
+++ b/gcc/diagnostic.cc
@@ -226,10 +226,7 @@ diagnostic_context::initialize (int n_opts)
   m_text_callbacks.m_begin_diagnostic = default_diagnostic_starter;
   m_text_callbacks.m_start_span = default_diagnostic_start_span_fn;
   m_text_callbacks.m_end_diagnostic = default_diagnostic_finalizer;
-  m_option_callbacks.m_option_enabled_cb = nullptr;
-  m_option_callbacks.m_option_state = nullptr;
-  m_option_callbacks.m_make_option_name_cb = nullptr;
-  m_option_callbacks.m_make_option_url_cb = nullptr;
+  m_option_mgr = nullptr;
   m_urlifier = nullptr;
   m_last_location = UNKNOWN_LOCATION;
   m_last_module = nullptr;
@@ -446,18 +443,12 @@ diagnostic_context::set_original_argv (unique_argv 
original_argv)
 }
 
 void
-diagnostic_context::
-set_option_hooks (diagnostic_option_enabled_cb option_enabled_cb,
- void *option_state,
- diagnostic_make_option_name_cb make_option_name_cb,
- diagnostic_make_option_url_cb make_option_url_cb,
- unsigned lang_mask)
-{
-  m_option_callbacks.m_option_enabled_cb = option_enabled_cb;
-  m_option_callbacks.m_option_state = option_state;
-  m_option_callbacks.m_make_option_name_cb = make_option_name_cb;
-  m_option_callbacks.m_make_option_url_cb = make_option_url_cb;
-  m_option_callbacks.m_lang_mask = lang_mask;
+diagnostic_context::set_option_manager (diagnostic_option_manager *mgr,
+   unsigned lang_mask)
+{
+  delete m_option_mgr;
+  m_option_mgr = mgr;
+  m_lang_mask = lang_mask;
 }
 
 void
diff --git a/gcc/diagnostic.h b/gcc/diagnostic.h
index 4d6147b87458..7244f425936c 100644
--- a/gcc/diagnostic.h
+++ b/gcc/diagnostic.h
@@ -181,14 +181,34 @@ typedef void (*diagnostic_finalizer_fn) 
(diagnostic_context *,
 const diagnostic_info *,
 diagnostic_t);
 
-typedef int (*diagnostic_option_enabled_cb) (int, unsigned, void *);
-typedef char *(*diagnostic_make_option_name_cb) (const diagnostic_context *,
-int,
-diagnostic_t,
-diagnostic_t);
-typedef char *(*diagnostic_make_option_url_cb) (const diagnostic_context *,
-   

[pushed: r15-3554] diagnostics: rename dc.printer to m_printer [PR116613]

[David Malcolm]

:dump): Likewise.
* supergraph.cc (supergraph::dump_dot_to_file): Likewise.
(superedge::dump): Likewise.
* svalue.cc (svalue::dump): Likewise.

gcc/c-family/ChangeLog:
PR other/116613
* c-format.cc (selftest::test_type_mismatch_range_labels): Rename
diagnostic_context's "printer" field to "m_printer".
(selftest::test_type_mismatch_range_labels): Likewise.
* c-opts.cc (c_diagnostic_finalizer): Likewise.

gcc/c/ChangeLog:
PR other/116613
* c-objc-common.cc (c_initialize_diagnostics): Rename
diagnostic_context's "printer" field to "m_printer".

gcc/cp/ChangeLog:
PR other/116613
* error.cc (cxx_initialize_diagnostics): Rename
diagnostic_context's "printer" field to "m_printer".
(cxx_print_error_function): Likewise.
(cp_diagnostic_starter): Likewise.
(cp_print_error_function): Likewise.
(print_instantiation_full_context): Likewise.
(print_instantiation_partial_context_line): Likewise.
(maybe_print_constexpr_context): Likewise.
(print_location): Likewise.
(print_constrained_decl_info): Likewise.
(print_concept_check_info): Likewise.
(print_constraint_context_head): Likewise.
(print_requires_expression_info): Likewise.
* module.cc (noisy_p): Likewise.

gcc/d/ChangeLog:
PR other/116613
* d-diagnostic.cc (d_diagnostic_report_diagnostic): Rename
diagnostic_context's "printer" field to "m_printer".

gcc/fortran/ChangeLog:
PR other/116613
* error.cc (gfc_clear_pp_buffer): Rename diagnostic_context's
"printer" field to "m_printer".
(gfc_warning): Likewise.
(gfc_diagnostic_build_kind_prefix): Likewise.
(gfc_diagnostic_build_locus_prefix): Likewise.
(gfc_diagnostic_starter): Likewise.
(gfc_diagnostic_starter): Likewise.
(gfc_diagnostic_start_span): Likewise.
(gfc_diagnostic_finalizer): Likewise.
(gfc_warning_check): Likewise.
(gfc_error_opt): Likewise.
(gfc_error_check): Likewise.

gcc/jit/ChangeLog:
PR other/116613
* jit-playback.cc (add_diagnostic): Rename diagnostic_context's
"printer" field to "m_printer".

gcc/testsuite/ChangeLog:
PR other/116613
* gcc.dg/plugin/analyzer_cpython_plugin.c (dump_refcnt_info):
Update for renaming of field "printer" to "m_printer".
* gcc.dg/plugin/diagnostic_group_plugin.c
(test_diagnostic_starter): Likewise.
(test_diagnostic_start_span_fn): Likewise.
(test_output_format::on_begin_group): Likewise.
(test_output_format::on_end_group): Likewise.
* gcc.dg/plugin/diagnostic_plugin_test_paths.c: Likewise.
* gcc.dg/plugin/diagnostic_plugin_test_show_locus.c
(custom_diagnostic_finalizer): Likewise.

Signed-off-by: David Malcolm 
---
 gcc/ada/gcc-interface/misc.cc |   6 +-
 gcc/analyzer/access-diagram.cc|   2 +-
 gcc/analyzer/analyzer-language.cc |   2 +-
 gcc/analyzer/analyzer.cc  |   4 +-
 gcc/analyzer/call-details.cc  |   2 +-
 gcc/analyzer/call-summary.cc  |   4 +-
 gcc/analyzer/checker-event.cc |   2 +-
 gcc/analyzer/constraint-manager.cc|   8 +-
 gcc/analyzer/diagnostic-manager.cc|   2 +-
 gcc/analyzer/engine.cc|   6 +-
 gcc/analyzer/kf-analyzer.cc   |   2 +-
 gcc/analyzer/pending-diagnostic.cc|   2 +-
 gcc/analyzer/program-point.cc |   4 +-
 gcc/analyzer/program-state.cc |   6 +-
 gcc/analyzer/ranges.cc|   4 +-
 gcc/analyzer/region-model-reachability.cc |   2 +-
 gcc/analyzer/region-model.cc  |   6 +-
 gcc/analyzer/region.cc|   4 +-
 gcc/analyzer/sm-malloc.cc |   6 +-
 gcc/analyzer/store.cc |  10 +-
 gcc/analyzer/supergraph.cc|   4 +-
 gcc/analyzer/svalue.cc|   2 +-
 gcc/attribs.cc|   6 +-
 gcc/c-family/c-format.cc  |   4 +-
 gcc/c-family/c-opts.cc|  13 +-
 gcc/c/c-objc-common.cc|   4 +-
 gcc/cp/error.cc   |  92 +
 gcc/cp/module.cc  |   2 +-
 gcc/d/d-diagnostic.cc |   5 +-
 gcc/diagnostic-format-json.cc |   9 +-
 gcc/diagnostic-format-sarif.cc|  33 ++--
 gcc/diagnostic-format-text.cc |  36 ++--
 gcc/diagnostic-format.h   |   2 +
 gcc/diagnostic-global-context.cc 

[pushed: r15-3553] SARIF output: fix schema URL [§3.13.3, PR116603]

[David Malcolm]

We were using
  
https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json
as the URL for the SARIF 2.1 schema, but this is now a 404.

Update it to the URL listed in the spec (§3.13.3 "$schema property"),
which is:
  
https://docs.oasis-open.org/sarif/sarif/v2.1.0/errata01/os/schemas/sarif-schema-2.1.0.json
and update the copy in
  gcc/testsuite/lib/sarif-schema-2.1.0.json
used by the "verify-sarif-file" DejaGnu directive to the version found at
that latter URL; the sha256 sum changes
from: 2b19d2358baef0251d7d24e208d05ffabf1b2a3ab5e1b3a816066fc57fd4a7e8
  to: c3b4bb2d6093897483348925aaa73af03b3e3f4bd4ca38cef26dcb4212a2682e

Doing so added a validation error on
  c-c++-common/diagnostic-format-sarif-file-pr111700.c
for which we emit this textual output:
  this-file-does-not-exist.c: warning: #warning message [-Wcpp]
with no line number, and these invalid SARIF regions within the
physical location of the warning:
  "region": {"startColumn": 2,
 "endColumn": 9},
  "contextRegion": {}

This is due to this directive:
  # 0 "this-file-does-not-exist.c"
with line number 0.

The patch fixes this by not creating regions that have startLine <= 0.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3553-g38dc2c64710aa0.

gcc/ChangeLog:
PR other/116603
* diagnostic-format-sarif.cc (SARIF_SCHEMA): Update URL.
(sarif_builder::maybe_make_region_object): Don't create regions
with startLine <= 0.
(sarif_builder::maybe_make_region_object_for_context): Likewise.

gcc/testsuite/ChangeLog:
PR other/116603
* gcc.dg/plugin/diagnostic-test-metadata-sarif.py (test_basics):
Update expected schema URL.
* gcc.dg/plugin/diagnostic-test-paths-multithreaded-sarif.py:
Likewise.
* gcc.dg/sarif-output/test-include-chain-1.py: Likewise.
* gcc.dg/sarif-output/test-include-chain-2.py: Likewise.
* gcc.dg/sarif-output/test-missing-semicolon.py: Likewise.
* gcc.dg/sarif-output/test-no-diagnostics.py: Likewise.
* gcc.dg/sarif-output/test-werror.py: Likewise.
* lib/sarif-schema-2.1.0.json: Update with copy downloaded from

https://docs.oasis-open.org/sarif/sarif/v2.1.0/errata01/os/schemas/sarif-schema-2.1.0.json

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc| 25 +--
 .../plugin/diagnostic-test-metadata-sarif.py  |  2 +-
 ...agnostic-test-paths-multithreaded-sarif.py |  2 +-
 .../sarif-output/test-include-chain-1.py  |  2 +-
 .../sarif-output/test-include-chain-2.py  |  2 +-
 .../sarif-output/test-missing-semicolon.py|  2 +-
 .../sarif-output/test-no-diagnostics.py   |  2 +-
 .../gcc.dg/sarif-output/test-werror.py|  2 +-
 gcc/testsuite/lib/sarif-schema-2.1.0.json | 73 ---
 9 files changed, 72 insertions(+), 40 deletions(-)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 9d9e7ae60734..e95f18f31bda 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -2221,7 +2221,10 @@ sarif_builder::get_sarif_column (expanded_location 
exploc) const
or return nullptr.
 
If COLUMN_OVERRIDE is non-zero, then use it as the column number
-   if LOC has no column information.  */
+   if LOC has no column information.
+
+   We only support text properties of regions ("text regions"),
+   not binary properties ("binary regions"); see 3.30.1.  */
 
 std::unique_ptr
 sarif_builder::maybe_make_region_object (location_t loc,
@@ -2244,11 +2247,16 @@ sarif_builder::maybe_make_region_object (location_t loc,
   if (exploc_finish.file !=exploc_caret.file)
 return nullptr;
 
+  /* We can have line == 0 in the presence of "#" lines.
+ SARIF requires lines > 0, so if we hit this case we don't have a
+ way of validly representing the region as SARIF; bail out.  */
+  if (exploc_start.line <= 0)
+return nullptr;
+
   auto region_obj = ::make_unique ();
 
   /* "startLine" property (SARIF v2.1.0 section 3.30.5) */
-  if (exploc_start.line > 0)
-region_obj->set_integer ("startLine", exploc_start.line);
+  region_obj->set_integer ("startLine", exploc_start.line);
 
   /* "startColumn" property (SARIF v2.1.0 section 3.30.6).
 
@@ -2316,11 +2324,16 @@ maybe_make_region_object_for_context (location_t loc,
   if (exploc_finish.file !=exploc_caret.file)
 return nullptr;
 
+  /* We can have line == 0 in the presence of "#" lines.
+ SARIF requires lines > 0, so if we hit this case we don't have a
+ way of validly representing the region as SARIF; bail out.  */
+  if (exploc_start.line <= 0)
+return nullptr;
+
   auto region_obj = ::make_unique ();
 
   /* "startLine" property (SARIF 

[pushed: r15-3551] analyzer: fix "unused variable 'summary_cast_reg'" warning

[David Malcolm]

I missed this in r15-1108-g70f26314b62e2d.

Successfully bootstrapped on x86_64-pc-linux-gnu.

Pushed as r15-3551-g6e35b0e8572a71.

gcc/analyzer/ChangeLog:
* call-summary.cc
(call_summary_replay::convert_region_from_summary_1): Drop unused
local "summary_cast_reg"

Signed-off-by: David Malcolm 
---
 gcc/analyzer/call-summary.cc | 2 --
 1 file changed, 2 deletions(-)

diff --git a/gcc/analyzer/call-summary.cc b/gcc/analyzer/call-summary.cc
index ec36fdfd9f1a..663a89567a79 100644
--- a/gcc/analyzer/call-summary.cc
+++ b/gcc/analyzer/call-summary.cc
@@ -724,8 +724,6 @@ call_summary_replay::convert_region_from_summary_1 (const 
region *summary_reg)
   break;
 case RK_CAST:
   {
-   const cast_region *summary_cast_reg
- = as_a  (summary_reg);
const region *summary_parent_reg = summary_reg->get_parent_region ();
const region *caller_parent_reg
  = convert_region_from_summary (summary_parent_reg);
-- 
2.26.3

Re: Handle 'NUM' in 'PUSH_INSERT_PASSES_WITHIN' (was: [PATCH 03/11] Handwritten part of conversion of passes to C++ classes)

[David Malcolm]

On Fri, 2024-06-28 at 15:06 +0200, Thomas Schwinge wrote:
> Hi!
> 
> As part of this:
> 
> On 2013-07-26T11:04:33-0400, David Malcolm 
> wrote:
> > This patch is the hand-written part of the conversion of passes
> > from
> > C structs to C++ classes.
> 
> > --- a/gcc/passes.c
> > +++ b/gcc/passes.c
> 
> ..., we did hard-code 'PUSH_INSERT_PASSES_WITHIN(PASS)' to always
> refer
> to the first instance of 'PASS':
> 
> >  #define PUSH_INSERT_PASSES_WITHIN(PASS) \
> >    { \
> > -    struct opt_pass **p = &(PASS).pass.sub;
> > +    struct opt_pass **p = &(PASS ## _1)->sub;
> 
> ..., however we did change 'NEXT_PASS(PASS, NUM)' to actually use
> 'NUM':
> 
> > -#define NEXT_PASS(PASS, NUM)  (p = next_pass_1 (p,
> > &((PASS).pass)))
> > +#define NEXT_PASS(PASS, NUM) \
> > +  do { \
> > +    gcc_assert (NULL == PASS ## _ ## NUM); \
> > +    if ((NUM) == 1)  \
> > +  PASS ## _1 = make_##PASS (ctxt_);  \
> > +    else \
> > +  {  \
> > +    gcc_assert (PASS ## _1); \
> > +    PASS ## _ ## NUM = PASS ## _1->clone (); \
> > +  }  \
> > +    p = next_pass_1 (p, PASS ## _ ## NUM);  \
> > +  } while (0)
> 
> This was never re-synchronized later on, and is problematic if you
> try to
> do something like this; change:
> 
>     [...]
>     NEXT_PASS (pass_postreload);
>     PUSH_INSERT_PASSES_WITHIN (pass_postreload)
>     NEXT_PASS (pass_postreload_cse);
>     [...]
>     NEXT_PASS (pass_cprop_hardreg);
>     NEXT_PASS (pass_fast_rtl_dce);
>     NEXT_PASS (pass_reorder_blocks);
>     [...]
>     POP_INSERT_PASSES ()
>     [...]
> 
> ... into:
> 
>     [...]
>     NEXT_PASS (pass_postreload);
>     PUSH_INSERT_PASSES_WITHIN (pass_postreload)
>     NEXT_PASS (pass_postreload_cse);
>     [...]
>     NEXT_PASS (pass_cprop_hardreg);
>     POP_INSERT_PASSES ()
>     NEXT_PASS (pass_fast_rtl_dce);
>     NEXT_PASS (pass_postreload);
>     PUSH_INSERT_PASSES_WITHIN (pass_postreload)
>     NEXT_PASS (pass_reorder_blocks);
>     [...]
>     POP_INSERT_PASSES ()
>     [...]
> 
> That is, interrupt the pass pipeline within 'pass_postreload', in
> order
> to unconditionally run 'pass_fast_rtl_dce' even if not running
> 'pass_postreload'.  What happens is that the second
> 'PUSH_INSERT_PASSES_WITHIN (pass_postreload)' overwrites the first
> 'PUSH_INSERT_PASSES_WITHIN (pass_postreload)' instead of applying to
> the
> second (preceding) 'NEXT_PASS (pass_postreload);'.
> 
> (I ran into this in context of what I tried in
> <https://inbox.sourceware.org/87ed8i2ekt@euler.schwinge.ddns.net>
> "nvptx vs. [PATCH] Add a late-combine pass [PR106594]"; discuss that
> specific use case over there, not here.)
> 
> OK to address this with the attached
> "Handle 'NUM' in 'PUSH_INSERT_PASSES_WITHIN'"?
> 
> This depends on
> <https://inbox.sourceware.org/87jzi9tgcw@euler.schwinge.ddns.net>
> "Rewrite usage comment at the top of 'gcc/passes.def'" to avoid
> running
> into the 'ERROR: Can't locate [...]' that I'm adding, while
> processing
> the 'PUSH_INSERT_PASSES_WITHIN (PASS)' in the usage comment at the
> top of
> 'gcc/passes.def', where 'NEXT_PASS (PASS)' only appears later.  ;-)
> 
> I've verified this does the expected thing for the main
> 'gcc/passes.def',
> and that 'PUSH_INSERT_PASSES_WITHIN' is not used/not applicable for
> 'PASSES_EXTRA' ('gcc/config/*/*-passes.def').

Thanks; patch LGTM.

Dave

[pushed 3/3] pretty-print: split up pretty_printer::format into subroutines

[David Malcolm]

The body of pretty_printer::format is almost 500 lines long,
mostly comprising two distinct phases.

This patch splits it up so that there are explicit subroutines
for the two different phases, reducing the scope of various
locals, and making it easier to e.g. put a breakpoint on phase 2.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3431-g07e74798b93c25.

gcc/ChangeLog:
* pretty-print-markup.h (pp_markup::context::context): Drop
params "buf" and "chunk_idx", initializing m_buf from pp.
pp_markup::context::m_chunk_idx): Drop field.
* pretty-print.cc (pretty_printer::format): Convert param
from a text_info * to a text_info &.  Split out phase 1
and phase 2 into subroutines...
(format_phase_1): New, from pretty_printer::format.
(format_phase_2): Likewise.
* pretty-print.h (pretty_printer::format): Convert param
from a text_info * to a text_info &.
(pp_format): Update for above change.  Assert that text_info is
    non-null.

Signed-off-by: David Malcolm 
---
 gcc/pretty-print-markup.h |   6 +-
 gcc/pretty-print.cc   | 232 +-
 gcc/pretty-print.h|   5 +-
 3 files changed, 131 insertions(+), 112 deletions(-)

diff --git a/gcc/pretty-print-markup.h b/gcc/pretty-print-markup.h
index ce2c5e9dbbe9..de9e4bda6ade 100644
--- a/gcc/pretty-print-markup.h
+++ b/gcc/pretty-print-markup.h
@@ -30,13 +30,10 @@ class context
 {
 public:
   context (pretty_printer &pp,
-  output_buffer &buf,
-  unsigned chunk_idx,
   bool "ed,
   pp_token_list *formatted_token_list)
   : m_pp (pp),
-m_buf (buf),
-m_chunk_idx (chunk_idx),
+m_buf (*pp_buffer (&pp)),
 m_quoted (quoted),
 m_formatted_token_list (formatted_token_list)
   {
@@ -52,7 +49,6 @@ public:
 
   pretty_printer &m_pp;
   output_buffer &m_buf;
-  unsigned m_chunk_idx;
   bool &m_quoted;
   pp_token_list *m_formatted_token_list;
 };
diff --git a/gcc/pretty-print.cc b/gcc/pretty-print.cc
index 115f376c4512..998e06e155f7 100644
--- a/gcc/pretty-print.cc
+++ b/gcc/pretty-print.cc
@@ -1589,35 +1589,79 @@ push_back_any_text (pp_token_list *tok_list,
Phase 3 is in pp_output_formatted_text, which pops the pp_formatted_chunks
instance.  */
 
+static void
+format_phase_1 (const text_info &text,
+   obstack &chunk_obstack,
+   pp_token_list **args,
+   pp_token_list ***formatters);
+
+static void
+format_phase_2 (pretty_printer *pp,
+   text_info &text,
+   obstack &chunk_obstack,
+   pp_token_list ***formatters);
+
 void
-pretty_printer::format (text_info *text)
+pretty_printer::format (text_info &text)
 {
-  output_buffer * const buffer = m_buffer;
+  pp_formatted_chunks *new_chunk_array = m_buffer->push_formatted_chunks ();
+  pp_token_list **args = new_chunk_array->m_args;
 
-  unsigned int chunk = 0, argno;
   pp_token_list **formatters[PP_NL_ARGMAX];
-
-  pp_formatted_chunks *new_chunk_array = buffer->push_formatted_chunks ();
-  pp_token_list **args = new_chunk_array->m_args;
+  memset (formatters, 0, sizeof formatters);
 
   /* Formatting phase 1: split up TEXT->format_spec into chunks in
  pp_buffer (PP)->args[].  Even-numbered chunks are to be output
  verbatim, odd-numbered chunks are format specifiers.
  %m, %%, %<, %>, %} and %' are replaced with the appropriate text at
  this point.  */
+  format_phase_1 (text, m_buffer->m_chunk_obstack, args, formatters);
 
-  memset (formatters, 0, sizeof formatters);
+  /* Note that you can debug the state of the chunk arrays here using
+   (gdb) call m_buffer->cur_chunk_array->dump()
+ which, given e.g. "foo: %s bar: %s" might print:
+   0: [TEXT("foo: ")]
+   1: [TEXT("s")]
+   2: [TEXT(" bar: ")]
+   3: [TEXT("s")]
+  */
+
+  /* Set output to the argument obstack, and switch line-wrapping and
+ prefixing off.  */
+  m_buffer->m_obstack = &m_buffer->m_chunk_obstack;
+  const int old_line_length = m_buffer->m_line_length;
+  const pp_wrapping_mode_t old_wrapping_mode = pp_set_verbatim_wrapping (this);
+
+  format_phase_2 (this, text, m_buffer->m_chunk_obstack, formatters);
+
+  /* If the client supplied a postprocessing object, call its "handle"
+ hook here.  */
+  if (m_format_postprocessor)
+m_format_postprocessor->handle (this);
+
+  /* Revert to normal obstack and wrapping mode.  */
+  m_buffer->m_obstack = &m_buffer->m_formatted_obstack;
+  m_buffer->m_line_length = old_line_length;
+  pp_wrapping_mode (this) = old_wrapping_mode;
+  clear_state ();
+}
 
+static void
+format_phase_1 (const text_info &text,
+ 

[pushed 1/3] pretty-print: naming cleanups

[David Malcolm]

put_buffer::m_cur_formatted_chunks): ...this.
(output_buffer::stream): Rename to...
(output_buffer::m_stream): ...this.
(output_buffer::line_length): Rename to...
(output_buffer::m_line_length): ...this.
(output_buffer::digit_buffer): Rename to...
(output_buffer::m_digit_buffer): ...this.
(output_buffer::flush_p): Rename to...
(output_buffer::m_flush_p): ...this.
(output_buffer_formatted_text): Prefix all output_buffer fields
with "m_".
(output_buffer_append_r): Likewise.
(output_buffer_last_position_in_text): Likewise.
(pretty_printer::set_output_stream): Likewise.
(pp_scalar): Likewise.
(pp_wide_int): Likewise.
* tree-pretty-print.cc (dump_generic_node): Likewise.
(dump_generic_node): Likewise.
(pp_double_int): Likewise.

Signed-off-by: David Malcolm 
---
 gcc/analyzer/analyzer-logging.cc |   2 +-
 gcc/c-family/c-ada-spec.cc   |   6 +-
 gcc/c-family/c-pretty-print.cc   |  18 +--
 gcc/c/c-objc-common.cc   |   4 +-
 gcc/cp/error.cc  |   6 +-
 gcc/diagnostic.cc|   2 +-
 gcc/dumpfile.cc  |   6 +-
 gcc/fortran/error.cc |   6 +-
 gcc/gimple-pretty-print.cc   |   2 +-
 gcc/json.cc  |   2 +-
 gcc/pretty-print-format-impl.h   |  34 +++---
 gcc/pretty-print.cc  | 192 +--
 gcc/pretty-print.h   |  52 +
 gcc/tree-pretty-print.cc |  10 +-
 14 files changed, 182 insertions(+), 160 deletions(-)

diff --git a/gcc/analyzer/analyzer-logging.cc b/gcc/analyzer/analyzer-logging.cc
index cceb4fe24a55..d3b04f2d9b10 100644
--- a/gcc/analyzer/analyzer-logging.cc
+++ b/gcc/analyzer/analyzer-logging.cc
@@ -51,7 +51,7 @@ logger::logger (FILE *f_out,
   m_pp (reference_pp.clone ())
 {
   pp_show_color (m_pp) = 0;
-  pp_buffer (m_pp)->stream = f_out;
+  pp_buffer (m_pp)->m_stream = f_out;
 
   /* %qE in logs for SSA_NAMEs should show the ssa names, rather than
  trying to prettify things by showing the underlying var.  */
diff --git a/gcc/c-family/c-ada-spec.cc b/gcc/c-family/c-ada-spec.cc
index e1b1b2a4b73f..fbba44362540 100644
--- a/gcc/c-family/c-ada-spec.cc
+++ b/gcc/c-family/c-ada-spec.cc
@@ -2485,13 +2485,13 @@ dump_ada_node (pretty_printer *pp, tree node, tree 
type, int spc,
  pp_minus (pp);
  val = -val;
}
- sprintf (pp_buffer (pp)->digit_buffer,
+ sprintf (pp_buffer (pp)->m_digit_buffer,
   "16#%" HOST_WIDE_INT_PRINT "x",
   val.elt (val.get_len () - 1));
  for (i = val.get_len () - 2; i >= 0; i--)
-   sprintf (pp_buffer (pp)->digit_buffer,
+   sprintf (pp_buffer (pp)->m_digit_buffer,
 HOST_WIDE_INT_PRINT_PADDED_HEX, val.elt (i));
- pp_string (pp, pp_buffer (pp)->digit_buffer);
+ pp_string (pp, pp_buffer (pp)->m_digit_buffer);
}
   break;
 
diff --git a/gcc/c-family/c-pretty-print.cc b/gcc/c-family/c-pretty-print.cc
index dd7eba123943..26038818d759 100644
--- a/gcc/c-family/c-pretty-print.cc
+++ b/gcc/c-family/c-pretty-print.cc
@@ -1043,7 +1043,7 @@ pp_c_integer_constant (c_pretty_printer *pp, tree i)
  wi = -wi;
}
   unsigned int prec = wi.get_precision ();
-  if ((prec + 3) / 4 > sizeof (pp_buffer (pp)->digit_buffer) - 3)
+  if ((prec + 3) / 4 > sizeof (pp_buffer (pp)->m_digit_buffer) - 3)
{
  char *buf = XALLOCAVEC (char, (prec + 3) / 4 + 3);
  print_hex (wi, buf);
@@ -1051,8 +1051,8 @@ pp_c_integer_constant (c_pretty_printer *pp, tree i)
}
   else
{
- print_hex (wi, pp_buffer (pp)->digit_buffer);
- pp_string (pp, pp_buffer (pp)->digit_buffer);
+ print_hex (wi, pp_buffer (pp)->m_digit_buffer);
+ pp_string (pp, pp_buffer (pp)->m_digit_buffer);
}
 }
 }
@@ -1138,11 +1138,11 @@ pp_c_floating_constant (c_pretty_printer *pp, tree r)
  log10(2) to 7 significant digits.  */
   int max_digits10 = 2 + (is_decimal ? fmt->p : fmt->p * 643L / 2136);
 
-  real_to_decimal (pp_buffer (pp)->digit_buffer, &TREE_REAL_CST (r),
-  sizeof (pp_buffer (pp)->digit_buffer),
+  real_to_decimal (pp_buffer (pp)->m_digit_buffer, &TREE_REAL_CST (r),
+  sizeof (pp_buffer (pp)->m_digit_buffer),
   max_digits10, 1);
 
-  pp_string (pp, pp_buffer(pp)->digit_buffer);
+  pp_string (pp, pp_buffer(pp)->m_digit_buffer);
   if (TREE_TYPE (r) == float_type_node)
 pp_character (pp, 'f');
   else if (TREE_TYPE (r) == long_double_type_node)
@@ -1170,9 +1170,9 @@ pp_c_floating_constant (c_pretty_printer *pp, tree r)
 static void
 pp_c_fixed_constant (c_pretty_printer *pp, tree r)
 {
-  fixed_to_decimal (pp_buffer (pp)->

[pushed 2/3] pretty-print: add selftest of pp_format's stack

[David Malcolm]

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3430-gd0891f3aa75d31.

gcc/ChangeLog:
* pretty-print-format-impl.h (pp_formatted_chunks::get_prev): New
accessor.
* pretty-print.cc (selftest::push_pp_format): New.
(ASSERT_TEXT_TOKEN): New macro.
(selftest::test_pp_format_stack): New test.
(selftest::pretty_print_cc_tests): New.

Signed-off-by: David Malcolm 
---
 gcc/pretty-print-format-impl.h |  3 ++
 gcc/pretty-print.cc| 78 ++
 2 files changed, 81 insertions(+)

diff --git a/gcc/pretty-print-format-impl.h b/gcc/pretty-print-format-impl.h
index c70f61ce1bab..ec4425c9dafb 100644
--- a/gcc/pretty-print-format-impl.h
+++ b/gcc/pretty-print-format-impl.h
@@ -376,6 +376,9 @@ public:
   void dump (FILE *out) const;
   void DEBUG_FUNCTION dump () const { dump (stderr); }
 
+  // For use in selftests
+  pp_formatted_chunks *get_prev () const { return m_prev; }
+
 private:
   /* Pointer to previous level on the stack.  */
   pp_formatted_chunks *m_prev;
diff --git a/gcc/pretty-print.cc b/gcc/pretty-print.cc
index 50aea69edd62..115f376c4512 100644
--- a/gcc/pretty-print.cc
+++ b/gcc/pretty-print.cc
@@ -3547,6 +3547,83 @@ test_custom_tokens_2 ()
"print_tokens was called");
 }
 
+/* Helper subroutine for test_pp_format_stack.
+   Call pp_format (phases 1 and 2), without calling phase 3.  */
+
+static void
+push_pp_format (pretty_printer *pp, const char *msg, ...)
+{
+  va_list ap;
+
+  va_start (ap, msg);
+  rich_location rich_loc (line_table, UNKNOWN_LOCATION);
+  text_info ti (msg, &ap, 0, nullptr, &rich_loc);
+  pp_format (pp, &ti);
+  va_end (ap);
+}
+
+#define ASSERT_TEXT_TOKEN(TOKEN, EXPECTED_TEXT)\
+  SELFTEST_BEGIN_STMT  \
+ASSERT_NE ((TOKEN), nullptr);  \
+ASSERT_EQ ((TOKEN)->m_kind, pp_token::kind::text); \
+ASSERT_STREQ   \
+  (as_a  (TOKEN)->m_value.get (),   \
+   (EXPECTED_TEXT));   \
+  SELFTEST_END_STMT
+
+
+/* Verify that the stack of pp_formatted_chunks works as expected.  */
+
+static void
+test_pp_format_stack ()
+{
+  auto_fix_quotes fix_quotes;
+
+  pretty_printer pp;
+  push_pp_format (&pp, "unexpected foo: %i bar: %qs", 42, "test");
+  push_pp_format (&pp, "In function: %qs", "test_fn");
+
+  /* Expect the top of the stack to have:
+ (gdb) call top->dump()
+ 0: [TEXT("In function: ")]
+ 1: [BEGIN_QUOTE, TEXT("test_fn"), END_QUOTE].  */
+
+  pp_formatted_chunks *top = pp_buffer (&pp)->m_cur_formatted_chunks;
+  ASSERT_NE (top, nullptr);
+  ASSERT_TEXT_TOKEN (top->get_token_lists ()[0]->m_first, "In function: ");
+  ASSERT_EQ (top->get_token_lists ()[1]->m_first->m_kind,
+pp_token::kind::begin_quote);
+  ASSERT_EQ (top->get_token_lists ()[2], nullptr);
+
+  /* Expect an entry in the stack below it with:
+ 0: [TEXT("unexpected foo: ")]
+ 1: [TEXT("42")]
+ 2: [TEXT(" bar: ")]
+ 3: [BEGIN_QUOTE, TEXT("test"), END_QUOTE].  */
+  pp_formatted_chunks *prev = top->get_prev ();
+  ASSERT_NE (prev, nullptr);
+  ASSERT_TEXT_TOKEN (prev->get_token_lists ()[0]->m_first, "unexpected foo: ");
+  ASSERT_TEXT_TOKEN (prev->get_token_lists ()[1]->m_first, "42");
+  ASSERT_TEXT_TOKEN (prev->get_token_lists ()[2]->m_first, " bar: ");
+  ASSERT_EQ (prev->get_token_lists ()[3]->m_first->m_kind,
+pp_token::kind::begin_quote);
+  ASSERT_EQ (prev->get_token_lists ()[4], nullptr);
+
+  ASSERT_EQ (prev->get_prev (), nullptr);
+
+  /* Pop the top of the stack.  */
+  pp_output_formatted_text (&pp);
+  ASSERT_EQ (pp_buffer (&pp)->m_cur_formatted_chunks, prev);
+  pp_newline (&pp);
+
+  /* Pop the remaining entry from the stack.  */
+  pp_output_formatted_text (&pp);
+  ASSERT_EQ (pp_buffer (&pp)->m_cur_formatted_chunks, nullptr);
+
+  ASSERT_STREQ (pp_formatted_text (&pp),
+   "In function: `test_fn'\nunexpected foo: 42 bar: `test'");
+}
+
 /* A subclass of pretty_printer for use by test_prefixes_and_wrapping.  */
 
 class test_pretty_printer : public pretty_printer
@@ -3976,6 +4053,7 @@ pretty_print_cc_tests ()
   test_merge_consecutive_text_tokens ();
   test_custom_tokens_1 ();
   test_custom_tokens_2 ();
+  test_pp_format_stack ();
   test_prefixes_and_wrapping ();
   test_urls ();
   test_urls_from_braces ();
-- 
2.26.3

Re: [PATCH] gdbhooks: Fix printing of vec with vl_ptr layout

[David Malcolm]

On Fri, 2024-08-30 at 12:08 +0100, Alex Coplan wrote:
> Hi,
> 
> As it stands, the pretty printing of GCC's vecs by gdbhooks.py only
> handles vectors with vl_embed layout.  As such, when encountering a
> vec
> with vl_ptr layout, GDB would print a diagnostic like:
> 
>   gdb.error: There is no member or method named m_vecpfx.
> 
> when (e.g.) any such vec occurred in a backtrace.  This patch extends
> VecPrinter.children to also handle vl_ptr vectors.
> 
> Manually tested by verifying that vl_embed vectors still print
> correctly
> and empty vl_ptr vectors no longer trigger errors.
> 
> OK for trunk? 

Thanks for fixing this.

+else:
+assert False, f"unxpected vec kind {kind}"

Typo: "unxpected" -> "unexpected"

Otherwise, looks good for trunk (with a ChangeLog).

Thanks again
Dave

[pushed 3/4] pretty-print: reimplement pp_format with a new struct pp_token

[David Malcolm]

list.
(pp_markup::context::end_quote): Likewise.
(pp_markup::context::begin_highlight_color): Likewise.
(pp_markup::context::end_highlight_color): Likewise.
(pp_markup::context::push_back_any_text): New.
(selftest::test_merge_consecutive_text_tokens): New.
(selftest::test_custom_tokens_1): New.
(selftest::test_custom_tokens_2): New.
(selftest::pp_printf_with_urlifier): Drop "urlifier" param from
call to pp_format.
(selftest::test_urlification): Add test of the example from
pretty-print-format-impl.h.
(selftest::pretty_print_cc_tests): Call the new selftest
functions.
* pretty-print.h (class quoting_info): Drop forward decl.
(class pp_token_list): New forward decl.
(printer_fn): Convert final param from const char ** to
pp_token_list &.
(class token_printer): New.
(class pretty_printer): Add pp_output_formatted_text as friend.
(pretty_printer::set_token_printer): New.
(pretty_printer::format): Drop urlifier param as this now happens
in phase 3.
(pretty_printer::m_format_decoder): Update comment.
(pretty_printer::m_token_printer): New field.
(pp_format): Drop urlifier param.
* tree-diagnostic.cc (default_tree_printer): Convert final param
from const char ** to pp_token_list &.
* tree-diagnostic.h: Likewise for decl.

gcc/fortran/ChangeLog:
    * error.cc (gfc_format_decoder): Convert final param from
const char **buffer_ptr to pp_token_list &formatted_token_list,
and update call to default_tree_printer accordingly.

Signed-off-by: David Malcolm 
---
 gcc/c/c-objc-common.cc |4 +-
 gcc/cp/error.cc|  105 +--
 gcc/diagnostic.cc  |2 +-
 gcc/dump-context.h |   40 +-
 gcc/dumpfile.cc|  217 +++---
 gcc/fortran/error.cc   |5 +-
 gcc/opt-problem.cc |3 +-
 gcc/pretty-print-format-impl.h |  407 ++-
 gcc/pretty-print-markup.h  |   10 +-
 gcc/pretty-print-urlifier.h|2 +-
 gcc/pretty-print.cc| 1176 ++--
 gcc/pretty-print.h |   43 +-
 gcc/tree-diagnostic.cc |2 +-
 gcc/tree-diagnostic.h  |2 +-
 14 files changed, 1483 insertions(+), 535 deletions(-)

diff --git a/gcc/c/c-objc-common.cc b/gcc/c/c-objc-common.cc
index fde9ae6ad667..9d39fcd4e442 100644
--- a/gcc/c/c-objc-common.cc
+++ b/gcc/c/c-objc-common.cc
@@ -34,7 +34,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "dwarf2.h"
 
 static bool c_tree_printer (pretty_printer *, text_info *, const char *,
-   int, bool, bool, bool, bool *, const char **);
+   int, bool, bool, bool, bool *, pp_token_list &);
 
 /* Info for C language features which can be queried through
__has_{feature,extension}.  */
@@ -318,7 +318,7 @@ pp_markup::element_quoted_type::print_type 
(pp_markup::context &ctxt)
 static bool
 c_tree_printer (pretty_printer *pp, text_info *text, const char *spec,
int precision, bool wide, bool set_locus, bool hash,
-   bool *quoted, const char **)
+   bool *quoted, pp_token_list &)
 {
   tree t = NULL_TREE;
   // FIXME: the next cast should be a dynamic_cast, when it is permitted.
diff --git a/gcc/cp/error.cc b/gcc/cp/error.cc
index 3cc0dd1cdfa9..420fad26b7b7 100644
--- a/gcc/cp/error.cc
+++ b/gcc/cp/error.cc
@@ -39,6 +39,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "cp-name-hint.h"
 #include "attribs.h"
 #include "pretty-print-format-impl.h"
+#include "make-unique.h"
 
 #define pp_separate_with_comma(PP) pp_cxx_separate_with (PP, ',')
 #define pp_separate_with_semicolon(PP) pp_cxx_separate_with (PP, ';')
@@ -110,7 +111,7 @@ static void cp_print_error_function (diagnostic_context *,
 const diagnostic_info *);
 
 static bool cp_printer (pretty_printer *, text_info *, const char *,
-   int, bool, bool, bool, bool *, const char **);
+   int, bool, bool, bool, bool *, pp_token_list &);
 
 /* Color names for highlighting "%qH" vs "%qI" values,
and ranges corresponding to them.  */
@@ -124,22 +125,50 @@ class deferred_printed_type
 {
 public:
   deferred_printed_type ()
-  : m_tree (NULL_TREE), m_buffer_ptr (NULL), m_verbose (false), m_quote (false)
+  : m_tree (NULL_TREE),
+m_printed_text (),
+m_token_list (nullptr),
+m_verbose (false), m_quote (false)
   {}
 
-  deferred_printed_type (tree type, const char **buffer_ptr, bool verbose,
+  deferred_printed_type (tree type,
+pp_token_list &token_list,
+bool verbose,
 bool quote)
-  : m_tree (t

[pushed 4/4] SARIF output: implement embedded URLs in messages (§3.11.6; PR other/116419)

[David Malcolm]

t for the context.
(diagnostic_output_format_init_sarif_stderr): Move responsibility
for setting the context's output format to within
diagnostic_output_format_init_sarif.
(diagnostic_output_format_init_sarif_file): Likewise.
(diagnostic_output_format_init_sarif_stream): Likewise.
(test_sarif_diagnostic_context::test_sarif_diagnostic_context):
Likewise.
(selftest::test_make_location_object): Provide an idx for the
result.
(selftest::get_result_from_log): New.
(selftest::get_message_from_log): New.
(selftest::test_message_with_embedded_link): New test.
(selftest::diagnostic_format_sarif_cc_tests): Call it.
* pretty-print-format-impl.h: Include "diagnostic-event-id.h".
(pp_token::kind): Add "event_id".
(struct pp_token_event_id): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
* pretty-print.cc (pp_token::dump): Handle kind::event_id.
(pretty_printer::format): Update handling of "%@" in phase 2
so that we add a pp_token_event_id, rather that the text "(N)".
(default_token_printer): Handle pp_token::kind::event_id by
printing the text "(N)".

gcc/testsuite/ChangeLog:
PR other/116419
* gcc.dg/sarif-output/bad-pragma.c: New test.
* gcc.dg/sarif-output/test-bad-pragma.py: New test.
* gcc.dg/sarif-output/test-include-chain-2.py
(test_location_relationships): Update expected text of event to
include an intra-sarif URI to the other event.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-event-id.h |   6 +
 gcc/diagnostic-format-sarif.cc| 605 +++---
 gcc/pretty-print-format-impl.h|  32 +
 gcc/pretty-print.cc   |  29 +-
 .../gcc.dg/sarif-output/bad-pragma.c  |  16 +
 .../gcc.dg/sarif-output/test-bad-pragma.py|  38 ++
 .../sarif-output/test-include-chain-2.py  |   6 +-
 7 files changed, 636 insertions(+), 96 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/sarif-output/bad-pragma.c
 create mode 100644 gcc/testsuite/gcc.dg/sarif-output/test-bad-pragma.py

diff --git a/gcc/diagnostic-event-id.h b/gcc/diagnostic-event-id.h
index 78c2ccbbc99d..8237ba34df33 100644
--- a/gcc/diagnostic-event-id.h
+++ b/gcc/diagnostic-event-id.h
@@ -41,6 +41,12 @@ class diagnostic_event_id_t
 
   bool known_p () const { return m_index != UNKNOWN_EVENT_IDX; }
 
+  int zero_based () const
+  {
+gcc_assert (known_p ());
+return m_index;
+  }
+
   int one_based () const
   {
 gcc_assert (known_p ());
diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 59d9cd721839..9d9e7ae60734 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -45,6 +45,8 @@ along with GCC; see the file COPYING3.  If not see
 #include "selftest-diagnostic-show-locus.h"
 #include "selftest-json.h"
 #include "text-range-label.h"
+#include "pretty-print-format-impl.h"
+#include "pretty-print-urlifier.h"
 
 /* Forward decls.  */
 class sarif_builder;
@@ -385,7 +387,12 @@ private:
 class sarif_result : public sarif_location_manager
 {
 public:
-  sarif_result () : m_related_locations_arr (nullptr) {}
+  sarif_result (unsigned idx_within_parent)
+: m_related_locations_arr (nullptr),
+  m_idx_within_parent (idx_within_parent)
+  {}
+
+  unsigned get_index_within_parent () const { return m_idx_within_parent; }
 
   void
   on_nested_diagnostic (diagnostic_context &context,
@@ -402,6 +409,7 @@ public:
 
 private:
   json::array *m_related_locations_arr; // borrowed
+  const unsigned m_idx_within_parent;
 };
 
 /* Subclass of sarif_object for SARIF "location" objects
@@ -460,7 +468,39 @@ private:
 /* Subclass of sarif_object for SARIF "codeFlow" objects
(SARIF v2.1.0 section 3.36).  */
 
-class sarif_code_flow : public sarif_object {};
+class sarif_code_flow : public sarif_object
+{
+public:
+  sarif_code_flow (sarif_result &parent,
+  unsigned idx_within_parent);
+
+  sarif_result &get_parent () const { return m_parent; }
+  unsigned get_index_within_parent () const { return m_idx_within_parent; }
+
+  sarif_thread_flow &
+  get_or_append_thread_flow (const diagnostic_thread &thread,
+diagnostic_thread_id_t thread_id);
+
+  sarif_thread_flow &
+  get_thread_flow (diagnostic_thread_id_t thread_id);
+
+  void add_location (sarif_thread_flow_location &);
+
+  sarif_thread_flow_location &
+  get_thread_flow_loc_obj (diagnostic_event_id_t event_id) const;
+
+private:
+  sarif_result &m_parent;
+  const unsigned m_idx_within_parent;
+
+  hash_map,
+  sarif_thread_flow *> m_thread_id_map; // borrowed ptr
+  json::array *m_thread_flows_arr; // borrowed
+
+  /* Vec of borrowed ptr, allo

[pushed 1/4] Use std::unique_ptr for optinfo_item

[David Malcolm]

As preliminary work towards an overhaul of how optinfo_items
interact with dump_pretty_printer, replace uses of optinfo_item * with
std::unique_ptr to make ownership clearer.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Successfully built stage1 cc1 on all configs via config-list.mk.
Pushed to trunk as r15-3309-g464a3d2fe53362.

gcc/ChangeLog:
* config/aarch64/aarch64.cc: Define INCLUDE_MEMORY.
* config/arm/arm.cc: Likewise.
* config/i386/i386.cc: Likewise.
* config/loongarch/loongarch.cc: Likewise.
* config/riscv/riscv-vector-costs.cc: Likewise.
* config/riscv/riscv.cc: Likewise.
* config/rs6000/rs6000.cc: Likewise.
* dump-context.h (dump_context::emit_item): Convert "item" param
from * to const &.
(dump_pretty_printer::stash_item): Convert "item" param from
optinfo_ * to std::unique_ptr.
(dump_pretty_printer::emit_item): Likewise.
* dumpfile.cc: Include "make-unique.h".
(make_item_for_dump_gimple_stmt): Replace uses of optinfo_item *
with std::unique_ptr.
(dump_context::dump_gimple_stmt): Likewise.
(make_item_for_dump_gimple_expr): Likewise.
(dump_context::dump_gimple_expr): Likewise.
(make_item_for_dump_generic_expr): Likewise.
(dump_context::dump_generic_expr): Likewise.
(make_item_for_dump_symtab_node): Likewise.
(dump_pretty_printer::emit_items): Likewise.
(dump_pretty_printer::emit_any_pending_textual_chunks): Likewise.
(dump_pretty_printer::emit_item): Likewise.
(dump_pretty_printer::stash_item): Likewise.
(dump_pretty_printer::decode_format): Likewise.
(dump_context::dump_printf_va): Fix overlong line.
(make_item_for_dump_dec): Replace uses of optinfo_item * with
std::unique_ptr.
(dump_context::dump_dec): Likewise.
(dump_context::dump_symtab_node): Likewise.
(dump_context::begin_scope): Likewise.
(dump_context::emit_item): Likewise.
* gimple-loop-interchange.cc: Define INCLUDE_MEMORY.
* gimple-loop-jam.cc: Likewise.
* gimple-loop-versioning.cc: Likewise.
* graphite-dependences.cc: Likewise.
* graphite-isl-ast-to-gimple.cc: Likewise.
* graphite-optimize-isl.cc: Likewise.
* graphite-poly.cc: Likewise.
* graphite-scop-detection.cc: Likewise.
* graphite-sese-to-poly.cc: Likewise.
* graphite.cc: Likewise.
* opt-problem.cc: Likewise.
* optinfo.cc (optinfo::add_item): Convert "item" param from
optinfo_ * to std::unique_ptr.
(optinfo::emit_for_opt_problem): Update for change to
dump_context::emit_item.
* optinfo.h: Add #error to fail immediately if INCLUDE_MEMORY
wasn't defined, rather than fail to find std::unique_ptr.
(optinfo::add_item): Convert "item" param from optinfo_ * to
std::unique_ptr.
* sese.cc: Define INCLUDE_MEMORY.
* targhooks.cc: Likewise.
* tree-data-ref.cc: Likewise.
* tree-if-conv.cc: Likewise.
* tree-loop-distribution.cc: Likewise.
* tree-parloops.cc: Likewise.
* tree-predcom.cc: Likewise.
* tree-ssa-live.cc: Likewise.
* tree-ssa-loop-ivcanon.cc: Likewise.
* tree-ssa-loop-ivopts.cc: Likewise.
* tree-ssa-loop-prefetch.cc: Likewise.
* tree-ssa-loop-unswitch.cc: Likewise.
* tree-ssa-phiopt.cc: Likewise.
* tree-ssa-threadbackward.cc: Likewise.
* tree-ssa-threadupdate.cc: Likewise.
* tree-vect-data-refs.cc: Likewise.
* tree-vect-generic.cc: Likewise.
* tree-vect-loop-manip.cc: Likewise.
* tree-vect-loop.cc: Likewise.
* tree-vect-patterns.cc: Likewise.
* tree-vect-slp-patterns.cc: Likewise.
* tree-vect-slp.cc: Likewise.
* tree-vect-stmts.cc: Likewise.
* tree-vectorizer.cc: Likewise.

gcc/testsuite/ChangeLog:
* gcc.dg/plugin/dump_plugin.c: Define INCLUDE_MEMORY.

Signed-off-by: David Malcolm 
---
 gcc/config/aarch64/aarch64.cc |   1 +
 gcc/config/arm/arm.cc |   1 +
 gcc/config/i386/i386.cc   |   1 +
 gcc/config/loongarch/loongarch.cc |   1 +
 gcc/config/riscv/riscv-vector-costs.cc|   1 +
 gcc/config/riscv/riscv.cc |   1 +
 gcc/config/rs6000/rs6000.cc   |   1 +
 gcc/dump-context.h|   7 +-
 gcc/dumpfile.cc   | 156 +++---
 gcc/gimple-loop-interchange.cc|   1 +
 gcc/gimple-loop-jam.cc|   1 +
 gcc/gimple-loop-versioning.cc |   1 +
 gcc/graphite-dependences.cc   |   1 +
 gcc/graphite-isl-ast-to-gimple.cc |   1 +
 gcc/graphite-optimize-isl.cc  |   1 +
 gcc/g

[pushed 2/4] pretty-print: move class chunk_info into its own header

[David Malcolm]

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3310-g68a0ca66972a06.

gcc/cp/ChangeLog:
* error.cc: Include "pretty-print-format-impl.h".

gcc/ChangeLog:
* dumpfile.cc: Include "pretty-print-format-impl.h".
* pretty-print-format-impl.h: New file, based on material from
pretty-print.h.
* pretty-print.cc: Include "pretty-print-format-impl.h".
* pretty-print.h (chunk_info): Replace full declaration with
a forward decl, moving full decl to pretty-print-format-impl.h.

Signed-off-by: David Malcolm 
---
 gcc/cp/error.cc|  1 +
 gcc/dumpfile.cc|  1 +
 gcc/pretty-print-format-impl.h | 70 ++
 gcc/pretty-print.cc|  1 +
 gcc/pretty-print.h | 45 +-
 5 files changed, 74 insertions(+), 44 deletions(-)
 create mode 100644 gcc/pretty-print-format-impl.h

diff --git a/gcc/cp/error.cc b/gcc/cp/error.cc
index 879e5a115cfe..3cc0dd1cdfa9 100644
--- a/gcc/cp/error.cc
+++ b/gcc/cp/error.cc
@@ -38,6 +38,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "c-family/c-type-mismatch.h"
 #include "cp-name-hint.h"
 #include "attribs.h"
+#include "pretty-print-format-impl.h"
 
 #define pp_separate_with_comma(PP) pp_cxx_separate_with (PP, ',')
 #define pp_separate_with_semicolon(PP) pp_cxx_separate_with (PP, ';')
diff --git a/gcc/dumpfile.cc b/gcc/dumpfile.cc
index 2971c69bb0a1..eb245059210a 100644
--- a/gcc/dumpfile.cc
+++ b/gcc/dumpfile.cc
@@ -42,6 +42,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "stringpool.h" /* for get_identifier.  */
 #include "spellcheck.h"
 #include "make-unique.h"
+#include "pretty-print-format-impl.h"
 
 /* If non-NULL, return one past-the-end of the matching SUBPART of
the WHOLE string.  */
diff --git a/gcc/pretty-print-format-impl.h b/gcc/pretty-print-format-impl.h
new file mode 100644
index ..e05ad388963d
--- /dev/null
+++ b/gcc/pretty-print-format-impl.h
@@ -0,0 +1,70 @@
+/* Implementation detail of pp_format.
+   Copyright (C) 2002-2024 Free Software Foundation, Inc.
+   Contributed by Gabriel Dos Reis 
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+<http://www.gnu.org/licenses/>.  */
+
+#ifndef GCC_PRETTY_PRINT_FORMAT_IMPL_H
+#define GCC_PRETTY_PRINT_FORMAT_IMPL_H
+
+#include "pretty-print.h"
+
+/* The chunk_info data structure forms a stack of the results from the
+   first phase of formatting (pp_format) which have not yet been
+   output (pp_output_formatted_text).  A stack is necessary because
+   the diagnostic starter may decide to generate its own output by way
+   of the formatter.  */
+class chunk_info
+{
+  friend class pretty_printer;
+  friend class pp_markup::context;
+
+public:
+  const char * const *get_args () const { return m_args; }
+  quoting_info *get_quoting_info () const { return m_quotes; }
+
+  void append_formatted_chunk (const char *content);
+
+  void pop_from_output_buffer (output_buffer &buf);
+
+private:
+  void on_begin_quote (const output_buffer &buf,
+  unsigned chunk_idx,
+  const urlifier *urlifier);
+
+  void on_end_quote (pretty_printer *pp,
+output_buffer &buf,
+unsigned chunk_idx,
+const urlifier *urlifier);
+
+  /* Pointer to previous chunk on the stack.  */
+  chunk_info *m_prev;
+
+  /* Array of chunks to output.  Each chunk is a NUL-terminated string.
+ In the first phase of formatting, even-numbered chunks are
+ to be output verbatim, odd-numbered chunks are format specifiers.
+ The second phase replaces all odd-numbered chunks with formatted
+ text, and the third phase simply emits all the chunks in sequence
+ with appropriate line-wrapping.  */
+  const char *m_args[PP_NL_ARGMAX * 2];
+
+  /* If non-null, information on quoted text runs within the chunks
+ for use by a urlifier.  */
+  quoting_info *m_quotes;
+};
+
+#endif /* GCC_PRETTY_PRINT_FORMAT_IMPL_H */
diff --git a/gcc/pretty-print.cc b/gcc/pretty-print.cc
index 1d91da828212..810c629ef116 100644
--- a/gcc/pretty-print.cc
+++ b/gcc/pretty-print.cc
@@ -24,6 +24,7 @@ along with GCC; see the file C

[pushed] diagnostics: consolidate on_{begin, end}_diagnostic into on_report_diagnostic

[David Malcolm]

Previously diagnostic_context::report_diagnostic had, after the call to
pp_format (phases 1 and 2 of formatting the message):

  m_output_format->on_begin_diagnostic (*diagnostic);
  pp_output_formatted_text (this->printer, m_urlifier);
  if (m_show_cwe)
print_any_cwe (*diagnostic);
  if (m_show_rules)
print_any_rules (*diagnostic);
  if (m_show_option_requested)
  print_option_information (*diagnostic, orig_diag_kind);
  m_output_format->on_end_diagnostic (*diagnostic, orig_diag_kind);

This patch replaces all of the above with a single call to

  m_output_format->on_report_diagnostic (*diagnostic, orig_diag_kind);

moving responsibility for phase 3 of formatting and printing the result
from diagnostic_context to the output format.

This simplifies diagnostic_context::report_diagnostic and allows us to
move the code that prints CWEs, rules, and option information in textual
form from diagnostic_context to diagnostic_text_output_format, where it
belongs.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3200-gac707d30ce449f.

gcc/ChangeLog:
* diagnostic-format-json.cc
(json_output_format::on_begin_diagnostic): Delete.
(json_output_format::on_end_diagnostic): Rename to...
(json_output_format::on_report_diagnostic): ...this and add call
to pp_output_formatted_text.
(diagnostic_output_format_init_json): Drop unnecessary calls
to disable textual printing of CWEs, rules, and options.
* diagnostic-format-sarif.cc (sarif_builder::end_diagnostic):
Rename to...
(sarif_builder::on_report_diagnostic): ...this and add call to
pp_output_formatted_text.
(sarif_output_format::on_begin_diagnostic): Delete.
(sarif_output_format::on_end_diagnostic): Rename to...
(sarif_output_format::on_report_diagnostic): ...this and update
call to m_builder accordingly.
(diagnostic_output_format_init_sarif): Drop unnecessary calls
to disable textual printing of CWEs, rules, and options.
* diagnostic.cc (diagnostic_context::print_any_cwe): Convert to...
(diagnostic_text_output_format::print_any_cwe): ...this.
(diagnostic_context::print_any_rules): Convert to...
(diagnostic_text_output_format::print_any_rules): ...this.
(diagnostic_context::print_option_information): Convert to...
(diagnostic_text_output_format::print_option_information):
...this.
(diagnostic_context::report_diagnostic): Replace calls to the
output format's on_begin_diagnostic, to pp_output_formatted_text,
printing CWE, rules, option info, and the call to the format's
on_end_diagnostic with a call to the format's
on_report_diagnostic.
(diagnostic_text_output_format::on_begin_diagnostic): Delete.
(diagnostic_text_output_format::on_end_diagnostic): Delete.
(diagnostic_text_output_format::on_report_diagnostic): New vfunc,
which effectively does the on_begin_diagnostic, the call to
pp_output_formatted_text, the calls for printing CWE, rules,
option info, and the call to the diagnostic_finalizer.
* diagnostic.h (diagnostic_output_format::on_begin_diagnostic):
Delete.
(diagnostic_output_format::on_end_diagnostic): Delete.
(diagnostic_output_format::on_report_diagnostic): New.
(diagnostic_text_output_format::on_begin_diagnostic): Delete.
(diagnostic_text_output_format::on_end_diagnostic): Delete.
(diagnostic_text_output_format::on_report_diagnostic): New.
(class diagnostic_context): Add friend class
diagnostic_text_output_format.
(diagnostic_context::get_urlifier): New accessor.
(diagnostic_context::print_any_cwe): Move decl...
(diagnostic_text_output_format::print_any_cwe): ...to here.
(diagnostic_context::print_any_rules): Move decl...
(diagnostic_text_output_format::print_any_rules): ...to here.
(diagnostic_context::print_option_information): Move decl...
(diagnostic_text_output_format::print_option_information): ...to
    here.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-json.cc  |  24 +--
 gcc/diagnostic-format-sarif.cc |  34 ++---
 gcc/diagnostic.cc  | 261 +
 gcc/diagnostic.h   |  28 ++--
 4 files changed, 171 insertions(+), 176 deletions(-)

diff --git a/gcc/diagnostic-format-json.cc b/gcc/diagnostic-format-json.cc
index b78cb92cfd7a..f2e9d0d79e51 100644
--- a/gcc/diagnostic-format-json.cc
+++ b/gcc/diagnostic-format-json.cc
@@ -47,13 +47,8 @@ public:
 m_cur_children_array = nullptr;
   }
   void
-  on_begin_diagnostic (const diagnostic_info &) final override
-  {
-/* No-op.  */
-  }
-  void
-  on_end_diagnostic (const diagnostic_info &diagnostic,
-diag

[pushed] testsuite: add event IDs to multithreaded event plugin test

[David Malcolm]

Add test coverage of "%@" in event messages in a multithreaded
execution path.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3199-g6a1c359e28442c.

gcc/testsuite/ChangeLog:
* gcc.dg/plugin/diagnostic-test-paths-multithreaded-inline-events.c:
Update expected output.
* gcc.dg/plugin/diagnostic-test-paths-multithreaded-sarif.py:
Likewise.
* gcc.dg/plugin/diagnostic-test-paths-multithreaded-separate-events.c:
Likewise.
* gcc.dg/plugin/diagnostic_plugin_test_paths.c
(test_diagnostic_path::add_event_2): Return the id of the added
event.
(test_diagnostic_path::add_event_2_with_event_id): New.
(example_4): Add event IDs to the deadlock messages indicating
where the locks where acquired.

Signed-off-by: David Malcolm 
---
 ...c-test-paths-multithreaded-inline-events.c |  4 +-
 ...agnostic-test-paths-multithreaded-sarif.py |  4 +-
 ...test-paths-multithreaded-separate-events.c |  4 +-
 .../plugin/diagnostic_plugin_test_paths.c | 46 +--
 4 files changed, 38 insertions(+), 20 deletions(-)

diff --git 
a/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-inline-events.c
 
b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-inline-events.c
index 333ef7359440..b306bcc1a0f3 100644
--- 
a/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-inline-events.c
+++ 
b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-inline-events.c
@@ -58,7 +58,7 @@ Thread: 'Thread 1'
|   NN |   acquire_lock_b ();
|  |   ^
|  |   |
-   |  |   (5) deadlocked due to waiting for lock b in thread 1...
+   |  |   (5) deadlocked due to waiting for lock b in thread 1 
(acquired by thread 2 at (4))...
|
 
 Thread: 'Thread 2'
@@ -67,6 +67,6 @@ Thread: 'Thread 2'
|   NN |   acquire_lock_a ();
|  |   ^
|  |   |
-   |  |   (6) ...whilst waiting for lock a in thread 2
+   |  |   (6) ...whilst waiting for lock a in thread 2 (acquired 
by thread 1 at (2))
|
  { dg-end-multiline-output "" } */
diff --git 
a/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-sarif.py 
b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-sarif.py
index cff78aa8ac8e..cb00faf1532a 100644
--- a/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-sarif.py
+++ b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-sarif.py
@@ -95,7 +95,7 @@ def test_result(sarif):
 == "lock a is now held by thread 1"
 assert tf0['locations'][2]['executionOrder'] == 5
 assert tf0['locations'][2]['location']['message']['text'] \
-== "deadlocked due to waiting for lock b in thread 1..."
+== "deadlocked due to waiting for lock b in thread 1 (acquired by 
thread 2 at (4))..."
 
 assert len(tf1['locations']) == 3
 assert tf1['locations'][0]['executionOrder'] == 3
@@ -106,4 +106,4 @@ def test_result(sarif):
 == "lock b is now held by thread 2"
 assert tf1['locations'][2]['executionOrder'] == 6
 assert tf1['locations'][2]['location']['message']['text'] \
-== "...whilst waiting for lock a in thread 2"
+== "...whilst waiting for lock a in thread 2 (acquired by thread 1 at 
(2))"
diff --git 
a/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-separate-events.c
 
b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-separate-events.c
index 914918bb9e16..90464320b8e7 100644
--- 
a/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-separate-events.c
+++ 
b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-separate-events.c
@@ -7,12 +7,12 @@ extern void acquire_lock_b(void);
 void foo ()
 { /* { dg-message "\\(1\\) entering 'foo'" } */
   acquire_lock_a (); /* { dg-message "\\(2\\) lock a is now held by thread 1" 
} */
-  acquire_lock_b (); /* { dg-message "\\(5\\) deadlocked due to waiting for 
lock b in thread 1\.\.\." } */
+  acquire_lock_b (); /* { dg-message "\\(5\\) deadlocked due to waiting for 
lock b in thread 1 \\(acquired by thread 2 at \\(4\\)\\)\.\.\." } */
 }
 
 void bar ()
 { /* { dg-message "\\(3\\) entering 'bar'" } */
   acquire_lock_b (); /* { dg-message "\\(4\\) lock b is now held by thread 2" 
} */
   acquire_lock_a (); /* { dg-warning "deadlock due to inconsistent lock 
acquisition order" } */
-  /* { dg-message "\\(6\\) \.\.\.whilst waiting for lock a in thread 2

[pushed] diagnostics: move output formats from diagnostic.{c, h} to their own files

[David Malcolm]

In particular, move the classic text output code to a
diagnostic-text.cc (analogous to -json.cc and -sarif.cc).

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3201-g92c5265d22afaa.

gcc/ChangeLog:
* Makefile.in (OBJS-libcommon): Add diagnostic-format-text.o.
* diagnostic-format-json.cc: Include "diagnostic-format.h".
* diagnostic-format-sarif.cc: Likewise.
* diagnostic-format-text.cc: New file, using material from
diagnostics.cc.
* diagnostic-global-context.cc: Include
"diagnostic-format.h".
* diagnostic-format-text.h: New file, using material from
diagnostics.h.
* diagnostic-format.h: New file, using material from
diagnostics.h.
* diagnostic.cc: Include "diagnostic-format.h" and
"diagnostic-format-text.h".
(diagnostic_text_output_format::~diagnostic_text_output_format):
Move to diagnostic-format-text.cc.
(diagnostic_text_output_format::on_report_diagnostic): Likewise.
(diagnostic_text_output_format::on_diagram): Likewise.
(diagnostic_text_output_format::print_any_cwe): Likewise.
(diagnostic_text_output_format::print_any_rules): Likewise.
(diagnostic_text_output_format::print_option_information):
Likewise.
* diagnostic.h (class diagnostic_output_format): Move to
diagnostic-format.h.
(class diagnostic_text_output_format): Move to
diagnostic-format-text.h.
(diagnostic_output_format_init): Move to
diagnostic-format.h.
(diagnostic_output_format_init_json_stderr): Likewise.
(diagnostic_output_format_init_json_file): Likewise.
(diagnostic_output_format_init_sarif_stderr): Likewise.
(diagnostic_output_format_init_sarif_file): Likewise.
(diagnostic_output_format_init_sarif_stream): Likewise.
* gcc.cc: Include "diagnostic-format.h".
* opts.cc: Include "diagnostic-format.h".

gcc/testsuite/ChangeLog:
* gcc.dg/plugin/diagnostic_group_plugin.c: Include
    "diagnostic-format-text.h".

Signed-off-by: David Malcolm 
---
 gcc/Makefile.in   |   1 +
 gcc/diagnostic-format-json.cc |   1 +
 gcc/diagnostic-format-sarif.cc|   1 +
 gcc/diagnostic-format-text.cc | 209 ++
 gcc/diagnostic-format-text.h  |  56 +
 gcc/diagnostic-format.h   |  83 +++
 gcc/diagnostic-global-context.cc  |   1 +
 gcc/diagnostic.cc | 176 +--
 gcc/diagnostic.h  |  85 +--
 gcc/gcc.cc|   1 +
 gcc/opts.cc   |   1 +
 .../gcc.dg/plugin/diagnostic_group_plugin.c   |   1 +
 12 files changed, 359 insertions(+), 257 deletions(-)
 create mode 100644 gcc/diagnostic-format-text.cc
 create mode 100644 gcc/diagnostic-format-text.h
 create mode 100644 gcc/diagnostic-format.h

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 8fba8f7db6a2..68fda1a75918 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1821,6 +1821,7 @@ OBJS = \
 OBJS-libcommon = diagnostic-spec.o diagnostic.o diagnostic-color.o \
diagnostic-format-json.o \
diagnostic-format-sarif.o \
+   diagnostic-format-text.o \
diagnostic-global-context.o \
diagnostic-macro-unwinding.o \
diagnostic-path.o \
diff --git a/gcc/diagnostic-format-json.cc b/gcc/diagnostic-format-json.cc
index f2e9d0d79e51..c94f5f73bb5a 100644
--- a/gcc/diagnostic-format-json.cc
+++ b/gcc/diagnostic-format-json.cc
@@ -27,6 +27,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "selftest-diagnostic.h"
 #include "diagnostic-metadata.h"
 #include "diagnostic-path.h"
+#include "diagnostic-format.h"
 #include "json.h"
 #include "selftest.h"
 #include "logical-location.h"
diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 554bf3cb2d5c..59d9cd721839 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -29,6 +29,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "diagnostic.h"
 #include "diagnostic-metadata.h"
 #include "diagnostic-path.h"
+#include "diagnostic-format.h"
 #include "json.h"
 #include "cpplib.h"
 #include "logical-location.h"
diff --git a/gcc/diagnostic-format-text.cc b/gcc/diagnostic-format-text.cc
new file mode 100644
index ..b984803ff380
--- /dev/null
+++ b/gcc/diagnostic-format-text.cc
@@ -0,0 +1,209 @@
+/* Classic text-based output of diagnostics.
+   Copyright (C) 1999-2024 Free Software Foundation, Inc.
+
+This file is part of GC

[pushed] testsuite: generalize support for Python tests for SARIF output

[David Malcolm]

In r15-2354-g4d1f71d49e396c I added the ability to use Python to write
tests of SARIF output via a new "run-sarif-pytest" based
on "run-gcov-pytest", with a sarif.py support script in
testsuite/gcc.dg/sarif-output.

This followup patch:
(a) removes the limitation of such tests needing to be in
testsuite/gcc.dg/sarif-output by moving sarif.py to testsuite/lib
and adding logic to add that directory to PYTHONPATH when invoking
pytest.

(b) uses this to replace fragile regexp-based tests in
gcc.dg/plugin/diagnostic-test-paths-multithreaded-sarif.c with
Python logic that verifies the structure within the generated JSON,
and to add test coverage for SARIF output relating to GCC plugins.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3198-gaa3b950291119a.

gcc/ChangeLog:
* diagnostic-format-sarif.cc: Add comments noting that we don't
yet capture any diagnostic_metadata::rules associated with a
diagnostic.

gcc/testsuite/ChangeLog:
* gcc.dg/plugin/diagnostic-test-metadata-sarif.c: New test,
based on diagnostic-test-metadata.c.
* gcc.dg/plugin/diagnostic-test-metadata-sarif.py: New script.
* gcc.dg/plugin/diagnostic-test-paths-multithreaded-sarif.c:
Replace scan-sarif-file directives with run-sarif-pytest, to
run...
* gcc.dg/plugin/diagnostic-test-paths-multithreaded-sarif.py:
...this new test.
* gcc.dg/plugin/plugin.exp (plugin_test_list): Add
diagnostic-test-metadata-sarif.c.
* gcc.dg/sarif-output/sarif.py: Move to...
* lib/sarif.py: ...here.
* lib/scansarif.exp (run-sarif-pytest): Prepend "lib" to
PYTHONPATH before running python scripts.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc|   6 +-
 .../plugin/diagnostic-test-metadata-sarif.c   |  17 +++
 .../plugin/diagnostic-test-metadata-sarif.py  |  55 +
 ...iagnostic-test-paths-multithreaded-sarif.c |  17 +--
 ...agnostic-test-paths-multithreaded-sarif.py | 109 ++
 gcc/testsuite/gcc.dg/plugin/plugin.exp|   4 +-
 .../{gcc.dg/sarif-output => lib}/sarif.py |   0
 gcc/testsuite/lib/scansarif.exp   |  16 +++
 8 files changed, 208 insertions(+), 16 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/plugin/diagnostic-test-metadata-sarif.c
 create mode 100644 
gcc/testsuite/gcc.dg/plugin/diagnostic-test-metadata-sarif.py
 create mode 100644 
gcc/testsuite/gcc.dg/plugin/diagnostic-test-paths-multithreaded-sarif.py
 rename gcc/testsuite/{gcc.dg/sarif-output => lib}/sarif.py (100%)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 963a185f6ced..1d99c904ff0c 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -580,7 +580,9 @@ public:
  (SARIF v2.1.0 section 3.27.13).
- doesn't capture -Werror cleanly
- doesn't capture inlining information (can SARIF handle this?)
-   - doesn't capture macro expansion information (can SARIF handle this?).  */
+   - doesn't capture macro expansion information (can SARIF handle this?).
+   - doesn't capture any diagnostic_metadata::rules associated with
+ a diagnostic.  */
 
 class sarif_builder
 {
@@ -1522,6 +1524,8 @@ sarif_builder::make_result_object (diagnostic_context 
&context,
}
 
   diagnostic.metadata->maybe_add_sarif_properties (*result_obj);
+
+  /* We don't yet support diagnostic_metadata::rule.  */
 }
 
   /* "level" property (SARIF v2.1.0 section 3.27.10).  */
diff --git a/gcc/testsuite/gcc.dg/plugin/diagnostic-test-metadata-sarif.c 
b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-metadata-sarif.c
new file mode 100644
index ..246a8429090d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-metadata-sarif.c
@@ -0,0 +1,17 @@
+/* { dg-do compile } */
+/* { dg-options "-fdiagnostics-format=sarif-file" } */
+
+extern char *gets (char *s);
+
+void test_cwe (void)
+{
+  char buf[1024];
+  gets (buf);
+}
+
+/* Verify that some JSON was written to a file with the expected name.  */
+/* { dg-final { verify-sarif-file } } */
+
+/* Use a Python script to verify various properties about the generated
+   .sarif file:
+   { dg-final { run-sarif-pytest diagnostic-test-metadata-sarif.c 
"diagnostic-test-metadata-sarif.py" } } */
diff --git a/gcc/testsuite/gcc.dg/plugin/diagnostic-test-metadata-sarif.py 
b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-metadata-sarif.py
new file mode 100644
index ..959e6f2e9942
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-metadata-sarif.py
@@ -0,0 +1,55 @@
+# We expect a warning with this textual form:
+#
+# . PATH/diagnostic-test-metadata-sarif.c: In function 'test_cwe':
+# . PATH/diagnostic-test-metadata-sarif.c:8:3: warning: never use 'gets' 
[CWE-242] [STR34-C]
+
+from 

[pushed] pretty-print: fixes to selftests

[David Malcolm]

Add selftest coverage for %{ and %} in pretty-print.cc

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3197-g276cc4324b9e8d.

gcc/ChangeLog:
* pretty-print.cc (selftest::test_urls): Make static.
(selftest::test_urls_from_braces): New.
(selftest::test_null_urls): Make static.
(selftest::test_urlification): Likewise.
(selftest::pretty_print_cc_tests): Call test_urls_from_braces.

Signed-off-by: David Malcolm 
---
 gcc/pretty-print.cc | 39 +++
 1 file changed, 35 insertions(+), 4 deletions(-)

diff --git a/gcc/pretty-print.cc b/gcc/pretty-print.cc
index 64713803dbe7..1d91da828212 100644
--- a/gcc/pretty-print.cc
+++ b/gcc/pretty-print.cc
@@ -3135,7 +3135,7 @@ test_prefixes_and_wrapping ()
 
 /* Verify that URL-printing works as expected.  */
 
-void
+static void
 test_urls ()
 {
   {
@@ -3169,9 +3169,40 @@ test_urls ()
   }
 }
 
+static void
+test_urls_from_braces ()
+{
+  {
+pretty_printer pp;
+pp.set_url_format (URL_FORMAT_NONE);
+pp_printf (&pp, "before %{text%} after",
+   "http://example.com";);
+ASSERT_STREQ ("before text after",
+ pp_formatted_text (&pp));
+  }
+
+  {
+pretty_printer pp;
+pp.set_url_format (URL_FORMAT_ST);
+pp_printf (&pp, "before %{text%} after",
+   "http://example.com";);
+ASSERT_STREQ ("before \33]8;;http://example.com\33\\text\33]8;;\33\\ 
after",
+ pp_formatted_text (&pp));
+  }
+
+  {
+pretty_printer pp;
+pp.set_url_format (URL_FORMAT_BEL);
+pp_printf (&pp, "before %{text%} after",
+   "http://example.com";);
+ASSERT_STREQ ("before \33]8;;http://example.com\atext\33]8;;\a after",
+ pp_formatted_text (&pp));
+  }
+}
+
 /* Verify that we gracefully reject null URLs.  */
 
-void
+static void
 test_null_urls ()
 {
   {
@@ -3221,8 +3252,7 @@ pp_printf_with_urlifier (pretty_printer *pp,
   va_end (ap);
 }
 
-
-void
+static void
 test_urlification ()
 {
   class test_urlifier : public urlifier
@@ -3424,6 +3454,7 @@ pretty_print_cc_tests ()
   test_pp_format ();
   test_prefixes_and_wrapping ();
   test_urls ();
+  test_urls_from_braces ();
   test_null_urls ();
   test_urlification ();
   test_utf8 ();
-- 
2.26.3

[pushed] json.h: fix typo in comment

[David Malcolm]

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-3196-gb835710328847a.

gcc/ChangeLog:
* json.h: Fix typo in comment about missing INCLUDE_MEMORY.

Signed-off-by: David Malcolm 
---
 gcc/json.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gcc/json.h b/gcc/json.h
index 21f71fe1c4ab..0bafa5fbea3f 100644
--- a/gcc/json.h
+++ b/gcc/json.h
@@ -27,7 +27,7 @@ along with GCC; see the file COPYING3.  If not see
json.h.  */
 
 #ifndef INCLUDE_MEMORY
-# error "You must define INCLUDE_MEMORY before including system.h to use 
make-unique.h"
+# error "You must define INCLUDE_MEMORY before including system.h to use 
json.h"
 #endif
 
 /* Implementation of JSON, a lightweight data-interchange format.
-- 
2.26.3

Re: [PATCH 0/7] v3 of libdiagnostics

[David Malcolm]

On Tue, 2024-08-20 at 11:49 +0200, Richard Biener wrote:
> On Thu, Aug 15, 2024 at 8:13 PM David Malcolm 
> wrote:
> > 
> > Here's v3 of my patch kit for "libdiagnostics", which makes GCC's
> > diagnostics subsystem available as a shared library; see:
> >   https://gcc.gnu.org/wiki/libdiagnostics
> 
> So this is to make use of this from gas?  

One of the clients of the library would be gas, yes (although
optionally, to avoid complicating bootstrap of binutils).

However other clients would be possible, including those that are not
under the "GNU toolchain" umbrella (provided that they can be linked
against GPLv3 code).

For example, I've been experimenting with Python bindings, which would
allow Python scripts to reuse GCC's diagnostics code (e.g. for SARIF,
fix-it hints, etc).  I've also been experimenting with IDE integration
(see PR 115970), and it would be nice if users of libdiagnostics got
that "for free".

Another example of client code is the sarif-replay tool in patch 7, as
a sarif consumer (if nothing else, writing this has exposed various
bugs in our existing SARIF-writing code).  The analyzer's integration
test suite generates 10s of thousands of .sarif files, so having
tooling to work with them is "scratching my own itch".

> Is the plan to move
> sources (and dependences) to the toplevel then, possibly
> building a static convenience lib for GCCs use?

The problem with moving things from "gcc" to, say, a new subdirectory
of the top level source directory is that our "gcc" subdirectory has a
lot of support code that would also need refactoring/moving.  Off the
top of my head:
- selftest framework
- DejaGnu .exp stuff below gcc/testsuite/lib
- C++11 support, such as our "make-unique.h"
- libcpp: the big one: diagnostics uses libcpp
- probably some configure/Makefile.in entanglements

Fixing the above would be a major task.

So the patch kit punts on this by adding/moving stuff within "gcc" (and
requiring an opt-in via --enable-libdiagnostics).

> 
> Note I'm missing documentation (which is probably there
> in the libdiagnostics.h header); an addition to sourcebuild.texi
> might be nice at least and documenting --enable-libdiagnostics
> in install.texi.

The libdiagnostics.h header has comments, but, yes, I should probably
add docs similar to that of libgccjit.h (tutorial and API reference). 
I'll do that for the next iteration of the patch.

Patch 2 of the kit documents --enable-libdiagnostics in install.texi. 
I'll add notes to sourcebuild.texi in the next iteration of the patch.


Thanks; hope the above makes sense.
Dave

> 
> > New in v3:
> > * it bootstraps and pass regression tests
> > * I added an opt-in configure flag: --enable-libdiagnostics, which
> >   must be enabled to build it (along with --enable-host-shared)
> > * a new "sarif-replay" command-line tool that takes .sarif files
> >   and replays the diagnostics within them as if they were GCC
> >   diagnostics, in GCC's textual format (i.e. GCC as a SARIF
> > *consumer*,
> >   as well as producer).  This is implemented on top of
> > libdiagnostics
> >   hence I've been "eating my own dogfood"
> > * support for execution paths in libdiagnostics API
> > * lots of fixes
> > 
> > Patch 1 has libdiagnostic.h, the public header file
> > Patch 2 has the implementation
> > Patch 3 has the C++ wrapper API I added in v2
> > Patch 4 has a refactoring of gcc-dg.exp I needed for patch 5.
> > Patch 5 has the testsuite for libdiagnostics itself
> > Patch 6 implements JSON parsing support
> > Patch 7 implements the sarif-replay command-line tool, and its
> >   testsuite, exercising various valid, invalid, and malformed
> >   input files.
> > 
> > Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu,
> > both with and without --enable-libdiagnostics.
> > With --enable-libdiagnostics the patch kit has this effect:
> > 
> >   # of .sum files: 20->22 (+2)
> >   FAIL: 110
> >   PASS: 617481->617679 (+198) 100.03%
> >   XFAIL: 4512
> >   XPASS: 13
> >   UNTESTED: 136
> >   UNSUPPORTED: 8058
> > 
> > where the two new .sum files are:
> > 
> >   BUILD/gcc/testsuite/libdiagnostics/libdiagnostics.sum:
> >     PASS: 132 tests
> > 
> >   BUILD/gcc/testsuite/sarif-replay/sarif-replay.sum:
> >     PASS: 66 tests
> > 
> > OK for trunk?
> > 
> > David Malcolm (7):
> >   libdiagnostics v3: header
> >   libdiagnostics v3: implementation
> >   libdiagnostics v3: add C++ wrapper API
> >   testsuite: 

[PATCH 2/7] libdiagnostics v3: implementation

[David Malcolm]

Changed in v3:
* Added a --enable-libdiagnostics to configure.ac.  It is disabled
  by default, and requires --enable-host-shared.
* Split out gcc/testsuite/libdiagnostics.dg/libdiagnostics.exp into
  another patch
* Update copyright year
* class diagnostic_logical_location: Add get_name_for_path_output.
* as_diagnostic_event_id: New function.
* on_begin_text_diagnostic: Make "info" const.
* diagnostic_manager::~diagnostic_manager: Free
  m_line_table.m_location_adhoc_data_map.data and
  m_line_table.info_ordinary.maps.
* diagnostic_manager::new_location_from_file_and_line: Move code into
  ensure_linemap_for_file_and_line.
* diagnostic_manager::new_location_from_file_line_column: Use
  ensure_linemap_for_file_and_line rather than always using LC_ENTER.
* class libdiagnostics_path_event: New.
* class libdiagnostics_path_thread: New.
* struct diagnostic_execution_path: New.
* diagnostic::diagnostic: Initialize m_path.
* diagnostic::add_execution_path: New.
* diagnostic::take_execution_path: New.
* diagnostic::m_path: New field.
* diagnostic_text_sink::diagnostic_text_sink: Initialize
  m_current_logical_loc.  Call diagnostic_urls_init, set_show_cwe,
  set_show_rules, and set m_show_column.
* diagnostic_text_sink::on_begin_text_diagnostic: Various fixes.
* sarif_sink::sarif_sink: Add "main_input_file" param.
* diagnostic_manager::write_patch: Update for change to pretty_printer.
* diagnostic_manager::new_execution_path:  New.
* diagnostic_manager_add_sarif_sink: Add "main_input_file" param.
* diagnostic_manager_debug_dump_location: Call diagnostic_finish.
* diagnostic_set_logical_location: Allow logical_loc to be NULL.
* diagnostic_add_execution_path: New.
* diagnostic_manager_new_execution_path: New.
* diagnostic_take_execution_path: New.
* diagnostic_execution_path_release: New.
* diagnostic_execution_path_add_event: New.
* diagnostic_execution_path_add_event_va: New.

Changed in v2:
* Changed diagnostic_location_t -> const diagnostic_physical_location *
* new entrypoint: diagnostic_finish_va
* new debugging entrypoints for dumping to a FILE *
* Makefile.in: dropped FULL_DRIVER_NAME from libdiagnostics
* fix up for my recent changes to gcc/diagnostic.h

Blurb from v1:

Here's a work-in-progress patch for GCC that adds the implementation
of libdiagnostics.  Various aspects of this need work; posting now
for early feedback on overall direction.

ChangeLog:
* configure: Regenerate.
* configure.ac: Add --enable-libdiagnostics.

gcc/ChangeLog:
* Makefile.in (enable_libdiagnostics): New.
(lang_checks): If libdiagnostics is enabled, add
check-libdiagnostics.
(ALL_HOST_OBJS): If libdiagnostics is enabled, add
$(libdiagnostics_OBJS).
(start.encap): Add LIBDIAGNOSTICS.
(libdiagnostics_OBJS): New.
(LIBDIAGNOSTICS_VERSION_NUM): New, adapted from code in
jit/Make-lang.in.
(LIBDIAGNOSTICS_MINOR_NUM): Likewise.
(LIBDIAGNOSTICS_RELEASE_NUM): Likewise.
(LIBDIAGNOSTICS_FILENAME): Likewise.
(LIBDIAGNOSTICS_IMPORT_LIB): Likewise.
(libdiagnostics): Likewise.
(LIBDIAGNOSTICS_AGE): Likewise.
(LIBDIAGNOSTICS_BASENAME): Likewise.
(LIBDIAGNOSTICS_SONAME): Likewise.
(LIBDIAGNOSTICS_LINKER_NAME): Likewise.
(LIBDIAGNOSTICS_COMMA): Likewise.
(LIBDIAGNOSTICS_VERSION_SCRIPT_OPTION): Likewise.
(LIBDIAGNOSTICS_SONAME_OPTION): Likewise.
(LIBDIAGNOSTICS_SONAME_SYMLINK): Likewise.
(LIBDIAGNOSTICS_LINKER_NAME_SYMLINK): Likewise.
(LIBDIAGNOSTICS_FILENAME): Likewise.
(libdiagnostics.serial): Likewise.
(LIBDIAGNOSTICS_EXTRA_OPTS): Likewise.
(install): If libdiagnostics is enabled, add
install-libdiagnostics.
(libdiagnostics.install-headers): New.
(libdiagnostics.install-common): New, adapted from code in
jit/Make-lang.in.
(install-libdiagnostics): New.
* configure: Regenerate.
* configure.ac (check_languages): Add check-libdiagnostics.
(--enable-libdiagnostics): New.
* diagnostic-event-id.h (diagnostic_event_id_t::zero_based): New.
* doc/install.texi (--enable-libdiagnostics): New.
* libdiagnostics.cc: New file.
* libdiagnostics.map: New file.

Signed-off-by: David Malcolm 
---
 configure |   42 +
 configure.ac  |   35 +
 gcc/Makefile.in   |  179 +++-
 gcc/configure |   26 +-
 gcc/configure.ac  |   16 +
 gcc/diagnostic-event-id.h |6 +
 gcc/doc/install.texi  |7 +
 gcc/libdiagnostics.cc | 1652 +
 gcc/libdiagnostics.map|   72 ++
 9 files changed, 2032 insertions(+), 3 deletions(-)
 create mode 100644 gcc/libdiagnostics.cc
 create mode 100644 gcc/libdiagnostics.map

diff --git a/configure b/configure
index 51bf1d1add18..be6e41439990 100755
--- a/conf

[PATCH 6/7] json: add json parsing support

[David Malcolm]

This patch implements JSON parsing support.

It's based on the parsing parts of the patch I posted here:
https://gcc.gnu.org/legacy-ml/gcc-patches/2017-08/msg00417.html
with the parsing moved to a separate source file and header, heavily
rewritten to capture source location information for JSON values, and
to capture errors via a result template.

I also added optional support for C and C++ style comments, which is
extremely useful in DejaGnu tests.

gcc/ChangeLog:
* Makefile.in (OBJS-libcommon): Add json-parsing.o.
* json-parsing.cc: New file.
* json-parsing.h: New file.
* json.cc (selftest::assert_print_eq): Remove "static".
* json.h (json::array::begin): New.
(json::array::end): New.
(json::array::length): New.
(json::array::get): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(is_a_helper ::test): New.
(selftest::assert_print_eq): New decl.
* selftest-run-tests.cc (selftest::run_tests): Call
selftest::json_parser_cc_tests.
* selftest.h (selftest::json_parser_cc_tests): New decl.

Signed-off-by: David Malcolm 
---
 gcc/Makefile.in   |2 +-
 gcc/json-parsing.cc   | 2394 +
 gcc/json-parsing.h|  113 ++
 gcc/json.cc   |2 +-
 gcc/json.h|  122 +-
 gcc/selftest-run-tests.cc |1 +
 gcc/selftest.h|1 +
 7 files changed, 2631 insertions(+), 4 deletions(-)
 create mode 100644 gcc/json-parsing.cc
 create mode 100644 gcc/json-parsing.h

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 3e4c7bd645f9..64dcaddfdfbe 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1832,7 +1832,7 @@ OBJS-libcommon = diagnostic-spec.o diagnostic.o 
diagnostic-color.o \
diagnostic-show-locus.o \
edit-context.o \
pretty-print.o intl.o \
-   json.o \
+   json.o json-parsing.o \
sbitmap.o \
vec.o input.o hash-table.o ggc-none.o memory-block.o \
selftest.o selftest-diagnostic.o sort.o \
diff --git a/gcc/json-parsing.cc b/gcc/json-parsing.cc
new file mode 100644
index ..78188c4fef9c
--- /dev/null
+++ b/gcc/json-parsing.cc
@@ -0,0 +1,2394 @@
+/* JSON parsing
+   Copyright (C) 2017-2024 Free Software Foundation, Inc.
+   Contributed by David Malcolm .
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+<http://www.gnu.org/licenses/>.  */
+
+#include "config.h"
+#define INCLUDE_MEMORY
+#include "system.h"
+#include "coretypes.h"
+#include "json-parsing.h"
+#include "pretty-print.h"
+#include "math.h"
+#include "make-unique.h"
+#include "selftest.h"
+
+using namespace json;
+
+/* Declarations relating to parsing JSON, all within an
+   anonymous namespace.  */
+
+namespace {
+
+/* A typedef representing a single unicode character.  */
+
+typedef unsigned unichar;
+
+/* An enum for discriminating different kinds of JSON token.  */
+
+enum token_id
+{
+  TOK_ERROR,
+
+  TOK_EOF,
+
+  /* Punctuation.  */
+  TOK_OPEN_SQUARE,
+  TOK_OPEN_CURLY,
+  TOK_CLOSE_SQUARE,
+  TOK_CLOSE_CURLY,
+  TOK_COLON,
+  TOK_COMMA,
+
+  /* Literal names.  */
+  TOK_TRUE,
+  TOK_FALSE,
+  TOK_NULL,
+
+  TOK_STRING,
+  TOK_FLOAT_NUMBER,
+  TOK_INTEGER_NUMBER
+};
+
+/* Human-readable descriptions of enum token_id.  */
+
+static const char *token_id_name[] = {
+  "error",
+  "EOF",
+  "'['",
+  "'{'",
+  "']'",
+  "'}'",
+  "':'",
+  "','",
+  "'true'",
+  "'false'",
+  "'null'",
+  "string",
+  "number",
+  "number"
+};
+
+/* Tokens within the JSON lexer.  */
+
+struct token
+{
+  /* The kind of token.  */
+  enum token_id id;
+
+  /* The location of this token within the unicode
+ character stream.  */
+  location_map::range range;
+
+  union
+  {
+/* Value for TOK_ERROR and TO

[PATCH 5/7] libdiagnostics v3: test suite

[David Malcolm]

Changed in v3:
* split out the C and C++ API tests into this patch
* heavily rewritten libdiagnostics.exp; added support for Python tests
* tests updated for API changes, rewritten and extended

gcc/testsuite/ChangeLog:
* libdiagnostics.dg/libdiagnostics.exp: New, adapted from jit.exp.
* libdiagnostics.dg/sarif.py: New.
* libdiagnostics.dg/test-dump.c: New test.
* libdiagnostics.dg/test-error-c.py: New test.
* libdiagnostics.dg/test-error-with-note-c.py: New test.
* libdiagnostics.dg/test-error-with-note.c: New test.
* libdiagnostics.dg/test-error-with-note.cc: New test.
* libdiagnostics.dg/test-error.c: New test.
* libdiagnostics.dg/test-error.cc: New test.
* libdiagnostics.dg/test-fix-it-hint-c.py: New test.
* libdiagnostics.dg/test-fix-it-hint.c: New test.
* libdiagnostics.dg/test-fix-it-hint.cc: New test.
* libdiagnostics.dg/test-helpers++.h: New test.
* libdiagnostics.dg/test-helpers.h: New test.
* libdiagnostics.dg/test-labelled-ranges.c: New test.
* libdiagnostics.dg/test-labelled-ranges.cc: New test.
* libdiagnostics.dg/test-labelled-ranges.py: New test.
* libdiagnostics.dg/test-logical-location-c.py: New test.
* libdiagnostics.dg/test-logical-location.c: New test.
* libdiagnostics.dg/test-metadata-c.py: New test.
* libdiagnostics.dg/test-metadata.c: New test.
* libdiagnostics.dg/test-multiple-lines-c.py: New test.
* libdiagnostics.dg/test-multiple-lines.c: New test.
* libdiagnostics.dg/test-no-column-c.py: New test.
* libdiagnostics.dg/test-no-column.c: New test.
* libdiagnostics.dg/test-no-diagnostics-c.py: New test.
* libdiagnostics.dg/test-no-diagnostics.c: New test.
* libdiagnostics.dg/test-note-with-fix-it-hint-c.py: New test.
* libdiagnostics.dg/test-note-with-fix-it-hint.c: New test.
* libdiagnostics.dg/test-text-sink-options.c: New test.
* libdiagnostics.dg/test-warning-c.py: New test.
* libdiagnostics.dg/test-warning-with-path-c.py: New test.
* libdiagnostics.dg/test-warning-with-path.c: New test.
* libdiagnostics.dg/test-warning.c: New test.
* libdiagnostics.dg/test-write-sarif-to-file-c.py: New test.
* libdiagnostics.dg/test-write-sarif-to-file.c: New test.
* libdiagnostics.dg/test-write-text-to-file.c: New test.

Signed-off-by: David Malcolm 
---
 .../libdiagnostics.dg/libdiagnostics.exp  | 296 ++
 gcc/testsuite/libdiagnostics.dg/sarif.py  |  23 ++
 gcc/testsuite/libdiagnostics.dg/test-dump.c   |  69 
 .../libdiagnostics.dg/test-error-c.py |  54 
 .../test-error-with-note-c.py |  50 +++
 .../libdiagnostics.dg/test-error-with-note.c  |  74 +
 .../libdiagnostics.dg/test-error-with-note.cc |  55 
 gcc/testsuite/libdiagnostics.dg/test-error.c  |  59 
 gcc/testsuite/libdiagnostics.dg/test-error.cc |  47 +++
 .../libdiagnostics.dg/test-fix-it-hint-c.py   |  46 +++
 .../libdiagnostics.dg/test-fix-it-hint.c  |  81 +
 .../libdiagnostics.dg/test-fix-it-hint.cc |  74 +
 .../libdiagnostics.dg/test-helpers++.h|  28 ++
 .../libdiagnostics.dg/test-helpers.h  |  72 +
 .../libdiagnostics.dg/test-labelled-ranges.c  |  69 
 .../libdiagnostics.dg/test-labelled-ranges.cc |  64 
 .../libdiagnostics.dg/test-labelled-ranges.py |  48 +++
 .../test-logical-location-c.py|  37 +++
 .../libdiagnostics.dg/test-logical-location.c |  79 +
 .../libdiagnostics.dg/test-metadata-c.py  |  45 +++
 .../libdiagnostics.dg/test-metadata.c |  61 
 .../test-multiple-lines-c.py  |  83 +
 .../libdiagnostics.dg/test-multiple-lines.c   |  76 +
 .../libdiagnostics.dg/test-no-column-c.py |  35 +++
 .../libdiagnostics.dg/test-no-column.c|  52 +++
 .../test-no-diagnostics-c.py  |  42 +++
 .../libdiagnostics.dg/test-no-diagnostics.c   |  25 ++
 .../test-note-with-fix-it-hint-c.py   |  54 
 .../test-note-with-fix-it-hint.c  |  69 
 .../test-text-sink-options.c  |  59 
 .../libdiagnostics.dg/test-warning-c.py   |  54 
 .../test-warning-with-path-c.py   | 108 +++
 .../test-warning-with-path.c  | 125 
 .../libdiagnostics.dg/test-warning.c  |  65 
 .../test-write-sarif-to-file-c.py |  55 
 .../test-write-sarif-to-file.c|  55 
 .../test-write-text-to-file.c |  47 +++
 37 files changed, 2435 insertions(+)
 create mode 100644 gcc/testsuite/libdiagnostics.dg/libdiagnostics.exp
 create mode 100644 gcc/testsuite/libdiagnostics.dg/sarif.py
 create mode 100644 gcc/testsuite/libdiagnostics.dg/test-dump.c
 create mode 100644 gcc/testsuite/libdiagnostics.dg/test-error-c.py
 create mode 100644 gcc/testsuite

[PATCH 4/7] testsuite: move dg-test cleanup code from gcc-dg.exp to its own file

[David Malcolm]

I need to use this cleanup logic for the testsuite for libdiagnostics
where it's too awkward to directly use gcc-dg.exp itself.

No functional change intended.

gcc/testsuite/ChangeLog:
* lib/dg-test-cleanup.exp: New file, from material moved from
lib/gcc-dg.exp.
* lib/gcc-dg.exp: Add load_lib of dg-test-cleanup.exp.
(cleanup-after-saved-dg-test): Move to lib/dg-test-cleanup.exp.
(dg-test): Likewise for override.
(initialize_prune_notes): Likewise.

libatomic/ChangeLog:
* testsuite/lib/libatomic.exp: Add
"load_gcc_lib dg-test-cleanup.exp".

libgomp/ChangeLog:
* testsuite/lib/libgomp.exp: Add
"load_gcc_lib dg-test-cleanup.exp".
libitm/ChangeLog:
* testsuite/lib/libitm.exp: Add
"load_gcc_lib dg-test-cleanup.exp".

libphobos/ChangeLog:
* testsuite/lib/libphobos-dg.exp: Add
"load_gcc_lib dg-test-cleanup.exp".

libstdc++-v3/ChangeLog:
* testsuite/lib/libstdc++.exp: Add
"load_gcc_lib dg-test-cleanup.exp".

libvtv/ChangeLog:
* testsuite/lib/libvtv.exp: Add
    "load_gcc_lib dg-test-cleanup.exp".

Signed-off-by: David Malcolm 
---
 gcc/testsuite/lib/dg-test-cleanup.exp| 116 +++
 gcc/testsuite/lib/gcc-dg.exp | 102 +---
 libatomic/testsuite/lib/libatomic.exp|   1 +
 libgomp/testsuite/lib/libgomp.exp|   1 +
 libitm/testsuite/lib/libitm.exp  |   1 +
 libphobos/testsuite/lib/libphobos-dg.exp |   1 +
 libstdc++-v3/testsuite/lib/libstdc++.exp |   1 +
 libvtv/testsuite/lib/libvtv.exp  |   1 +
 8 files changed, 123 insertions(+), 101 deletions(-)
 create mode 100644 gcc/testsuite/lib/dg-test-cleanup.exp

diff --git a/gcc/testsuite/lib/dg-test-cleanup.exp 
b/gcc/testsuite/lib/dg-test-cleanup.exp
new file mode 100644
index ..b2b8507a0320
--- /dev/null
+++ b/gcc/testsuite/lib/dg-test-cleanup.exp
@@ -0,0 +1,116 @@
+#   Copyright (C) 1997-2024 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING3.  If not see
+# <http://www.gnu.org/licenses/>.
+
+# We need to make sure that additional_* are cleared out after every
+# test.  It is not enough to clear them out *before* the next test run
+# because gcc-target-compile gets run directly from some .exp files
+# (outside of any test).  (Those uses should eventually be eliminated.)
+
+# Because the DG framework doesn't provide a hook that is run at the
+# end of a test, we must replace dg-test with a wrapper.
+
+if { [info procs saved-dg-test] == [list] } {
+rename dg-test saved-dg-test
+
+# Helper function for cleanups that should happen after the call
+# to the real dg-test, whether or not it returns normally, or
+# fails with an error.
+proc cleanup-after-saved-dg-test { } {
+   global additional_files
+   global additional_sources
+   global additional_sources_used
+   global additional_prunes
+   global compiler_conditional_xfail_data
+   global shouldfail
+   global expect_ice
+   global testname_with_flags
+   global set_target_env_var
+   global set_compiler_env_var
+   global saved_compiler_env_var
+   global keep_saved_temps_suffixes
+   global nn_line_numbers_enabled
+   global multiline_expected_outputs
+   global freeform_regexps
+   global save_linenr_varnames
+
+   set additional_files ""
+   set additional_sources ""
+   set additional_sources_used ""
+   set additional_prunes ""
+   set shouldfail 0
+   set expect_ice 0
+   if [info exists set_target_env_var] {
+   unset set_target_env_var
+   }
+   if [info exists set_compiler_env_var] {
+   restore-compiler-env-var
+   unset set_compiler_env_var
+   unset saved_compiler_env_var
+   }
+   if [info exists keep_saved_temps_suffixes] {
+   unset keep_saved_temps_suffixes
+   }
+   unset_timeout_vars
+   if [info exists compiler_conditional_xfail_data] {
+   unset compiler_conditional_xfail_data
+   }
+   if [info exists testname_with_flags] {
+   unset testname_with_flags
+   }
+   set nn_line_numbers_enabled 0
+   set multiline_expected_outputs []
+   set freefor

[PATCH 3/7] libdiagnostics v3: add C++ wrapper API

[David Malcolm]

Changed in v3:
* Moved the testsuite to a separate patch
* Updated copyright year
* class text_sink: New.
* class file: Add default ctor, copy ctor, move ctor; make m_inner
  non-const
* class physical_location: Add default ctor
* class logical_location: Make m_inner non-const
* class execution_path: New
* class diagnostic: Add member functions: add_rule, take_execution_path,
  finish_va
* class manager: Add alternate ctor; add m_owned bool and use in dtor;
  delete copy ctor; add move ctor; add member functions set_tool_name,
  set_full-name, set_version_string, set_version_url,
  new_execution_path.  Add param "main_input_file" to add_sarif_sink.

Blurb from v2:

This is new in v2: a C++ wrapper API that provides some syntactic sugar for
calling into libdiagnostics.{h,so}.

I've been "eating my own dogfood" with this by using it to write a simple
client that reads a SARIF file and dumps it using the text sink:
  https://github.com/davidmalcolm/libdiagnostics-sarif-dump

gcc/ChangeLog:
* libdiagnostics++.h: New file.
---
 gcc/libdiagnostics++.h | 595 +
 1 file changed, 595 insertions(+)
 create mode 100644 gcc/libdiagnostics++.h

diff --git a/gcc/libdiagnostics++.h b/gcc/libdiagnostics++.h
new file mode 100644
index ..14c84934a446
--- /dev/null
+++ b/gcc/libdiagnostics++.h
@@ -0,0 +1,595 @@
+/* A C++ wrapper API around libdiagnostics.h for emitting diagnostics.
+   Copyright (C) 2023-2024 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 3, or (at your option)
+any later version.
+
+GCC is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+.  */
+
+#ifndef LIBDIAGNOSTICSPP_H
+#define LIBDIAGNOSTICSPP_H
+
+#include "libdiagnostics.h"
+
+namespace libdiagnostics {
+
+typedef diagnostic_line_num_t line_num_t;
+typedef diagnostic_column_num_t column_num_t;
+
+class file;
+class physical_location;
+class logical_location;
+class execution_path;
+class group;
+class manager;
+class diagnostic;
+
+/* Wrapper around a borrowed diagnostic_text_sink *.  */
+
+class text_sink
+{
+public:
+  text_sink (diagnostic_text_sink *inner)
+  : m_inner (inner)
+  {
+  }
+
+  void
+  set_source_printing_enabled (int value)
+  {
+diagnostic_text_sink_set_source_printing_enabled (m_inner, value);
+  }
+
+  void
+  set_colorize (enum diagnostic_colorize colorize)
+  {
+diagnostic_text_sink_set_colorize (m_inner, colorize);
+  }
+
+  void
+  set_labelled_source_colorization_enabled (int value)
+  {
+diagnostic_text_sink_set_labelled_source_colorization_enabled (m_inner,
+  value);
+  }
+
+  diagnostic_text_sink *m_inner;
+};
+
+/* Wrapper around a const diagnostic_file *.  */
+
+class file
+{
+public:
+  file () : m_inner (nullptr) {}
+  file (const diagnostic_file *file) : m_inner (file) {}
+  file (const file &other) : m_inner (other.m_inner) {}
+  file &operator= (const file &other) { m_inner = other.m_inner; return *this; 
}
+
+  const diagnostic_file * m_inner;
+};
+
+/* Wrapper around a const diagnostic_physical_location *.  */
+
+class physical_location
+{
+public:
+  physical_location () : m_inner (nullptr) {}
+
+  physical_location (const diagnostic_physical_location *location)
+  : m_inner (location)
+  {}
+
+  const diagnostic_physical_location *m_inner;
+};
+
+/* Wrapper around a const diagnostic_logical_location *.  */
+
+class logical_location
+{
+public:
+  logical_location () : m_inner (nullptr) {}
+
+  logical_location (const diagnostic_logical_location *logical_loc)
+  : m_inner (logical_loc)
+  {}
+
+  const diagnostic_logical_location *m_inner;
+};
+
+/* RAII class around a diagnostic_execution_path *.  */
+
+class execution_path
+{
+public:
+  execution_path () : m_inner (nullptr), m_owned (false) {}
+
+  execution_path (diagnostic_execution_path *path)
+  : m_inner (path), m_owned (true)
+  {}
+
+  execution_path (const diagnostic_execution_path *path)
+  : m_inner (const_cast (path)),
+m_owned (false)
+  {}
+
+  execution_path (const execution_path &other) = delete;
+  execution_path &operator= (const execution_path &other) = delete;
+
+  execution_path (execution_path &&other)
+  : m_inner (other.m_inner),
+m_owned (other.m_owned)
+  {
+other.m_inner = nullptr;
+other.m_owned = false;
+  }
+
+  execution_path &operator= (execution_path &&other)
+  {
+m_inner = other.m_inner;
+m_owned = other.m_owned;
+other.m_inner = nullptr;
+other.m_owned 

[PATCH 1/7] libdiagnostics v3: header

[David Malcolm]

Changed in v3:
* Added support for execution paths
* Moved the test cases to another patch
* diagnostic_manager_add_sarif_sink: add param "main_input_file"
* Added diagnostic_text_sink_set_colorize
* Added DIAGNOSTIC_LEVEL_SORRY
* Updated copyright year

Changed in v2:
* Changed from diagnostic_location_t -> const diagnostic_physical_location *
* Add entrypoint: diagnostic_finish_va
* add new type diagnostic_text_sink, and new entrypoints for
  enabling/disabling options on it
* add new debugging entrypoints for dumping objects to a FILE *
* new test cases

Blurb from v1:

Here's a work-in-progress patch for GCC that adds a libdiagnostics.h
header describing the public interface, along with various testcases
that show usage examples for the API.  Various aspects of this need
work; posting now for early feedback on overall direction.

How does the interface look?

gcc/ChangeLog:
* libdiagnostics.h: New file.
---
 gcc/libdiagnostics.h | 691 +++
 1 file changed, 691 insertions(+)
 create mode 100644 gcc/libdiagnostics.h

diff --git a/gcc/libdiagnostics.h b/gcc/libdiagnostics.h
new file mode 100644
index ..43138a911123
--- /dev/null
+++ b/gcc/libdiagnostics.h
@@ -0,0 +1,691 @@
+/* A pure C API for emitting diagnostics.
+   Copyright (C) 2023-2024 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 3, or (at your option)
+any later version.
+
+GCC is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+.  */
+
+#ifndef LIBDIAGNOSTICS_H
+#define LIBDIAGNOSTICS_H
+
+#include 
+#include 
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+/**
+ Compatibility macros.
+ **/
+
+/* This macro simplifies testing whether we are using gcc, and if it
+   is of a particular minimum version. (Both major & minor numbers are
+   significant.)  This macro will evaluate to 0 if we are not using
+   gcc at all.  */
+#define LIBDIAGNOSTICS_GCC_VERSION (__GNUC__ * 1000 + __GNUC_MINOR__)
+
+/**
+ Macros for attributes.
+ **/
+
+# if (LIBDIAGNOSTICS_GCC_VERSION >= 3003)
+#  define LIBDIAGNOSTICS_PARAM_MUST_BE_NON_NULL(ARG_NUM) __attribute__ 
((__nonnull__ (ARG_NUM)))
+# else
+#  define LIBDIAGNOSTICS_PARAM_MUST_BE_NON_NULL(ARG_NUM)
+# endif /* GNUC >= 3.3 */
+
+#define LIBDIAGNOSTICS_PARAM_CAN_BE_NULL(ARG_NUM)
+  /* empty; for the human reader */
+
+#define LIBDIAGNOSTICS_PARAM_GCC_FORMAT_STRING(FMT_ARG_NUM, ARGS_ARG_NUM) \
+  LIBDIAGNOSTICS_PARAM_MUST_BE_NON_NULL (FMT_ARG_NUM)
+  /* In theory we'd also add
+   __attribute__ ((__format__ (__gcc_diag__, FMT_ARG_NUM, ARGS_ARG_NUM)))
+ if LIBDIAGNOSTICS_GCC_VERSION >= 4001
+ However, doing so leads to warnings from -Wformat-diag, which is part
+ of -Wall but undocumented, and much fussier than I'd want to inflict
+ on users of libdiagnostics.  */
+
+/**
+ Data structures and types.
+ All structs within the API are opaque.
+ **/
+
+/* An opaque bundle of state for a client of the library.
+   Has zero of more "sinks" to which diagnostics are emitted.
+   Responsibilities:
+   - location-management
+   - caching of source file content
+   - patch generation.  */
+typedef struct diagnostic_manager diagnostic_manager;
+
+/* Types relating to diagnostic output sinks.  */
+
+typedef struct diagnostic_text_sink diagnostic_text_sink;
+
+/* An enum for determining if we should colorize a text output sink.  */
+enum diagnostic_colorize
+{
+  DIAGNOSTIC_COLORIZE_IF_TTY,
+  DIAGNOSTIC_COLORIZE_NO,
+  DIAGNOSTIC_COLORIZE_YES
+};
+
+/* An enum for choosing the SARIF version for a SARIF output sink.
+   Eventually the SARIF output may support multiple SARIF versions.  */
+
+enum diagnostic_sarif_version
+{
+  DIAGNOSTIC_SARIF_VERSION_2_1_0
+};
+
+/* Types relating to "physical" source locations i.e. locations within
+   specific files expressed via line/column.  */
+
+/* Opaque type describing a particular input file.  */
+typedef struct diagnostic_file diagnostic_file;
+
+/* Opaque type representing a key into a database of source locations within
+   a diagnostic_manager.  Locations are created by various API 

[PATCH 0/7] v3 of libdiagnostics

[David Malcolm]

Here's v3 of my patch kit for "libdiagnostics", which makes GCC's
diagnostics subsystem available as a shared library; see:
  https://gcc.gnu.org/wiki/libdiagnostics

New in v3:
* it bootstraps and pass regression tests
* I added an opt-in configure flag: --enable-libdiagnostics, which
  must be enabled to build it (along with --enable-host-shared)
* a new "sarif-replay" command-line tool that takes .sarif files
  and replays the diagnostics within them as if they were GCC
  diagnostics, in GCC's textual format (i.e. GCC as a SARIF *consumer*,
  as well as producer).  This is implemented on top of libdiagnostics
  hence I've been "eating my own dogfood"
* support for execution paths in libdiagnostics API
* lots of fixes

Patch 1 has libdiagnostic.h, the public header file
Patch 2 has the implementation
Patch 3 has the C++ wrapper API I added in v2
Patch 4 has a refactoring of gcc-dg.exp I needed for patch 5.
Patch 5 has the testsuite for libdiagnostics itself
Patch 6 implements JSON parsing support
Patch 7 implements the sarif-replay command-line tool, and its
  testsuite, exercising various valid, invalid, and malformed
  input files.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu,
both with and without --enable-libdiagnostics.
With --enable-libdiagnostics the patch kit has this effect:

  # of .sum files: 20->22 (+2)
  FAIL: 110
  PASS: 617481->617679 (+198) 100.03%
  XFAIL: 4512
  XPASS: 13
  UNTESTED: 136
  UNSUPPORTED: 8058

where the two new .sum files are:

  BUILD/gcc/testsuite/libdiagnostics/libdiagnostics.sum: 
PASS: 132 tests

  BUILD/gcc/testsuite/sarif-replay/sarif-replay.sum:
PASS: 66 tests

OK for trunk?

David Malcolm (7):
  libdiagnostics v3: header
  libdiagnostics v3: implementation
  libdiagnostics v3: add C++ wrapper API
  testsuite: move dg-test cleanup code from gcc-dg.exp to its own file
  libdiagnostics v3: test suite
  json: add json parsing support
  libdiagnostics: add a "sarif-replay" command-line tool [PR96032]

 configure |   42 +
 configure.ac  |   35 +
 contrib/regenerate-sarif-spec-index.py|   60 +
 gcc/Makefile.in   |  191 +-
 gcc/configure |   26 +-
 gcc/configure.ac  |   16 +
 gcc/diagnostic-event-id.h |6 +
 gcc/doc/install.texi  |   13 +
 gcc/json-parsing.cc   | 2394 +
 gcc/json-parsing.h|  113 +
 gcc/json.cc   |2 +-
 gcc/json.h|  122 +-
 gcc/libdiagnostics++.h|  595 
 gcc/libdiagnostics.cc | 1652 
 gcc/libdiagnostics.h  |  691 +
 gcc/libdiagnostics.map|   72 +
 gcc/libsarifreplay.cc | 1747 
 gcc/libsarifreplay.h  |   59 +
 gcc/sarif-replay.cc   |  239 ++
 gcc/sarif-spec-urls.def   |  496 
 gcc/selftest-run-tests.cc |1 +
 gcc/selftest.h|1 +
 gcc/testsuite/lib/dg-test-cleanup.exp |  116 +
 gcc/testsuite/lib/gcc-dg.exp  |  106 +-
 gcc/testsuite/lib/sarif-replay-dg.exp |   90 +
 gcc/testsuite/lib/sarif-replay.exp|  204 ++
 .../libdiagnostics.dg/libdiagnostics.exp  |  296 ++
 gcc/testsuite/libdiagnostics.dg/sarif.py  |   23 +
 gcc/testsuite/libdiagnostics.dg/test-dump.c   |   69 +
 .../libdiagnostics.dg/test-error-c.py |   54 +
 .../test-error-with-note-c.py |   50 +
 .../libdiagnostics.dg/test-error-with-note.c  |   74 +
 .../libdiagnostics.dg/test-error-with-note.cc |   55 +
 gcc/testsuite/libdiagnostics.dg/test-error.c  |   59 +
 gcc/testsuite/libdiagnostics.dg/test-error.cc |   47 +
 .../libdiagnostics.dg/test-fix-it-hint-c.py   |   46 +
 .../libdiagnostics.dg/test-fix-it-hint.c  |   81 +
 .../libdiagnostics.dg/test-fix-it-hint.cc |   74 +
 .../libdiagnostics.dg/test-helpers++.h|   28 +
 .../libdiagnostics.dg/test-helpers.h  |   72 +
 .../libdiagnostics.dg/test-labelled-ranges.c  |   69 +
 .../libdiagnostics.dg/test-labelled-ranges.cc |   64 +
 .../libdiagnostics.dg/test-labelled-ranges.py |   48 +
 .../test-logical-location-c.py|   37 +
 .../libdiagnostics.dg/test-logical-location.c |   79 +
 .../libdiagnostics.dg/test-metadata-c.py  |   45 +
 .../libdiagnostics.dg/test-metadata.c |   61 +
 .../test-multiple-lines-c.py  |   83 +
 .../libdiagnostics.dg/test-multiple-lines.c   |   76 +
 .../libdiagnostics.dg/test-no-column-c.py |   35 +
 .../libdiagnostics.dg/test-no-column.c|   52 +
 .../test-no-dia

Re: [PATCH v2] c++: improve diagnostic of 'return's in coroutines

[David Malcolm]

On Thu, 2024-08-08 at 22:29 +0200, Arsen Arsenović wrote:
> Tested on x86_64-pc-linux-gnu.  I have blinking tsan test results
> again,
> but I think they're bogus (I'll re-test on physical hardware before
> pushing if needed).
> 
> I'm somewhat curious of we should do a similar change WRT
> RETURN_EXPRs
> in the C FE (currently, the C FE uses the operand location for its
> RETURN_EXPR locations, or the return kw if missing, which I suspect
> is
> why malloc-CWE-401-example.c fails in C today, which appears
> confirmed
> by -Wsystem-headers de-suppressing it).

FWIW I've now filed the false negative for malloc-CWE-401-example.c as:
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=116304


Dave

Re: [PATCH] c++: permit errors inside uninstantiated templates [PR116064]

[David Malcolm]

On Tue, 2024-08-06 at 17:52 -0400, Jason Merrill wrote:
> On 8/6/24 5:47 PM, Patrick Palka wrote:
> > On Tue, 6 Aug 2024, Jason Merrill wrote:
> > 
> > > On 8/6/24 2:00 PM, Patrick Palka wrote:
> > > > On Tue, 6 Aug 2024, Jason Merrill wrote:
> > > > 
> > > > > On 8/5/24 6:09 PM, Patrick Palka wrote:
> > > > > > On Mon, 5 Aug 2024, Jason Merrill wrote:
> > > > > > 
> > > > > > > On 8/5/24 3:47 PM, Patrick Palka wrote:
> > > > > > > > On Mon, 5 Aug 2024, Jason Merrill wrote:
> > > > > > > > 
> > > > > > > > > On 8/5/24 1:14 PM, Patrick Palka wrote:
> > > > > > > > > > On Mon, 5 Aug 2024, Jason Merrill wrote:
> > > > > > > > > > 
> > > > > > > > > > > On 8/2/24 4:18 PM, Patrick Palka wrote:
> > > > > > > > > > > > On Fri, 2 Aug 2024, Patrick Palka wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > On Fri, 2 Aug 2024, Jason Merrill wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > On 8/1/24 2:52 PM, Patrick Palka wrote:
> > > > > > > > > > > > > > > In recent versions of GCC we've been
> > > > > > > > > > > > > > > diagnosing more
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > kinds of
> > > > > > > > > > > > > > > errors inside a template ahead of time. 
> > > > > > > > > > > > > > > This is a
> > > > > > > > > > > > > > > largely
> > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > thing
> > > > > > > > > > > > > > > because it catches bugs, typos, dead code
> > > > > > > > > > > > > > > etc sooner.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > But if the template never gets
> > > > > > > > > > > > > > > instantiated then such
> > > > > > > > > > > > > > > errors
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > harmless, and can be inconvenient to work
> > > > > > > > > > > > > > > around if
> > > > > > > > > > > > > > > say
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > code
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > question is third party and in
> > > > > > > > > > > > > > > maintenence mode.  So
> > > > > > > > > > > > > > > it'd
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > "maintenance"
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Fixed
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > diff --git a/gcc/cp/error.cc
> > > > > > > > > > > > > > > b/gcc/cp/error.cc
> > > > > > > > > > > > > > > index d80bac822ba..0bb0a482e28 100644
> > > > > > > > > > > > > > > --- a/gcc/cp/error.cc
> > > > > > > > > > > > > > > +++ b/gcc/cp/error.cc
> > > > > > > > > > > > > > > @@ -165,6 +165,58 @@ class
> > > > > > > > > > > > > > > cxx_format_postprocessor :
> > > > > > > > > > > > > > > public
> > > > > > > > > > > > > > > format_postprocessor
> > > > > > > > > > > > > > >   deferred_printed_type m_type_b;
> > > > > > > > > > > > > > >     };
> > > > > > > > > > > > > > >     +/* A map from TEMPLATE_DECL to
> > > > > > > > > > > > > > > the location of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > error (if
> > > > > > > > > > > > > > > any)
> > > > > > > > > > > > > > > +   within the template that we
> > > > > > > > > > > > > > > permissivly downgraded
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > warning.
> > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > "permissively"
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Fixed
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > +relaxed_template_errors_t
> > > > > > > > > > > > > > > *relaxed_template_errors;
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +/* Callback function
> > > > > > > > > > > > > > > diagnostic_context::m_adjust_diagnostic_i
> > > > > > > > > > > > > > > nfo.
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +   In -fpermissive mode we downgrade
> > > > > > > > > > > > > > > errors within a
> > > > > > > > > > > > > > > template
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > +   warnings, and only issue an error if
> > > > > > > > > > > > > > > we later need
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > instantiate
> > > > > > > > > > > > > > > +   the template.  */
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +static void
> > > > > > > > > > > > > > > +cp_adjust_diagnostic_info
> > > > > > > > > > > > > > > (diagnostic_context
> > > > > > > > > > > > > > > *context,
> > > > > > > > > > > > > > > + 
> > > > > > > > > > > > > > > diagnostic_info
> > > > > > > > > > > > > > > *diagnostic)
> > > > > > > > > > > > > > > +{
> > > > > > > > > > > > > > > +  tree ti;
> > > > > > > > > > > > > > > +  if (diagnostic->kind == DK_ERROR
> > > > > > > > > > > > > > > +  && context->m_permi

[pushed] diagnostics: SARIF output: fix "executionSuccessful" §3.20.14 [PR116177]

[David Malcolm]

Previously the invocation's "executionSuccessful" property (§3.20.14)
was only false if there was an ICE.

Update it so that it will also be false if we will exit with a non-zero
exit code (due to errors, Werror, and "sorry").

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-2769-g77f36e8016e11c.

gcc/ChangeLog:
PR other/116177
* diagnostic-format-sarif.cc (sarif_invocation::prepare_to_flush):
If the diagnostics would lead to us exiting with a failure code,
then emit "executionSuccessful": False (SARIF v2.1.0 section
§3.20.14).
* diagnostic.cc (diagnostic_context::execution_failed_p): New.
* diagnostic.h (diagnostic_context::execution_failed_p): New decl.
* toplev.cc (toplev::main): Use it for determining returned value.

gcc/testsuite/ChangeLog:
PR other/116177
* gcc.dg/sarif-output/include-chain-2.c: Remove pruning of
"exit status is 1", as we expect this to exit with 0.
* gcc.dg/sarif-output/no-diagnostics.c: New test.
* gcc.dg/sarif-output/test-include-chain-1.py
(test_execution_unsuccessful): Add.
* gcc.dg/sarif-output/test-include-chain-2.py
(test_execution_successful): Add.
* gcc.dg/sarif-output/test-missing-semicolon.py
(test_execution_unsuccessful): Add.
* gcc.dg/sarif-output/test-no-diagnostics.py: New test.
* gcc.dg/sarif-output/test-werror.py: New test.
* gcc.dg/sarif-output/werror.c: New test.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc|  2 +
 gcc/diagnostic.cc | 13 +++
 gcc/diagnostic.h  |  2 +
 .../gcc.dg/sarif-output/include-chain-2.c |  5 ---
 .../gcc.dg/sarif-output/no-diagnostics.c  | 13 +++
 .../sarif-output/test-include-chain-1.py  | 11 ++
 .../sarif-output/test-include-chain-2.py  | 11 ++
 .../sarif-output/test-missing-semicolon.py| 11 ++
 .../sarif-output/test-no-diagnostics.py   | 31 +++
 .../gcc.dg/sarif-output/test-werror.py| 39 +++
 gcc/testsuite/gcc.dg/sarif-output/werror.c| 18 +
 gcc/toplev.cc |  2 +-
 12 files changed, 152 insertions(+), 6 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/sarif-output/no-diagnostics.c
 create mode 100644 gcc/testsuite/gcc.dg/sarif-output/test-no-diagnostics.py
 create mode 100644 gcc/testsuite/gcc.dg/sarif-output/test-werror.py
 create mode 100644 gcc/testsuite/gcc.dg/sarif-output/werror.c

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 7c2e96f4f746..963a185f6ced 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -811,6 +811,8 @@ void
 sarif_invocation::prepare_to_flush (diagnostic_context &context)
 {
   /* "executionSuccessful" property (SARIF v2.1.0 section 3.20.14).  */
+  if (context.execution_failed_p ())
+m_success = false;
   set_bool ("executionSuccessful", m_success);
 
   /* "toolExecutionNotifications" property (SARIF v2.1.0 section 3.20.21).  */
diff --git a/gcc/diagnostic.cc b/gcc/diagnostic.cc
index 71d2f44e40c8..3fc81ad47f56 100644
--- a/gcc/diagnostic.cc
+++ b/gcc/diagnostic.cc
@@ -399,6 +399,19 @@ diagnostic_context::finish ()
   m_original_argv = nullptr;
 }
 
+/* Return true if sufficiently severe diagnostics have been seen that
+   we ought to exit with a non-zero exit code.  */
+
+bool
+diagnostic_context::execution_failed_p () const
+{
+  /* Equivalent to (seen_error () || werrorcount), but on
+ this context, rather than global_dc.  */
+  return (m_diagnostic_count [DK_ERROR]
+ || m_diagnostic_count [DK_SORRY]
+ || m_diagnostic_count [DK_WERROR]);
+}
+
 void
 diagnostic_context::set_output_format (diagnostic_output_format *output_format)
 {
diff --git a/gcc/diagnostic.h b/gcc/diagnostic.h
index 79386ccbf856..83180ded414d 100644
--- a/gcc/diagnostic.h
+++ b/gcc/diagnostic.h
@@ -392,6 +392,8 @@ public:
 
   void finish ();
 
+  bool execution_failed_p () const;
+
   void set_original_argv (unique_argv original_argv);
   const char * const *get_original_argv ()
   {
diff --git a/gcc/testsuite/gcc.dg/sarif-output/include-chain-2.c 
b/gcc/testsuite/gcc.dg/sarif-output/include-chain-2.c
index 3f984f48979b..a04b647d259e 100644
--- a/gcc/testsuite/gcc.dg/sarif-output/include-chain-2.c
+++ b/gcc/testsuite/gcc.dg/sarif-output/include-chain-2.c
@@ -27,11 +27,6 @@ PATH/include-chain-2.h:6:3: warning: double-'free' of 'ptr' 
[CWE-415] [-Wanalyze
 
 #include "include-chain-2.h"
 
-/* We expect a failing compile due to the errors, but the use of 
-   -fdiagnostics-format=sarif-file means there should be no output to stderr.
-   DejaGnu injects this message; ignore it:
-   { dg-prune-output "exit status i

Re: [PATCH 1/2] gdbhooks: Make dot viewer configurable

[David Malcolm]

On Thu, 2024-08-01 at 10:14 +0100, Alex Coplan wrote:
> Hi,
> 
> This adds a new GDB parameter 'gcc-dot-cmd' which allows the user to
> configure the command used to render the CFG within dot-fn.
> 
> E.g. with this patch the user can change their dot viewer like so:
> 
> (gdb) show gcc-dot-cmd
> The current value of 'gcc-dot-cmd' is "dot -Tx11".
> (gdb) set gcc-dot-cmd xdot
> (gdb) dot-fn # opens in xdot
> 
> The second patch in this series adds a hook which users can define in
> their .gdbinit in order to be called when the GCC extensions have
> finished loading, thus allowing users to automatically configure
> gcc-dot-cmd as desired in their .gdbinit.
> 
> Manually tested by debugging an x86 -> aarch64 cross, changing the
> parameter, and invoking dot-fn.
> 
> OK to install?

Thanks; both patches look good to me.
Dave

> 
> Thanks,
> Alex
> 
> gcc/ChangeLog:
> 
> * gdbhooks.py (GCCDotCmd): New.
> (gcc_dot_cmd): New. Use it ...
> (DotFn.invoke): ... here.

[PATCH] testsuite: add print-stack.exp

[David Malcolm]

I wrote this support file to help me debug Tcl issues in the
testsuite.

Adding a call to:

  print_stack_backtrace

somewhere in a .exp file (along with "load_lib print-stack.exp") leads
to the interpreter printing a backtrace in a form that e.g. Emacs can
consume, with filename:linenum: lines, and quoting the line of .exp
source code.

Fer example, adding a print_stack_backtrace to scansarif.exp in
run-sarif-pytest I get this output:

VVV START OF BACKTRACE VVV
  /home/david/coding/gcc-newgit/src/gcc/testsuite/lib/scansarif.exp:142: frame 
16 in proc print_stack_backtrace
142 | print_stack_backtrace
  : frame 15 in proc run-sarif-pytest
  : frame 14 in proc dg-final-proc
  /usr/share/dejagnu/dg.exp:851: frame 13 in proc dg-final-proc
851 |   if {[catch "dg-final-proc $prog" errmsg]} {
  : frame 12 in proc saved-dg-test
  /home/david/coding/gcc-newgit/src/gcc/testsuite/lib/gcc-dg.exp:1080: frame 11 
in proc saved-dg-test
1080 |  if { [ catch { eval saved-dg-test $args } errmsg ] } {
  /usr/share/dejagnu/dg.exp:559: frame 10 in proc dg-test
559 |   dg-test $testcase $options ${default-extra-options}
  
/home/david/coding/gcc-newgit/src/gcc/testsuite/gcc.dg/sarif-output/sarif-output.exp:28:
 frame 9
28 | dg-runtest [lsort [glob -nocomplain $srcdir/$subdir/*.c]] "" ""
  : frame 8
  : frame 7
  /usr/share/dejagnu/runtest.exp:1460: frame 6
1460 |  if { [catch "uplevel #0 source $test_file_name"] == 1 } {
  /usr/share/dejagnu/runtest.exp:1886: frame 5 in proc dg-runtest
1886 |  runtest $test_name
  /usr/share/dejagnu/runtest.exp:1845: frame 4 in proc dg-runtest
1845 |  foreach test_name [lsort [find ${dir} *.exp]] {
  /usr/share/dejagnu/runtest.exp:1788: frame 3 in proc dg-runtest
1788 |  foreach dir "${test_top_dirs}" {
  /usr/share/dejagnu/runtest.exp:1669: frame 2 in proc dg-runtest
1669 | foreach pass $multipass {
  /usr/share/dejagnu/runtest.exp:1619: frame 1 in proc dg-runtest
1619 | foreach current_target $target_list {
^^^  END OF BACKTRACE  ^^^

and can click on the lines in Emacs's compilation buffer to take
me to the relevant places.

I found this made it *much* easier to debug my .exp files.  That
said, I'm uncomfortable with Tcl, and so
(a) there may be a better way of doing this
(b) I may have made mistakes

OK for trunk?

gcc/testsuite/ChangeLog:
* lib/print-stack.exp: New file.

Signed-off-by: David Malcolm 
---
 gcc/testsuite/lib/print-stack.exp | 59 +++
 1 file changed, 59 insertions(+)
 create mode 100644 gcc/testsuite/lib/print-stack.exp

diff --git a/gcc/testsuite/lib/print-stack.exp 
b/gcc/testsuite/lib/print-stack.exp
new file mode 100644
index 000..5688d0a63de
--- /dev/null
+++ b/gcc/testsuite/lib/print-stack.exp
@@ -0,0 +1,59 @@
+# Copyright (C) 2024 Free Software Foundation, Inc.
+#  Contributed by David Malcolm .
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING3.  If not see
+# <http://www.gnu.org/licenses/>.
+
+# Get the 1-based line for LINENUM from FILENAME as a string
+
+proc get_line { filename linenum } {
+set f [open $filename]
+set lines [split [read $f] \n]
+close $f
+return [lindex $lines [expr $linenum - 1] ]
+}
+
+# Print a backtrace of the Tcl interpreter's stack, showing
+# frames, levels, source file and line where available.
+
+proc print_stack_backtrace {} {
+set current_frame_level [info frame]
+puts "VVV START OF BACKTRACE VVV"
+for {set i [expr $current_frame_level - 1]} {$i > 0} {incr i -1} {
+   set frame [info frame $i]
+   if { [dict exists $frame "level"] } {
+   set level_num [dict get $frame "level"]
+   set relative_level_offset [expr 1 - $level_num]
+   set level [info level $relative_level_offset]
+   set procname [lindex $level 0]
+   # TODO: args = rest of $level, but this can be very long
+   } else {
+   set procname ""
+   }
+   set suffix ""
+   if { $procname != "" } {
+   set suffix " in proc $procname"
+   }
+   if { [dict get $frame "type"] == "source" } {
+   set fname [dict get $frame "file"]
+   set line [dict g

[pushed] testsuite: split out parts of jit.dg/jit.exp into a new lib/valgrind.exp

[David Malcolm]

I want to reuse some of the support for valgrind in jit.exp
in my upcoming testsuite for https://gcc.gnu.org/wiki/libdiagnostics
so this patch splits that out into a valgrind.exp.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-2468-g64fbaf36a3834c.

gcc/testsuite/ChangeLog:
* jit.dg/jit.exp: Add load_lib valgrind.exp.
(proc report_leak): Move to valgrind.exp, and add argument
leak_report_function rather than hardcoding xfail.
(parse_valgrind_logfile): Likewise.
(fixed_host_execute): Pass xfail to parse_valgrind_logfile.
* lib/valgrind.exp: New file, based on the above.

Signed-off-by: David Malcolm 
---
 gcc/testsuite/jit.dg/jit.exp   | 47 +++
 gcc/testsuite/lib/valgrind.exp | 58 ++
 2 files changed, 62 insertions(+), 43 deletions(-)
 create mode 100644 gcc/testsuite/lib/valgrind.exp

diff --git a/gcc/testsuite/jit.dg/jit.exp b/gcc/testsuite/jit.dg/jit.exp
index 893ff5f6dd0..57b133b6d8c 100644
--- a/gcc/testsuite/jit.dg/jit.exp
+++ b/gcc/testsuite/jit.dg/jit.exp
@@ -38,6 +38,7 @@ load_lib gcc.exp
 load_lib g++.exp
 load_lib dejagnu.exp
 load_lib target-supports-dg.exp
+load_lib valgrind.exp
 
 # Skip these tests for targets that don't support -lgccjit
 if { ![check_effective_target_lgccjit] } {
@@ -47,48 +48,6 @@ if { ![check_effective_target_lgccjit] } {
 # The default do-what keyword.
 set dg-do-what-default compile
 
-# Look for lines of the form:
-#   definitely lost: 11,316 bytes in 235 blocks
-#   indirectly lost: 352 bytes in 4 blocks
-# Ideally these would report zero bytes lost (which is a PASS);
-# for now, report non-zero leaks as XFAILs.
-proc report_leak {kind name logfile line} {
-set match [regexp "$kind lost: .*" $line result]
-if $match {
-   verbose "Saw \"$result\" within \"$line\"" 4
-   # Extract bytes and blocks.
-   # These can contain commas as well as numerals,
-   # but we only care about whether we have zero.
-   regexp "$kind lost: (.+) bytes in (.+) blocks" \
-   $result -> bytes blocks
-   verbose "bytes: '$bytes'" 4
-   verbose "blocks: '$blocks'" 4
-   if { $bytes == 0 } {
-   pass "$name: $logfile: $result"
-   } else {
-   xfail "$name: $logfile: $result"
-   }
-}
-}
-
-proc parse_valgrind_logfile {name logfile} {
-verbose "parse_valgrind_logfile: $logfile" 2
-if [catch {set f [open $logfile]}] {
-   fail "$name: unable to read $logfile"
-   return
-}
-
-while { [gets $f line] >= 0 } {
-   # Strip off the PID prefix e.g. ==7675==
-   set line [regsub "==\[0-9\]*== " $line ""]
-   verbose $line 2
-
-   report_leak "definitely" $name $logfile $line
-   report_leak "indirectly" $name $logfile $line
-}
-close $f
-}
-
 # Given WRES, the result from "wait", issue a PASS
 # if the spawnee exited cleanly, or a FAIL for various kinds of
 # unexpected exits.
@@ -327,7 +286,9 @@ proc fixed_host_execute {args} {
  
 if $run_under_valgrind {
upvar 2 name name
-   parse_valgrind_logfile $name $valgrind_logfile
+   # Use xfail to report leaks, as libgccjit isn't yet clean of
+   # memory leaks (PR jit/63854)
+   parse_valgrind_logfile $name $valgrind_logfile xfail
 }
 
 # force a close of the executable to be safe.
diff --git a/gcc/testsuite/lib/valgrind.exp b/gcc/testsuite/lib/valgrind.exp
new file mode 100644
index 000..7d4f7ce51da
--- /dev/null
+++ b/gcc/testsuite/lib/valgrind.exp
@@ -0,0 +1,58 @@
+# Copyright (C) 2014-2024 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING3.  If not see
+# <http://www.gnu.org/licenses/>.
+
+# Look for lines of the form:
+#   definitely lost: 11,316 bytes in 235 blocks
+#   indirectly lost: 352 bytes in 4 blocks
+# Report zero bytes lost as a a PASS.
+# Use LEAK_REPORT_FUNCTION to report non-zero bytes lost (either fail or xfail)
+
+proc report_leak {kind name logfile line leak_report_function} {
+set match [regexp "$kind lost: .*" $line result]
+if $match {
+   verbose "Saw \"$res

[pushed] diagnostics: handle logical locations with NULL name

[David Malcolm]

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Successful run of analyzer integration tests on x86_64-pc-linux-gnu.
Pushed to trunk as r15-2467-g55982d1682921f.

gcc/ChangeLog:
* diagnostic-path.cc
(thread_event_printer::print_swimlane_for_event_range): Gracefully
handle logical_location::get_name_for_path_output returning null.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-path.cc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gcc/diagnostic-path.cc b/gcc/diagnostic-path.cc
index b497d89d059..37751843f9a 100644
--- a/gcc/diagnostic-path.cc
+++ b/gcc/diagnostic-path.cc
@@ -860,7 +860,8 @@ public:
 if (const logical_location *logical_loc = range->m_logical_loc)
   {
label_text name (logical_loc->get_name_for_path_output ());
-   pp_printf (pp, "%qs: ", name.get ());
+   if (name.get ())
+ pp_printf (pp, "%qs: ", name.get ());
   }
 if (range->m_start_idx == range->m_end_idx)
   pp_printf (pp, "event %i",
-- 
2.26.3

[pushed] testsuite: drop unused import within sarif.py

[David Malcolm]

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.

Pused to trunk as r15-2466-g5cb7adeaf5420c.

gcc/testsuite/ChangeLog:
* gcc.dg/sarif-output/sarif.py: Drop unused import of gzip.

Signed-off-by: David Malcolm 
---
 gcc/testsuite/gcc.dg/sarif-output/sarif.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/gcc/testsuite/gcc.dg/sarif-output/sarif.py 
b/gcc/testsuite/gcc.dg/sarif-output/sarif.py
index a34678791ac..7daf35b5819 100644
--- a/gcc/testsuite/gcc.dg/sarif-output/sarif.py
+++ b/gcc/testsuite/gcc.dg/sarif-output/sarif.py
@@ -1,4 +1,3 @@
-import gzip
 import json
 import os
 
-- 
2.26.3

[pushed] diagnostics: SARIF output: capture unlabelled secondary locations

[David Malcolm]

This patch extends
* the work done in r15-2291-gd7a688fc960f78 to capture labels
  on location ranges in rich_locations in SARIF form as
  "annotations" (§3.28.6)
* the work done in r15-2354-g4d1f71d49e396c to support
  related locations (§3.27.22 and §3.34)

so that all location ranges in a rich_location now get captured in
the SARIF output:
- those with a label are handled as before as "annotations" (§3.28.6),
  per r15-2291-gd7a688fc960f78
- those without a label now get captured, in the result's
  "relatedLocations" (§3.27.22)

For example, given:

  int missing_semicolon (void)
  {
return 42
  }

for which the textual output looks like this:

  PATH/missing-semicolon.c: In function 'missing_semicolon':
  PATH/missing-semicolon.c:9:12: error: expected ';' before '}' token
  9 |   return 42
|^
|;
 10 | }
| ~

with this patch the SARIF output now has this for the result's location:

   "relationships": [{"target": 0,
  "kinds": ["relevant"]}]}],

where the result gains a related location :

  "relatedLocations": [{"physicalLocation": {"artifactLocation": { [...snip...] 
},
 "region": {"startLine": 10,
"startColumn": 1,
"endColumn": 2},
 "contextRegion": {"startLine": 10,
   "snippet": 
{"text": "}\n"}}},
"id": 0}]}]}]}

i.e. that the error also has the secondary location at the trailing
close brace which has the relationship "relevant" to the primary
location (at the suggested insertion point).

The patch also adds test coverage for the SARIF encoding of the fix-it hint.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Successful run of analyzer integration tests on x86_64-pc-linux-gnu.
Pushed to trunk as r15-2465-ga874b8301d9aa0.

gcc/ChangeLog:
* diagnostic-format-sarif.cc
(sarif_location_manager::worklist_item::unlabelled_secondary_location):
New enum value.
(sarif_location_manager::m_unlabelled_secondary_locations): New
field.
(sarif_location_manager::process_worklist_item): Handle unlabelled
secondary locations.
(sarif_builder::make_location_object): Generalize code to handle
ranges within a rich_location so as well as using annotations for
those with labels, we now add related locations for those without
labels.

gcc/testsuite/ChangeLog:
* gcc.dg/sarif-output/missing-semicolon.c: New test.
* gcc.dg/sarif-output/sarif.py (get_location_physical_region): New.
(get_location_snippet_text): New.
* gcc.dg/sarif-output/test-missing-semicolon.py: New test.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc| 54 +++--
 .../gcc.dg/sarif-output/missing-semicolon.c   | 22 ++
 gcc/testsuite/gcc.dg/sarif-output/sarif.py|  3 +
 .../sarif-output/test-missing-semicolon.py| 79 +++
 4 files changed, 152 insertions(+), 6 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/sarif-output/missing-semicolon.c
 create mode 100644 gcc/testsuite/gcc.dg/sarif-output/test-missing-semicolon.py

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 2232883281b..7c2e96f4f74 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -323,7 +323,11 @@ public:
 {
  /* Process a #include relationship where m_location_obj
was #included-d at m_where.  */
- included_from
+ included_from,
+
+ /* Process a location_t that was added as a secondary location
+   to a rich_location without a label.  */
+ unlabelled_secondary_location
 };
 
 worklist_item (sarif_location &location_obj,
@@ -369,6 +373,7 @@ private:
 
   std::list m_worklist;
   std::map m_included_from_locations;
+  std::map m_unlabelled_secondary_locations;
 };
 
 /* Subclass of sarif_object for SARIF "result" objects
@@ -559,6 +564,7 @@ public:
- diagnostic groups (see limitations below)
- logical locations (e.g. cfun)
- labelled ranges (as annotations)
+   - secondary ranges without labels (as related locations)
 
Known limitations:
- GCC supports one-deep nesting of diagnostics (via auto_diagnostic_group),
@@ -566,9 +572,6 @@ public:
  diagnostics (e.g. we ignore fix-it hints on them)
- although we capture command-line arguments (section 3.20.2), we don't
  yet capture response files.
-   - doesn't capture secondary

[pushed] diagnostics: SARIF output: eliminate some uses of "line_table" global

[David Malcolm]

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Successful run of analyzer integration tests on x86_64-pc-linux-gnu.
Pushed to trunk as r15-2464-gc990667996ff79.

gcc/ChangeLog:
* diagnostic-format-sarif.cc (sarif_builder::sarif_builder): Assert
that m_line_maps is nonnull.
(diagnostic_output_format_init_sarif_stderr): Add "line_maps"
param and pass to format ctor.
(diagnostic_output_format_init_sarif_file): Likewise.
(diagnostic_output_format_init_sarif_stream): Likewise.
* diagnostic.cc (diagnostic_output_format_init): Pass "line_table"
as line_maps param to the above.
* diagnostic.h (diagnostic_output_format_init_sarif_stderr): Add
"line_maps" param.
(diagnostic_output_format_init_sarif_file): Likewise.
(diagnostic_output_format_init_sarif_stream): Likewise.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc | 14 +++---
 gcc/diagnostic.cc  |  2 ++
 gcc/diagnostic.h   |  3 +++
 3 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 5900b9ea9b7..2232883281b 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -1302,6 +1302,8 @@ sarif_builder::sarif_builder (diagnostic_context &context,
   m_tabstop (context.m_tabstop),
   m_formatted (formatted)
 {
+  gcc_assert (m_line_maps);
+
   /* Mark MAIN_INPUT_FILENAME_ as the artifact that the tool was
  instructed to scan.
  Only quote the contents if it gets referenced by physical locations,
@@ -2988,13 +2990,15 @@ diagnostic_output_format_init_sarif (diagnostic_context 
&context)
 
 void
 diagnostic_output_format_init_sarif_stderr (diagnostic_context &context,
+   const line_maps *line_maps,
const char *main_input_filename_,
bool formatted)
 {
+  gcc_assert (line_maps);
   diagnostic_output_format_init_sarif (context);
   context.set_output_format
 (new sarif_stream_output_format (context,
-line_table,
+line_maps,
 main_input_filename_,
 formatted,
 stderr));
@@ -3005,14 +3009,16 @@ diagnostic_output_format_init_sarif_stderr 
(diagnostic_context &context,
 
 void
 diagnostic_output_format_init_sarif_file (diagnostic_context &context,
+ const line_maps *line_maps,
  const char *main_input_filename_,
  bool formatted,
  const char *base_file_name)
 {
+  gcc_assert (line_maps);
   diagnostic_output_format_init_sarif (context);
   context.set_output_format
 (new sarif_file_output_format (context,
-  line_table,
+  line_maps,
   main_input_filename_,
   formatted,
   base_file_name));
@@ -3022,14 +3028,16 @@ diagnostic_output_format_init_sarif_file 
(diagnostic_context &context,
 
 void
 diagnostic_output_format_init_sarif_stream (diagnostic_context &context,
+   const line_maps *line_maps,
const char *main_input_filename_,
bool formatted,
FILE *stream)
 {
+  gcc_assert (line_maps);
   diagnostic_output_format_init_sarif (context);
   context.set_output_format
 (new sarif_stream_output_format (context,
-line_table,
+line_maps,
 main_input_filename_,
 formatted,
 stream));
diff --git a/gcc/diagnostic.cc b/gcc/diagnostic.cc
index 46cddfe94d1..71d2f44e40c 100644
--- a/gcc/diagnostic.cc
+++ b/gcc/diagnostic.cc
@@ -1857,12 +1857,14 @@ diagnostic_output_format_init (diagnostic_context 
&context,
 
 case DIAGNOSTICS_OUTPUT_FORMAT_SARIF_STDERR:
   diagnostic_output_format_init_sarif_stderr (context,
+ line_table,
  main_input_filename_,
  json_formatting);
   break;
 
 case DIAGNOSTICS_OUTPUT_FORMAT_SARIF_FILE:
   diagnostic_output_format_init_sarif_file (context,
+   line_table,
  

[pushed] diagnostics: SARIF output: tweak ASCII art in comment

[David Malcolm]

Pushed to trunk as r15-2463-gf829e627f40c95.

gcc/ChangeLog:
* diagnostic-format-sarif.cc: Tweak ASCII art in comment
to show edges for both directions in the digraph.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 84b3e651e46..5900b9ea9b7 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -290,17 +290,17 @@ private:
. |  have 'char'"|| | with type 'int'")|
. | in include-chain-1-2.h| | in include-chain-1-1.h   |
. +---+ +--+
-   .   |  |
-   .   | included-by  | included-by
-   .   V  V
+   .^ |^ |
+   .   includes | | included-by   includes | | included-by
+   .| V| V
.  ++++
.  |"id": 1 ||"id": 3 |
.  | #include "include-chain-1-2.h" || #include "include-chain-1-1.h" |
.  | in include-chain-1.h   || in include-chain-1.h   |
.  ++++
-   . |  |
-   . | included-by  | included-by
-   . V  V
+   .   ^ |   ^|
+   .  includes | | included-by  includes || included-by
+   .   | V   |V
.  ++
.  |"id": 4 |
.  | The  #include "include-chain-1.h"  |
-- 
2.26.3

Re: [PATCH] doc: Improve punctuation and grammar in -fdiagnostics-format docs

[David Malcolm]

On Fri, 2024-03-15 at 13:02 +, Jonathan Wakely wrote:
> OK for trunk?

LGTM, thanks

Dave

> 
> -- >8 --
> 
> The hyphen can be misunderstood to mean "emitted to -" i.e. stdout.
> Refer to both forms by name, rather than using "the former" for one
> and
> referring to the other by name.
> 
> gcc/ChangeLog:
> 
> * doc/invoke.texi (Diagnostic Message Formatting Options):
> Replace hyphen with a new sentence. Replace "the former" with
> the actual value.
> ---
>  gcc/doc/invoke.texi | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> index 85c938d4a14..d850b5fcdcc 100644
> --- a/gcc/doc/invoke.texi
> +++ b/gcc/doc/invoke.texi
> @@ -5737,8 +5737,9 @@ named @file{@var{source}.sarif}, respectively.
>  
>  The @samp{json} format is a synonym for @samp{json-stderr}.
>  The @samp{json-stderr} and @samp{json-file} formats are identical,
> apart from
> -where the JSON is emitted to - with the former, the JSON is emitted
> to stderr,
> -whereas with @samp{json-file} it is written to
> @file{@var{source}.gcc.json}.
> +where the JSON is emitted to.  With @samp{json-stderr}, the JSON is
> emitted
> +to stderr, whereas with @samp{json-file} it is written to
> +@file{@var{source}.gcc.json}.
>  
>  The emitted JSON consists of a top-level JSON array containing JSON
> objects
>  representing the diagnostics.

[pushed] diagnostics: SARIF output: capture #include information (PR 107941; §3.34)

[David Malcolm]

sts on x86_64-pc-linux-gnu.
Pushed to trunk as r15-2354-g4d1f71d49e396c.

gcc/ChangeLog:
PR middle-end/107941
* diagnostic-format-sarif.cc: Define INCLUDE_LIST and INCLUDE_MAP.
(enum class location_relationship_kind): New.
(diagnostic_artifact_role::scanned_file): New value.
(class sarif_location_manager): New.
(class sarif_result): Derive from sarif_location_manager rather
than directly from sarif_object.
(sarif_result::add_related_location): Convert to vfunc
implementation.
(sarif_location::m_relationships_map): New field.
(class sarif_location_relationship): New.
(class sarif_ice_notification): Derive from sarif_location_manager
rather than directly from sarif_object.
(sarif_builder::take_current_result): New.
(sarif_builder::m_line_maps): New field.
(sarif_builder::m_cur_group_result): Convert to std::unique_ptr.
(sarif_artifact::add_role): Skip scanned_file.
(get_artifact_role_string): Handle scanned_file.
(sarif_location_manager::add_relationship_to_worklist): New.
(sarif_location_manager::process_worklist): New.
(sarif_location_manager::process_worklist_item): New.
(sarif_result::on_nested_diagnostic): Pass *this to
make_location_object.
(sarif_location::lazily_add_id): New.
(sarif_location::get_id): New.
(get_string_for_location_relationship_kind): New.
(sarif_location::lazily_add_relationship): New.
(sarif_location::lazily_add_relationship_object): New.
(sarif_location::lazily_add_relationships_array): New.
(sarif_ice_notification::sarif_ice_notification): Fix overlong line.
Pass *this to make_locations_arr.
(sarif_ice_notification::add_related_location): New.
(sarif_location_relationship::sarif_location_relationship): New.
(sarif_location_relationship::get_target_id): New.
(sarif_location_relationship::lazily_add_kind): New.
(sarif_builder::sarif_builder): Add "line_maps" param and use it
to initialize m_line_maps.
(sarif_builder::end_diagnostic): Update for m_cur_group_result
becoming a std::unique_ptr.  Don't append to m_results_array yet.
(sarif_builder::end_group): Append m_cur_group_result to
m_results_array here, rather than in end_diagnostic.
(sarif_builder::make_result_object): Pass result_obj to
make_locations_arr and to make_code_flow_object.
(sarif_builder::make_locations_arr): Add "loc_mgr" param and pass
it to make_location_object.
(sarif_builder::make_location_object): For two overloads, add
"loc_mgr" param and call add_any_include_chain on the location.
(sarif_builder::add_any_include_chain): New.
(sarif_builder::make_location_object): New overload.
(sarif_builder::make_code_flow_object): Add "result" param and
pass it to make_thread_flow_location_object.
(sarif_builder::make_thread_flow_location_object): Add "result"
param and pass it to make_location_object.
(sarif_builder::get_or_create_artifact): Handle scanned_file.
(sarif_output_format::~sarif_output_format): Assert that there
isn't a pending result.
(sarif_output_format::sarif_output_format): Add "line_maps" param
and pass it to m_builder's ctor.
(sarif_stream_output_format::sarif_stream_output_format): Add
"line_maps" param and pass it to base class ctor.
(sarif_file_output_format::sarif_file_output_format): Likewise.
(diagnostic_output_format_init_sarif_stderr): Pass "line_table"
global to format.
(diagnostic_output_format_init_sarif_file): Likewise.
(diagnostic_output_format_init_sarif_stream): Likewise.
(test_sarif_diagnostic_context::test_sarif_diagnostic_context):
Likewise.
(buffered_output_format::buffered_output_format): Likewise.
(selftest::test_make_location_object): Likewise.
(selftest::test_make_location_object): Create a sarif_result for
use when calling make_location_object.
* diagnostic.cc (diagnostic_context::finish): End any active
diagnostic groups.
(diagnostic_context::report_diagnostic): Assert that we're within
a diagnostic group.
* diagnostic.h (diagnostic_report_diagnostic): Add
begin_group/end_group pair around call to
diagnostic_context::report_diagnostic.
* selftest-diagnostic.cc (test_diagnostic_context::report): Add
begin_group/end_group pair around diagnostic_impl call.

gcc/testsuite/ChangeLog:
PR middle-end/107941
* gcc.dg/sarif-output/include-chain-1-1.h: New test.
* gcc.dg/sarif-output/include-chain-1-2.h: New test.
* gcc.dg/sarif-ou

[PATCH 10/16] diagnostics: SARIF output: potentially add escaped renderings of source (§3.3.4)

[David Malcolm]

.h: New file, taking class text_range_label from
gcc-rich-location.h.

libcpp/ChangeLog:
* include/rich-location.h
(semi_embedded_vec::semi_embedded_vec): Add copy ctor.
(rich_location::rich_location): Remove "= delete" from decl of
copy ctor.  Add deleted decl of move ctor.
(rich_location::operator=): Remove "= delete" from decl of
    copy assignment.  Add deleted decl of move assignment.
(fixit_hint::fixit_hint): Add copy ctor decl.  Add deleted decl of
move.
(fixit_hint::operator=): Add copy assignment decl.  Add deleted
decl of move assignment.
* line-map.cc (rich_location::rich_location): New copy ctor.
(fixit_hint::fixit_hint): New copy ctor.

Signed-off-by: David Malcolm 
---
 gcc/Makefile.in   |   1 +
 gcc/diagnostic-format-sarif.cc| 229 --
 gcc/diagnostic-show-locus.cc  |  98 
 gcc/gcc-rich-location.h   |  17 --
 gcc/selftest-diagnostic-show-locus.h  |  82 +++
 gcc/selftest-json.cc  | 119 +
 gcc/selftest-json.h   | 100 
 gcc/selftest-run-tests.cc |   1 +
 gcc/selftest.h|   1 +
 ...diagnostic-format-sarif-file-Wbidi-chars.c |   9 +
 .../diagnostic_plugin_test_show_locus.c   |   1 +
 gcc/text-range-label.h|  42 
 libcpp/include/rich-location.h|  31 ++-
 libcpp/line-map.cc|  28 +++
 14 files changed, 669 insertions(+), 90 deletions(-)
 create mode 100644 gcc/selftest-diagnostic-show-locus.h
 create mode 100644 gcc/selftest-json.cc
 create mode 100644 gcc/selftest-json.h
 create mode 100644 gcc/text-range-label.h

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index f4bb4a88cf31..3798caaf60f7 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1836,6 +1836,7 @@ OBJS-libcommon = diagnostic-spec.o diagnostic.o 
diagnostic-color.o \
vec.o input.o hash-table.o ggc-none.o memory-block.o \
selftest.o selftest-diagnostic.o sort.o \
selftest-diagnostic-path.o \
+   selftest-json.o \
selftest-logical-location.o \
text-art/box-drawing.o \
text-art/canvas.o \
diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 6f61d89363f2..847e1eb9bdfc 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -37,9 +37,16 @@ along with GCC; see the file COPYING3.  If not see
 #include "ordered-hash-map.h"
 #include "sbitmap.h"
 #include "make-unique.h"
+#include "selftest.h"
+#include "selftest-diagnostic.h"
+#include "selftest-diagnostic-show-locus.h"
+#include "selftest-json.h"
+#include "text-range-label.h"
 
 /* Forward decls.  */
 class sarif_builder;
+class content_renderer;
+  class escape_nonascii_renderer;
 
 /* Subclasses of sarif_object.
Keep these in order of their descriptions in the specification.  */
@@ -284,6 +291,20 @@ public:
  sarif_builder &builder);
 };
 
+/* Abstract base class for use when making an  "artifactContent"
+   object (SARIF v2.1.0 section 3.3): generate a value for the
+   3.3.4 "rendered" property.
+   Can return nullptr, for "no property".  */
+
+class content_renderer
+{
+public:
+  virtual ~content_renderer () {}
+
+  virtual std::unique_ptr
+  render (const sarif_builder &builder) const = 0;
+};
+
 /* A class for managing SARIF output (for -fdiagnostics-format=sarif-stderr
and -fdiagnostics-format=sarif-file).
 
@@ -312,7 +333,6 @@ public:
  property (SARIF v2.1.0 section 3.14.11), as invocation objects
  (SARIF v2.1.0 section 3.20), but we'd want to capture the arguments to
  toplev::main, and the response files.
-   - doesn't capture escape_on_output_p
- doesn't capture secondary locations within a rich_location
  (perhaps we should use the "relatedLocations" property: SARIF v2.1.0
  section 3.27.22)
@@ -379,7 +399,8 @@ private:
   std::unique_ptr
   maybe_make_physical_location_object (location_t loc,
   enum diagnostic_artifact_role role,
-  int column_override);
+  int column_override,
+  const content_renderer 
*snippet_renderer);
   std::unique_ptr
   make_artifact_location_object (location_t loc);
   std::unique_ptr
@@ -390,7 +411,8 @@ private:
   maybe_make_region_object (location_t loc,
int column_override) const;
   std::unique_ptr
-  maybe_make_region_object_for_context (location_t loc) const;
+  maybe_make_region_object_for_context (location_t loc,
+  

[PATCH 16/16] diagnostics: SARIF output: tweak output for UNKNOWN_LOCATION

[David Malcolm]

gcc/ChangeLog:
* diagnostic-format-sarif.cc (sarif_builder::make_locations_arr):
Don't add entirely empty location objects, such as for
UNKNOWN_LOCATION.
(test_sarif_diagnostic_context::test_sarif_diagnostic_context):
Add param "main_input_filename".
(selftest::test_simple_log): Provide above param.  Verify that
"locations" is empty.
(selftest::test_simple_log_2): New.
(selftest::diagnostic_format_sarif_cc_tests): Call it.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc | 123 ++---
 1 file changed, 115 insertions(+), 8 deletions(-)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 816f3210036e..1fc45c9b4b39 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -1134,8 +1134,12 @@ sarif_builder::make_locations_arr (const diagnostic_info 
&diagnostic,
   if (auto client_data_hooks = m_context.get_client_data_hooks ())
 logical_loc = client_data_hooks->get_current_logical_location ();
 
-  locations_arr->append
-(make_location_object (*diagnostic.richloc, logical_loc, role));
+  auto location_obj
+= make_location_object (*diagnostic.richloc, logical_loc, role);
+  /* Don't add entirely empty location objects to the array.  */
+  if (!location_obj->is_empty ())
+locations_arr->append (std::move (location_obj));
+
   return locations_arr;
 }
 
@@ -2452,12 +2456,12 @@ namespace selftest {
 class test_sarif_diagnostic_context : public test_diagnostic_context
 {
 public:
-  test_sarif_diagnostic_context ()
+  test_sarif_diagnostic_context (const char *main_input_filename)
   {
 diagnostic_output_format_init_sarif (*this);
 
 m_format = new buffered_output_format (*this,
-  "MAIN_INPUT_FILENAME",
+  main_input_filename,
   true);
 set_output_format (m_format); // give ownership;
   }
@@ -2609,14 +2613,14 @@ test_make_location_object (const line_table_case &case_)
   }
 }
 
-/* Test of reporting a diagnostic to a diagnostic_context and
-   examining the generated sarif_log.
+/* Test of reporting a diagnostic at UNKNOWN_LOCATION to a
+   diagnostic_context and examining the generated sarif_log.
Verify various basic properties. */
 
 static void
 test_simple_log ()
 {
-  test_sarif_diagnostic_context dc;
+  test_sarif_diagnostic_context dc ("MAIN_INPUT_FILENAME");
 
   rich_location richloc (line_table, UNKNOWN_LOCATION);
   dc.report (DK_ERROR, richloc, nullptr, 0, "this is a test: %i", 42);
@@ -2719,7 +2723,109 @@ test_simple_log ()
   }
 
   // 3.27.12:
-  EXPECT_JSON_OBJECT_WITH_ARRAY_PROPERTY (result, "locations");
+  auto locations
+   = EXPECT_JSON_OBJECT_WITH_ARRAY_PROPERTY (result, "locations");
+  ASSERT_EQ (locations->size (), 0);
+}
+  }
+}
+
+/* As above, but with a "real" location_t.  */
+
+static void
+test_simple_log_2 (const line_table_case &case_)
+{
+  auto_fix_quotes fix_quotes;
+
+  const char *const content
+/* 011
+   123456789012345.  */
+= "unsinged int i;\n";
+  diagnostic_show_locus_fixture f (case_, content);
+  location_t line_end = linemap_position_for_column (line_table, 31);
+
+  /* Don't attempt to run the tests if column data might be unavailable.  */
+  if (line_end > LINE_MAP_MAX_LOCATION_WITH_COLS)
+return;
+
+  test_sarif_diagnostic_context dc (f.get_filename ());
+
+  const location_t typo_loc
+= make_location (linemap_position_for_column (line_table, 1),
+linemap_position_for_column (line_table, 1),
+linemap_position_for_column (line_table, 8));
+
+  rich_location richloc (line_table, typo_loc);
+  dc.report (DK_ERROR, richloc, nullptr, 0,
+"did you misspell %qs again?",
+"unsigned");
+
+  auto log_ptr = dc.flush_to_object ();
+
+  // 3.13 sarifLog:
+  auto log = log_ptr.get ();
+
+  auto runs = EXPECT_JSON_OBJECT_WITH_ARRAY_PROPERTY (log, "runs"); // 3.13.4
+  ASSERT_EQ (runs->size (), 1);
+
+  // 3.14 "run" object:
+  auto run = (*runs)[0];
+
+  {
+// 3.14.23:
+auto results = EXPECT_JSON_OBJECT_WITH_ARRAY_PROPERTY (run, "results");
+ASSERT_EQ (results->size (), 1);
+
+{
+  // 3.27 "result" object:
+  auto result = (*results)[0];
+  ASSERT_JSON_STRING_PROPERTY_EQ (result, "ruleId", "error");
+  ASSERT_JSON_STRING_PROPERTY_EQ (result, "level", "error"); // 3.27.10
+
+  {
+   // 3.27.11:
+   auto message
+ = EXPECT_JSON_OBJECT_WITH_OBJECT_PROPERTY (result, "message");
+   ASSERT_JSON_STRING_PROPERTY_EQ (message,

[PATCH 14/16] diagnostics: SARIF output: add "annotations" property (§3.28.6)

[David Malcolm]

This patch extends our SARIF output so that if a diagnostic has any
labelled source ranges, the "location" object gains an "annotations"
property capturing them (§3.28.6).

For example, given this textual output:

../../src/gcc/testsuite/gcc.dg/bad-binary-ops.c: In function ‘test_2’:
../../src/gcc/testsuite/gcc.dg/bad-binary-ops.c:31:11: error: invalid operands 
to binary + (have ‘struct s’ and ‘struct t’)
   30 |   return (some_function ()
  |   
  |   |
  |   struct s
   31 |   + some_other_function ());
  |   ^ ~~
  | |
  | struct t

the SARIF output gains this within the result's location[0]:

   "annotations": [{"startLine": 30,
"startColumn": 11,
"endColumn": 27,
"message": {"text": "struct s"}},
   {"startLine": 31,
"startColumn": 13,
"endColumn": 35,
"message": {"text": "struct t"}}]}]},

gcc/ChangeLog:
* diagnostic-format-sarif.cc
(sarif_builder::make_location_object): Add "annotations" property if
there are any labelled ranges (§3.28.6).
(selftest::test_make_location_object): Verify annotations are added
to location_obj.
* json.h (json::array::size): New.
(json::array::operator[]): New.
* selftest-json.cc
(selftest::expect_json_object_with_array_property): New.
* selftest-json.h
(selftest::expect_json_object_with_array_property): New decl.
(EXPECT_JSON_OBJECT_WITH_ARRAY_PROPERTY): New macro.

gcc/testsuite/ChangeLog:
* c-c++-common/diagnostic-format-sarif-file-Wbidi-chars.c: Verify
that we have an "annotations" property for the labelled
ranges (§3.28.6).

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc| 70 ++-
 gcc/json.h|  3 +
 gcc/selftest-json.cc  | 16 +
 gcc/selftest-json.h   | 14 
 ...diagnostic-format-sarif-file-Wbidi-chars.c |  8 +++
 5 files changed, 110 insertions(+), 1 deletion(-)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 775d01f75744..afb29eab5839 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -345,6 +345,7 @@ public:
- CWE metadata
- diagnostic groups (see limitations below)
- logical locations (e.g. cfun)
+   - labelled ranges (as annotations)
 
Known limitations:
- GCC supports one-deep nesting of diagnostics (via auto_diagnostic_group),
@@ -361,7 +362,6 @@ public:
  ("artifact.hashes" property (SARIF v2.1.0 section 3.24.11).
- doesn't capture the "analysisTarget" property
  (SARIF v2.1.0 section 3.27.13).
-   - doesn't capture labelled ranges
- doesn't capture -Werror cleanly
- doesn't capture inlining information (can SARIF handle this?)
- doesn't capture macro expansion information (can SARIF handle this?).  */
@@ -1210,6 +1210,38 @@ sarif_builder::make_location_object (const rich_location 
&rich_loc,
   /* "logicalLocations" property (SARIF v2.1.0 section 3.28.4).  */
   set_any_logical_locs_arr (*location_obj, logical_loc);
 
+  /* "annotations" property (SARIF v2.1.0 section 3.28.6).  */
+  {
+/* Create annotations for any labelled ranges.  */
+std::unique_ptr annotations_arr = nullptr;
+for (unsigned int i = 0; i < rich_loc.get_num_locations (); i++)
+  {
+   const location_range *range = rich_loc.get_range (i);
+   if (const range_label *label = range->m_label)
+ {
+   label_text text = label->get_text (i);
+   if (text.get ())
+ {
+   location_t range_loc = rich_loc.get_loc (i);
+   auto region
+ = maybe_make_region_object (range_loc,
+ rich_loc.get_column_override ());
+   if (region)
+ {
+   if (!annotations_arr)
+ annotations_arr = ::make_unique ();
+   region->set
+ ("message", make_message_object (text.get ()));
+   annotations_arr->append (std::move (region));
+ }
+ }
+ }
+  }
+if (annotations_arr)
+  location_obj->set ("annotations",
+ std::move (annotations_arr));
+  }
+
   /* A flag for hinting that the diagnostic involves issues at the
  level of character encodings (such as homoglyphs, or misleading
  bidir

[PATCH 12/16] diagnostics: SARIF output: add "arguments" property (§3.20.2)

[David Malcolm]

gcc/ChangeLog:
* diagnostic-format-sarif.cc (sarif_invocation::sarif_invocation):
Add "original_argv" param and use it to populate "arguments"
property (§3.20.2).
(sarif_builder::sarif_builder): Pass argv to m_invocation_obj's
ctor.
* diagnostic.cc (diagnostic_context::initialize): Initialize
m_original_argv.
(diagnostic_context::finish): Clean up m_original_argv.
(diagnostic_context::set_original_argv): New.
* diagnostic.h: Include "unique-argv.h".
(diagnostic_context::set_original_argv): New decl.
(diagnostic_context::get_original_argv): New decl.
(diagnostic_context::m_original_argv): New field.
* toplev.cc: Include "unique-argv.h".
(general_init): Add "original_argv" param and move it to global_dc.
(toplev::main): Stash a copy of the original argv before expansion,
and pass it to general_init for use by SARIF output.
* unique-argv.h: New file.

gcc/jit/ChangeLog:
* jit-playback.cc (jit::playback_context::compile) Add a trailing
null to argvec.

gcc/testsuite/ChangeLog:
* c-c++-common/diagnostic-format-sarif-file-1.c: Verify that we
have an "arguments" property (§3.20.2).

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc| 25 +--
 gcc/diagnostic.cc | 17 +
 gcc/diagnostic.h  | 10 +++
 gcc/jit/jit-playback.cc   |  6 +-
 .../diagnostic-format-sarif-file-1.c  |  5 ++
 gcc/toplev.cc | 13 +++-
 gcc/unique-argv.h | 67 +++
 7 files changed, 132 insertions(+), 11 deletions(-)
 create mode 100644 gcc/unique-argv.h

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 9be84fb268a5..6c7216651627 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -125,7 +125,8 @@ class sarif_tool_component : public sarif_object {};
 class sarif_invocation : public sarif_object
 {
 public:
-  sarif_invocation (sarif_builder &builder);
+  sarif_invocation (sarif_builder &builder,
+   const char * const *original_argv);
 
   void add_notification_for_ice (diagnostic_context &context,
 const diagnostic_info &diagnostic,
@@ -329,10 +330,8 @@ public:
- GCC supports one-deep nesting of diagnostics (via auto_diagnostic_group),
  but we only capture location and message information from such nested
  diagnostics (e.g. we ignore fix-it hints on them)
-   - doesn't yet capture command-line arguments: would be run.invocations
- property (SARIF v2.1.0 section 3.14.11), as invocation objects
- (SARIF v2.1.0 section 3.20), but we'd want to capture the arguments to
- toplev::main, and the response files.
+   - although we capture command-line arguments (section 3.20.2), we don't
+ yet capture response files.
- doesn't capture secondary locations within a rich_location
  (perhaps we should use the "relatedLocations" property: SARIF v2.1.0
  section 3.27.22)
@@ -513,10 +512,20 @@ sarif_object::get_or_create_properties ()
 
 /* class sarif_invocation : public sarif_object.  */
 
-sarif_invocation::sarif_invocation (sarif_builder &builder)
+sarif_invocation::sarif_invocation (sarif_builder &builder,
+   const char * const *original_argv)
 : m_notifications_arr (::make_unique ()),
   m_success (true)
 {
+  // "arguments" property (SARIF v2.1.0 section 3.20.2)
+  if (original_argv)
+{
+  auto arguments_arr = ::make_unique ();
+  for (size_t i = 0; original_argv[i]; ++i)
+   arguments_arr->append_string (original_argv[i]);
+  set ("arguments", std::move (arguments_arr));
+}
+
   // "workingDirectory" property (SARIF v2.1.0 section 3.20.19)
   if (const char *pwd = getpwd ())
 set ("workingDirectory",
@@ -752,7 +761,9 @@ sarif_builder::sarif_builder (diagnostic_context &context,
  const char *main_input_filename_,
  bool formatted)
 : m_context (context),
-  m_invocation_obj (::make_unique (*this)),
+  m_invocation_obj
+(::make_unique (*this,
+ context.get_original_argv ())),
   m_results_array (new json::array ()),
   m_cur_group_result (nullptr),
   m_seen_any_relative_paths (false),
diff --git a/gcc/diagnostic.cc b/gcc/diagnostic.cc
index aa8afd521fa2..c70c394f7ccd 100644
--- a/gcc/diagnostic.cc
+++ b/gcc/diagnostic.cc
@@ -261,6 +261,7 @@ diagnostic_context::initialize (int n_opts)
   m_includes_seen = nullptr;
   m_client_data_hooks = nullptr;
   m_diagrams.m_theme = nullptr;
+  m_original_argv = nullptr;
 
  

[PATCH 11/16] diagnostics: SARIF output: add "workingDirectory" property (§3.20.19)

[David Malcolm]

gcc/ChangeLog:
* diagnostic-format-sarif.cc
(sarif_builder::make_artifact_location_object): Make public.
(sarif_invocation::sarif_invocation): Add param "builder".
Use it to potentially populate the "workingDirectory" property
with the result of pwd (§3.20.19).
(sarif_builder::sarif_builder): Pass *this to m_invocation_obj's
ctor.

gcc/testsuite/ChangeLog:
* c-c++-common/diagnostic-format-sarif-file-1.c: Verify that we have
a "workingDirectory" property.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc| 15 ++-
 .../c-c++-common/diagnostic-format-sarif-file-1.c |  1 +
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 847e1eb9bdfc..9be84fb268a5 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -125,7 +125,7 @@ class sarif_tool_component : public sarif_object {};
 class sarif_invocation : public sarif_object
 {
 public:
-  sarif_invocation ();
+  sarif_invocation (sarif_builder &builder);
 
   void add_notification_for_ice (diagnostic_context &context,
 const diagnostic_info &diagnostic,
@@ -378,6 +378,9 @@ public:
   std::unique_ptr
   maybe_make_artifact_content_object (const char *filename) const;
 
+  std::unique_ptr
+  make_artifact_location_object (const char *filename);
+
 private:
   std::unique_ptr
   make_result_object (diagnostic_context &context,
@@ -404,8 +407,6 @@ private:
   std::unique_ptr
   make_artifact_location_object (location_t loc);
   std::unique_ptr
-  make_artifact_location_object (const char *filename);
-  std::unique_ptr
   make_artifact_location_object_for_pwd () const;
   std::unique_ptr
   maybe_make_region_object (location_t loc,
@@ -512,10 +513,14 @@ sarif_object::get_or_create_properties ()
 
 /* class sarif_invocation : public sarif_object.  */
 
-sarif_invocation::sarif_invocation ()
+sarif_invocation::sarif_invocation (sarif_builder &builder)
 : m_notifications_arr (::make_unique ()),
   m_success (true)
 {
+  // "workingDirectory" property (SARIF v2.1.0 section 3.20.19)
+  if (const char *pwd = getpwd ())
+set ("workingDirectory",
+ builder.make_artifact_location_object (pwd));
 }
 
 /* Handle an internal compiler error DIAGNOSTIC occurring on CONTEXT.
@@ -747,7 +752,7 @@ sarif_builder::sarif_builder (diagnostic_context &context,
  const char *main_input_filename_,
  bool formatted)
 : m_context (context),
-  m_invocation_obj (::make_unique ()),
+  m_invocation_obj (::make_unique (*this)),
   m_results_array (new json::array ()),
   m_cur_group_result (nullptr),
   m_seen_any_relative_paths (false),
diff --git a/gcc/testsuite/c-c++-common/diagnostic-format-sarif-file-1.c 
b/gcc/testsuite/c-c++-common/diagnostic-format-sarif-file-1.c
index 50375465483d..0a3778323792 100644
--- a/gcc/testsuite/c-c++-common/diagnostic-format-sarif-file-1.c
+++ b/gcc/testsuite/c-c++-common/diagnostic-format-sarif-file-1.c
@@ -32,6 +32,7 @@
  { dg-final { scan-sarif-file "\"informationUri\": \"" } }
 
  { dg-final { scan-sarif-file "\"invocations\": \\\[" } }
+   { dg-final { scan-sarif-file {"workingDirectory": } } }
{ dg-final { scan-sarif-file "\"toolExecutionNotifications\": \\\[\\\]" 
} }
{ dg-final { scan-sarif-file "\"executionSuccessful\": true" } }
 
-- 
2.26.3

[PATCH 13/16] diagnostics: SARIF output: add "{start, end}TimeUtc" properties (§§3.20.7-8)

[David Malcolm]

gcc/ChangeLog:
* diagnostic-format-sarif.cc
(make_date_time_string_for_current_time): New.
(sarif_invocation::sarif_invocation): Set "startTimeUtc"
property (§3.20.7).
(sarif_invocation::prepare_to_flush): Set "endTimeUtc"
property (§3.20.8).

gcc/testsuite/ChangeLog:
* c-c++-common/diagnostic-format-sarif-file-1.c: Verify that we have
"startTimeUtc" and "endTimeUtc" properties of the correct form.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc| 28 +++
 .../diagnostic-format-sarif-file-1.c  |  5 
 2 files changed, 33 insertions(+)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 6c7216651627..775d01f75744 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -119,6 +119,26 @@ class sarif_tool : public sarif_object {};
 
 class sarif_tool_component : public sarif_object {};
 
+/* Make a JSON string for the current date and time.
+   See SARIF v2.1.0 section 3.9 "Date/time properties".
+   Given that we don't run at the very beginning/end of the
+   process, it doesn't make sense to be more accurate than
+   the current second.  */
+
+static std::unique_ptr
+make_date_time_string_for_current_time ()
+{
+  time_t t = time (nullptr);
+  struct tm *tm = gmtime (&t);
+  char buf[256];
+  snprintf (buf, sizeof (buf) - 1,
+   ("%04i-%02i-%02iT"
+"%02i:%02i:%02iZ"),
+   tm->tm_year + 1900, tm->tm_mon + 1, tm->tm_mday,
+   tm->tm_hour, tm->tm_min, tm->tm_sec);
+  return ::make_unique (buf);
+}
+
 /* Subclass of sarif_object for SARIF "invocation" objects
(SARIF v2.1.0 section 3.20).  */
 
@@ -530,6 +550,10 @@ sarif_invocation::sarif_invocation (sarif_builder &builder,
   if (const char *pwd = getpwd ())
 set ("workingDirectory",
  builder.make_artifact_location_object (pwd));
+
+  // "startTimeUtc" property (SARIF v2.1.0 section 3.20.7)
+  set ("startTimeUtc",
+make_date_time_string_for_current_time ());
 }
 
 /* Handle an internal compiler error DIAGNOSTIC occurring on CONTEXT.
@@ -559,6 +583,10 @@ sarif_invocation::prepare_to_flush (diagnostic_context 
&context)
  this object (SARIF v2.1.0 section 3.8) e.g. for recording time vars.  */
   if (auto client_data_hooks = context.get_client_data_hooks ())
 client_data_hooks->add_sarif_invocation_properties (*this);
+
+  // "endTimeUtc" property (SARIF v2.1.0 section 3.20.8);
+  set ("endTimeUtc",
+make_date_time_string_for_current_time ());
 }
 
 /* class sarif_artifact : public sarif_object.  */
diff --git a/gcc/testsuite/c-c++-common/diagnostic-format-sarif-file-1.c 
b/gcc/testsuite/c-c++-common/diagnostic-format-sarif-file-1.c
index c9ad0d238195..fdf602eaae7b 100644
--- a/gcc/testsuite/c-c++-common/diagnostic-format-sarif-file-1.c
+++ b/gcc/testsuite/c-c++-common/diagnostic-format-sarif-file-1.c
@@ -37,6 +37,11 @@
3.20.2 invocation "arguments" property:
{ dg-final { scan-sarif-file {"arguments": \[} } }
 
+   Expect "startTimeUtc" and "endTimeUtc" properties of the form
+   "-nn-nnTnn:nn:nnZ" (3.20.7 and 3.20.8):
+   { dg-final { scan-sarif-file {"startTimeUtc": 
"[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9][0-9]:[0-9][0-9]:[0-9][0-9]Z"} 
} }
+   { dg-final { scan-sarif-file {"endTimeUtc": 
"[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9][0-9]:[0-9][0-9]:[0-9][0-9]Z"} 
} }
+
{ dg-final { scan-sarif-file {"workingDirectory": } } }
{ dg-final { scan-sarif-file "\"toolExecutionNotifications\": \\\[\\\]" 
} }
{ dg-final { scan-sarif-file "\"executionSuccessful\": true" } }
-- 
2.26.3

[PATCH 09/16] diagnostics: JSON output: use std::unique_ptr throughout

[David Malcolm]

No functional change intended.

gcc/ChangeLog:
* diagnostic-format-json.cc: Include "make-unique.h".
(json_output_format::m_toplevel_array): Convert to
std::unique_ptr.
(json_output_format::json_output_format): Update accordingly.
(json_output_format::~json_output_format): Remove manual
"delete" of field.
(json_from_expanded_location): Convert return type to
std::unique_ptr.
(json_from_location_range): Likewise.  Use nullptr rather than
NULL.
(json_from_fixit_hint): Convert return type to std::unique_ptr.
(json_from_metadata): Likewise.
(make_json_for_path): Likewise.
(json_output_format::on_end_diagnostic): Use std::unique_ptr
throughout.
(json_file_output_format::~json_file_output_format): Use nullptr.
(selftest::test_unknown_location): Update to use std::unique_ptr.
(selftest::test_bad_endpoints): Likewise.  Replace NULL with
nullptr.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-json.cc | 122 +-
 1 file changed, 62 insertions(+), 60 deletions(-)

diff --git a/gcc/diagnostic-format-json.cc b/gcc/diagnostic-format-json.cc
index 55ba39e0c532..b78cb92cfd7a 100644
--- a/gcc/diagnostic-format-json.cc
+++ b/gcc/diagnostic-format-json.cc
@@ -30,6 +30,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "json.h"
 #include "selftest.h"
 #include "logical-location.h"
+#include "make-unique.h"
 
 /* Subclass of diagnostic_output_format for JSON output.  */
 
@@ -62,7 +63,7 @@ protected:
   json_output_format (diagnostic_context &context,
  bool formatted)
   : diagnostic_output_format (context),
-m_toplevel_array (new json::array ()),
+m_toplevel_array (::make_unique ()),
 m_cur_group (nullptr),
 m_cur_children_array (nullptr),
 m_formatted (formatted)
@@ -75,31 +76,30 @@ protected:
   {
 m_toplevel_array->dump (outf, m_formatted);
 fprintf (outf, "\n");
-delete m_toplevel_array;
 m_toplevel_array = nullptr;
   }
 
 private:
   /* The top-level JSON array of pending diagnostics.  */
-  json::array *m_toplevel_array;
+  std::unique_ptr m_toplevel_array;
 
   /* The JSON object for the current diagnostic group.  */
-  json::object *m_cur_group;
+  json::object *m_cur_group; // borrowed
 
   /* The JSON array for the "children" array within the current diagnostic
  group.  */
-  json::array *m_cur_children_array;
+  json::array *m_cur_children_array; // borrowed
 
   bool m_formatted;
 };
 
 /* Generate a JSON object for LOC.  */
 
-static json::value *
+static std::unique_ptr
 json_from_expanded_location (diagnostic_context &context, location_t loc)
 {
   expanded_location exploc = expand_location (loc);
-  json::object *result = new json::object ();
+  std::unique_ptr result = ::make_unique  ();
   if (exploc.file)
 result->set_string ("file", exploc.file);
   result->set_integer ("line", exploc.line);
@@ -130,26 +130,29 @@ json_from_expanded_location (diagnostic_context &context, 
location_t loc)
 
 /* Generate a JSON object for LOC_RANGE.  */
 
-static json::object *
+static std::unique_ptr
 json_from_location_range (diagnostic_context &context,
  const location_range *loc_range, unsigned range_idx)
 {
   location_t caret_loc = get_pure_location (loc_range->m_loc);
 
   if (caret_loc == UNKNOWN_LOCATION)
-return NULL;
+return nullptr;
 
   location_t start_loc = get_start (loc_range->m_loc);
   location_t finish_loc = get_finish (loc_range->m_loc);
 
-  json::object *result = new json::object ();
-  result->set ("caret", json_from_expanded_location (context, caret_loc));
+  std::unique_ptr result = ::make_unique  ();
+  result->set ("caret",
+  json_from_expanded_location (context, caret_loc));
   if (start_loc != caret_loc
   && start_loc != UNKNOWN_LOCATION)
-result->set ("start", json_from_expanded_location (context, start_loc));
+result->set ("start",
+json_from_expanded_location (context, start_loc));
   if (finish_loc != caret_loc
   && finish_loc != UNKNOWN_LOCATION)
-result->set ("finish", json_from_expanded_location (context, finish_loc));
+result->set ("finish",
+json_from_expanded_location (context, finish_loc));
 
   if (loc_range->m_label)
 {
@@ -163,15 +166,17 @@ json_from_location_range (diagnostic_context &context,
 
 /* Generate a JSON object for HINT.  */
 
-static json::object *
+static std::unique_ptr
 json_from_fixit_hint (diagnostic_context &context, const fixit_hint *hint)
 {
-  json::object *fixit_obj = new json::object ();
+  std::unique_ptr fixit_obj = ::make_unique  ();
 
   locati

[PATCH 08/16] diagnostics: SARIF output: use std::unique_ptr throughout

[David Malcolm]

No functional change intended.

gcc/analyzer/ChangeLog:
* checker-event.cc (maybe_add_sarif_properties): Update setting
of "original_fndecl" to use typesafe unique_ptr variant of
json::object::set.

gcc/ChangeLog:
* diagnostic-format-sarif.cc: Include "make-unique.h".  Convert
raw pointers to std::unique_ptr throughout to indicate ownership,
adding comments in the few places where pointers are borrowed.
Use typesafe unique_ptr variants of json::object::set and
json::array::append throughout to make types of properties more
explicit, whilst using "auto" to reduce typing.
Use "nullptr" rather than "NULL" throughout.
* diagnostic-format-sarif.h (make_sarif_logical_location_object):
Use std::unique_ptr for return type.

Signed-off-by: David Malcolm 
---
 gcc/analyzer/checker-event.cc  |   5 +-
 gcc/diagnostic-format-sarif.cc | 667 -
 gcc/diagnostic-format-sarif.h  |   2 +-
 3 files changed, 337 insertions(+), 337 deletions(-)

diff --git a/gcc/analyzer/checker-event.cc b/gcc/analyzer/checker-event.cc
index 593f364e1d66..2f1438c983a5 100644
--- a/gcc/analyzer/checker-event.cc
+++ b/gcc/analyzer/checker-event.cc
@@ -161,8 +161,9 @@ maybe_add_sarif_properties (sarif_object 
&thread_flow_loc_obj) const
   if (m_original_fndecl != m_effective_fndecl)
 {
   tree_logical_location logical_loc (m_original_fndecl);
-  props.set (PROPERTY_PREFIX "original_fndecl",
-make_sarif_logical_location_object (logical_loc));
+  props.set
+   (PROPERTY_PREFIX "original_fndecl",
+make_sarif_logical_location_object (logical_loc));
 }
   if (m_original_depth != m_effective_depth)
 props.set_integer (PROPERTY_PREFIX "original_depth", m_original_depth);
diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 6aba81c6ac9b..6f61d89363f2 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -36,6 +36,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "diagnostic-format-sarif.h"
 #include "ordered-hash-map.h"
 #include "sbitmap.h"
+#include "make-unique.h"
 
 /* Forward decls.  */
 class sarif_builder;
@@ -125,7 +126,7 @@ public:
   void prepare_to_flush (diagnostic_context &context);
 
 private:
-  json::array *m_notifications_arr;
+  std::unique_ptr m_notifications_arr;
   bool m_success;
 };
 
@@ -180,7 +181,7 @@ private:
 class sarif_result : public sarif_object
 {
 public:
-  sarif_result () : m_related_locations_arr (NULL) {}
+  sarif_result () : m_related_locations_arr (nullptr) {}
 
   void
   on_nested_diagnostic (diagnostic_context &context,
@@ -193,9 +194,9 @@ public:
 
 private:
   void
-  add_related_location (sarif_location *location_obj);
+  add_related_location (std::unique_ptr location_obj);
 
-  json::array *m_related_locations_arr;
+  json::array *m_related_locations_arr; // borrowed
 };
 
 /* Subclass of sarif_object for SARIF "location" objects
@@ -226,10 +227,12 @@ class sarif_thread_flow : public sarif_object
 public:
   sarif_thread_flow (const diagnostic_thread &thread);
 
-  void add_location (sarif_thread_flow_location *thread_flow_loc_obj);
+  void
+  add_location
+(std::unique_ptr thread_flow_loc_obj);
 
 private:
-  json::array *m_locations_arr;
+  json::array *m_locations_arr; // borrowed
 };
 
 /* Subclass of sarif_object for SARIF "threadFlowLocation" objects
@@ -340,80 +343,81 @@ public:
 
   void flush_to_file (FILE *outf);
 
-  json::array *make_locations_arr (const diagnostic_info &diagnostic,
-  enum diagnostic_artifact_role role);
-  sarif_location *
+  std::unique_ptr
+  make_locations_arr (const diagnostic_info &diagnostic,
+ enum diagnostic_artifact_role role);
+  std::unique_ptr
   make_location_object (const rich_location &rich_loc,
const logical_location *logical_loc,
enum diagnostic_artifact_role role);
-  sarif_message *
+  std::unique_ptr
   make_message_object (const char *msg) const;
-  sarif_message *
+  std::unique_ptr
   make_message_object_for_diagram (diagnostic_context &context,
   const diagnostic_diagram &diagram);
-  sarif_artifact_content *
+  std::unique_ptr
   maybe_make_artifact_content_object (const char *filename) const;
 
 private:
-  sarif_result *
+  std::unique_ptr
   make_result_object (diagnostic_context &context,
  const diagnostic_info &diagnostic,
  diagnostic_t orig_diag_kind);
   void
-  set_any_logical_locs_arr (sarif_location *location_obj,
+  set_any_logical_locs_arr (sarif_location &location_obj,
const logical_location *logical_l

[PATCH 15/16] diagnostics: add selftests for SARIF output

[David Malcolm]

The existing DejaGnu-based tests for our SARIF output used regexes
to verify the JSON at the string level, which lets us test for
the presence of properties, but doesn't check the overall structure.

This patch uses the selftest framework to verify the structure of
the tree of JSON values for a log containing one diagnostic.

No functional change intended.

gcc/ChangeLog:
* diagnostic-format-sarif.cc (sarif_builder::flush_to_object):
New, using code moved from...
(sarif_builder::end_group): ...here.
(class selftest::test_sarif_diagnostic_context): New.
(selftest::test_simple_log): New.
(selftest::diagnostic_format_sarif_cc_tests): Call it.
* json.h (json::object::is_empty): New.
* selftest-diagnostic.cc (test_diagnostic_context::report): New.
* selftest-diagnostic.h (test_diagnostic_context::report): New
decl.
* selftest-json.cc (selftest::assert_json_string_eq): New.
(selftest::expect_json_object_with_string_property): New.
(selftest::assert_json_string_property_eq): New.
* selftest-json.h (selftest::assert_json_string_eq): New decl.
(ASSERT_JSON_STRING_EQ): New macro.
(selftest::expect_json_object_with_string_property): New decl.
(EXPECT_JSON_OBJECT_WITH_STRING_PROPERTY): New macro.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc | 185 -
 gcc/json.h |   2 +
 gcc/selftest-diagnostic.cc |  14 +++
 gcc/selftest-diagnostic.h  |  10 ++
 gcc/selftest-json.cc   |  34 +-
 gcc/selftest-json.h|  27 +
 6 files changed, 264 insertions(+), 8 deletions(-)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index afb29eab5839..816f3210036e 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -380,6 +380,7 @@ public:
 const diagnostic_diagram &diagram);
   void end_group ();
 
+  std::unique_ptr flush_to_object ();
   void flush_to_file (FILE *outf);
 
   std::unique_ptr
@@ -860,6 +861,20 @@ sarif_builder::end_group ()
   m_cur_group_result = nullptr;
 }
 
+/* Create a top-level object, and add it to all the results
+   (and other entities) we've seen so far, moving ownership
+   to the object.  */
+
+std::unique_ptr
+sarif_builder::flush_to_object ()
+{
+  m_invocation_obj->prepare_to_flush (m_context);
+  std::unique_ptr top
+= make_top_level_object (std::move (m_invocation_obj),
+std::move (m_results_array));
+  return top;
+}
+
 /* Create a top-level object, and add it to all the results
(and other entities) we've seen so far.
 
@@ -868,12 +883,8 @@ sarif_builder::end_group ()
 void
 sarif_builder::flush_to_file (FILE *outf)
 {
-  m_invocation_obj->prepare_to_flush (m_context);
-  std::unique_ptr top
-= make_top_level_object (std::move (m_invocation_obj),
-std::move (m_results_array));
+  std::unique_ptr top = flush_to_object ();
   top->dump (outf, m_formatted);
-  m_invocation_obj = nullptr;
   fprintf (outf, "\n");
 }
 
@@ -2434,6 +2445,54 @@ diagnostic_output_format_init_sarif_stream 
(diagnostic_context &context,
 
 namespace selftest {
 
+/* A subclass of sarif_output_format for writing selftests.
+   The JSON output is cached internally, rather than written
+   out to a file.  */
+
+class test_sarif_diagnostic_context : public test_diagnostic_context
+{
+public:
+  test_sarif_diagnostic_context ()
+  {
+diagnostic_output_format_init_sarif (*this);
+
+m_format = new buffered_output_format (*this,
+  "MAIN_INPUT_FILENAME",
+  true);
+set_output_format (m_format); // give ownership;
+  }
+
+  std::unique_ptr flush_to_object ()
+  {
+return m_format->flush_to_object ();
+  }
+
+private:
+  class buffered_output_format : public sarif_output_format
+  {
+  public:
+buffered_output_format (diagnostic_context &context,
+   const char *main_input_filename_,
+   bool formatted)
+  : sarif_output_format (context, main_input_filename_, formatted)
+{
+}
+bool machine_readable_stderr_p () const final override
+{
+  return false;
+}
+std::unique_ptr flush_to_object ()
+{
+  return m_builder.flush_to_object ();
+}
+  };
+
+  buffered_output_format *m_format; // borrowed
+};
+
+/* Test making a sarif_location for a complex rich_location
+   with labels and escape-on-output.  */
+
 static void
 test_make_location_object (const line_table_case &case_)
 {
@@ -2550,12 +2609,128 @@ test_make_location_object (const line_table_case 
&case_)
   }
 }
 
+/* Test of reporting a diagnostic to a diagnostic_context and
+   examining the generated sarif_log.
+   Verify various basic properties. */
+
+sta

[PATCH 06/16] diagnostics: output formats: use references for non-null pointers

[David Malcolm]

No functional change intended.

gcc/ChangeLog:
* diagnostic-format-json.cc (json_from_expanded_location): Make
"static". Pass param "context" by reference, as it cannot be null.
(json_from_location_range): Likewise for param "context".
(json_from_fixit_hint): Likewise.
(make_json_for_path): Likewise.
(json_output_format::on_end_diagnostic): Update for above changes.

(diagnostic_output_format_init_json::diagnostic_output_format_init_json):
Pass param "context" by reference, as it cannot be null.
(diagnostic_output_format_init_json_stderr): Likewise.
(diagnostic_output_format_init_json_file): Likewise.
(selftest::test_unknown_location): Update for above changes.
(selftest::test_bad_endpoints): Likewise.
* diagnostic-format-sarif.cc (sarif_builder::m_context): Convert
from pointer to reference.
(sarif_invocation::add_notification_for_ice): Convert both params
from pointers to references.
(sarif_invocation::prepare_to_flush): Likewise for "context".
(sarif_result::on_nested_diagnostic): Likewise for "context" and
"builder".
(sarif_result::on_diagram): Likewise.
(sarif_ice_notification::sarif_ice_notification): Likewise.
(sarif_builder::sarif_builder): Likewise for "context".
(sarif_builder::end_diagnostic): Likewise.
(sarif_builder::emit_diagram): Likewise.
(sarif_builder::make_result_object): Likewise.
(make_reporting_descriptor_object_for_warning): Likewise.
(sarif_builder::make_locations_arr): Update for change to m_context.
(sarif_builder::get_sarif_column): Likewise.
(sarif_builder::make_message_object_for_diagram): Convert "context"
from pointer to reference.
(sarif_builder::make_tool_object): Likewise for "m_context".
(sarif_builder::make_driver_tool_component_object): Likewise.
(sarif_builder::get_or_create_artifact): Likewise.
(sarif_builder::maybe_make_artifact_content_object): Likewise.
(sarif_builder::get_source_lines): Likewise.
(sarif_output_format::on_end_diagnostic): Update for above changes.
(sarif_output_format::on_diagram): Likewise.
(sarif_output_format::sarif_output_format): Likewise.
(diagnostic_output_format_init_sarif): Convert param "context"
from pointer to reference.
(diagnostic_output_format_init_sarif_stderr): Likewise.
(diagnostic_output_format_init_sarif_file): Likewise.
(diagnostic_output_format_init_sarif_stream): Likewise.
* diagnostic.cc (diagnostic_output_format_init): Likewise.
* diagnostic.h (diagnostic_output_format_init): Likewise.
(diagnostic_output_format_init_json_stderr): Likewise.
(diagnostic_output_format_init_json_file): Likewise.
(diagnostic_output_format_init_sarif_stderr): Likewise.
(diagnostic_output_format_init_sarif_file): Likewise.
(diagnostic_output_format_init_sarif_stream): Likewise.
(json_from_expanded_location): Delete decl.
* gcc.cc (driver_handle_option): Update for change to
    diagnostic_output_format_init.
* opts.cc (common_handle_option): Likewise.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-json.cc  |  56 +--
 gcc/diagnostic-format-sarif.cc | 166 -
 gcc/diagnostic.cc  |   2 +-
 gcc/diagnostic.h   |  15 ++-
 gcc/gcc.cc |   3 +-
 gcc/opts.cc|   3 +-
 6 files changed, 122 insertions(+), 123 deletions(-)

diff --git a/gcc/diagnostic-format-json.cc b/gcc/diagnostic-format-json.cc
index 4dc0f264fc70..1bf8da663cc2 100644
--- a/gcc/diagnostic-format-json.cc
+++ b/gcc/diagnostic-format-json.cc
@@ -94,8 +94,8 @@ private:
 
 /* Generate a JSON object for LOC.  */
 
-json::value *
-json_from_expanded_location (diagnostic_context *context, location_t loc)
+static json::value *
+json_from_expanded_location (diagnostic_context &context, location_t loc)
 {
   expanded_location exploc = expand_location (loc);
   json::object *result = new json::object ();
@@ -103,7 +103,7 @@ json_from_expanded_location (diagnostic_context *context, 
location_t loc)
 result->set_string ("file", exploc.file);
   result->set_integer ("line", exploc.line);
 
-  const enum diagnostics_column_unit orig_unit = context->m_column_unit;
+  const enum diagnostics_column_unit orig_unit = context.m_column_unit;
   struct
   {
 const char *name;
@@ -115,22 +115,22 @@ json_from_expanded_location (diagnostic_context *context, 
location_t loc)
   int the_column = INT_MIN;
   for (int i = 0; i != ARRAY_SIZE (column_fields); ++i)
 {
-  context->m_column_unit = column_fields[i].unit;
-  const i

[PATCH 04/16] gcov: reduce use of naked "new" for json output

[David Malcolm]

No functional change intended.

gcc/ChangeLog:
* gcov.cc (output_intermediate_json_line): Use
json::object::set_integer to avoid naked "new".

Signed-off-by: David Malcolm 
---
 gcc/gcov.cc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gcc/gcov.cc b/gcc/gcov.cc
index e76a314041cd..85fdac4368e8 100644
--- a/gcc/gcov.cc
+++ b/gcc/gcov.cc
@@ -1285,8 +1285,8 @@ output_intermediate_json_line (json::array *object,
const int covered = info.popcount ();
 
json::object *cond = new json::object ();
-   cond->set ("count", new json::integer_number (count));
-   cond->set ("covered", new json::integer_number (covered));
+   cond->set_integer ("count", count);
+   cond->set_integer ("covered", covered);
 
json::array *mtrue = new json::array ();
json::array *mfalse = new json::array ();
-- 
2.26.3

[PATCH 03/16] analyzer: reduce use of naked "new" for json dumps

[David Malcolm]

No functional change intended.

gcc/analyzer/ChangeLog:
* call-string.cc (call_string::to_json): Avoid naked "new".
* constraint-manager.cc (bounded_range::set_json_attr): Likewise.
(equiv_class::to_json): Likewise.
(constraint::to_json): Likewise.
(bounded_ranges_constraint::to_json): Likewise.
* diagnostic-manager.cc (saved_diagnostic::to_json): Likewise.
(saved_diagnostic::maybe_add_sarif_properties): Likewise.
* engine.cc (exploded_node::to_json): Likewise.
(exploded_edge::to_json): Likewise.
* program-point.cc (program_point::to_json): Likewise.
* program-state.cc (program_state::to_json): Likewise.
* sm.cc (state_machine::to_json): Likewise.
* store.cc (binding_cluster::to_json): Likewise.
(store::to_json): Likewise.
* supergraph.cc (supernode::to_json): Likewise.
(superedge::to_json): Likewise.

Signed-off-by: David Malcolm 
---
 gcc/analyzer/call-string.cc|  9 +++--
 gcc/analyzer/constraint-manager.cc | 12 ++--
 gcc/analyzer/diagnostic-manager.cc | 14 +++---
 gcc/analyzer/engine.cc | 13 ++---
 gcc/analyzer/program-point.cc  | 11 ---
 gcc/analyzer/program-state.cc  |  2 +-
 gcc/analyzer/sm.cc |  2 +-
 gcc/analyzer/store.cc  |  6 +++---
 gcc/analyzer/supergraph.cc | 17 -
 9 files changed, 39 insertions(+), 47 deletions(-)

diff --git a/gcc/analyzer/call-string.cc b/gcc/analyzer/call-string.cc
index 23880e3f5419..c404c09ca0df 100644
--- a/gcc/analyzer/call-string.cc
+++ b/gcc/analyzer/call-string.cc
@@ -111,12 +111,9 @@ call_string::to_json () const
   for (const call_string::element_t &e : m_elements)
 {
   json::object *e_obj = new json::object ();
-  e_obj->set ("src_snode_idx",
- new json::integer_number (e.m_callee->m_index));
-  e_obj->set ("dst_snode_idx",
- new json::integer_number (e.m_caller->m_index));
-  e_obj->set ("funcname",
- new json::string (function_name (e.m_caller->m_fun)));
+  e_obj->set_integer ("src_snode_idx", e.m_callee->m_index);
+  e_obj->set_integer ("dst_snode_idx", e.m_caller->m_index);
+  e_obj->set_string ("funcname", function_name (e.m_caller->m_fun));
   arr->append (e_obj);
 }
 
diff --git a/gcc/analyzer/constraint-manager.cc 
b/gcc/analyzer/constraint-manager.cc
index 29539060ebdd..62d3b84bb745 100644
--- a/gcc/analyzer/constraint-manager.cc
+++ b/gcc/analyzer/constraint-manager.cc
@@ -479,7 +479,7 @@ bounded_range::set_json_attr (json::object *obj, const char 
*name, tree value)
   pretty_printer pp;
   pp_format_decoder (&pp) = default_tree_printer;
   pp_printf (&pp, "%E", value);
-  obj->set (name, new json::string (pp_formatted_text (&pp)));
+  obj->set_string (name, pp_formatted_text (&pp));
 }
 
 
@@ -1140,7 +1140,7 @@ equiv_class::to_json () const
   pretty_printer pp;
   pp_format_decoder (&pp) = default_tree_printer;
   pp_printf (&pp, "%qE", m_constant);
-  ec_obj->set ("constant", new json::string (pp_formatted_text (&pp)));
+  ec_obj->set_string ("constant", pp_formatted_text (&pp));
 }
 
   return ec_obj;
@@ -1397,9 +1397,9 @@ constraint::to_json () const
 {
   json::object *con_obj = new json::object ();
 
-  con_obj->set ("lhs", new json::integer_number (m_lhs.as_int ()));
-  con_obj->set ("op", new json::string (constraint_op_code (m_op)));
-  con_obj->set ("rhs", new json::integer_number (m_rhs.as_int ()));
+  con_obj->set_integer ("lhs", m_lhs.as_int ());
+  con_obj->set_string ("op", constraint_op_code (m_op));
+  con_obj->set_integer ("rhs", m_rhs.as_int ());
 
   return con_obj;
 }
@@ -1485,7 +1485,7 @@ bounded_ranges_constraint::to_json () const
 {
   json::object *con_obj = new json::object ();
 
-  con_obj->set ("ec", new json::integer_number (m_ec_id.as_int ()));
+  con_obj->set_integer ("ec", m_ec_id.as_int ());
   con_obj->set ("ranges", m_ranges->to_json ());
 
   return con_obj;
diff --git a/gcc/analyzer/diagnostic-manager.cc 
b/gcc/analyzer/diagnostic-manager.cc
index 51304b0795b6..92e30bd049bd 100644
--- a/gcc/analyzer/diagnostic-manager.cc
+++ b/gcc/analyzer/diagnostic-manager.cc
@@ -746,17 +746,17 @@ saved_diagnostic::to_json () const
   json::object *sd_obj = new json::object ();
 
   if (m_sm)
-sd_obj->set ("sm", new json::string (m_sm->get_name ()));
-  sd_obj->set ("enode", new json::integer_number (m_enode->m_index));
-  sd_obj->set ("snode", new json::integer_number (m_snode->m_index));
+sd_obj-&

[PATCH 05/16] diagnostics: SARIF output: add sarif_object subclasses throughout

[David Malcolm]

No functional change intended.

gcc/ChangeLog:
* diagnostic-format-sarif.cc: Introduce subclasses of sarif_object
for all aspects of the spec that we're using.  Replace almost all
usage of json::object with uses of these subclasses, the only
remaining use of json::object being for originalUriBaseIds, as per
SARIF 2.1.0 §3.14.14.  This stronger typing makes it considerably
easier to maintain validity against the schema.
* diagnostic-format-sarif.h (class sarif_logical_location): New.
(make_sarif_logical_location_object): Convert return type from
json::object * to sarif_logical_location *.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-sarif.cc | 605 +
 gcc/diagnostic-format-sarif.h  |   9 +-
 2 files changed, 399 insertions(+), 215 deletions(-)

diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 7105f7853318..c35d8788d6de 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -36,18 +36,87 @@ along with GCC; see the file COPYING3.  If not see
 #include "ordered-hash-map.h"
 #include "sbitmap.h"
 
+/* Forward decls.  */
 class sarif_builder;
 
-/* Subclass of json::object for SARIF invocation objects
+/* Subclasses of sarif_object.
+   Keep these in order of their descriptions in the specification.  */
+class sarif_artifact_content; // 3.3
+class sarif_artifact_location; // 3.4
+class sarif_message; // 3.11
+class sarif_multiformat_message_string; // 3.12
+class sarif_log; // 3.13
+class sarif_run; // 3.14
+class sarif_tool; // 3.18
+class sarif_tool_component; // 3.19
+class sarif_invocation; // 3.20
+class sarif_artifact; // 3.24
+class sarif_result; // 3.27
+class sarif_location; // 3.28
+class sarif_physical_location; // 3.29
+class sarif_region; // 3.30
+class sarif_logical_location; // 3.33
+class sarif_code_flow; // 3.36
+class sarif_thread_flow; // 3.37
+class sarif_thread_flow_location; // 3.38
+class sarif_reporting_descriptor; // 3.49
+class sarif_reporting_descriptor_reference; // 3.53
+class sarif_tool_component_reference; // 3.54
+class sarif_fix; // 3.55
+class sarif_artifact_change; // 3.56
+class sarif_replacement; // 3.57
+class sarif_ice_notification; // 3.58
+
+/* Declarations of subclasses of sarif_object.
+   Keep these in order of their descriptions in the specification.  */
+
+/* Subclass of sarif_object for SARIF "artifactContent" objects
+   (SARIF v2.1.0 section 3.3).  */
+
+class sarif_artifact_content : public sarif_object {};
+
+/* Subclass of sarif_object for SARIF "artifactLocation" objects
+   (SARIF v2.1.0 section 3.4).  */
+
+class sarif_artifact_location : public sarif_object {};
+
+/* Subclass of sarif_object for SARIF "message" objects
+   (SARIF v2.1.0 section 3.11).  */
+
+class sarif_message : public sarif_object {};
+
+/* Subclass of sarif_object for SARIF "multiformatMessageString" objects
+   (SARIF v2.1.0 section 3.12).  */
+
+class sarif_multiformat_message_string : public sarif_object {};
+
+/* Subclass of sarif_object for SARIF "log" objects
+   (SARIF v2.1.0 section 3.13).  */
+
+class sarif_log : public sarif_object {};
+
+/* Subclass of sarif_object for SARIF "run" objects
+   (SARIF v2.1.0 section 3.14).  */
+
+class sarif_run : public sarif_object {};
+
+/* Subclass of sarif_object for SARIF "tool" objects
+   (SARIF v2.1.0 section 3.18).  */
+
+class sarif_tool : public sarif_object {};
+
+/* Subclass of sarif_object for SARIF "toolComponent" objects
+   (SARIF v2.1.0 section 3.19).  */
+
+class sarif_tool_component : public sarif_object {};
+
+/* Subclass of sarif_object for SARIF "invocation" objects
(SARIF v2.1.0 section 3.20).  */
 
 class sarif_invocation : public sarif_object
 {
 public:
-  sarif_invocation ()
-  : m_notifications_arr (new json::array ()),
-m_success (true)
-  {}
+  sarif_invocation ();
 
   void add_notification_for_ice (diagnostic_context *context,
 const diagnostic_info &diagnostic,
@@ -104,7 +173,7 @@ private:
   bool m_embed_contents;
 };
 
-/* Subclass of sarif_object for SARIF result objects
+/* Subclass of sarif_object for SARIF "result" objects
(SARIF v2.1.0 section 3.27).  */
 
 class sarif_result : public sarif_object
@@ -122,42 +191,95 @@ public:
   sarif_builder *builder);
 
 private:
-  void add_related_location (json::object *location_obj);
+  void
+  add_related_location (sarif_location *location_obj);
 
   json::array *m_related_locations_arr;
 };
 
-/* Subclass of sarif_object for SARIF notification objects
-   (SARIF v2.1.0 section 3.58).
+/* Subclass of sarif_object for SARIF "location" objects
+   (SARIF v2.1.0 section 3.28).  */
 
-   This subclass is specifically for notifying when an
-   internal compiler error occurs.  */

[PATCH 07/16] json: support std::unique_ptr in array::append and object::set

[David Malcolm]

This patch uses templates to add overloads of json::array::append and
json::object::set taking std::unique_ptr where T is a subclass of
json::value.

Doing so makes it much easier to track memory ownership and enforce
schema validity when constructing non-trivial JSON; using the wrong
kind of JSON value leads to compile-time errors like the following:

error: cannot convert ‘unique_ptr’ to ‘unique_ptr’
  629 |   location_obj->set ("message", std::move (message_obj));
  |~~^
  |  |
  |  
unique_ptr

No functional change intended.

gcc/ChangeLog:
* diagnostic-format-json.cc: Define INCLUDE_MEMORY.
* diagnostic-format-sarif.cc: Likewise.
* dumpfile.cc: Likewise.
* gcov.cc: Likewise.
* json.cc: Likewise.  Include "make-unique.h".
(selftest::test_formatting): Exercise overloads of
array::append and object::set that use unique_ptr.
* json.h: Require INCLUDE_MEMORY to have been defined.
(json::object::set): Add a template to add a family of overloads
taking a std::unique_ptr
(json::array::append): Likewise.
* optinfo-emit-json.cc: Define INCLUDE_MEMORY.
* optinfo.cc: Likewise.
* timevar.cc: Likewise.
* toplev.cc: Likewise.
* tree-diagnostic-client-data-hooks.cc: Likewise.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-format-json.cc|  1 +
 gcc/diagnostic-format-sarif.cc   |  1 +
 gcc/dumpfile.cc  |  1 +
 gcc/gcov.cc  |  1 +
 gcc/json.cc  | 17 +++
 gcc/json.h   | 38 
 gcc/optinfo-emit-json.cc |  1 +
 gcc/optinfo.cc   |  1 +
 gcc/timevar.cc   |  1 +
 gcc/toplev.cc|  1 +
 gcc/tree-diagnostic-client-data-hooks.cc |  1 +
 11 files changed, 58 insertions(+), 6 deletions(-)

diff --git a/gcc/diagnostic-format-json.cc b/gcc/diagnostic-format-json.cc
index 1bf8da663cc2..55ba39e0c532 100644
--- a/gcc/diagnostic-format-json.cc
+++ b/gcc/diagnostic-format-json.cc
@@ -20,6 +20,7 @@ along with GCC; see the file COPYING3.  If not see
 
 
 #include "config.h"
+#define INCLUDE_MEMORY
 #include "system.h"
 #include "coretypes.h"
 #include "diagnostic.h"
diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index d6de5806f5ac..6aba81c6ac9b 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -20,6 +20,7 @@ along with GCC; see the file COPYING3.  If not see
 
 
 #include "config.h"
+#define INCLUDE_MEMORY
 #define INCLUDE_VECTOR
 #include "system.h"
 #include "coretypes.h"
diff --git a/gcc/dumpfile.cc b/gcc/dumpfile.cc
index 82bd8b06bebf..6353c0857449 100644
--- a/gcc/dumpfile.cc
+++ b/gcc/dumpfile.cc
@@ -18,6 +18,7 @@ along with GCC; see the file COPYING3.  If not see
 <http://www.gnu.org/licenses/>.  */
 
 #include "config.h"
+#define INCLUDE_MEMORY
 #include "system.h"
 #include "coretypes.h"
 #include "options.h"
diff --git a/gcc/gcov.cc b/gcc/gcov.cc
index 85fdac4368e8..aa016c658ce0 100644
--- a/gcc/gcov.cc
+++ b/gcc/gcov.cc
@@ -32,6 +32,7 @@ along with Gcov; see the file COPYING3.  If not see
 
 #include "config.h"
 #define INCLUDE_ALGORITHM
+#define INCLUDE_MEMORY
 #define INCLUDE_VECTOR
 #define INCLUDE_STRING
 #define INCLUDE_MAP
diff --git a/gcc/json.cc b/gcc/json.cc
index 86490259dabf..275ef486faf1 100644
--- a/gcc/json.cc
+++ b/gcc/json.cc
@@ -19,11 +19,13 @@ along with GCC; see the file COPYING3.  If not see
 <http://www.gnu.org/licenses/>.  */
 
 #include "config.h"
+#define INCLUDE_MEMORY
 #include "system.h"
 #include "coretypes.h"
 #include "json.h"
 #include "pretty-print.h"
 #include "math.h"
+#include "make-unique.h"
 #include "selftest.h"
 
 using namespace json;
@@ -499,28 +501,31 @@ test_writing_literals ()
   ASSERT_PRINT_EQ (literal (false), true, "false");
 }
 
-/* Verify that nested values are formatted correctly when written.  */
+/* Verify that nested values are formatted correctly when written.
+
+   Also, make use of array::append(std::unique_ptr) and
+   object::set (const char *key, std::unique_ptr v).*/
 
 static void
 test_formatting ()
 {
   object obj;
   object *child = new object;
-  object *grandchild = new object;
+  std::unique_ptr grandchild = ::make_unique ();
 
   obj.set_string ("str", "bar");
   obj.set ("child", child);
   obj.set_integer ("int", 42);
 
-  child->set ("grandchild", gr

[PATCH 02/16] json: add array::append_string

[David Malcolm]

No functional change intended.

gcc/analyzer/ChangeLog:
* supergraph.cc (supernode::to_json): Avoid naked "new" by using
json::array::append_string.
(supernode::to_json): Likewise.

gcc/ChangeLog:
* diagnostic-format-sarif.cc (sarif_artifact::populate_roles):
Avoid naked "new" by using json::array::append_string.
(sarif_builder::maybe_make_kinds_array): Likewise.
* json.cc (json::array::append_string): New.
(selftest::test_writing_arrays): Use it.
* json.h (json::array::append_string): New decl.
* optinfo-emit-json.cc (optrecord_json_writer::pass_to_json):
Avoid naked "new" by using json::array::append_string.
(optrecord_json_writer::optinfo_to_json): Likewise.

Signed-off-by: David Malcolm 
---
 gcc/analyzer/supergraph.cc | 4 ++--
 gcc/diagnostic-format-sarif.cc | 8 
 gcc/json.cc| 9 -
 gcc/json.h | 1 +
 gcc/optinfo-emit-json.cc   | 4 ++--
 5 files changed, 17 insertions(+), 9 deletions(-)

diff --git a/gcc/analyzer/supergraph.cc b/gcc/analyzer/supergraph.cc
index 4dc7942b26aa..20c62856d4b6 100644
--- a/gcc/analyzer/supergraph.cc
+++ b/gcc/analyzer/supergraph.cc
@@ -747,7 +747,7 @@ supernode::to_json () const
pretty_printer pp;
pp_format_decoder (&pp) = default_tree_printer;
pp_gimple_stmt_1 (&pp, stmt, 0, (dump_flags_t)0);
-   phi_arr->append (new json::string (pp_formatted_text (&pp)));
+   phi_arr->append_string (pp_formatted_text (&pp));
   }
 snode_obj->set ("phis", phi_arr);
   }
@@ -762,7 +762,7 @@ supernode::to_json () const
pretty_printer pp;
pp_format_decoder (&pp) = default_tree_printer;
pp_gimple_stmt_1 (&pp, stmt, 0, (dump_flags_t)0);
-   stmt_arr->append (new json::string (pp_formatted_text (&pp)));
+   stmt_arr->append_string (pp_formatted_text (&pp));
   }
 snode_obj->set ("stmts", stmt_arr);
   }
diff --git a/gcc/diagnostic-format-sarif.cc b/gcc/diagnostic-format-sarif.cc
index 225476995d13..7105f7853318 100644
--- a/gcc/diagnostic-format-sarif.cc
+++ b/gcc/diagnostic-format-sarif.cc
@@ -435,7 +435,7 @@ sarif_artifact::populate_roles ()
 if (bitmap_bit_p (m_roles, i))
   {
enum diagnostic_artifact_role role = (enum diagnostic_artifact_role)i;
-   roles_arr->append (new json::string (get_artifact_role_string (role)));
+   roles_arr->append_string (get_artifact_role_string (role));
   }
   set ("roles", roles_arr);
 }
@@ -1394,13 +1394,13 @@ sarif_builder::maybe_make_kinds_array 
(diagnostic_event::meaning m) const
   json::array *kinds_arr = new json::array ();
   if (const char *verb_str
= diagnostic_event::meaning::maybe_get_verb_str (m.m_verb))
-kinds_arr->append (new json::string (verb_str));
+kinds_arr->append_string (verb_str);
   if (const char *noun_str
= diagnostic_event::meaning::maybe_get_noun_str (m.m_noun))
-kinds_arr->append (new json::string (noun_str));
+kinds_arr->append_string (noun_str);
   if (const char *property_str
= diagnostic_event::meaning::maybe_get_property_str (m.m_property))
-kinds_arr->append (new json::string (property_str));
+kinds_arr->append_string (property_str);
   return kinds_arr;
 }
 
diff --git a/gcc/json.cc b/gcc/json.cc
index 53edca520556..86490259dabf 100644
--- a/gcc/json.cc
+++ b/gcc/json.cc
@@ -286,6 +286,13 @@ array::append (value *v)
   m_elements.safe_push (v);
 }
 
+void
+array::append_string (const char *utf8_value)
+{
+  gcc_assert (utf8_value);
+  append (new json::string (utf8_value));
+}
+
 /* class json::float_number, a subclass of json::value, wrapping a double.  */
 
 /* Implementation of json::value::print for json::float_number.  */
@@ -432,7 +439,7 @@ test_writing_arrays ()
   arr.append (new json::string ("foo"));
   ASSERT_PRINT_EQ (arr, true, "[\"foo\"]");
 
-  arr.append (new json::string ("bar"));
+  arr.append_string ("bar");
   ASSERT_PRINT_EQ (arr, true,
   "[\"foo\",\n"
   " \"bar\"]");
diff --git a/gcc/json.h b/gcc/json.h
index ad4f8c448f8c..d3493a72d525 100644
--- a/gcc/json.h
+++ b/gcc/json.h
@@ -130,6 +130,7 @@ class array : public value
   void print (pretty_printer *pp, bool formatted) const final override;
 
   void append (value *v);
+  void append_string (const char *utf8_value);
 
  private:
   auto_vec m_elements;
diff --git a/gcc/optinfo-emit-json.cc b/gcc/optinfo-emit-json.cc
index 1fa82d438db9..faae95fc232a 100644
--- a/gcc/optinfo-emit-json.cc
+++ b/gcc/optinfo-emit-json.cc
@@ -259,7 +259,7 @@ optrecord_json_writer::pass_to_json (opt_pass *pass)
 optgroup->name != NULL; optgroup++)
   if (optgroup->value != OPTGROUP

[PATCH 01/16] json: add dump overload for easier debugging

[David Malcolm]

This has saved me a lot of typing in the debugger.

gcc/ChangeLog:
* json.cc (value::dump): New overload, taking no params.
* json.h (value::dump): New decl.

Signed-off-by: David Malcolm 
---
 gcc/json.cc | 10 ++
 gcc/json.h  |  1 +
 2 files changed, 11 insertions(+)

diff --git a/gcc/json.cc b/gcc/json.cc
index b3106f39cedf..53edca520556 100644
--- a/gcc/json.cc
+++ b/gcc/json.cc
@@ -90,6 +90,16 @@ value::dump (FILE *outf, bool formatted) const
   pp_flush (&pp);
 }
 
+/* A convenience function for debugging.
+   Dump to stderr with formatting, and a trailing newline. */
+
+void
+value::dump () const
+{
+  dump (stderr, true);
+  fprintf (stderr, "\n");
+}
+
 /* class json::object, a subclass of json::value, representing
an ordered collection of key/value pairs.  */
 
diff --git a/gcc/json.h b/gcc/json.h
index 97c68116b329..ad4f8c448f8c 100644
--- a/gcc/json.h
+++ b/gcc/json.h
@@ -83,6 +83,7 @@ class value
   virtual void print (pretty_printer *pp, bool formatted) const = 0;
 
   void dump (FILE *, bool formatted) const;
+  void DEBUG_FUNCTION dump () const;
 };
 
 /* Subclass of value for objects: a collection of key/value pairs
-- 
2.26.3

[pushed 00/16] Revamp of JSON/SARIF output

[David Malcolm]

The following patch kit overhauls various aspects of JSON/SARIF output:

* patch 1: adds a simpler json::value::dump () method to make it easier
  to debug JSON-handling code

* patches 2-9 are cleanups/refactorings of the json-handling code and
  the JSON/SARIF output formats with no functional change intended.
  Among other things, they make it more type-safe (useful when enforcing
  a schema), and use std::unique_ptr in many places to indicate
  ownership.

* patches 10-16: add new functionality to our SARIF output whilst
  building out selftest coverage for it.

See the individual patches for more details.

Lightly tested with valgrind.

Successfully bootstrapped with GCC 4.8.5 on
powerpc64le-unknown-linux-gnu (cfarm135).

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.

Successful run of analyzer integration tests on x86_64-pc-linux-gnu.

Pushed to trunk as r15-2278-g6baa26c3d6ceec through
r15-2293-g142003df61b45b.

David Malcolm (16):
  json: add dump overload for easier debugging
  json: add array::append_string
  analyzer: reduce use of naked "new" for json dumps
  gcov: reduce use of naked "new" for json output
  diagnostics: SARIF output: add sarif_object subclasses throughout
  diagnostics: output formats: use references for non-null pointers
  json: support std::unique_ptr in array::append and object::set
  diagnostics: SARIF output: use std::unique_ptr throughout
  diagnostics: JSON output: use std::unique_ptr throughout
  diagnostics: SARIF output: potentially add escaped renderings of
source (§3.3.4)
  diagnostics: SARIF output: add "workingDirectory" property (§3.20.19)
  diagnostics: SARIF output: add "arguments" property (§3.20.2)
  diagnostics: SARIF output: add "{start,end}TimeUtc" properties
(§§3.20.7-8)
  diagnostics: SARIF output: add "annotations" property (§3.28.6)
  diagnostics: add selftests for SARIF output
  diagnostics: SARIF output: tweak output for UNKNOWN_LOCATION

 gcc/Makefile.in   |1 +
 gcc/analyzer/call-string.cc   |9 +-
 gcc/analyzer/checker-event.cc |5 +-
 gcc/analyzer/constraint-manager.cc|   12 +-
 gcc/analyzer/diagnostic-manager.cc|   14 +-
 gcc/analyzer/engine.cc|   13 +-
 gcc/analyzer/program-point.cc |   11 +-
 gcc/analyzer/program-state.cc |2 +-
 gcc/analyzer/sm.cc|2 +-
 gcc/analyzer/store.cc |6 +-
 gcc/analyzer/supergraph.cc|   21 +-
 gcc/diagnostic-format-json.cc |  167 +-
 gcc/diagnostic-format-sarif.cc| 1738 -
 gcc/diagnostic-format-sarif.h |9 +-
 gcc/diagnostic-show-locus.cc  |   98 +-
 gcc/diagnostic.cc |   19 +-
 gcc/diagnostic.h  |   25 +-
 gcc/dumpfile.cc   |1 +
 gcc/gcc-rich-location.h   |   17 -
 gcc/gcc.cc|3 +-
 gcc/gcov.cc   |5 +-
 gcc/jit/jit-playback.cc   |6 +-
 gcc/json.cc   |   36 +-
 gcc/json.h|   45 +
 gcc/optinfo-emit-json.cc  |5 +-
 gcc/optinfo.cc|1 +
 gcc/opts.cc   |3 +-
 gcc/selftest-diagnostic-show-locus.h  |   82 +
 gcc/selftest-diagnostic.cc|   14 +
 gcc/selftest-diagnostic.h |   10 +
 gcc/selftest-json.cc  |  163 ++
 gcc/selftest-json.h   |  141 ++
 gcc/selftest-run-tests.cc |1 +
 gcc/selftest.h|1 +
 .../diagnostic-format-sarif-file-1.c  |   11 +
 ...diagnostic-format-sarif-file-Wbidi-chars.c |   17 +
 .../diagnostic_plugin_test_show_locus.c   |1 +
 gcc/text-range-label.h|   42 +
 gcc/timevar.cc|1 +
 gcc/toplev.cc |   14 +-
 gcc/tree-diagnostic-client-data-hooks.cc  |1 +
 gcc/unique-argv.h |   67 +
 libcpp/include/rich-location.h|   31 +-
 libcpp/line-map.cc|   28 +
 44 files changed, 2179 insertions(+), 720 deletions(-)
 create mode 100644 gcc/selftest-diagnostic-show-locus.h
 create mode 100644 gcc/selftest-json.cc
 create mode 100644 gcc/selftest-json.h
 create mode 100644 gcc/text-range-label.h
 create mode 100644 gcc/unique-argv.h

-- 
2.26.3

[pushed] diagnostics: add highlight-a vs highlight-b in colorization and pp_markup

[David Malcolm]

ree-color-no-highlight-colors.C to
show_template_tree_color_plugin.c.
* g++.dg/plugin/show-template-tree-color-labels.C: Update expected
output to reflect use of highlight-a and highlight-b to contrast
mismatches.
* g++.dg/plugin/show-template-tree-color-no-elide-type.C:
Likewise.
* g++.dg/plugin/show-template-tree-color-no-highlight-colors.C:
New test.
* g++.dg/plugin/show-template-tree-color.C: Update expected output
to reflect use of highlight-a and highlight-b to contrast
mismatches.
* g++.dg/warn/Wformat-gcc_diag-1.C: New test.
* g++.dg/warn/Wformat-gcc_diag-2.C: New test.
* g++.dg/warn/Wformat-gcc_diag-3.C: New test.
* gcc.dg/bad-binary-ops-highlight-colors.c: New test.
* gcc.dg/format/colors.c: New test.
* gcc.dg/plugin/diagnostic_plugin_show_trees.c (show_tree): Pass
nullptr for new param of gcc_rich_location::add_expr.

libcpp/ChangeLog:
* include/rich-location.h (location_range::m_highlight_color): New
field.
(rich_location::rich_location): Add optional label_highlight_color
param.
(rich_location::set_highlight_color): New decl.
(rich_location::add_range): Add optional label_highlight_color
param.
(rich_location::set_range): Likewise.
* line-map.cc (rich_location::rich_location): Add
"label_highlight_color" param and pass it to add_range.
(rich_location::set_highlight_color): New.
(rich_location::add_range): Add "label_highlight_color" param.
(rich_location::set_range): Add "highlight_color" param.

Signed-off-by: David Malcolm 
---
 gcc/c-family/c-common.cc  |  17 +-
 gcc/c-family/c-common.h   |   6 +-
 gcc/c-family/c-format.cc  | 229 +-
 gcc/c-family/c-format.h   |   1 +
 gcc/c-family/c-type-mismatch.cc   |  10 +-
 gcc/c-family/c-type-mismatch.h|   3 +-
 gcc/c/c-objc-common.cc|  30 ++-
 gcc/c/c-typeck.cc |  46 +++-
 gcc/common.opt|   4 +
 gcc/common.opt.urls   |   3 +
 gcc/coretypes.h   |   2 +
 gcc/cp/call.cc|  75 --
 gcc/cp/cp-tree.h  |  13 +-
 gcc/cp/error.cc   | 107 +---
 gcc/cp/typeck.cc  |  20 +-
 gcc/diagnostic-color.cc   |   4 +-
 gcc/diagnostic-format-json.cc |   1 +
 gcc/diagnostic-format-sarif.cc|   1 +
 gcc/diagnostic-highlight-colors.h |  56 +
 gcc/diagnostic-path.cc|   2 +-
 gcc/diagnostic-show-locus.cc  |  58 -
 gcc/diagnostic.h  |   4 +
 gcc/doc/invoke.texi   |  51 +++-
 gcc/doc/ux.texi   |  87 +++
 gcc/gcc-rich-location.cc  |  12 +-
 gcc/gcc-rich-location.h   |  20 +-
 gcc/gcc.cc|   4 +
 gcc/lto-wrapper.cc|   7 +
 gcc/opts-common.cc|   2 +
 gcc/opts-global.cc|   5 +
 gcc/opts.cc   |   5 +
 gcc/pretty-print-markup.h |  75 ++
 gcc/pretty-print.cc   | 113 +
 gcc/pretty-print.h|  18 ++
 gcc/substring-locations.cc|  21 +-
 gcc/substring-locations.h |   3 +
 .../bad-binary-ops-highlight-colors.C |  34 +++
 .../bad-binary-ops-no-highlight-colors.C  |  29 +++
 gcc/testsuite/g++.dg/plugin/plugin.exp|   1 +
 .../plugin/show-template-tree-color-labels.C  |  20 +-
 .../show-template-tree-color-no-elide-type.C  |   8 +-
 ...-template-tree-color-no-highlight-colors.C |  32 +++
 .../g++.dg/plugin/show-template-tree-color.C  |   8 +-
 .../g++.dg/warn/Wformat-gcc_diag-1.C  | 200 +++
 .../g++.dg/warn/Wformat-gcc_diag-2.C  |  52 
 .../g++.dg/warn/Wformat-gcc_diag-3.C  |  54 +
 .../gcc.dg/bad-binary-ops-highlight-colors.c  |  35 +++
 .../bad-binary-ops-no-highlight-colors.c  |  30 +++
 gcc/testsuite/gcc.dg/format/colors.c  |  27 +++
 .../plugin/diagnostic_plugin_show_trees.c |   2 +-
 gcc/toplev.cc |   2 +
 gcc/tree-pretty-print-markup.h|  84 +++
 libcpp/include/rich-location.h|  14 +-
 libcpp/line-map.cc|  29 ++-
 54 files changed, 1580 insertions(+), 196 deletions(-)
 create mode 100644 gcc/diagnostic-highlight-colors.h
 create mode 100644 gcc/pretty-print-markup.h
 create 

Re: [PATCH] [analyzer] [testsuite] avoid unexpected null dereference warning

[David Malcolm]

On Thu, 2024-07-11 at 10:36 -0300, Alexandre Oliva wrote:
> 
> The analyzer testsuite, on a customer's own operating system, reports
> a potential NULL pointer dereference in flex-without-call-
> summaries.c.
> I'm not sure why it shows up on that system, but not on others, but
> the test is not meant to test for that warning, so I'm silencing it.

Hi Alex

I intended that particular test as an integration test, to try to make
sure that -fanalyzer is silent on the result of a minimal "flex"
script.  So if we're actually getting warnings, that's undesirable, and
I'm not sure the approach in your patch is the right one.

Given that the testcase includes various C headers, there's going to be
some variability with different configurations.  Are you able to share
the preprocessed sources with me?  Maybe we should eliminate those
#includes from that test and convert it to something that's more
config-independent?

Thanks
Dave


> 
> Regstrapped on x86_64-linux-gnu, also tested with gcc-13 targeting
> aarch64.  Ok to install?
> 
> 
> for  gcc/testsuite/ChangeLog
> 
> * c-c++-common/analyzer/flex-without-call-summaries.c:
> Disable
> null dereference analyzer warnings.
> ---
>  .../analyzer/flex-without-call-summaries.c |    1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/gcc/testsuite/c-c++-common/analyzer/flex-without-call-
> summaries.c b/gcc/testsuite/c-c++-common/analyzer/flex-without-call-
> summaries.c
> index c6ecb25d25d59..1aad2bc896b7e 100644
> --- a/gcc/testsuite/c-c++-common/analyzer/flex-without-call-
> summaries.c
> +++ b/gcc/testsuite/c-c++-common/analyzer/flex-without-call-
> summaries.c
> @@ -3,6 +3,7 @@
>  
>  /* { dg-additional-options "-fno-analyzer-call-summaries" } */
>  
> +/* { dg-additional-options "-Wno-analyzer-null-dereference" } */
>  /* { dg-additional-options "-Wno-analyzer-too-complex" } */
>  /* { dg-additional-options "-D_POSIX_SOURCE" } */
>  
> 

Re: [PATCH v2] Fix Xcode 16 build break with NULL != nullptr

[David Malcolm]

On Wed, 2024-07-10 at 09:43 +, Daniel Bertalan wrote:
> As of Xcode 16 beta 2 with the macOS 15 SDK, each re-inclusion of the
> stddef.h header causes the NULL macro in C++ to be re-defined to an
> integral constant (__null). This makes the workaround in d59a576b8
> ("Redefine NULL to nullptr") ineffective, as other headers that are
> typically included after system.h (such as obstack.h) do include
> stddef.h too.

Thanks for the patch; the analyzer parts look good to me.

Dave

> 
> This can be seen by running the sample below through `clang++ -E`
> 
>     #include 
>     #define NULL nullptr
>     #include 
>     NULL
> 
> The relevant libc++ change is here:
> https://github.com/llvm/llvm-project/commit/2950283dddab03c183c1be2d7de9d4999cc86131
> 
> Filed as FB14261859 to Apple and added a comment about it on LLVM PR
> 86843.
> 
> This fixes the cases in --enable-languages=c,c++,objc,obj-c++,rust
> build
> where NULL being an integral constant instead of a null pointer
> literal
> (therefore no longer implicitly converting to a pointer when used as
> a
> template function's argument) caused issues.
> 
>     gcc/value-pointer-equiv.cc:65:43: error: no viable conversion
> from `pair::type, typename
> __unwrap_ref_decay::type>' to 'const pair'
> 
>     65 |   const std::pair  m_marker = std::make_pair
> (NULL, NULL);
>    |  
> ^~~
> 
> As noted in the previous commit though, the proper solution would be
> to
> phase out the usages of NULL in GCC's C++ source code.
> 
> gcc/analyzer/ChangeLog:
> 
> * diagnostic-manager.cc (saved_diagnostic::saved_diagnostic):
> Change NULL to nullptr.
> (struct null_assignment_sm_context): Likewise.
> * infinite-loop.cc: Likewise.
> * infinite-recursion.cc: Likewise.
> * varargs.cc (va_list_state_machine::on_leak): Likewise.
> 
> gcc/rust/ChangeLog:
> 
> * metadata/rust-imports.cc
> (Import::try_package_in_directory):
> Change NULL to nullptr.
> 
> gcc/ChangeLog:
> 
> * value-pointer-equiv.cc: Change NULL to nullptr.
> ---
>  gcc/analyzer/diagnostic-manager.cc | 18 +-
>  gcc/analyzer/infinite-loop.cc  |  2 +-
>  gcc/analyzer/infinite-recursion.cc |  2 +-
>  gcc/analyzer/varargs.cc    |  2 +-
>  gcc/rust/metadata/rust-imports.cc  |  2 +-
>  gcc/value-pointer-equiv.cc |  2 +-
>  6 files changed, 14 insertions(+), 14 deletions(-)
> 
> diff --git a/gcc/analyzer/diagnostic-manager.cc
> b/gcc/analyzer/diagnostic-manager.cc
> index fe943ac61c9e..51304b0795b6 100644
> --- a/gcc/analyzer/diagnostic-manager.cc
> +++ b/gcc/analyzer/diagnostic-manager.cc
> @@ -679,12 +679,12 @@ saved_diagnostic::saved_diagnostic (const
> state_machine *sm,
>    m_stmt (ploc.m_stmt),
>    /* stmt_finder could be on-stack; we want our own copy that can
>   outlive that.  */
> -  m_stmt_finder (ploc.m_finder ? ploc.m_finder->clone () : NULL),
> +  m_stmt_finder (ploc.m_finder ? ploc.m_finder->clone () : nullptr),
>    m_loc (ploc.m_loc),
>    m_var (var), m_sval (sval), m_state (state),
> -  m_d (std::move (d)), m_trailing_eedge (NULL),
> +  m_d (std::move (d)), m_trailing_eedge (nullptr),
>    m_idx (idx),
> -  m_best_epath (NULL), m_problem (NULL),
> +  m_best_epath (nullptr), m_problem (nullptr),
>    m_notes ()
>  {
>    /* We must have an enode in order to be able to look for paths
> @@ -1800,10 +1800,10 @@ public:
> stmt,
> stack_depth,
> sm,
> -   NULL,
> +   nullptr,
> src_sm_val,
> dst_sm_val,
> -   NULL,
> +   nullptr,
> dst_state,
> src_node));
>  return false;
> @@ -1993,9 +1993,9 @@ struct null_assignment_sm_context : public
> sm_context
> m_sm,
> var_new_sval,
> from, to,
> -   NULL,
> +   nullptr,
> *m_new_state,
> -   NULL));
> +   nullptr));
>    }
>  
>    void set_next_state (const gimple *stmt,
> @@ -2019,9 +2019,9 @@ struct null_assignment_sm_context : public
> sm_context
> m_sm,
> sval,
> from, to,
> -   NULL,
> +   nullptr,
> *m_new_state,
> -   

[pushed] diagnostics: use refs rather than pointers for diagnostic_{path, context}

[David Malcolm]

Use const & rather than const * in various places where it can't be null
and can't change.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-1913-g2d6e6a77e840b7.

gcc/ChangeLog:
* diagnostic-path.cc: Replace "const diagnostic_path *" with
"const diagnostic_path &" throughout, and "diagnostic_context *"
with "diagnostic context &".
* diagnostic.cc (diagnostic_context::show_any_path): Pass
reference in call to print_path.
* diagnostic.h (diagnostic_context::print_path): Convert param
to a reference.

Signed-off-by: David Malcolm 
---
 gcc/diagnostic-path.cc | 101 -
 gcc/diagnostic.cc  |   2 +-
 gcc/diagnostic.h   |   2 +-
 3 files changed, 52 insertions(+), 53 deletions(-)

diff --git a/gcc/diagnostic-path.cc b/gcc/diagnostic-path.cc
index e470bd29fdd..f279aead6c5 100644
--- a/gcc/diagnostic-path.cc
+++ b/gcc/diagnostic-path.cc
@@ -225,7 +225,7 @@ class path_label : public range_label
 {
  public:
   path_label (const diagnostic_context &ctxt,
- const diagnostic_path *path,
+ const diagnostic_path &path,
  unsigned start_idx)
   : m_ctxt (ctxt), m_path (path), m_start_idx (start_idx), m_effects (*this)
   {}
@@ -233,7 +233,7 @@ class path_label : public range_label
   label_text get_text (unsigned range_idx) const final override
   {
 unsigned event_idx = m_start_idx + range_idx;
-const diagnostic_event &event = m_path->get_event (event_idx);
+const diagnostic_event &event = m_path.get_event (event_idx);
 
 /* Get the description of the event, perhaps with colorization:
normally, we don't colorize within a range_label, but this
@@ -305,7 +305,7 @@ class path_label : public range_label
   const diagnostic_event &get_event (unsigned range_idx) const
   {
 unsigned event_idx = m_start_idx + range_idx;
-return m_path->get_event (event_idx);
+return m_path.get_event (event_idx);
   }
 
   const diagnostic_event *get_prev_event (unsigned range_idx) const
@@ -313,11 +313,11 @@ class path_label : public range_label
 if (m_start_idx + range_idx == 0)
   return nullptr;
 unsigned event_idx = m_start_idx + range_idx - 1;
-return &m_path->get_event (event_idx);
+return &m_path.get_event (event_idx);
   }
 
   const diagnostic_context &m_ctxt;
-  const diagnostic_path *m_path;
+  const diagnostic_path &m_path;
   unsigned m_start_idx;
   path_label_effects m_effects;
 };
@@ -508,7 +508,7 @@ struct event_range
   };
 
   event_range (const diagnostic_context &ctxt,
-  const diagnostic_path *path, unsigned start_idx,
+  const diagnostic_path &path, unsigned start_idx,
   const diagnostic_event &initial_event,
   per_thread_summary &t,
   bool show_event_links)
@@ -559,7 +559,7 @@ struct event_range
unsigned new_ev_idx,
bool check_rich_locations)
   {
-if (!can_consolidate_events (*m_path,
+if (!can_consolidate_events (m_path,
 m_initial_event, m_start_idx,
 new_ev, new_ev_idx,
 check_rich_locations))
@@ -574,7 +574,7 @@ struct event_range
   = get_per_source_line_info (exploc.line);
 const diagnostic_event *prev_event = nullptr;
 if (new_ev_idx > 0)
-  prev_event = &m_path->get_event (new_ev_idx - 1);
+  prev_event = &m_path.get_event (new_ev_idx - 1);
 const bool has_in_edge = (prev_event
  ? prev_event->connect_to_next_event_p ()
  : false);
@@ -604,7 +604,7 @@ struct event_range
   /* Print the events in this range to DC, typically as a single
  call to the printer's diagnostic_show_locus.  */
 
-  void print (diagnostic_context *dc, pretty_printer *pp,
+  void print (diagnostic_context &dc, pretty_printer *pp,
  diagnostic_source_effect_info *effect_info)
   {
 location_t initial_loc = m_initial_event.get_location ();
@@ -612,13 +612,13 @@ struct event_range
 /* Emit a span indicating the filename (and line/column) if the
line has changed relative to the last call to
diagnostic_show_locus.  */
-if (dc->m_source_printing.enabled)
+if (dc.m_source_printing.enabled)
   {
expanded_location exploc
  = linemap_client_expand_location_to_spelling_point
  (line_table, initial_loc, LOCATION_ASPECT_CARET);
-   if (exploc.file != LOCATION_FILE (dc->m_last_location))
- diagnostic_start_span (dc) (dc, exploc);
+   if (exploc.file != LOCATION_FILE (dc.m_last_location))
+ diagnostic_start_span (&dc) (&dc, explo

[pushed] c-format.cc: add ctors to format_check_results and format_check_context

[David Malcolm]

This is a minor cleanup I spotted whilst working on another patch.
No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as r15-1900-g113b5ce0610207.

gcc/c-family/ChangeLog:
* c-format.cc (format_check_results::format_check_results): New
ctor.
(struct format_check_context): Add ctor; add "m_" prefix to all
fields.
(check_format_info): Use above ctors.
(check_format_arg): Update for "m_" prefix to
format_check_context.

Signed-off-by: David Malcolm 
---
 gcc/c-family/c-format.cc | 63 +++-
 1 file changed, 37 insertions(+), 26 deletions(-)

diff --git a/gcc/c-family/c-format.cc b/gcc/c-family/c-format.cc
index 7a5ffc25602c..5bfd2fc4469e 100644
--- a/gcc/c-family/c-format.cc
+++ b/gcc/c-family/c-format.cc
@@ -1021,6 +1021,20 @@ static int n_format_types = ARRAY_SIZE 
(format_types_orig);
many leaves resulting from nested conditional expressions.  */
 struct format_check_results
 {
+  format_check_results (location_t format_string_loc_)
+  : number_non_literal (0),
+number_extra_args (0),
+extra_arg_loc (UNKNOWN_LOCATION),
+number_dollar_extra_args (0),
+number_wide (0),
+number_non_char (0),
+number_empty (0),
+number_unterminated (0),
+number_other (0),
+format_string_loc (format_string_loc_)
+  {
+  }
+
   /* Number of leaves of the format argument that could not be checked
  as they were not string literals.  */
   int number_non_literal;
@@ -1050,10 +1064,21 @@ struct format_check_results
 
 struct format_check_context
 {
-  format_check_results *res;
-  function_format_info *info;
-  tree params;
-  vec *arglocs;
+  format_check_context (format_check_results *res,
+   function_format_info *info,
+   tree params,
+   vec *arglocs)
+  : m_res (res),
+m_info (info),
+m_params (params),
+m_arglocs (arglocs)
+  {
+  }
+
+  format_check_results *m_res;
+  function_format_info *m_info;
+  tree m_params;
+  vec *m_arglocs;
 };
 
 /* Return the format name (as specified in the original table) for the format
@@ -1539,10 +1564,8 @@ static void
 check_format_info (function_format_info *info, tree params,
   vec *arglocs)
 {
-  format_check_context format_ctx;
   unsigned HOST_WIDE_INT arg_num;
   tree format_tree;
-  format_check_results res;
   /* Skip to format argument.  If the argument isn't available, there's
  no work for us to do; prototype checking will catch the problem.  */
   for (arg_num = 1; ; ++arg_num)
@@ -1558,26 +1581,14 @@ check_format_info (function_format_info *info, tree 
params,
   if (format_tree == 0)
 return;
 
-  res.number_non_literal = 0;
-  res.number_extra_args = 0;
-  res.extra_arg_loc = UNKNOWN_LOCATION;
-  res.number_dollar_extra_args = 0;
-  res.number_wide = 0;
-  res.number_non_char = 0;
-  res.number_empty = 0;
-  res.number_unterminated = 0;
-  res.number_other = 0;
-  res.format_string_loc = input_location;
-
-  format_ctx.res = &res;
-  format_ctx.info = info;
-  format_ctx.params = params;
-  format_ctx.arglocs = arglocs;
+  format_check_results res (input_location);
+
+  format_check_context format_ctx (&res, info, params, arglocs);
 
   check_function_arguments_recurse (check_format_arg, &format_ctx,
format_tree, arg_num, OPT_Wformat_);
 
-  location_t loc = format_ctx.res->format_string_loc;
+  location_t loc = format_ctx.m_res->format_string_loc;
 
   if (res.number_non_literal > 0)
 {
@@ -1659,10 +1670,10 @@ check_format_arg (void *ctx, tree format_tree,
  unsigned HOST_WIDE_INT arg_num)
 {
   format_check_context *format_ctx = (format_check_context *) ctx;
-  format_check_results *res = format_ctx->res;
-  function_format_info *info = format_ctx->info;
-  tree params = format_ctx->params;
-  vec *arglocs = format_ctx->arglocs;
+  format_check_results *res = format_ctx->m_res;
+  function_format_info *info = format_ctx->m_info;
+  tree params = format_ctx->m_params;
+  vec *arglocs = format_ctx->m_arglocs;
 
   int format_length;
   HOST_WIDE_INT offset;
-- 
2.26.3

[pushed] analyzer: handle at -O0 [PR115724]

[David Malcolm]

At -O0, glibc's:

__extern_always_inline void
error (int __status, int __errnum, const char *__format, ...)
{
  if (__builtin_constant_p (__status) && __status != 0)
__error_noreturn (__status, __errnum, __format, __builtin_va_arg_pack ());
  else
__error_alias (__status, __errnum, __format, __builtin_va_arg_pack ());
}

becomes just:

__extern_always_inline void
error (int __status, int __errnum, const char *__format, ...)
{
  if (0)
__error_noreturn (__status, __errnum, __format, __builtin_va_arg_pack ());
  else
__error_alias (__status, __errnum, __format, __builtin_va_arg_pack ());
}

and thus calls to "error" are calls to "__error_alias" by the
time -fanalyzer "sees" them.

Handle them with more special-casing in kf.cc.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Successful run of analyzer integration tests on x86_64-pc-linux-gnu.
Pushed to trunk as r15-1845-ga6fdb1a2a29061.

gcc/analyzer/ChangeLog:
PR analyzer/115724
* kf.cc (register_known_functions): Add __error_alias and
__error_at_line_alias.

gcc/testsuite/ChangeLog:
PR analyzer/115724
* c-c++-common/analyzer/error-pr115724.c: New test.

Signed-off-by: David Malcolm 
---
 gcc/analyzer/kf.cc|  4 +
 .../c-c++-common/analyzer/error-pr115724.c| 86 +++
 2 files changed, 90 insertions(+)
 create mode 100644 gcc/testsuite/c-c++-common/analyzer/error-pr115724.c

diff --git a/gcc/analyzer/kf.cc b/gcc/analyzer/kf.cc
index 4213b89ac9fb..5c3a71fbb49c 100644
--- a/gcc/analyzer/kf.cc
+++ b/gcc/analyzer/kf.cc
@@ -2325,6 +2325,10 @@ register_known_functions (known_function_manager &kfm,
 kfm.add ("__errno_location", make_unique ());
 kfm.add ("error", make_unique (3));
 kfm.add ("error_at_line", make_unique (5));
+/* Variants of "error" and "error_at_line" seen by the
+   analyzer at -O0 (PR analyzer/115724).  */
+kfm.add ("__error_alias", make_unique (3));
+kfm.add ("__error_at_line_alias", make_unique (5));
   }
 
   /* Other implementations of C standard library.  */
diff --git a/gcc/testsuite/c-c++-common/analyzer/error-pr115724.c 
b/gcc/testsuite/c-c++-common/analyzer/error-pr115724.c
new file mode 100644
index ..ae606ad89d6a
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/analyzer/error-pr115724.c
@@ -0,0 +1,86 @@
+/* Verify that the analyzer handles the no-optimization case in
+   glibc's  when error,error_at_line calls become
+   __error_alias and __error_at_line_alias.  */
+
+typedef __SIZE_TYPE__ size_t;
+#define EXIT_FAILURE 1
+#define __extern_always_inline  extern inline __attribute__ 
((__always_inline__)) __attribute__ ((__gnu_inline__))
+
+int errno;
+
+/* Adapted from glibc's bits/error.h.  */
+
+extern void __error_alias (int __status, int __errnum,
+  const char *__format, ...)
+  __attribute__ ((__format__ (__printf__, 3, 4)));
+extern void __error_noreturn (int __status, int __errnum,
+  const char *__format, ...)
+  __attribute__ ((__noreturn__, __format__ (__printf__, 3, 4)));
+
+/* If we know the function will never return make sure the compiler
+   realizes that, too.  */
+__extern_always_inline void
+error (int __status, int __errnum, const char *__format, ...)
+{
+  if (__builtin_constant_p (__status) && __status != 0)
+__error_noreturn (__status, __errnum, __format, __builtin_va_arg_pack ());
+  else
+__error_alias (__status, __errnum, __format, __builtin_va_arg_pack ());
+}
+
+extern void __error_at_line_alias (int __status, int __errnum,
+  const char *__fname,
+  unsigned int __line,
+  const char *__format, ...)
+  __attribute__ ((__format__ (__printf__, 5, 6)));
+extern void __error_at_line_noreturn (int __status, int __errnum,
+ const char *__fname,
+ unsigned int __line,
+ const char *__format,
+ ...)
+  __attribute__ ((__noreturn__, __format__ (__printf__, 5, 6)));
+
+/* If we know the function will never return make sure the compiler
+   realizes that, too.  */
+__extern_always_inline void
+error_at_line (int __status, int __errnum, const char *__fname,
+  unsigned int __line, const char *__format, ...)
+{
+  if (__builtin_constant_p (__status) && __status != 0)
+__error_at_line_noreturn (__status, __errnum, __fname, __line, __format,
+ __builtin_va_arg_pack ());
+  else
+__error_at_line_alias (__status, __errnum, __fname, __line,
+  __format, __builtin_va_arg_pack ());
+}
+
+
+struct list {
+size_t size;
+void (*destroy)(void *data);
+  

gcc-patches@gcc.gnu.org

[David Malcolm]

These are never nullptr and never change, so use a reference rather
than a pointer.

No functional change intended.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Successful run of analyzer integration tests on x86_64-pc-linux-gnu.
Pushed to trunk as r15-1846-gf8c130cdf53165.

gcc/analyzer/ChangeLog:
* diagnostic-manager.cc
(diagnostic_manager::add_events_for_eedge): Pass sm_ctxt by
reference.
* engine.cc (impl_region_model_context::on_condition): Likewise.
(impl_region_model_context::on_bounded_ranges): Likewise.
(impl_region_model_context::on_phi): Likewise.
(exploded_node::on_stmt): Likewise.
* sm-fd.cc: Update all uses of sm_context * to sm_context &.
* sm-file.cc: Likewise.
* sm-malloc.cc: Likewise.
* sm-pattern-test.cc: Likewise.
* sm-sensitive.cc: Likewise.
* sm-signal.cc: Likewise.
* sm-taint.cc: Likewise.
* sm.h: Likewise.
* varargs.cc: Likewise.

gcc/testsuite/ChangeLog:
* gcc.dg/plugin/analyzer_gil_plugin.c: Update all uses of
sm_context * to sm_context &.

Signed-off-by: David Malcolm 
---
 gcc/analyzer/diagnostic-manager.cc|   2 +-
 gcc/analyzer/engine.cc|   8 +-
 gcc/analyzer/sm-fd.cc | 318 +-
 gcc/analyzer/sm-file.cc   |  38 +--
 gcc/analyzer/sm-malloc.cc | 194 +--
 gcc/analyzer/sm-pattern-test.cc   |  14 +-
 gcc/analyzer/sm-sensitive.cc  |  22 +-
 gcc/analyzer/sm-signal.cc |  20 +-
 gcc/analyzer/sm-taint.cc  | 122 ---
 gcc/analyzer/sm.h |   8 +-
 gcc/analyzer/varargs.cc   |  54 +--
 .../gcc.dg/plugin/analyzer_gil_plugin.c   |  46 +--
 12 files changed, 419 insertions(+), 427 deletions(-)

diff --git a/gcc/analyzer/diagnostic-manager.cc 
b/gcc/analyzer/diagnostic-manager.cc
index 20e793d72c19..fe943ac61c9e 100644
--- a/gcc/analyzer/diagnostic-manager.cc
+++ b/gcc/analyzer/diagnostic-manager.cc
@@ -2239,7 +2239,7 @@ diagnostic_manager::add_events_for_eedge (const 
path_builder &pb,
&iter_point,
emission_path,
pb.get_ext_state 
());
-   sm.on_stmt (&sm_ctxt, dst_point.get_supernode (), stmt);
+   sm.on_stmt (sm_ctxt, dst_point.get_supernode (), stmt);
// TODO: what about phi nodes?
  }
  }
diff --git a/gcc/analyzer/engine.cc b/gcc/analyzer/engine.cc
index f5fad5b2e470..c9f204b13e70 100644
--- a/gcc/analyzer/engine.cc
+++ b/gcc/analyzer/engine.cc
@@ -969,7 +969,7 @@ impl_region_model_context::on_condition (const svalue *lhs,
   m_old_state->m_checker_states[sm_idx],
   m_new_state->m_checker_states[sm_idx],
   m_path_ctxt);
-  sm.on_condition (&sm_ctxt,
+  sm.on_condition (sm_ctxt,
   (m_enode_for_diag
? m_enode_for_diag->get_supernode ()
: NULL),
@@ -996,7 +996,7 @@ impl_region_model_context::on_bounded_ranges (const svalue 
&sval,
   m_old_state->m_checker_states[sm_idx],
   m_new_state->m_checker_states[sm_idx],
   m_path_ctxt);
-  sm.on_bounded_ranges (&sm_ctxt,
+  sm.on_bounded_ranges (sm_ctxt,
(m_enode_for_diag
 ? m_enode_for_diag->get_supernode ()
 : NULL),
@@ -1037,7 +1037,7 @@ impl_region_model_context::on_phi (const gphi *phi, tree 
rhs)
   m_old_state->m_checker_states[sm_idx],
   m_new_state->m_checker_states[sm_idx],
   m_path_ctxt);
-  sm.on_phi (&sm_ctxt, m_enode_for_diag->get_supernode (), phi, rhs);
+  sm.on_phi (sm_ctxt, m_enode_for_diag->get_supernode (), phi, rhs);
 }
 }
 
@@ -1559,7 +1559,7 @@ exploded_node::on_stmt (exploded_graph &eg,
   unknown_side_effects);
 
   /* Allow the state_machine to handle the stmt.  */
-  if (sm.on_stmt (&sm_ctxt, snode, stmt))
+  if (sm.on_stmt (sm_ctxt, snode, stmt))
unknown_side_effects = false;
 }
 
diff --git a/gcc/analyzer/sm-fd.cc b/gcc/analyzer/sm-fd.cc
index ded20576fd10..3396b1d11228 100644
--- a/gcc/analyzer/sm-fd.cc
+++ b/gcc/analyzer/sm-fd.cc
@@ -119,10 +119,10 @@ public:
 return m_start;
   }
 
-  bool on_stmt (sm_context *sm_ctxt, const supernode *node,
+  bool on_stmt (sm_context &sm_ctx

Re: [RFC][PATCH v1] Provide more contexts for -Warray-bounds warning messages

[David Malcolm]

On Tue, 2024-07-02 at 18:02 -0400, David Malcolm wrote:
> On Tue, 2024-07-02 at 16:17 +, Qing Zhao wrote:

[...snip...]
> 
> > +    path.add_event (cur_ch->condition, NULL_TREE, 0,
> > +   "when the condition is evaluated to %s",
> > +   cur_ch->is_true_path ? "true" : "false");
> 
> add_event's 2nd param is the function in which the event occurs,
> which
> you may want to set to cfun, and set the stack depth to 1, rather
 
 cfun's decl, that is, sorry.


[...snip...]

> 
> Or, all those
> 
>   if (for_array_bound)
> warned = warning_at (richloc, OPT_Warray_bounds_, 
>  /* etc */);
> 
> could become:
> 
>   if (for_array_bound)
>     {
>   copy_history_diagnostic_path path (location, stmt);
>   warned = warning_at (richloc, OPT_Warray_bounds_, 
>    /* etc */);
>     }

Or rather:

  if (for_array_bound)
{
  copy_history_location richloc (location, stmt);
  warned = warning_at (&richloc, OPT_Warray_bounds_, 
   /* etc */);
}

where the copy_history_location is a new rich_location subclass, and it
only populates the path if it's asked for one (when we're actually
emitting the diagnostic).

> or somesuch, explicitly deferring creation of the path until we're
> actually about to emit a warning, and putting the logic in
> copy_history_diagnostic_path's ctor.

[...snip...]


Dave

Re: [RFC][PATCH v1] Provide more contexts for -Warray-bounds warning messages

[David Malcolm]

On Tue, 2024-07-02 at 16:17 +, Qing Zhao wrote:
> due to code duplication from jump threading [PR109071]
> Control this with a new option -fdiagnostic-try-to-explain-harder.

The name -fdiagnostic-try-to-explain-harder seems a little too "cute"
to me, but I can't think of a better name.

Various comments inline below...  I'm sorry I didn't take a close look
at the copy history implementation; I'm hoping Richi will dig into that
part of the patch.

> 
> This patch has been tested with -fdiagnostic-try-to-expain-harder on
> by
> default to bootstrap gcc and regression testing on both x86 and
> aarch64,
> resolved all bootstrap issues and regression testing issues. 
> 
> I need some help in the following two items:
> 1. suggestions on better documentation wordings for the new option.
> 2. checking on the new data structures copy_history and the
>    memory management for it.
>    In the beginning, I tried to use the GGC for it.
>    but have met quite some issues (possibly because the gimple and
> the containing
>    basic block elimination), then I gave up the GGC scheme and
> instead
>    used the obstack and manually clean and deleted the obstack in the
> end of
>    the compilation.  
> 
> The following is more details on the patchs
> 
> Thanks a lot!
> 
> Qing
> 
> $ cat t.c
> extern void warn(void);
> static inline void assign(int val, int *regs, int *index)
> {
>   if (*index >= 4)
>     warn();
>   *regs = val;
> }
> struct nums {int vals[4];};
> 
> void sparx5_set (int *ptr, struct nums *sg, int index)
> {
>   int *val = &sg->vals[index];
> 
>   assign(0,    ptr, &index);
>   assign(*val, ptr, &index);
> }
> 
> $ gcc -Wall -O2  -c -o t.o t.c
> t.c: In function ‘sparx5_set’:
> t.c:12:23: warning: array subscript 4 is above array bounds of
> ‘int[4]’ [-Warray-bounds=]
>    12 |   int *val = &sg->vals[index];
>   |   ^~~
> t.c:8:18: note: while referencing ‘vals’
>     8 | struct nums {int vals[4];};
>   |  ^~~~
> 
> In the above, Although the warning is correct in theory, the warning
> message
> itself is confusing to the end-user since there is information that
> cannot
> be connected to the source code directly.
> 
> It will be a nice improvement to add more information in the warning
> message
> to report where such index value come from.
> 
> In order to achieve this, we add a new data structure copy_history to
> record
> the condition and the transformation that triggered the code
> duplication.
> Whenever there is a code duplication due to some specific
> transformations,
> such as jump threading, loop switching, etc, a copy_history structure
> is
> created and attached to the duplicated gimple statement.
> 
> During array out-of-bound checking or other warning checking, the
> copy_history
> that was attached to the gimple statement is used to form a sequence
> of
> diagnostic events that are added to the corresponding rich location
> to be used
> to report the warning message.
> 
> This behavior is controled by the new option -fdiagnostic-try-to-
> explain-harder
> which is off by default.
> 
> With this change, by adding -fdiagnostic-try-to-explain-harder,
> the warning message for the above testing case is now:
> 
> t.c: In function ‘sparx5_set’:
> t.c:12:23: warning: array subscript 4 is above array bounds of
> ‘int[4]’ [-Warray-bounds=]
>    12 |   int *val = &sg->vals[index];
>   |   ^~~
>   event 1
>     |
>     |    4 |   if (*index >= 4)
>     |  |  ^
>     |  |  |
>     |  |  (1) when the condition is evaluated to true
>     |
> t.c:8:18: note: while referencing ‘vals’
>     8 | struct nums {int vals[4];};
>   |  ^~~~

BTW I notice in the above example you have the extra vertical line
below "event 1" here:

  event 1
|
|4 |   if (*index >= 4)
|  |  ^
|  |  |
|  |  (1) when the condition is evaluated to true
|

whereas in the testcase it looks like you're expecting the simpler:

  event 1
   4 |   if (*index >= 4)
 |  ^
 |  |
 |  (1) when the condition is evaluated to true

which is due to r15-533-g3cd267446755ab, so I think the example text is
slightly outdated.

I wonder if the wording of the event could be improved to better
explain to the user what the optimizer is "thinking".

Perhaps it could be two events:

(1) when specializing the code for both branches...
(2) ...and considering the 'true' branch...

or something like that?  (I'm not sure)

Is there a more complicated testcase with multiple events?

Various other points about the path:

When the analyzer builds paths for its diagnostics, it adds a final
event that repeats the problem.  This is somewhat tautological, but
might make the path more readable - I'm not sure.  Consider:

t.c: In function ‘sparx5_set’:
t.c:12:23: warning: array subscript 4 is above array bounds of
‘int[4]’ [-Warray-bounds=]
   12 |   int *val = &sg->vals[in

Re: [PATCH] libgccjit: Add support for machine-dependent builtins

[David Malcolm]

On Thu, 2023-11-23 at 17:17 -0500, Antoni Boucher wrote:
> Hi.
> I did split the patch and sent one for the bfloat16 support and
> another
> one for the vector support.
> 
> Here's the updated patch for the machine-dependent builtins.
> 

Thanks for the patch; sorry about the long delay in reviewing it.

CCing Jan and Uros re the i386 part of that patch; for reference the
patch being discussed is here:
  https://gcc.gnu.org/pipermail/gcc-patches/2023-November/638027.html

> From e025f95f4790ae861e709caf23cbc0723c1a3804 Mon Sep 17 00:00:00 2001
> From: Antoni Boucher 
> Date: Mon, 23 Jan 2023 17:21:15 -0500
> Subject: [PATCH] libgccjit: Add support for machine-dependent builtins

[...snip...]

> diff --git a/gcc/config/i386/i386-builtins.cc 
> b/gcc/config/i386/i386-builtins.cc
> index 42fc3751676..5cc1d6f4d2e 100644
> --- a/gcc/config/i386/i386-builtins.cc
> +++ b/gcc/config/i386/i386-builtins.cc
> @@ -225,6 +225,22 @@ static GTY(()) tree ix86_builtins[(int) 
> IX86_BUILTIN_MAX];
>  
>  struct builtin_isa ix86_builtins_isa[(int) IX86_BUILTIN_MAX];
>  
> +static void
> +clear_builtin_types (void)
> +{
> +  for (int i = 0 ; i < IX86_BT_LAST_CPTR + 1 ; i++)
> +ix86_builtin_type_tab[i] = NULL;
> +
> +  for (int i = 0 ; i < IX86_BUILTIN_MAX ; i++)
> +  {
> +ix86_builtins[i] = NULL;
> +ix86_builtins_isa[i].set_and_not_built_p = true;
> +  }
> +
> +  for (int i = 0 ; i < IX86_BT_LAST_ALIAS + 1 ; i++)
> +ix86_builtin_func_type_tab[i] = NULL;
> +}
> +
>  tree get_ix86_builtin (enum ix86_builtins c)
>  {
>return ix86_builtins[c];
> @@ -1483,6 +1499,8 @@ ix86_init_builtins (void)
>  {
>tree ftype, decl;
>  
> +  clear_builtin_types ();
> +
>ix86_init_builtin_types ();
>  
>/* Builtins to get CPU type and features. */

Please can one of the i386 maintainers check this?
(CCing Jan and Uros: this is for the case where the compiler code runs
multiple times in-process due to being linked into libgccjit.so.  We
want to restore state within i386-builtins.cc to an initial state, and
ensure that no GC-managed objects persist from previous in-memory
compiles).

> diff --git a/gcc/jit/docs/topics/compatibility.rst
b/gcc/jit/docs/topics/compatibility.rst
> index ebede440ee4..764de23341e 100644
> --- a/gcc/jit/docs/topics/compatibility.rst
> +++ b/gcc/jit/docs/topics/compatibility.rst
> @@ -378,3 +378,12 @@ alignment of a variable:
>  
>  ``LIBGCCJIT_ABI_25`` covers the addition of
>  :func:`gcc_jit_type_get_restrict`
> +
> +.. _LIBGCCJIT_ABI_26:
> +
> +``LIBGCCJIT_ABI_26``
> +
> +
> +``LIBGCCJIT_ABI_26`` covers the addition of a function to get target 
> builtins:
> +
> +  * :func:`gcc_jit_context_get_target_builtin_function`
> diff --git a/gcc/jit/docs/topics/functions.rst 
> b/gcc/jit/docs/topics/functions.rst
> index cf5cb716daf..e9b77fdb892 100644
> --- a/gcc/jit/docs/topics/functions.rst
> +++ b/gcc/jit/docs/topics/functions.rst
> @@ -140,6 +140,25 @@ Functions
>uses such a parameter will lead to an error being emitted within
>the context.
>  
> +.. function::  gcc_jit_function *\
> +   gcc_jit_context_get_target_builtin_function (gcc_jit_context 
> *ctxt,\
> +const char *name)
> +
> +   Get the :type:`gcc_jit_function` for the built-in function with the
> +   given name.  For example:

Might be nice to add the "(sometimes called intrinsic functions)" text
you have in the header here.

[...snip]

> diff --git a/gcc/jit/dummy-frontend.cc b/gcc/jit/dummy-frontend.cc
> index a729086bafb..3ca9702d429 100644
> --- a/gcc/jit/dummy-frontend.cc
> +++ b/gcc/jit/dummy-frontend.cc

[...]

> @@ -29,8 +30,14 @@ along with GCC; see the file COPYING3.  If not see
>  #include "options.h"
>  #include "stringpool.h"
>  #include "attribs.h"
> +#include "jit-recording.h"
> +#include "print-tree.h"
>  
>  #include 
> +#include 
> +#include 
> +
> +using namespace gcc::jit;
>  
>  /* Attribute handling.  */
>  
> @@ -86,6 +93,11 @@ static const struct attribute_spec::exclusions 
> attr_const_pure_exclusions[] =
>ATTR_EXCL (NULL, false, false, false)
>  };
>  
> +hash_map target_builtins{};

I was wondering if this needs a GTY marker, but I don't think it does:
presumably it's only used within jit_langhook_parse_file where no GC
can happen - unless jit_langhook_write_globals makes use of it?

> +std::unordered_map
target_function_types
> +{};
> +recording::context target_builtins_ctxt{NULL};

Please add a comment to target_builtins_ctxt saying what it's for.  As
far as I can tell, it's for getting at recording::types from
tree_type_to_jit_type; we then use a new "copy" mechanism to copy
objects from target_builtins_ctxt for use with the real
recording::context.

This feels ugly, but maybe it's the only way to make it work.

Could tree_type_to_jit_type take a recording::context as a param?  The
only non-recursive uses of tree_type_to_jit_type seem to be in
jit_langhook_builtin_funct

Re: [PATCH] libgccjit: Fix get_size of size_t

[David Malcolm]

On Wed, 2024-02-21 at 14:16 -0500, Antoni Boucher wrote:
> On Thu, 2023-12-07 at 19:57 -0500, David Malcolm wrote:
> > On Thu, 2023-12-07 at 17:26 -0500, Antoni Boucher wrote:
> > > Hi.
> > > This patch fixes getting the size of size_t (bug 112910).
> > > 
> > > There's one issue with this patch: like every other feature that
> > > checks
> > > for target-specific stuff, it requires a compilation before
> > > actually
> > > fetching the size of the type.
> > > Which means that getting the size before a compilation might be
> > > wrong
> > > (and I actually believe is wrong on x86-64).
> > > 
> > > I was wondering if we should always implicitely do the first
> > > compilation to gather the correct info: this would fix this issue
> > > and
> > > all the others that we have due to that.
> > > I'm not sure what would be the performance implication.
> > 
> > Maybe introduce a new class target_info which contains all the
> > information we might want to find via a compilation, and have the
> > top-
> > level recording::context have a pointer to it, which starts as
> > nullptr,
> > but can be populated on-demand the first time something needs it?
> 
> That would mean that we'll need to populate it for every top-level
> context, right? Would the idea be that we should then use child
> contexts to have the proper information filled?
> If so, how is this different than just compiling two contexts like
> what
> I currently do?
> This would also mean that we'll do an implicit compilation whenever
> we
> use an API that needs this info, right? Wouldn't that be unexpected?

I was thinking a compilation with an empty playback::context to lazily
capture the target data.

My hope was that this would make things easier for users.  But you're
the one using this API, so if you're more comfortable with the explicit
initial compilation approach, let's go with that.

If so, this is OK for trunk - but we might want to add a note to the
documentation about the double-compilation workaround.

Dave


> 
> Thanks for the idea.
> 
> > 
> > > 
> > > Another solution that I have been thinking about for a while now
> > > would
> > > be to have another frontend libgccaot (I don't like that name),
> > > which
> > > is like libgccjit but removes the JIT part so that we get access
> > > to
> > > the
> > > target stuff directly and would remove the need for having a
> > > seperation
> > > between recording and playback as far as I understand.
> > > That's a long-term solution, but I wanted to share the idea now
> > > and
> > > gather your thoughts on that.
> > 
> > FWIW the initial version of libgccjit didn't have a split between
> > recording and playback; instead the client code had to pass in a
> > callback to call into the various API functions (creating tree
> > nodes).
> > See:
> > https://gcc.gnu.org/legacy-ml/gcc-patches/2013-10/msg00228.html
> > 
> > Dave
> > 
> 

Re: Frontend access to target features (was Re: [PATCH] libgccjit: Add ability to get CPU features)

[David Malcolm]

On Sun, 2024-03-10 at 12:05 +0100, Iain Buclaw wrote:
> Excerpts from David Malcolm's message of März 5, 2024 4:09 pm:
> > On Thu, 2023-11-09 at 19:33 -0500, Antoni Boucher wrote:
> > > Hi.
> > > See answers below.
> > > 
> > > On Thu, 2023-11-09 at 18:04 -0500, David Malcolm wrote:
> > > > On Thu, 2023-11-09 at 17:27 -0500, Antoni Boucher wrote:
> > > > > Hi.
> > > > > This patch adds support for getting the CPU features in
> > > > > libgccjit
> > > > > (bug
> > > > > 112466)
> > > > > 
> > > > > There's a TODO in the test:
> > > > > I'm not sure how to test that gcc_jit_target_info_arch
> > > > > returns
> > > > > the
> > > > > correct value since it is dependant on the CPU.
> > > > > Any idea on how to improve this?
> > > > > 
> > > > > Also, I created a CStringHash to be able to have a
> > > > > std::unordered_set. Is there any built-in way
> > > > > of
> > > > > doing
> > > > > this?
> > > > 
> > > > Thanks for the patch.
> > > > 
> > > > Some high-level questions:
> > > > 
> > > > Is this specifically about detecting capabilities of the host
> > > > that
> > > > libgccjit is currently running on? or how the target was
> > > > configured
> > > > when libgccjit was built?
> > > 
> > > I'm less sure about this part. I'll need to do more tests.
> > > 
> > > > 
> > > > One of the benefits of libgccjit is that, in theory, we support
> > > > all
> > > > of
> > > > the targets that GCC already supports.  Does this patch change
> > > > that,
> > > > or
> > > > is this more about giving client code the ability to determine
> > > > capabilities of the specific host being compiled for?
> > > 
> > > This should not change that. If it does, this is a bug.
> > > 
> > > > 
> > > > I'm nervous about having per-target jit code.  Presumably
> > > > there's a
> > > > reason that we can't reuse existing target logic here - can you
> > > > please
> > > > describe what the problem is.  I see that the ChangeLog has:
> > > > 
> > > > > * config/i386/i386-jit.cc: New file.
> > > > 
> > > > where i386-jit.cc has almost 200 lines of nontrivial code. 
> > > > Where
> > > > did
> > > > this come from?  Did you base it on existing code in our source
> > > > tree,
> > > > making modifications to fit the new internal API, or did you
> > > > write
> > > > it
> > > > from scratch?  In either case, how onerous would this be for
> > > > other
> > > > targets?
> > > 
> > > This was mostly copied from the same code done for the Rust and D
> > > frontends.
> > > See this commit and the following:
> > > https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=b1c06fd9723453dd2b2ec306684cb806dc2b4fbb
> > > The equivalent to i386-jit.cc is there:
> > > https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=22e3557e2d52f129f2bbfdc98688b945dba28dc9
> > 
> > [CCing Iain and Arthur re those patches; for reference, the patch
> > being
> > discussed is attached to :
> > https://gcc.gnu.org/pipermail/jit/2024q1/001792.html ]
> > 
> > One of my concerns about this patch is that we seem to be gaining
> > code
> > that's per-(frontend x config) which seems to be copied and pasted
> > with
> > a search and replace, which could lead to an M*N explosion.
> > 
> 
> That's certainly the case with the configure/make rules. Itself I
> think
> is copied originally from the {cpu_type}-protos.h machinery.
> 
> It might be worth pointing out that the c-family of front-ends don't
> have separate headers because their per-target macros are defined in
> {cpu_type}.h directly - for better or worse.
> 
> > Is there any real difference between the per-config code for the
> > different frontends, or should there be a general "enumerate all
> > features of the target" hook that's independent of the frontend?
> > (but
> > perhaps calls into it).
> > 
> 
> As far as I understand, the configure parts should all be identical
> between tm_p, tm_d, tm_rust, ..., so would benefit f

Re: [PATCH] libgccjit: Add support for the type bfloat16

[David Malcolm]

On Wed, 2024-02-21 at 10:56 -0500, Antoni Boucher wrote:
> Thanks for the review.
> Here's the updated patch.

Thanks for the update patch; sorry for the delay in reviewing.

The updated patch looks good for trunk.

Dave

> 
> On Fri, 2023-12-01 at 12:45 -0500, David Malcolm wrote:
> > On Thu, 2023-11-16 at 17:20 -0500, Antoni Boucher wrote:
> > > I forgot to attach the patch.
> > > 
> > > On Thu, 2023-11-16 at 17:19 -0500, Antoni Boucher wrote:
> > > > Hi.
> > > > This patch adds the support for the type bfloat16 (bug 112574).
> > > > 
> > > > This was asked to be splitted from a another patch sent here:
> > > > https://gcc.gnu.org/pipermail/jit/2023q1/001607.html
> > > > 
> > > > Thanks for the review.
> > > 
> > 
> > Thanks for the patch.
> > 
> > > diff --git a/gcc/jit/jit-playback.cc b/gcc/jit/jit-playback.cc
> > > index 18cc4da25b8..7e1c97a4638 100644
> > > --- a/gcc/jit/jit-playback.cc
> > > +++ b/gcc/jit/jit-playback.cc
> > > @@ -280,6 +280,8 @@ get_tree_node_for_type (enum gcc_jit_types
> > > type_)
> > >  
> > >  case GCC_JIT_TYPE_FLOAT:
> > >    return float_type_node;
> > > +    case GCC_JIT_TYPE_BFLOAT16:
> > > +  return bfloat16_type_node;
> > 
> > The code to create bfloat16_type_node (in build_common_tree_nodes)
> > is
> > guarded by #ifdef HAVE_BFmode, so we should probably have a test
> > for
> > this in case GCC_JIT_TYPE_BFLOAT16 to at least add an error message
> > when it's NULL_TREE, rather than silently returning NULL_TREE and
> > crashing.
> > 
> > [...]
> > 
> > > diff --git a/gcc/testsuite/jit.dg/test-bfloat16.c
> > > b/gcc/testsuite/jit.dg/test-bfloat16.c
> > > new file mode 100644
> > > index 000..6aed3920351
> > > --- /dev/null
> > > +++ b/gcc/testsuite/jit.dg/test-bfloat16.c
> > > @@ -0,0 +1,37 @@
> > > +/* { dg-do compile { target x86_64-*-* } } */
> > > +
> > > +#include 
> > > +#include 
> > > +
> > > +#include "libgccjit.h"
> > > +
> > > +/* We don't want set_options() in harness.h to set -O3 so our
> > > little local
> > > +   is optimized away. */
> > > +#define TEST_ESCHEWS_SET_OPTIONS
> > > +static void set_options (gcc_jit_context *ctxt, const char
> > > *argv0)
> > > +{
> > > +}
> > 
> > 
> > Please add a comment to all-non-failing-tests.h noting the
> > exclusion
> > of
> > this test case from the array.
> > 
> > [...]
> > 
> > > diff --git a/gcc/testsuite/jit.dg/test-types.c
> > > b/gcc/testsuite/jit.dg/test-types.c
> > > index a01944e35fa..9e7c4f3e046 100644
> > > --- a/gcc/testsuite/jit.dg/test-types.c
> > > +++ b/gcc/testsuite/jit.dg/test-types.c
> > > @@ -1,3 +1,4 @@
> > > +#include 
> > >  #include 
> > >  #include 
> > >  #include 
> > > @@ -492,4 +493,5 @@ verify_code (gcc_jit_context *ctxt,
> > > gcc_jit_result *result)
> > >  
> > >    CHECK_VALUE (gcc_jit_type_get_size (gcc_jit_context_get_type
> > > (ctxt, GCC_JIT_TYPE_FLOAT)), sizeof (float));
> > >    CHECK_VALUE (gcc_jit_type_get_size (gcc_jit_context_get_type
> > > (ctxt, GCC_JIT_TYPE_DOUBLE)), sizeof (double));
> > > +  CHECK_VALUE (gcc_jit_type_get_size (gcc_jit_context_get_type
> > > (ctxt, GCC_JIT_TYPE_BFLOAT16)), sizeof (__bfloat16));
> > 
> > 
> > This is only going to work on targets which #ifdef HAVE_BFmode, so
> > this
> > CHECK_VALUE needs to be conditionalized somehow, to avoid having
> > this,
> > test-combination, and test-threads from bailing out early on
> > targets
> > without BFmode.
> > 
> > Dave
> > 
>