Hello Akim, I attached the patches and took into account the comment underlining (I did not notice the style until you pointed it out, sorry about that). Thank you for your patience and help!
Adela În sâm., 30 ian. 2021 la 09:49, Akim Demaille <[email protected]> a scris: > Hi Adela, > > > Le 29 janv. 2021 à 15:43, Adela Vais <[email protected]> a écrit : > > > > Hello Akim, > > > > I made the token constructors work. Doing this, I realized that the > Symbol > > constructors needed some changes, which I did in D. > > After our discussion from Tuesday, I do agree that it's not the best > > approach. > > The code is quite unnecessarily complex, so I will rewrite the > constructors > > to be generated with M4, to be easier to maintain. > > > > Now, the problem is that token constructors need the new version of the > > Symbol constructors to be working correctly. > > So it's your choice if you decide to install it like this or wait until I > > rewrite the constructors. > > I'm installing the first of your patches, which fixes style issues. > Please, be sure to make commit messages that fit with 76 columns. > And when it applies, describe the changes in the files within the > file section. So I rewrote your first commit message as follows: > > Author: Adela Vais <[email protected]> > Date: Mon Jan 25 17:59:06 2021 +0200 > > d: tests: various style fixes > > * tests/calc.at, tests/scanner.at: Here. > (yylex): Report values directly, without storing them to the union > first. > > instead of > > d: tests: various style fixes > > Now the tests report values in yylex() directly, without storing them > to the union first. > > * tests/calc.at, tests/scanner.at: Here. > > > Let's wait for the version with m4, to keep a simple history in > the repository. No hurry, 3.8 is not ready... > > But please, even though it's unpleasant and LITS, keep the comments > correctly underlined: > > > +# _b4_constructor_maker_define_types(SYMBOL-NUM) > +# ---------------------------------- > +# Declare the case branches in the Symbol constructor for SYMBOL-NUM > values. > > > +# AT_YYLEX_RETURN_VAL > +# --------------- > +m4_define([AT_YYLEX_RETURN_VAL], > > etc. > > Cheers!
From a7e0a8c5344ffae02a029e8aec2b836eeee1bed0 Mon Sep 17 00:00:00 2001 From: Adela Vais <[email protected]> Date: Mon, 25 Jan 2021 22:53:30 +0200 Subject: [PATCH for Dlang support 3/3] d: add token constructors support The user can return from yylex() by calling the Symbol method of the same name as the TokenKind reported, and adding the parameters for value and location if necessary. These methods generate compile-time errors if the parameters are not correlated. Token constructors work with both %union and api.value.type union. * data/skeletons/d.m4: Here. * tests/calc.at: Test it. --- data/skeletons/d.m4 | 27 +++++++++++++++++++++++++++ tests/calc.at | 42 ++++++++++++++++++++++++++---------------- 2 files changed, 53 insertions(+), 16 deletions(-) diff --git a/data/skeletons/d.m4 b/data/skeletons/d.m4 index d2fa6e29..0aae111e 100644 --- a/data/skeletons/d.m4 +++ b/data/skeletons/d.m4 @@ -208,6 +208,32 @@ b4_symbol_foreach([b4_token_enum])dnl m4_define([b4_symbol_translate], [[_($1)]]) + +# _b4_token_constructor_define +# ---------------------------- +# Define make_symbol for a value type. +m4_define([_b4_token_constructor_define], +[b4_token_visible_if([$1], +[[ + static auto ]b4_symbol([$1], [id])[(]b4_symbol_if([$1], [has_type], +[b4_union_if([b4_symbol([$1], [type]], +[[typeof(YYSemanticType.]b4_symbol([$1], [type])[]])) [val]])dnl +[]b4_locations_if([b4_symbol_if([$1], [has_type], [[, ]])[Location l]])[) + { + return Symbol(TokenKind.]b4_symbol([$1], [id])[]b4_symbol_if([$1], [has_type], + [[, val]])[]b4_locations_if([[, l]])[); + }]])]) + +# b4_token_constructor_define +# --------------------------- +# Define the overloaded versions of make_symbol for all the value types. +m4_define([b4_token_constructor_define], +[[ + /* Implementation of token constructors for each symbol type visible to + * the user. The code generates static methods that have the same names + * as the TokenKinds. + */]b4_symbol_foreach([_b4_token_constructor_define])]) + ## -------------- ## ## Symbol kinds. ## ## -------------- ## @@ -593,5 +619,6 @@ m4_define([b4_symbol_type_define], SymbolKind token() { return kind; } Value value() { return value_; }]b4_locations_if([[ Location location() { return location_; }]])[ +]b4_token_ctor_if([b4_token_constructor_define])[ } ]]) diff --git a/tests/calc.at b/tests/calc.at index 042cac7b..7475535d 100644 --- a/tests/calc.at +++ b/tests/calc.at @@ -519,6 +519,16 @@ m4_copy([_AT_DATA_CALC_Y(c)], [_AT_DATA_CALC_Y(c++)]) ## Calc in D. ## ## ----------- ## +# AT_YYLEX_RETURN_VAL +# ------------------- +# Produce the return value for yylex(). +m4_define([AT_YYLEX_RETURN_VAL], +[return dnl +AT_TOKEN_CTOR_IF( + [[Symbol.]AT_TOKEN_PREFIX[$1](m4_ifval([$2], [$2])[]AT_LOCATION_IF([m4_ifval([$2], [, ])[location]])[);]], + [[Symbol(TokenKind.]AT_TOKEN_PREFIX[$1]m4_ifval([$2], [, $2])[]AT_LOCATION_IF([[, location]])[);]])] +) + # AT_CALC_MAIN(d). m4_define([AT_CALC_MAIN(d)], [[int main (string[] args) @@ -595,11 +605,10 @@ if (isInputRange!R && is (ElementType!R : dchar)) // EOF. if (input.empty) - return Symbol(TokenKind.]AT_TOKEN_PREFIX[EOF]AT_LOCATION_IF([[, location]])[); - + ]AT_YYLEX_RETURN_VAL([EOF])[ // Numbers. if (input.front.isNumber) - return Symbol(TokenKind.]AT_TOKEN_PREFIX[NUM, parseInt]AT_LOCATION_IF([[, location]])[); + ]AT_YYLEX_RETURN_VAL([NUM], [parseInt])[ // Individual characters auto c = input.front;]AT_LOCATION_IF([[ @@ -616,22 +625,22 @@ if (isInputRange!R && is (ElementType!R : dchar)) if (c == '#') { stderr.writeln (]AT_LOCATION_IF([location, ": ", ])["syntax error: invalid character: '#'"); - return Symbol(TokenKind.]AT_TOKEN_PREFIX[YYerror]AT_LOCATION_IF([[, location]])[); + ]AT_YYLEX_RETURN_VAL([YYerror])[ } switch (c) { - case '+': return Symbol(TokenKind.]AT_TOKEN_PREFIX[PLUS]AT_LOCATION_IF([[, location]])[); - case '-': return Symbol(TokenKind.]AT_TOKEN_PREFIX[MINUS]AT_LOCATION_IF([[, location]])[); - case '*': return Symbol(TokenKind.]AT_TOKEN_PREFIX[STAR]AT_LOCATION_IF([[, location]])[); - case '/': return Symbol(TokenKind.]AT_TOKEN_PREFIX[SLASH]AT_LOCATION_IF([[, location]])[); - case '(': return Symbol(TokenKind.]AT_TOKEN_PREFIX[LPAR]AT_LOCATION_IF([[, location]])[); - case ')': return Symbol(TokenKind.]AT_TOKEN_PREFIX[RPAR]AT_LOCATION_IF([[, location]])[); - case '\n': return Symbol(TokenKind.]AT_TOKEN_PREFIX[EOL]AT_LOCATION_IF([[, location]])[); - case '=': return Symbol(TokenKind.]AT_TOKEN_PREFIX[EQUAL]AT_LOCATION_IF([[, location]])[); - case '^': return Symbol(TokenKind.]AT_TOKEN_PREFIX[POW]AT_LOCATION_IF([[, location]])[); - case '!': return Symbol(TokenKind.]AT_TOKEN_PREFIX[NOT]AT_LOCATION_IF([[, location]])[); - default: return Symbol(TokenKind.]AT_TOKEN_PREFIX[YYUNDEF]AT_LOCATION_IF([[, location]])[); + case '+': ]AT_YYLEX_RETURN_VAL([PLUS])[ + case '-': ]AT_YYLEX_RETURN_VAL([MINUS])[ + case '*': ]AT_YYLEX_RETURN_VAL([STAR])[ + case '/': ]AT_YYLEX_RETURN_VAL([SLASH])[ + case '(': ]AT_YYLEX_RETURN_VAL([LPAR])[ + case ')': ]AT_YYLEX_RETURN_VAL([RPAR])[ + case '\n': ]AT_YYLEX_RETURN_VAL([EOL])[ + case '=': ]AT_YYLEX_RETURN_VAL([EQUAL])[ + case '^': ]AT_YYLEX_RETURN_VAL([POW])[ + case '!': ]AT_YYLEX_RETURN_VAL([NOT])[ + default: ]AT_YYLEX_RETURN_VAL([YYUNDEF])[ } } } @@ -1499,7 +1508,8 @@ AT_CHECK_CALC_LALR1_D([%locations %define parse.lac full %define parse.error det #AT_CHECK_CALC_LALR1_D([%locations %define parse.error detailed %debug %verbose %parse-param {semantic_value *result}{int *count}{int *nerrs}]) #AT_CHECK_CALC_LALR1_D([%locations %define parse.error detailed %debug %define api.prefix {calc} %verbose %parse-param {semantic_value *result}{int *count}{int *nerrs}]) -AT_CHECK_CALC_LALR1_D([%define parse.error custom %define api.value.type union]) +AT_CHECK_CALC_LALR1_D([%define api.token.constructor %locations %define parse.error custom %define api.value.type union]) +AT_CHECK_CALC_LALR1_D([%define api.token.constructor %locations %define parse.error detailed]) # ----------------------- # # LALR1 Java Calculator. # -- 2.25.1
From 079ef175ece0f53ed11fd9484cff1a6a4122281a Mon Sep 17 00:00:00 2001 From: Adela Vais <[email protected]> Date: Sat, 13 Mar 2021 22:15:05 +0200 Subject: [PATCH for Dlang support 1/3] d: rewrite Symbol's constructors in M4 The D code was becoming too complex. M4 is easier to maintain in the long run. * data/skeletons/d.m4: Here. --- data/skeletons/d.m4 | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/data/skeletons/d.m4 b/data/skeletons/d.m4 index 83eb0517..e4e60c8c 100644 --- a/data/skeletons/d.m4 +++ b/data/skeletons/d.m4 @@ -448,6 +448,22 @@ alias Position = ]b4_position_type[;]])[ ]]) +# b4_basic_symbol_constructor_define +# ---------------------------------- +# Create Symbol struct constructors for all the visible types. +m4_define([b4_basic_symbol_constructor_define], +[b4_token_visible_if([$1], +[ this(TokenKind token]b4_symbol_if([$1], [has_type], +[[, typeof(YYSemanticType.]b4_symbol([$1], [type])dnl +[) val]])[]b4_locations_if([[, Location loc]])[) + { + kind = yytranslate_(token);]b4_symbol_if([$1], [has_type], [[ + value_.]b4_symbol([$1], [type])[ = val;]])[]b4_locations_if([ + location_ = loc;])[ + } +])]) + + # b4_symbol_type_define # --------------------- # Define symbol_type, the external type for symbols used for symbol @@ -462,20 +478,8 @@ m4_define([b4_symbol_type_define], private SymbolKind kind; private Value value_;]b4_locations_if([[ private Location location_;]])[ - this(TokenKind token]b4_locations_if([[, Location loc]])[) - { - kind = yytranslate_(token);]b4_locations_if([ - location_ = loc;])[ - } - static foreach (member; __traits(allMembers, YYSemanticType)) - { - this(TokenKind token, typeof(mixin("YYSemanticType." ~ member)) val]b4_locations_if([[, Location loc]])[) - { - kind = yytranslate_(token); - mixin("value_." ~ member ~ " = val;");]b4_locations_if([ - location_ = loc;])[ - } - } + +]b4_type_foreach([b4_basic_symbol_constructor_define])[ SymbolKind token() { return kind; } Value value() { return value_; }]b4_locations_if([[ Location location() { return location_; }]])[ -- 2.25.1
From dcf910f30e95fb3088ed2284c9528dfe5aeb4ab6 Mon Sep 17 00:00:00 2001 From: Akim Demaille <[email protected]> Date: Thu, 21 Jan 2021 21:45:45 +0100 Subject: [PATCH for Dlang support 2/3] d: add api.value.type union support The union of the values is handled by the backend. In D, unions can hold classes, structs, etc., so this is more similar to the C++ api.value.type variant. * data/skeletons/d.m4, data/skeletons/lalr1.d: Here. * tests/calc.at, tests/local.at: Test it. --- data/skeletons/d.m4 | 120 +++++++++++++++++++++++++++++++++++++++-- data/skeletons/lalr1.d | 2 +- tests/calc.at | 11 ++-- tests/local.at | 2 + 4 files changed, 124 insertions(+), 11 deletions(-) diff --git a/data/skeletons/d.m4 b/data/skeletons/d.m4 index e4e60c8c..d2fa6e29 100644 --- a/data/skeletons/d.m4 +++ b/data/skeletons/d.m4 @@ -275,7 +275,7 @@ m4_define([b4_declare_symbol_enum], ]]) -# b4-case(ID, CODE, [COMMENTS]) +# b4_case(ID, CODE, [COMMENTS]) # ----------------------------- m4_define([b4_case], [ case $1:m4_ifval([$3], [ b4_comment([$3])]) $2 @@ -302,6 +302,109 @@ m4_define([b4_location_type], b4_percent_define_ifdef([[location_type]],[b4_perc m4_define([b4_position_type], b4_percent_define_ifdef([[position_type]],[b4_percent_define_get([[position_type]])],[YYPosition])) +## ---------------- ## +## api.value.type. ## +## ---------------- ## + + +# ---------------------- # +# api.value.type=union. # +# ---------------------- # + +# b4_symbol_type_register(SYMBOL-NUM) +# ----------------------------------- +# Symbol SYMBOL-NUM has a type (for union) instead of a type-tag. +# Extend the definition of %union's body (b4_union_members) with a +# field of that type, and extend the symbol's "type" field to point to +# the field name, instead of the type name. +m4_define([b4_symbol_type_register], +[m4_define([b4_symbol($1, type_tag)], + [b4_symbol_if([$1], [has_id], + [b4_symbol([$1], [id])], + [yykind_[]b4_symbol([$1], [number])])])dnl +m4_append([b4_union_members], +m4_expand([m4_format([ %-40s %s], + m4_expand([b4_symbol([$1], [type]) b4_symbol([$1], [type_tag]);]), + [b4_symbol_tag_comment([$1])])])) +]) + + +# b4_type_define_tag(SYMBOL1-NUM, ...) +# ------------------------------------ +# For the batch of symbols SYMBOL1-NUM... (which all have the same +# type), enhance the %union definition for each of them, and set +# there "type" field to the field tag name, instead of the type name. +m4_define([b4_type_define_tag], +[b4_symbol_if([$1], [has_type], + [m4_map([b4_symbol_type_register], [$@])]) +]) + + +# b4_symbol_value_union(VAL, SYMBOL-NUM, [TYPE]) +# ---------------------------------------------- +# Same of b4_symbol_value, but when api.value.type=union. +m4_define([b4_symbol_value_union], +[m4_ifval([$3], + [(*($3*)(&$1))], + [m4_ifval([$2], + [b4_symbol_if([$2], [has_type], + [($1.b4_symbol([$2], [type_tag]))], + [$1])], + [$1])])]) + + +# b4_value_type_setup_union +# ------------------------- +# Setup support for api.value.type=union. Symbols are defined with a +# type instead of a union member name: build the corresponding union, +# and give the symbols their tag. +m4_define([b4_value_type_setup_union], +[m4_define([b4_union_members]) +b4_type_foreach([b4_type_define_tag]) +m4_copy_force([b4_symbol_value_union], [b4_symbol_value]) +]) + + +# _b4_value_type_setup_keyword +# ---------------------------- +# api.value.type is defined with a keyword/string syntax. Check if +# that is properly defined, and prepare its use. +m4_define([_b4_value_type_setup_keyword], +[b4_percent_define_check_values([[[[api.value.type]], + [[none]], + [[union]], + [[union-directive]], + [[yystype]]]])dnl +m4_case(b4_percent_define_get([[api.value.type]]), + [union], [b4_value_type_setup_union])]) + + +# b4_value_type_setup +# ------------------- +# Check if api.value.type is properly defined, and possibly prepare +# its use. +b4_define_silent([b4_value_type_setup], +[ +# Define default value. +b4_percent_define_ifdef([[api.value.type]], [], +[# %union => api.value.type=union-directive +m4_ifdef([b4_union_members], +[m4_define([b4_percent_define_kind(api.value.type)], [keyword]) +m4_define([b4_percent_define(api.value.type)], [union-directive])], +[# no tag seen => api.value.type={int} +m4_if(b4_tag_seen_flag, 0, +[m4_define([b4_percent_define_kind(api.value.type)], [code]) +m4_define([b4_percent_define(api.value.type)], [int])], +[# otherwise api.value.type=yystype +m4_define([b4_percent_define_kind(api.value.type)], [keyword]) +m4_define([b4_percent_define(api.value.type)], [yystype])])])]) + +# Set up. +m4_bmatch(b4_percent_define_get_kind([[api.value.type]]), + [keyword], [_b4_value_type_setup_keyword]) +]) + + ## ----------------- ## ## Semantic Values. ## ## ----------------- ## @@ -454,11 +557,18 @@ alias Position = ]b4_position_type[;]])[ m4_define([b4_basic_symbol_constructor_define], [b4_token_visible_if([$1], [ this(TokenKind token]b4_symbol_if([$1], [has_type], -[[, typeof(YYSemanticType.]b4_symbol([$1], [type])dnl -[) val]])[]b4_locations_if([[, Location loc]])[) +[[, ]b4_union_if([], [[typeof(YYSemanticType.]])b4_symbol([$1], [type])dnl +[]b4_union_if([], [[) ]])[ val]])[]b4_locations_if([[, Location loc]])[) { - kind = yytranslate_(token);]b4_symbol_if([$1], [has_type], [[ - value_.]b4_symbol([$1], [type])[ = val;]])[]b4_locations_if([ + kind = yytranslate_(token);]b4_union_if([b4_symbol_if([$1], [has_type], [[ + static foreach (member; __traits(allMembers, YYSemanticType)) + { + static if (is(typeof(mixin("value_." ~ member)) == ]b4_symbol([$1], [type])[)) + { + mixin("value_." ~ member ~ " = val;"); + } + }]])], [b4_symbol_if([$1], [has_type], [[ + value_.]b4_symbol([$1], [type])[ = val;]])])[]b4_locations_if([ location_ = loc;])[ } ])]) diff --git a/data/skeletons/lalr1.d b/data/skeletons/lalr1.d index 4b287866..96eb259d 100644 --- a/data/skeletons/lalr1.d +++ b/data/skeletons/lalr1.d @@ -206,7 +206,7 @@ public struct ]b4_location_type[ private immutable bool yy_location_is_class = false; -]])])m4_ifdef([b4_user_union_members], [private union YYSemanticType +]])])[]b4_value_type_setup[]m4_ifdef([b4_user_union_members], [private union YYSemanticType { b4_user_union_members };], diff --git a/tests/calc.at b/tests/calc.at index 0e76c282..042cac7b 100644 --- a/tests/calc.at +++ b/tests/calc.at @@ -645,11 +645,11 @@ m4_define([_AT_DATA_CALC_Y(d)], alias semantic_value = int; } /* Exercise %union. */ -%union +]AT_UNION_IF([[]], [[%union { semantic_value ival; -}; -%printer { yyo.write($$); } <ival>; +};]])[ +%printer { yyo.write($$); } <]AT_UNION_IF([[int]], [[ival]])[>; %code { ]AT_TOKEN_TRANSLATE_IF([[ @@ -670,8 +670,8 @@ m4_define([_AT_DATA_CALC_Y(d)], /* Bison Declarations */ %token EOF 0 ]AT_TOKEN_TRANSLATE_IF([_("end of file")], ["end of input"])[ -%token <ival> NUM "number" -%type <ival> exp +%token <]AT_UNION_IF([[int]], [[ival]])[> NUM "number" +%type <]AT_UNION_IF([[int]], [[ival]])[> exp %token EQUAL "=" MINUS "-" @@ -1499,6 +1499,7 @@ AT_CHECK_CALC_LALR1_D([%locations %define parse.lac full %define parse.error det #AT_CHECK_CALC_LALR1_D([%locations %define parse.error detailed %debug %verbose %parse-param {semantic_value *result}{int *count}{int *nerrs}]) #AT_CHECK_CALC_LALR1_D([%locations %define parse.error detailed %debug %define api.prefix {calc} %verbose %parse-param {semantic_value *result}{int *count}{int *nerrs}]) +AT_CHECK_CALC_LALR1_D([%define parse.error custom %define api.value.type union]) # ----------------------- # # LALR1 Java Calculator. # diff --git a/tests/local.at b/tests/local.at index 8411bfa3..f8836282 100644 --- a/tests/local.at +++ b/tests/local.at @@ -319,6 +319,8 @@ m4_pushdef([AT_UNION_IF], [m4_bmatch([$3], [%define api\.value\.type union], [$1], [$2])]) m4_pushdef([AT_VARIANT_IF], [m4_bmatch([$3], [%define api\.value\.type variant], [$1], [$2])]) +m4_pushdef([AT_UNION_IF], +[m4_bmatch([$3], [%define api\.value\.type union], [$1], [$2])]) m4_pushdef([AT_API_prefix], [m4_bmatch([$3], [%define api\.prefix {.*}], [m4_bregexp([$3], [%define api\.prefix {\([^\}]*\)}], [\1])], -- 2.25.1
