Again, for the purposes of the code examples above, assume: {foo, bar} = {1, 2}
Additionally, - Lines beginning with # == indicate what the compiler expands an expression to. - Lines beginning with # => represent the result of evaluating that expression. - Lines beginning with *# !> * represent an exception. On Wednesday, June 28, 2023 at 9:30:41 PM UTC-5 Christopher Keele wrote: > I've figured out the tokenizer enough to prototype this as new operator; > working title the "tagged variable literal" operator (not in love with that > name). I'm using a dollar sign ($) to represent it. > > It has the same issues as before, as I've ungracefully wedged it between > the capture operator and other precedences, but now is logically separated > from the capture operator. Weird stuff still happens without wrapping it in > parens in certain contexts, for example; but I think it's enough to > continue discussion around this proposal if we want to refocus it around a > new operator. > > I'm happy to refine the branch further and work on a PR, but would need > much guidance, and so would rather leave it as is for now without more > feedback on the proposal and related blessings, as I would need more > core-team support to implement it than I did defguard. Still sounds really > fun to do. > > The source code for this new fork of Elixir is available here > <https://github.com/elixir-lang/elixir/compare/main...christhekeele:elixir:tagged-variable-literals> > for > experimentation. For convenience, here are the examples in this proposal > reworked to use a dedicated $ operator for compile-time tagged variable > literals. They all work in iex on my fork, although many obvious usages of > it do not without more work: > > *Bare Tagged Variable Literals* > > $:foo > # == *{:foo, foo}* > # => {:foo, 1} > $"foo" > # == *{"foo", foo}* > # => {"foo", 1} > > If bare usage is supported, this expansion would work as expected in match > and guard contexts as well, since it expands before variable references are > resolved: > > {:foo, baz} = $:foo > *# == {:foo, baz} = {:foo, foo}* > # => {:foo, 1} > baz > # => 1 > > *Tagged Variable Literals in **Lists* > > Since tagged variable expressions are allowed in lists, this can be used > to construct Keyword lists from the local variable scope elegantly: > > list = [$:foo, $:bar] > # == *list = [{:foo, foo}, {:bar, bar}]* > # => [foo: 1, bar: 2] > > This would work with other list operators like *|*: > > baz = 3 > list = [$:baz | list] > # == *list = [**{:baz, baz} **| **list**]* > # => [baz: 3, foo: 1, bar: 2] > > And list destructuring: > > {foo, bar, baz} = {nil, nil, nil} > [$:baz, $:foo, $:bar] = list > *# == [{:baz, baz}, {:foo, foo}, {:bar, bar}] = list* > # => [baz: 3, foo: 1, bar: 2] > {foo, bar, baz} > # => {1, 2, 3} > > *Tagged Variable Literals in **Maps* > > With a small change to the parser, > <https://github.com/elixir-lang/elixir/commit/119bd6da351e8fe2ab94e86a8456ffc521ce865d#diff-7e4167a9de48e2dcae64fae18a5b2ddad1d4aeff8f2dde274eb6f127ef65ac11R615> > we > can allow this expression inside map literals. Because this expression > individually gets expanded into a tagged-tuple before the map associations > list as a whole are processed, it allow this syntax to work in all existing > map/struct constructs, like map construction: > > map = %{$:foo, $"bar"} > *# == %{:foo => foo, "bar" => bar}* > # => %{:foo => 1, "bar" => 2} > > Map updates: > > foo = 3 > map = %{map | $:foo} > *# == %{map | :foo => foo}* > # => %{:foo => 3, "bar" => 2} > > And map destructuring: > > {foo, bar} = {nil, nil} > %{$:foo, &"bar"} = map > *# == %{:foo => foo, "bar" => bar} = map* > # => %{:foo => 3, "bar" => 2} > {foo, bar} > # => {3, 2} > > On Wednesday, June 28, 2023 at 8:36:15 PM UTC-5 Paul Schoenfelder wrote: > >> I do think there is value in proposing the "tagged variable captures" >> idea separately, but at the same time, your solution for field punning is >> part of the value proposition there. That said, as you've already noted, it >> is very easy for the conversation to get bogged down when more than one >> thing is being discussed at a time. >> >> This is a very salient point. How do you feel about introducing a new >> operator for this sugar, such as $:foo? >> >> >> The first thing that sticks out to me is that there are a variety of >> places where atoms starting with `$` occur in practice (particularly around >> ETS), so I could see things like `$:$$` appearing in code, which is >> just...no. Of course, an argument could be made that one should just not do >> that, but it is something to consider. Obviously, you can't get rid of the >> `:` for the same reason. >> >> But the idea of an operator more generally? I guess it would really >> depend on the specific choice. I don't like it in principle, but I'd want >> to cast my vote with a specific syntax in question, such as those you've >> proposed. As I mentioned in my previous reply, I really think the best path >> for Elixir with regard to field punning is to solve the syntax ambiguities >> that prevent the "obvious" syntax for it, e.g. `%{foo, bar} = baz`, and >> only focus on supporting atom keys. That may not be possible without >> backwards-incompatible changes to the grammar, in which case it's something >> to throw on the wishlist of things that could go in an eventual Elixir 2.0. >> >> I think it's important to cast the feature in a broader context, because >> I think everyone would agree that field punning is a nice-to-have. But is >> the tradeoff in complexity for the language really worth it? The more >> explicit syntax is (perhaps) more annoying to write, but I think the vast >> majority would agree that it is simple, clear, and easy to reason about. >> When we're arguing for field punning, we're really arguing for a >> significant benefit when writing code, but only in the "obvious" syntax I >> gave an example of above do I think one can argue that there is any benefit >> in terms of readability, and even then it is a small benefit. It adds >> cognitive overhead, particularly for new Elixir developers, as one must >> desugar the syntax in their head. I don't think that cognitive overhead is >> significant, but it is only one thing amongst many that one must carry >> around in their head when working with Elixir code - we should aim to >> reduce that overhead rather than add to it. >> >> Anyway, I don't think I'm adding anything new to the arguments that have >> been made in the past, so I don't want to derail your proposal here, or add >> to the noise, particularly with regard to the "tagged variable captures" >> portion, which deserves its own consideration. I will leave it up to the >> community at large to decide, but just want to say thanks again for putting >> so much effort into summarizing the current state of the discussion and >> implementing a prototype of your proposal - it certainly gives it a lot >> more weight to me. >> >> Paul >> >> >> On Wed, Jun 28, 2023, at 8:45 PM, Christopher Keele wrote: >> >> > My thoughts on the proposal itself aside, I’d just like to say that I >> think you’ve set a great example of what proposals on this list should look >> like. Well done! >> >> Much appreciated! >> >> > I have an almost visceral reaction to the use of capture syntax for >> this though. >> >> > I think calling the `&…` syntax “capture syntax” is actually >> misleading, and only has that name because it can be used to construct >> closures by “capturing” a function name, but it is more accurate to >> consider it closure syntax, in my opinion. >> >> This is a very salient point. How do you feel about introducing a new >> operator for this sugar, such as $:foo? >> On Wednesday, June 28, 2023 at 7:41:05 PM UTC-5 Paul Schoenfelder wrote: >> >> >> My thoughts on the proposal itself aside, I’d just like to say that I >> think you’ve set a great example of what proposals on this list should look >> like. Well done! >> >> I have an almost visceral reaction to the use of capture syntax for this >> though, and I don’t believe any of the languages you mentioned that support >> field punning do so in this fashion. They all use a similar intuitive >> syntax where the variable matches the field name, and they don’t make any >> effort to support string keys. >> >> If Elixir is to ever support field punning, I strongly believe it should >> follow their example. However, there are reasons why Elixir cannot do so >> due to syntax ambiguities (IIRC). In my mind, that makes any effort to >> introduce this feature a non-starter, because code should be first and >> foremost easy to read, and I have yet to see a proposal for this that >> doesn’t make the code harder to read and understand, including this one. >> >> I’d like to have field punning, but by addressing, if possible, the core >> issue that is blocking it. If that can’t be done, I just don’t think the >> cost of overloading unrelated syntax is worth it. I think calling the `&…` >> syntax “capture syntax” is actually misleading, and only has that name >> because it can be used to construct closures by “capturing” a function >> name, but it is more accurate to consider it closure syntax, in my opinion. >> Overloading it to mean capturing things in a more general sense will be >> confusing for everyone, and would only work in a few restricted forms, >> which makes it more difficult to teach and learn. >> >> That’s my two cents anyway, I think you did a great job with the >> proposal, but I’m very solidly against it as the solution to the problem >> being solved. >> >> Paul >> >> >> >> On Wed, Jun 28, 2023, at 7:56 PM, Christopher Keele wrote: >> >> This is a formalization of my concept here >> <https://groups.google.com/g/elixir-lang-core/c/oFbaOT7rTeU/m/BWF24zoAAgAJ>, >> as a first-class proposal for explicit discussion/feedback, since I now >> have a working prototype >> <https://github.com/elixir-lang/elixir/compare/main...christhekeele:elixir:tagged-variable-capture> >> . >> >> *Goal* >> >> The aim of this proposal is to support a commonly-requested feature: >> *short-hand >> construction and pattern matching of key/value pairs of associative data >> structures, based on variable names* in the current scope. >> >> *Context* >> >> Similar shorthand syntax sugar exists in many programming languages >> today, known variously as: >> >> - Field Punning <https://dev.realworldocaml.org/records.html> — OCaml >> - Record Puns >> >> <https://ghc.gitlab.haskell.org/ghc/doc/users_guide/exts/record_puns.html> >> — Haskell >> - Object Property Value Shorthand >> >> <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Object_initializer#property_definitions> >> >> — ES6 Javascript >> >> This feature has been in discussion for a decade, on this mailing list (1 >> <https://groups.google.com/g/elixir-lang-core/c/4w9eOeLvt-8/m/WOkoPSMm6kEJ>, >> 2 >> <https://groups.google.com/g/elixir-lang-core/c/NoUo2gqQR3I/m/WTpArTGMKSIJ>, >> 3 >> <https://groups.google.com/g/elixir-lang-core/c/3XrVXEVSixc/m/NHU2M4QFAQAJ>, >> 4 >> <https://groups.google.com/g/elixir-lang-core/c/OvSQkvXxsmk/m/bKKHbBxiCwAJ>, >> 5 >> <https://groups.google.com/g/elixir-lang-core/c/XxnrGgZsyVc/m/1W-d_XAlBgAJ> >> , 6 <https://groups.google.com/g/elixir-lang-core/c/oFbaOT7rTeU>) >> and the Elixir forum (1 >> <https://elixirforum.com/t/proposal-add-field-puns-map-shorthand-to-elixir/15452>, >> >> 2 >> <https://elixirforum.com/t/shorthand-for-passing-variables-by-name/30583>, >> 3 >> <https://elixirforum.com/t/if-you-could-change-one-thing-in-elixir-language-what-you-would-change/19902/17>, >> >> 4 >> <https://elixirforum.com/t/has-map-shorthand-syntax-in-other-languages-caused-you-any-problems/15403>, >> >> 5 >> <https://elixirforum.com/t/es6-ish-property-value-shorthands-for-maps/1524>, >> 6 >> <https://elixirforum.com/t/struct-creation-pattern-matching-short-hand/7544>), >> >> and has motivated many libraries (1 >> <https://github.com/whatyouhide/short_maps>, 2 >> <https://github.com/meyercm/shorter_maps>, 3 >> <https://hex.pm/packages/shorthand>, 4 <https://hex.pm/packages/synex>). >> These narrow margins cannot fit the full history of possibilities, >> proposals, and problems with this feature, and I will not attempt to >> summarize them all. For context, I suggest reading this mailing list >> proposal >> <https://groups.google.com/g/elixir-lang-core/c/XxnrGgZsyVc/m/1W-d_XAlBgAJ> >> and this community discussion >> <https://elixirforum.com/t/proposal-add-field-puns-map-shorthand-to-elixir/15452> >> in >> particular. >> >> However, in summary, this particular proposal tries to solve a couple of >> past sticking points: >> >> 1. Atom vs String >> >> <https://groups.google.com/g/elixir-lang-core/c/NoUo2gqQR3I/m/IpZQHbZk4xEJ> >> key support >> 2. Visual clarity >> >> <https://groups.google.com/g/elixir-lang-core/c/XxnrGgZsyVc/m/NBkAVto0BAAJ> >> that atom/string matching is occurring >> 3. Limitations of string-based sigil parsing >> >> <https://groups.google.com/g/elixir-lang-core/c/XxnrGgZsyVc/m/TiZw6xM3BAAJ> >> 4. Easy confusion >> >> <https://groups.google.com/g/elixir-lang-core/c/XxnrGgZsyVc/m/WRhXxHDfBAAJ> >> with tuples >> >> I have a working fork of Elixir here >> <https://github.com/christhekeele/elixir/tree/tagged-variable-capture> >> where this proposed syntax can be experimented with. Be warned, it is buggy. >> >> *Proposal: Tagged Variable Captures* >> >> I propose we overload the unary capture operator (*&*) to accept >> compile-time atoms and strings as arguments, for example *&:foo* and >> *&"bar"*. This would *expand at compile time* into *a tagged tuple with >> the atom/string and a variable reference*. For now, I am calling this a >> *"tagged-variable >> capture"* to differentiate it from a function capture. >> >> For the purposes of this proposal, assume: >> >> {foo, bar} = {1, 2} >> >> Additionally, >> >> - Lines beginning with *# == * indicate what the compiler expands an >> expression to. >> - Lines beginning with *# => * represent the result of evaluating >> that expression. >> - Lines beginning with *# !> * represent an exception. >> >> *Bare Captures* >> >> I'm not sure if we should support *bare* tagged-variable capture, but it >> is illustrative for this proposal, so I left it in my prototype. It would >> look like: >> >> &:foo >> *# == **{:foo, foo}* >> *# => *{:foo, 1} >> &"foo" >> *# == **{"foo", foo}* >> *# => *{"foo", 1} >> >> If bare usage is supported, this expansion would work as expected in >> match and guard contexts as well, since it expands before variable >> references are resolved: >> >> {:foo, baz} = &:foo >> *# == {:foo, baz} = {:foo, foo}* >> *# => *{:foo, 1} >> baz >> *# => *1 >> >> *List Captures* >> >> Since capture expressions are allowed in lists, this can be used to >> construct Keyword lists from the local variable scope elegantly: >> >> list = [&:foo, &:bar] >> *# == **list = [{:foo, foo}, {:bar, bar}]* >> *# => *[foo: 1, bar: 2] >> >> This would work with other list operators like *|*: >> >> baz = 3 >> list = [&:baz | list] >> *# == **list = [**{:baz, baz} **| **list**]* >> *# => *[baz: 3, foo: 1, bar: 2] >> >> And list destructuring: >> >> {foo, bar, baz} = {nil, nil, nil} >> [&:baz, &:foo, &:bar] = list >> *# == [{:baz, baz}, {:foo, foo}, {:bar, bar}] = list* >> *# => *[baz: 3, foo: 1, bar: 2] >> {foo, bar, baz} >> *# => *{1, 2, 3} >> >> *Map Captures* >> >> With a small change to the parser, >> <https://github.com/elixir-lang/elixir/commit/0a4f5376c0f9b4db7d71514d05df6b8b6abc96a9> >> >> we can allow this expression inside map literals. Because this expression >> individually gets expanded into a tagged-tuple before the map associations >> list as a whole are processed, it allow this syntax to work in all existing >> map/struct constructs, like map construction: >> >> map = %{&:foo, &"bar"} >> *# == %{:foo => foo, "bar" => bar}* >> *# => *%{:foo => 1, "bar" => 2} >> >> Map updates: >> >> foo = 3 >> map = %{map | &:foo} >> *# == %{map | :foo => foo}* >> *# => *%{:foo => 3, "bar" => 2} >> >> And map destructuring: >> >> {foo, bar} = {nil, nil} >> %{&:foo, &"bar"} = map >> *# == %{:foo => foo, "bar" => bar} = map* >> *# => *%{:foo => 3, "bar" => 2} >> {foo, bar} >> *# => *{3, 2} >> >> *Considerations* >> >> Though just based on an errant thought >> <https://groups.google.com/g/elixir-lang-core/c/oFbaOT7rTeU/m/BWF24zoAAgAJ> >> that popped into my head yesterday, I'm unreasonably pleased with how well >> this works and reads in practice. I will present my thoughts here, though >> again I encourage you to grab my branch >> <https://github.com/christhekeele/elixir/tree/tagged-variable-capture>, >> compile >> it from source >> <https://github.com/christhekeele/elixir/tree/tagged-variable-capture#compiling-from-source>, >> and >> play with it yourself! >> >> *Pro: solves existing pain points* >> >> As mentioned, this solves flaws previous proposals suffer from: >> >> 1. Atom vs String >> >> <https://groups.google.com/g/elixir-lang-core/c/NoUo2gqQR3I/m/IpZQHbZk4xEJ> >> key >> support >> This supports both. >> 2. Visual clarity >> >> <https://groups.google.com/g/elixir-lang-core/c/XxnrGgZsyVc/m/NBkAVto0BAAJ> >> that >> atom/string matching is occurring >> This leverages the appropriate literal in question within the syntax >> sugar. >> 3. Limitations of string-based sigil parsing >> >> <https://groups.google.com/g/elixir-lang-core/c/XxnrGgZsyVc/m/TiZw6xM3BAAJ> >> This is compiler-expansion-native. >> 4. Easy confusion >> >> <https://groups.google.com/g/elixir-lang-core/c/XxnrGgZsyVc/m/WRhXxHDfBAAJ> >> with >> tuples >> %{&:foo, &"bar"} is very different from {foo, bar}, instead of >> 1-character different. >> >> Additionally, it solves my main complaint with historical proposals: >> syntax to combine a variable identifier with a literal must either obscure >> that we are building an identifier, or obscure the key/string typing of the >> literal. >> >> I'm proposing overloading the capture operator rather than introducing a >> new operator because the capture operator already has a semantic >> association with messing with variable scope, via the nested integer-based >> positional function argument syntax (ex *& &1*). >> >> By using the capture operator we indicate that we are messing with an >> identifier in scope, but via a literal atom/string we want to associate >> with, to get the best of both worlds. >> >> *Pro: works with existing code* >> >> The capture today operator has well-defined compile-time-error semantics >> if you try to pass it an atom or a string. All compiling Elixir code today >> will continue to compile as before. >> >> *Pro: works with existing tooling* >> >> By overloading an existing operator, this approach works seamlessly for >> me with the syntax highlighters I have tried it with so far, and reasonable >> with the formatter. >> >> In my experimentation I've found that the formatter wants to rewrite *&:baz >> *to *(&:baz)* pretty often. That's good, because there are several edge >> cases in my prototype where not doing so causes it to behave strangely; I'm >> sure it's resolving ambiguities that would occur in function captures that >> impact my proposal in ways I have yet fully anticipated. >> >> *Pros: minimizes surface area of the language* >> >> By overriding the capture operator instead of introducing a new operator >> or sigil, we are able to keep the surface area of this feature slim. >> >> *Cons: overloads the capture operator* >> >> Of course, much of the virtues of this proposal comes from overloading >> the capture operator. But it is an already semantically fraught syntactic >> sugar construct that causes confusion to newcomers, and this would place >> more strain on it. >> >> We would need to augment it with more than the meager error message >> modification >> <https://github.com/elixir-lang/elixir/commit/3d83d21ada860d03cece8c6f90dbcf7bf9e737ec#diff-92b98063d1e86837fae15261896c265ab502b8d556141aaf1c34e67a3ef3717cL199-R207> >> in >> my prototype, as well as documentation and anticipate a new wave of >> questions from the community upon release. >> >> This inelegance really shows when considering embedding a tagged variable >> capture inside an anonymous function capture, ex *& &1 = &:foo*. In my >> prototype I've chosen to allow this rather than error on "nested captures >> not allowed" (would probably become: "nested *function* captures not >> allowed"), but I'm not sure I found all the edge-cases of mixing them in >> all possible constructions. >> >> Additionally, since my proposal now allows the capture operator as an >> associative element inside map literal parsing, that would change the >> syntax error reported by providing a function capture as an associative >> element to be generated during expansion rather than during parsing. I am >> not fluent enough in leex to have have updated the parser to preserve the >> exact old error, but serendipitously what it reports in my prototype today >> is pretty good regardless, but I prefer the old behaviour: >> >> Old: >> %{& &1} >> *# !> **** (SyntaxError) syntax error before '}'* >> *# !> * | >> *# !> * 1 | %{& &1} >> *# !> * | ^ >> New: >> %{& &1} >> *# => error: expected key-value pairs in a map, got: & &1* >> *# => ** (CompileError) cannot compile code (errors have been logged)* >> >> *Cons: here there be dragons I cannot see* >> >> I'm quite sure a full implementation would require a lot more knowledge >> of the compiler than I am able to provide. For example, *&:foo = &:foo >> *raises >> an exception where *(&:foo) = &:foo* behaves as expected. I also find >> the variable/context/binding environment implementation in the erlang part >> of the compiler during expansion to be impenetrable, and I'm sure my >> prototype fails on edge cases there. >> >> *Open Question: the pin operator* >> >> As this feature constructs a variable ref for you, it is not clear if/how >> we should support attempts to pin the generated variable to avoid new >> bindings. In my prototype, I have tried to support the pin operator via the >> *&^:atom *syntax, though I'm pretty sure it's super buggy on bare >> out-of-data-structure cases and I only got it far enough to work in >> function heads for basic function head map pattern matching. >> >> *Open Question: charlists* >> >> I did not add support for charlist tagged variable captures in my >> prototype, as it would be more involved to differentiate a capture of list >> mean to become a tagged tuple from a list representing the AST of a >> function capture. I would not lose a lot of sleep over this. >> >> *Open Question: allowed contexts* >> >> Would we even want to allow this syntax construct outside of map >> literals? Or list literals? >> >> I can certainly see people abusing the >> bare-outside-of-associative-datastructure syntax to make some neigh >> impenetrable code where it's really unclear where assignment and pattern >> matching is occuring, and relatedly this is where I see a lot of odd >> edge-case behaviour in my prototype. I allowed it to speed up the >> implementation, but it merits more discussion. >> >> On the other hand, this does seem like an... interesting use-case: >> >> error = "rate limit exceeded" >> &:error *# return error tuple* >> >> *Thanks for reading! What do you think?* >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elixir-lang-co...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-core/ad7e0313-4207-4cb7-a5f3-d824735830abn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elixir-lang-core/ad7e0313-4207-4cb7-a5f3-d824735830abn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elixir-lang-co...@googlegroups.com. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-core/2b46232e-04f1-4b21-87e6-9c098741cd36n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elixir-lang-core/2b46232e-04f1-4b21-87e6-9c098741cd36n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/8b4fa272-f13f-4342-b258-2efb99cc3928n%40googlegroups.com.