Yes, good one, sorry I didn't clarify. One of the points of my talk was that I'm using the parser to parse user-written scripts, that look like elixir but are actually interpreted in a different way. And currently there is no way to safely parse Elixir-ish code which contains unknown identifiers.
So in my talk I worked around that problem by escaping all such atoms in a predefined way, but after the talk José suggested using a callback function. By the way, the slides are here: https://docs.google.com/presentation/d/15_xKuL_H4Eu-EkGarxVixCk192858avE1ef1gmcVKoc/edit?usp=sharing video hopefully up soon. Arjan On Wednesday, April 10, 2019 at 3:00:30 PM UTC+2, Allen Madsen wrote: > > For those of us who haven't seen your talk, what's the reason for wanting > to add this? > > Allen Madsen > http://www.allenmadsen.com > > > On Wed, Apr 10, 2019 at 7:50 AM Arjan Scherpenisse <[email protected] > <javascript:>> wrote: > >> Hello all, >> >> As I discussed with José on ElixirconfEU after my talk on the parser, I >> would like to add an option to the tokenizer to be able to still being able >> to parse Elixir code without creating atoms. >> >> The solution would be to have a callback function as an option to the >> tokenizer (exposed through `Code.string_to_quoted/2`), that would get >> called in the event of the tokenizer encountering an unexisting atom. >> Instead of raising, a callback function could be called with the token and >> the tokenizer metadata. The callback function returns the data structure >> that would be put in the AST instead of the atom. For instance, like in my >> talk, an 'atom marker' {:":", "atomname"}. >> >> The callback gets 4 arguments: >> - atom name (string) >> - file >> - line >> - column >> >> The default behaviour of this function could be to raise an error, like >> the existing_atoms_only: true option does now. So two questions before I >> implement this: >> >> 1) How should we call this option? I was thinking of either one of these: >> - nonexisting_atom_callback: >> - on_nonexisting_atom: >> >> 2) Should the callback function option be only applicable when >> existing_atoms_only: true is passed in as well? Or would it be like this: >> >> Code.string_to_quoted(string_w_new_atoms) → normal behaviuor, creates >> atoms >> Code.string_to_quoted(string_w_new_atoms, existing_atoms_only: true) → >> raises ("builtin" callback behaviour) >> Code.string_to_quoted(string_w_new_atoms, nonexisting_atom_callback: >> &mycallback/4) → gets called >> >> in the last case, existing_atoms_only: true is implicit. >> However that creates a confusing situation when combining >> existing_atoms_only: false + nonexisting_atom_callback . So I prefer to be >> explicit: >> >> Code.string_to_quoted(string_w_new_atoms, nonexisting_atom_callback: >> &mycallback/4) → raises RuntimeError >> >> Code.string_to_quoted(string_w_new_atoms, existing_atoms_only: true, >> nonexisting_atom_callback: &mycallback/4) → mycallback gets called >> >> >> Arjan >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-core/ac2699c7-af26-4b0d-9734-c385aa59aef2%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elixir-lang-core/ac2699c7-af26-4b0d-9734-c385aa59aef2%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/e5c10522-18fa-493a-9b27-c1a3a33917fa%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
