I'd originally looked for tests in the spec, browsers, and other languages to compare against.
W3C's current test suite doesn't seem comprehensive: https://github.com/web-platform-tests/wpt/blob/master/encoding/replacement-encodings.any.js Go has a few token tests that are relatively intelligible: https://cs.opensource.google/go/go/+/refs/tags/go1.21.3:src/bytes/bytes_test.go;l=1157 On Tue, Oct 31, 2023 at 2:09 PM José Valim <jose.va...@dashbit.co> wrote: > Does the specification provide tests for us to include? Otherwise we can > include enough tests for full line coverage and a “brute force”/property > test commented out. > > I would say the name “replace_invalid” is excellent. > > On Tue, Oct 31, 2023 at 18:52 Cameron Duley <cameron.dule...@gmail.com> > wrote: > >> This was the final version I'd landed on for UTF-8: >> >> https://github.com/elixir-unicode/unicode/blob/main/lib/unicode/validation/utf8.ex >> >> Along with the following modules for testing: >> >> https://github.com/elixir-unicode/unicode/blob/main/test/support/unicode_validation_helpers.ex >> >> https://github.com/elixir-unicode/unicode/blob/main/test/unicode_validation_test.exs >> >> I think it's ideal functionality to have in the String module, and the >> implementation's "reasonable enough" until a native solution's available in >> OTP. >> >> Testing is my only uncertainty - How much is prudent and in what manner? >> On Tuesday, October 31, 2023 at 12:35:48 PM UTC-4 José Valim wrote: >> >>> Folks, I am following up on this, where did we land? >>> >>> The new implementation is roughly ~70LOC for UTF-8, so at first I don't >>> see an issue with adding it to Elixir. However, the Elixir version would be >>> UTF-8 only (part of the String module). >>> >>> Thoughts? >>> >>> On Saturday, October 7, 2023 at 10:51:34 PM UTC+2 Kip wrote: >>> >>>> Cameron, I think this would be a useful addition to the Unicode library >>>> <https://github.com/elixir-unicode/unicode> I maintain. If that works >>>> for you, please open an issue there and we can collaborate. I think it >>>> being part of the Erlang `:unicode` module makes good sense too as José >>>> says but that's a longer "sales" and implementation cycle. >>>> >>>> On Saturday, October 7, 2023 at 7:40:50 PM UTC+11 José Valim wrote: >>>> >>>>> Hi Cameron, >>>>> >>>>> If the goal is to include this handling for UTF-16 and UTF-32, I >>>>> suggest proposing this to Erlang/OTP as new functions in the "unicode" >>>>> module. Otherwise, Elixir only has facilities to deal with UTF-8. You >>>>> could >>>>> propose such a feature in their issues tracker. >>>>> >>>>> Also note that "rolling your own" or "depending on packages" is >>>>> usually not enough reasons for adding features to Elixir. Otherwise, one >>>>> could easily argue Decimal and Jason would be more important additions to >>>>> the language. :) We do describe which features we would consider part of >>>>> the language here: https://elixir-lang.org/development.html >>>>> >>>>> Other than that, awesome job on the library and benchmarks. :) >>>>> >>>>> On Sat, Oct 7, 2023 at 1:03 AM Kip <kipc...@gmail.com> wrote: >>>>> >>>>>> Your implementation is definitely fast and memory efficient so I >>>>>> retract my implementation comments. Now that I've run the benchmarking >>>>>> script and tested out a few different approaches leveraging the std lib I >>>>>> understand better why you've taken the approach you have. Nice work. >>>>>> >>>>>> On Saturday, October 7, 2023 at 9:26:37 AM UTC+11 Kip wrote: >>>>>> >>>>>>> Cameron, I think this is a useful proposal. Elixir has means to >>>>>>> check validity (String.valid?/1) and a mechanism to split valid and >>>>>>> invalid >>>>>>> code points (String.chunk/2 with the :valid trait). But there isn't, to >>>>>>> my >>>>>>> knowledge, a means to coerce validity. A couple of thoughts: >>>>>>> >>>>>>> 1. Since Elixir strings are, by definition, UTF8, I don't know that >>>>>>> special handling of UTF16 and UTF32 code points makes much sense - >>>>>>> although >>>>>>> I accept this may be more Unicode compliant. >>>>>>> 2. What would the function be called? Since we have String.valid?/1 >>>>>>> maybe String.validate/2 with an option `replace_invalid: utf8_string`. >>>>>>> The >>>>>>> default `:replace_invalid` could be U+FFFD or it could be `nil`. >>>>>>> If the default is `nil` then there could also be a `String.validate!/2` >>>>>>> that raises if there is no `:replace_invalid` option. >>>>>>> 3. I think the implementation could leverage the code of >>>>>>> `String.chunk/2` which uses `String.next_codepoint/1`. That would >>>>>>> simplify >>>>>>> implementation and be more consistent in code style. >>>>>>> >>>>>>> On Friday, October 6, 2023 at 12:24:28 PM UTC+11 >>>>>>> cameron...@gmail.com wrote: >>>>>>> >>>>>>>> As far as I can tell, neither Elixir nor Erlang have a built in >>>>>>>> function for replacing invalid sequences in Unicode. There's a >>>>>>>> suggested >>>>>>>> method on this page >>>>>>>> <https://www.unicode.org/versions/Unicode15.0.0/UnicodeStandard-15.0.pdf#page=153> >>>>>>>> of the Unicode standard for handling this. Several other languages ( >>>>>>>> Go <https://pkg.go.dev/bytes#ToValidUTF8>, Python >>>>>>>> <https://docs.python.org/3/library/stdtypes.html#bytes.decode>, C# >>>>>>>> <https://github.com/dotnet/docs/issues/13547>, etc) now follow >>>>>>>> this spec. >>>>>>>> >>>>>>>> Invalid Unicode's encountered frequently enough that I think it's >>>>>>>> worth incorporating a solution into Elixir itself. >>>>>>>> >>>>>>>> Present alternatives to handling invalid unicode (and json by >>>>>>>> extension <https://github.com/michalmuskala/jason/issues/174>) are: >>>>>>>> >>>>>>>> - Crashing (not ideal in many cases) >>>>>>>> - Roll your own (lot of overhead for accidental complexity) >>>>>>>> - Depend on a package (+1 package towards dependency hell) >>>>>>>> >>>>>>>> This is my college try >>>>>>>> <https://github.com/Moosieus/UniRecover/tree/main>, but I'm >>>>>>>> certain there's a performant and far cleaner solution to be had in pure >>>>>>>> Elixir. If not, perhaps this is a request for OTP. >>>>>>>> >>>>>>> >>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elixir-lang-core" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to elixir-lang-co...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elixir-lang-core/197620a2-6a96-41c6-a6e7-5da03e351080n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/197620a2-6a96-41c6-a6e7-5da03e351080n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elixir-lang-core+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-core/26e4c761-caf0-46ea-ab08-a33407febdb8n%40googlegroups.com >> <https://groups.google.com/d/msgid/elixir-lang-core/26e4c761-caf0-46ea-ab08-a33407febdb8n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elixir-lang-core+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LKEOb%3D7WMeUhots27dj%3DAwvhLF3o%2Bo7Q2%2BhZXMNQi8YA%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LKEOb%3D7WMeUhots27dj%3DAwvhLF3o%2Bo7Q2%2BhZXMNQi8YA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Thanks, Cameron Duley -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAH7jZqe%3D%3DKUPG6O3v_%3DSr%3DAjzv28X5x1QvpF-bqJ96XWOVPu5w%40mail.gmail.com.
