You're absolutely right. I just meant that the tokenizer interface wouldn't be completely replaced. There are still some corner cases where it will be necessary.
On Friday, August 9, 2019 at 11:41:17 PM UTC+2, burak serdar wrote: > > On Fri, Aug 9, 2019 at 3:32 PM Jonathan Hall <fli...@flimzy.com > <javascript:>> wrote: > > > > I think you're right that most people are frustrated by the encoder, but > as I mentioned in another message just a bit ago, the same fundamental > problem exists with decoder, and for certain workloads, I believe it should > be solved. > > > > Having said that, I think tackling the writer first definitely makes the > most sense. > > > > With the decoder, at least it's possible, although terribly cumbersome, > to cobble together a solution with the tokenizer interface of json.Decoder. > And truth be told, my proposal wouldn't really elminate the use of the > tokenizer interface--but it would make it possible to use it on a burried > type (i.e. within a UnmarshalJSONStream() method), to achieve the benefit > of stream reading. > > It may not be terribly cumbersome, and may not need the tokenizer > interface. I have a json streaming library (streaming in the sense > that multiple json docs one after the other, not one large doc) based > on the std encoder/decoder, and something like that can be developed > to deal with large json docs. > > https://github.com/bserdar/jsonstream > > > > > > > > > > > > On Friday, August 9, 2019 at 10:17:41 PM UTC+2, burak serdar wrote: > >> > >> On Fri, Aug 9, 2019 at 2:10 PM Robert Engels <ren...@ix.netcom.com> > wrote: > >> > > >> > I'm sorry, maybe I didn't understand your original concern. There is > an example of doing stream based processing in the json package (using > Decoder). > >> > > >> > How is this not sufficient? > >> > > >> > The only problem I see with it is that it doesn't allow error > correction/continuation, but in the modern world that seems rather rare (or > very difficult to do well). > >> > >> I was thinking similarly and after reading those github issues, it > >> looks like the main problem is with Encoder, and not with Decoder. > >> Encoder's problem can be solved by providing an unbuffered output > >> option that directly writes to the io.Writer. > >> > >> I like the idea of stream-friendly marshaler/unmarshaler interfaces. > >> > >> > > >> > -----Original Message----- > >> > From: Jonathan Hall > >> > Sent: Aug 9, 2019 11:00 AM > >> > To: golang-nuts > >> > Subject: Re: [go-nuts] RFC for opt-in streaming support in > encoding/json package > >> > > >> > An interesting observation. > >> > > >> > Although in a sense, we already have the decoding half of that > low-level streaming API exposed by way of the `json.Decoder` type. > >> > > >> > Would you be suggesting that a separate, stream-based API makes sense > even within the stdlib? > >> > > >> > I'm not really sure what that separate API would look like, or do > differently than my proposal (I'm open to new ideas, though). > >> > > >> > Given that the "Go way" of handling streams is with > io.Reader/io.Writer (as opposed to events, for example), and the internal > implementation of `json/encoding` is already so close to that, I wonder if > the APIs would end up looking very much the same, anyway. > >> > > >> > Jonathan > >> > > >> > > >> > On Friday, August 9, 2019 at 5:53:41 PM UTC+2, Robert Engels wrote: > >> >> > >> >> In other environments (e.g. Java), the streaming processors are > distinct from the instance oriented - usually for good reason as the APIs > are very different - the former is usually event based, and the latter > returns realized instances as a whole or an error. The streaming processors > can often skip ill-formed entities, and/or have them manipulated during > decoding. > >> >> > >> >> -----Original Message----- > >> >> From: Jonathan Hall > >> >> Sent: Aug 9, 2019 10:38 AM > >> >> To: golang-nuts > >> >> Subject: Re: [go-nuts] RFC for opt-in streaming support in > encoding/json package > >> >> > >> >> Thanks for the reply. My responses inline below. > >> >> > >> >> On Friday, August 9, 2019 at 5:14:52 PM UTC+2, burak serdar wrote: > >> >>> > >> >>> On Fri, Aug 9, 2019 at 8:53 AM Jonathan Hall <fli...@flimzy.com> > wrote: > >> >>> > > >> >>> > Can you say more? Better in which way? > >> >>> > >> >>> Better in the way that it wouldn't change existing code. > >> >> > >> >> > >> >> That doesn't seem like a benefit, in its own right > >> >> > >> >> I understand the desire not to just change code for its own sake, or > add extra features nobody needs. But people have been asking for these > types of features for several years. This doesn't seem like a frivolous > code change to me. > >> >> > >> >>> > >> >>> Also, I think > >> >>> the use cases for existing and proposed json encoders/decoders are > >> >>> different enough to justify a separate implementation. > >> >> > >> >> > >> >> I don't think I agree with this. > >> >> > >> >> The proposal deals with a subset of current use cases, but not, > strictly speaking, a _different set_ of use cases. And the number of > commentators on the issues linked above, I think lends weight to the idea > that the use cases this proposal addresses are not insignificant, or > fundamentally "different". > >> >> > >> >> If I were to fork the standard `encoding/json` library, and add my > proposed functionality, the code would still be 95% the same. Standard > reasons for sharing code apply, as far as I can tell. > >> >> > >> >> -- > >> >> You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > >> >> To unsubscribe from this group and stop receiving emails from it, > send an email to golan...@googlegroups.com. > >> >> To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/df688733-a2e9-4bc8-aa7b-09267827007a%40googlegroups.com. > > > >> >> > >> >> > >> >> > >> >> > >> > -- > >> > You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send an email to golan...@googlegroups.com. > >> > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/718e2b9e-39e2-44f2-9308-8c94e31afbff%40googlegroups.com. > > > >> > > >> > > >> > > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send an email to golan...@googlegroups.com. > >> > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/924142684.7377.1565381399326%40wamui-agami.atl.sa.earthlink.net. > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to golan...@googlegroups.com <javascript:>. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/4d901ba9-5724-45cd-b7f7-24fccd9ef5db%40googlegroups.com. > > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/22f450b6-2752-43aa-89c8-41a216a2ab1b%40googlegroups.com.
