[go-nuts] How to stream across json-seq RFC-7464

[Greg Saylor]

Good afternoon,

For a case where there's a file containing a sequence of hashes (it could 
be arrays too, as the underlying object type seems irrelevant) as per 
RFC-7464.  I cannot figure out how to handle this in a memory efficient way 
that doesn't involve pulling each blob 

I've tried to express this on Go playground 
here: https://play.golang.org/p/Aqx0gnc39rn
Note that I'm using exponent-io/jsonpath as the JSON decoder, but certainly 
that could be swapped for something else.

In essence here is an example of the input bytes:

{
   "elements" : [
      {
         "Space" : "YCbCr",
         "Point" : {
            "Cb" : 0,
            "Y" : 255,
            "Cr" : -10
         }
      },
      {
         "Point" : {
            "B" : 255,
            "R" : 98,
            "G" : 218
         },
         "Space" : "RGB"
      }
   ]
}
{
   "elements" : [
      {
         "Space" : "YCbCr",
         "Point" : {
            "Cb" : 3000,
            "Y" : 355,
            "Cr" : -310
         }
      },
      {
         "Space" : "RGB",
         "Point" : {
            "B" : 355,
            "G" : 318,
            "R" : 108
         }
      }
   ]
}
{
   "elements" : [
      {
         "Space" : "YCbCr",
         "Point" : {
            "Cr" : -410,
            "Cb" : 400,
            "Y" : 455
         }
      },
      {
         "Space" : "RGB",
         "Point" : {
            "B" : 455,
            "R" : 118,
            "G" : 418
         }
      }
   ]
}

I can iterate through that with this code:

w := json.NewDecoder(bytes.NewReader(j))
for w.More() {
var v interface{}
w.Decode(&v)
fmt.Printf("%+v\n", v)
}

This works, but the downside is that each {...} of bytes has to be pulled 
into memory.  And the functions that is called is already designed to 
receive an io.Reader and parse the VERY large inner blob in an efficient 
manner.

So in principal, this is kinda want I want to do, but maybe I'm looking at 
it all wrong:


w := json.NewDecoder(bytes.NewReader(j))
for w.More() {
reader2 := ???? //Some io.Reader that represents each of the 3 json-seq 
blocks
secondDecoder(reader2)
}

func secondDecoder(reader io.Reader) {
w2 := json.NewDecoder(reader)
var v interface{}
w2.Decode(&v)
fmt.Printf("%+v\n", v)
}

Any ideas on how to solve this problem?

I should note that it is not possible for the input to change in this case 
as the system that consumes it is not the same one that has been generating 
it for the past 5 years.

Thanks!

- Greg

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/a68b4b7b-efe0-407c-bdd8-59cdc5ad7816n%40googlegroups.com.