Chris, 1) I've attached a file with a total of four records. 3 with no complex inner fields, and the last one has the complex inner fields. I get the error on the last record: complaining about the Dimensions field.
2) The data definitely seems to be nested objects (at least as I understand
them).
thx!
mew
On Tue, Oct 10, 2023 at 5:17 PM Chris Sampson
<chris.samp...@naimuri.com.invalid> wrote:
> Mew,
>
> Are you able to provide examples of the additional data?
>
> Are you able to (easily) see whether any of your data has these inner
> fields as anything other than nested objects, e.g. maybe the field is a
> string in one of your top-level objects?
>
>
>
> On Tue, 10 Oct 2023, 21:47 Mark Woodcock, <woodc...@usna.edu.invalid>
> wrote:
>
> > Chris,
> >
> > 1) I've upgraded to 1.23.2 (which appears to be the latest and
> greatest).
> >
> > 2) I've tested the JoltTransformRecord with
> > a) JsonTreeReader w/ InferredSchema
> > b) JsonRecordSetWriter w/ InheritsSchema
> > c) a GetFile processor which grabs a text file with the various bits of
> > test data
> >
> > It appears that your suspicions are correct:
> > i) if I test with just that single record as the entire content of the
> > file, the processor is successful.
> > ii) if I test with multiple records, none of which have the complicated
> > inner field, all is successful.
> > c) if I test with multiple records, where at least one has the
> complicated
> > inner field, I get the earlier noted error.
> >
> > IOW, yep, it only happens with *more* data.
> >
> > bummer,
> >
> > mew
> >
> > On Tue, Oct 10, 2023 at 2:43 PM Chris Sampson
> > <chris.samp...@naimuri.com.invalid> wrote:
> >
> > > Using your example (single JSON Object and Jolt Spec) seems to work
> fine
> > > in both JoltTransformJSON and JoltTransformRecord when run on the
> current
> > > main branch (which is for the upcoming 2.0.0 release).
> > >
> > > To test, I setup a GenerateFlowFile processor to output the example
> JSON
> > > you gave, then sent that through both of the Jolt processors using a
> > > JsonTreeReader with “Inferred Schema”, and a JsonRecordSetWriter that
> > > “Inherits Schema” for the Record processor.
> > >
> > > If you run *just* your example from this email chain through the Jolt
> > > processors on the version of NiFi you’re using, do you see the errors
> you
> > > mention, or does that only happen with more data?
> > >
> > >
> > > Cheers,
> > >
> > > ---
> > > Chris Sampson
> > > IT Consultant
> > > chris.samp...@naimuri.com
> > >
> > >
> > > > On 10 Oct 2023, at 15:45, Mark Woodcock <woodc...@usna.edu.INVALID>
> > > wrote:
> > > >
> > > > Hmmmm,
> > > >
> > > > One small problem: While JOLTTransformJSON is quite lovely (a) it
> has
> > a
> > > > great "advanced" interface that allows one to test their spec and
> json
> > > > inputs and (b) it actually works for the cases that I noted...it
> treats
> > > the
> > > > input a single blob of JSON. Unfortunately, my input files are
> > > collections
> > > > of JSON records (which--less the noted problem--JOLTTransformRecord
> > does
> > > > quite nicely with)--that's literally how they arrive, not the result
> of
> > > me
> > > > formatting them at all.
> > > >
> > > > Is there a way to get JTJ to treat the input as records?
> > > > Does 1.22 or 1.23 have the fix for JTR?
> > > >
> > > > thx,
> > > >
> > > > mew
> > > >
> > > >
> > > > On Mon, Oct 9, 2023 at 3:21 PM Mark Woodcock <woodc...@usna.edu>
> > wrote:
> > > >
> > > >> confirmed: version 1.21.
> > > >> How recent is the fix?
> > > >>
> > > >> thx,
> > > >>
> > > >> mew
> > > >>
> > > >>
> > > >> On Sun, Oct 8, 2023 at 11:39 PM Mark Woodcock <woodc...@usna.edu>
> > > wrote:
> > > >>
> > > >>> Matt,
> > > >>>
> > > >>> Unfortunately (at home now) the details are all at work at the
> > moment,
> > > >>> but I know that I didn't start this work until April (at the
> > > earliest), so
> > > >>> I'm surely using at least 1.21; is the fix more recent than that?
> > > {If so,
> > > >>> perhaps there is a bug.}
> > > >>>
> > > >>> Fortunately, yea, JSON out is the intent; I need the data to be in
> > that
> > > >>> format to set up a subsequent transform to AVRO, so it seems there
> > are
> > > two
> > > >>> possible ways out (depending on which version I'm running):
> upgrade
> > or
> > > >>> change processors. So, at least there is a path.
> > > >>>
> > > >>> thx,
> > > >>>
> > > >>> mew
> > > >>>
> > > >>>
> > >
> > >
> >
>
short.json
Description: application/json
