Mew,
Are you able to provide examples of the additional data?
Are you able to (easily) see whether any of your data has these inner
fields as anything other than nested objects, e.g. maybe the field is a
string in one of your top-level objects?
On Tue, 10 Oct 2023, 21:47 Mark Woodcock, <woodc...@usna.edu.invalid> wrote:
> Chris,
>
> 1) I've upgraded to 1.23.2 (which appears to be the latest and greatest).
>
> 2) I've tested the JoltTransformRecord with
> a) JsonTreeReader w/ InferredSchema
> b) JsonRecordSetWriter w/ InheritsSchema
> c) a GetFile processor which grabs a text file with the various bits of
> test data
>
> It appears that your suspicions are correct:
> i) if I test with just that single record as the entire content of the
> file, the processor is successful.
> ii) if I test with multiple records, none of which have the complicated
> inner field, all is successful.
> c) if I test with multiple records, where at least one has the complicated
> inner field, I get the earlier noted error.
>
> IOW, yep, it only happens with *more* data.
>
> bummer,
>
> mew
>
> On Tue, Oct 10, 2023 at 2:43 PM Chris Sampson
> <chris.samp...@naimuri.com.invalid> wrote:
>
> > Using your example (single JSON Object and Jolt Spec) seems to work fine
> > in both JoltTransformJSON and JoltTransformRecord when run on the current
> > main branch (which is for the upcoming 2.0.0 release).
> >
> > To test, I setup a GenerateFlowFile processor to output the example JSON
> > you gave, then sent that through both of the Jolt processors using a
> > JsonTreeReader with “Inferred Schema”, and a JsonRecordSetWriter that
> > “Inherits Schema” for the Record processor.
> >
> > If you run *just* your example from this email chain through the Jolt
> > processors on the version of NiFi you’re using, do you see the errors you
> > mention, or does that only happen with more data?
> >
> >
> > Cheers,
> >
> > ---
> > Chris Sampson
> > IT Consultant
> > chris.samp...@naimuri.com
> >
> >
> > > On 10 Oct 2023, at 15:45, Mark Woodcock <woodc...@usna.edu.INVALID>
> > wrote:
> > >
> > > Hmmmm,
> > >
> > > One small problem: While JOLTTransformJSON is quite lovely (a) it has
> a
> > > great "advanced" interface that allows one to test their spec and json
> > > inputs and (b) it actually works for the cases that I noted...it treats
> > the
> > > input a single blob of JSON. Unfortunately, my input files are
> > collections
> > > of JSON records (which--less the noted problem--JOLTTransformRecord
> does
> > > quite nicely with)--that's literally how they arrive, not the result of
> > me
> > > formatting them at all.
> > >
> > > Is there a way to get JTJ to treat the input as records?
> > > Does 1.22 or 1.23 have the fix for JTR?
> > >
> > > thx,
> > >
> > > mew
> > >
> > >
> > > On Mon, Oct 9, 2023 at 3:21 PM Mark Woodcock <woodc...@usna.edu>
> wrote:
> > >
> > >> confirmed: version 1.21.
> > >> How recent is the fix?
> > >>
> > >> thx,
> > >>
> > >> mew
> > >>
> > >>
> > >> On Sun, Oct 8, 2023 at 11:39 PM Mark Woodcock <woodc...@usna.edu>
> > wrote:
> > >>
> > >>> Matt,
> > >>>
> > >>> Unfortunately (at home now) the details are all at work at the
> moment,
> > >>> but I know that I didn't start this work until April (at the
> > earliest), so
> > >>> I'm surely using at least 1.21; is the fix more recent than that?
> > {If so,
> > >>> perhaps there is a bug.}
> > >>>
> > >>> Fortunately, yea, JSON out is the intent; I need the data to be in
> that
> > >>> format to set up a subsequent transform to AVRO, so it seems there
> are
> > two
> > >>> possible ways out (depending on which version I'm running): upgrade
> or
> > >>> change processors. So, at least there is a path.
> > >>>
> > >>> thx,
> > >>>
> > >>> mew
> > >>>
> > >>>
> >
> >
>
