Chris,
1) I've upgraded to 1.23.2 (which appears to be the latest and greatest).
2) I've tested the JoltTransformRecord with
a) JsonTreeReader w/ InferredSchema
b) JsonRecordSetWriter w/ InheritsSchema
c) a GetFile processor which grabs a text file with the various bits of
test data
It appears that your suspicions are correct:
i) if I test with just that single record as the entire content of the
file, the processor is successful.
ii) if I test with multiple records, none of which have the complicated
inner field, all is successful.
c) if I test with multiple records, where at least one has the complicated
inner field, I get the earlier noted error.
IOW, yep, it only happens with *more* data.
bummer,
mew
On Tue, Oct 10, 2023 at 2:43 PM Chris Sampson
<chris.samp...@naimuri.com.invalid> wrote:
> Using your example (single JSON Object and Jolt Spec) seems to work fine
> in both JoltTransformJSON and JoltTransformRecord when run on the current
> main branch (which is for the upcoming 2.0.0 release).
>
> To test, I setup a GenerateFlowFile processor to output the example JSON
> you gave, then sent that through both of the Jolt processors using a
> JsonTreeReader with “Inferred Schema”, and a JsonRecordSetWriter that
> “Inherits Schema” for the Record processor.
>
> If you run *just* your example from this email chain through the Jolt
> processors on the version of NiFi you’re using, do you see the errors you
> mention, or does that only happen with more data?
>
>
> Cheers,
>
> ---
> Chris Sampson
> IT Consultant
> chris.samp...@naimuri.com
>
>
> > On 10 Oct 2023, at 15:45, Mark Woodcock <woodc...@usna.edu.INVALID>
> wrote:
> >
> > Hmmmm,
> >
> > One small problem: While JOLTTransformJSON is quite lovely (a) it has a
> > great "advanced" interface that allows one to test their spec and json
> > inputs and (b) it actually works for the cases that I noted...it treats
> the
> > input a single blob of JSON. Unfortunately, my input files are
> collections
> > of JSON records (which--less the noted problem--JOLTTransformRecord does
> > quite nicely with)--that's literally how they arrive, not the result of
> me
> > formatting them at all.
> >
> > Is there a way to get JTJ to treat the input as records?
> > Does 1.22 or 1.23 have the fix for JTR?
> >
> > thx,
> >
> > mew
> >
> >
> > On Mon, Oct 9, 2023 at 3:21 PM Mark Woodcock <woodc...@usna.edu> wrote:
> >
> >> confirmed: version 1.21.
> >> How recent is the fix?
> >>
> >> thx,
> >>
> >> mew
> >>
> >>
> >> On Sun, Oct 8, 2023 at 11:39 PM Mark Woodcock <woodc...@usna.edu>
> wrote:
> >>
> >>> Matt,
> >>>
> >>> Unfortunately (at home now) the details are all at work at the moment,
> >>> but I know that I didn't start this work until April (at the
> earliest), so
> >>> I'm surely using at least 1.21; is the fix more recent than that?
> {If so,
> >>> perhaps there is a bug.}
> >>>
> >>> Fortunately, yea, JSON out is the intent; I need the data to be in that
> >>> format to set up a subsequent transform to AVRO, so it seems there are
> two
> >>> possible ways out (depending on which version I'm running): upgrade or
> >>> change processors. So, at least there is a path.
> >>>
> >>> thx,
> >>>
> >>> mew
> >>>
> >>>
>
>
