Hi Matt,
Many of the deployments I get involved with are not in a Hadoop ecosystem,
so building in resilience becomes part of the challenge, especially when
data input is out of your control and often has bad data. In these
situations, one thing I like doing is a test flow using DuplicateFlowFile
and DistributeLoad configured as a valve so the main queue isn't
backlogged. You can then use live data to work on massaging bad input.
Manually creating schemas for record based processors and JoltTransformJSON
becomes a large part of the job.
In one such experiment I was either trying to work around GenerateRecord
not having capability for some data types (e.g. Boolean) and faking it with
a jolt transform, or I was putting parent key/values back into a content
after splitting on a sub-arrray. Either way, I ended up with a jolt
transform inserting a timestamp.
Current time is pretty useful for downstream systems to understand latency
between an event and it getting to that system, I often ensure Nifi is
dropping it into content.
I'm not near my test system to get the exact spec file but will endeavour
to at the next opportunity.
On Fri, 18 Aug 2023, 07:13 Matt Burgess, <mattyb...@gmail.com> wrote:
> Matthew,
>
> What was your second case trying to use ${now():toNumber()} ? My unit
> test evaluated the expression into an integer and it worked (versus
> putting the expression in quotes which would make it a string).
>
> Regards,
> Matt
>
> On Tue, Aug 15, 2023 at 4:09 AM Matthew Hawkins <hawko2...@gmail.com>
> wrote:
> >
> > Hi Matt,
> >
> > The tag will be ${firstname}, the spec is correct.
> > (well, I also get lastname, but it's unimportant)
> >
> > Weird thing is this failed similarly with a file based input on the
> Record
> > based jolt processor as well on a completely different system.
> >
> > Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
> > source with
> include-graph,include-media,include-rules,include-sql-reporting
> >
> > In the second case I was trying to get ${now():toNumber()} into a json
> > record using either default or modify-overwrite-beta (tried both). The
> > following success processor failed on reading schema as the literal
> > ${now...} was a string not a long and the JVM refused the type cast. If I
> > insert the transform directly then it still fails with a type cast
> problem
> > from generic Object :/ (that was midnight last night, and I tossed my
> hands
> > in the air and turned the server off)
> >
> > I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
> > postgresql. Trying to be a good nifi citizen and use record based
> > processing where possible. Since I don't run Hadoop at home, it's all
> > manual schemas and using postgres as the perm data store. I'm using json
> as
> > mqtt does it intrinsically and I can jolt transform other data into json
> > form and then store it easily in postgres. Well, that was the idea 😁
> >
> > Kr,
> >
> > On Sun, 13 Aug 2023, 10:44 Matt Burgess, <mattyb...@gmail.com> wrote:
> >
> > > Just to follow up, I added a unit test to put EL in a JOLT spec and it
> > > worked. I noticed you referred to "attrname" in your post but your
> > > spec refers to "firstname", is that a typo?
> > >
> > > Regards,
> > > Matt
> > >
> > > On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess <mattyb...@apache.org>
> wrote:
> > > >
> > > > I added file support to JoltTransformJSON under NIFI-4957 [1], a
> first
> > > > glance at the code seems like it should work, but I'll try to
> > > > reproduce it and follow up, thanks for bringing this to our
> attention!
> > > >
> > > > Regards,
> > > > Matt
> > > >
> > > > [1] https://issues.apache.org/jira/browse/NIFI-4957
> > > >
> > > > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins <hawko2...@gmail.com
> >
> > > wrote:
> > > > >
> > > > > Hi devs,
> > > > >
> > > > > Using 1.23.0 I have a simple transform adding some flowfile
> attributes
> > > into
> > > > > the content.
> > > > > When putting the spec directly to the processor it works fine. If I
> > > have
> > > > > the spec in an external file however it seems to put in the literal
> > > string
> > > > > ${attrname} - ie the EL didn't process.
> > > > >
> > > > > Have I done something incorrectly or should I be now asking for an
> > > account
> > > > > to log a bug?
> > > > >
> > > > > Sample spec run using Chain DSL:
> > > > >
> > > > > [{
> > > > > "operation": "shift",
> > > > > "spec": {
> > > > > "@": "values"
> > > > > },
> > > > > {
> > > > > "operation": "default",
> > > > > "spec": {
> > > > > "firstname": "${firstname}"
> > > > > }
> > > > > }
> > > > > ]
> > > > >
> > > > > Kind regards,
> > > > > Matthew
> > >
>
