Re: JoltTransformJSON EL when using file
Hi Matt,
Had this happen again.
Spec:
[{
"operation": "default",
"spec": {
"PartitionID": "${distribute.load.relationship}"
}}]
I'm abusing DistributeLoad in round robin mode to add a cycling id in a
range to the content by sending all relationships to the next processor.
Using JoltTransformRecord, running this spec as a file reference results in
the literal characters ${distribute.load.relationship} being added to the
content. Running the same spec inline results in the correct expectation of
the attribute value being inserted.
Removing the double quotes around the EL expression results in a Java
exception for the spec being invalid. Doing the same with JoltTransformJSON
results in a Java exception for the type conversion failure when
marshalling the object.
On Thu, 24 Aug 2023, 14:06 Matthew Hawkins, wrote:
> Hi Matt,
>
> Many of the deployments I get involved with are not in a Hadoop ecosystem,
> so building in resilience becomes part of the challenge, especially when
> data input is out of your control and often has bad data. In these
> situations, one thing I like doing is a test flow using DuplicateFlowFile
> and DistributeLoad configured as a valve so the main queue isn't
> backlogged. You can then use live data to work on massaging bad input.
> Manually creating schemas for record based processors and JoltTransformJSON
> becomes a large part of the job.
>
> In one such experiment I was either trying to work around GenerateRecord
> not having capability for some data types (e.g. Boolean) and faking it with
> a jolt transform, or I was putting parent key/values back into a content
> after splitting on a sub-arrray. Either way, I ended up with a jolt
> transform inserting a timestamp.
> Current time is pretty useful for downstream systems to understand latency
> between an event and it getting to that system, I often ensure Nifi is
> dropping it into content.
>
> I'm not near my test system to get the exact spec file but will endeavour
> to at the next opportunity.
>
>
>
> On Fri, 18 Aug 2023, 07:13 Matt Burgess, wrote:
>
>> Matthew,
>>
>> What was your second case trying to use ${now():toNumber()} ? My unit
>> test evaluated the expression into an integer and it worked (versus
>> putting the expression in quotes which would make it a string).
>>
>> Regards,
>> Matt
>>
>> On Tue, Aug 15, 2023 at 4:09 AM Matthew Hawkins
>> wrote:
>> >
>> > Hi Matt,
>> >
>> > The tag will be ${firstname}, the spec is correct.
>> > (well, I also get lastname, but it's unimportant)
>> >
>> > Weird thing is this failed similarly with a file based input on the
>> Record
>> > based jolt processor as well on a completely different system.
>> >
>> > Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
>> > source with
>> include-graph,include-media,include-rules,include-sql-reporting
>> >
>> > In the second case I was trying to get ${now():toNumber()} into a json
>> > record using either default or modify-overwrite-beta (tried both). The
>> > following success processor failed on reading schema as the literal
>> > ${now...} was a string not a long and the JVM refused the type cast. If
>> I
>> > insert the transform directly then it still fails with a type cast
>> problem
>> > from generic Object :/ (that was midnight last night, and I tossed my
>> hands
>> > in the air and turned the server off)
>> >
>> > I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
>> > postgresql. Trying to be a good nifi citizen and use record based
>> > processing where possible. Since I don't run Hadoop at home, it's all
>> > manual schemas and using postgres as the perm data store. I'm using
>> json as
>> > mqtt does it intrinsically and I can jolt transform other data into json
>> > form and then store it easily in postgres. Well, that was the idea 😁
>> >
>> > Kr,
>> >
>> > On Sun, 13 Aug 2023, 10:44 Matt Burgess, wrote:
>> >
>> > > Just to follow up, I added a unit test to put EL in a JOLT spec and it
>> > > worked. I noticed you referred to "attrname" in your post but your
>> > > spec refers to "firstname", is that a typo?
>> > >
>> > > Regards,
>> > > Matt
>> > >
>> > > On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess
>> wrote:
>> > > >
>> > > > I added file support to JoltTransformJSON under NIFI-4957 [1], a
>> first
>> > > > glance at the code seems like it should work, but I'll try to
>> > > > reproduce it and follow up, thanks for bringing this to our
>> attention!
>> > > >
>> > > > Regards,
>> > > > Matt
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/NIFI-4957
>> > > >
>> > > > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins <
>> hawko2...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > Hi devs,
>> > > > >
>> > > > > Using 1.23.0 I have a simple transform adding some flowfile
>> attributes
>> > > into
>> > > > > the content.
>> > > > > When putting the spec directly to the processor it works fine. If
>> I
>> > > have
>> > > > > the spec in an external file however it seems to put in the
>
Re: JoltTransformJSON EL when using file
Hi Matt,
Many of the deployments I get involved with are not in a Hadoop ecosystem,
so building in resilience becomes part of the challenge, especially when
data input is out of your control and often has bad data. In these
situations, one thing I like doing is a test flow using DuplicateFlowFile
and DistributeLoad configured as a valve so the main queue isn't
backlogged. You can then use live data to work on massaging bad input.
Manually creating schemas for record based processors and JoltTransformJSON
becomes a large part of the job.
In one such experiment I was either trying to work around GenerateRecord
not having capability for some data types (e.g. Boolean) and faking it with
a jolt transform, or I was putting parent key/values back into a content
after splitting on a sub-arrray. Either way, I ended up with a jolt
transform inserting a timestamp.
Current time is pretty useful for downstream systems to understand latency
between an event and it getting to that system, I often ensure Nifi is
dropping it into content.
I'm not near my test system to get the exact spec file but will endeavour
to at the next opportunity.
On Fri, 18 Aug 2023, 07:13 Matt Burgess, wrote:
> Matthew,
>
> What was your second case trying to use ${now():toNumber()} ? My unit
> test evaluated the expression into an integer and it worked (versus
> putting the expression in quotes which would make it a string).
>
> Regards,
> Matt
>
> On Tue, Aug 15, 2023 at 4:09 AM Matthew Hawkins
> wrote:
> >
> > Hi Matt,
> >
> > The tag will be ${firstname}, the spec is correct.
> > (well, I also get lastname, but it's unimportant)
> >
> > Weird thing is this failed similarly with a file based input on the
> Record
> > based jolt processor as well on a completely different system.
> >
> > Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
> > source with
> include-graph,include-media,include-rules,include-sql-reporting
> >
> > In the second case I was trying to get ${now():toNumber()} into a json
> > record using either default or modify-overwrite-beta (tried both). The
> > following success processor failed on reading schema as the literal
> > ${now...} was a string not a long and the JVM refused the type cast. If I
> > insert the transform directly then it still fails with a type cast
> problem
> > from generic Object :/ (that was midnight last night, and I tossed my
> hands
> > in the air and turned the server off)
> >
> > I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
> > postgresql. Trying to be a good nifi citizen and use record based
> > processing where possible. Since I don't run Hadoop at home, it's all
> > manual schemas and using postgres as the perm data store. I'm using json
> as
> > mqtt does it intrinsically and I can jolt transform other data into json
> > form and then store it easily in postgres. Well, that was the idea 😁
> >
> > Kr,
> >
> > On Sun, 13 Aug 2023, 10:44 Matt Burgess, wrote:
> >
> > > Just to follow up, I added a unit test to put EL in a JOLT spec and it
> > > worked. I noticed you referred to "attrname" in your post but your
> > > spec refers to "firstname", is that a typo?
> > >
> > > Regards,
> > > Matt
> > >
> > > On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess
> wrote:
> > > >
> > > > I added file support to JoltTransformJSON under NIFI-4957 [1], a
> first
> > > > glance at the code seems like it should work, but I'll try to
> > > > reproduce it and follow up, thanks for bringing this to our
> attention!
> > > >
> > > > Regards,
> > > > Matt
> > > >
> > > > [1] https://issues.apache.org/jira/browse/NIFI-4957
> > > >
> > > > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins >
> > > wrote:
> > > > >
> > > > > Hi devs,
> > > > >
> > > > > Using 1.23.0 I have a simple transform adding some flowfile
> attributes
> > > into
> > > > > the content.
> > > > > When putting the spec directly to the processor it works fine. If I
> > > have
> > > > > the spec in an external file however it seems to put in the literal
> > > string
> > > > > ${attrname} - ie the EL didn't process.
> > > > >
> > > > > Have I done something incorrectly or should I be now asking for an
> > > account
> > > > > to log a bug?
> > > > >
> > > > > Sample spec run using Chain DSL:
> > > > >
> > > > > [{
> > > > > "operation": "shift",
> > > > > "spec": {
> > > > > "@": "values"
> > > > > },
> > > > > {
> > > > > "operation": "default",
> > > > > "spec": {
> > > > > "firstname": "${firstname}"
> > > > > }
> > > > > }
> > > > > ]
> > > > >
> > > > > Kind regards,
> > > > > Matthew
> > >
>
Re: JoltTransformJSON EL when using file
Matthew,
What was your second case trying to use ${now():toNumber()} ? My unit
test evaluated the expression into an integer and it worked (versus
putting the expression in quotes which would make it a string).
Regards,
Matt
On Tue, Aug 15, 2023 at 4:09 AM Matthew Hawkins wrote:
>
> Hi Matt,
>
> The tag will be ${firstname}, the spec is correct.
> (well, I also get lastname, but it's unimportant)
>
> Weird thing is this failed similarly with a file based input on the Record
> based jolt processor as well on a completely different system.
>
> Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
> source with include-graph,include-media,include-rules,include-sql-reporting
>
> In the second case I was trying to get ${now():toNumber()} into a json
> record using either default or modify-overwrite-beta (tried both). The
> following success processor failed on reading schema as the literal
> ${now...} was a string not a long and the JVM refused the type cast. If I
> insert the transform directly then it still fails with a type cast problem
> from generic Object :/ (that was midnight last night, and I tossed my hands
> in the air and turned the server off)
>
> I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
> postgresql. Trying to be a good nifi citizen and use record based
> processing where possible. Since I don't run Hadoop at home, it's all
> manual schemas and using postgres as the perm data store. I'm using json as
> mqtt does it intrinsically and I can jolt transform other data into json
> form and then store it easily in postgres. Well, that was the idea 😁
>
> Kr,
>
> On Sun, 13 Aug 2023, 10:44 Matt Burgess, wrote:
>
> > Just to follow up, I added a unit test to put EL in a JOLT spec and it
> > worked. I noticed you referred to "attrname" in your post but your
> > spec refers to "firstname", is that a typo?
> >
> > Regards,
> > Matt
> >
> > On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess wrote:
> > >
> > > I added file support to JoltTransformJSON under NIFI-4957 [1], a first
> > > glance at the code seems like it should work, but I'll try to
> > > reproduce it and follow up, thanks for bringing this to our attention!
> > >
> > > Regards,
> > > Matt
> > >
> > > [1] https://issues.apache.org/jira/browse/NIFI-4957
> > >
> > > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins
> > wrote:
> > > >
> > > > Hi devs,
> > > >
> > > > Using 1.23.0 I have a simple transform adding some flowfile attributes
> > into
> > > > the content.
> > > > When putting the spec directly to the processor it works fine. If I
> > have
> > > > the spec in an external file however it seems to put in the literal
> > string
> > > > ${attrname} - ie the EL didn't process.
> > > >
> > > > Have I done something incorrectly or should I be now asking for an
> > account
> > > > to log a bug?
> > > >
> > > > Sample spec run using Chain DSL:
> > > >
> > > > [{
> > > > "operation": "shift",
> > > > "spec": {
> > > > "@": "values"
> > > > },
> > > > {
> > > > "operation": "default",
> > > > "spec": {
> > > > "firstname": "${firstname}"
> > > > }
> > > > }
> > > > ]
> > > >
> > > > Kind regards,
> > > > Matthew
> >
Re: JoltTransformJSON EL when using file
Hi Matt,
The tag will be ${firstname}, the spec is correct.
(well, I also get lastname, but it's unimportant)
Weird thing is this failed similarly with a file based input on the Record
based jolt processor as well on a completely different system.
Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
source with include-graph,include-media,include-rules,include-sql-reporting
In the second case I was trying to get ${now():toNumber()} into a json
record using either default or modify-overwrite-beta (tried both). The
following success processor failed on reading schema as the literal
${now...} was a string not a long and the JVM refused the type cast. If I
insert the transform directly then it still fails with a type cast problem
from generic Object :/ (that was midnight last night, and I tossed my hands
in the air and turned the server off)
I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
postgresql. Trying to be a good nifi citizen and use record based
processing where possible. Since I don't run Hadoop at home, it's all
manual schemas and using postgres as the perm data store. I'm using json as
mqtt does it intrinsically and I can jolt transform other data into json
form and then store it easily in postgres. Well, that was the idea 😁
Kr,
On Sun, 13 Aug 2023, 10:44 Matt Burgess, wrote:
> Just to follow up, I added a unit test to put EL in a JOLT spec and it
> worked. I noticed you referred to "attrname" in your post but your
> spec refers to "firstname", is that a typo?
>
> Regards,
> Matt
>
> On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess wrote:
> >
> > I added file support to JoltTransformJSON under NIFI-4957 [1], a first
> > glance at the code seems like it should work, but I'll try to
> > reproduce it and follow up, thanks for bringing this to our attention!
> >
> > Regards,
> > Matt
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-4957
> >
> > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins
> wrote:
> > >
> > > Hi devs,
> > >
> > > Using 1.23.0 I have a simple transform adding some flowfile attributes
> into
> > > the content.
> > > When putting the spec directly to the processor it works fine. If I
> have
> > > the spec in an external file however it seems to put in the literal
> string
> > > ${attrname} - ie the EL didn't process.
> > >
> > > Have I done something incorrectly or should I be now asking for an
> account
> > > to log a bug?
> > >
> > > Sample spec run using Chain DSL:
> > >
> > > [{
> > > "operation": "shift",
> > > "spec": {
> > > "@": "values"
> > > },
> > > {
> > > "operation": "default",
> > > "spec": {
> > > "firstname": "${firstname}"
> > > }
> > > }
> > > ]
> > >
> > > Kind regards,
> > > Matthew
>
Re: JoltTransformJSON EL when using file
Just to follow up, I added a unit test to put EL in a JOLT spec and it
worked. I noticed you referred to "attrname" in your post but your
spec refers to "firstname", is that a typo?
Regards,
Matt
On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess wrote:
>
> I added file support to JoltTransformJSON under NIFI-4957 [1], a first
> glance at the code seems like it should work, but I'll try to
> reproduce it and follow up, thanks for bringing this to our attention!
>
> Regards,
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-4957
>
> On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins wrote:
> >
> > Hi devs,
> >
> > Using 1.23.0 I have a simple transform adding some flowfile attributes into
> > the content.
> > When putting the spec directly to the processor it works fine. If I have
> > the spec in an external file however it seems to put in the literal string
> > ${attrname} - ie the EL didn't process.
> >
> > Have I done something incorrectly or should I be now asking for an account
> > to log a bug?
> >
> > Sample spec run using Chain DSL:
> >
> > [{
> > "operation": "shift",
> > "spec": {
> > "@": "values"
> > },
> > {
> > "operation": "default",
> > "spec": {
> > "firstname": "${firstname}"
> > }
> > }
> > ]
> >
> > Kind regards,
> > Matthew
Re: JoltTransformJSON EL when using file
I added file support to JoltTransformJSON under NIFI-4957 [1], a first
glance at the code seems like it should work, but I'll try to
reproduce it and follow up, thanks for bringing this to our attention!
Regards,
Matt
[1] https://issues.apache.org/jira/browse/NIFI-4957
On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins wrote:
>
> Hi devs,
>
> Using 1.23.0 I have a simple transform adding some flowfile attributes into
> the content.
> When putting the spec directly to the processor it works fine. If I have
> the spec in an external file however it seems to put in the literal string
> ${attrname} - ie the EL didn't process.
>
> Have I done something incorrectly or should I be now asking for an account
> to log a bug?
>
> Sample spec run using Chain DSL:
>
> [{
> "operation": "shift",
> "spec": {
> "@": "values"
> },
> {
> "operation": "default",
> "spec": {
> "firstname": "${firstname}"
> }
> }
> ]
>
> Kind regards,
> Matthew
JoltTransformJSON EL when using file
Hi devs,
Using 1.23.0 I have a simple transform adding some flowfile attributes into
the content.
When putting the spec directly to the processor it works fine. If I have
the spec in an external file however it seems to put in the literal string
${attrname} - ie the EL didn't process.
Have I done something incorrectly or should I be now asking for an account
to log a bug?
Sample spec run using Chain DSL:
[{
"operation": "shift",
"spec": {
"@": "values"
},
{
"operation": "default",
"spec": {
"firstname": "${firstname}"
}
}
]
Kind regards,
Matthew
