Re: AttributeRollingWindow 1.9.2 Value to Track UUID possible ?
Steve,
In a flow-based system like NiFi it can be difficult to tell when
there are "no more events with the same key". However if you have a
notion of the max amount of time where events would appear, you might
be able to use MergeRecord or MergeContent with a Correlation
Attribute Name of "eventUuid" with a Min and Max Bin Age of that time
interval (and make sure the Max Records is set higher than you'd ever
expect for the number of events in the system over that time). When
MergeRecord emits a flow file, you are assured that all the records
have the same eventUuid, and then you might use QueryRecord to execute
SQL on the records which allows you to do some aggregations of the
measures.
This won't be a perfect system as MergeRecord/Content only has the
number of bins (unique eventUuids, e.g.) you specify, so it's possible
that a single flow file does not contain all events for that UUID over
that time period. You may be able to use an incremental pipeline like
MergeRecord -> PartitionRecord -> MergeRecord -> QueryRecord. If
there's a timestamp field in each record you could have something like
"MIN(timestamp) as min_timestamp, MAX(timestamp) as max_timestamp,
AVG(myEventMeasure) GROUP BY eventUuid" to do the aggregates. It might
not be an exact window size but "max_timestamp - min_timestamp" would
give you the width of the window for the records in each flow file.
Queries over streams is an active research area, the folks working on
Apache Calcite are doing some really cool things, and we've got
Calcite powering QueryRecord under the hood, so I'm hoping we can
continue to enjoy the fruits of their labor :)
Regards,
Matt
On Fri, Aug 23, 2019 at 4:03 PM Steve Robert
wrote:
>
> Hi Brian ,
>
> thank you for your reply.
> To be precise I studied the possibility to open a window on an attribute
> value and close it after an user-defined time and if there are no more
> events with the same key.
> In flink side lot operator send Event with SUCCESS OR FAIL . operator is
> running in paralleles.
> I wanted to be able to do a aggregation to determine if there was an error or
> not during a life cycle iteration. each life cycle is unique and associated
> with a UUID
>
> Le ven. 23 août 2019 à 21:00, Bryan Bende a écrit :
>>
>> I think AttributeRollingWindow was made to aggregate a numeric
>> attribute. So for example, if all flows files had an attribute called
>> "amount" that was an integer, then you could say Value to Track =
>> ${amount} and it would aggregate those values over window.
>>
>> In your case the attribute you have is not the value to aggregate, its
>> a key. The value in your case is a constant of 1 where you want to
>> increment the aggregated value by 1 whenever you see the key.
>>
>> I guess it depends what you want to do with the aggregated value, but
>> your situation seems closer to a counter which could be tracked with
>> UpdateCounter setting the Counter Name to ${eventUuid} and the Delta
>> to 1, but then I don't know exactly how you would use this counter
>> later.
>>
>> On Fri, Aug 23, 2019 at 1:59 PM Steve Robert
>> wrote:
>> >
>> > Hi Guys ,
>> >
>> > I apologize in advance if my question seems trivial but I am new on Nifi.
>> > I'm studying Nifi for an integration with Flink that I'm used to.
>> >
>> > Since flink I send events using Site-to-Site to Nifi.
>> > the Flowfile have attribute
>> > "eventUuid":"97f82c90-0782-4aab-8850-56ee60b0b73d"
>> >
>> > I would like to do an aggregation on events based on the value of this
>> > attribute,
>> >
>> > so I look at the documentation of AttributeRollingWindow.
>> >
>> > I understood by looking at the errors that this operator does not accept
>> > strings and therefore can not track a value on String
>> > Value to Track : ${eventUuid} will throw NumberFormatExceptions.
>> >
>> > Value to Track : ${eventUuid:toNumber()} will not work because UUID 128
>> > bit can't be convert to an 32 bytes number;
>> >
>> > Is this a limitation or a bad approach on my part? I can not use an event
>> > on the timestamp here because this UUID is generated at the beginning of
>> > the workflow
>> > Thank a lot for your help
>> > Steve
>> >
>> >
RE: Installing nifi-registry as service
If I run "sudo /etc/init.d/nifi-registry start", it will start the service
correctly. Thanks a lot.
Best,
Qiong
-Original Message-
From: Qiong Wei
Sent: Friday, August 23, 2019 1:02 PM
To: users@nifi.apache.org
Subject: RE: Installing nifi-registry as service
Yeah, I checked the code, and run the install command again. The SVC_FILE was
created under /etc/init.d, however, it complains unit nifi-registry.service not
found when starting the service.
Thanks,
Qiong
-Original Message-
From: Bryan Bende
Sent: Friday, August 23, 2019 11:54 AM
To: users@nifi.apache.org
Subject: Re: Installing nifi-registry as service
I haven't had time to try it out, but the code for the install part is in
nifi-registry.sh here:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi-registry%2Fblob%2Fmaster%2Fnifi-registry-core%2Fnifi-registry-resources%2Fsrc%2Fmain%2Fresources%2Fbin%2Fnifi-registry.sh%23L170-L248data=02%7C01%7Cqiwe%40microsoft.com%7C95a00420b20241c995b808d72804c165%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637021873199151809sdata=fGu%2Bhq3xhF6YRN9j5JLk4itWJY7%2Feb%2FYSS2Jg%2BCsPi0%3Dreserved=0
It should be creating the service file in /etc/init.d based on these lines:
initd_dir='/etc/init.d'
SVC_FILE="${initd_dir}/${SVC_NAME}"
And then it calls cat to SVC_FILE with inline content.
-Bryan
On Fri, Aug 23, 2019 at 1:41 PM Qiong Wei wrote:
>
> Hi Bryan,
>
> Thanks a lot for your response.
>
> After the install command, it shows service nifi-registry installed, but
> actually I cannot find nifi-registry under /etc/init.d. The OS I'm using is
> Ubuntu 18.04-LTS.
>
> Thanks,
> Qiong
>
> -Original Message-
> From: Bryan Bende
> Sent: Friday, August 23, 2019 9:57 AM
> To: users@nifi.apache.org
> Subject: Re: Installing nifi-registry as service
>
> Hello,
>
> Does it look like the service is correctly installed in
> /etc/init.d/nifi-registry ?
>
> Also, what operating system?
>
> -Bryan
>
> On Fri, Aug 23, 2019 at 12:26 PM Qiong Wei wrote:
> >
> > Hi All,
> >
> >
> >
> > I’m new to NiFi-Registry, now trying to install NiFi-Registry as a service,
> > here are the command I run:
> >
> >
> >
> > sudo ./opt/nifi-registry/bin/nifi-registry.sh install
> >
> > Output: Service nifi-registry installed
> >
> > sudo service nifi-registry start
> >
> > Output: Failed to start nifi-registry.service: Unit nifi-registry.service
> > not found.
> >
> >
> >
> > Do you guys have any idea about this issue? Thanks a lot.
> >
> >
> >
> > Best,
> >
> > Qiong
Re: AttributeRollingWindow 1.9.2 Value to Track UUID possible ?
Hi Brian ,
thank you for your reply.
To be precise I studied the possibility to open a window on an attribute
value and close it after an user-defined time and if there are no more
events with the same key.
In flink side lot operator send Event with SUCCESS OR FAIL . operator is
running in paralleles.
I wanted to be able to do a aggregation to determine if there was an error
or not during a life cycle iteration. each life cycle is unique and
associated with a UUID
Le ven. 23 août 2019 à 21:00, Bryan Bende a écrit :
> I think AttributeRollingWindow was made to aggregate a numeric
> attribute. So for example, if all flows files had an attribute called
> "amount" that was an integer, then you could say Value to Track =
> ${amount} and it would aggregate those values over window.
>
> In your case the attribute you have is not the value to aggregate, its
> a key. The value in your case is a constant of 1 where you want to
> increment the aggregated value by 1 whenever you see the key.
>
> I guess it depends what you want to do with the aggregated value, but
> your situation seems closer to a counter which could be tracked with
> UpdateCounter setting the Counter Name to ${eventUuid} and the Delta
> to 1, but then I don't know exactly how you would use this counter
> later.
>
> On Fri, Aug 23, 2019 at 1:59 PM Steve Robert
> wrote:
> >
> > Hi Guys ,
> >
> > I apologize in advance if my question seems trivial but I am new on Nifi.
> > I'm studying Nifi for an integration with Flink that I'm used to.
> >
> > Since flink I send events using Site-to-Site to Nifi.
> > the Flowfile have attribute
> "eventUuid":"97f82c90-0782-4aab-8850-56ee60b0b73d"
> >
> > I would like to do an aggregation on events based on the value of this
> attribute,
> >
> > so I look at the documentation of AttributeRollingWindow.
> >
> > I understood by looking at the errors that this operator does not accept
> strings and therefore can not track a value on String
> > Value to Track : ${eventUuid} will throw NumberFormatExceptions.
> >
> > Value to Track : ${eventUuid:toNumber()} will not work because UUID
> 128 bit can't be convert to an 32 bytes number;
> >
> > Is this a limitation or a bad approach on my part? I can not use an
> event on the timestamp here because this UUID is generated at the beginning
> of the workflow
> > Thank a lot for your help
> > Steve
> >
> >
>
RE: Installing nifi-registry as service
Yeah, I checked the code, and run the install command again. The SVC_FILE was
created under /etc/init.d, however, it complains unit nifi-registry.service not
found when starting the service.
Thanks,
Qiong
-Original Message-
From: Bryan Bende
Sent: Friday, August 23, 2019 11:54 AM
To: users@nifi.apache.org
Subject: Re: Installing nifi-registry as service
I haven't had time to try it out, but the code for the install part is in
nifi-registry.sh here:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi-registry%2Fblob%2Fmaster%2Fnifi-registry-core%2Fnifi-registry-resources%2Fsrc%2Fmain%2Fresources%2Fbin%2Fnifi-registry.sh%23L170-L248data=02%7C01%7Cqiwe%40microsoft.com%7Cd238debc72044d71f3d808d727fb4a99%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637021832550575713sdata=kVEcAs9SLL%2FmWeMa6VvEGt3nprU58llYnFGmDGPSLnw%3Dreserved=0
It should be creating the service file in /etc/init.d based on these lines:
initd_dir='/etc/init.d'
SVC_FILE="${initd_dir}/${SVC_NAME}"
And then it calls cat to SVC_FILE with inline content.
-Bryan
On Fri, Aug 23, 2019 at 1:41 PM Qiong Wei wrote:
>
> Hi Bryan,
>
> Thanks a lot for your response.
>
> After the install command, it shows service nifi-registry installed, but
> actually I cannot find nifi-registry under /etc/init.d. The OS I'm using is
> Ubuntu 18.04-LTS.
>
> Thanks,
> Qiong
>
> -Original Message-
> From: Bryan Bende
> Sent: Friday, August 23, 2019 9:57 AM
> To: users@nifi.apache.org
> Subject: Re: Installing nifi-registry as service
>
> Hello,
>
> Does it look like the service is correctly installed in
> /etc/init.d/nifi-registry ?
>
> Also, what operating system?
>
> -Bryan
>
> On Fri, Aug 23, 2019 at 12:26 PM Qiong Wei wrote:
> >
> > Hi All,
> >
> >
> >
> > I’m new to NiFi-Registry, now trying to install NiFi-Registry as a service,
> > here are the command I run:
> >
> >
> >
> > sudo ./opt/nifi-registry/bin/nifi-registry.sh install
> >
> > Output: Service nifi-registry installed
> >
> > sudo service nifi-registry start
> >
> > Output: Failed to start nifi-registry.service: Unit nifi-registry.service
> > not found.
> >
> >
> >
> > Do you guys have any idea about this issue? Thanks a lot.
> >
> >
> >
> > Best,
> >
> > Qiong
Re: AttributeRollingWindow 1.9.2 Value to Track UUID possible ?
I think AttributeRollingWindow was made to aggregate a numeric
attribute. So for example, if all flows files had an attribute called
"amount" that was an integer, then you could say Value to Track =
${amount} and it would aggregate those values over window.
In your case the attribute you have is not the value to aggregate, its
a key. The value in your case is a constant of 1 where you want to
increment the aggregated value by 1 whenever you see the key.
I guess it depends what you want to do with the aggregated value, but
your situation seems closer to a counter which could be tracked with
UpdateCounter setting the Counter Name to ${eventUuid} and the Delta
to 1, but then I don't know exactly how you would use this counter
later.
On Fri, Aug 23, 2019 at 1:59 PM Steve Robert
wrote:
>
> Hi Guys ,
>
> I apologize in advance if my question seems trivial but I am new on Nifi.
> I'm studying Nifi for an integration with Flink that I'm used to.
>
> Since flink I send events using Site-to-Site to Nifi.
> the Flowfile have attribute "eventUuid":"97f82c90-0782-4aab-8850-56ee60b0b73d"
>
> I would like to do an aggregation on events based on the value of this
> attribute,
>
> so I look at the documentation of AttributeRollingWindow.
>
> I understood by looking at the errors that this operator does not accept
> strings and therefore can not track a value on String
> Value to Track : ${eventUuid} will throw NumberFormatExceptions.
>
> Value to Track : ${eventUuid:toNumber()} will not work because UUID 128 bit
> can't be convert to an 32 bytes number;
>
> Is this a limitation or a bad approach on my part? I can not use an event on
> the timestamp here because this UUID is generated at the beginning of the
> workflow
> Thank a lot for your help
> Steve
>
>
Re: Installing nifi-registry as service
I haven't had time to try it out, but the code for the install part is
in nifi-registry.sh here:
https://github.com/apache/nifi-registry/blob/master/nifi-registry-core/nifi-registry-resources/src/main/resources/bin/nifi-registry.sh#L170-L248
It should be creating the service file in /etc/init.d based on these lines:
initd_dir='/etc/init.d'
SVC_FILE="${initd_dir}/${SVC_NAME}"
And then it calls cat to SVC_FILE with inline content.
-Bryan
On Fri, Aug 23, 2019 at 1:41 PM Qiong Wei wrote:
>
> Hi Bryan,
>
> Thanks a lot for your response.
>
> After the install command, it shows service nifi-registry installed, but
> actually I cannot find nifi-registry under /etc/init.d. The OS I'm using is
> Ubuntu 18.04-LTS.
>
> Thanks,
> Qiong
>
> -Original Message-
> From: Bryan Bende
> Sent: Friday, August 23, 2019 9:57 AM
> To: users@nifi.apache.org
> Subject: Re: Installing nifi-registry as service
>
> Hello,
>
> Does it look like the service is correctly installed in
> /etc/init.d/nifi-registry ?
>
> Also, what operating system?
>
> -Bryan
>
> On Fri, Aug 23, 2019 at 12:26 PM Qiong Wei wrote:
> >
> > Hi All,
> >
> >
> >
> > I’m new to NiFi-Registry, now trying to install NiFi-Registry as a service,
> > here are the command I run:
> >
> >
> >
> > sudo ./opt/nifi-registry/bin/nifi-registry.sh install
> >
> > Output: Service nifi-registry installed
> >
> > sudo service nifi-registry start
> >
> > Output: Failed to start nifi-registry.service: Unit nifi-registry.service
> > not found.
> >
> >
> >
> > Do you guys have any idea about this issue? Thanks a lot.
> >
> >
> >
> > Best,
> >
> > Qiong
AttributeRollingWindow 1.9.2 Value to Track UUID possible ?
Hi Guys ,
I apologize in advance if my question seems trivial but I am new on Nifi.
I'm studying Nifi for an integration with Flink that I'm used to.
Since flink I send events using Site-to-Site to Nifi.
the Flowfile have attribute "eventUuid":"
97f82c90-0782-4aab-8850-56ee60b0b73d"
I would like to do an aggregation on events based on the value of this
attribute,
so I look at the documentation of AttributeRollingWindow.
I understood by looking at the errors that this operator does not accept
strings and therefore can not track a value on String
*Value to Track : ${eventUuid} will throw NumberFormatExceptions.*
*Value to Track : ${eventUuid:toNumber()} will not work because UUID 128
bit can't be convert to an 32 bytes number;*
Is this a limitation or a bad approach on my part? I can not use an event
on the timestamp here because this UUID is generated at the beginning of
the workflow
Thank a lot for your help
Steve
RE: Installing nifi-registry as service
Hi Bryan, Thanks a lot for your response. After the install command, it shows service nifi-registry installed, but actually I cannot find nifi-registry under /etc/init.d. The OS I'm using is Ubuntu 18.04-LTS. Thanks, Qiong -Original Message- From: Bryan Bende Sent: Friday, August 23, 2019 9:57 AM To: users@nifi.apache.org Subject: Re: Installing nifi-registry as service Hello, Does it look like the service is correctly installed in /etc/init.d/nifi-registry ? Also, what operating system? -Bryan On Fri, Aug 23, 2019 at 12:26 PM Qiong Wei wrote: > > Hi All, > > > > I’m new to NiFi-Registry, now trying to install NiFi-Registry as a service, > here are the command I run: > > > > sudo ./opt/nifi-registry/bin/nifi-registry.sh install > > Output: Service nifi-registry installed > > sudo service nifi-registry start > > Output: Failed to start nifi-registry.service: Unit nifi-registry.service not > found. > > > > Do you guys have any idea about this issue? Thanks a lot. > > > > Best, > > Qiong
Re: Installing nifi-registry as service
Hello, Does it look like the service is correctly installed in /etc/init.d/nifi-registry ? Also, what operating system? -Bryan On Fri, Aug 23, 2019 at 12:26 PM Qiong Wei wrote: > > Hi All, > > > > I’m new to NiFi-Registry, now trying to install NiFi-Registry as a service, > here are the command I run: > > > > sudo ./opt/nifi-registry/bin/nifi-registry.sh install > > Output: Service nifi-registry installed > > sudo service nifi-registry start > > Output: Failed to start nifi-registry.service: Unit nifi-registry.service not > found. > > > > Do you guys have any idea about this issue? Thanks a lot. > > > > Best, > > Qiong
Installing nifi-registry as service
Hi All, I'm new to NiFi-Registry, now trying to install NiFi-Registry as a service, here are the command I run: 1. sudo ./opt/nifi-registry/bin/nifi-registry.sh install Output: Service nifi-registry installed 1. sudo service nifi-registry start Output: Failed to start nifi-registry.service: Unit nifi-registry.service not found. Do you guys have any idea about this issue? Thanks a lot. Best, Qiong
JVM Proxy in Bootstrap
Hello I have attempted to set authenticated proxy configuration on NIfi start-up by setting properties in bootstrap with no success. Is it possible? Currently getting 407 due to authentication errors. The configs do not seem to be working java.arg.19=-Dhttp.proxyHost=proxy java.arg.20=-Dhttp.proxyPort= java.arg.23=-Dhttp.proxyUser=myuser java.arg.24=-Dhttp.proxyPassword=mypassword #Java version 8u111+ java.arg.27=-Djdk.http.auth.tunneling.disabledSchemes= java.arg.28=-Djdk.http.auth.proxying.disabledSchemes= Just want to double-check with the community before banging my head against the wall for another day. Thank you Noe
